I just off the phone with an old High School chum.  We got to talking about Bottom-Up Thinking as a way to set direction when a situation is unclear or changing too quickly. It was in the context of career, so I gave a lot of unsolicited advice … ‘natch.

However, it got me to thinking about all the messed up, and the few really great technical projects I’ve worked on.  I think one of the most important things to do for every phase of any project is to decide if it is time to work Top-Down or Bottom-Up.  I think for Software Design one must always work Bottom Up:

1. To understand requirements clearly, it’s best to mock up, or prototype software to show stakeholders.  Paper or a whiteboard is the place to start.  It should be iterative with the prototypes getting fancier and the requirements firming up.
2. Already know your requirements?  Think again.  Go back to step #1.
3. Prototypes should always run and always be accessible, you never know who will get shown the proto and come up with the game changing idea.
4. Think you know what will change the game?  Think again.  Go back to step #3.
5. As soon as the ideas in the product firm up and the prototype represents them, it is time for User Interface (UX) testing.  It is critical to design with the customer in mind asap.
6. Think that silly users are not going to be able to understand the lofty ideas behind your story board, powerpoint, or web prototype?  Think again.  They get it better than you.  After all, if they don’t understand it, why would they buy it?  Back to step #5 with you.
7. Prototypes become commercial software through a process of Re-factoring.  Re-factoring changes incomplete, flaky or non-mission-critical components with stuff that is ready for prime time.  The point is to be iterative, always have the software in working order, and develop tests+specs as you go along.
8. Is the prototype technology too week and not suitable for heavy duty commercial use?  Do you need some new language, tool, technique or paradigm to get your design implemented?  Think again.  All kinds of cheap technology powers critical systems the world over.  Google started with a freaking Perl script.  Elitism doesn’t make money.  You got it, back to step #7.
9. If you’ve gotten to this point.  Ship It!
10. Are you wondering if this is really ready for primetime?  Maybe we need to make it run on a cloud, or have a real sysadmin, or advertise, or, or, I don’t know, do something else.  Look, software is never done.  It is much better to fail fast, and still be able to get back up, than sit around waiting for stars to line up.  The real learning doesn’t start until the product ships.  Back to step #9, and no excuses this time

Some folks today will call this Agile.  But its been around a long time in engineering.  Before the Internet, back when Pyramids were being built.  Engineers noodle around too much, have lots of ideas, and tend to work top down.  I am one, I know my type.  The truth is, it really is a jungle out there.  Work Bottom Up, in lots of little steps, up to survive Software Design.

I think groups going Agile need the one sheet Principles Behind The Agile Manifesto.  Short and clear, it has a good chance of galvanizing change.  In true contrarian manner though, I’d like to point out why blindly following these principles is a bad idea.

1. Our highest priority is to satisfy the customer
   through early and continuous delivery
   of valuable software.

   Most teams need to focus on delivering must faster with smaller iterations.  However, customers are not always satisfied just by valuable software.  We need to make sure we look at every aspect of what customers want.  Make sure you have an analytic method to listen to customers!  I’m certain you’ll be surprised.

2. Welcome changing requirements, even late in
   development. Agile processes harness change for
   the customer’s competitive advantage.

   Yup, there’s no point in complaining about late requirements.  They happen, and they are usually important.  However, it is critical to note that the vast majority of the quality, cost and delay problems in software stem directly from late requirements.  Get involved early to help make requirements development, from a business perspective, successful.

3. Deliver working software frequently, from a
   couple of weeks to a couple of months, with a
   preference to the shorter timescale.

   Yup, the more often a complete release cycle is done, the better off everyone is.  However, customers can be irritated and confused by software that constantly changes.  If so, find a way to make releases to a select group of “mavens” that appreciate the chance to use software before public release.

4. Business people and developers must work
   together daily throughout the project.

   Duh.  However, having this ideal and getting it to fly is something all together different.  Talented project management, a good understanding of the cultures and corporate cultures involved, and solid requirements processes are needed to survive this.  Ideas are one thing, knowing what actions to take is another.  If you’re not succeeding here, don’t wait.  Get help.

5. Build projects around motivated individuals.
   Give them the environment and support they need,
   and trust them to get the job done.

   I would just like to point out however that human beings are complicated.  If we don’t have goals and measures, subjectivity will lead to judgment, and then to emotionalism.  Not good.  We all like to think we’re above this.  We aren’t.  Get a performance methodology in place and use it.  I like HP’s Management By Objective.  In the long run, it will create the fairness to allow trust to happen.

6. The most efficient and effective method of
   conveying information to and within a development
   team is face-to-face conversation.

   This is a red herring.  Psychologists continue to discover all the ways humans use body language, gestures, micro-facial gestures, etc. to communicate while we are face to face.  That said, I’ve had super effective technical cooperation with people I’ve never seen.  Mind you, I don’t have a bullet proof recipe for virtual teams, but I’ve had them work wonderfully.

7. Working software is the primary measure of progress.

   Yes, yes, we’re programmers after all, right?  Wrong.  When we ship a product, what we really have is a relationship with a customer.  They give us money (hopefully) and we give them what they want.  It may be software, support or back rubs.  Maybe even prestige!  You know about the customer always being right, right?  As programmers we need to make sure we are getting measurable feedback that we are on track to a happy customer.  Think analytics.

8. Agile processes promote sustainable development.
   The sponsors, developers, and users should be able
   to maintain a constant pace indefinitely.

   In theory this is true.  It will rarely be true.  Reality can be crazy.  I’m not saying that being out of balance is a good thing.  However, if we accept requirements change at the last minute, we need to be ready to work quickly.  Life happens, sometimes fast.

9. Continuous attention to technical excellence
   and good design enhances agility.

   So often when deadlines hit the emphasis on doing the best job possible on the software slides out the window. However, engineers tend to noodle over stuff.  Do not get stuck looking for the “right” design. That’s what smaller iterations are helpful in doing, refactoring towards better designs.

10. Simplicity–the art of maximizing the amount
    of work not done–is essential.

    Just realize, doing something complicated is easy.  Simple is hard.  There may be several iterations of complicated before everything gets boiled down to simple.  However, always ship the simplest and most usable interface, no exceptions. If you’ve got tests, you can always clean up what’s behind the curtain.

11. The best architectures, requirements, and designs
    emerge from self-organizing teams.

    Yup, as long as the team has direction and no-one is actively trying to kill another team member.  There is nothing toxic about roles and responsibilities handed out by leadership.  The best teams self-organize, others might need direct help and facilitation.  Just be nice about it.

12. At regular intervals, the team reflects on how
    to become more effective, then tunes and adjusts
    its behavior accordingly.
    
    Continuous improvement is absolutely critical for everyone in the software field.  Fall behind and we are penalizing ourselves and our teams.  It’s easy to calculate the cost of everyone in the same room, and stop doing postmortems.  What is the cost of having poorly trained and performing folks adding to your source code base?

Okay, so I agree with this last principle without reservations.  With Agility, like any fad, there is an underlying truth to the doctrine.  However, blindly following a doctrine, even a good one like Agility, is a lemming’s plan.

Isms in my opinion are not good. A person should not believe in an ism – he should believe in himself. I quote John Lennon: “I don’t believe in Beatles – I just believe in me”. A good point there. Of course, he was the Walrus. I could be the Walrus – I’d still have to bum rides off of people.  — Ferris Bueller

• Principles behind the Agile Manifesto
• Deal With It: Retrospectives Are Postmortems
• Ferris Bueller’s Day Off
• A Firm Agile-ism Debunking from a Google Engineer
  

One of the latest fads in programming is Agile Software Development.  Most fads, after a while, take on a disturbing quality as the bandwagon swells to the teetering point. We’ve gotten to that disturbing evolutionary point with Agile Programming.  Talks and material from Linda Rising are the clearest indicators I’ve seen that Agile Programming’s days as a fad are numbered.

Don’t get me wrong, there is lots of potentially useful practices that come from the agile programming community.  My very first software mentor (way back when) gave me the solid advice to work in small iterations and always have my software in working order with a set of tests. Agile practices like Scrum and XP have lots of good ideas for how to take this kind of small iterative thinking and apply it to entire teams.  Software systems like Ruby on Rails provide domain specific tools that cater to agile development. Good stuff on the whole.

However pundits and critics alike can be horrible advisers.  Linda Rising does for Agile Programming what the Dancing Wu Li Masters did for Quantum Physics: water down science with entertaining but ridiculous parallels and wild extrapolations. Here are some highlights I took from my viewing of Perfection Is An Unrealistic Goal:

1. We are hard-wired to lie, even to ourselves.
2. We are hopelessly inept at scheduling and we will never improve.
3. If we knew what people thought about us, we’d never get out of bed.

The thesis goes like this:  We are social animals so we will subvert the truth to be accepted by others, even subconsciously.  Therefore we will offer the most optimistic schedule always.  This is a characteristic we all have because it has been inherited from our ape ancestors.  Rubbish…

Software schedules are hard. There is more than one way to do anything in software.  The result is that creating software, even low level coding, is more a design process than a construction process. Construction can be mapped out, metrics developed and predictions made.  Design processes are not deterministic like production processes.  The result is that humans don’t have the data to predict, so we estimate, or guess, sometimes with data, and sometimes without.  No new news here!

My absolute favorite is Linda Rising on Collaboration, Bonobos and the Brain:

Actually chimpanzees are also promiscuous, but the difference is that chimpanzees don’t enjoy pairing, it’s for procreation only. The power struggles in the chimpanzee cultures are about bananas, the food, and about sex. Usually only the alpha male or his supporters are the ones who are mating with the available females. The struggles that go on there are power struggle. Whereas the bonobos don’t have those power issues and they resolve them with sex, which is just a ritual.

…

My point is that bonobos have found a ritual for resolving conflicts and for working together and sharing limited resources. What we found in the Agile community are a lot of rituals or practices that connect with our inner bonobo. They are not really part of the chimpanzee culture at all. We are looking at small teams that communicate, that work well together, certainly the bonobo culture represents an attempt for a community, to all to communicate with each other; it’s not just the alpha male who gets to mate, but everyone mates with everyone.

How Linda knows that chimps don’t enjoy sex, but bonobos do is beyond me. I see her meaning though.  We programmers need to look more like the happy bonobo pictured above: unreasoning, rested, promiscuous folks who don’t have to work too much. Hey, sign me up!

If you’ve read my discussion of using analytics in design project management, you might be asking how anyone can plan for a design project.  After all, if I claim that design is a non-deterministic process, how could it possibly be planned for? Design would then be a pretty bad business risk, right? Short answer: yup.

The long answer is a little more hopeful, but we have to change the way we plan for design processes if we are to grapple with the uncertainty (and hence risk) that goes along with design.  If you’ve ever studied personal productivity, or (gasp) ever procrastinated on a big project, you may have learned the cardinal rule of getting things done: do the big stuff first.

The same rule works well for design.  Don’t sweat the small stuff.  Use the time for planning as a way to discover what the big problems are, where the big value is, where the gaps in information or technology lie. When the project starts, make sure folks stay focused on this big stuff first. With the big stuff out of the way, you’ll be on track.

Where people get messed up is with big projects.  With the stakes and cost high, Microsoft Project comes out and people start writing down whatever they can think of that needs to get accomplished during the design process.  Then the inevitable happens, in the push to show progress (percent complete, usually) people look for items they can get done and check off the list.  These end up being the small easy items. Sigh…

The result is that the big issues don’t get worked on until the end.  By then, time has expired doing easy stuff that will usually have to be re-done once the big decisions finally get made.  Big design projects typically die this way, a billion mosquitoes irritate the project to near-death, and then at the end, a giant tiger leaps out of the jungle and eats everyone alive.

A better path is to worry about the tigers up front and ignore the mosquitos.  In fact, do yourself a favor, don’t even write those pesky little tasks down in a project plan.  Save the planning just for the big problems, big value opportunities, big information gaps and big technological issues. Put the little items that come up on a list (I call it a backlog list) so you *will* forget about them and stay focused on the big stuff. Agile software development folks will see where I am going with this kind of talk.  But I really just like to think of it as dodging the mosquitoes to get the big game at the end of the day.

I alluded to design being a risky business proposition at the beginning of this article.  The truth is: betting ones bread and butter on a non-deterministic process has to be risky. Design is the creation of intellecutal property.  It is grappling with the unknown.  Design is big game hunting.  When big game hunting, don’t grapple with mosquitos, just focus on the tiger, and maybe, just maybe, you’ll be the big game hunter you planned on being as opposed to a snack for a big cat.