Agile development and the Software Researcher

October 3, 2006 at 10:42 am

A while back I came across this post about agile.


I originally was going to write a comment about that post, but I think the commenters have done a good job at that. I will point out that the environment described in the post is a lot like many of the internet boom startups, few of which are around right now.


But in writing some thoughts down, I realized that I was really writing about why I found agile to be interesting. So that’s what this post is about.


I’ve worked on a ton of projects, some short, some long, some small teams, some big teams. But fundamentally, writing software is an exercise in:



  • Making progress in the face of imperfect information

and



  • Managing change

These shouldn’t be a surprise to anybody. When we start coding, we always have imperfect information. We don’t know if our design will really meet the customer’s needs, we don’t know how long it’s going to take us to write, we don’t know if our architecture is feasible to build, we don’t know if it will perform well enough,  etc. And things will change along the way. Market conditions will change. Customers will modify their priorities. New technology will become available. People will join or leave the team.


There’s a name for trying do something in that sort of environment, and it’s not “engineering”.


It’s research.


Applied research, to be specific.


And doing research requires a different approach than engineering.


Way back in 1961, President Kennedy made his famous “moon speech“.  At that time, the only US astronaut was Alan Shepard, who had flown a 15 minute sub-orbital flight. The ability to send people to the moon seemed out of reach. But the leaders at NASA came up with a plan – they decided on three different programs: Mercury, Gemini, and Apollo. Each program had a theme and a set of goals, and each flight had a set of smaller goals.


Sometimes things worked, sometimes they didn’t. After each flight, they looked at what had happened, decided how to adapt, and went on to the next flight. And over the span of 8 years, they accomplished their goal. All by following the “design a little, build a little, fly a little” approach.


After Apollo, NASA decided to build the shuttle. They did a lot of design up front, made a lot of promises (both schedule and capabilities) and then started building. They ran into lots of difficulties with the main engines (ultimately solved), and the thermal protection system (not really solved). Ultimately, they finished the vehicle, but it took years longer than they expected and ultimately didn’t do what they had designed it to do or what they needed it to do. And it has very high maintenance costs.


The analogy to software development projects should be obvious. Shuttle is like most big software projects – lots of planning up front, lots of promises, followed by delays as problems are addressed, and then the ultimate release of something that doesn’t measure up to the original idea. We convince ourselves that “this time it will be different”, despite the fact that the same thing has happened every time we take this approach.


Incremental development accepts the realities of the environment, and takes a different approach. You know that you don’t know enough to finish the project and you know that things are going to change, so you bite off a manageable portion – say a month’s worth of work – do enough design, write some code, and then see how well it works.


And after that month, you modify your plan based on what you learned.


Design a little
Code a little
Fly a little


That’s what agile means to me.

Tags: