There was a post on an internal alias about moving a team that has not been creating any developer-written tests to one that does TDD. I wrote a reply that I think may be of more general interest…

Developers are fluent in code. I think any time you are talking to developers about engineering practices, you need to be showing them code. I think showing TDD workflow is very important. I also have stopped using the term “Unit test” as it means 45 different things before 10am.

I’m a big fan of selling the problem rather than the solution. Here are some problems that I think resonate with the developers I know:

Time Savings

Nearly every group with existing code has a section of code that either is or has been a bug factory, and most developers can make a decent estimate of what parts of new code are “tricky”. If you use TDD (or just write tests for) that set of code, you can save yourself a huge amount of time debugging. You can also factor in the difficulty of debugging certain parts of code – there is a big benefit there.

Flexiblity

People also tend to own code that a) needs changes and b) is brittle and c) nobody understands any more, and everybody knows how much of a pain that is.

Freedom to focus on new stuff

All devs like to work on new stuff rather than fixing old bugs. If you have good tests along the way, you won’t have to be dragged away from the new stuff to fix the old stuff as often.

Pride

Everybody likes to write code that works well, nobody likes to own the bug factory. In fact, I think people leave groups to get away from something they know is a bug factory but nobody else does.

Permission

Group dynamics often push devs toward meeting a team’s definition of code complete rather than spending the time writing tests. Now, I happen to think that TDD often gets you to code complete sooner (in a tradition milestone approach), but if you’re just learning it, that isn’t the case. You need an explicit declaration that it’s okay to be spending the time writing the tests

Tests as examples

If you are creating a component that is consumed by somebody else, you can save a bunch of your time by having unit tests. Not only do you spend less time responding to email and doing bugfixes, the tests often provide very nice examples of how to use your component.

You may not that I don’t list “Design by example” there at all.  I think that design by example is a much better way to create software, but it’s an experiential thing rather than something you can put on a powerpoint deck.

Hope that helps.

One of my readers asked whether there were any UI unit testing tools.

While I have seen some ASP.net tools like this, in general I’d expect that you would unit test a UI by making the UI a very thin layer (one that doesn’t really need testing), and writing the unit tests to talk to the layer underneath.

Though I haven’t had the opportunity to try it on a full project, I think that Presenter First has a lot going for it.

O’reilly publishes Beautiful Code

Jonathan Edwards counters with a beautiful explanation.

Now, I haven’t read the new book, but I have a strong resonance with what Edwards wrote.  You should definitely read the whole thing, but I few sentences jumped out at me.

A lesson I have learned the hard way is that we aren’t smart enough. Even the most brilliant programmers routinely make stupid mistakes. Not just typos, but basic design errors that back the code into a corner, and in retrospect should have been obvious.

and

It seems that infatuation with a design inevitably leads to heartbreak, as overlooked ugly realities intrude. 

Precisely.

If there’s anything that agile says, it says that we should build things simply and with a eye to revision because we not only are we “just not smart enough”, there are too many unknowns when we start.

The problem with “beautiful code” as a concept is that it is closely related to “beautiful design”, and I’ve mostly come to the conclusion that any design effort that takes more than, say, 30 minutes is a waste of time.

The concept also gets confused about what the goal of software is anyway. The goal is not to have beautiful, elegant, code. The goal is to have *useful* code that does what you need it to do.

Discuss and comment

Thanks for your comments.

I decided to go ahead and write the unit tests for that layer, both because I knew what not writing them would be like, and I wanted to play with wrapping/mocking a system service.

I also decided – as some of you commented – to do the right thing and encapsulate it into a class. That would have happened long ago, but though I’ve written it several times, I don’t think I’ve ever duplicated it within a single codebase – and the codebases where I did write it are pretty disparate. Now, I have something where I could at least move the source file around…

Writing tests for this was a bit weird, because in some sense what I needed to do was figure out what the system behavior was, break that down, write a test against my objects, and then write mocks that allowed me to simulate the underlying behavior.

So, for example, I created a test to enumerate a single file in a single directory, wrote a wrapper around DirectoryInfo, and then created a mock on that object so I could write GetFiles() to pass back what I wanted. And so on with multiple files, sub-directories, etc.

So, I did that, went to write the little bit of code that I needed in the real version (to use the real GetFiles() calls and package the data up), hooked it up to my real code, and it worked.

*But*, when I went back and looked at the code, I found that what I had really done was create two sets of code. There was the real code that called the system routines and shuffled the data into my wrapped classes. And then there was my mock code that let me control what files and directories got returned. But there wasn’t any common code that was shared.

So, my conclusion is that I really didn’t get anything out of the tests I wrote, because the tests only tested the mocks that I wrote rather than the real code, because the only real code was the code that called the system functions.

In this case, TDD didn’t make sense, and I will probably pull those tests out of the system.TDD may make sense the next level up, where I’ve written a new encapsulation around directory traversal, but it seems like the only code there is hookup code.

So, the result of my experiement was that, in this case, writing the tests was the wrong thing to do.

I’ve been writing a small utility to help us do some configuration setup for testing. It needs to walk a directory structure, find all instances of a specific xml file, and then make some modifications to the file.

I TDD’d the class that does the XML file stuff, and I’m confident that it’s working well. I’m now going to do the class to walk of the directory structure and find the files.

And there is my dilemna. I don’t know if I’m going to do TDD on that.

I know exactly how to write it, I’ve written it before, and my experience is that that is code that never changes nor breaks. And, figuring out how to write tests for it is going to be somewhat complex because I’ll have to isolate out the live file system parts.

So, I’ve already decided what I’m going to do, but I’m curious what you think. Does YAGNI apply to test code, or is that the first step to the dark side?

I was reading a “Joel” post (I like Joel’s writing, but I wish that he allowed comments) entitled “The Econ 101 Management Method“, which I find myself mostly in agreement.

I’d like to expand a bit in the area of metrics – specifically what I call “surrogate metrics”.

Most software development teams are associated with what business guys call a “P&L center”, which in simple terms means that it’s part of the business that will either make or lose money. The measure of whether the group is making or losing money is an example of a metric, and it’s a good metric, in the sense that it measures exactly what it says it’s going to measure.

How important that particularly metric is to a company and what other metrics are also important is a different subject. As is the siren song of metrics in general.

The subject of this post is metrics don’t measure the thing that you want to measure, but are *believed to* correlate with the thing that you want to measure.

Say, for example, that you’re a software company, and you’ve heard through the grapevine that customers are unhappy with the service that they are getting through your support forums. A little research shows that some people aren’t getting prompt answers.

So, you spend some time writing a reporting layer on top of the forum software that tracks the time between when a post first shows up and it has a response from somebody in your company. You run it on some historical data, and see that the response time averages out at 36 hours, which makes you unhappy. You work with your group, tell them that they need to do better, and over the next month, the average response time goes down to 12 hours, and you’re happy that you’ve solved the problem.

Did you do a good job? Is the problem fixed? Discuss…

The answer to my questions is a rousing “who knows?” It’s possible that the problem is fixed, and it’s also possible that it’s still as bad as before. That’s because “response time” is a surrogate measure of the thing that you really care about, customer satisfaction.  You chose it because it was a *easily measurable*.

Which I guess does lead me towards discussing the siren song of metrics in general. There’s a real bias in some business cultures towards measuring a lot of metrics. As Joel points out, this leads to people gaming the system, which is an obvious issue. But even if people don’t game the system, surrogate metrics can, at best, suggest when something is bad, but they can never tell you when something is good enough.

Some people would argue that you should still collect the metrics you can, but I think you just shouldn’t bother with surrogate measures. Measure the things that you truly care about, and don’t mess up your culture and reward system by measuring the surrogates. And if you can’t measure the thing you really care about objectively, if it’s too hard or too expensive, you’ll just have to deal with the the uncertainty.

In my example, if you care about customer satisfaction in your support forums, then you need to ask customers whether their support experience was acceptable. There are lots of ways of doing this, and you can often use the same process to allow customers who had a bad experience to escalate it.

So, what is your favorite real measure and surrogate measure that you’ve seen? 

Software is always built on other software – your dependencies – and the dependencies that you choose have a considerable influence on your success. Choose the existing technology that you know, and you have good predictability, but you might not produce a great product, or it might take too long to finish. Choose a hot new technology, and it’s harder to predict what will happen. Maybe the benefits will be great and you’ll finish faster (ASP.NET vs old ASP…). Maybe things won’t be as good as promised (insert the name of a technology that you were “disappointed with” in the past).

Or maybe it’s not finished when you need it. Welcome to the wonderful world of co-development, where you are depending on features that aren’t implemented yet. How do you reduce the risk of features/APIs not showing up, or being substantially different than you expected?

Well, the first (and best) way to reduce this risk is simply not to do it. If you only depend on features and APIs that are currently available, you know they are there.

If you can’t wait a full release cycling, then perhaps you can take some sort of incremental approach, where you plan to use feature <X> but don’t *commit* to using it until its actually there. My preference would be an agile approach (such as Scrum), so that when feature <X> shows up, it’s actually finished and working.

That’s really just the same thing I said first – don’t take on the dependency until something is done.

But what do you do if you really need that feature – if your plans would be derailed unless the other team finishes the feature? I have four things in mind that can help:

Accept the Risk

First, you have to accept that you are taking on risk. Software scheduling beyond a period of a month or two is not only an unsolved problem, I believe it isn’t a tractable problem. Decades of project slippage have demonstrated that, and we should just embrace the uncertainty involved rather than trying to “do better”.

Note that while there are teams out there that can give good estimates for tasks in the next month (and perhaps up to two months), you can’t assume that you are dealing with such a team. There are many teams who are essentially unpredictable even in short timeframes.

Understand the Risk

Second, you need to understand the risk. This will require you to work with the team that’s building whatever you are needing. You need to understand where your feature ranks in the things that they are doing. It might be a feature that they absolutely have to have to ship, or it might be a “nice to have” feature. You need to understand this. It’s closely related to how close your requested feature is to their main charter. You do not want to be the outlier group amongst all their clients, the customer they don’t want to have.

You also need to understand when they’re building the feature. If it’s very early in the cycle, then it’s likely to get done. If it’s late in the cycle, it’s less likely to get done.

If they don’t think of features in this way and/or are working on features in parallel, it’s more risky.

It would also help to understand what development methodology they use, and their history of being done when they guess they will be done.

Plan for Mitigation

What are you going to do if things don’t work out, if the feature is late or is cut? Even in the best organizations, people get married, are out for months on medical leave, have accidents, or leave to form their own companies.

What is your group going to do when this happens?

Track the Risk

In an ideal world, the group you depend on would give you regular updates about the feature you’re waiting for. And some groups do do this, but it’s your risk, and you’re going to need to stay on top of it. The details of that depend on the groups involved.

Accept the Outcome

If things work, great. But if the feature doesn’t show up, remember that you were the one who accepted the risk in the first place.  

We’ve been spending some time trying to figure out what our team can get accomplished in the next 6 months or so. We have a feature list that we’re considering, and we need to apply some numbers to it.

So, I decided to try using Planning Poker. I made up some index cards with values in dev-weeks, in the values 1/4, 1/2, 1, 2, and 4.

It has taken a fair amount of time to get through this process, but the numbers we got are a lot better than we would get with other methods, and more importantly, the dev team has a much better idea about what and how they’ll be building whatever we are building (I apologize for being evasive, but it’s a bit of an organizational disease right now).

Note that I’m not recommending all-up estimating. In general, I don’t think you should be costing more than a month or so worth of work at a time, as the quality of the estimates won’t be great. In this case, we needed a go/no-go number, and the estimates that we got were good enough.

Last night about 1 AM we returned from our 48 hour enforced vacation in the 19th century.

As I’m sure you have heard, we had a pretty big windstorm, and we spent two days hunkered down in from of our gas fireplace. It was far more boring than dangerous, though cleaning out the fridge this morning was a bit harrowing. It’s amazing how many useless condiments you can accumulate over the years.

Our house emerged unscathed. We took out the big doug firs in our front yard a few years ago, and this is the kind of storm that would have dropped them on the house.

The holiday lights, however, did less well. The santa and reindeer plywood cutouts got tossed down the slope, most of the globes got tossed out of the big tree (and the remaining ones aren’t fully functional). And the spiral tree and base on top of the garage – weighing in at around 50 pounds got picked up and tossed off the garage, landing on the guy wires to my tree of lights and bending the main support pole creatively (and significantly). After I just got finished repairing the rodent damage.

I’ll also need to see if the controller getting wet caused any issues.

Tommorrow I get to drive up to our ski cabin, to see if there are any trees down there. I’m taking my chainsaw…

We’ve been doing some planning ’round these parts – planning that I unfortunately can’t talk about – but it’s led to a fair amount of discussion about architecture, both inside the team and outside the team.

Which has got me thinking about reuse.

Reuse has been one of the Holy Grails of software development for a long time, along with… Well, work with me, I’m sure there are others. True AI!. That’s another.

Anyway, reuse has been discussed since time immemorial (October 13th, 1953), for some pretty sound reasons:

• Software is hard and expensive to develop


• We already break things apart into “components” to simplify development


• We’ve all written subroutines that are used in multiple places.

It seems that if we did a little more planning, paid a little more attention, were just a little smarter, we could build our components in a more general way, and others could benefit from them.

And yet, people have been trying to do this for a long time, and have mostly failed at it. There are successes – widely-used successes – but they’re fairly small in number. Surprisingly, people are still optimistic about going down the reuse path, and since they are likely to fail anyway, I therefore present some rules that can help them get there faster.

Eric’s N rules for failing at reuse

Authoring reusable components:

• It really helps to have delusions of grandeur and importance. You are going to be the ones who succeed at doing what others have failed at.
  


• Pick a wide/diverse scope. You’re not building a good UI framework for client applications, your framework is going to work for both client and web applications.
  


• Plenty of technical challenge. That’s what makes it fun and keeps you interested.
  


• No immediate clients. You are building a *Component*, and when you are done, clients can use it.

In my experience, that’s more than enough by itself, but it helps if you can throw in some obscure algorithms and quirky coding styles. I’m already assuming that you don’t have any real tests.

Consuming other people’s components:

• You absolutely *must* sign up to work with something as it is being developed. There is simply no better way to waste vast amounts of time. Unfinished implementations block you, regressions set you back, and even if you don’t get those, you at least get hours of refactoring to switch to the new version. It’s best if you do this on a “milestone” basis, roughly every 6 weeks or so. That’s short enough that you’re randomized often, but long enough that waiting for bug fixes is really painful.
  


• Commit to using the other component early and deeply. If somebody asks, “what will you do if <x> is too buggy, too late, or doesn’t do what you need?”, you can’t have an answer.
  


• Make sure that your use of the component is a variant of what the component is really being used for. In other words, the creator of the component is building the thing to do <x>, and you want to do <y>, which is kindof like <x>.
  


• A quick prototype should be enough to ensure that you can do it.
  


• If you can’t get the authoring group to commit to producing what you want, take a snapshot of their code and work on it yourself.
  


• Don’t plan any schedule time or impact to deal with issues that come up.
  


• If possible, buy the component you’re using. Because if you paid for it, the quality will be higher, and they’ll have to give you good support.

I hope these tips help you.

If you’re a bit leery of reuse, then good for you. I have only a few thoughts to offer:

If you’re thinking about doing something, it’s always a build vs buy decision. Even the best general-purpose framework out there is just that – a general-purpose framework. It’s not designed to do exactly what you want to do.

In the abstract, there are three phases of using a component in your code:

Phase 1 is great. The component is doing what you want, and it’s quick and easy to do it. Let’s say for sake of argument that this gets you to the 80% point in your project, and it gets you there quick.

Phase 2 is a harder. You’re starting to reach the limits of the component, and it’s tough to get it to do what you want. Tough enough that it’s taking more time, and you’re using up the time that you saved in phase 1. But you still feel like it was the right decision.

Phase 3 is much harder. It’s taken you as long to get here as a custom-written solution would have taken, and making further progress is considerably slower than if you had written everything. Worse, you can see the point where you’ll reach a wall where you can’t do anything more, and it’s close.

Different projects obviously reach different phases. Some never venture out of phase 1, and others are deep in phase 3. It’s hard to tell where you’ll end up, but if a given component is central to what you do, you are much more likely to end up in phase 3.  

The obvious problem is that prototyping is always done in phase 1, and the rapid progress you make there is oh-so-tempting. The whole application UI is laid out in a week of work using Avalon. I got this demo with moving pictures done in 3 days using XNA. We all want to believe that it’s really going to be that easy.

Stay strong against the lure of the siren song.