We make estimates of effort all the time. No big deal. But estimating software projects is a big problem, often a nightmare. Why should software be uniquely difficult?
Estimating effort is a reasonable thing to want
How long will it take you to get to the meeting? When can I have the summary report? How long will it take to crank out the press release?
We ask for and get estimations of effort all the time. It's a reasonable thing to want. If you think something will take a day and the person tells you it will take a week, probably something is wrong, and it's helpful to expose the issue and resolve it.
Estimating software effort seems like a reasonable thing to want
If estimating effort before you start is generally reasonable, it would seem to be even more a good idea when it comes to software. After all, software projects tend to be expensive and take a long time. Worse, they have a nasty habit of running over time, over budget, and then not even giving you what you expected. So it would appear that creating detailed requirements and then providing careful time and effort estimates would be even more advised in software than in other fields. At least then we can track our progress see have a chance to fix things when they go off track.
Mainstream thinking in software management thoroughly endorses this view. The highest virtue a software effort can have, in this view, is being "in control." A software project effort is deemed to be "in control" when it is working its way up to creating estimates (i.e., gathering requirements, etc.), or logging work against estimates already made and tracking how well the estimates are being met.
There is a body of work on this subject that is both broad and deep. You may be impressed by this, but actually it's an indication of how difficult giving good estimates is, and how rarely it's accomplished. When projects go badly, get "out of control," the blame is usually placed on the process not being taken seriously enough. Inevitably, people with experience respond to bad experiences by laying on the process deeper and thicker, and making sure the estimates are long enough to account for all sorts of unforeseen events. In other words, even more time is spent on things other than actual programming, and the estimates are padded like crazy -- even more so than last time.
This, in a nutshell, is why programming groups get larger and larger; accomplish less and less; and never do anything new. It's why startups with bright programmers, unencumbered by the ball-and-chain of estimation and process, usually are the ones that accomplish the cool new stuff that programming groups with literally hundreds more programmers cannot.
Software Estimations are Different
The facts on the ground are that estimate-based programming leads to no good. How can we make sense out of this? Is programming really that different? Yes, programming really is way different than all those things that are reasonable to estimate. Here are some of the ways to understand this:
Huge variations in Productivity
Software estimates are in terms of the work -- not the person doing the work. But there are HUGE variations in the productivity of different programmers, more than 10X. So when you estimate that a given task will take ten days, is that 10 days of the 1X-er's time (the 10X-er will get it done in a day), or 10 days of the 10X-er's time (the 1X-er will take 100 days, and the 10-20% of programmers who are 0X-er's will never deliver)?
Skeptical of this huge variability? Here's a summary of the literature. McConnell concludes, "...the body of research that supports the 10x claim is as solid as any research that’s been done in software engineering."
What kills you with any estimate are the unknowns. The newer the kind of software you're developing, the greater the unknowns tend to be. Worse of all are the things you don't know you don't know. If estimating resolved the unknowns, I would warm up to it -- but in practice, estimating tends to make everyone feel better about how "in control" the project is, and delay the discovery of the unknowns until a later time, when they are even less convenient, a.k.a. "nightmares."
How do you discover unknowns sooner? Dive in and start doing the work sooner!
You can only estimate if you know what you're building. The more exact the requirements, the better the estimate -- or so the theory goes. It's why everyone who's big into estimating is even bigger into requirements, and all sorts of squishy-sounding things like getting all the "stake-holders" on board.
In the real world, requirements change. Call it "evolution" if you want, but it always happens -- worse of all is when the real requirements in the real world are evolving while the project's requirements remain static -- in the name of maintaining the "integrity" of the software project, real world be damned! Only in academia (or silo-ed software departments of bureaucracies) are requirements immune to outside-world changes, discoveries made during the project, etc.
Learning by Doing
All programming starts with thinking. Thinking about the existing code, the tools, the architectural structure, the functionality that needs to be built and where and how you'll accomplish that functionality in the code. An integral part of this process is exploration -- filling in your knowledge of missing pieces as you think things through. In fact, it's hard to separate exploration from thinking; as you think, you discover holes in your knowledge, which you go off and fill, which helps flesh out and evolve your thinking. The more clear and complete your combination of thinking and exploring, the more cleanly, concisely and quickly you'll code it.
In practice, the best people write and change code as they go through this process. Nothing like trying to actually write the code to clarify the issues with writing it. The best way to discover road-blocks? Get on the road!
Do the people who create estimates do this? If not, what possible value is their estimate? If so, why don't they dive in and do the work, while everything is in the foreground of their mind?
There is more than one way to peel an apple (what a stupid phrase -- who wouldn't want to just eat the darn thing, apple peels are really good!), and there is more than one way to get stuff done with computers. I'm in the middle of looking at some new functionality an Oak company is trying to build. Naturally, many people at the company want to take their existing, comfortable implementation path and apply it to the new problem. It's a good deal of work, involving a DBMS, stored procedures, a couple development environments, some business logic, a rules engine, a workflow engine and some UI designing. There is no doubt in anyone's mind, including mine, that the existing tool set can get the job done.
A kind of estimating causes me to strongly prefer the new implementation path -- I'm sure it will be a lot less work, with results that will be more pleasing to consumers. A sense of estimation is indeed important, as one of the factors that helps you choose the optimal implementation path for getting a given job done.
The software approaches and implementation paths taken by most companies for getting their work done were established many Moore's-Law-generations ago, when computers were 10, 30, or 100 times slower and/or smaller and/or more expensive than they are today. For example, most programmers continue to think in terms of databases, when Moore's Law has changed the game for databases and for storage.
It is far more fruitful to figure out how to take advantage of changes like this than to put effort into estimating the work using what are likely to be obsolete paradigms for building the software. And the best way to accomplish this is usually to try it.
Asking how long something will take to program seems so reasonable, so innocent. Who would ever guess that it tends to start a cascade of disastrous consequences?
The truth is that estimation does play a role in good software development, but in a completely subsidiary way, as a minor aspect of your technical and conceptual approach to building the software. It's like running a race: yes, you have in mind your past times, but if you ever get into making estimates and then padding them and then judging success by hitting your estimated time, you're going to lose the race. It's just not what winners do. Winners focus on ... winning!