The Black Liszt

Summary: Software Project Management

Software project management is a major aspect of software development. People generally only think about it when changes to it are being considered. Most of the time it’s just part of life. It determines your role in a software group, how to interact with others in your group and is the central function determining whether efforts are successes or failures.

Some form of project management is required for all software projects – otherwise they are considered out-of-control amateur efforts. Sadly, this is a nearly universal bad idea – project management has proven to be a disaster for software development.

A core trouble is that the ideas of software project management are based on inappropriate physical metaphors such as architecting and building a house.

https://www.blackliszt.com/2018/12/software-planning-is-impossible.html

I wrote software for a number of years. I got the stuff done people wanted quickly. Then I joined a company that was infected by experienced professional managers and got project management force-fed to me.

https://www.blackliszt.com/2012/10/software-project-management-book.html

I studied more and found there were many thousands of books on the subject!

https://www.blackliszt.com/2012/10/there-are-lots-and-lots-of-books-on-software-project-management.html

I learned that there were intense disputes in the industry about the subject, particularly concerning the widely-mocked “waterfall” method that was giving way to supposedly better Agile methods.

https://www.blackliszt.com/2013/03/software-comparing-waterfall-and-agile.html

Regardless of the religious sect of project management to which you belong, the key factor is how much time you spend writing code versus doing anything but writing code. With rare exceptions, spending more time writing code rather than just about anything else leads to better productivity.

https://www.blackliszt.com/2012/09/software-productivity-the-abp-factor.html

Why is project management so bad? Don’t you want to know how long something will take and what it will cost before you dive in? Of course you do! That’s why it refuses to die. The absolute core issue is that all forms of project management optimize for predictability, starting of course with estimation.

https://www.blackliszt.com/2012/06/the-nightmare-of-software-estimation.html

When you dig in, it turns out that all those estimates and dates lead to dysfunction. Someone who says they’ll run a mile in 20 minutes and makes 19 is praised, while someone who hopes to do it in 10 minutes but makes 11 is a failure.

https://www.blackliszt.com/2010/10/software-project-management-dates-are-evil.html

The whole notion of releases needs to change.

https://www.blackliszt.com/2011/02/software-project-management-releases-are-stupid-and-out-moded.html

In spite of all the fervor about Agile etc., the fact is that nearly all software projects are way too expensive and slow and have crappy results.

https://www.blackliszt.com/2021/02/why-cant-big-companies-build-or-even-buy-sofware-that-works.html

https://www.blackliszt.com/2014/04/delivering-software-is-a-nightmare.html

The solution is widely proven by small groups that really need to win instead of getting praise from experienced managers. They use grow-the-baby techniques with post-hoc design and wartime methods.

https://www.blackliszt.com/2011/06/software-how-to-move-quickly-while-not-breaking-anything.html

https://www.blackliszt.com/2022/09/better-software-and-happier-customers-with-post-hoc-design.html

https://www.blackliszt.com/2013/05/wartime-software-optimizing-for-speed.html

The book I wrote has lots of explanation about the issues.

https://www.blackliszt.com/2012/10/software-project-management-book.html

 

Software Planning is Impossible

What else is new? Everyone knows that software planning is, well, just impossible. Live with it!

I mean something else entirely. I mean that software planning, in the usual sense of the word, is literally impossible to do. Planning for a new house? OK. Planning a new road or intersection? You may not like the cost, disruption or time, but it can be done. "Planning" in the sense that everyone means it and uses it for everything else, is literally impossible for software.

Here's why, in a nutshell: a "software plan" that is the exact equivalent of an architectural plan for a building, including the materials list is the software itself. Anything less is a vague sketch of what part of the building might look like, something a builder asked to quote on it would say, "I'll be glad to give you a quote when you give me a plan. Which this is not."

Let me explain.

Building plans

Most people understand the basics of planning a building, say a house.

You start by answering some very basic questions, like How Big, and Where. This location and size information determine much of the eventual cost of the building.

You then go with architecture, i.e., the guts of building planning. Starting with size and place, you then go into the general shape and style of the house, things like How Many of What Kind of rooms and finishes. You may interact with the architect, creating several rounds of plans. As you agree to one level of generality, you dig deeper, ending with things like colors, appliances, and materials (wood clapboard vs. Hardie board, etc.). If the architect is modern, he'll show you 3-D renderings and give you 3-D walk-throughs of fully furnished rooms.

Here's an overview that I built using common software, two snapshots from different directions:

Notice the landscape, the terrain, the bushes, the shadows -- these are just snapshots from a complete 360 degree view. You can pick the angle, distance, height and even the position of the sun. Those are images of real clapboards, doors and windows that are commercially available, everything.

Here's part of a floor plan I built with the same software:

 

And here's a variation of that plan with a bunch of furniture added and some other changes:

Here's a sample 3D interior view from the software's website:

The software enables you to not just have pictures, but to do a full 3-D live walk-through, just like in a video game.

If the architect uses modern software, the software will not only assure that the building is structurally sound, the software will fill in all the structural elements, including electrical and mechanical, and produce an item-level materials list.

That's the plan! You can give it to builders for time and money quotes, and to the local building department for permits. Once you make an agreement with the builder, off starts the construction project. Hopefully it finishes about when promised, and you end up paying the agreed-upon amount. Done!

When it's done, unless the builder has screwed up, it will look in physical life just like the drawings and renderings, with all the selected materials.

Software planning

Software planning is basically the same as building planning, right? Except even better! With buildings, you're digging holes, pouring concrete, putting up framing, and all sorts of time-consuming, physical things that are costly to buy and install. By contrast, software is just writing code, a very non-physical activity -- any change you want or mistake you make is easily fixed, without ripping out building materials or breaking up concrete. Moreover, building planning is like old-style "waterfall" project management. Agile, the modern trend in software, isn't possible with buildings.

Why is it, then, that everyone with practical experience knows that software planning is a mess, a total nightmare, pretty much no matter what planning regime you follow? Why do you think the industry keeps coming up with new ways of planning things? For building planning, the methods were always effective, and with the increased use of software, the methods have stayed the same, but the ability to render the results to clients and automate error-prone architecture work has gotten better and better. For software planning, we experience a ever-ending sequence of planning "revolutions," each of which promises to eliminate the perennial problems. Except the benefits are always in the future, and the future somehow never happens. What's going on here?

At root, the cause of the problem is simple: the analogy of building planning and software planning doesn't work! But we keep trying.

Here's why: a plan for a building at the level of detail I described above is truly comprehensive. The ability to render detail in 3D makes that clear, and the ability to generate the structural elements and item-level material list makes it crystal clear. While it's incredibly comprehensive for a building plan, no software plan comes close to that level of detail in terms of building software!

The reason is simple:

A building (or road or bridge) is passive, built by workers and machines from a carefully assembled inventory of passive parts and materials, things like lumber, wiring, pipes, siding, concrete, etc. A program is active, a carefully organized set of instructions and data (mostly instructions, i.e., software) that do things in response to active inputs and stored data, as a result of which stored data is added or altered and new data and/or actions are created in the outside world (e.g., new screen displays, messages sent to distant servers).

This difference is a BIG DEAL. It's EVERYTHING!

The Building metaphor for software is bogus

Think back to the beautiful building plan I showed above, complete with its ability to enable a person to do a virtual walk-through of the building. That's like a software plan, right? It's kind of like what people call wire frames, mock-ups of the UI, right?

Now imagine that the software plan we're building is for a combat-oriented, multi-player video game. How close is that ready-to-build physical building plan to a plan for the video game? Ahhh, maybe 5%? The easiest 5%, for sure.

The detailed building plan we can walk through is like the "world" that video games create. Except the world is always changing. No building plan can deal with a player racing through the building, crashing through a window and firing a blast that blows a hole in a wall -- a hole that is customized to the location and angle of the shooter, the type of wall and the kind of blaster used. The wall explosion could further impact another game player on either side of the wall, in real time. This is the hard part, and where the vast majority of effort of building a video game goes. It's the action part.

Is building commercial transaction software like building a video game? No, of course not. There are some elements of video games that are unique. In other ways, building commercial software is harder than building video games! Video games are completely self-contained "worlds." For better or worse, commercial software is very much part of an existing, amazingly extensive world -- of other software! Existing software that is always changing, has bugs, is extremely elaborate, and whose actions depend not only on what you ask of it, but what's been asked in the past -- context. Commercial software is a complex series of actions, many of which involve interactions with other software that is no less complex.

What's important in software, and takes the VAST majority of the work, is the action part. A building is passive. It just sits there. Even the "passive" parts of software are created, on the fly, by the action part. For example, when you first go to a web page, it might be fairly passive. But the second you click on something, action starts. Once you login, the action gets fierce, and is created, on the fly, just for you: with data that's been stored for you and things on the screen that are customized for you.

This is the difference between walking into one of those Amazon retail book stores, which are what they are, and going to the Amazon website as a logged-in visitor. In the store, personalization is performed by a worker who walks up to you, who of course doesn't know your complete history of interactions with Amazon, which the software does and reflects in how it interacts with you. This goes all the way down to simple things; for example, if I go to the web page of a book I bought years ago but forgot, the software helpfully tells me I already own the book, would I like to get it again?

Conclusion

As usual, metaphors lead us astray -- even the universally-accepted metaphors for software. This is really sad, and the fact that it's gone on for decades is even sadder, with no signs of changing for the better. I've gone into the sources of the bogus building planning metaphor in great detail in my book on Software Project management, along with more detail on why the metaphor doesn't work.

All this history and logic is besides the point, in a way: the important thing is that, even today, the bogus building metaphor for software enjoys near-universal acceptance, in spite of its persisting monumental failures. Instead, the "leading minds" of the industry yammer on about irrelevancies like Agile and Scrum that may make people feel better, but don't change the underlying problem.

You wish for a better world? Be one of the renegades who actually gets stuff done; try Wartime software. Here's the background.

Healthcare Innovation: EMR Procurement is Broken

Computers and software get faster and less expensive at a dramatic rate. Healthcare systems implementing computers and software gets slower and more expensive at a dramatic rate. Why is buying the thing getting so much worse at the same time as the thing being bought is getting so much better?

There is only one explanation: the procurement process used by the large organizations is broken. Badly. It doesn’t need “improvement” or “innovation.” Today's standard procurement system needs to be thrown out. We need to start over.

Improving Computers: Incredibly complex, successful

It's just unbelievable how much better, faster, smaller and cheaper computers get. The rate of improvement is unprecedented in human experience. Nothing comes close.

Even software that used to be unavailable or expensive to buy is free! Think "open source," for example.

Procurement of computers: Basically simple, a worsening mess

Most large organizations are just awful at building, acquiring and implementing computer hardware and software. It's not getting better.

How do these organizations respond to their screw-ups? They look closely at what went wrong, "improve" the process ... and make it even worse.

Back to paper

Organizations really try to make it work. The US Coast Guard, for example, embarked on a project to plan the implementation of a leading EMR in 2010, and committed $14 million just to do some planning. Six years later they cancelled it:

They knew something was really wrong, not just the usual wrong stuff. You know things are tough when they go back to paper:

On the other hand, they can provide all the health services their members need with paper alone! Tells you something about today's EMR's, doesn't it?

New York City's HHC EMR procurement

NYC's public hospital system is big. They're also a pioneer in using EMR's. They've won awards because of it, and gotten $200 million from the feds for achieving full "meaningful use" status with it. They go way beyond just electronic recording; they manage diseases, do screening, and lots of advanced things. See this from a January 2013 HHC press release:

By their own report and by the judgment of important institutions, HHC has this EMR thing nailed!

HHC has bigger problems than EMR. It is in big financial trouble, and it's getting worse. See this report from late 2013:

They've got an industry-leading, award-winning, value-enhancing, WORKING EMR, and huge financial troubles. Sounds like an ideal time to buy a new, system-wide EMR, right??!!

Well, that's what they did. In same early 2013 press release in which they bragged about their wonderful existing EMR, quoted above, they announced they were buying a new one:

A close reading of the press release leaves one wondering why they had to have a new one: the touted wonders of the new system line up pretty well with the wonders of their award-winning existing system. I guess they really hankered after that new-EMR smell. And with the big award from the feds, $200M of the $302M is already paid for, leaving just pocket change to pay for it!

But then, less than a month after this press release, HHC issued an RFP for extensive additional services. Here is a list of the consultant skills they wanted to hire:

Strange. I got the distinct impression from reading the press release about getting Epic that the $302 million covered everything. What's this about?

The July 2015 board meeting of HHC had an update on how things were going. Things weren't working out too well on the cost side; those numbers are millions, by the way:

Wow. Just a bit more that $302 million. Digging a little further in the Board minutes, we find a little about the $113 Million. Here's the summary:

Or maybe not -- this $119 million is for consultants, not FTE's, and it's per year -- the surprise on-going cost of getting that shiny-new-EMR-system smell into the hospital! And you have to notice that 20 vendors will be awarded contracts -- because everyone knows that having lots of temporary people from lots of vendors working on a single project is the proven way to maximize coordination and minimize surprises. Sure.

Oh well, at least things are "under control." Or maybe not. Just a month later, in August 2015 we learned:

And the dates? Let's talk about something else, because "full operation by 2017 for all users" is history, along with the $302 million cost and lots of other things for this troubled system.

Conclusion

HHC is a nice, big, fat example -- but it's not unusual. This is what EMR procurement is like, over and over again. The buyers just keep shuffling ahead to the painful and prolonged slaughter, like cattle. Few organizations are as smart as the US Coast Guard, and decide that paper is just fine. Big-organization procurement in general is broken, and EMR procurement in particular is badly broken. Every attempt to "fix" it seems to make it worse. The procurement process needs to be thrown out. We need to start over.

But start over with what? I will summarize how in forthcoming posts. But it's not mysterious. It's just an application of the proven methods described in my books and summarized in various posts on this blog.

Excellence in Government IT

Consider the sets "Excellence" and "Government IT." There is a great deal of evidence that these are non-overlapping sets. Put another way, the phrase "excellence in government IT" is an oxymoron. Of course, there are people who think otherwise. Mostly, these are government workers and their enablers.

Digital Government Awards

It appears there are organizations promoting and celebrating "digital government." Who knew?

Part of what these guys do is hold awards ceremonies honoring the best, the brightest and the most accomplished. There was an awards ceremony for New York in 2014.

30 people were individually honored for Outstanding IT Service and Support. In addition, 10 awards were given in various categories. One of the categories is related to one of my favorite subjects. The award, "Demonstrated Excellence in Project Management," is a double killer: excellence in project management, which you mostly demonstrate by chucking it over the side of the boat, and excellence in government IT, which is pretty much the null set. So "government project management?" If there ever was a candidate for something emptier than the null set, that's got to be near the head of the line.

One naturally wonders what magic project won this coveted award. This project was so good that the leader was also awarded the Best of New York Leadership Award. Here are the highlights:
This is a bit hard to figure out. Mostly, it appears, he spent money and outsourced work. He put a little data center into a big central one, and by the way bought a bunch of new equipment (that's what "modernizing WCB's infrastructure" means), and he dumped thousands of cases to an outsourcer ("third-party administrator" sounds more official, doesn't it?), I guess because those poor government workers were just overworked.

But I was unsatisfied. I really wanted to know how he got the top award for project management. So I clicked to find out: And I was rewarded with this page, from the organization that leads, promotes and awards excellence in digital government:

I was truly impressed. I always wondered how all those government agencies, some of which are bound to have bright people who truly want to serve the public, managed to deliver such uniformly expensive, inefficient, labor-intensive systems that often don't work. Now we have the answer: they have an organization that leads them and shows them how its done!

By giving awards, they in effect define excellence down. Think about this guy singled out for the leadership award: he bought a bunch of equipment (for less? more? who knows?), moved to another data center and outsourced some work. That's the best of the best! Think about what everyone else accomplished during the year!

Project Management: the Zombies have Won

If you're a professional software project manager, I have a suggestion. Why don't you become a consultant with Mary Kay or Avon so you can do something more worthwhile with your life?

Oh, boy, that was mean. But if you can stand it, read on.

Project Management in General, and in Software

Project management is a well-developed body of theory and practice. In most fields to which it is applied, it is the only responsible way to run things. Period.

So you'd think it would be a winner in software, which badly needs something to get it manageable. It's really hard to believe that normal project management techniques and practices wouldn't apply to software development pretty much the same way they apply to other things. But they don't.

We now have literally decades of experience showing that project management, when applied to software, simply and categorically does not work. I've covered this subject quite a bit on this blog, and devoted a whole book to exactly how and why it does not work.

It is one of the many sad results of the mad refusal of the whole software industry to pay attention to history that this fact is not one of the first things taught in school.

Project Management in Software

As it is, project management for software is a skill you can acquire. There are piles of books. There are certifications. Many of the people who go into the field are nice, well-meaning people. I like most of the ones I've met. One guy I know even teaches courses in it; from his description, it sounds like his course would be great!

But there's a problem. Not all programmers admire or even respect project managers. There are good reasons for not wanting your project to be infected with the disease of project management. But most programmers aren't particularly intellectual about it. They just want to be left alone! Some of them feel strongly about it. So I would advise project managers to watch their step.

And if you are going to get into project management and make a success out of, do try to take a course like the one my friend teaches, not one like Dogbert's:

If you avoid the Dogbert course, your life expectancy will be considerably longer.

 

Software: Comparing Waterfall and Agile

Lots of people talk about the evils of waterfall-style development. They aspire to move to something they think is better. Agile is high on most short lists for the something better. How different are waterfall and agile? Answer: not much.

Waterfall

The Waterfall model is an ordered, systematic method for determining what a computer system needs to do (the requirements) and then getting it done and into production. Like this:

The method is well-named. It really does look like a waterfall, like that big one famous for honeymoon visits on the St Lawrence River:

Above is a picture of Niagara Falls I look a little while ago, and is good for understanding software waterfalls. See the big river of water flowing from the upper right? See how everything is clear as it starts to fall? Then you see there's all the mist, making it very hard to see anything clearly at the end. Kind of like most software projects... This one gives you a good sense of the transition from clarity to mist:

Of course everyone hopes for the good outcome, for the rainbow emerging out of the mist:

But, I'm sad to say, the experience of Ms. Annie Edson Taylor comes closer to the common experience of waterfall software development:

While there is a vast array of software development philosophies, waterfall appears to be the standard against which most of them are compared; her concluding remarks saying it all: "nobody ought ever do that again."

Agile

Naturally, people look for better ways, and find lots and lots of ways that are thought to be better. It is incredible the number of software development philosophies there are. They go on and on! At least in my experience, Agile is the one I most often hear as a replacement for waterfall.

Like with all these things, people have a lot to say about Agile. There are books and books and conferences and training and certification, endlessly. Here is a summary diagram, given at roughly the same level of detail as the waterfall diagram above:

Lots of strong claims are made for Agile. It's faster, leads to better results, etc. Stuff that everyone says they want. But what are the real differences?

Comparing waterfall and agile

Take a close look at the two diagrams. Both of them start from requirements and go through design, development, test, integration and delivery. Here's the difference: with waterfall, you determine all the requirements up front and then drive through to delivery. The requirements are fixed, and you determine the time from there. In Agile, you determine a bunch of starting requirements, deliver them in a fixed time period (for example 2 to 6 weeks), and then get another set of requirements, and keep cycling until the project is done.

Waterfall: first fix the requirements, figure out the time.

Agile: fix the time periods, and then repeat until you're done.

Putting all the rhetoric aside, the difference between the two methods is simple: one determines the time from fixed requirements, and the other takes fixed time periods and fits requirements into them as appropriate. In other words, Agile is little more than a series of time-fixed waterfalls!

Remember, it's all just Process!

It's easy to get caught up in all this and forget that the most important thing isn't what makes Waterfall and Agile different -- it's how they're the same. Not exactly the same, but the same kind of thing: process!

You can build 100,000 lines of really crappy code using Agile. You can build 10,000 lines of great code that accomplishes the same thing using Waterfall. Or the other way round.

In Simple Terms

In simple terms, Waterfall is:

Do once: {Define. Design. Do. Check. Deliver.}

and Agile is:

Do until done: {Define. Design. Do. Check. Deliver.}

Conclusion

There are many good things about Agile. It's more iterative and can allow for more feedback loops than pure Waterfall. But its difference from Waterfall is easily exaggerated, which helps explain why the results in practice are so often disappointing. In the end, switching the precedence of the two key variables (requirements and time) can't make that much difference when the fundamentals of software and its Postulates are not addressed.

The Disease of Software Project Management

There are a lot of books on the market about project management in general and software project management in particular. More than 6,000 of them.

They all appear to think that software project management is a good thing -- at least the brand they preach.

I've threatened to publish a book saying that it ain't so. Giving details, arguments and examples. Sounds radical -- but it's not. Most sensible, productive software people know that software project management's effectiveness is best compared to the fineness of the emperor's new clothes:

In publishing this book, I'm not doing any more than the little boy in the story, who cried out "But he's not wearing anything at all!" In other words, I'm just saying what everyone who isn't blind already knows.

The book is now available on Amazon for Kindle. I even made a nerdy cover for it:

My hope with this book is to assure the people who know there's something deeply wrong with project management orthodoxy that they're sane people. but living in an asylum which the inmates have taken over. I hope the book will arm them with the concepts they need to make a break for it, so they can experience the fresh air and freedom they deserve.

Software Project Management Book

I've written a fair amount about software project management in this blog. I've also written a short book about it. Like the software quality book, so far I've only distributed it privately. But also like that book, I'm thinking of publishing it as a Kindle book.

Tid-bits on the blog

It's hard to be seriously involved with software and avoid run-in's (not to mention complete co-option) with project management. You can hardly start to think about writing some code without someone popping out with "how long do you think it will take," the question of estimates. If you resist or act uncomfortable, you're put on the spot. Everyone, you see, wants their software group to be as predictable as though it were a software factory. The people who talk this way clearly don't understand that dates are evil, but there are so many of them, it's like you live in a land of zombies.

Background

While many programmers resist it, they most often accept project management as a necessary evil, as something that they can't avoid. As they age, sadly, most programmers accept this perverse thought as though it were a natural accoutrement of adulthood: wild young programmers may resist the bridle, but mature ones accept that it's part of life.

I too resisted it, and I too came to appreciate some of the rhetoric of software project management. But then reality intervened.

A bit more than 20 years ago I ran a small software group doing pioneering work in document imaging and workflow. A new management team took over, and were appalled that we just wrote code. I was guilty of about the worst thing a manager could be accused of (in their eyes): running an out-of-control, seat-of-the-pants operation in which people just did stuff, without the comfort and support of project management.

Things changed. Expensive project management software got bought.
Expensive consultants came in and lots of formerly productive people sat in excruciatingly long training classes. For days! Then we settled into a regimen in which lots of reports and dense charts
were generated regularly, and we threw around terms like "critical path."

Well, we "got under control." And stopped writing much code. And fell behind the market.
As we became more predictable, we became more inflexible. Timelines stretched out so far that sales people lost heart. It was sad.

After that baptism by torture, which was followed by many more, I really began to think about what was going on when I got involved with Oak. I had a chance to see lots of companies producing software with varying doses of project management involved.

I noticed that the Indian Outsourcing companies were pushing project management big-time, and winning business with it. It must be a good idea, right? When you dove into the details, they did not win by being faster and more flexible. They were completely rigid and slower. But predictable and marginally less expensive. Here's the bottom line: they won business by costing less. They cost less because they paid their programmers only about one tenth of the equivalent programmer in the U.S. But their methods had so much overhead that they staffed every project so much that the final bill to the customer ended up being only about 30% less than doing it in-house. So, oddly enough, the Outsourcers with their devotion to project management proved the point of how bad it is.

On the positive side, I saw entrepreneurial companies doing more work with less, having more flexibility, less overhead, and shorter cycle times. Had they found clever new ways to implement project management? No. They just found better ways to develop better software with fewer bugs, more quickly. That's all!

Systematic Thought about Project Management

These experiences led me to try to understand what project management was really all about -- why everyone kept trying to apply it to software, why it never works (except if you don't care about time or money), and what the alternatives are.

It was a long journey, and I was surprised that I ended up with a short book. As I state in the book:

“Project management” is as effective at guiding software projects to success as hopping and grunting is at helping pool balls to drop in the intended pockets – it may be entertaining to watch, but it has no constructive impact on the outcome. More important, to the extent that we focus on our hopping and grunting technique, we fail to pay attention to what really matters – hitting the ball correctly with the queue. Similarly, in software projects, the more things get off track, the more we seem to focus on project management hopping and grunting activities, so much so that the shaking floor actually makes things worse.

Project Management needs to be taken down a few notches

Part of the problem is that it just doesn't work. Another part is that everyone with experience knows it doesn't work. The crowning part of the problem is that even people who know it doesn't work and put it to the side when they really have to get something done, continue to kow-tow to it. This is illustrated by a story I personally experienced that I tell in the book.

 I recently spent some time with the seasoned, non-technical leader of one of our portfolio companies, and some of his lead technical people. We discussed one of their most successful products. The CEO described how he got involved with a couple customers who had a problem that no one could solve, how he promised them a solution and got his programming team to throw something together that sort of worked. They then scrambled, fixing problems and coming out with a flurry of new releases, always listening to the customer and evolving their code until things settled down, the customer’s needs were met and the company had a new product line.

“Of course,” said the CEO, glancing over at his technical people, “that was the wrong way to do things. Later, we settled down and got back to proper project management.” Of course – the CEO had to intervene and make sure something important actually got done. Later, “project management,” i.e., doing very little but trying hard to do that little on time and on budget, could be allowed to return.

Conclusion

Again, I'm thinking of pulling the trigger on the Project Management book. But first I need to finish formatting it.

Update:

Trigger pulled. Book available.

There are Lots and Lots of Books on Software Project Management

There are an amazing number of books on software project management, each promising to tell you how to "manage" your way to software success. Amazon lists over 6,000 of them! There seems to be no end of books that claim to do it "better," or claim mastery over some sub-specialty.

Here are some books listed on Amazon. There are books (of course) for dummies:

Books from major publishers and computer societies: 

Here's one emphasizing numbers, from a major publisher and a big specialist consultancy:

This one appears to be a hit -- it's got over 30 reviews on Amazon, most favorable. It's from a major publisher, about a particular flavor of software project management:

This project management thing is serious stuff -- my next example is in its tenth edition(!), and is one of the books you need to read to be certified as a Project Managment Professional:

Oh, and about that exam -- you don't want to flunk it, do you? You better pick this up to help you succeed:

And now, we've gotten all the way to item 30 in a list of 6.225 items, just skimming the highlights of about one half of one percent of the list!

What can possible have gone un-said about that deep subject of software project management?

How about that it's a pernicious disease, and does more harm than good?

 

 

The Nightmare of Software Estimation

We make estimates of effort all the time. No big deal. But estimating software projects is a big problem, often a nightmare. Why should software be uniquely difficult?

Estimating effort is a reasonable thing to want

How long will it take you to get to the meeting? When can I have the summary report? How long will it take to crank out the press release?

We ask for and get estimations of effort all the time. It's a reasonable thing to want. If you think something will take a day and the person tells you it will take a week, probably something is wrong, and it's helpful to expose the issue and resolve it.

Estimating software effort seems like a  reasonable thing to want

If estimating effort before you start is generally reasonable, it would seem to be even more a good idea when it comes to software. After all, software projects tend to be expensive and take a long time. Worse, they have a nasty habit of running over time, over budget, and then not even giving you what you expected. So it would appear that creating detailed requirements and then providing careful time and effort estimates would be even more advised in software than in other fields. At least then we can track our progress see have a chance to fix things when they go off track.

Mainstream thinking in software management thoroughly endorses this view. The highest virtue a software effort can have, in this view, is being "in control." A software project effort is deemed to be "in control" when it is working its way up to creating estimates (i.e., gathering requirements, etc.), or logging work against estimates already made and tracking how well the estimates are being met.

There is a body of work on this subject that is both broad and deep. You may be impressed by this, but actually it's an indication of how difficult giving good estimates is, and how rarely it's accomplished. When projects go badly, get "out of control," the blame is usually placed on the process not being taken seriously enough. Inevitably, people with experience respond to bad experiences by laying on the process deeper and thicker, and making sure the estimates are long enough to account for all sorts of unforeseen events. In other words, even more time is spent on things other than actual programming, and the estimates are padded like crazy -- even more so than last time.

This, in a nutshell, is why programming groups get larger and larger; accomplish less and less; and never do anything new. It's why startups with bright programmers, unencumbered by the ball-and-chain of estimation and process, usually are the ones that accomplish the cool new stuff that programming groups with literally hundreds more programmers cannot.

Software Estimations are Different

The facts on the ground are that estimate-based programming leads to no good. How can we make sense out of this? Is programming really that different? Yes, programming really is way different than all those things that are reasonable to estimate. Here are some of the ways to understand this:

Huge variations in Productivity

Software estimates are in terms of the work -- not the person doing the work. But there are HUGE variations in the productivity of different programmers, more than 10X. So when you estimate that a given task will take ten days, is that 10 days of the 1X-er's time (the 10X-er will get it done in a day), or 10 days of the 10X-er's time (the 1X-er will take 100 days, and the 10-20% of programmers who are 0X-er's will never deliver)?

Skeptical of this huge variability? Here's a summary of the literature. McConnell concludes, "...the body of research that supports the 10x claim is as solid as any research that’s been done in software engineering."

Unknowns

What kills you with any estimate are the unknowns. The newer the kind of software you're developing, the greater the unknowns tend to be. Worse of all are the things you don't know you don't know. If estimating resolved the unknowns, I would warm up to it -- but in practice, estimating tends to make everyone feel better about how "in control" the project is, and delay the discovery of the unknowns until a later time, when they are even less convenient, a.k.a. "nightmares."

How do you discover unknowns sooner? Dive in and start doing the work sooner!

Requirements Evolution

You can only estimate if you know what you're building. The more exact the requirements, the better the estimate -- or so the theory goes. It's why everyone who's big into estimating is even bigger into requirements, and all sorts of squishy-sounding things like getting all the "stake-holders" on board.

In the real world, requirements change. Call it "evolution" if you want, but it always happens -- worse of all is when the real requirements in the real world are evolving while the project's requirements remain static -- in the name of maintaining the "integrity" of the software project, real world be damned! Only in academia (or silo-ed software departments of bureaucracies) are requirements immune to outside-world changes, discoveries made during the project, etc.

Learning by Doing

All programming starts with thinking. Thinking about the existing code, the tools, the architectural structure, the functionality that needs to be built and where and how you'll accomplish that functionality in the code. An integral part of this process is exploration -- filling in your knowledge of missing pieces as you think things through. In fact, it's hard to separate exploration from thinking; as you think, you discover holes in your knowledge, which you go off and fill, which helps flesh out and evolve your thinking. The more clear and complete your combination of thinking and exploring, the more cleanly, concisely and quickly you'll code it.

In practice, the best people write and change code as they go through this process. Nothing like trying to actually write the code to clarify the issues with writing it. The best way to discover road-blocks? Get on the road!

Do the people who create estimates do this? If not, what possible value is their estimate? If so, why don't they dive in and do the work, while everything is in the foreground of their mind?

Implementation paths

There is more than one way to peel an apple (what a stupid phrase -- who wouldn't want to just eat the darn thing, apple peels are really good!), and there is more than one way to get stuff done with computers. I'm in the middle of looking at some new functionality an Oak company is trying to build. Naturally, many people at the company want to take their existing, comfortable implementation path and apply it to the new problem. It's a good deal of work, involving a DBMS, stored procedures, a couple development environments, some business logic, a rules engine, a workflow engine and some UI designing. There is no doubt in anyone's mind, including mine, that the existing tool set can get the job done.

But there's another implementation path. The functionality is pretty simple, basically offering choices to consumers and getting them to fill in fields, with some branching and edit checking. The whole thing can be done using documents, XML and javascript, with a highly responsive UI. Taking this new approach would require a tiny fraction of the effort that using the existing implementation approach would require. Anyone involved can see it. It meets the inter-ocular test. Is it being warmly embraced by everyone at the company? Of course not!

A kind of estimating causes me to strongly prefer the new implementation path -- I'm sure it will be a lot less work, with results that will be more pleasing to consumers. A sense of estimation is indeed important, as one of the factors that helps you choose the optimal implementation path for getting a given job done.

Moore's Law

The software approaches and implementation paths taken by most companies for getting their work done were established many Moore's-Law-generations ago, when computers were 10, 30, or 100 times slower and/or smaller and/or more expensive than they are today. For example, most programmers continue to think in terms of databases, when Moore's Law has changed the game for databases and for storage.

It is far more fruitful to figure out how to take advantage of changes like this than to put effort into estimating the work using what are likely to be obsolete paradigms for building the software. And the best way to accomplish this is usually to try it.

Conclusion

Asking how long something will take to program seems so reasonable, so innocent. Who would ever guess that it tends to start a cascade of disastrous consequences?

The truth is that estimation does play a role in good software development, but in a completely subsidiary way, as a minor aspect of your technical and conceptual approach to building the software. It's like running a race: yes, you have in mind your past times, but if you ever get into making estimates and then padding them and then judging success by hitting your estimated time, you're going to lose the race. It's just not what winners do. Winners focus on ... winning!

Software Project Management: "Releases" are Stupid and Out-moded

Tell me, do you use Google by any chance? Yes? OK, now tell me which release of Google you use now, which ones you've used recently, and which one you like best; and, by the way, how was the upgrade process? Oh, you don't know? There's no way to find out, you say?

What the heck is WRONG with those people at Google? Don't they know that modern software development processes demand a disciplined release and planning methodology? There is NO WAY customers will put up with a product when the vendor just shoves new releases at them any time they feel like it, without even proper notification! And what customer is going to sign up with a vendor who won't commit to a future roadmap, with at least a year's worth of features laid out, tied to hard-commit release dates? That Google, they violate every rule of the game, there's no way they're going to make it as a software/service company!

What, how can it be! NOOOOoooooo.... Google is the world's most valuable software company!!?? When they don't even have the most basic element of proper software methodology, releases?? I must be sleeping! This must be a nightmare!!

Get over it!

Here are the facts:

• Classic project management has been a disaster for software development.
• All the heavy-weight process things that people do to make things better ... invariably makes them worse!
• The classic big-bang release is the cornerstone of the temple of evil.
• The classic little-bang release (a.k.a. "agile") is a brick in the temple of evil.
• "Releases" are ... stupid! ... and not only that, they're ... outmoded!

 There is life after "releases." It is a better life. The software is better. The users are happier. Go there. Enjoy it.

Software Project Management: Dates are Evil

The problem with estimates and dates in software is simple: nearly everyone wants them, and having target dates usually  makes things worse. Worse! Are there exceptions? Yes – but not many!

I’ve written agonizingly loooooong papers on this subject. But this is just a blog, so here is a bite-size thought to chew on.

Software estimates of dates might be a good thing if the thing we were estimating were something we have done over and over again, and had reasonable standards to apply. But we don’t.

Think of things like your commute to work, and the commutes of other people to the same place. Say you tell a friend your commute this morning took half an hour. Your friend will immediately be able to react to that information by saying, “whoa, was there an accident?” or “how come they didn’t catch you, lead-foot?” Your friend has a clear idea of the range of reasonable times for a “normal” commute, and can use that knowledge to judge on his own whether the one this morning was fast or slow, and by a lot or a little.

Now suppose a friend tells you that he's just finished designing a database, and that it took a week. Lots of people have designed databases. Say you’ve designed lots on your own. How do you respond? Is your friend a superman, for designing the database in record time, or a slug? The fact is, without knowing a whole lot about the details of the database to be designed, you have no idea. Even if your friend tells you some basics like the number of tables and columns, you still really have no idea. Was there a starting model? Are there special performance, size, interface or compatibility requirements? Were there a bunch of conflicting interests to account for? Depending on lots of details like these, even knowing the “raw” size of the database schema that was designed, you still have no way of judging.

Lots of interesting things come from this observation. One of them was taken up and exploited by the big services firms long ago. Here it is: the people who look at your project plan typically don’t have a clue – not even a foggy idea – of whether the times and resources assigned are reasonable. What the manager typically wants is confidence that, whatever resource and time commitments may be made, they are kept. So project management, in this context, is all about having lots and lots of highly technical looking stuff with charts and diagrams and fancy reports to justify what are – all too often -- completely outrageous resource and time commitments. No one looking at them has any way of knowing that they’re outrageous. So they get approved. The expectations are eventually met. And everyone’s a hero!

Speed vs. Predictability on the Race Track

Imagine we’re on a race track and there are some people running the 100 meter. Most of the people are outstanding runners, so they’re always competing with each other. In every race, all but one of them is a loser, and they tend to set aggressive goals of the times they’re going to make, which they frequently miss by a little.

The typical project manager would look at them as a bunch of unpredictable losers! They usually miss their goals, after all, and they practically always lose the contest they enter!

Now someone sidles up to the disappointed project man and says that he’s a different breed. “When I say I’m going to do something, I do it” is the line. So our runner says, see that 100 meter track over there? I’m going to go up to it within the next 15 minutes, and I’m going to tell you when I’m starting. You time me. I’m going to cross the finish line with 60 seconds. And I’m going to try to exceed your expectations.

Naturally, the project man is impressed. This is what he’s been looking for, a winner who does what he says he’s going to do.

Ten minutes later, the guy goes to the starting line, announces his start, and 55 seconds or so later, crosses the finish line, ahead of schedule! Wow! He started ahead of time and took even less time than he told me – this guy is a star!

Speed or Predictability: You Choose

What this boils down to is that you’ve got a choice. Which do you want: speed or predictability?

If you want predictability, dates are your friend. By all means, set a 60 second target for the hundred meter dash, and consistently beat your estimate. You’ll look like a hero to the ignorant, and you’ll do your part in flushing your business down the toilet.

If on the other hand you want speed, you will avoid setting date targets like the evil, mutilating plague they are. You will clearly define the target a hundred meters from here, and encourage your swift runners to start before the words get out of your mouth, and arrive yesterday. You may cause a bit of confusion among business people to whom this way of working seems strange and new, but they’ll quickly adapt as they see high spirits, pride of accomplishment … and results, the like of which they’ve never seen before.