Outstanding software development groups are truly outstanding. In many sporting events, the winner is often determined by a margin of a fraction of a percent. In software, the best teams are at least 10X more productive than the competition, and sometimes 100X. Here's the most amazing thing of all: the way the super teams do it is not secret!
Discovering the secrets
For most of my software development career I was only vaguely aware of productivity differences. My job was rarely to judge and evaluate -- it was to get stuff done, quickly and well. I figured out some basic stuff to the extent it helped me get stuff done, and hired people who would contribute. I quickly realized that education was irrelevant, as was experience in a particular technology or tool set. What mattered most was raw horsepower, ability to concentrate, willingness to work and the drive to learn.
I think my first wake-up call was a consulting project I did for Sallie Mae during the early 1990's, when I was one of three groups advising the company on a massive re-engineering project using document imaging and workflow technologies. One of the groups was me; the second was a local group with 5 people on the project full time; the third was a national consulting firm that had at least 15 people on the project full time with various senior people and "experts" floating around. I was the only person involved with any of the groups who had not only used the technologies, but created them.
To make a long story short, the other groups performed standard analysis and vendor evaluation, and proposed an expensive process that would result in a 30% productivity gain, if all went well. I dove into the actual work Sallie Mae was doing, noticed some things about the work, and proposed an inexpensive process that would result in a 5X productivity gain. I called the method I used "zero-based re-engineering," combining two fashionable-at-the-time concepts, "zero-based budgeting" and "process re-engineering." Did it catch on? Did the dramatic advantages of the approach lead to widespread use? Nope.
It took years for me to figure out that what I had experienced with Sallie was the tip of the iceberg, both with giant bureaucracies like Sallie Mae and with the "technology expert" consulting firms.
It's developer's gold -- but you can't give it away
About ten years later (yes, I know, I'm slow), I started noticing similar stark contrasts among groups of technology companies and the way they built software. The things I noticed were at heart simple -- but compared to existing practice, radical. And I now finally realize, revolutionary. Man, I'm really getting the formula for the secret sauce here! I've got to be really careful, and make sure I only whisper these secrets to our portfolio companies. I can't let them get out; if one of our company's competitors got wind of the methods and rolled them out before we did, we'd get killed.
So for the first couple of years, I was really careful with these "secrets." It was only spoken, never written, and only on a need-to-know basis. Some of the companies I would visit wouldn't say they had "secrets" -- but they did things differently from most groups, and I eagerly learned from then. I'd visit other companies which could really benefit by employing one or more of these "secrets." I whispered to the smart, motivated people who are leaders in companies in which we've invested where the gold is -- and it's really close, right over there!! -- and they're ho-hum, thanks a lot, so glad you let me know, please tell me when you plan to visit again, good bye. In other cases, there was apparent enthusiasm, followed by nothing. I thought the problem might be understanding. So I followed up the verbal with e-mails to solidify the ideas.
Learning more and revealing the secrets
Meanwhile, I'm getting a broader knowledge of how the super-developers get it done. It appears there's a spectrum of techniques. Some people use lots of one and little of another, but they all optimize for speed and quality instead of expectations.
I did notice that putting the ideas in writing helped. So I graduated from over-long emails to PDF papers. I told people the ideas, then gave them the papers. Or the other way round. Having the papers seemed to increase the chances that our companies would take the ideas seriously. I kept augmenting and editing the papers as I learned things, and peppered them with real-life examples. Some of the papers went through a couple dozen editions. By about 5 years ago, I had over ten papers comprising over 800 pages.
Why won't people grab the gold?
As I saw more examples of great results, I become ever-more mystified about the laggards who just don't get it.
Now, to be clear, these "laggards" are so only in relative terms. Most of them are way smarter and more accomplished than their compatriots are in the giant legacy companies. In the eyes of the world, the vast majority are leading edge. I started noticing that what I saw as laggards were nearly universally practicing methods that the software industry as a whole has labelled "leading edge," things like unit testing, Extreme programming or Agile. So on a bell curve of established practices, they were definitely leading edge.
My dim light bulb finally started flickering on. Almost no one decides important things like software methods on their own! They pick whatever brand of recognized external authority they're most comfortable with and which best matches their self-image, and try to follow the established method.
Understanding the resistance
But these are programmers, darn it! The methods aren't exotic, requiring a thorough grounding in quantum string theory -- in most cases, they're simpler than the standard stuff. So where does the reluctance come from?
This is a BIG subject. There are lots of aspects of it, and I'll return to the subject in future posts. But the bottom line is simple: if programmers want to advance in their careers, for the most part they have to accede to the demands of non-technical managers. The non-technical managers have no clue what programming is about, except that it's full of incomprehensible gobbledygook. Understandably, they want to feel confident of what's going on in the department they're in charge of, most of whose people are engaged in activities that are opaque to them. So they absolutely require the use of standard methods, measures and goals that are entirely consistent with other disciplines in which things get built, like buildings and cars. Anything else feels like a "lack of discipline."
Most of the "secret" methods did not follow the "makes managers comfortable" formula. So the vast majority of organizations would reject their use, regardless of their provable power and effectiveness. For me, keeping things secret was not the problem -- it was getting the ideas acted on. There are indeed amazing methods used by software super-developers to get things done 10X or more better than anyone else. But because the methods are not among those talked about by the public "experts" in the industry and do not resemble how houses or cars are built, they aren't a candidate for being used by the vast majority of developers, who do not make decisions for themselves based on substance.
Going fully public
That's when I decided there was no harm and possible benefit in going fully public with the things I'd been noticing over the years. That's when I decided to fix up the papers, turn them into books, and publish them in paperback and on Kindle. Here is a little more background, some illustrative blog posts and links to the books that I've already published.
I am working on book versions of the remaining papers, some of which are more revolutionary than what I've already published. Going through the process of crystallizing what I've learned over the years has taught me a great deal about knowledge itself and innovation, since part of what I've done is observe dramatic advances and seen how the knowledge is acquired and spread.