The Black Liszt

Deep-Seated Resistance to Software Innovation

Everyone says they're in favor of innovation. Some organizations promote innovation with lots of publicity. Many organizations even have CIO's (Chief Innovation Officers) to make sure it gets done. But the reality is that resistance to innovation runs strong and deep in organizations; the larger the organization, the greater the resistance usually is. The reason is simple: innovation threatens the power and position of people who have it. They feel they have nothing to gain and much to lose.

It's not just psychology. Innovation resistance throws up barriers that are thick and high. See this for examples.

A good way to understand the resistance is look at sound technologies that have been proven in practice that could be more widely applied, but are ignored and/or actively resisted by the organizations that could benefit from them. I have called these In-Old-vations. Here is an innovation that is still waiting for its time in the sun, and here's one that's over 50 years old that is still being rolled out VERY slowly.

In this post I will illustrate the resistance to technology innovation in a little-known extreme example: the people in charge of Britain's war effort resisted innovations that could help them win the war. They were literally in war and losing and decided, in effect, that they'd rather lose. Sounds ridiculous, I know, but this is normal behavior of people in organizations of all kinds.

The Battle of Britain

Britain was at war, facing Hitler’s much larger, better-prepared military, who had already rolled over its adversaries. Life and death. Literally. The established departments did all they could do defend from attacks. The so-called Battle of Britain is well-known. What is not as widely known is the battle in the seas. German submarines were sinking British ships at an alarming rate. The Navy had no answers other than to do what they were already doing harder.

The situation was desperate. If there was ever a time to "think outside the box" it would seem this was it. The response of the Navy to new things? NO WAY. Amazing new weapons developed by uncertified people outside the normal departmental structures? NO WAY. Once those weapons are built and proven, use them to stop the submarines that were destroying boats and killing men by the thousands? NO WAY!!

Of course, you might think that someone would have known that the fairly recent great innovation in flying machines was achieved by "amateurs" flying in the face of the establishment and the acknowledged expert in flying as I describe here. You might think that Navy men would remember that perhaps the greatest innovation in naval history was invented by Navy-related people. But no. Protecting our power and the authority of our experts is FAR more important than a little thing like losing a war!

The story of the new way to fight submarines is told in this book:

Someone who was not part of the Navy establishment invented a whole new approach to fighting submarines. The person wasn't a certified, official expert. He was rejected by all relevant authorities and experts. Fortunately for the survival of England, Churchill made sure the concept was implemented and tested. The new devices were delivered to a ship.

This all took time and it was not until the spring of 1943 that the first Hedgehogs were being installed on Royal Navy vessels. When Commander Reginald Whinney took command of the HMS Wanderer, he was told to expect the arrival of a highly secret piece of equipment. ‘At more or less the last minute, the bits and pieces for an ahead-throwing anti-submarine mortar codenamed “hedgehog” arrived.’ As Whinney watched it being unpacked on the Devonport quayside, he was struck by its bizarre shape. ‘How does this thing work, sir?’ he asked, ‘and when are we supposed to use it?’ He was met with a shrug. ‘You’ll get full instructions.' Whinney glanced over the Hedgehog’s twenty-four mortars and was ‘mildly suspicious’ of this contraption that had been delivered in an unmarked van coming from an anonymous country house in Buckinghamshire. He was not alone in his scepticism. Many Royal Navy captains were ‘used to weapons which fired with a resounding bang’, as one put it, and were ‘not readily impressed with the performance of a contact bomb which exploded only on striking an unseen target’. They preferred to stick with the tried and tested depth charge when attacking U-boats, even though it had a hit rate of less than one in ten. Jefferis’s technology was too smart to be believed.

Here's what the new mortars looked like:

What happened? It was transformative:

Over the course of the next twelve days, Williamson achieved a record unbeaten in the history of naval warfare. He and his men sank a further five submarines, all destroyed by Hedgehogs. Each time.

If resistance to change to true technological innovation is so strong when you’re desperate, literally at death’s door, how do you think it’s going to be in everyday life? The rhetoric is that we all we love innovation! The reality is that anything that threatens anybody or anything about the status quo is to be ignored, shoved to the side and left to die. Anyone who makes noise about it obviously isn’t a team player and should find someplace to work where they’ll be happier. And so on.

Conclusion

Innovation happens. Often nothing "new" needs to be invented -- "just" a pathway through the resistance to make it happen. Here is a description of the main patterns followed by successful innovations. If you have an innovation or want to innovate, you should be aware of the deep-seated resistance to innovation and rather than meeting it head-on, craft a way to make it happen without head-on war. Go for it!

The Fundamentals of Speed-optimized Software Development

How do you write software more quickly? How do you engage and win with Wartime Software? There is a set of techniques that enable dramatic speed-ups in producing effective, high-quality software, and they share a set of underlying principles. These fundamental principles of software development are practically never discussed, yet they are almost painfully simple. Understanding and applying them can help you adapt existing speed-optimized methods to your needs and evolve new ones.

Mainstream thinking about speed

Very few groups optimize for speed in software development. By far the most common factor that is optimized is expectations. Given that everyone involved in software knows about, believes in and practices expectation-based development, the options for speeding things up aren't great.

The most widely used method for speeding things up is simple: pressuring everyone involved to go faster. Do whatever it takes, darn it, we're going to bring this project in on time! What this amounts to is:

• People work harder, longer hours and extra days. Result: sloppiness, lack of care, increasing number of mistakes and omissions, greater risk.

• People squeeze or omit essential steps, particularly later in the project. Result: poor quality and even disasters encountered during roll-out.

• Fashionable new methods are introduced that re-arrange things or add steps. Result: see above.

The fundamentals of achieving speed

The fundamental principles of achieving speed in software are extremely simple: take fewer, more effective steps to the goal and eliminate overhead. When framed in all the stuff we know about software development, this can be abstract and difficult to apply in practice. But it's truly simple. Think about walking to a goal. The principles amount to:

Walk to the destination as directly as possible; avoid stopping and re-tracing your steps.

That's not so hard, is it?

I know, it is hard, because standard software development methods are worlds away from thinking about things in these terms. But that's because they're optimized for expectations and not for speed!

Replacing lots and lots with a little

Most of the steps in mainstream software development are there for a good reason. While concepts like project management may have been developed outside of software, they have been adapted to it. So if you're going to toss out some standard expectations-based activity, you'd better be sure there is speed-optimized activity that gets the same job done -- at least as well, and with lots less time and effort.

With all the effort and work that's gone into software development methods over the years, there's not a lot of improvement to be made by tweaking this or enhancing that. Radical methods are required to accomplish radical change. So it's important to ignore all the intermediate steps and focus on the only goal that matters: customer satisfaction. Period. Everything else is a stand-in, an intermediary, or supposed to contribute to that end.

If this is the goal, one of the fundamental principles of Wartime Software, of optimizing for speed and results, is this:

DO globally optimize for the goal.

DO NOT sub-optimize for any intermediate goals.

Once you globally optimize for reaching your goal, then you realize that there is a HUGE amount of redundancy in the normal "rigorous" software development process. The redundancy isn't obvious or even visible because mostly it's many repetitions with slight variations, each confined in a silo, performed by different people in different environments using different tools. So it doesn't look redundant, when examined in isolation. But when viewed globally? MASSIVE redundancy, and an opportunity to replace lots and lots of overlapping stuff with a little bit, done right.

Conclusion

Wartime software is not achieved by doing a re-organization or tweaking a process. It's achieved by a total re-conceptualization of the optimal way to get from Point A to Point B in software. Once you understand the fundamental principle of global optimization, with the goal of eliminating overhead and redundancy, all the detailed speed-optimized techniques follow like Q.E.D.

Wartime Software Book Available

I've been threatening to release my book on Wartime Software. It is now available as a Kindle book.

Wartime Software is all about writing software when competition and speed matter. It's about releasing more often. It's about using new methods, as different as building bridges in peacetime and in time of war.

Here is the introduction, which should give you the idea.

Most people assume there is one “right” way to build software, and that’s that. While there are various fashion trends that infect software from time to time, none of them are as different as they like to think they are.

There are some important but little-discussed facts about the mainstream consensus of software development:

• It is mostly organized to give non-technical people confidence that things are OK, meaning on-time and on-budget. Its highest principle is predictability. Not speed.
• It mostly doesn’t work. Studies support what everyone in the field knows: most projects fail outright, or have their goals changed to avoid admitting failure.

So what we have are methods that are slow – and produce crappy results! What happened to slow but sure, or slow but steady? What we’ve got is slow and stupid.

If everyone you compete against uses the same crappy methods, you’ll be OK. Your projects will be perpetually late and disappointing, but so will everyone else’s, so you’ll be performing “up to standard.”

But what if you’re not? What if you’re competing against a group that gets way more done in much less time? I’m not talking 10 or 20% here; I’m talking many whole-number factors, like 10, 50 or more. What’s going to happen? It’s simple: you’re going to lose! If that’s OK with you, stop reading right now, close your eyes, and get lost in your muzak. You’ll be happier.

If your goal is to learn the standard, accepted techniques of software as widely practiced, don't waste your time with this book. But if you're pioneering or really under the gun and need to find a way to program the way software ninjas program, you'll find some useful information in this book.

Wartime Software: Optimizing for Speed

Software Development is a mission-critical issue for increasing numbers of organizations, particularly the growing number of "software-enabled service" organizations. Which makes it all the more surprising that there is a lack of concensus about to best do it.

I've written about software development quite a bit on this blog. Now, I'm in the final stages of preparing my small book on Wartime Software Development for publication as an inexpensive Kindle book. This post about bridges in war and peace gives some of the flavor. 

Wartime Software is all about optimizing the process for speed instead of predictabillity. Here's a short excerpt from the book about what optimizing for speed really means.

The usual procedures for producing code are supremely arrogant. They are arrogant because we decide that we can figure out what the customer wants, and the customer should simply wait while we “get it right.” We’re so sure that we know what the customer wants that we build it, and not just any old way, but we build it industrial strength, loaded up with piles of documentation, test plans for every little jot and tittle, so that when we (finally) roll it out, it’s on silver platters and with bands playing, with code ready to stand the test of time…and sadly, all too often, we’re wrong! We’ve misunderstood the customer, built things they don’t want, failed to build things they do want, built some things they need in confusing, incomplete or simply perverse ways. We frequently spend a year solving last year’s problem, and when we deliver our well-intentioned mess next year, the customer and the market have moved on and sometimes our competitors have leapfrogged us. Most software projects resemble your worst nightmare of a pork-barrel politics public works project, like the “bridge to nowhere,” the project in Alaska that projected nearly $400 million to build a bridge as long as the Golden Gate bridge and higher than the Brooklyn Bridge to Gravina Island, an island with only 50 residents, no stores, no restaurants and no paved roads. Who cares how well the bridge was designed?

The design of the bridge (or the software) is not the most important thing – the most important thing is the unmet needs of the people who will use the thing you intend to build. And so the number one priority is to discover what those needs are, from the only authoritative source. And by the way, the customer’s opinions may be more relevant than your opinions, but they are not truly authoritative – only the customer’s actions are authoritative.

And that means that you have to find a way to write code really quickly, so that you can turn your ideas (that hopefully you’ve mostly stolen from customers or other successful services) into services, modify them quickly based on customer feedback, and either discard them and move on, or evolve them until you’ve improved your service, using the real actions of real customers at every step of the path to make your critical decisions. You have to optimize all your processes for speed in order to pull this off.

And remember – if you’re not doing things this way, you’re probably building a software “bridge to nowhere.”

The Dirty Secret of Peace-time Software Development

We use a large number of intimidating words and abstruse concepts to make our software development methods sound like they're highly evolved and refined. But all too often they take way too long to produce software that doesn't work and isn't what we need.

The Propaganda

The normal, "peace-time" process of software development sounds pretty deep. There are requirements.

There are designs.

There is all sorts of testing to assure quality.

That's just for a start -- there's loads more.

And on top of it all, there are levels and levels of analysis to assure you've got a repeatable process that is documented, measured, and continuously improved.

The Reality

When we try to create software in this way, we resemble Dr. Frankenstein in his laboratory,

with his carefully crafted plans to bring something into being that has never before existed.

And when you've done all that impressive-sounding stuff, what do you have???

This...

is what you have.

He's late.

He cost way too much.

He doesn't work.

And worst of all...

...he's not what you wanted in the first place!

Conclusion

The reality is all those software methods we argue about amount to pretty much the same thing. They are all "peace-time" software methods. They promise to be careful and deliberate. They promise to deliver safety and predictability. But they can't! Which explains why, in the vast majority of cases, they don't! That's the dirty secret of all those high-minded concepts and abstruse words -- they put a high-minded gloss on incompetence and ineptitude.

Bridges and Software in Peace and War

We build bridges in times of peace. They take a long time to build; they tend to last a long time, but sometimes they crash. We also build bridges in war-time. Working in the face of enemy fire, they get built really quickly, and tend to serve the purpose well.

What is war-time software? Are there methods that enable us to build software in a fraction of the usual time in highly competitive circumstances, while still serving its purpose well? The answer is yes.

Peace-time bridge building

The bridge over the Firth of Forth in Scotland was the world's first major steel bridge. It took about seven years to build, was completed in 1890, and is in use to this day.

(Credit)

As many as 4,000 men worked on the bridge at a time, with 57 losing their lives.

The Golden Gate bridge in San Francisco is more recent, having been completed in 1937 after about 4.5 years of work.

(Credit)

Peace-time bridge building: the results

I've given just a couple of examples, but they are typical: bridges take years to build in peace-time, and people die while building them. And while we expect them to never crash, in fact they do. It's not as rare as you may think! Here's a collapse of a bridge in Canada in 1907 killing 95 people:

The Silver Bridge was built in 1928 over the Ohio River. Here is in when it was completed.

It collapsed in 1967 during heavy rush hour traffic. 46 people were killed.

And here's a portion of the Route 95 in Connecticut that collapsed in 1983:

There are many more examples. Peace-time bridges take years to build and are expected to work without problems, but in fact they sometimes collapse and kill people.

War-time bridge building

Building bridges in war time is a whole different matter. The bridges aren't allowed to collapse and kill people any more than those built in peace-time, and the loads they're required to carry can be much greater. Frequently they are built under enemy fire. Yes, they look different and are constructed using different techniques:

(Credit)

But that's the whole point. The time constraints are severe: instead of years to build a bridge, it must be done in days.

Here's the story of the bridge pictured above:

It was during this week, in late March of 1945, that the U.S. Third Army under Gen. Patton, began its famous bridging and crossing operations of the Rhine.

The first unit to cross was the 5th Infantry Division that used assault rafts to cross the raging Rhine ... in the early morning hours of March 23. ... By 1880 that evening, a class 40 M-2 treadway bridge was taking traffic. The following day, a second 1,280 foot class 24 bridge was completed in the same area. It was later upgraded to a class M-40 bridge. Without the benefit of aerial bombardment or artillery preparation, units landed quickly and established a beachhead that was seven miles wide and six miles deep in less than 24 hours...When daylight came, the Luftwaffe attacked the enclave with 154 aircraft in an attempt to dislodge the foothold on the east bank. Effective anti-aircraft fires brought down 18 of the attacking planes and destroyed 15 more.

By March 27, five divisions with supporting troops and supplies had crossed the three bridges constructed at Oppenheim. The entire 6th Armored Division crossed in less than 17 hours. During the period of March 24-31, a total of 60,000 vehicles passed over these bridges.

Peace-time software

Most of the software built today is built using "peace-time" methods. Those methods are so ubiquitous that they are simply considered to be "the right way to do things." We document everything. We have a nicely, orderly flow from requirements through design, coding, testing and deployment. Whether waterfall or "agile" is used, everyone is given time to do their job, and frequently asked how long it will take. Estimates are critical, and the most important thing is delivering on the expectations you set.

In this environment, it's important to make sure your estimates are long enough to account for things you forgot about. Taking a long time to get a job done isn't a problem; taking longer than you said you would take is the problem.

War-time software

So what is war-time software? It's a looooong subject, and can't be done right in a short post. But the principles should be obvious from the bridge-building metaphor:

• Time is the most important thing; if you take a year to do what the other guy gets done in a month, you've lost the war.
• Solving immediate problems is far more important than effort put towards some imagined future.
• Something is better than nothing.
• Finding and fixing problems is more important than preventing them.
• Did I mention that nothing is more important than speed, except possibly avoiding getting killed (usually)?

Those war-time bridge-building guys made it up as they went along, but they couldn't have done it without an elaborate tool kit, appropriate supplies and matching skills and procedures. The scene on the river may seem chaotic, but there's a pattern and lots of coordinated activity, with everyone working towards a common goal: the least they can possibly do that gets things safely over the river. When is peace-time software ever subjected to that kind of parsimonious discipline?

War-time software development is development that is organized and optimized for speed: getting the least acceptable solution built and deployed in the shortest possible amount of time, and rapidly iterating from there. And then doing it again. Obviously, you spend time gathering and organizing your supplies and improving your technique.

War-time software is not doing things the usual way, only skipping steps, doing things sloppily and writing half-done crap code. That's doing a bad job using peace-time methods. War-time software is doing things in a war-time way using war-time techniques.

Conclusion

Are you truly operating in peace-time? Is your competition frozen? Do you have no time constraints or money limitations? Then, by all means, continue to use peace-time software methods -- take huge amounts of money, incredible amounts of time, document and plan and manage everything with precision, and build your software. Software that will crash when you least expect it.

If, on the other hand, you are at war, and if you, you know, want to, like, survive -- well, you may want to consider building software that actually meets the immediate need.

 

Find a way to get that data, those screens and workflows over that threshold, soldier. Now! Yes, I know that in your previous life, it would have taken you a week to write a proposal for creating a plan to get it done. These screens, workflows and databases are going to be on the other side of that threshold in under a week, while enemy forces and programmers are doing their best to kill us in the marketplace. Move it!

War-time software. It's the way to win.