The Black Liszt

Summary: Wartime Software to Win the War

The vast majority of software development is organized to deliver software that meets a set of requirements on time and on budget. In other words, it’s organized to meet expectations.

What if you don’t have the time or money to do this? What if you’re focused on a need unmet by existing software and you’re driven to create software to meet that need? This is wartime software – get it done or lose the war. It’s optimized for speed, with customer satisfaction and unfailing quality as a given.

In this summary, we’ll cover relevant blog posts and books.

No matter how you do it, there are things about the software development process that are common.

https://www.blackliszt.com/2013/03/software-development-process-simple-terms.html

How do you measure success in software development? Agreeing on the measure of success is crucial to achieving it.

https://www.blackliszt.com/2013/03/software-postulate-the-measure-of-success.html

Why is so much software so bad? there are multiple reasons.

https://www.blackliszt.com/2011/06/why-computer-software-is-so-bad.html

https://www.blackliszt.com/2022/04/how-to-fix-software-development-and-security-a-brief-history.html

The near-universal focus on procedural language is the most important underlying cause of bad software development. It's not something a few tweaks can fix.

https://www.blackliszt.com/2024/08/why-is-writing-computer-software-dysfunctional.html

Peacetime Software Development

Peacetime software optimizes for predictability, i.e., delivering the requirements on time and budget. Wartime software optimizes for speed.

https://www.blackliszt.com/2010/10/software-project-management-dates-are-evil.html

Being on time and budget means making an estimate, which experienced people know is nearly impossible.

https://www.blackliszt.com/2012/06/the-nightmare-of-software-estimation.html

Estimates are also hard because software productivity is hard to measure.

https://www.blackliszt.com/2012/09/what-is-software-productivity.html

Whatever the productivity is, doubling the amount of work to be done using popular peace-time development techniques is sure to make it worse.

https://www.blackliszt.com/2012/09/software-productivity-divisors.html

Peacetime development involves loads of activities that aren’t programming.

https://www.blackliszt.com/2012/09/software-productivity-the-abp-factor.html

Project Management is supposed to assure that software development is “under control.” It’s actually a method to assure that programmers spend as little time as possible programming.

https://www.blackliszt.com/2012/10/software-project-management-book.html

https://www.blackliszt.com/2012/10/the-disease-of-software-project-management.html

The vast majority of software development bears a striking resemblance to the meticulous planning and building of Frankenstein’s monster.

https://www.blackliszt.com/2012/04/the-dirty-secret-of-peace-time-software-development.html

You get to near the end of a software project that has been “going well” and you suddenly discover the disaster.

https://www.blackliszt.com/2014/06/building-software-the-bad-old-way-and-the-good-new-way.html

With the usual measures, standard methods deliver terrible results.

https://www.blackliszt.com/2011/07/software-quality-theory-and-reality.html

The normal methods make common sense, like planning and making a Thanksgiving meal – and usually yield the classic mess you have after the meal.

https://www.blackliszt.com/2011/11/thanksgiving-meals-and-software-turkeys.html

Most peacetime software managers know they should act like they’re modern, criticizing old-fashioned waterfall and embracing Agile. Little change in results.

https://www.blackliszt.com/2013/03/software-comparing-waterfall-and-agile.html

Classic software development efforts focus on process instead of the substance of the software itself.

https://www.blackliszt.com/2013/03/process-and-substance-in-software-development.html

Most important decisions about software aren’t made by programmers, which is a big factor in things going wrong.

https://www.blackliszt.com/2014/01/who-makes-the-software-decisions.html

https://www.blackliszt.com/2014/10/joe-torre-and-software-development.html

Standard development embraces the metaphor of “building” software like a house, imagining that there’s a time when it’s done.

https://www.blackliszt.com/2013/10/when-is-software-development-done.html

Another factor in building software is “just” having competent people who do their jobs.

https://www.blackliszt.com/2014/04/delivering-software-is-a-nightmare.html

Why do all the experts cling to the expectation-based approach to software? The history of scurvy is a close parallel to software development diseases.

https://www.blackliszt.com/2014/02/lessons-for-software-from-the-history-of-scurvy.html

A major issue with expectation-based software is incentives. Among other things, when you’re judged on expectations, you’re incented to make huge estimates of the amount of work.

https://www.blackliszt.com/2015/04/software-problems-the-role-of-incentives.html

You might think that because software is related to math that it’s precise. There are huge factors like the exponential growth of computer power that are ignored in Computer Science and peacetime development.

https://www.blackliszt.com/2015/05/math-and-computer-science-vs-software-development.html

There are many paths people take to feel better about software development; they may sell well at the beginning but make things worse.

https://www.blackliszt.com/2015/06/software-problems.html

Software development is bad, and what makes it even worse it that people try to hide the failures, re-brand failures as success, and above all avoid tracking the outcomes.

https://www.blackliszt.com/2024/08/what-is-your-software-project-managers-batting-average.html

Wartime Software Development

The essence of wartime software development is that everything is optimized for speed of development.

https://www.blackliszt.com/2013/05/wartime-software-optimizing-for-speed.html

Great software development resembles growing a baby.

https://www.blackliszt.com/2011/06/software-how-to-move-quickly-while-not-breaking-anything.html

Just like with growing a baby, wartime software can’t be allowed to die.

https://www.blackliszt.com/2012/04/field-tested-software.html

A good way to see the contrast between the mainstream way of software development and the method used by software ninjas is to compare building bridges in peacetime and war time.

https://www.blackliszt.com/2012/03/bridges-and-software-in-peace-and-war.html

When you focus on speed, there is a simple concept that explains how it’s better.

https://www.blackliszt.com/2015/12/the-fundamentals-of-speed-optimized-software-development.html

https://www.blackliszt.com/2015/10/building-better-software-more-quickly.html

Is wartime software development like regular development only faster with fewer distractions? No. There are also essential differences in method and code standards.

https://www.blackliszt.com/2020/02/how-to-build-applications-that-can-be-changed-quickly.html

If software is always being changed, you want to make change easy. It’s what the average lay person wants from software.

https://www.blackliszt.com/2022/10/how-to-improve-software-productivity-and-quality-common-sense-approach.html

Here’s the idea of code without redundancies.

https://www.blackliszt.com/2020/06/the-map-for-building-optimal-software.html

While the software world may not have caught on to the value of no redundancy, the ordinary world of machine design is all over it.

https://www.blackliszt.com/2020/05/lessons-for-better-software-from-washing-machine-design.html

Here’s a broader exploration of focusing on redundancy.

https://www.blackliszt.com/2014/03/how-to-evaluate-programming-languages.html

Technical debt should mostly mean too much redundancy.

https://www.blackliszt.com/2020/03/how-to-pay-down-technical-debt.html

Technically, the way to eliminate redundancy is to move concepts from code to metadata.

https://www.blackliszt.com/2022/10/how-to-improve-software-productivity-and-quality-code-and-metadata.html

More about how to make quickly-changeable code.

https://www.blackliszt.com/2022/05/how-to-improve-software-productivity-and-quality-schema-enhancements.html

Code without redundancy is simple. It’s an ancient idea.

https://www.blackliszt.com/2020/03/william-occam-inventor-method-for-building-optimal-software.html

Here’s a summary of the method and techniques.

https://www.blackliszt.com/2022/09/better-software-and-happier-customers-with-post-hoc-design.html

Here's an example of how one group evolved from trying to improve quality and availability to wartime software:

https://www.blackliszt.com/2024/06/the-amazing-path-to-high-quality-fast-to-change-software.html

Of course, following good business and product strategies is an essential aspect of wartime software -- among other things, carefully selecting customer focus is key, Like conquering Europe in World War 2 by landing on a small number of beaches in Normandy instead of invading the whole coastline.

https://www.blackliszt.com/2023/07/summary-software-innovation.html

Wartime Software Book

Here is a brief introduction with links to my book on Wartime Software.

https://www.blackliszt.com/2013/05/wartime-software-book-available.html

 

What is your Software Project Manager's Batting Average?

How often do software projects fail? While it’s never talked about and rarely makes the news, the fact is that software projects fail with alarming frequency. Exactly how often do they fail? The shocking answer is, despite there being august professors of Computer Science and Computer Engineering falling out of the woodwork, doing research, pronouncing their deep knowledge in papers and teaching the next generation of students, the truly shocking answer is that no one knows! Or cares! Oh, they say they care about teaching people how to create great software, all you have to do is learn the intricacies of Object-Oriented Programming (or whatever). But none of them studies the results. There is no science in what is called “computer Science.”  They all declare they know how to create good software – so long as no one takes the trouble to measure and count, judge and evaluate, to see how often it happens.

In the past, I’ve compared bridges falling with software failures. We can understand this from another angle by looking at … baseball.

Baseball Science

Aaron Judge is an amazing professional baseball player, part of the New York Yankees. He’s an excellent fielder, but he’s best known as a hitter. In his nine seasons so far, he’s already hit over 300 home runs, with 50 so far as of this writing. His batting average this season is an awesome .331 so far, with a .288 average for his career.

Let’s think about that enviable .331 batting average he’s got so far this year. Any player would love to have such an excellent average. But that’s in the context of professional baseball. Simple arithmetic tells you that this “excellent” average means that he only gets a hit in one third of his at-bats! More than two thirds of the time he strikes out or otherwise fails to get a hit! What would you think of someone who managed to drive safely to work just a third of the time, getting into accidents or otherwise screwing up two thirds of the time? What would you think of a car that only worked a third of the time you tried to drive it? And so on … you get the idea.

Why is this? Are all these highly paid pro ball players really losers? Of course not. You can see what’s going on if you watch the home run derby, when top hitters go the plate and get soft, easy pitches from a pitcher standing behind a screen so they don’t get hit. Those guys nearly always hit the ball, and lots of them are home runs! But we all know that’s a once-a-year event.

Most of the time, there’s a highly skilled pitcher on the mound whose goal in life is to strike out the batter or sucker him into hitting an out. Pitchers like Gerrit Cole, who is so good that he’s being paid $36 million dollars this year.

When you’re a batter and walk up to the plate facing Gerrit Cole, you know you’ve got someone who’s in an excellent position to lower your batting average.

So is creating software like trying to get a hit with a fearsome skilled pitcher on the mound? Or is it more like trying to drive your car to work or designing and building a reliable car that just works, with remarkably few exceptions?

The sad fact is that building software resembles the home run derby, except that instead of trying to hit the ball out of the park, all the batter has to do is … not miss. With a rule like this, you’d expect something close to a 1.000 batting average. They try to make it even easier in the software world by messing with requirements, tripling estimates and doing everything they can to make the project a “success.”

Is software really that bad? Yup. Just for fun, I’m going to share a couple of the rarely publicized stories of software failures I made note of a dozen years ago. With things like ransomware exposing the ugly underside of most software operations -- not to mention the awfulness of computer security in general -- you can be sure things haven’t gotten better.

Sample Failures

In 2014, the VA admitted that they have over 57,000 patients waiting for their fist visit. What's their excuse?

"The official also said the VA needs to update its scheduling software package, which the department has been using since 1985. “It predates the internet and Blockbuster’s rise and fall,” he said."

Does that count as a software failure? It's more like the software department died years ago and no one noticed.

Here's a good one from 2012:

The U.S. Air Force has decided to scrap a major ERP (enterprise resource planning) software project after spending US$1 billion, concluding that finishing it would cost far too much more money for too little gain.

Dubbed the Expeditionary Combat Support System (ECSS), the project has racked up $1.03 billion in costs since 2005, “and has not yielded any significant military capability,” an Air Force spokesman said in an emailed statement Wednesday. “We estimate it would require an additional $1.1B for about a quarter of the original scope to continue and fielding would not be until 2020. The Air Force has concluded the ECSS program is no longer a viable option for meeting the FY17 Financial Improvement and Audit Readiness (FIAR) statutory requirement. Therefore, we are cancelling the program and moving forward with other options in order to meet both requirements.”

The Air Force will instead need to use its “existing and modified logistics systems for 2017 audit compliance,” the statement adds.

They started spending money in 2005, spent over a billion dollars by 2012, got nothing of value, estimated they'd need to spend the same again for eight more years to get about a quarter of the original plan done. If there were anyone paying attention to batting averages in software, that would likely be a winner.

Here is rare-to-find information on the frequency of failures from a book.

The odds of a large project finishing on time are close to zero. The odds of a large project being canceled are an even-money bet (Jones 1991).

In 1998, Peat Marwick found that about 35 percent of 600 firms surveyed had at least one runaway software project (Rothfeder 1988). The damage done by runaway software projects makes the Las Vegas prize fights look as tame as having high tea with the queen. Allstate set out in 1982 to automate all of its office operations. They set a 5-year timetable and an $8 million budget. Six years and $15 million later, Allstate set a new deadline and readjusted its sights on a new budget of $100 million. In 1988, Westpac Banking Corporation decided to redefine its information systems. It set out on a 5-year, $85 million project. Three years later, after spending $150 million with little to show for it, Westpac cut its losses, canceled the project, and eliminated 500 development jobs (Glass 1992). Even Vegas prize fights don't get this bloody.

If you care to look, you will find loads more examples of failures the group has been unable to keep secret. The failures keep rolling in spite of the huge efforts to reduce requirements, inflate estimates, extend time lines, increase staff and everything else. You have to ask the question::how many software successes are really failures in disguise? If anyone were serious about calculating software batting averages, this would be a key factor.

This pattern has resulted in some fairly widespread humor that you can be sure isn't mentioned in project management meetings. For example, here are the stages of a software development project:

1. Enthusiasm
2. Disillusionment
3. Panic and Hysteria
4. Search for the Guilty
5. Punishment of the Innocent
6. Praise and Honor for the nonparticipants

Why do Software projects fail and how do you win?

When lots of human beings work at something for a long time, they tend to figure out how to do it. Building software appears to be a huge exception to that rule. With decades of experience under our belt, why is software the exception?

This is a long subject. I have gone into great detail spelling out the causes ... and the cures!

Start with history and evolution:

https://www.blackliszt.com/2023/08/summary-computer-software-history-and-evolution.html

Everyone knows that software project management is essential to producing software that works, on time and on budget. In spite of decades of "innovation," it doesn't get better. The winners follow a different set of rules.

https://www.blackliszt.com/2023/04/summary-software-project-management.html

Software quality assurance is an important specialty within the non-science of computing, but in spite of all the time and money spent, quality continues to be a major issue. There are solutions that have been proven in practice that are ignored by the experts and authorities.

https://www.blackliszt.com/2023/04/summary-software-quality-assurance.html

How do you win with software? Nearly everyone starts with requirements, makes estimates and is judged whether they deliver on time and on budget. This optimizes for expectations and is a proven path to failure. The winning path optimizes for speed, customer satisfaction and continuous quality. It's what the people who need to win do.

https://www.blackliszt.com/2023/07/summary-wartime-software-to-win-the-war.html

 

Why is Writing Computer Software Mostly Dysfunctional?

While not much discussed or broadly recognized, the vast majority of efforts to build computer software are disasters. They take too long, cost too much, and result in varying degrees of crap. Lots of solutions have been been promoted and tried. None of them have worked.

There are exceptions, of course. The exceptions prove that it really is possible to create good software quickly and efficiently. It is highly unlikely that the cause is that the methods taught in academia are good, but screwed up when applied; instead, it’s likely that nearly everyone has it wrong, just like doctors did when they bled patients to cure them and refused to sanitize when operating on patients.

A major cause of the dysfunction can be found in the approach to building software that was taken for good reason in the early days of computing. This approach was necessary for the first decades of computing. As computers grew more powerful, the necessity of doing things the same way began to fade away and finally disappear, Many of the early cumbersome practices were discarded, but the key focus has remained the core of software development to this day.

What is this universally accepted, unquestioned aspect of building software that does so much harm? Simple: it’s obsessing on software imperative language, relegating data definitions and attributes to a necessary annoyance, confined to a tiny island in an obscure corner of the vast sea of procedural language.

Does this sound awful? No. Everyone who does it doesn't think they're obsessing -- they're just working, writing their code! Similarly, getting water from the local community well didn’t sound awful in the 1800’s – until people finally found out about water contaminated with diseases like cholera. It took decades for the need for sanitation to be taken seriously, when the result of doing it poorly was death! We can hope that procedural language obsession will in the future be recognized as the main source of disease in software.

Early Computing

The roots of the language obsession are in the earliest days of computing. It was one thing to build a machine, and quite another to get it to do what you wanted it to do. The start was plugs and switches. Then the stored program computer was given step-by-step instructions in binary machine language. In the 1950’s first FORTRAN then COBOL were invented to make the process of creating the precise instructions needed easier, while still enabling the computer to operate at maximum speed.Those were indeed big advances.

In the 1960’s it still took a great deal of careful work to get results in a timely manner from computers. While languages like FORTRAN made writing easier, the fact that a compiler translated them to maximum speed machine language made their use acceptable.

The Apollo space capsule had a custom-built guidance system that was essential to its operation. Here is Margaret Hamilton next to a stack of code she and her team wrote for the Apollo Mission computers.

The Apollo guidance computer was a fast machine for its day, but the programmers had to get all the power out of it that they could to guide the capsule in real time. This is an extreme example, but 20 years into the computer revolution, everyone focused on using compiled procedural languages to get performance, and assembler language when necessary.

It was already evident that getting programs written quickly and well was incredibly hard. In fact, a big conference was held in 1968 to address what was called the "crisis" in software. Nothing got fixed. Meanwhile, efforts continued then and to this day to invent new programming languages that would miraculously make the problem go away. Nothing has changed for the better.

Partial steps towards declarative

From the early days to the present, there have been isolated efforts to go beyond simple definitions of data for procedural commands to operate on. Generally, the idea is that procedural commands spell out HOW to accomplish a task, while data definitions and attributes define WHAT the task is. It's like having a map (what is there) and directions (how to get from place to place on the map). See this for an explanation.

The invention of the SQL database was a small but important early step in this direction. SQL is all declarative. It is centered around a schema (a set of data definitions organized in rows and columns). The SELECT statement states what you want from the database, but not how to get it. WHAT not HOW!

You would think this would have led to a revolution in language (HOW) obsession. It didn't. In fact, because the language obsession stayed in charge, in some ways things got worse.

A few years after the DBMS revolution, people started putting big collections of historic data into what were called data warehouses. The idea was to make reporting easier without impacting production databases. Before long, OLAP (OnLine Analytical Processing) was invented to complement existing OLTP (OnLine Transaction Processing). While there were many differences, the core of OLAP was having a schema definition in the form of a star (star schema), with a central table of transactions and tables related to it containing the attributes (dimensions) of the transactions, typically organized in hierarchies. So there would be a time dimension (days, weeks, months), a location dimension (office, region, state) and others as relevant (sales, profits, department, etc.). After constructing such a thing, it was easy to get things like the change in sales from month to month in the hardware department without writing code.

OLAP was and is powerful. It assigned attributes to data in hierarchies, with unchanging programs that made it easy to navigate around. You could add attributes, dimensions, etc. without changing code! What an idea! But the idea was strictly confined to the isolated, distant island of OLAP and had no impact on software as a whole. The procedural language obsession continued without pause.

Declarative front and center

Procedural code is necessary. Code is what makes a machine run. However, the time for near-exclusive obsession about procedural code has long since passed. Limitations of computer speed and storage space were a legitimate reason to obsess about using the speed you had optimally. Think about cars. Engineers worked for decades to get the maximum speed of cars from about 10MPH to finally breaking 100. Now it's in the hundreds. In a much shorter period of time, computers have increased in speed by millions of times. Computer speed is rarely an issue.

How can we spend a little of the mountains of excess, unused computer speed to help make creating computer software less dysfunctional? Maybe instead of concentrating on procedural languages, there's a another way to get computer software to work quickly and well?

There is a proven path. It's obsessing about the WHAT is to be done. Obsessing about the data and everything we know about the data -- its attributes. This means applying the fruitful but limited approach we took with OLAP, and extending it as far as possible. In other words, instead of creating a set of directions from every possible starting point to every possible destination, we create a map in the form of metadata and a tiny, rarely-changing direction-generating program that takes as input starting and ending points and generates directions. You know, like those direction-generating program that are so handy in cars? That's how they work! See: https://www.blackliszt.com/2020/06/the-map-for-building-optimal-software.html

When we do this, we'll advance from the endless complexities of the solar system as described by Ptolemy to the simple, clear and accurate one described by Newton. What Newton did for understanding the movements of the planets, Metadata obsession will do for software. See this for more: https://www.blackliszt.com/2022/10/how-to-improve-software-productivity-and-quality-code-and-metadata.html

Software development is stuck in the endlessly complex epicycles of Ptolemy; we need to get to Newton.

 

The Amazing path to High Quality Fast-to-change Software

Everyone wants software that does what it’s supposed to do, runs fast without down time and can be changed quickly without causing problems. Who doesn’t want this? For more details on software goals, see this.

Everyone claims that their methods are great at achieving those goals. Sadly, such assertions are mostly baseless and in fact the touted methods do a terrible job. But they’re standard practice!

I’ve written extensively about they way to meet the goals in dozens of blog posts and a couple of books. Here is a specific step-by step way to move from standard architecture to one that meets the goals in simple terms. It’s a modified version of a path taken by a small, rapidly growing software company that I worked with. Incremental steps of this kind, with value at each step, are usually superior to the massive re-write approach that some software people are tempted to take.

When the new tech leader took over, there was no QA. So he first put in simple, UI, click-script based QA, and a full build process. Having done that, he wanted to move from his existing code organization to components to make change cleaner. But he had a reliability problem – it was good, but because of the nature of this customer base, it needed to be near-perfect. He started by thinking about adding a second data center, putting in database replication and then somehow doing a switch-over when the primary database went down.

Replication

He first focused on replication. All the transactions would arrive at a primary data center, with results stored on disk. The disk contents would be replicated to a backup data center, so that when it needed to take over, everything would be in place. It’s easy to set up a storage system to do this.

The trouble is that replication & availability functions have been moving up the stack, for good reason. While it may be trickier to set up database replication, the results are usually much better.

DBMS replication

Even when handling replication at the DBMS level, the DBMS transaction is guaranteed, but the user’s task may be just half done. The database will think everything is cool, but unless the application is 100% lined up, the results could be bogus.Yuck.

Application-level replication

Serious applications tend to have application-level replication. This basically means recording the field-level changes a user makes during an application session, from start through confirmation. This can and should be done in a generic way, so that when the application adds or changes a field, the recording, transfer and replay at the other site doesn’t need to change. This has the same net effect as database replication and avoids the issues.

Application log

Building an app-level log for replication is VERY close to building a … user transaction log! Which you’d like to have anyway! So make sure the extra information is there. This has many uses, including being able to show the user what they did and when they did it.

Instead of needing a big, intense, fast connection between the sites like you need for storage or database replication, you just need to ship and apply the session transaction log, which you can do at the end.

Replaying application transaction logs

Application transaction logs solve the reliability problem and give the user full access to their history with the application. They are also the crucial foundation of an incredibly important architectural advance: champion/challenger QA. This is how you can dump those QA scripts and enable rapid testing of new features for rapid, problem-free deployment.

In addition to having two identical copies of the application running in the two data centers, you bring up the proposed new version, Once you’ve got that, you can replay logs against a new copy of the application, and make sure you get the same results as last time. Guess what – that’s real-life QA! The only extra thing you need is the comparison, which again can be built once, regardless of whether there are 10 functions or 10,000. See this for much more.

Live parallel test

Once you’ve got that, you can implement live parallel test, which takes all the strain and risk out of releasing. It means doing the replication to another copy of the stack, just like you do for DR, except it’s the new code. If everything works well for however long you feel is necessary, all you do is switch which copy of the application sends results back to the user.

What’s gone when you do this thing that a small number of successful organizations do to their great benefit? Among other things: QA scripting. Double coding. Fixing QA bugs. Changing the script when you change the app. Unit testing. Test-driven development. Etc.

Moving to components

Yes, components and layers and objects and microservices are what experts are in favor of. Beware of experts! And above all, even though you may not give a talk at a conference about it, migrate your code to ... yes, a monolithic architecture.

Can’t you use queuing when it seems to make sense? Of course you can! But it should be something in-memory and simple like redis queues.

Active data migration

It’s standard practice to store all user and transaction data in a DBMS, and to call on the data from the DBMS when the user wants to do something. Sadly, this standard approach has long-since become obsolete as I explain in this post from more than a decade ago. It’s is a killer of application speed, flexibility and everything else. And there are proven in-memory databases like redis that can do the vast majority of jobs. If you’re dying to use a DBMS, they continue to be useful for archive and reporting!

Doing this will also tremendously simplify your champion/challenger testing.

The final push to speed of application change

Doing the foregoing things to your application will help on multiple dimensions. The final push, which can be step by step, is the one that will do the most to enable unanticipated changes to be made to your application. Conceptually, it’s pretty simple. What makes change hard? Finding all the places that need to be changed. What if the number of places you need to go to make a change shrinks, approaching over time a single place needing to be changed? You’re in the winner’s circle! Bonus: as you do this, the number of lines of code in your application will actually decrease! There’s lots to be said about this, but the core concept and value really are as simple as this.

What are you waiting for? Get started! High quality, fast-to-change software is yours for the taking ... err ... fast for the building.

How to Improve Software Productivity and Quality: The Common Sense Approach

I talked with a frustrated executive in a computer software company. I was about to visit their central development location for the first time, and he wanted to make sure I asked sufficiently penetrating questions so that I would find out what was “really” going on.

He explained that while he had written software, it was only for a few years in the distant past, and things had changed a great deal since his day. His current job in product marketing didn’t really require him to get into any details of the development shop, and in fact he preferred to stay out of the details for several reasons: (1) he was completely out of date with current technology and methods; (2) he didn’t want his thinking constrained by what the programmers declared was possible; (3) it was none of his business.

He had developed a keen interest in what was going on in the software group, however, because he realized that it had a dramatic effect on his ability to successfully market the product. His complaints were personal and based on his own experience, but they were fairly typical, which is why I’m recounting his tale of woe here.

The Lament

The layman’s lament was an interesting mish-mash of two basic themes:

• I’m not getting the results I need. There are certain results that I really need for my business. My competitors seem to be able to get those results, and I can’t. Basically, I want more features in each release, more frequent releases, more control and visibility on new features, fewer bugs in new releases, and the ability to make simple-sounding changes quickly. Our larger competitors seem to be able to move more quickly than we do.
• I think the way the developers’ work is old-fashioned, and if it were brought up-to-date, I would get the results I need. What they do seems to be “waterfall,” with lots of documentation that doesn’t say a lot. There must be something better, along the lines of what we used to call RAD (rapid application development). They only have manual testing, nothing automated, and they tell me it will be years before they can build automated testing! And shouldn’t they be using object-oriented methods? Wouldn’t that provide more re-use, so that things can be built and changed more quickly? They have three tiers, but when I want to change something, the code always seems to be in the wrong tier and takes forever. They’re talking about re-writing everything from scratch using the latest  technology, but I’m afraid it will take a long time and there won’t be anything that benefits me.

Basically, he was saying that he wants more things, quicker, and better quality. He also advanced some theories for why he’s not getting those things and how they might be achieved, but of course he couldn’t push his theories too hard, because he lacked experience and in-depth knowledge of the newer methods. He even claimed, in classic “the grass is greener” style, that practically everyone accomplishes these things, and he was nearly alone in being deprived of them – not true!

The usual dynamics of a technology group explaining itself to “outsiders” was also at work here – if you just listen to the technology managers, things are pretty good. The methods are modern and the operation is efficient and productive. There are all sorts of improvements that could be made with additional money for people and tools, of course, but for a group that’s been under continual pressure to build new features, support cranky customers and meet accelerated deadlines with fewer resources, they’re doing amazingly well. The non-technology executives tend to feel that this is all a front, and that results really could be better with more modern methods and tools. The technology managers, for their part, feel like they’re flying passenger planes listening to a bunch of desk-bound ignoramuses complain about their inability to deliver the passengers safely and on-time while upgrading the engine and cockpit systems at the same time. These people have no idea what building automated testing (for example) really takes, they’re thinking. The non-technology people don’t really want to talk about automated testing, of course – they’re the ones taking the direct heat from customers who get hurt by bugs in the new release, and aren’t even getting proposals from the technology management of how this noxious problem can be eliminated. Well, if you can’t tell me how to solve the problem (and you should be able to), how about this (automated testing, object-oriented, micro-services, etc.)??

It goes on and on. The business executives put a cap on it, sigh, maybe throw a tantrum or two, but basically try to live with a situation they know could be better than it is. Inexperienced executives refuse to put with this crap, and bring in new management, consultants, do outsourcing, etc. Their wrath is felt! Sadly, though, the result is typically a dramatic increase in costs, better-looking reporting, but basically the status quo in terms of results, with success being defined downwards to make everything look good. The inexperienced executive is now experienced, and reverts to plan A.

The technology manager does his version of the same dance. The experienced manager tries to keep things low-key and leaves lots of room for coping with disasters and the unexpected. Inexperienced technology managers refuse to tolerate the tyranny of low expectations; they strive for real excellence, using modern tools and methods. Sadly, though, the result is typically a dramatic increase in costs, better-sounding reports, but basically the status quo in terms of tangible results. The new methods are great, but we’re still recovering from the learning curve; that was tense and risky, I’m lucky I survived, that’s the last time I’m trying something like that again!

The Hope

The non-technology executive is sure there’s an answer here, and it isn’t just that he’s dumb. He keeps finding reason to hope that higher productivity with high quality and rapid cycles can be achieved. In my experience, the most frequent (rational) basis for that hope is a loose understanding of the database concept of normalization, and the thought that it should enable wide-spread changes to be made quickly and easily. Suppose the executive looks at a set of functionally related screens and wants some button or style change to be applied to each screen. It makes sense that there should be one place to go to make that change, because surely all those functionally related screens are based on something in common, a template or pattern of some kind. What if the zip code needs to be expanded from five digits to nine? The executive can understand that you’d have to go to more than one place to make the change, because the zip code is displayed on screens, used in application code and stored in the database, but there should be less than a handful of places to change, not scores or hundreds!

But somehow, each project gets bogged down in a morass of detail. When frustration causes the executive to dive into “why can’t you…” the eyes normally glaze over in the face of massive amounts of endless gobbledy-gook. What bugs some of the more inquisitive executives is how what should be one task ends up being lots and lots of tasks? With computers to do all the grunt work, there’s bound to be a way to turn what sounds, feels and seems like one thing (adding a search function to all the screens) actually be one thing – surely there must be! And if everything you can think of is in just one place, surely you should be able to go to that one place and change it! Don’t they do something like that with databases?

There is a realistic basis for hope

I’ve spent more of my life on the programmers’ side of the table than the executives’, so I can go on, with passion and enthusiasm, about the ways that technology-ignorant executives reduce the productivity, effectiveness and quality of tech groups, not to mention the morale! The more technical detail they think they know, the worse it seems to be.

That having been said, the executive’s lament is completely justified, and his hope for better days is actually reasonable (albeit not often realized).

What his hope needs to be realized is there to be exactly one place in the code where every completely distinct entity is defined, and all information about it is stated. For example, there should be exactly one place where we define what we mean by “city.” This is like having domains and normalization in database design, only extended further.

The definition of “city” needs to have everything we know about cities in that one place. It needs to include information that we need to store it (for example, its data type and length), to process it (for example, the code that verifies that a new instance of city is valid) and to display it (for example, its label). The information needs to incorporate both data (e.g. display label) and code (e.g. the input edit check) if needed to get the job done. This is like an extended database schema; a variety of high-level software design environments have something similar to this.

It must be possible to create composite entities in this way as well, for example address. A single composite entity would typically include references to other entities (for example, city), relationships among those other entities and unique properties of the composite entity (for example, that it’s called an “address.” This composite-making ability should be able to be extended to any number of levels. If there are composites that are similar, the similarity should be captured, so that only what makes the entity unique is expressed in the entity itself. A common example of this is home address and business address.

Sometimes entities need to be related to each other in detailed ways. For example, when checking for city, you might have a list of cities, and for each the state it’s in, and maybe even the county, which may have its own state-related lists.

The same principle should apply to entities buried deep in the code. For example, a sort routine probably has no existence in terms of display or storage, but there should usually be just one sort routine. Again, if there are multiple entities that are similar, it is essential that the similarities be placed in one entity and the unique parts in another. Simple parameterization is an approach that does this.

Some of these entities will need to cross typical software structure boundaries in order to maintain our prime principle here of having everything in exactly one place. For example, data entities like city and state need to have display labels, but there needs to be one single place where the code to display an entity’s label is defined. Suppose you want a multi-lingual application? This means that the single place where labels are displayed needs to know that all labels are potentially multi-lingual, needs to know which the current language is, and needs to be able to display the current language’s label for the current entity. It also means that wherever we define a label, we need to be able to make entries for each defined language. This may sound a bit complicated at first reading, but it actually makes sense, and has the wonderful effect of making an application completely multi-lingual.

In order to keep to the principle of each entity defined once, we need the ability to make relationships between entities. The general concept of inheritance, more general than found in object-oriented languages, is what we need here. It’s like customizing a standard-model car, where you want to leave some things off, add some things and change some things.

There’s lots more detail we could go into, but for present purposes I just want to illustrate the principle of “each entity defined in one place,” and to illustrate that “entity” means anything that goes into a program at any level. By defining an entity in one place, we can group things, reference things, and abstract their commonality wherever it is found, not just in a simple hierarchy, and not limited to functions or data definitions or anything else.

While this is a layman’s description, it should be possible to see that IF programs could be constructed in this way, the layman’s hope would be fulfilled. What the layman wants is pretty simple, and actually would be if programs were written in the way he assumes. The layman assumes that there’s one way to get to the database. He assumes that if you have a search function on a screen, it’s no big deal to put a search function on every screen. He assumes that if he wants a new function that has a great deal in common with an existing function, the effort to create the new function is little more than the effort to define the differences. He assumes that look and feel is defined centrally, and is surprised when the eleventh of anything feels, looks or acts differently than the prior ten.

Because he has these assumptions in his mind, he’s surprised when a change in one place breaks something that he doesn’t think has been changed (the infamous side-effect), because he assumes you haven’t been anywhere near that other place. He really doesn’t understand regression testing, in which you test all the stuff that you didn’t think you touched, to make sure it still works. Are these programmers such careless fools that, like children in a trinket shop, they break things while walking down the aisle to somewhere else, and you have to do a complete inventory of the store when the children leave?

Programs are definitely not generally written in conformance with the layman’s assumptions; that’s why there’s such a horrible disconnect between the layman and the techies. The techies have a way of building code, generally a way that they’ve received from those who came before them, that can be made to work, albeit with considerable effort. They may try to normalize their database schemas and apply object principles to their code, but in the vast majority of cases, the layman’s assumption of a single, central definition of every “thing,” and the ability to change that thing and have the side-effects ripple silently and effectively through the application, does not exist, is not articulated, not thought about, and is in no way a goal of the software organization. It’s not even something they’ve heard talked about in some book they keep meaning to get to. It’s just not there.

I assert that it is possible to write programs in a way that realizes the layman’s hope.

I’ve done it myself and I’ve seen others do it. The results are amazing. It’s harder to do than you would ideally like because of a lack of infrastructure already available to support this style of writing, but in spite of this, it’s not hard to write. Moreover, once the initial investment in structure has been made, the ability to make changes quickly and with high quality quickly pays back the investment.

The main obstacle for everyone is that there is tremendous inertia, and the techniques that provide a basis for the hope, while reasonable and achievable, are far out of the mainstream of software thinking. I have seen people who have good resumes but are stupid or lazy look at projects that have been constructed according to the “one entity – one definition” principle and simply declare them dead on arrival, complete re-write required. But I have also encountered projects in domain areas where there is no tradition at all in building things in this way in which the people have invented the principles completely on their own.

The “principle of non-redundancy” has far-reaching technical consequences and ends up being pretty sophisticated, but at its heart is simple: things are hard to do when you have to go many places or touch many things to get them done. When the redundancy in program representation (ignoring for the moment differences between code, program data and meta-data) is eliminated, making changes or additions to programs is optimally agile. In other words, with program representation of this type, it is as easy and quick as it can possibly be to make changes to the program. In general, this will be far quicker than most programs in their current highly redundant form.

The layman’s hope that improvements can be made in software productivity, quality and cycle is realistic, and based on creating a technical reality behind the often-discussed concepts of “components” and “building blocks” that is quite different from the usual embodiment.

I have no idea why this approach to building software, which is little but common sense, isn't taught in schools and widely practiced. For those who know and practice it, the approach of "Occamality" (define everything in exactly one place) gives HUGE competitive advantages.

How to Improve Software Productivity and Quality: Code and Metadata

In the long-fought war to improve software programming productivity, there have been offensives on many fronts, but precious little genuine progress made. We give our programmers the fanciest, most high-tech equipment imaginable – and it is orders of magnitude faster and more powerful than the equipment available to earlier generations – but this new equipment has made only marginal difference. While relieving programmers of the burden of punch cards helps, the latest generation of programmers are not getting the job done much better than their comparatively low-tech predecessors.

Form and Content

As usual, most efforts to improve the situation focus on one of two general approaches: form or content.

People who focus on form tend to think that the process of getting software written is what’s important. They talk about how one “methodology” is better than another. They think about how people are selected, trained, organized and motivated. As you might imagine, the spectrum of methodologies is a broad one. On one end of the spectrum is the linear approach, which starts from the generation of business requirements for software, and ends with testing, installation and ongoing maintenance. On the other end of the spectrum is the circular, interactive approach, in which a small working program is built right away, and gradually enhanced by programmers who interact closely with the eventual end-users of the program. There are any number of methodologies between these two extremes, each of which claims an ideal combination of predictable linearity with creative interactivity.

People who focus on content tend to think that the language in which programs are written and the structure and organization of programs are what’s important. Naturally, they like design and programming tools that give specific support to their preferred language and/or architecture. There tends to be a broad consensus of support at the leading edge around just a couple of languages and structures, while the majority of programmers struggle with enhancing  and maintaining programs originally written according to some earlier generation’s candidate for “best language” or “best architecture.” At the same time, many of those programmers make valiant attempts to renovate older programs so that they more closely resemble the latest design precepts, frequently creating messes. Regardless of the generation, programmers quickly get the idea that making changes to programs is their most time-consuming activity (apart, of course, from never-ending meetings), and so they focus on ways to organize programs to minimize the cost of change. This leads to a desire to build “components” that can be “assembled” into working programs, and naturally to standards for program components to “talk” with each other.

Lots of effort, not much progress

The net effect of all these well-intentioned efforts, on both the form and content sides, has been a little bit of progress and a great deal of churn. Having some agreed-on methodology leads to better predictability and generally better results than having none; it seems that having a beat to which everyone marches in unison, even if it’s not the best beat, leads to better results than having a great beat that many people don’t know or simply refuse to march to. What is the best methodology in any one case depends a great deal on both how much is already known about the problem to be solved and how smart and broadly skilled the participants in the project are. The more qualified the people and the less known about the target, the more appropriate it is to be on the “interactive” end of the spectrum; think highly qualified and trained special forces going after a well-defended target in enemy territory – creativity, teamwork and extraordinary skills are what you need. The more ordinary the participants and with better-understood objectives, the more appropriate being somewhere towards the “linear” end of the spectrum; think of a large army pressing an offensive against a broad front – with so many people, they can’t all be extraordinary, and you want coordinated, linear planning, because too much local initiative will lead to chaos.

As to content, while it is clear that the latest programming languages encourage common-sense concepts like modular organization and exposing clear interfaces, good programmers did that and more decades ago in whatever language they were writing, including assembler, and bad programmers can always find a way to make messes. And even if you use the best tools and interface methods, incredible churn is created by frequent shifts in what is considered to be the best architecture, practically obsoleting prior generations. Probably the single biggest movement over the last several decades has been the gradual effort to take important things about programs that originally could be discovered only by inspecting the source code of the program and making those things available for discovery and use by people and programs, without access to the source code. The first major wave of this movement led to exposure of much of a program’s persistent data, in the form of DBMS schemas; the other major wave of this movement (in many embodiments, from SAA to SOAP to microservices) exposes a program’s interfaces, in the form of callable functions and their parameters. This has been done in the belief that it will enable us to make important changes to some programs without access to or impact on others, and thus approach the dream of programming by assembling fixed components, like building blocks or legos, into completed programs or buildings. The belief in this dream has been affected very little by decades of evidence that legos don’t build things that adults want to live in.

I believe that while considerations of form are incredibly important, and when done inappropriately can drag down or even sink any programming project, there is little theoretical headway to be made by improvements in methodology. I think most of the relevant ideas for good methodology have already been explored from multiple angles, with the exception of how to match methodology to people and project requirements – no one methodology is the best for each project and each collection of people.  But even when you’ve picked the best methodology, the NBA players will always beat the high school team, as long as the NBA players execute well on the motivational and teamwork aspect that any good methodology incorporates. With methodology we’re more in need of good execution and somehow assembling a talented and experienced team than we are of fresh new ideas.

Ptolemy, Copernicus, Kepler, Newton

Content, however, is another story altogether. I think our current best languages and architectures will look positively medieval from the perspective of a better approach to content. I think there is a possible revolution here, one that can bring about dramatic improvements in productivity, but which requires an entirely new mind-set, as different as Copernicus and Ptolemy, as different as Einstein and Newton.

Having said that, let me also say that the “new” ideas are by no means completely new. Many existing products and projects have exploited important parts of these ideas. What is mostly new here is not any particular programming technique, but an overriding vision and approach that ties various isolated programming techniques into a unified, consistent whole. Kepler’s equations described the motion of planets with accuracy equal to Newton’s – in fact, you can derive one set of equations mathematically from the other; but Newton provided the vision (gravity) and the tools (calculus) that transformed some practical techniques (Kepler’s equations) into the basis of a new view of the world. That’s why physics up to the end of the nineteenth century was quite justifiably characterized as “Newtonian” and not “Keplerian.”

Ptolemy and Newton looked at the same set of objects; Ptolemy picked the closest one, the earth, to serve as the center of thinking, while still incorporating the rest.

His main goal was to describe the motions of what you could see, focused on the matter. Copernicus noted that things got simpler and more accurate if you picked one farther away, the Sun, to serve as the center of things.

Kepler made things better by noticing the curves made by the planets. Newton then took the crucial step: instead of focusing on matter, he focused on energy (gravity in this case), and wrote an equation describing how gravity works,

which creates changes in the location and velocity of matter. In programming, we have clear equivalents of matter and energy: matter is data (whether or not it is persistent), and energy is instructions (lines of code, regardless of the language it’s in). In COBOL, for example, this division is made explicit in the data division and the procedure division. In modern languages the two are more intermixed, but it remains clear whether any particular line describes data or describes an action to take (an if statement, assignment statement, etc.).

Now, in spite of what you may have been taught in school, Ptolemy’s method works. In fact, it would be possible (though not particularly desirable) to update his approach with modern observations and methods, and have it produce results that are nearly identical to those possible today; in fact, only when you get to relativity does Ptolemy’s method break down altogether -- but remember that the effects of relativity are so small, it takes twentieth century instruments to detect them. But no one bothers to do this, because the energy-centered approach developed by Newton and refined by his successors is so much simpler, cleaner and efficient.

Similarly, there is no doubt that the instructions-centric approach to programming works. But what comes out of it is complicated, ugly and inefficient. The Newtonian breakthrough in programming is replacing writing and organizing instructions (and oh, by the way, there’s some data too) as the center of what we do with defining and describing data (and oh, by the way, there are some instructions too). The instruction-centered approach yields large numbers of instructions with a small amount of associated data definitions; the data-centered approach yields large numbers of data definitions and descriptions operated on by a significant body of unchanging, standard instructions and small numbers of instructions specifically written for a particular program. In the instruction-centered approach, we naturally worry about how to organize collections of instructions so that we can write fewer of them, and arrive at concepts like objects, components and class inheritance (a subclass inherits and can override its parent’s methods (instructions)). In the data-centered approach, we naturally worry about how to organize collections of data definitions so that we can have the minimal set of them, and arrive at concepts like meta-data, standard operations (e.g. pre-written meta-data-driven instructions enabling create, query, select, update and delete of a given collection of data) and data inheritance (a child data definition, individual or group, inherits and can override its parent’s definitional attributes). In short, we separate out everything we observe into a small, unchanging core (like Newton's gravity) that produces ever-changing results in a diverse landscape.

We shift our perspective in this way not because it enables us to accomplish something that can’t be accomplished by the current perspective, but because it proves to be cleaner, simpler, more efficient, easier to change, etc.

Instructions, data and maps

Only by understanding the details of this approach can it really be appreciated, but the metaphor of driving instructions and maps is appropriate and should make the core idea clear.

Suppose your job is to drive between two locations, and the source and destination location are always changing. There are two general approaches for giving you directions:

1. Turn-by-turn directions (the instruction-driven, action-oriented approach)
2. A map, with source and destination marked (the data-driven, matter-oriented approach)

The advantage of directions is that they make things easy for the driver (the driver is like the computer in this case). You pick up one step, drive it, then pick up the next, drive it, and so on until the end. You don’t have to think in advance. All you have to do is follow directions, which tell you explicitly what to do, in action-oriented terms (turn here, etc.).

The problem with directions come in a couple of circumstances:

• You have to have a huge number of directions to cover all possible starting points and destinations, and there is a great deal of overlap between sets of directions
• If there is a problem not anticipated by the directions, such as an accident or road construction, you have to guess and get lucky to get around the problem and get back on track.

A map, on the other hand, gives you most of the information you need to generate your own directions between any two given points. The map provides the information; you generate the actions from that information. With a direction-generating program and some parameters like whether to use toll roads, a generic program can generate directions on the fly. With a program like Wayz, real-time changes due to updated traffic information can even be generated.

The Wayz program itself isn't updated very often -- it's mostly the map and road conditions. Same thing with a program written using this approach; the core capabilities are written in instructions, while the details of the input, storage, processing and output are all described in a "map" of the data and what is to be done with it.

Conclusion

It's pretty simple: programming today largely follows the method of Ptolemy, resulting in an explosion of software epi-cycles to get anything done. Attempts to keep things easily changeable sound promising but never work out in practice.

The way forward is to focus instead on what there is and what is to be done with it (data and metadata), with a small amount of Wayz-like code to take any user or data stream from its starting point to its destination with easy changes as needed.

Better Software and Happier Customers with Post-hoc Design

What can you possibly mean by "post-hoc design?" Yes, I know it means "after-the-fact design," using normal English. It's nonsense! First you design something. Then you build it. Period.

Got your attention, have I? I agree that "post-hoc design" sounds like nonsense. I never heard of it or considered it for decades. But then I did. Before long I saw that great programmers used it to create effective high-quality, loved-by-customers software very quickly.

The usual way to build software: design then build

The way to build good software is obviously to think about it first. Who does anything important without having a plan? Start by getting requirements from the best possible source, as detailed as possible. Then consider scale and volume. Then start with architecture and drill down to design.

When experienced people do architecture and design, they know that requirements often "evolve." So it's important to generalize the design anticipating the changes and likely future requirements. Then you make plans and can start building. Test and verify as you drive towards alpha then beta testing. You know the drill. Anything but this general approach is pure amateur-hour.

I did this over and over. Things kept screwing up. The main issue was requirements "evolution," which is something I knew would happen! Some of the changes seemed like they were from left field, and meant that part of my generalized architecture not only failed to anticipate them, but actually made it harder to meet them! Things that I anticipated might happen which I wove into the design never happened. Not only had I wasted the time designing and building, the weren't-needed parts of the design often made it hard for me to build the new things that came along that I had failed to anticipate.

I assumed that the problem was that I didn't spend enough time doing the architecture and design thinking, and I hadn't been smart enough about it. Next time I would work harder and smarter and things would go more smoothly. Never happened. How about requirements? Same thing. The people defining the requirements did the best they could, but were also surprised when things came along, and embarrassed when things they were sure would be important weren't.

After a long time -- decades! -- I finally figured out that the problem was in principle unsolvable. You can't plan for the future in software. Because you can't perfectly predict the future! What you are sure will happen doesn't, and what you never thought about happens.Time spent on anything but doing and learning as you go along is wasted time.

The winning way to build software: Build then Design

Build first. Then and only then do the design for the software you've already built. Sounds totally stupid. That's part of why I throw in some Latin to make it sound exotic: "Post-hoc design," i.e., after-the-fact design.

When you design before you build, you can't possibly know what you're doing. You spend a bunch of time doing things that turn out to be wrong, and making the build harder and longer than it needs to be. When you build in small increments with customer/user input and feedback at each step, keeping the code as simple as possible, you keep everything short and direct. You might even build a whole solution for a customer this way -- purposely NOT thinking about what other customers might need, but driving with lots of hard-coding to exactly what THIS customer needs. Result: the customer watches their solution grow, each step (hopefully) doing something useful, guides it as needed, and gets exactly what they need in the shortest possible time. What's bad about a happy customer?

Of course, if you've got the typical crew of Design-first-then-build programmers, they're going to complain about the demeaning, unprofessional approach they're being forced to take. They might cram in O-O classes and inheritance as a sop to their pride; if they do, they should be caught and chastised! They will grumble about the enormous mountain of "technical debt" being created. Shut up and code! Exactly and only what's needed to make this customer happy!

When the code is shown to another customer, they might love some things, not need some other things and point out some crucial things they need aren't there. Response: the nearly-mutinous programmers grab a copy of the code and start hacking at it, neutering what isn't needed, changing here and adding there. They are NOT permitted to "enhance" the original code, but hack a copy of it to meet the new customer's need. At this point, some of the programmers might discover that they like the feeling of making a customer happy more quickly than ever before.

After doing this a couple times, exactly when is a matter of judgement, it will be time to do the "design" on the software that's already been built. Cynics might call this "paying off tech debt," except it's not. You change the code so that it exactly and only meets the requirements of the design you would have made to build these and only these bodies of code. You take the several separate bodies of code (remember, you did evil copy-and-modify) and create from them a single body of code that can do what any of the versions can do.

When you do this, it's essential that you NOT anticipate future variations -- which will lead to the usual problems of design-first. The pattern for accomplishing this is the elimination of redundancy, i.e., Occamality. When you see copy/modify versions of code, you replace them with a single body of code with the variations handled in the simplest way possible -- for example, putting the variations into a metadata table.

This isn't something that's done just once. You throw in a post-hoc design cycle whenever it makes sense, usually when you have an unwieldy number of similar copies.

As time goes on, an ever-growing fraction of a new user's needs can be met by simple parameter and table settings of the main code line, and an ever-shrinking fraction met by new code.

Post-Hoc Design

Ignoring the pretentious name, post-hoc design is the simplest and most efficient way to build software that makes customers happy while minimizing the overall programming effort. The difference is a great reduction in wasted time designing and building, and in the time to customer satisfaction. Instead of a long requirements gathering and up-front design trying valiantly to get it right for once, resulting in lots of useless code that makes it harder to build what it turns out is actually needed, you hard-code direct to working solutions, and then periodically perform code unification whose purpose is to further shorten the time to satisfaction of new customers. To the extent that a "design" is a structure for code that enables a single body of code to be easily configured to meet diverse needs, doing the design post-hoc assures zero waste and error.

What is the purpose of architecture and design anyway? It is to create a single body of code (with associated parameters and control tables) that meets the needs of many customers with zero changes to the code itself. The usual method is outside-in, gaze into the future. Post-hoc design is inside-out, study what you built to make a few customers happy, and reduce the number of source code copies to zero while reducing the lines of code to a minimum. The goal is post-hoc design is to minimize the time and effort to satisfy the next customer, and that's achieved by making the code Occamal, i.e., eliminating redundancies of all kinds. After all, what makes code hard to change? Finding all the places where something is defined. If everything is defined in exactly one place, once you've found it, change is easy.

Post-hoc design is a process that should continue through the whole life of a body of code. It prioritizes satisfaction of the customer in front of your face. It breaks the usual model of do one thing to build code and another to modify it. In the early days of what would normally be called a code "build," the code works, but only does a subset of what it is likely to end up doing. When customers see subsets of this kind, it's amazing how it impacts their view of their requirements! "I love that. I could start using it today if only this and that were added!" It's called "grow the baby," an amazing way to achieve both speed and quality.

New name for an old idea

All I'm doing with "Post-hoc design" is putting a name and some system around a practice that, while scorned by academia and banned by professional managers, has a long history of producing best-in class results. I'm far from the first person who has noticed the key elements of post-hoc design.

Linus Torvalds (key author of Linux, the world's leading operating system) is clearly down on the whole idea of up-front design:

Don’t ever make the mistake [of thinking] that you can design something better than what you get from ruthless massively parallel trial-and-error with a feedback cycle. That’s giving your intelligence much too much credit.

Gall's Law is a clear statement of the incremental approach:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.

The great computer scientist Donald Knuth, author of the multi-volume Art of Computer Programming, was a master of shifting between assembler language programming and abstract algorithms and back, the key activities of the speed-to-solution and post-hoc abstraction phases of the method I've described here.

People who discover the power and beauty of high-level, abstract ideas often make the mistake of believing that concrete ideas at lower levels are worthless and might as well be forgotten. On the contrary, the best computer scientists are thoroughly grounded in basic concepts of how computers actually work. The essence of computer science is an ability to understand many levels of abstraction simultaneously.

Thanks to Daniel Lemire for alerting me to these quotes.

Conclusion

Post-hoc design is based on the idea that software is only "built" once, and after that always changed. So why not apply the optimal process of changing software from day one? And then alternate between as-fast-as-possible driving to the next milestone, with periodic clean-ups to make fast driving to the next goal optimal? Post-hoc design is a cornerstone of the process of creating happy customers and optimal code.  It also happens to conform to the goals of software architecture. Post-hoc design is like first fighting a battle, and then, once the battle is over and you've won, cleaning and repairing everything, incorporating what you learned from the battle just past so that everything is ready for the next battle. Post-hoc design is the way to win.

 

Samuel Johnson's Dictionary and Writing Software

Samuel Johnson's famous Dictionary of English was written hundreds of years ago. Nonetheless, it has a great deal to teach us about software.

English is a language, of course, and we know there are numerous computer languages. Each kind of language has words, and it's essential to spell the words correctly so that the reader of what you write knows what you mean to say. Similarly with grammar and usage. When explaining the use of words, it's helpful to give examples. Of course there are more words in human languages than in computer languages, so the book defining a human language tends to be longer, as you can easily see by picking up a copy of Kernighan & Ritchie's The C Programming Language and any English dictionary.

The Dictionary

In 1755, Samuel Johnson published "A Dictionary of the English Language."

His was far from the first such dictionary -- there were already dozens in existence. But his became the most influential dictionary of English for over 150 years.

Much like a top modern programmer, he furiously wrote away with great energy and concentration. He originally claimed he would finish it in 3 years, but it ended up taking him seven. He did everything himself, with just a small amount of clerical help.

Meanwhile, L'Academie Francaise had forty scholars working for forty years to do a similar job for French. Johnson is said to have commented on this:

This is the proportion. Let me see; forty times forty is sixteen hundred. As three to sixteen hundred, so is the proportion of an Englishman to a Frenchman.

In software, there is sometimes talk of programmers who are 10X more productive than the average programmer. Johnson claimed to be more than 500X! When he was "really" only over 200X. Such an exaggerator... Of course in programming, there are people and small teams that perform 100X better than the ancient, lumbering giants "competing" against them.

With such a vast amount of work, it's not surprising that some humor managed to find its way into the work. For example:

Lexicographer: A writer of dictionaries; a harmless drudge, that busies himself in tracing the original, and detailing the signification of words.

Oats: a grain which in England is generally given to horses, but in Scotland supports the people.

Software

If you're a high-productivity programmer, I suspect the parallels have leaped out at you. Here are a couple:

• Programmers have strong opinions, and sometimes express them with wonderful humor. Here are dictionary examples and language opinions.
  
• The productivity difference between a top-down typical big-company MBA-driven software project tends to take 10 to 100's of times more effort than a similar project done by a great programmer or small team. See this and this.
  

The lesson here is clear: when it comes to writing software, be more like Samuel Johnson and less like the French Academy.

How to Improve Software Productivity and Quality: Schema Enhancements

Most efforts to improve programmer productivity and software quality fail to generate lasting gains. New languages, new project management and the rest are decades-long disappointments – not that anyone admits failure, of course.

The general approach of software abstraction, i.e., moving program definition from imperative code to declarative metadata, has decades of success to prove its viability. It’s a peculiar fact of software history and Computer Science that the approach is not mainstream. So much the more competitive advantage for hungry teams that want to fight the entrenched software armies and win!

The first step – and it’s a big one! – on the journey to building better software more quickly is to migrate application functionality from lines of code to attributes in central schema (data) definitions.

Data Definitions and Schemas

Every software language has two kinds of statements: statements that define and name data and statements that do things that are related to getting, processing and storing data. Definitions are like a map of what exists. Action statements are like sets of directions for going between places on a map. The map/directions metaphor is key here.

In practice, programmers tend to first create the data definitions and then proceed to spend the vast majority of their time and effort creating and evolving the action statements. If you look at most programs, the vast majority of the lines are “action” lines.

The action lines are endlessly complex, needing books to describe all the kinds of statements, the grammar, the available libraries and frameworks, etc. The data definitions are extremely simple. They first and foremost name a piece of data, and then (usually) give its type, which is one of a small selection of things like integer, character, and floating point (a number that has decimal digits). There are often some grouping and array options that allow you to put data items into a block (like address with street, town and state) and sets (like an array for days in a year).

One of the peculiar elements of software language evolution is whether the data used in a program is defined in a single place or multiple places. You would think – correctly! – that the sensible choice is a single definition. That was the case for the early batch-oriented languages like COBOL, which has a shared copybook library of data definitions. A single definition was a key aspect of the 4-GL languages that fueled their high productivity.

Then the DBMS grew as a standard part of the software toolkit; each DBMS has its own set of data definitions, called a “schema.” Schemas enable each piece of data to have a name, a data type and be part of a grouping (table). That’s pretty much it! Then software began to be developed in layers, like UI, server and database, each with its own data/schema definitions and language. Next came services and distributed applications, each with its own data definitions and often written in different languages. Each of these things need to “talk” with each other, passing and getting back data, with further definitions for the interfaces.

The result of all this was an explosion of data definitions, with what amounts to the same data being defined multiple times in multiple languages and locations in a program.

In terms of maps and directions, this is very much like having many different collections of directions, each of which has exactly and only the parts of the map those directions traverse. Insane!

The BIG First Step towards Productivity and Quality

The first big step towards sanity, with the nice side effect of productivity and quality, is to centralize all of a program’s data definitions in a single place. Eliminate the redundancy!

Yes, it may take a bit of work. The central schema would be stored in a multi-part file in a standardized format, with selectors and generators for each program that shared the schema. Each sub-program (like a UI or service) would generally only use some of the program’s data, and would name the part it used in a header. A translator/generator would then grab the relevant subset of definitions and generate them in the format required for the language of the program – generally not a hard task, and one that in the future should be provided as a widely-available toolset.

Why bother? Make your change in ONE place, and with no further work it’s deployed in ALL relevant places. Quality (no errors, no missing a place to change) and productivity (less work). You just have to bend your head around the "radical" thought that data can be defined outside of a program.

If you're scratching your head and thinking that this approach doesn't fit into the object-oriented paradigm in which data definitions are an integral part of the code that works with them, i.e. a Class, you're right. Only by breaking this death-grip can we eliminate the horrible cancer of redundant data definitions that make bodies of O-O code so hard to write and change. That is the single biggest reason why O-O is bad -- but there are more!

The BIG Next Step towards Productivity and Quality

Depending on your situation, this can be your first step.

Data definitions, as you may know, are pretty sparse. There is a huge amount of information we know about data that we normally express in various languages, often in many places. When we put a field on a screen, we may:

• Set permissions to make it not visible, read-only or editable.
• If the field can be entered, it may be required or optional
• Display a label for the field
• Control the size and format of the field to handle things like selecting from a list of choices or entering a date
• Check the input to make sure it’s valid, and display an error message if it isn’t
• Fields may be grouped for display and be given a label, like an address

Here's the core move: each one of the above bullet items -- and more! -- should be defined as attributes of the data/schema definition. In other words, these things shouldn't be arguments of functions or otherwise part of procedural code. They should be just like the Type attribute of a data definition is, an attribute of the data definition.

This is just in the UI layer. Why not take what’s defined there and apply it as required at the server and database layers – surely you want the same error checking there as well, right?

Another GIANT step forward

Now we get to some fun stuff. You know all that rhetoric about “inheritance” you hear about in the object-oriented world? The stuff that sounds good but never much pans out? In schemas and data definitions, inheritance is simple and … it’s effective! It’s been implemented for a long time in the DBMS concept of domains, but it makes sense to greatly extend it and make it multi-level and multi-parent.

You’ve gone to the trouble of defining the multi-field group of address. There may be variations that have lots in common, like billing and shipping address. Why define each kind of address from scratch? Why not define the common parts once and then say what’s unique about shipping and billing?

Once you’re in the world of inheritance, you start getting some killer quality and productivity. Suppose it’s decades ago and the USPS has decided to add another 4 digits to the zip code. Bummer. If you’re in the enhanced schema world, you just go into the master definition, make the change, and voila! Every use of zip code is now updated.

Schema updating with databases

Every step you take down the road of centralized schema takes some work but delivers serious benefits. So let’s turn to database schema updates.

Everyone who works with a database knows that updating the database schema is a process. Generally you try to make updates backwards compatible. It’s nearly always the case that the database schema change has to be applied to the test version of the database first. Then you update the programs that depend on the new or changed schema elements and test with the database. When it’s OK, you do the same to the production system, updating the production database first before releasing the code that uses it.

Having a centralized schema that encompasses all programs and databases doesn’t change this, but makes it easier – fewer steps with fewer mistakes. First you make the change in the centralized schema. Then it’s a process of generating the data definitions first for the test systems (database and programs) and then to the production system. You may have made just a couple changes to the centralized schema, but because of inheritance and all the data definitions that are generated, you might end up with dozens of changes in your overall system – UI pages, back end services, API calls and definitions and the database schema. Making an omission or mistake on just one of the dozens of changes means a bug that has to be found and fixed.

Conclusion

I’ve only scratched the surface of a huge subject in this post. But in practice, it’s a hill you can climb. Each step yields benefits, and successive steps deliver increasingly large results in terms of productivity and quality. The overall picture should be clear: you are taking a wide variety of data definitions expressed in code in different languages and parts of a system and step by step, collapsing them into a small number of declarative, meta-data attributes of a centralized schema. A simple generator (compile-time or run-time) can turn the centralized information into what’s needed to make the system work.

In doing this, you have removed a great deal of redundancy from your system. You’ve made it easier to change. While rarely looked on as a key thing to strive for, the fact that the vast majority of what we do to software is change it makes non-redundancy the most important measure of goodness that software can have.

What I've described here are just the first steps up the mountain. Near the mountain's top, most of a program's functionality is defined by metadata!

FWIW, the concept I'm explaining here is an OLD one. It's been around and been implemented to varying extents in many successful production systems. It's the core of climbing the tree of abstraction. When and to the extent it's been implemented, the productivity and quality gains have in fact been achieved. Ever hear of the RAILS framework in Ruby, implementing the DRY (Don't Repeat Yourself) concept? A limited version of the same idea. Apple's credit card runs on a system built on these principles today. This approach is practical and proven. But it's orthogonal to the general thoughts about software that are generally taught in Computer Science and practiced in mainstream organizations.

This means that it's a super-power that software ninjas can use to program circles around the lumbering armies of mainstream software development organizations.

The Map for Building Optimal Software

So you want to build optimal software, do you? What’s “optimal” in software, anyway? Is it building the software that’s needed quickly and well, with no bugs? Is it building software that’s easy to enhance, adding new features – including ones you never thought of – with minimal fuss and bother? Is it building software that’s easy to scale endlessly with little trouble? How about all of the above, all at once? Yup, that would be optimal, sure enough.

I’ve described in general terms about optimal software. Here’s a map, a specific example, of how to get there.

Going to new places

Let’s think about going places we haven’t been before, or taking routes that are new to us. What are the options for figuring out how to do it?

If you’re talking with someone who is familiar with the place to which you want to go, you might ask them how to get there. If there are more than a couple of turns, you might try to simplify the directions or write them down. If you’re on your own, you might consult a map. If you’re computer-oriented you might use one of the on-line mapping programs. If you’re driving in a car, you might use a navigation system.

No matter what you do, you’ll end up (let’s say) getting in a car and driving. When you drive, one way or another, you’ll be following directions and making choices of turns. Nothing happens unless you drive and to drive you need directions.

It’s obvious that while you need directions, a fixed set of directions by itself is adequate only if nothing goes wrong – there are no accidents, no construction detours, no recent changes to the roads, etc. The minute there is any problem, you need a map or a live, map-driven navigation system.

What you do when you look at a map, or what a navigation system does, is execute a generic direction-creating algorithm. Either your brain or a computer looks at the map, sees where you are and where you want to go, and figures out the sequence of roads and turns to get you there. Is there an obstruction? The nav system doesn't care -- it just computes a fresh set of directions for you, based on where you are now.

Here's an example. It shows two different routes for me to drive from my home to the wonderfully close source of some of the world's best home-made ice cream: Denville Dairy.

A direction-creating algorithm is a relatively small amount of code that is independent of the map that it uses. You may need to enhance it when different kinds of things are introduced into the map (for example, you decide to support avoiding toll roads or taking ferries), but once you’ve added the capability, it should automatically apply to all the maps you have.

So “where” is the application functionality here? The ability to create directions is in a small amount of abstract code. This code accepts a reference to a map, a starting point, an ending point and perhaps some parameters, like whether to minimize distance or time or avoid toll roads. The actual directions themselves are drawn from the substance of the map. In this case,

• what you’ve got is the map;
• what you do to what you’ve got is create directions.

Isn’t it interesting that most of the time and effort is creating the set of maps, without which there are no directions? And isn’t it interesting that the maps are data?

Maps are:

• Extensive
• Subject to frequent updating and change

An error in a map is bad, but should not “crash the program” – it should just lead to bad results, and should be easily fixed, without going into debuggers, single-stepping and other geeky things.

The direction-creating program is:

• Short
• Infrequently updated
• Unaffected by additional maps
• Unaffected by changes to maps

Best of all, the direction-creating program is unaffected by the driver not following the directions – the program just builds new directions to the destination from wherever the car happens to be.

Errors in this program are crippling, but it’s likely that errors will be found early in the process, after which it won’t change much.

Maps and writing optimal code

So what does this have to do with writing code? Lots.

A map (meta-data) is an efficient, compact, re-purpose-able representation of what the user cares about. It is truly Occamal, without redundancy. Each piece of knowledge you have about the world is represented in the map exactly once.

A reasonably well-written direction-generating program is an efficient, compact representation of strategies for generating routes through the entire universe of possible maps. Each algorithm or strategy for generating directions should be represented in the program exactly once.

Combined, the map and the direction-generating program are a pretty good template/metaphor for an Occamal program. Here’s the pattern:

• You should make everything you possibly can into a “map” (metadata).
• Abstract actions that cannot be reduced to metadata should be coded once, and should be “driven” to the largest extent possible by the metadata.

The map to building optimal software is a simple idea: build the smallest amount of code you possibly can, and put as much information as possible into passive, declarative meta-data. That will get you to your destination safely and in the shortest amount of time without breaking laws.

William of Occam is the Inventor of the Method for Building Optimal Software

Lots of people would like the credit for establishing, once and for all, the core, immutable principle of optimal software design/architecture – the method for measuring which, among an infinite number of embodiments of software requirements, is provably the best among them. I am one of the crowd that would love to take credit! Alas, I and my competitors have been beaten to the punch, by a mere 700 years or so, by a guy named William of Occam.

As it turns out, the software crowd is, as usual, late to the party: loads of luminaries in other fields have recognized William’s achievement, viz., Occam’s Razor, for the indispensable concept that it is. Various software folks have been nipping around the edges of it, for example with the DRY (don’t repeat yourself) principle that’s gotten some attention. But they haven’t grasped the fundamental, touches-nearly-everything, deep nature of the Razor. Too bad! Crappy software continues to be churned out by the ton by people who are enthralled by false idols, convinced they’re on the righteous path toward software excellence, when in fact they’re doing something else, words for which should not be used in polite company.

William of Occam (ca. 1285 to 1349) was an English logician and Franciscan friar.

He is credited with formulating a principle that is already widely applied to various aspects of computer systems. In those areas of computing to which it has been applied, it reigns supreme – it unquestionably supplies the most fundamental, relevant and optimal understanding and solutions to the relevant problems, so much so that the people involved don’t think about it or question it.

However, there are large areas of computing to which Occam’s razor has not been applied. Worse, it is not even one of the candidates under consideration. As a result, those aspects of computing are fractured, inefficient, unpredictable, and driven by fashion and politics. Even while the practitioners call themselves "computer scientists," which is something that none of them are.

Everyone involved knows that the whole process of specifying, designing, building, testing and supporting software is hopelessly inefficient, unpredictable and error-prone. The leading methodology that concentrates on the process of building software, “project management,” is theoretically bankrupt and in any case has an empirical track record of failure. In terms of the content of good software, there are fierce battles among competing approaches, none of which is anything but a collection of unsupported and unfounded assertions, and which in practice don’t contribute to building good software.

Occam’s razor leads to the principles on which good software may be built, and supplies a single simple, widely applicable theme that, when applied, cuts away (as a razor should) all the inefficiency, inapplicability and generally what we don’t like about software. Once the principle is understood and widely applied, I expect it will become the un-discussed and undisputed standard for how software is built, just as it has in the other areas of computing to which it has been applied.

What is this amazing standard? It’s simple. I mean literally simple. That’s what Occam’s Razor is all about.

Occam’s razor is:

    Entia non sunt multiplicanda praeter necessitatem.

    No more things should be presumed to exist than are absolutely necessary.

Occam’s razor applied to software would be:

    No more software entities should be created than are absolutely necessary.

Another way of expressing this is that all redundancy, of any kind and in any way, should be eliminated from a piece of software. This often requires writing imperative code to implement any and all abstract actions, and to create largely declarative meta-data to specify everything that is specific to an application. The meta-data, of course, should also have no redundancy; this is normally achieved by use of multiple inheritance with override. See this for more.

To be extremely brief, the reason why eliminating redundancy is good is that practically everything that happens to a piece of software is that it is enhanced, altered, bug-fixed or otherwise modified. Creation happens just once; everything that follows is growth and change. Implementing change of any kind to a program takes the least effort with the least risk if everything is defined in exactly one place – once you’ve found the place, you make the change and you’re done. Among the infinite universe of programs solving the same problem that work, the speed, cost and risk of change is the overriding virtue, both in technical and business terms.

Thanks to William of Occam, I can’t claim to have invented the core principle of software, and the way to express the measure of goodness for software. William has me beat by over 700 years. Thanks a lot, Bill! Lots of people have applied Occam’s Razor to lots of tough things with optimal success, even in things closely related to software, such as information theory. Way too late again. About all I can do is mess with his name and make up a term that expresses goodness in software. Here it is: I propose that a piece of software can be measured by its “Occamality.” The more “Occamal” it is, the better it is. And I propose that Occamality is strictly correlated with the extent to which there is no redundancy of any kind in a program, which is almost always strictly correlated with the program being at the high end of the hierarchy of abstraction.

How to Build Applications that can be Changed Quickly

Nearly everyone who talks about fast and reliable application building seems to talk about the process (Agile etc.) and/or a couple architectural things (object-oriented, micro-services, etc.). From what I can tell, these haven’t led to any break-throughs in application velocity or predictability. However, it’s clear that there are methods for high-velocity application growth. A wide variety of small, motivated groups have re-invented them over the years. I have had the privilege of observing these groups and their methods, and seen the impact when they add or drop velocity-aiding methods.

Here is an overview of some of the methods I’ve seen used. The improvements can range from cutting time in half to whole number factors, things like turning weeks into days or in some cases hours.

Forget requirements, focus on change

The normal requirements process starts with the often non-technical people who have a need. The need can be expressed directly, or through refusal to buy/use a product, which gets the attention of sales and marketing people. There then ensues a process of requirements gathering and refinement. Wire-frame UI’s may be involved. Finally, the requirements are ready for engineering, and away we go on hopefully not too many “sprints.” This is an “outside-in” approach.

Speed is achieved by taking an “inside-out” approach – instead of focusing almost exclusively on requirements, center your efforts on the code and what it can do or easily be changed to do today to go in the direction of the requirements. Take steps to change and add to the existing code to meet the unmet needs. Do this in small steps, getting feedback at each step. When you’re done, go back and “clean up” the code to eliminate redundancies as much as possible, to make the next change as easy as possible.

What this does is ground the changes in the existing code, resulting in the changes and additions being smaller than they otherwise would be. As a additional benefit, it usually results in a greater uniformity of both the code and the interfaces (UI and API) to the code.

Minimize the customization and optimization of the code and UI

When product people think about a new process that has UI, they usually try to do a good job, understanding all the details and nuances involved. The result of this sophistication is often requirements that are finely adapted to the particular function being performed. What’s wrong with this, you may wonder? One problem is that this approach ends up taking longer to built and results in more code. More important is the users – the less someone has to learn to use a new feature, the more quickly and easily they’ll be able to use it. Familiarity is incredibly important. Which means, among other things, that literally using rarely changing templates to generate interfaces will both make any new interface element recognizable and easy to learn, but also lead to the least amount of code changes to get the job done.

This means that the fineness and sophistication of product people can be a problem. The more time they spend on a subject and the more they think and reflect on it, the worse the temptation is likely to get to do something other than just crank out another instance of a familiar pattern.  Resist!

Avoid planning and designing for change and future requirements

When starting a project, accomplished software professionals often insist on a period devoted to design or architecture, in which the overall technical approach to the project is determined. When things go wrong later, inadequate or insufficient up-front design is often cited as the reason. “We’re gong to insist on being able to do it right next time, so this won’t happen again.” But all too often, projects slip and slide regardless of any up-front design work.

Generally, up-front design time is actually counter-productive. The reasons are simple: you spend time “designing” when you could have been building; the result of the design is often building for anticipated change, which takes time; when the anticipated change doesn’t happen and unanticipated change arrives, the code added because of the design almost always makes things harder.

None of this happens when you build incrementally and strive toward uniformity, eliminating code redundancies and moving functionality from code into meta-data.

Wartime software

The methods I have described as wartime software maximize speed and efficiency of building new software that meets business needs. See here for details. Generally, the methods include:

• Eliminate overhead
• Avoid things resembling “project management”
• Optimize for speed instead of expectations
• Use production environments for testing instead of sand boxes
• Use comparison QA; avoid test-driven development, unit testing and test scripts
• Minimize the use of documents, meetings and other internal overhead
• The main architectural value should be speed and scale
• Avoid technical fashions

Move up the mountain of abstraction

When you think about code and methods, the main values should be the elimination of redundancy in both data definitions and code, and migrating functionality from code into meta-data. See this and this.

The other methods described here are important for moving quickly, and are complementary to this one. But moving up Abstraction Mountain is the most powerful of the methods, often yielding qualitative improvements.

Conclusion

This is a short post, and the methods are described briefly. In a different world, each topic here would be at least a course in the Computer Science curriculum, and in a couple cases multiple books. I don't claim anything I've said here is fully described. But every single item has been proven in practice by many groups over many years, in different ways, always resulting in better quality software that meets business needs being built dramatically more quickly than other methods could possibly have achieved.

My goal here is simply to provide a short list of the main methods for fast development in a single place.

Recurring Software Fashion Nightmares

Computer software is plagued by nightmares. The nightmares vary.

Sometimes they are fundamentally sound ideas that are pursued in the wrong way, in the wrong circumstances or at the wrong time. Therefore they fail – but usually come back, sometimes with a different name or emphasis.

Sometimes they’re just plain bad ideas, but sound good and are cleverly promoted, and sound like they may be relevant to solving widely acknowledged problems. Except they just don’t work. Sometimes these fundamentally bad ideas resurface, sometimes with a new name.

Sometimes they’re a good idea for an extremely narrow problem that is wildly applied beyond its area of applicability.

The worst nightmares are the ones that slowly evolve, take hold, and become widely accepted as part of what “good programmers” believe and do. Some of these even become part of what is ludicrously called “computer science.” Foundation stones such as these, accepted and taught without proof, evidence or even serious analysis, make it clear to any objective observer that computer “science” is anything but scientific, and that computer “engineering” is a joke. If the bridges and buildings designed by mechanical engineers collapsed with anything close to the frequency that software systems malfunction, the bums and frauds in charge would have long since been thrown out. But since bad software that breaks is so widespread, and since after all it “merely” causes endless delays and trouble, but doesn’t dump cars and trucks into the river, people just accept it as how things are. Sad.

Following is a brief review of sample nightmares in each of these categories. Some of what I describe as nightmares are fervently believed by many people. Some have staked their professional lives on them. I doubt any of the true adherents will be moved by my descriptions – why should they be? It was never about evidence or proof for the faithful to start with, so why should things change now?

Sound Ideas – but not here and now

• Machine learning, AI, OR methods, most analytics.
  • These things are wonderful. I love them. When properly applied in the right way by competent people against the right problem at the right time, amazing things can be accomplished. But those simple-sounding conditions are rarely met, and as a result, most efforts to apply these amazing techniques fail. See this.
    

Bad ideas cleverly promoted

• Microservices.
  • Microservices are currently what modern, transformative CTO’s shove down the throats of their innocent organizations, promising great things, including wonderful productivity and an end to that horror of horrors, the “monolithic code base.” While rarely admitted, this is little but a re-incarnation of the joke of Extreme Programming from a couple decades ago, with slightly modified rhetoric. A wide variety of virtues are supposed to come from micro-services, above all scalability and productivity, but it’s all little but arm-waving. If software valued evidence even a little, micro-services would be widely accepted as the bad joke it is.
• Big Data
  • There's nothing wrong in principle with collecting data and analyzing it. But in practice, the whole big data effort is little but an attempt to apply inapplicable methods to random collections of data, hoping to achieving generic but unspecified benefits. See here.
    

Great ideas for a narrow problem

• Blockchain.
  • My favorite example of an excellent solution to a problem that has been applied way beyond its area of legitimate application is blockchain. Bitcoin was a brilliant solution to the problem of having a currency that works with no one in charge. How do you get people to “guard the vault” and not turn into crooks? The way the mining is designed with its incentives and distributed consensus scheme brilliantly solves the problem. However, the second you start having loads of crypto-currencies, things get weaker. Once you introduce “smart contracts,” there’s a fly in the ointment. Take away the currency and make it blockchain and you’ve got a real disaster. Then “improve” it and make it private and you’ve got yourself a full-scale contradiction in terms that is spectacularly useless. See here.

Bad ideas that are part of the Computer Science and Engineering canon

• Object-orientation.
  • Languages can have varying ways of expressed data structures and relating to them. A particular variant on this issue called “object-orientation” emerged decades ago. After enjoying a heyday of evangelism during the first internet bubble, O-O languages are taught nearly universally in academia to the exclusion of all others, and adherence to the O-O faith is widely taken to be a sign of how serious a CS person you are. For some strange reason, no one seems to mention the abject failure of O-O databases. And no one talks about serious opposing views, such as that of Linus Torvalds, the creator/leader of the linux open source movement, which powers 90% of the world's web servers. Programmers who value results have long since moved on, but academia and industry as a whole remains committed to the false god of O-O-ism.
• Most project management methods.
  • I’ve written a great deal about this, including a book. Whether it’s the modern fashion of Agile with its various add-ons such as SCRUM, project management methods, ripped from other disciplines and applied mindlessly to software programming, have been an unmitigated disaster. The response? It ranges from “you’re not doing it right,” to “you need to use cool new method X.” Yet, project management is a profession, with certifications and course on it taught in otherwise respectable schools. A true nightmare. See these.
    
• Most software QA methods.
  • All software professionals accept what they’ve been taught, which is that some kind of software automation is an essential part of building software that works and keeping it working. Except the methods are horrible, wasteful, full of holes and rarely make things much better. See this.
    

Conclusion

I make no claim that the list of items I’ve discussed here is comprehensive. There are things I could have included that I have left out. But this is a start. Furthermore, if just half the items I’ve listed were tossed and replaced with things that actually, you know, worked, much better software would be produced more quickly than it is today. I don’t call for perfection – but some progress would sure be nice!

Evidence-based Software

Have you heard of "evidence-based medicine?" It's a relatively new trend in medicine based on the idea that what doctors do should be based on the evidence of what works and what doesn't. What's scary as a patient is the thought that this is a new idea. What is it replacing? Voo-doo-based medicine?

At least the field of medicine has accepted that evidence matters. So much better than not!

Let's turn to software. Have you ever heard of evidence-based software? Of course not! There is no such thing! How software is built is based on loads of things, but sadly, evidence is not among them. Among other things, this explains why software projects fail, and/or result in expensive, rigid bloat-ware that is riddled with errors and security holes.

The Golden Globes 2016

One of the reasons to watch the Golden Globes awards ceremony is for the fashion. Everyone knows it -- which is why there's a multi-hour Red Carpet pre-show, and even a pre-show to the red carpet show.

You watch the show if you want to see what the new fashions are. You wouldn't want to look silly, would you? If you watched this year's show, you could see Amanda Peet looking really nice:

And you could see Sarah Jessica Parker looking like something else altogether:

I heard the expert on one of the shows talking about the new colors and lines in the dresses, something we'd see more of in the upcoming year.

What's the "best" fashion? The one leading people seem to like. What will be the best fashion next year? About all you can be certain about is that it will be something different from what was most-liked this year.

Software development fashions

The methods used in software development are selected with just about the same criteria as the leading fashions in dresses. Who's wearing what? What do leading people think? What did I use (wear) last time that got admiring looks?

Fashions come into software development. They get promoted. They get used and mis-used, adapted and morphed. Programmers take them with varying degrees of seriousness. Wherever you're programming, you have to more or less go along with the prevailing fashion. If everyone else crosses themselves, you'd better too. If there's a daily stand-up, you'd better stand up when everyone else does, and not look too abstracted or bored.

Effectiveness, Productivity and Quality

In fashion, you want the admiration of other people who look at what you're wearing. In software, since you spend most of your time building software, it's nice to have the admiration of people who look at you building software.

But unfortunately, other points of view sometime intrude. Managers want to know about budgets and productivity and deadlines. After the software is put into use, there are ignorant and annoying users to contend with. What you've worked so hard to build is never enough. They complain about it! Crashes, performance, quality issues? Sometimes people get upset. And security? Rule number one is keep it quiet! The last thing we need is this getting into the papers!

Then you find out that most outsiders could care less what goes on in the sausage factory. Whether it's organized or chaotic, ugly or pretty, in the end all they seem to care about is how the sausage tastes. These simple-minded people can only keep one thing in their heads at a time, and that one thing is most often: the results!

Wouldn't it be nice if we had a way of picking through the dozens of software methods that are in widespread use, and based on the evidence, settle on just a couple that were the best that actually ... produced the best results!!?? Or maybe that's just too radical a thought.

That's why we need something like evidence-based software -- or at least acknowledgement that it could help things out.

Coda: EBSE: Evidence-Based Software Engineering

I started writing this blog post based on the comparison to evidence-based medicine as a way to frame the fashion-based chaos that surprisingly rules the day in this highly exacting field of work. I certainly had never heard the phrase "evidence-based software." But as a check before clicking "publish," I thought I'd better do a quick search. Imagine my surprise when I found that there is, indeed, something called EBSE, evidence-based software engineering, explicitly inspired by the analogy in medicine!

I've interacted with a large number of software engineering groups over the last twenty-plus years, and been inside a few for many years prior to that. The groups have been highly varied and diverse, to put it mildly. I've seen loads of trends, languages, methodologies and tools. And never -- not once! -- have I heard of the existence of EBSE. It should be just what we need, right?

So I dove in. It's sad. Or pathetic. Both, actually.

There's a moribund website on the subject:

• It doesn't have a domain name, it's just hosted at some obscure university in the UK midlands.
• The last "news" is from 2011. Not much happenin'...
• All the "evidence" appears to come from published academic papers -- you know, those things that practicing software people absolutely depend on.
• "The core tool of the evidence-based paradigm is the Systematic Literature Review (SLR)..." The SLR is basically a meta-analysis of lots of published academic papers. Whoopee!
• The whole thing is organized "using the knowledge areas defined by the SWEBOK."
• I couldn't find a single useful thing in the whole pile of words.

The "SWEBOK"??? Another thing I've never heard of. It turns out it's an official IEEE guide to the Software Engineering Body of Knowledge. This essential guide tells us everything that leading academics are convinced are must-knows, "generally accepted knowledge about software engineering." If only I had known! Think how much trouble I could have saved myself and others over the years! Best of all, it's practically up-to-date -- just over 12 years old!

EBSE and SWEBOK are great demonstrations of just how bad things are in the software field: even when you start with a great metaphor, you still make no progress if you continue to accept as gospel the broken assumptions that the field's academics take to be eternal TRUTH. The sad fact is, math and computer science are at fundamental odds with effective software development. As I've shown. Sad, but true.

Having something like evidence-based medicine for software instead of the ugly, ineffective chaos we have today would be nice. EBSE is a nice name, but as a reality, a non-starter.

The Fundamentals of Speed-optimized Software Development

How do you write software more quickly? How do you engage and win with Wartime Software? There is a set of techniques that enable dramatic speed-ups in producing effective, high-quality software, and they share a set of underlying principles. These fundamental principles of software development are practically never discussed, yet they are almost painfully simple. Understanding and applying them can help you adapt existing speed-optimized methods to your needs and evolve new ones.

Mainstream thinking about speed

Very few groups optimize for speed in software development. By far the most common factor that is optimized is expectations. Given that everyone involved in software knows about, believes in and practices expectation-based development, the options for speeding things up aren't great.

The most widely used method for speeding things up is simple: pressuring everyone involved to go faster. Do whatever it takes, darn it, we're going to bring this project in on time! What this amounts to is:

• People work harder, longer hours and extra days. Result: sloppiness, lack of care, increasing number of mistakes and omissions, greater risk.

• People squeeze or omit essential steps, particularly later in the project. Result: poor quality and even disasters encountered during roll-out.

• Fashionable new methods are introduced that re-arrange things or add steps. Result: see above.

The fundamentals of achieving speed

The fundamental principles of achieving speed in software are extremely simple: take fewer, more effective steps to the goal and eliminate overhead. When framed in all the stuff we know about software development, this can be abstract and difficult to apply in practice. But it's truly simple. Think about walking to a goal. The principles amount to:

Walk to the destination as directly as possible; avoid stopping and re-tracing your steps.

That's not so hard, is it?

I know, it is hard, because standard software development methods are worlds away from thinking about things in these terms. But that's because they're optimized for expectations and not for speed!

Replacing lots and lots with a little

Most of the steps in mainstream software development are there for a good reason. While concepts like project management may have been developed outside of software, they have been adapted to it. So if you're going to toss out some standard expectations-based activity, you'd better be sure there is speed-optimized activity that gets the same job done -- at least as well, and with lots less time and effort.

With all the effort and work that's gone into software development methods over the years, there's not a lot of improvement to be made by tweaking this or enhancing that. Radical methods are required to accomplish radical change. So it's important to ignore all the intermediate steps and focus on the only goal that matters: customer satisfaction. Period. Everything else is a stand-in, an intermediary, or supposed to contribute to that end.

If this is the goal, one of the fundamental principles of Wartime Software, of optimizing for speed and results, is this:

DO globally optimize for the goal.

DO NOT sub-optimize for any intermediate goals.

Once you globally optimize for reaching your goal, then you realize that there is a HUGE amount of redundancy in the normal "rigorous" software development process. The redundancy isn't obvious or even visible because mostly it's many repetitions with slight variations, each confined in a silo, performed by different people in different environments using different tools. So it doesn't look redundant, when examined in isolation. But when viewed globally? MASSIVE redundancy, and an opportunity to replace lots and lots of overlapping stuff with a little bit, done right.

Conclusion

Wartime software is not achieved by doing a re-organization or tweaking a process. It's achieved by a total re-conceptualization of the optimal way to get from Point A to Point B in software. Once you understand the fundamental principle of global optimization, with the goal of eliminating overhead and redundancy, all the detailed speed-optimized techniques follow like Q.E.D.

Building Better Software More Quickly

How can you build software more quickly, without just cutting corners, taking risks and turning out crap? The prevailing view in the industry is that building software is a tough job, and it just plain takes a long time. Different factions heavily promote methods that are supposedly better, but in the end, the fancy new methods are either minor variations on standard practice or make things worse.

We don't need to speculate to decide that great software can be built much more quickly than nearly everyone thinks -- there are people who are doing it today. I've written about this extensively. But in most environments, including government contracting and large corporations, the speed methods are forbidden. Yes, forbidden, as unthinkable as trying to walk in the office barefoot. The people who get it done don't just add some secret sauce to the standard recipes -- they have whole new ways of framing the problem.

Getting from the start to the finish

Most programming jobs are like getting from Point A (where your software is today) to Point B (where you'd like it to be).
All the widely accepted methods for getting from Point A to Point B take what they sincerely believe to be the shortest path.

Standard ways to travel

There are standard, authorized, certified and audited ways to make the trip. Some people feel passionately that their way is better than the others, but they all take a long time, and there are frequent wrecks on the road. The journey is always an arduous one, and has many phases, each requiring specialists with lots of experience and training to get the job done while avoiding disaster. Which nonetheless happens often enough. Even when you make it safely to the destination, all too often you either arrive late or without all the stuff you expected to arrive with.

Everyone agrees that there are lots of steps in getting software done, steps that require education, experience and skill of different kinds, and each step builds on successful completion of earlier steps.

In the early stages of the trip, you do lots of planning, because you know how tough trips like this tend to be. You go back and forth over nearly the same ground, never making lots of progress. It’s climbing a mountain, and the switchbacks are just a necessary part of what you have to do to get to the peak. Once you’re there, you can finally see your goal at the bottom of the mountain. There’s a clear path to get there. The path continues to have lots of risks, but at least you have a definite plan, you’re know exactly where you’re going (you can see it!) and how you’re going to get there. It feels more like you’re going down the mountain.

When you’ve got point B in sight, it’s important to move deliberately and carefully. If you skip steps, you can easily wreck everything and fall off a cliff. But still, it’s easier going because everything is spelled out and written down. You’ve brought new members onto the team, specialists who have gone down mountains like this many times.

Even though you're going in a straight line, it turns out it's not a straight line on level ground! It's a straight line that goes over a very high mountain, one with vertical walls, gaps and all sort of progress-killing obstacles. There are all sorts of people promoting ways of getting to point B that are supposedly faster or less error-prone. But in the end, you still have to climb the mountain and then go down to the destination. All the same work ends up getting done, but just in different combinations and arrangements and orders.

Speed-optimizing methods

There are a couple wacko’s out there who talk about nothing but “speed.” Right. Going faster is definitely the fastest way to end up in the ditch.

Yes, that's what nearly everyone thinks, and especially people with experience, people who have been part of or been subjected to the wide variety of disasters caused by train-wreck software development efforts (oops, sorry, I'm being redundant here).

When you look at what the people who get things done quickly actually do, you find that they reject the whole paradigm of project-management-oriented software development. They think about the whole process in different terms. There's still a Point B, but instead of accepting the common delusion that you can figure out how comfortable a bed is before you lie on it, they realize that you have to get close to where Point B probably is, and then look around and refine it, maybe by quite a bit! Maybe Point B sounded wonderful based on the speculate reports received while standing on Point A, but maybe once you're in the general region of Point B, nearby Point B1 is clearly superior. So you shift to it. Look around. Figure something else out, and shift to Point B2. Wow, how great is Point B2 -- let's build some comfy chairs and a roof so we can stay here a while -- but be sure to build the roof this way, and with a skylight over there, which is pretty obvious while sitting on a camp chair in the right place, but simply couldn't have been known way over there at Point A.

Conclusion

The speedy programmers who want to get to a destination find ways to avoid making arduous, risky climbs over mountains; instead they zoom to the general place quickly, learn stuff and refine. The detailed methods that speed-oriented developers use can seem mysterious, until you see that they're looking at things from an entirely different point of view.

Large Organization Software Fails: the case of Microsoft Windows

Large organizations have trouble building software. This has been true since the dawn of software history, and shows no signs of changing. The decades-long, rolling disaster of Microsoft Windows is a great example of this. I've been hit personally with this. Recent experiences with Windows 8 have renewed my appreciation of the breadth and depth of the on-going awfulness of Windows.

Windows Screen Saver

I got a new computer. It had Windows 8. I was setting up my new machine and I wanted to do something simple. I had remembered that in some earlier version of Windows, you could get the screen saver to display the file name of the photo it was showing. This was useful if you wanted to get your hands on the photo that just flashed by. It's a pretty small feature, but one anyone who stores photos on their PC could find it useful.

So I drilled in to the screen saver.

I went into the settings, and didn't see the control I was hoping would be there.

So I clicked on Help, something I rarely do, but what the heck, that's what it's there for. Here's what I got: The content is missing!

It's a little thing. It's not like my computer crashed. In the world of books, it's like a footnote was missing -- hey, that's an idea, let's compare the new edition of Windows to the new edition of a book!

Software and Books

Most of us know how to judge books. If a book is poorly produced, like the pages tear easily and the type is hard to read, most of us will toss it aside -- it may have great content, but it's not worth reading. If we get past the first impression, we'll dive in and start reading. The next potential barrier is how well the book has been edited. If the book is full of spelling, usage and grammatical errors, many of us will think poorly of the author, the editor and the publishing house -- the author shouldn't have made the mistakes in the first place, the editor should have caught and corrected them, and the publishing house shouldn't have put sloppy trash in print. Then and only then do we get to the style and substance of the book.

I read a lot of books from many publishers in many genres -- fiction, history, science, etc.  -- and I'm happy to report that I rarely encounter a published book that has editing errors.

And by the time a particularly timeless book gets to later editions? There are never errors.

In that context, how is Windows 8?

I've got the latest version of Windows, 8.1, running on a new machine. It's hardly a first edition. Microsoft pours out updates, and I'm up to date. Here's a snapshot:

Note the scroll bar -- there were hundreds more updates that had been applied.

The lovely option that lets you see the file name along with the picture was in an earlier version of Windows. Making a new edition of software isn't that much different than making a new edition of a book -- basically, unless you add or change something, it stays the same. In this case, someone had to make a conscious decision to drop an isolated, harmless feature that gave value to many customers.

Why would someone do that? It's more trouble to drop a feature than just let it ride along on the next edition, so someone had to actively remove it. There is no conceivable objection to the feature. While not everyone would want it, since it's an opt-in feature, it harms no one. It's like someone deciding to drop a short appendix from a book -- not everyone will want it, but those who do value it. In the paper publishing world, dropping it might save a page or two. But in the electronic world? There's no conceivable reason.

I don't claim for a second that displaying the file name on the screen saver is important. I simply claim that the decision to drop it exemplifies the pervasive anti-customer attitude of the Microsoft organization, which unfortunately is typical of large software-building organizations in general.

It's the missing Help file though, that really set me off. Again, it's a trivial error, like dropping a footnote. But why would you do it? How could it possibly slip though what should be a totally automated editing/QA process?? It may somehow be complicated in the labyrinthine world of Windows development, but it's a fixable thing. You have a program that assures that for each instance of Help there's a corresponding piece of content, and for each piece of content there's a way to reach it. There either is no such program or it's broken. In the overall scheme of things (Windows remains horrifically slow, it freezes and crashes, etc.) it's a small thing, but surely by the edition of Windows 8 I am suffering with it would have been found and fixed?

Conclusion

Software is all about productivity, attention to detail and automation. Unless you've got a de facto monopoly, software is also about meeting customer needs. Large organizations in general (for example government, big corporations) and Microsoft in particular don't get that, in spite of the billions they spend on development and (supposedly) quality. I would love to be able to say it's getting better, but most of the evidence is on the other side. Which is why, among other things, good software will continue to be produced mostly by organizations that are small and willing to do things the "wrong" way.

The Science of Drugs vs. the Science of Computers and Software

Prescription drugs are important elements of our lives. There is a strict, scientific, testing-based process to assure that drugs that become widely used are safe and effective, with known side effects. Computers and software are also important elements of our lives. There is a chaotic, fashion-trend-based process used to select the mixture of tools and techniques used to build, maintain and operate our IT systems, resulting in widespread failures, along with cost and quality problems. Worse, there is no recognition that this is the state of affairs, and no movement to correct the situation.

Pharmaceuticals

Everyone knows that drugs are important, and an important part of our economy. Here are some numbers from the CDC.
In 2013, we spent about $271 Billion dollars on prescription drugs. That's quite a bit, but just about 10% of national health spending.

I won't recount the process drugs go through to get approval from the FDA, but I think everyone knows it's an elaborate, multi-year and multi-stage process, with testing at each step to assure that we know how a proposed new drug will work in human beings. While I have my complaints, there is a process, and it's scientific and evidence-based.

IT

The IT industry is also a large one. Here's a breakdown of it worldwide.

There are conflicting estimates of its size in the US, but here's a representative one.

Note that the definition of IT does not include the activities of well-known IT-centric companies like Google.

I was fascinated to see that in 2013, IT was three times the size of the entire pharmaceutical industry. Amazing.

Drugs and IT

Drugs are developed by scientists. They are vetted by a strict scientific process. Only drugs that make it through all the tests are widely used. As a result, the vast majority of drugs are used safely and effectively by the vast majority of patients, with a few experiencing side effects that have already been identified.

IT is run by professionals and staffed with computer scientists and engineers, using tools and techniques developed over many years by scientists and engineers. No matter how high-profile and important the project, regardless of the involvement by government or private companies, a shocking fraction of IT projects end up late, too costly, ineffective or worse. Industry-accepted certifications seem to make no difference. New methods and techniques emerge, become talked about and are deployed widely without any evidence-based process being used to assure their safety and effectiveness. The industry is rife with warring camps, each passionately committed to the effectiveness of their set of tools and techniques. But there isn't even postmortem testing to see which ones were better at gaining its adherents admittance into IT "heaven."

Conclusion

I think the FDA-run drug acceptance process could be much better than it is. But the important thing is, everyone involved in prescription drugs understands and acts scientifically about the process. No one, including me, wants that to change.

The IT industry is at least three times the size of the drug industry. There are computer science and computer engineering departments in every major university, and their graduates staff the industry. It's hard to imagine that they don't understand science, scientific process and evidence-based reasoning. However: they adhere to faith-based processes and vendor-driven products that yield horrible results year after year. None of them say, "hey, this stinks, maybe we can apply that thing that Galileo, Newton and Einstein did, what's it called, science?"

The last thing I want is government involvement in IT, given how horribly government handles its own IT affairs, and I'm not suggesting it here. But it's a sign of just how bad things are in IT that the bureaucratic, government-run FDA does a more scientific job with drugs than anyone does with IT.

How to Feel Better about Software while making it worse

Everyone knows it's hard to build software. Even projects that are judged "successful" are often fraught with problems. The odd thing is that many of the steps people take to reduce the risk and increase the odds of success actually make things worse!

Trying to reduce the risk of software projects

At some level, everyone knows that software projects are risky and often fail. They really want to avoid failure, but the second one guy starts babbling about "object-oriented frameworks" and another guy rattles on about "Agile and a great SCRUM master," normal people get even more worried. "How can I avoid being road kill" is the fear causing the roiling of the intestines. So they insist on things that make them feel safe, all of which (perversely) are most likely to increase the time, cost and risk of failure!

These safe-feeling but risk-increasing items include (but are not limited to):

• Outsourcing the project using normal procurement channels and methods
• Selecting a large vendor to do the work
• Requiring lots of certifications among the organizations and people doing the work
• Selecting independent auditing, testing and other functions to assure the work is done well
• Interviewing the people in charge of the work, and accepting only those who make you feel comfortable

Each one of these merits an essay explaining why such common-sense steps make things worse. Empirically, they do. The spate of failures among the Obamacare implementations are a recent poster child, since the implementations involved most of the above "safety-increasing" elements.

Outsourcing

Outsourcing is a favorite. Huge organizations outsource all the time, even their whole IT function. But there is no evidence that the organizations that do the outsourcing do any better than the flailing organization that outsources. There is exactly one guarantee: having the work done under a different roof means that you are largely free of the responsibility, and largely from the stress of seeing the sausage factory in action.

Large Vendor

Choosing a large vendor is a tried and true way to make the buyer feel better and safer. You wouldn't buy a car built by someone you'd never heard of, would you? Of course not! So sensible people insist on dealing only with large, well-established vendors. Unfortunately for those sensible people, the things that work in most of our lives cause failure in software. Too bad!

Certifications

You wouldn't go to a restaurant that had failed a health inspection, would you? Or go to a doctor who had lost his license? Of course not. So a good way to feel safe is to find out what certifications are floating around the software industry and make sure your vendor has lots of them. Nice idea. Makes sense in most fields. But not in software. In software, you can be pretty sure that the more certifications they have, the worse they are at building software.

Independent checking

How do you know if they're really doing the work they say they're doing? We get our books audited by an outside firm, so doesn't it make sense to have the software audited by an outside firm of experts? Makes common sense. However, this is yet another example of how common sense makes things worse in software.

Personal interviewing

When all else fails, use your in-depth knowledge and experience with people to do your selecting. The trouble with this nice idea is that the person you're dealing with deals with yokels like you all day long, and you're not nearly as good as you think you are. Worse, the person you're interviewing either personally does the work (unlikely), in which case you have no clue at all, or they're just a sales person (most likely), in which case you're seriously outgunned. Forget it.

Conclusion

If software were easy, everyone would learn how to do it as kids, and be able to pick it up again after years of not having done it. We all know how to make risky decisions and processes less risky. The trouble is that most of those methods, which work pretty well in most of our lives, come up short in the wacky world of software, frequently making things worse.

Math and Computer Science vs. Software Development

In a prior post, I demonstrated the close relationship between math and computer science in academia. Many posts in this blog have delved into the pervasive problems of software development. I suggest that there is a fundamental conflict between the perspectives of math and computer science on the one hand, and the needs of effective, high quality software development on the other hand. The more you have computer science, the worse your software is; the more you concentrate on building great software, the more distant you grow from computer science.

If this is true, it explains a great deal of what we observe in reality. And if true, it defines and/or confirms some clear paths of action in developing software.

A Math book helped me understand this

I've always loved math, though math (at least at the higher levels) hasn't always loved me. So I keep poking at it. Recently, I've been going through a truly enjoyable book on math by Alex Bellos.

It's well worth reading for many reasons. But this is the passage that shed light on something I've been struggling with literally for decades.

When we learn to count, we're learning math that's been around for thousands of years. It's the same stuff! Likewise when we learn to add and subtract. And multiply. When we get into geometry, which for most people is in high school, we're catching up to the Greeks of two thousand years ago.

As Alex says, "Math is the history of math." As he says, kids who are still studying math by the age of 18 have gotten all the way to the 1700's!

These are not new facts for me. But somehow when he put together the fact that "math does not age" with the observation that in applied science "theories are undergoing continual refinement," it finally clicked for me.

Computers Evolve faster than anything has ever evolved

Computers evolve at a rate unlike anything else in human experience, a fact that I've harped on. I keep going back to it because we keep applying methods developed for things that evolve at normal rates (i.e., practically everything else) to software, and are surprised when things don't turn out well. The software methods that highly skilled software engineers use are frequently shockingly out of date, and the methods used for management (like project management) are simply inapplicable. Given this, it's surprising, and a tribute to human persistance and hard work, that software ever works.

This is what I knew. It's clear, and seems inarguable to me. Even though I'm fully aware that the vast majority of computer professionals simply ignore the observation, it's still inarguable. The old "how fast do you have to run to avoid being eaten by the lion" joke applies to the situation. In the case of software development, all the developers just stroll blithely along, knowing that the lions are going to to eat a fair number of them (i.e., their projects are going to fail), and so they concentrate on distracting management from reality, which usually isn't hard.

What is now clear to me is the role played by math, computer science and the academic establishment in creating and sustaining this awful state of affairs, in which outright failure and crap software is accepted as the way things are. It's not a conspiracy -- no one intends to bring about this result, so far as I know. It's just the inevitable consequence of having wrong concepts.

Computer Science and Software Development

There are some aspects of software development which are reasonably studied using methods that are math-like. The great Donald Knuth made a career out of this; it's valuable work, and I admire it. Not only do I support the approach when applicable, I take it myself in some cases, for example with Occamality.

But in general, most of software development is NOT eternal. You do NOT spend your time learning things that were first developed in the 1950's, and then if you're good get all the way up the 1970's, leaving more advanced software development from the 1980's and on to the really smart people with advanced degrees. It's not like that!

Yes, there are things that were done in the 1950's that are still done, in principle. We still mostly use "von Neumann architecture" machines. We write code in a language and the machine executes it. There is input and output. No question. It's the stuff "above" that that evolves in order to keep up with the opportunities afforded by Moore's Law, the incredible increase of speed and power.

In math, the old stuff remains relevant and true. You march through history in your quest to get near the present in math, to work on the unsolved problems and explore unexplored worlds.

In software development, you get trapped by paradigms and systems that were invented to solve a problem that long since ceased being a problem. You think in terms and with concepts that are obsolete. In order to bring order to the chaos, you import methods that are proven in a variety of other disciplines, but which wreck havoc in software development.

People from a computer science background tend to have this disease even worse than the average software developer. Their math-computer-science background taught them the "eternal truth" way of thinking about computers, rather than the "forget the past, what is the best thing to do NOW" way of thinking about computers. Guess which group focusses most on getting results? Guess which group would rather do things the "right" way than deliver high quality software quickly, whatever it takes?

Computer Science vs. Software Development

The math view of history, which is completely valid and appropriate for math, is that you're always building on the past, standing on the shoulders of giants.

The software development view of history is that while some general things don't change (pay attention to detail, write clean code, there is code and data, inputs and outputs), many important things do change, and the best results are obtained by figuring out optimal approaches (code, technique, methods) for the current situation.

When math-CS people pay attention to software, they naturally tend to focus on things that are independent of the details of particular computers. The Turing machine is a great example. It's an abstraction that has helped us understand whether something is "computable." Computability is something that is independent (as it should be) of any one computer. It doesn't change as computers get faster and less expensive. Like the math people, the most prestigious CS people like to "prove" things. Again, Donald Knuth is the poster child. His multi-volume work solidly falls in this tradition, and exemplifies the best that CS brings to software development.

The CS mind wants to prove stuff, wants to find things that are deeply and eternally true and teach others to apply them.

The Software Development mind wants to leverage the CS stuff when it can help, but mostly concentrates on the techniques and methods that have been made possible by recent advances in computer capabilities. By concentrating on the newly-possible approaches, the leading-edge software person can beat everyone else using older tools and methods, delivering better software more quickly at lower cost.

The CS mind tends to ignore ephemeral details like the cost of memory and how much is easily available, because things like that undergo constant change. If you do something that depends on rapidly shifting ground like that, it will soon be irrelevant. True!

In contrast, the Software Development mind jumps on the new stuff, caring only that it is becoming widespread, and tries to be among the first to leverage the newly-available power.

The CS mind sits in an ivory tower among like-minded people like math folks, sometimes reading reports from the frontiers, mostly discarding the information as not changing the fundamentals. The vast majority of Software Development people live in the comfortable cities surrounding the ivory towers doing things pretty much the way they always have ("proven techniques!"). Meanwhile, the advanced Software Development people are out there discovering new continents, gold and silver, and bringing back amazing things that are highly valued at home, though not always at first, and often at odds with establishment practices.

Qualifications

Yes, I'm exaggerating the contrast between CS and Software Development. Sometimes developers are crappy because they are clueless about simple concepts taught in CS intro classes. Sometimes great CS people are also great developers, and sometimes CS approaches are hugely helpful in understanding development. I'm guilty of this myself! For example, I think the fact that computers evolve with unprecedented speed is itself an "eternal" (at least for now) fact that needs to be understood and applied. I argue strongly that this fact, when applied, changes the way to optimally build software. In fact, that's the argument I'm making now!

Nonetheless, the contrast between CS-mind and Development-mind exists. I see it in the tendency to stick to practices that are widely used, accepted practices, but are no longer optimal, given the advances in computers. I see it in the background of developers' preferences, attitudes and general approaches.

Conclusion

The problem in essence is simple:

Math people learn the history of math, get to the present, and stand on the shoulders of giants to advance it.

Good software developers master the tools they've been given, but ignore and discard the detritous of the past, and invent software that exploits today's computer capabilities to solve today's problems.

Most software developers plod ahead, trying to apply their obsolete tools and methods to problems that are new to them, ignoring the new capabilities that are available to them, all the while convinced that they're being good computer science and math wonks, standing on the shoulders of giants like you're supposed to do.

The truly outstanding people may take computer science and math courses, but when they get into software development, figure out that a whole new approach is needed. They come to the new approach, and find that it works, it's fun, and they can just blow past everyone else using it. Naturally, these folks don't join big software bureaucracies and do what everyone else does. They somehow find like-minded people and kick butt. They take from computer science in the narrow areas (typically algorithms) where it's useful, but then take an approach that is totally different for the majority of their work.

Software Problems: the Role of Incentives

When lots of human beings work at something for a long time, they tend to figure out how to do it. Building software appears to be a huge exception to that rule. With decades of experience under our belt, why is it that we still can't build good software?

One of the reasons software projects so often fail and improved methods aren't used appears to be that the people involved have perverse incentives.

Incentives

Everyone knows about incentives. They work. Even when we know someone is using incentives to get us to do something, we're more likely to do the thing with incentives than without them.

Perverse Incentives

Whether an incentive is perverse or not is in the eye of the beholder. From the incented person's point of view, an incentive is an incentive, and as we know, incentives work. But we normally call incentives "perverse" when they incent people to do something that most other people would agree is a bad thing.

Perverse Incentives: Mortgages

The housing boom leading up to the financial crash of 2007 was clearly driven by perverse incentives on multiple fronts. Borrowers were tempted to take what seemed to be easy money. Mortgage companies could make piles of money in fees by packaging up risky mortgages and passing them on. Rating agencies could collect loads of fees by not looking too closely. And the bankers at the top of the food chain made themselves lots of money by creating and selling fancy instruments that ignored the underlying realities and the ultimate consequences of their actions. Then it all came crashing down. Many were hurt, the big guys who made the most money least of all.

Perverse Incentives:The VA System

It has recently come out that more than 120,000 veterans are experiencing long waits for care at VA hospitals, even while official reports showed minimal wait times, enabling managers to collect incentive payments. If there ever was a case of perverse incentives leading to bad behavior, this is it.

Perverse Incentives in Software

Software is so rational, so organized, the people involved are so smart and well-educated -- surely perverse incentives aren't driving behavior in software, are they?

Sorry, sweetie, perverse incentives are a human issue. Humans respond to incentives, perverse or otherwise. And as it turns out, there is a rogue's gallery of perverse incentives operating in software -- I will only scratch the surface here!

Estimates

Estimates are perverse all by themselves.

They are also a GIANT BILLBOARD incenting EVERYONE involved in the process to make any estimate as long as they can possibly get away with; and since very few people (often including the programmer involved!) has any idea how long something *should* take, the estimates are typically accepted as is; but then, manager often double the estimates before passing them on. Why is this perverse?

The organization probably would like to get something done in the shortest reasonable time. But the programmers and project people are measured on whether they beat or miss the estimate. The longer the estimate, the better the chances of avoiding failure. It's that simple. It just makes it all the more maddening that, even with inflated estimates, things still go wrong!

Requirements

The whole modern software development process starts from requirements. Gamesmanship around requirements is therefore front-and-central. Estimates are based on requirements, and therefore controlling and fixing the requirements is central to the effort of creating "success." The system may fail, the users may hate it, but if it meets the "requirements," the people running the project get to declare "success." What you'd like is for the project to succeed when the needs of the business are met. The perverse incentive is for the people delivering the system to define "meeting the requirements" and then control the requirements to assure that they're met, regardless of what disasters happen to the business.

False reporting

Just like at the VA, project managers are highly incented to avoid reporting problems -- typically using big fancy reports that are chock full of meaningful-seeming stuff but are in fact just garbage. Just like in the mortgage-driven financial crisis, everyone involved is incented to declare success, take their rewards, and kick the can down the road for the next guy. Eventually, with shocking speed, it all comes crashing down, just like the financial system, and just like the mere 4 days between the laudatory article about how great Cover Oregon was going to be and the admission of total failure.

False Assessments

Here's where the rubber meets the road. Who is incented to blow the whistle on a failing software project? How, when and by whom is a software project judged to have failed? Most importantly: what are the consequences of having failed?

We all know the answer. Who has even heard of a software engineer who was fired for failure to deliver? And the people in charge? Never. It wasn't their fault! And the project didn't fail anyway! The requirements changed every month, the target kept moving, and blah, blah, blah.

Conclusion

Your kid comes up to you and asks, "can I play my video game now?" You briefly think about how your question when you were that age was "Can I go out and play now," but the kid isn't interested, and is bouncing around waiting for your "sure." Being the aspiring adult you are, you act responsibly and ask "Have you done your homework?" There's a brief pause. The kid is doing a quick risk-reward ratio calculation. If he says "yes," he probably gets to do what he wants. But you might ask to check. Hmmm.

This is the breeding ground of perverse incentives. We all learn to balance honesty, openness and getting what we want. Some of us go for honesty and openness, deciding that anything else just isn't worth it. But loads of people make an informed judgment on a case-by-case basis, much like the kid and his homework.

Whatever the morality of the case, the facts are clear: software projects fail left and right, and perverse incentives are a significant factor in making them fail. Without changing the incentives, we're unlike to abandon the Bad Old Way of building software and achieve success.

Joe Torre and Software Development

Joe Torre had an outstanding run as manager of the NY Yankees baseball team. While managing baseball seems pretty distant from managing software development, there are nonetheless a couple of important lessons to be learned. Put simply, baseball has it right and software has it wrong: if we chose software managers using the common-sense methods that are widely accepted in baseball, our software development track record would emerge from its current long, dismal, always-agonizing depression.

Joe Torre

Maybe not everyone knows who Joe Torre is. Now retired, he was a baseball player and manager.

Joe Torre had an excellent career as a player, from 1960 to 1977. He was an all-star 9 times, was NL MVP once, and was the NL batting champion once. Unusually for a baseball player, he had extended playing time at multiple positions: catcher, first base and third base.

He went on to have a stellar career as a manager, from 1977 to 2010. His Yankees won the World Series 4 times. He was AL manager of the year twice. His NY Yankees #6 was retired.

Players and Managers

Are most managers former players? Is Joe Torre the exception? Loads of baseball fans imagine they can do a better job than their home team manager. The owners have their own opinions on the subject. How hard can it be?

I looked into this question. There is a list of every baseball team manager from the start of the game. The list gives lots of information, including the manager’s history as a player (or not).

Here are the facts: as of today, there have been 686 managers of major league baseball teams, starting in 1871. Of those, 566 were former players, while 120 were never players. So the numbers show that the vast majority of managers have been former players. Just 17% of managers since 1871 were never players.

Is it just Sports?

My go-to example in music is, of course, Franz Liszt, who excelled as a performer, composer and conductor.

But he was hardly alone. The NY Times says

Times have not really changed. In Bach's day composers played their music at keyboards and conducted the instrumentalists about them. Beethoven conducted. So did Berlioz, Mendelssohn, Wagner, Mahler and Strauss. In our day composers are still conducting...

Here’s Gerard Baker, managing editor of the Wall Street Journal.
Mr. Baker is an accomplished writer, an excellent reader and judge of other peoples’ writing.

A pattern seems to be emerging here

Yes, it’s a pattern. Can you imagine a CFO who can’t add? A managing editor who not only can’t write, but can’t even read? How about a museum director who not only isn’t an artist, but can’t see?

Let’s apply the pattern to software!

Oh! Ummmm...

"Well, software is just another thing that can be managed by good management techniques!"

"I don’t need to know the details – I manage for results!"

Can we talk about something else now please?

Conclusion

The best qualifications for managing software in general and programmers in particular has never been a hot topic. In spite of all the evidence of massive failure, I doubt it will become a hot topic any time soon. But it should be! Just think about the basics here: however peculiar you may think writers are, do you really think editors don’t need to be able to read and write themselves? You may think of accountants as people with thick glasses hunched over desks with green-shaded lights, but do you really think the CFO doesn’t need to be able to add? Programmers may be weird, but doesn’t similar thinking apply?

Postscript

While 83% of baseball managers were players, 17% were not, among them some excellent managers. I'm not saying that only former programmers can manage programming efforts, and I know a couple truly excellent non-programmer managers. But in each case, they do interesting special things that are not widely understood that enable them to achieve excellent results.

Building Software: the Bad Old Way and the Good New Way

Software is hard to build. There are lots of failures. When the stakes are high, all parties concerned are “highly qualified” and failure isn’t an option – there are still lots of failures.

What is it about software? If cars failed anywhere close to the rate at which software fails, everyone would be afraid to ride in them. If houses failed at the rate of software, we’d see an explosion in people living in tents. Like it or not, building software is different from building almost anything else.

The reason why there are so many software failures is simple: most people think that building software is pretty much the same as building anything else, and they apply the same methods and criteria to it. Even collections of supposedly smart people who start out designing processes specific to software end up throwing in the towel and admitting that their processes are applicable to nearly anything – which is a good way of distracting people from the fact that they don’t work for software!

Knowledge of the bad old way to build software is widespread. It’s pretty simple, when you come right down to it. It’s not much different than, say, building a house. You start with requirements (how many bedrooms do you need, etc.) and budget. You work with an architect and get a set of plans (the detailed design). You put it out to bid, and select a contractor based on price, time and quality. The contractor gets permits, builds the house, you make progress payments along the way, there are inspections, a final payment and finally you move in.

When you try to apply this process to software, things fall apart quickly. I won’t go through the awful details, but imagine you weren’t allowed anywhere close to the job site until move-in day, and that most of the work is done by kids who are learning as they go, led by managers who have never used the tools or materials the kids are using. The Dirty Secret of Peacetime Development is gruesome.

The Good New Way to build software is just as simple, but rarely practiced. Here it is. Have the developers talk with the users and figure out what’s needed. Not for weeks – for a couple of hours. Then they should build something, taking hours or a couple of days at most. The “something” may not do much, but it should kinda work. Then they should show it to the users and have another discussion about what should be added, what should be changed. The programmers get back to work. No control freaks allowed! “That’s not what I meant” and “I forgot to add that” are just fine. It’s best if everyone agrees that progress is more important than perfection. When the software gets to elementary school age, it should be sent to school – how “real” users react to your precious child will add value and give perspective. If things are going well, the kid can skip grades. Repeating grades is OK too. Gradually you let the kid out on play dates and even summer camp. There’s never a date when the kid is “done;” there’s just increasing independence, and fewer visits back home.

One of the biggest advantages of this method is the elimination of the classic “big surprise,” that heart-pounding moment of the big demo a few days prior to launch. In an article published September 24, 2013, officials of the Oregon Health Exchange stated:

The big surprise happened just 4 days later, on September 28, 2013:

There you have it. The nearly-ubiquitous Bad Old Way of building software, as illustrated at great trouble and expense by Cover Oregon (among many others).

The Bad Old Way masquerades as a heart stress test and as reputation roulette for practically everyone involved. On the other hand you have the Good New Way, the apparently chaotic but amazingly effective way of building software, as exemplified by start-ups, people under pressure and generally speaking programmers who don’t have the time, money or perhaps patience to build things the bad way, and who end up adopting Wartime Software methods. And simply whupping the competition.

 

Delivering Software is a Nightmare

Blackliszt is down!

That led me to reflect on how nice things are when they work, but how many things can go wrong.

Building and delivering software is nightmare-hard. Given all the difficulties, it sometimes amazes me that anything ever works. This is not news – it’s just the way things are. But I’m going to illustrate it with current events.

Attacks from the outside

My blogging platform has been under attack for days. Here's what I got when I went there.

Here's a little bit about what's going on.

Bad guys attacking! For days and days at a time!

Attacks from the inside

It's bad enough that there are hordes of tough, effective bad guys roaming around outside the walls looking to cause trouble. But loads of folks inside the castle walls, people who are supposed to be good guys, aren't.

This is a huge problem. It isn't widely accepted.

Brute incompetence in government systems

There’s an on-going, massive failure of government computer systems. 
 
10 days ago as of this writing. A whole nationwide part of the Department of Justice has been thrown back to pre-computer days. Details were revealed here.

When you build things the right way, no fault can take a system down. When you build things the wrong way, it might take an hour or two to fix things. If you build a system an unimaginably stupid way, it might take a day. Like an anti-Manhattan Project, no bureaucracy other than the government could possibly have a system that someone would know would take a couple weeks to repair. But that's how it is here!

Amazing brain-dead fails in non-government systems

I got an e-mail today telling me about a wonderful new storage system. They claimed that Gartner had dubbed them a “cool vendor,” which I’d never heard Gartner doing and sounded weird.

So I thought I’d check it out. Here’s the result.

This wasn’t a fluke. Their whole website, not just the e-mail landing page, gave the same result. Good work, guys! Good to see the private sector showing everyone how the discipline of profit-making leads to great computer systems results!

Chaos and obsolescence among the “experts”

When you walk into any computer systems organization, you have no idea what you’re in for. This is because most computer organizations make the Keystone Kops

look like Eliot Ness and the Untouchables.

Put aside all the politics and infighting; software “experts” can be continents and decades away from each other in terms of how to get things done. Most software people can’t even follow the arguments, much less decide what the right answer is. It’s amazing these people manage to get out of bed in the morning.

When it “works” is it vaguely usable?

It’s not under attack, from the outside or the inside. It’s not broken. The internal comedy of errors has managed to deliver a piece of software that arguably “works.” Wonderful!

Sadly, this is where reasonable debate on software would start, not end. Because there are endless ways to get things done in software, and endless ways to interact with users. When you have software companies with loads of employees, reputedly smart, with tens of billions of dollars in the bank, and they trumpet their proud new creation -- and it’s a barely-usable piece of crap, the debate should be about how crappy-but-not-broken software gets shipped. Instead it’s far worse.

Conclusion

This is why smart, motivated start-ups beat rich established companies all the time. You’ve got 100 programmers for every one of mine? Great! The odds are on my side! Of course, the truth is that start-ups mostly fail. Their stuff doesn’t work or nobody cares or they just think they’re great when really they’re just like everyone else.

Nonetheless, a tough, smart, hard-working team can be literally 100 times more effective than the established players, including the cool modern ones like social media companies. Because building and delivering software that actually works and people want is a shockingly difficult thing to do.

Lessons for Software from the History of Scurvy

Software is infected by horrible diseases. These awful diseases cause painfully long gestation periods requiring armies of support people, after which deformed, barely-alive products struggle to be useful, live crippled existences, and are finally forgotten. Software that functions reasonably well is surprisingly rare, and even then typically requires extensive support staffs to remain functional.

Similarly, sailors suffered from the dread disease of scurvy until quite recently in human history. The history of scurvy sheds surprising light on the diseases which plague software. I hope applying the lessons of scurvy will lead to a world of disease-free, healthy software sooner than would otherwise happen.

Scurvy

Scurvy is caused by a lack of vitamin C. It's a rotten disease. First you get depressed and weak. Then you pant while walking and your bones hurt. Next your skin goes bad,

your gums rot and your teeth fall out.

You get fevers and convulsions. And then you die. Yuck.

The Impact of scurvy

Scurvy has been known since the Egyptians and Greeks. Between 1500 and 1800, it's been estimated that it killed 2 million sailors. For example, in 1520, Magellan lost 208 out of a crew of 230, mainly to scurvy. During the Seven Years' War, the Royal Navy reported that it conscripted 184,899 sailors, of whom 133,708 died, mostly due to scurvy. Even though most British sailors were scurvy-free by then, expeditions to the Antarctic in the early 20th century were plagued by scurvy.

The Long path to Scurvy prevention and cure

The cure for scurvy was discovered repeatedly. In 1614 a book was published by the Surgeon General of the East India company with a cure. Another was published in 1734 with a cure. Some admirals kept their sailors healthy by providing them daily doses of fresh citrus. In 1747 the Scottish Naval Surgeon James Lind proved (in the first-ever clinical trial!) that scurvy could be prevented and cured by eating citrus fruit.

Finally, during the Napoleonic Wars, the British Navy implemented the use of fresh lemons and solved the problem. In 1867, the Scot Lachlan Rose invented a method to preserve lime juice without alcohol, and daily doses of the new product were soon standard for sailors, which is how "limey" became synonymous with "sailor."

Competing Theories and Establishment Resistance

The effective cures that had been known and used by some people for centuries were not in a vacuum. There were competing theories. Cures included urine mouthwashes, sulphuric acid and bloodletting. As recently as 100 years ago, the prevailing theory was that scurvy was caused by "tainted" meat. How could this be?

We've seen this movie before. Over and over again. I told the story of Lister and the discovery of antiseptic surgery -- and the massive resistance to the new method by the leading authorities at the time.

Software Diseases

This brings us back to software. However esoteric and difficult it may be, software is a human endeavor: people create, change and use software and the devices it powers. Like any human endeavor, some of what happens is because of the subject matter, but a great deal is due to human nature. People are, after all, people, regardless of what they do. Patients were killed for lack of antiseptic surgery -- and the surgical establishment fought it tooth and nail. Millions of sailors were killed by scurvy, when a cure had been known, practiced and proved for centuries. Why would we expect any other reaction to cures for software diseases, when the "only" consequence of the diseases are explosive growth in the time, cost and risk to build and maintain software, which is nonetheless crappy and late?

Is there a general outcry about this dismal software situation? No! Why would anyone expect there would be? Everyone thinks it's just the way software is, just like they thought scurvy in sailors and deaths after surgery were part of life. Government software screws up,

software from major corporations is awful,

software from cool new social media companies is inexcusably bad. Examples of bad software can be listed for endless, boring, tedious, like forever lengths.

Toward Healthy Software Development

If I had spent my life in the normal way (for a software guy), I wouldn't be on this kick. But I didn't and I am on this most-software-sucks kick. Early on, I had enough exposure to large-group software practices to convince me that I wanted none of it. I'd rather actually get stuff done, thank you very much. Now, looking at many young software ventures over a period of a couple decades, the patterns have emerged clearly.

I have described the main sources of the problems. I have described the key features of disease-free software development. I have explained the main sources of the resistance to a cure, for example in this post. And I have no illusion that things will change any time soon.

It will sure be nice when the pockets of healthy software excellence that I see proliferate more quickly than they are, and when an anti-establishment consensus consolidates and gains visibility more quickly than it is. In the meantime, there is good news: groups that use healthy, disease-free software methods will have a massive competitive advantage over the rest. It's like ninjas vs. a collection of retired security guards. It's just not fair!

Who Makes the Software Decisions?

When the home team loses game after game, everyone starts wondering who's in charge, and shouldn't changes be made? Well, this is exactly what's happening in software. It's gotten so bad that it's making the front page of the tabloids.

Who's in charge in software? Who makes the decisions? Could it possibly be that the wrong people are empowered to make crucial decisions in software, and that things will only get better if we make a change?

Software decision makers

Who makes important decisions in software? The answer is obvious: anyone but programmers (ABP)! Programmers are the ones who do the real work: create and modify source code. They work with various tool sets in one or more programming environments and create the code that leads to the results needed by the business. You might think that programmers, therefore, would be front-and-center in selecting the toolsets and methods to use to get the desired results most effectively. This is rarely the case. Such decisions are typically made by people who are not, and in many cases never were, programmers.

What the industry thinks

I receive e-mail solicitations every day for various software products. I'm often invited to attend seminars or webcasts. Here is a typical example I received recently:

I'm not drawing attention to it because it's exceptional in any way. The next part of the solicitation is also typical, who should attend:

Who should attend? ABP, of course! Again, the company that sent the solicitation is not doing anything wrong. In fact, they're being smart. They are inviting the people who make important decisions about programming, i.e., ABP.

How things work in other fields

In just about any field you can think of, the more highly specialized and skilled the person doing the work, the more involved that person tends to be in all important decisions about the work. While kids starting out in baseball are told what gloves and bats to use, accomplished players have their own gloves and bats they have selected.

Even when the front-line people in other fields don't make the ultimate decisions, the important managers who do tend to be former front-line people.

Software is different

Things are different in software, of course. In sports, anyone can watch the game. They see the players on the field or court. TV commentators can circle a player on the screen and tell you to watch the thing they did, whether stupid or smart. Whatever it was, it makes sense to the viewer.

There is no equivalent in software! Software is invisible! (to everyone but programmers...) All those important decision-makers ever see is reports. Not only don't they open the hood and look underneath, they can't even see the car! The decision-makers largely rely on rumors and hearsay, but nonetheless develop strong opinions about how best to win on a playing field they can't see, where a game is played they can't play, following rules about which they are entirely ignorant. Hmmm, how is this going to turn out, I wonder???

It doesn't have to be this way!

There are places where business-as-usual in software decision-making is ... blatantly violated! Someone who ... wrote the code! ... is actually in charge of things. Of course, since he's a guy entirely without management training -- OMG, he doesn't even have an MBA, that's how bad it is! -- the place must be a disaster, right?

One such place is Athena Health. Athena powers doctor's offices. I first encountered them more than ten years years ago, when I had the pleasure of having a phone interview with the guy who wrote their code. I was hearing lots of skepticism, which is why I was having the call. This was still the time when internet bubble thinking ruled technology, and the rumor was that this guy was using "toy" technology and building something that "wasn't scalable." Heh.

The real problem was that the guy who wrote the code's skill was ... get ready for this ... writing good code, and making excellent decisions about writing code along the way! He had no experience in or talent for telling ignorant investors what they expected to hear. Bless him!

We invested, and the company has done, and continues to do, great. A couple years ago, I was pleased to have Ed Park, who is now EVP and COO of the company, attend my nerdfest, a gathering of top CTO's of companies I'm associated with. Here he is explaining something.

Ed wrote Athena's original code. He still knows stuff, and continues to make decisions based on the substance, not just go-through-the-motions process.

Conclusion

While lots of spinning goes on to disguise the fact, software projects typically fail, and even ones that "succeed" have crappy software. The Post was right (see above) to feature the question, "who makes the software decisions?" The industry's answer is clearly and unambiguously ABP (anyone but programmers). If you want the software your organization produces to sort of, actually, you know, work, you might want to think about removing the "ABP" restriction from the job requirement for software decision-makers.

When is Software Development "Done?"

Almost any activity you can think of, from building a road to composing a symphony, gets to a point where it's done. If not, something awful has happened, and you declare failure and move on. Software projects seem to be different, for no obvious reason. Quite frequently, software isn't a throw-it-out failure, but then it's not done either. What's going on here?

Building a house

Why can't software be like building a house? My uncle built a house for himself back in the 1950's. First came the foundation:

He did a great deal of the work himself, as much as he could:

All the way up to finishing the chimney for the fireplace and furnace:

And then it was done! He and his wife could enjoy a nice time with their nephews in front of the fireplace of the completed house:

Which actually was completed, unlike all those software projects, which drag on and on, refusing to get completed or to die. Perhaps this is why books and movies about zombies have become so popular!?

If houses were like software...

If houses were like software, instead of actually being done with them, they'd all be like the house built by Sarah Winchester, who bought an unfinished farm house in 1884, and spent the 38 years from then until her death having it worked on and expanded continuously, all day and all night. Here's a clip about it from an old magazine:

Building Software

Frequently, software projects are just failures. In spite of the traditional massive padding of estimates, things take even longer than projected. After the usual remedies (denial, punishing the innocent, rewarding the guilty, etc.) are exhausted, more money and resources are thrown at the project to "rescue" it. This inevitably has the effect of adding to the mountain of evidence supporting the thesis advanced by Fred Brooks in his classic "Mythical Man-Month" that adding resources to a late project makes it even later. Finally, the project is declared to be a "success" and promptly put on the shelf, never to be mentioned in polite company again, or the project in rare cases is declared a failure so that blame can be put onto the innocent target of some politically powerful person's agenda.

However, there are exceptions. I see such exceptions constantly in the growing, innovative companies I work with. These companies don't just grow. They learn, experiment, evolve, extend and sometimes take great leaps. As modern companies, they do this in close collaboration with their software, and frequently software is all or a major part of the service they provide.

Instead of thinking of the software as a house that needs to be designed and built, it's more appropriate to think of these companies as starting out with baby software that needs to keep growing and becoming stronger and more independent, like an infant grows to be a toddler and so on. If you stop developing software in this context, you guarantee the demise of the business. With a static business, it's appropriate to think of "finish or fail" as the relevant choices for software. With an innovative, growing business, it's appropriate to think of "evolve or die" as the relevant choices for software.

Conclusion

Everyone wants software to be like everything else: figure out what you want, build it, declare completion or failure, and move on. But when software is the engine that runs your business and you're trying to get on track to be a big success, the rules are different. In that case, the rules for software are: make the most important changes, figure out what is most important next, do it, clean up the software a bit, run some experiments, refine the winning approach, and keep evolving. Work fast, work accurate, be responsive, always learn, and keep learning. That's how you win with software.

 

Wartime Software Book Available

I've been threatening to release my book on Wartime Software. It is now available as a Kindle book.

Wartime Software is all about writing software when competition and speed matter. It's about releasing more often. It's about using new methods, as different as building bridges in peacetime and in time of war.

Here is the introduction, which should give you the idea.

Most people assume there is one “right” way to build software, and that’s that. While there are various fashion trends that infect software from time to time, none of them are as different as they like to think they are.

There are some important but little-discussed facts about the mainstream consensus of software development:

• It is mostly organized to give non-technical people confidence that things are OK, meaning on-time and on-budget. Its highest principle is predictability. Not speed.
• It mostly doesn’t work. Studies support what everyone in the field knows: most projects fail outright, or have their goals changed to avoid admitting failure.

So what we have are methods that are slow – and produce crappy results! What happened to slow but sure, or slow but steady? What we’ve got is slow and stupid.

If everyone you compete against uses the same crappy methods, you’ll be OK. Your projects will be perpetually late and disappointing, but so will everyone else’s, so you’ll be performing “up to standard.”

But what if you’re not? What if you’re competing against a group that gets way more done in much less time? I’m not talking 10 or 20% here; I’m talking many whole-number factors, like 10, 50 or more. What’s going to happen? It’s simple: you’re going to lose! If that’s OK with you, stop reading right now, close your eyes, and get lost in your muzak. You’ll be happier.

If your goal is to learn the standard, accepted techniques of software as widely practiced, don't waste your time with this book. But if you're pioneering or really under the gun and need to find a way to program the way software ninjas program, you'll find some useful information in this book.

Wartime Software: Optimizing for Speed

Software Development is a mission-critical issue for increasing numbers of organizations, particularly the growing number of "software-enabled service" organizations. Which makes it all the more surprising that there is a lack of concensus about to best do it.

I've written about software development quite a bit on this blog. Now, I'm in the final stages of preparing my small book on Wartime Software Development for publication as an inexpensive Kindle book. This post about bridges in war and peace gives some of the flavor. 

Wartime Software is all about optimizing the process for speed instead of predictabillity. Here's a short excerpt from the book about what optimizing for speed really means.

The usual procedures for producing code are supremely arrogant. They are arrogant because we decide that we can figure out what the customer wants, and the customer should simply wait while we “get it right.” We’re so sure that we know what the customer wants that we build it, and not just any old way, but we build it industrial strength, loaded up with piles of documentation, test plans for every little jot and tittle, so that when we (finally) roll it out, it’s on silver platters and with bands playing, with code ready to stand the test of time…and sadly, all too often, we’re wrong! We’ve misunderstood the customer, built things they don’t want, failed to build things they do want, built some things they need in confusing, incomplete or simply perverse ways. We frequently spend a year solving last year’s problem, and when we deliver our well-intentioned mess next year, the customer and the market have moved on and sometimes our competitors have leapfrogged us. Most software projects resemble your worst nightmare of a pork-barrel politics public works project, like the “bridge to nowhere,” the project in Alaska that projected nearly $400 million to build a bridge as long as the Golden Gate bridge and higher than the Brooklyn Bridge to Gravina Island, an island with only 50 residents, no stores, no restaurants and no paved roads. Who cares how well the bridge was designed?

The design of the bridge (or the software) is not the most important thing – the most important thing is the unmet needs of the people who will use the thing you intend to build. And so the number one priority is to discover what those needs are, from the only authoritative source. And by the way, the customer’s opinions may be more relevant than your opinions, but they are not truly authoritative – only the customer’s actions are authoritative.

And that means that you have to find a way to write code really quickly, so that you can turn your ideas (that hopefully you’ve mostly stolen from customers or other successful services) into services, modify them quickly based on customer feedback, and either discard them and move on, or evolve them until you’ve improved your service, using the real actions of real customers at every step of the path to make your critical decisions. You have to optimize all your processes for speed in order to pull this off.

And remember – if you’re not doing things this way, you’re probably building a software “bridge to nowhere.”

Software Development: the Relationship between Speed and Release Frequency

There is a deep, fundamental relationship between the velocity of software development and the frequency of releases. I hope this relationship will be studied in detail and everything about it understood, but the basics are clear: with minor qualifications, the more frequently you release your software, the more rapidly it will advance by every relevant measure. It will advance not only in feature/function, but in quality!

Mainstream thinking on Releases and Development Speed

The relationship I propose, "more releases = more features & better quality," is counter to the vast majority of mainstream thinking in software. In fact, in those terms, it's counter-intuitive. Here's why.

Think about software development in the simplest possible terms. You've got define it, plan it, do it, check it and release it. Five basic steps, which apply across a wide variety of process methodologies. Each step takes some time, right? After you do the work, you've got to check it and then release it. And you can't just check what you did -- you also have to make sure you didn't break anything that used to work, the "keep it right" part of quality, which grows ever larger as your software evolves.

This "check and release" process is a kind of necessary evil, the way most people think of it, and as quality failures hit you, it tends to get bigger and longer. A clever project manager (an oxymoron if there ever was one, except when intended ironically, as it is here) will naturally think, gee, let's go from 6 releases a year to 4. By cutting the overhead of the two extra releases, we'll be able to buy some development time back.

Yup, that really is how people think! Fewer releases = more time to do other stuff = we get more done.

Not!

A Real-life Example

A good example of a company that illustrates the proper relationship between release frequency and development speed is RebelMouse. RebelMouse is a next-generation, socially-fueled publishing platform. It can be used to turn boring-appearing blogs like BlackLiszt from this:

to this, a snapshot from my RebelMouse page:

v

Increasingly, they are used by big-media places, for example for Glee:

and the recently released real-time publishing curation features were used for The Following to create a social firestorm:

RebelMouse -- the Facts

The CTO/founder of RebelMouse is Paul Berry. Here he is below explaining something to his fellow nerds at the nerdfest I held a while ago.

RebelMouse has grown like crazy in its short life. Currently there are about 280,000 websites powered by RebelMouse, and that number is growing over 100% month-to-month. Their sites have over 2 million unique visitors a month.

Does RebelMouse have just a handful of releases a year? Duhhh. Try over 10 a day. A day! And there are more than 30 developers, who are not all in the same location.

Digging in

There is a lot to be said on this subject. For now, I'm just going to keep it to a single simple but important observation.

The relationship between development speed and frequency of releases does not hold up at a fine-grained level; so, for example, given two organizations, one of which has a release every 10 weeks and the other every 11 weeks, any difference in speed will be random. Similarly, if the two organizations release 5 times a day and 10 times a day, any difference in speed will also be random. But at a coarse-grained level, I observe large differences. HUGE differences.

Conclusion

RebelMouse is far from the only example, but they show the relationship between development speed and release fequency very nicely. They move much more quickly than most development organizations of their size -- in fact, they manage to push hundreds of releases in the time most organizations would have been able to limp through an "agile" (heh) development cycle or two.

 

Software: Comparing Waterfall and Agile

Lots of people talk about the evils of waterfall-style development. They aspire to move to something they think is better. Agile is high on most short lists for the something better. How different are waterfall and agile? Answer: not much.

Waterfall

The Waterfall model is an ordered, systematic method for determining what a computer system needs to do (the requirements) and then getting it done and into production. Like this:

The method is well-named. It really does look like a waterfall, like that big one famous for honeymoon visits on the St Lawrence River:

Above is a picture of Niagara Falls I look a little while ago, and is good for understanding software waterfalls. See the big river of water flowing from the upper right? See how everything is clear as it starts to fall? Then you see there's all the mist, making it very hard to see anything clearly at the end. Kind of like most software projects... This one gives you a good sense of the transition from clarity to mist:

Of course everyone hopes for the good outcome, for the rainbow emerging out of the mist:

But, I'm sad to say, the experience of Ms. Annie Edson Taylor comes closer to the common experience of waterfall software development:

While there is a vast array of software development philosophies, waterfall appears to be the standard against which most of them are compared; her concluding remarks saying it all: "nobody ought ever do that again."

Agile

Naturally, people look for better ways, and find lots and lots of ways that are thought to be better. It is incredible the number of software development philosophies there are. They go on and on! At least in my experience, Agile is the one I most often hear as a replacement for waterfall.

Like with all these things, people have a lot to say about Agile. There are books and books and conferences and training and certification, endlessly. Here is a summary diagram, given at roughly the same level of detail as the waterfall diagram above:

Lots of strong claims are made for Agile. It's faster, leads to better results, etc. Stuff that everyone says they want. But what are the real differences?

Comparing waterfall and agile

Take a close look at the two diagrams. Both of them start from requirements and go through design, development, test, integration and delivery. Here's the difference: with waterfall, you determine all the requirements up front and then drive through to delivery. The requirements are fixed, and you determine the time from there. In Agile, you determine a bunch of starting requirements, deliver them in a fixed time period (for example 2 to 6 weeks), and then get another set of requirements, and keep cycling until the project is done.

Waterfall: first fix the requirements, figure out the time.

Agile: fix the time periods, and then repeat until you're done.

Putting all the rhetoric aside, the difference between the two methods is simple: one determines the time from fixed requirements, and the other takes fixed time periods and fits requirements into them as appropriate. In other words, Agile is little more than a series of time-fixed waterfalls!

Remember, it's all just Process!

It's easy to get caught up in all this and forget that the most important thing isn't what makes Waterfall and Agile different -- it's how they're the same. Not exactly the same, but the same kind of thing: process!

You can build 100,000 lines of really crappy code using Agile. You can build 10,000 lines of great code that accomplishes the same thing using Waterfall. Or the other way round.

In Simple Terms

In simple terms, Waterfall is:

Do once: {Define. Design. Do. Check. Deliver.}

and Agile is:

Do until done: {Define. Design. Do. Check. Deliver.}

Conclusion

There are many good things about Agile. It's more iterative and can allow for more feedback loops than pure Waterfall. But its difference from Waterfall is easily exaggerated, which helps explain why the results in practice are so often disappointing. In the end, switching the precedence of the two key variables (requirements and time) can't make that much difference when the fundamentals of software and its Postulates are not addressed.