The Black Liszt

Summary: Bad Software Metaphors lead to bad results

Most of the people who talk about software can’t create it. They can’t even read it with understanding. So how is anyone who not only can’t write software but can’t even see it supposed to understand it, much less manage it?

https://www.blackliszt.com/2016/12/managing-what-you-cant-see.html

https://www.blackliszt.com/2017/04/managing-software-thats-invisible-to-you.html

A standard way about talking about software has emerged among the people to whom it’s invisible: physical-world metaphors. The metaphors are so ubiquitous that they are also used by programmers.

Because the metaphors are nearly always seriously inappropriate and people act as though they are great, horrible things result.

https://www.blackliszt.com/2016/11/software-myths-and-myth-conceptions.html

https://www.blackliszt.com/2021/09/the-catastrophic-effect-of-bad-metaphors-in-software.html

Wouldn’t it be nice if we could churn out software as reliably and predictably as a factory?

https://www.blackliszt.com/2012/04/how-effective-are-software-factories.html

The building metaphor is widely used. Buildings have architects, and so does software.

https://www.blackliszt.com/2018/12/software-planning-is-impossible.html

https://www.blackliszt.com/2017/06/software-myth-conceptions-building-materials.html

You architect and build and house. It’s a passive physical object. Software is NOT a physical object; it’s mostly is a set of actions to be taken by a computer.

https://www.blackliszt.com/2021/07/fundamental-concept-software-is-not-a-passive-thing-it-is-actions.html

The notion that the core organizing principle of software is that it's a collection of “objects” has had catastrophic effects.

https://www.blackliszt.com/2021/11/the-promising-origins-of-object-oriented-programming.html

Once you’ve built a house, you hope that modifying it is rare. Software, by contrast, undergoes nearly continuous modification.

https://www.blackliszt.com/2013/10/when-is-software-development-done.html

https://www.blackliszt.com/2021/07/fundamental-concept-software-is-rarely-built-mostly-it-is-changed.html

We maintain our cars, homes and devices. Heating and cooling systems need regular maintenance. So do our bodies! If we don’t care for our bodies properly, they break down! Software, by sharp contrast, never needs to be maintained. NEVER!

https://www.blackliszt.com/2022/01/software-never-needs-to-be-maintained.html

Waterfall and Agile are metaphors widely used in software project management. They hurt without helping.

https://www.blackliszt.com/2013/03/software-comparing-waterfall-and-agile.html

Software is amazing. We interact with what it does when using our computers and phones and when playing video games. Understanding what goes on behind the scenes is a whole different thing. It's understandable that metaphors are how people think about it. Sadly, the inappropriate metaphors result in bad software management.

Software NEVER needs to be “Maintained”

We maintain our cars, homes and devices. Heating and cooling systems need regular maintenance. So do our bodies! If we don’t care for our bodies properly, they break down! Software, by sharp contrast, never needs to be maintained. NEVER! Using the word “maintenance” to describe applying a “maintenance update” to software is beyond misleading. More accurate would be to say “a new version of the software that was crippled by a horrible design error that our perpetually broken quality processes failed to catch.” That’s not “maintenance.” It’s an urgent “factory recall” to fix a design error that infects every car (copy of the software) that was built using the flawed design.

Software is different than almost everything

Software is unlike nearly everything in our experience. It is literally invisible. Even “experts” have trouble understanding a given body of code, much less the vast continent of code it interacts with. Naturally, we apply real-world metaphors to give us a chance of understanding it. While sometimes helpful, the metaphors often prove to be seriously misleading, giving nearly everyone a deeply inaccurate view of the underlying invisible reality. The notion of “software maintenance” is a classic example. The flaw is similar to the words “software factory.”

Maintaining anything physical centers around either preventing or repairing things that break due to simple wear-and-tear or an adverse event. We change the oil in a car because it degrades with use. We change the filters in heating and cooling units because they get clogged up with the gunk from the air that passes through them. We sharpen knives that have dulled as a result of use. We maintain our homes and yards. It’s the physical world and things happen.

In the invisible, non-physical world of software, by contrast, a body of software is the same after years of use as it was the moment it was created. Nothing gets worn down. Nothing gets clogged. An inspection after years of heavy use would show that every single bit, every one and zero, was the same as it was when it was created. Of course there are memory crashes, hacker changes, etc. It’s not that software is impervious to being changed; it’s just that software is unchanged as a result of being used – unlike everything in the normal physical world, which one way or another, is changed by its environment – everything from clothes getting wrinkled or dirty from wear to seats being worn down by being sat upon.

The meaning of software maintenance

When a car is proven to have a design flaw, auto manufacturers are reluctant to ship everyone a new car in which the original design flaw has been corrected. Instead, they issue a recall notice to each affected owner, urging them to bring their car to the nearest dealership to have repair done to the car that corrects the design flaw. It’s inconvenient for the owner, but far less expensive for the manufacturer. With software, by contrast, all the software vendor has to do is make a corrected version of the software available for download and installation, the software equivalent of shipping everyone a new car! It’s no more expensive to “ship” hundreds of megabytes of “brand-new” code than it is a tiny bit. Such are the wonders of software.

Software factory recalls are part of everyday life. Software creators are maddeningly unable to create error-free software that is also cyber-secure. See this.

We’ve all become accustomed to the Three Stooges model of building software.

There are highly paid hordes of cossetted employees enjoying free lunches and lounging on bean bags on luxurious campuses, “hard at work” creating leading edge software whose only consistent feature is that it’s late, expensive and chock full of bugs and security flaws.

While the Three Stooges and their loyal armies of followers are busily at work creating standards, regulations and academic departments devoted to churning out well-indoctrinated new members of the Stooge brigades, rebels are quietly at work creating software that is needed to meet the needs of under-served customers, using tools and methods that … gasp! … actually work. What an idea!

The good news is that the rebels are often richly rewarded for their apostasy by customers who eagerly use the results of their work. It’s a good thing for the customers that the totalitarian masters of the Three Stooges software status quo are no better at enforcing their standards than they are at building software that, you know, works.

The catastrophic effect of bad metaphors in software

Software is invisible. The vast majority of people who talk about it have little or no practical experience writing it. But software is incredibly important and people need to be able to talk about it: creating it, modifying it, installing and using it and making it work as intended. How do you talk about something that's invisible? Answer: you use real-world metaphors that everyone knows and talk in those terms. Given the importance of the process of creating and modifying software, you also use real-world metaphors for the processes, methods and techniques that people use for those things on software.

Given the high cost, long time periods and risky outcomes of software development, real-world metaphors are used as models for how to create what seem like the equivalent things in the invisible world. Software professionals use the same metaphors and think in the same terms as their non-programming counterparts, the more so as advancement in software normally correlates with doing less hands-on programming.

Before long, the metaphors turn into highly detailed, structured things embodied in software tools for software management, and regulations for how software must be built in order to be acceptable for acquisition by governments and major enterprises. The software in medical devices, for example, is highly regulated in this way, everything from requirements to testing and modifications.

These metaphors are great for communication among people involved in software in some way, but disastrous for building fast, efficient software, quickly and effectively. The metaphors are largely to blame for the rolling crisis/disaster of software that was first recognized over 50 years ago.

Small groups of smart, hard-working software nerds sometimes break out of metaphor prison and use techniques that naturally grow out of the experience of programmers who live in the land of software, which, though invisible to the vast majority of people, is vividly visible to them. They are usually simple ideas that come out of the writing real-world software that meets needs and evolves with experience. Top programmers often think of themselves as "lazy," meaning that they are aware of how they spend their time and find ways to avoid doing things that don't advance the cause. They visualize the world of software they're working on, always looking for ways to spend less time writing less code to get more stuff done without errors.

The methods that they use rarely conform to any of the established software methodologies and typically make non-software people uncomfortable. I've observed and described many of those techniques, which as a whole I term "wartime software."

In this post I will identify and summarize some of the widely accepted metaphors that continue to wreak havoc on software development

The factory metaphor

The metaphor of the software factory tends to float around. Wouldn’t it be nice if we could churn out software as reliably and predictably as a factory? We’d put the requirements and other goals and constraints into the factory, and after a series of careful, orderly processing steps, out would come a wonderful piece of software! Unfortunately, factories make copies of things that are already designed and tested. All the effort goes in to making the design of a new car or drug; sometimes the design is no good and has to be discarded. Once you’ve made test versions and made sure they operate the way you planned, you turn the design over to the factory for making lots of identical copies of it. See this for more.

The building metaphor

Buildings and roads are widely used as metaphors for software. Lots of the words are the same: we have requirements, architects, standards that must be met, common design patterns. We have project plans with resource requirements and target dates. We might put the project out to bid. We use specialists for each aspect of the work. We build in phases and have regular reviews by the owners and quality inspectors. At the end we have walk-throughs, which may result in punch lists of defects to be corrected. Then we have the closing and we’re off and running!

It’s a powerful metaphor. It’s better than the factory metaphor because each house is uniquely built by a team, just like each piece of software is uniquely built by a team. The trouble is that a house just sits there. It’s a passive physical object. Software is NOT a physical object; it’s mostly is a set of actions to be taken by a computer, along with some built-in data. See this for more detail.

There's another issue with the metaphor. Buildings are built and then mostly used. Modifications are sometimes made, but they are normally small compared to the effort put into the original building. Software, on the other hand, is mostly changed. Vastly more effort is typically put into changing a body of software over time than was spent with the original building. Most of what most employees of software departments do is make changes. See this for more detail.

The Architect metaphor

The architect metaphor is closely related to the building metaphor. It's just as bad for the same reasons. Here is a detailed description that uses architecture software to illustrate the difference.

Waterfall and Agile

The waterfall method is the universally derided classic method of applying project management to building software. After years of failure with increasingly rigorous and detailed project planning to improve software development outcomes, it was supposed that waterfall just wasn't flexible enough. It was too rigid. More agility to respond to emerging knowledge and conditions would solve the problems. Thus, Agile software development method was invented, purposely built on the metaphor of a nimble-footed engineering team going through the process with lots of re-calibration and obstacle avoidance.

Hah. Nothing has changed except the rhetoric. Waterfall is a method that fixes the requirements and figures out the time to build them. Agile is a method that fixes the time periods and figures out what can be built in each. The difference in practice is that Agile has more overhead, and no better results. See this for more.

Objects in software

Could there be a more inappropriate metaphor? A bundle of data definitions and code is an "object?" A thing? Sorry, programs are NOT things. They're not passive; programs are actions!

Object-orientation started as an obscure design pattern for creating simulation programs, grew into a fad and has evolved into a standard part of software orthodoxy. I remember long ago hearing the sales pitch for the early O-O languages like Smalltalk. The business sales pitch was that it would greatly enhance the speed and quality of software development, while enabling changes to be more easily accomplished. The tech pitch was that by creating "objects" consisting of a definition of a related set of data and by including in that definition all the code that would read and write the data, concerns would be isolated and mistakes avoided by having data manipulated only by "its own" code.

The common metaphor was that objects were like Lego blocks, lending themselves to easily build elaborate structures and making changes to them without collateral damage. By throwing in things like inheritance -- which was always illustrated by physical things like the hierarchy of species, families, etc. -- the Lego effect was even stronger.

As it turned out, O-O has been a giant step backwards for software. While appropriate in some situations, as a rigid structure with no exceptions it imposes difficult and unnatural constraints on how code and data can be related and arranged, making everything more complicated than it needs to be. Lego blocks are great as physical objects, but their software version is a failure.

The walls and defenses metaphor

This one is huge in cybersecurity. Everyone talks about putting up walls to protect our computers, establishing and maintaining an effective periphery. We talk about attackers who want to "penetrate" our security defenses.

A great deal of software security is built according to this metaphor. The fact that software is invisible to everyone who makes important decisions about computer security leads directly to the string of monstrous security disasters that continue to this day. Here is a good way using baseball to understand the impact of the invisibility. Here is a detailed description of the impact of invisibility.

Conclusion

We all think using metaphors. It's not inherently a bad thing. But when the metaphor has a strong real-world grounding that everyone understands, it can lead to disaster when applied to software in ways that don't match the correct understanding of the software itself. This is unfortunately the case for a great deal of software methods and processes, and helps explain the ongoing software disaster that was first recognized over 50 years ago.

 

Fundamental Concept: Software is not a passive thing, it is actions

Even lay people have heard that there are computer languages. They've heard that the languages process data -- they take data in and put it out. They gather there are commands in the language to do stuff to data. So of course software is actions.

In spite of this knowledge, something very different is in the forefront of peoples' thinking about software, both in the industry and not. This different way of thinking results from the fact that software is invisible to nearly everyone, and that in order to think and communicate about it, language is used that assumes software is a thing.

We talk about a body of software, the 2021 version of a piece of software, as though it were a car. Speaking of cars, we talk about how software crashes. We talk about how software has bugs, a phrase that originated from a bug found in a long-ago computer that had broken.

We talk about creating requirements for a new piece of software that someone will architect and the building may be put out to bid. The rest of the software building process sounds remarkably like building a house. We think of a time when the building project is done and we can start using the house.

The trouble is that a house just sits there. It’s a passive physical object. Software is NOT a physical object; it’s mostly is a set of actions to be taken by a computer, along with some built-in data. A piece of software is like a data factory – it takes data in, makes all sorts of changes, sometimes puts some of it in a storage room, and spits out data in all sorts of volumes and formats. A software program responds to each piece of data it gets, e.g. mouse click or data file; software is an actor, while a house is a thing that does nothing. Designing a piece of software is more like writing a play for an improv theater – instead of having a fixed script, the play can go in all sorts of directions depending on the input it gets from the audience. The improv theater script even has to tell the actor that when an audience member uses a swear word, the actor has to respond saying that those words aren’t permitted, would he like to try again. This is NOTHING like designing a house! Here is more detail.

Suppose you use an advanced architecture program and design a house in great detail. Modern programs go down to the level of the framing, the details of the steps, doors and windows, literally every object that will be part of the eventual house. When you're done with such a design, you can give it to the building department for approval and to a construction company to build. If they follow the instructions correctly, you'll get exactly what you saw in the 3-D renderings and the live-action walk-throughs. Here's the killer: the equivalent of such a detailed design for computer software is ... a computer program! Or at least enough detail in pseudo-code, data definitions and screen designs so that coding is fairly mechanical. A detailed design for a computer program is like the results of extensive interactions by the script-writing team for a new episode of a series TV show, with recordings and lots of notes. Chances are a junior script-writer who has been intensely following everything with some participation is assigned the job of producing a clean draft of the script, ready for refinement and editing. That is the program, ready for actors to give it a studio reading, like you would run a test version of the software for the developers.

Software isn't a thing, regardless of how often we talk about it as though it were. It isn't a thing, even though we nearly always use methods like project management developed for building things for organizing its creation. Software is a collection of actions, like an incredibly detailed script for improvisational theater with loads of conditional branching and the ability to bring things up that were talked about before in creative and useful ways.

This matters because how you think influences what you do. If you're thinking about running a marathon while you're climbing a mountain, chances are you'll slip and fall. If you're thinking about baseball plays while you're trying to choreograph a ballet, I suspect you won't win any awards. But at least those are thinking about actions. If you're thinking about the design of a book while you're writing a comedy, I doubt you'll get many laughs.

This fundamental concept is a core aspect of all of programming. The fact that people so often have the wrong things in their heads while they are dealing with software has profound effects.

And ... just as bad, we mostly don't "build" software -- mostly we change it!

Fundamental Concept: Software is rarely Built, mostly it is Changed

Nearly everything we hear about the construction of software is that it is built. This is a fundamental misconception that pervades academia and practice. This misconception is a major underlying cause of why software has so many problems getting created in a timely way and functioning with acceptable speed and quality.

The software-is-built misconception is woven into every aspect of software education and practice. You are first taught how to write software programs; all the exercises start with a blank screen or sheet of paper. You’re taught how to test and run them. In software engineering you’re taught how to perform project management, from requirements to deployment. You may learn how to automate software quality testing. Everything about these processes assumes that software is built, like a house or a car.

The idea that software is built isn’t wrong – it just applies to a tiny fraction of the total effort that goes in to a body of software from birth to its final resting place in the great bit-bucket in the sky. The vast majority of the effort goes into changing software – making it do something different than the version you’re working on does. Maybe you’re fixing a bug, maybe you’re adding a feature or adapting to a new environment. Sometimes the changes are extensive, like when you’re changing to a new UI framework. For a typical  body of software, however much effort is put into the first production version, a minimum of 10X and often over 100X of the work goes into changing it.

We don't think of software as ever-changing, ever-growing partly because we think in terms of physical metaphors. We use the same project management techniques that are used for houses that we use for software. While sometimes houses are changed, mostly they're just built and then used for years. It's an extremely rare house that is endlessly extended the way software is. See this for more.

When it comes to enterprise software, the fraction of effort that goes into change sky rockets even more. A good example is the kind of software that runs hospitals and health systems. The software already exists, and it’s already had 100X the effort put into change than the original building. For example, standard software from the leading industry supplier was installed in the NYC hospital system. The original contract -- just to buy and install existing production software -- was for over $300M and 4 years! A couple years later the carefully planned project cost more than doubled. All because of changes. Here are details.

Why does thinking software is built rather than changed matter?

The idea that software is about building and not changing is responsible for most of the processes, procedures and methods that are overwhelmingly common practice either by custom or by regulation. The mismatch between the practices and the fact that software is nearly always changed instead of being created in the first place results in the never-ending nightmare of software disasters that plague us. The never-ending flow of new tools, languages and methods are futile attempts to solve these problems. No one would find the new things attractive if there weren’t a problem to fixed – a problem everyone knows and that no one wants to explicitly discuss!

Example: Project Management, Architecture, Design and Coding

Mostly what happens when you get a job in software is that you join a group that works with a substantial existing body of code. You may be a project manager, a designer, a customizer, any number of things, all of which revolve around the body of code.

Yes, you have to start any project from an understanding of what you want to accomplish. Here's the thing: what you want to get done is in the context of the existing code. What this means is that you have to understand as much of the code as possible to do things. If what you want to do is fairly circumscribed in the existing code, you may get away with ignoring the rest -- which is where comprehensive, nothing-left-out regression testing comes in, which means better make darn sure you didn't break anything.

In this context, lots of skills are relevant. But the overwhelming number one thing is ... knowledge of the ins and outs of the existing code base!

I've worked in a couple places where there were large code bases of this kind. Forget about departmental hierarchy -- the go-to person in every case of doing anything important was the person who best knew that particular aspect or section of the code.

In this context, what's the point of all the fancy project management stuff people think is the cornerstone of effective, predictable work? When you're traveling through an unknown land with lots of mountains, jungles, rivers and swamps, what do you want above all else (besides nice paved roads that don't exist)? You want an experienced GUIDE, someone who's been all over the place and knows the best way to get there from here.

Kind of turns all the project management stuff on its head, doesn't it?

Example: Software QA

Most software QA revolves around writing test scripts of one kind or another that mirror the new code you write: if the code does X, the test should assure that the code does X correctly. This is true whether you're doing test-driven development or whether QA people write scripts. As you build the code, you're supposed to build the test code. When you make a new release, you run integration tests that include running all the individual test scripts.

Everyone with experience knows the problems: there are never enough tests, they don't cover all the conditions, there are omissions and errors in the tests, with the result that new releases put into production often have problems.

The root cause of this is the never-spoken assumption that software is about building, when the reality is that it's mostly changed. What this means is that the focus should be on the existing code and what it does -- do the changes that you made break anything that used to work? When you make changes, you should be thinking champion/challenger like in data science: the new code is the challenger: did it make things better, and did it (probably unintentionally) make anything worse? The best way to do this is to run the same data through the production program and the challenger program in parallel, and have a tool that compares the output. Looking at the output will tell you what you want to know. Even better, running the new code in live-parallel-production, with lots of real customer data going through both versions but only the current champion output going to the customer provides a safe way to see if the customers are OK, while testing out the new functionality on the champion code only to make sure it does what you wanted.

Who needs test scripts, TDD and the rest? They're backwards! See this for more.

Conclusion

Software is literally invisible to the vast majority of people. Therefore, we all talk and think about software in terms of metaphors. That would be OK if the metaphors matched reality; unfortunately, the metaphors don’t match reality, and play a major role in software dysfunction. In real life, we build houses, bridges, and other structures -- we mostly don't change them.

So what are the methods that are appropriate when you recognize that the main thing you do with software is change it instead of building it? I've done my best to abstract the methods I've seen the best people use, calling them (for want of a better term) war-time software. Instead of creating a whole adult human at once, like Frankenstein, it's like starting with a baby that can't do much yet, but is alive. The software effort is one of growing the baby instead of constructing a Frankenstein.

 

Software Myth-Conceptions: Building Materials

An amazing amount of discussion about software is governed by metaphors. This is not surprising, since software is so abstract, and invisible to most people. So a shared physical metaphor helps everyone communicate.

The trouble comes when the metaphor doesn't match with the reality of software, as it often doesn't. In those cases, what usually seems to happen is that what everyone understands about the metaphor is assumed to be true of the software. Bad things then happen.

One of the most common metaphors for software is a physical building. Buildings normally have architectural plans that are determined before ground is broken, with the architect being in charge. Then some digging happens and a foundation is created. From then on the framing is built, the electrical and mechanicals are installed, and the finishes are applied inside and out. Now the building is ready for use. Something everyone can understand, right? It makes sense that in software we also have architects, carefully planned and laid foundations, workers who fill things out according to plan, and finally finishes applied.

Software is usually built according to the physical building metaphor. Things like Agile methodology don't change things very much -- they just mean that instead of a complete plan of everything from the beginning, things happen in phases. The physical building metaphor for software is unacknowledged, unchallenged ... and pernicious. Ignoring the metaphor is one of the main things that smart software people who follow Wartime Software practices do to beat the competition and win.

Temporary physical buildings

When we think of software in terms of the building metaphor, we nearly always think of "permanent" buildings, ones with poured foundations and all the rest. Of course, you darned well better have it all planned out right -- altering a foundation is no easy job!

What about the technology used to build temporary buildings? These aren't just tents for camping; some of them are extensive. A great example is the Frieze Art Fair, an international art fair held in NYC every year. It's huge; over 200 galleries exhibited last year. Here is a view of the extensive building from the air:

Here's a view of part of the exhibition area on the inside:

There were even a couple of restaurants:

All this was assembled in under a week, used for the time of the fair, and then torn down and carted away. No poured foundations. No big deal!

Architectural programs

Architects use software to do their work today. There are varying levels of programs they can use. I used a semi-professional one to design a house. The program was amazing. It helped me with everything. Here's part of a drawing from my project:

See the steps in the lower left? The software made it easy. I didn't draw every step. There was a staircase module which did most of the work for me. Same thing with roofs, walls, windows, doors, appliances and pretty much everything. At any time, it could generate a list of the building materials required.

In the physical world, I would need those building materials. But what if I were designing a building that existed only in a virtual reality world? I wouldn't have to go farther than the design -- the design itself is all that would be needed! Now let's turn to software. Suppose I'm designing a UI and I want a footer on every page, and a logo on the top right corner. If I'm reasonably competent with software, I don't have to count the pages and place the logo and footer as many times as there are pages -- I do it once, in a page master!

In the plans for the physical house, while it's easy for me to pop 10 identical windows into the drawing, when it gets built, someone has to build and install 10 physical windows. In the software world, the realization is done by more software. In the physical world, if I want a change to all 10 windows after they're installed ... ugh! Go back to all 10 and re-do them. In software world, I go to the footer or logo definition, change what I want, and boom! It's done!

Yes, you can design software that doesn't work this way. Yes, all too many people do it the wrong way. They're stupid. What's important is that software can be designed the smart way, and it's not hard, so often enough, it is.

The building metaphor

So what does this mean? A couple things.

Using the building metaphor encourages software to be built the wrong way. It encourages people to think about it inappropriately. The smarter you are in building software, the farther away from the physical building metaphor you get. The metaphor even gets you to think about roles badly -- the "architect" is someone who does lots of planning but no real work. Lots of workers are needed to build, workers who follow exact instructions. Then checkers who walk around with the plans making sure the workers did their job. All of this is counter-productive! It leads to bad process and bad software.

Let's knock the building metaphor down, bury it and move on.

 

Software Myths and Myth-Conceptions

Elephants are hard to figure out for people who are blind, but at least blind people can touch the elephant. Software is invisible, can't be touched and isn't anything like things we're familiar with. It's much harder to figure out, which is why what passes for understanding of software is a collection of unrelated-to-reality myths.

Elephants and Invisible Software

What would we think of elephants if the vast majority of people had never seen one in any way, even a picture? What if the few people who had encountered an elephant were blind and their experience limited to touching one part, for example a leg or the trunk?
And what if elephants were nonetheless a very important part of daily life, both personal and commercial? This is hardly thinkable, but that's what software is.

Myth-conceptions

Software is important and ubiquitous, but also invisible. The information we have about it is mixed, inconsistent and mutually incompatible. The vast majority of people, including programmers, resolve this by accepting a set of myths about what this unseen world is like. The myths tend to make sense and explain what software is about in a satisfying way. Most programmers, with their real but typically skewed experience with software, accept the same myths, but add details. The few programmers who can't or won't accept the common software myths find it difficult to communicate with the majority who hold firm to the myths.

Software "myth-conceptions" of this kind aren't just stories, or ways of talking about software. They are widely shared concepts and metaphors that guide how software is designed, built and maintained. Since many of them have been imposed on software from the outside, they tend not to fit with the reality of software -- like things designed for chipmunks, but imposed on elephants. The result is that the process of building software is really expensive and hard, with crappy results.

I have, through hard personal experience over many years, grasped the mythical nature of many widespread thoughts about software. I hope someone will systematically lay out this world of myths that weighs down and perverts software. I have already described some of the myths in prior posts and hope to go into more depth in future posts.

The Software Factory

The job of a factory is to make high quality things. There are inputs, processes, stages, and work in process. At the end, you get what you expected, when you expected it. Let's make software work like a factory -- a software factory! Hah! Here is more detail.

Software is like a Building

First you have to have a plan. You figure out what you need and work with an architect to lay out the details.  Then you put the job out to bid. You pick the best bidder by some criteria. The software builder gives you a plan with dates, and often expects partial payments based on progress. The builder gets permits and approvals.The site is prepared, the foundation is dug, footings are poured, the basement created, and the structure is built and the outside put on. The inside work is done -- first the plumbing and heating, then walls, floors, appliances and finishes. Final inspections take place, and the building is ready for use.

Sounds practically like software, right? Yup! This is one of the prime myths; its fit with reality is marginal. Acting on this myth is a prime cause of software development disasters.

Software is just another thing to Manage

Lots of activities involving different skills happen in an organization. They all need to be managed. Management is a skill that can be taught -- that's what getting an MBA is all about, after all. Some organizations need to build software, just like they need to set up a new office in Chicago. It's just another activity that needs a skilled, experienced person to manage it.

Not! Here's a post that goes into it.

Advances in Software

There's a widespread myth that software is one of those things that advances at a fierce rate. Maybe you were trained in a certain kind of software and were good at it, but after a couple of years you become hopelessly out of date and effectively unemployable. 

For certain kinds of low-level software roles like Novell Sysadmin, this is an appropriate thought. But in general, it's dead wrong. Software advances incredibly slowly; often it moves sideways, and sometimes goes backwards. When software is created for a new hardware environment, it is often extremely primitive, with all the "advances" that have been thoroughly proven by widespread use simply ignored. Much of what is presented as software "advances" is little but a fashion.

Software and Security

Securing software is much like securing a building or country. You have to have a wall. The wall should be strong and high. It should have lots of defenses against being breached. It must be constantly monitored. The places where people can enter must have extra protection, and everyone who enters must have ID and submit to a search. There are well-understood methods for accomplishing all this, and conforming to accepted procedure is essential.

That's the myth. The myth guides the vast majority of cyber-security thinking. It is pernicious and wrong. I've written about this already, and will write more.

Modern Software Methods

It's true (according to the myth) that software has been a chancy thing to build. There have been lots of delays, overruns and out-right failures. But now, with method X, we've put all that behind us. We finally understand how to create a flexible, dynamic process that puts all that behind us. We are all (for example) Agile with certified Scrum Masters.

This myth has persisted for decades. All that changes is the value of X. This myth is a perennial favorite. I've written about Agile specifically here, and will write more.

Software Innovation

Software is an exciting field, with innovation constantly frothing about. Software engineers are highly creative people -- after all, every line of software they write has never been written before! It's no wonder that the cup of software is just bubbling over with innovation.

Nice thought. But the fact is, after all these years, software projects fail left and right. Managers grasp at all sorts of things in vain attempts to increase the odds of success. Innovation is all about making something better; the challenge in software is the struggle to rise to the level of adequacy.

Conclusion

The myth-conceptions go on and on. There are a myriad of "solutions" floating around for any given software problem. The vast majority of the solutions make things worse.

There are in fact, fast, effective ways to build software. But the methods are flagrantly at odds with most of the myths that dominate the industry. So the methods remain marginal, employed mostly by small groups of outsiders who are desperate, and really want to win.