The Black Liszt

How to Improve Software Productivity and Quality: The Common Sense Approach

I talked with a frustrated executive in a computer software company. I was about to visit their central development location for the first time, and he wanted to make sure I asked sufficiently penetrating questions so that I would find out what was “really” going on.

He explained that while he had written software, it was only for a few years in the distant past, and things had changed a great deal since his day. His current job in product marketing didn’t really require him to get into any details of the development shop, and in fact he preferred to stay out of the details for several reasons: (1) he was completely out of date with current technology and methods; (2) he didn’t want his thinking constrained by what the programmers declared was possible; (3) it was none of his business.

He had developed a keen interest in what was going on in the software group, however, because he realized that it had a dramatic effect on his ability to successfully market the product. His complaints were personal and based on his own experience, but they were fairly typical, which is why I’m recounting his tale of woe here.

The Lament

The layman’s lament was an interesting mish-mash of two basic themes:

• I’m not getting the results I need. There are certain results that I really need for my business. My competitors seem to be able to get those results, and I can’t. Basically, I want more features in each release, more frequent releases, more control and visibility on new features, fewer bugs in new releases, and the ability to make simple-sounding changes quickly. Our larger competitors seem to be able to move more quickly than we do.
• I think the way the developers’ work is old-fashioned, and if it were brought up-to-date, I would get the results I need. What they do seems to be “waterfall,” with lots of documentation that doesn’t say a lot. There must be something better, along the lines of what we used to call RAD (rapid application development). They only have manual testing, nothing automated, and they tell me it will be years before they can build automated testing! And shouldn’t they be using object-oriented methods? Wouldn’t that provide more re-use, so that things can be built and changed more quickly? They have three tiers, but when I want to change something, the code always seems to be in the wrong tier and takes forever. They’re talking about re-writing everything from scratch using the latest  technology, but I’m afraid it will take a long time and there won’t be anything that benefits me.

Basically, he was saying that he wants more things, quicker, and better quality. He also advanced some theories for why he’s not getting those things and how they might be achieved, but of course he couldn’t push his theories too hard, because he lacked experience and in-depth knowledge of the newer methods. He even claimed, in classic “the grass is greener” style, that practically everyone accomplishes these things, and he was nearly alone in being deprived of them – not true!

The usual dynamics of a technology group explaining itself to “outsiders” was also at work here – if you just listen to the technology managers, things are pretty good. The methods are modern and the operation is efficient and productive. There are all sorts of improvements that could be made with additional money for people and tools, of course, but for a group that’s been under continual pressure to build new features, support cranky customers and meet accelerated deadlines with fewer resources, they’re doing amazingly well. The non-technology executives tend to feel that this is all a front, and that results really could be better with more modern methods and tools. The technology managers, for their part, feel like they’re flying passenger planes listening to a bunch of desk-bound ignoramuses complain about their inability to deliver the passengers safely and on-time while upgrading the engine and cockpit systems at the same time. These people have no idea what building automated testing (for example) really takes, they’re thinking. The non-technology people don’t really want to talk about automated testing, of course – they’re the ones taking the direct heat from customers who get hurt by bugs in the new release, and aren’t even getting proposals from the technology management of how this noxious problem can be eliminated. Well, if you can’t tell me how to solve the problem (and you should be able to), how about this (automated testing, object-oriented, micro-services, etc.)??

It goes on and on. The business executives put a cap on it, sigh, maybe throw a tantrum or two, but basically try to live with a situation they know could be better than it is. Inexperienced executives refuse to put with this crap, and bring in new management, consultants, do outsourcing, etc. Their wrath is felt! Sadly, though, the result is typically a dramatic increase in costs, better-looking reporting, but basically the status quo in terms of results, with success being defined downwards to make everything look good. The inexperienced executive is now experienced, and reverts to plan A.

The technology manager does his version of the same dance. The experienced manager tries to keep things low-key and leaves lots of room for coping with disasters and the unexpected. Inexperienced technology managers refuse to tolerate the tyranny of low expectations; they strive for real excellence, using modern tools and methods. Sadly, though, the result is typically a dramatic increase in costs, better-sounding reports, but basically the status quo in terms of tangible results. The new methods are great, but we’re still recovering from the learning curve; that was tense and risky, I’m lucky I survived, that’s the last time I’m trying something like that again!

The Hope

The non-technology executive is sure there’s an answer here, and it isn’t just that he’s dumb. He keeps finding reason to hope that higher productivity with high quality and rapid cycles can be achieved. In my experience, the most frequent (rational) basis for that hope is a loose understanding of the database concept of normalization, and the thought that it should enable wide-spread changes to be made quickly and easily. Suppose the executive looks at a set of functionally related screens and wants some button or style change to be applied to each screen. It makes sense that there should be one place to go to make that change, because surely all those functionally related screens are based on something in common, a template or pattern of some kind. What if the zip code needs to be expanded from five digits to nine? The executive can understand that you’d have to go to more than one place to make the change, because the zip code is displayed on screens, used in application code and stored in the database, but there should be less than a handful of places to change, not scores or hundreds!

But somehow, each project gets bogged down in a morass of detail. When frustration causes the executive to dive into “why can’t you…” the eyes normally glaze over in the face of massive amounts of endless gobbledy-gook. What bugs some of the more inquisitive executives is how what should be one task ends up being lots and lots of tasks? With computers to do all the grunt work, there’s bound to be a way to turn what sounds, feels and seems like one thing (adding a search function to all the screens) actually be one thing – surely there must be! And if everything you can think of is in just one place, surely you should be able to go to that one place and change it! Don’t they do something like that with databases?

There is a realistic basis for hope

I’ve spent more of my life on the programmers’ side of the table than the executives’, so I can go on, with passion and enthusiasm, about the ways that technology-ignorant executives reduce the productivity, effectiveness and quality of tech groups, not to mention the morale! The more technical detail they think they know, the worse it seems to be.

That having been said, the executive’s lament is completely justified, and his hope for better days is actually reasonable (albeit not often realized).

What his hope needs to be realized is there to be exactly one place in the code where every completely distinct entity is defined, and all information about it is stated. For example, there should be exactly one place where we define what we mean by “city.” This is like having domains and normalization in database design, only extended further.

The definition of “city” needs to have everything we know about cities in that one place. It needs to include information that we need to store it (for example, its data type and length), to process it (for example, the code that verifies that a new instance of city is valid) and to display it (for example, its label). The information needs to incorporate both data (e.g. display label) and code (e.g. the input edit check) if needed to get the job done. This is like an extended database schema; a variety of high-level software design environments have something similar to this.

It must be possible to create composite entities in this way as well, for example address. A single composite entity would typically include references to other entities (for example, city), relationships among those other entities and unique properties of the composite entity (for example, that it’s called an “address.” This composite-making ability should be able to be extended to any number of levels. If there are composites that are similar, the similarity should be captured, so that only what makes the entity unique is expressed in the entity itself. A common example of this is home address and business address.

Sometimes entities need to be related to each other in detailed ways. For example, when checking for city, you might have a list of cities, and for each the state it’s in, and maybe even the county, which may have its own state-related lists.

The same principle should apply to entities buried deep in the code. For example, a sort routine probably has no existence in terms of display or storage, but there should usually be just one sort routine. Again, if there are multiple entities that are similar, it is essential that the similarities be placed in one entity and the unique parts in another. Simple parameterization is an approach that does this.

Some of these entities will need to cross typical software structure boundaries in order to maintain our prime principle here of having everything in exactly one place. For example, data entities like city and state need to have display labels, but there needs to be one single place where the code to display an entity’s label is defined. Suppose you want a multi-lingual application? This means that the single place where labels are displayed needs to know that all labels are potentially multi-lingual, needs to know which the current language is, and needs to be able to display the current language’s label for the current entity. It also means that wherever we define a label, we need to be able to make entries for each defined language. This may sound a bit complicated at first reading, but it actually makes sense, and has the wonderful effect of making an application completely multi-lingual.

In order to keep to the principle of each entity defined once, we need the ability to make relationships between entities. The general concept of inheritance, more general than found in object-oriented languages, is what we need here. It’s like customizing a standard-model car, where you want to leave some things off, add some things and change some things.

There’s lots more detail we could go into, but for present purposes I just want to illustrate the principle of “each entity defined in one place,” and to illustrate that “entity” means anything that goes into a program at any level. By defining an entity in one place, we can group things, reference things, and abstract their commonality wherever it is found, not just in a simple hierarchy, and not limited to functions or data definitions or anything else.

While this is a layman’s description, it should be possible to see that IF programs could be constructed in this way, the layman’s hope would be fulfilled. What the layman wants is pretty simple, and actually would be if programs were written in the way he assumes. The layman assumes that there’s one way to get to the database. He assumes that if you have a search function on a screen, it’s no big deal to put a search function on every screen. He assumes that if he wants a new function that has a great deal in common with an existing function, the effort to create the new function is little more than the effort to define the differences. He assumes that look and feel is defined centrally, and is surprised when the eleventh of anything feels, looks or acts differently than the prior ten.

Because he has these assumptions in his mind, he’s surprised when a change in one place breaks something that he doesn’t think has been changed (the infamous side-effect), because he assumes you haven’t been anywhere near that other place. He really doesn’t understand regression testing, in which you test all the stuff that you didn’t think you touched, to make sure it still works. Are these programmers such careless fools that, like children in a trinket shop, they break things while walking down the aisle to somewhere else, and you have to do a complete inventory of the store when the children leave?

Programs are definitely not generally written in conformance with the layman’s assumptions; that’s why there’s such a horrible disconnect between the layman and the techies. The techies have a way of building code, generally a way that they’ve received from those who came before them, that can be made to work, albeit with considerable effort. They may try to normalize their database schemas and apply object principles to their code, but in the vast majority of cases, the layman’s assumption of a single, central definition of every “thing,” and the ability to change that thing and have the side-effects ripple silently and effectively through the application, does not exist, is not articulated, not thought about, and is in no way a goal of the software organization. It’s not even something they’ve heard talked about in some book they keep meaning to get to. It’s just not there.

I assert that it is possible to write programs in a way that realizes the layman’s hope.

I’ve done it myself and I’ve seen others do it. The results are amazing. It’s harder to do than you would ideally like because of a lack of infrastructure already available to support this style of writing, but in spite of this, it’s not hard to write. Moreover, once the initial investment in structure has been made, the ability to make changes quickly and with high quality quickly pays back the investment.

The main obstacle for everyone is that there is tremendous inertia, and the techniques that provide a basis for the hope, while reasonable and achievable, are far out of the mainstream of software thinking. I have seen people who have good resumes but are stupid or lazy look at projects that have been constructed according to the “one entity – one definition” principle and simply declare them dead on arrival, complete re-write required. But I have also encountered projects in domain areas where there is no tradition at all in building things in this way in which the people have invented the principles completely on their own.

The “principle of non-redundancy” has far-reaching technical consequences and ends up being pretty sophisticated, but at its heart is simple: things are hard to do when you have to go many places or touch many things to get them done. When the redundancy in program representation (ignoring for the moment differences between code, program data and meta-data) is eliminated, making changes or additions to programs is optimally agile. In other words, with program representation of this type, it is as easy and quick as it can possibly be to make changes to the program. In general, this will be far quicker than most programs in their current highly redundant form.

The layman’s hope that improvements can be made in software productivity, quality and cycle is realistic, and based on creating a technical reality behind the often-discussed concepts of “components” and “building blocks” that is quite different from the usual embodiments.

I have no idea why approach to building software, which is little but common sense, isn't taught in schools and widely practiced. For those who know and practice it, the approach of "Occamality" (define everything in exactly one place) gives HUGE competitive advantages.

How to Improve Software Productivity and Quality: Code and Metadata

In the long-fought war to improve software programming productivity, there have been offensives on many fronts, but precious little genuine progress made. We give our programmers the fanciest, most high-tech equipment imaginable – and it is orders of magnitude faster and more powerful than the equipment available to earlier generations – but this new equipment has made only marginal difference. While relieving programmers of the burden of punch cards helps, the latest generation of programmers are not getting the job done much better than their comparatively low-tech predecessors.

Form and Content

As usual, most efforts to improve the situation focus on one of two general approaches: form or content.

People who focus on form tend to think that the process of getting software written is what’s important. They talk about how one “methodology” is better than another. They think about how people are selected, trained, organized and motivated. As you might imagine, the spectrum of methodologies is a broad one. On one end of the spectrum is the linear approach, which starts from the generation of business requirements for software, and ends with testing, installation and ongoing maintenance. On the other end of the spectrum is the circular, interactive approach, in which a small working program is built right away, and gradually enhanced by programmers who interact closely with the eventual end-users of the program. There are any number of methodologies between these two extremes, each of which claims an ideal combination of predictable linearity with creative interactivity.

People who focus on content tend to think that the language in which programs are written and the structure and organization of programs are what’s important. Naturally, they like design and programming tools that give specific support to their preferred language and/or architecture. There tends to be a broad consensus of support at the leading edge around just a couple of languages and structures, while the majority of programmers struggle with enhancing  and maintaining programs originally written according to some earlier generation’s candidate for “best language” or “best architecture.” At the same time, many of those programmers make valiant attempts to renovate older programs so that they more closely resemble the latest design precepts, frequently creating messes. Regardless of the generation, programmers quickly get the idea that making changes to programs is their most time-consuming activity (apart, of course, from never-ending meetings), and so they focus on ways to organize programs to minimize the cost of change. This leads to a desire to build “components” that can be “assembled” into working programs, and naturally to standards for program components to “talk” with each other.

Lots of effort, not much progress

The net effect of all these well-intentioned efforts, on both the form and content sides, has been a little bit of progress and a great deal of churn. Having some agreed-on methodology leads to better predictability and generally better results than having none; it seems that having a beat to which everyone marches in unison, even if it’s not the best beat, leads to better results than having a great beat that many people don’t know or simply refuse to march to. What is the best methodology in any one case depends a great deal on both how much is already known about the problem to be solved and how smart and broadly skilled the participants in the project are. The more qualified the people and the less known about the target, the more appropriate it is to be on the “interactive” end of the spectrum; think highly qualified and trained special forces going after a well-defended target in enemy territory – creativity, teamwork and extraordinary skills are what you need. The more ordinary the participants and with better-understood objectives, the more appropriate being somewhere towards the “linear” end of the spectrum; think of a large army pressing an offensive against a broad front – with so many people, they can’t all be extraordinary, and you want coordinated, linear planning, because too much local initiative will lead to chaos.

As to content, while it is clear that the latest programming languages encourage common-sense concepts like modular organization and exposing clear interfaces, good programmers did that and more decades ago in whatever language they were writing, including assembler, and bad programmers can always find a way to make messes. And even if you use the best tools and interface methods, incredible churn is created by frequent shifts in what is considered to be the best architecture, practically obsoleting prior generations. Probably the single biggest movement over the last several decades has been the gradual effort to take important things about programs that originally could be discovered only by inspecting the source code of the program and making those things available for discovery and use by people and programs, without access to the source code. The first major wave of this movement led to exposure of much of a program’s persistent data, in the form of DBMS schemas; the other major wave of this movement (in many embodiments, from SAA to SOAP to microservices) exposes a program’s interfaces, in the form of callable functions and their parameters. This has been done in the belief that it will enable us to make important changes to some programs without access to or impact on others, and thus approach the dream of programming by assembling fixed components, like building blocks or legos, into completed programs or buildings. The belief in this dream has been affected very little by decades of evidence that legos don’t build things that adults want to live in.

I believe that while considerations of form are incredibly important, and when done inappropriately can drag down or even sink any programming project, there is little theoretical headway to be made by improvements in methodology. I think most of the relevant ideas for good methodology have already been explored from multiple angles, with the exception of how to match methodology to people and project requirements – no one methodology is the best for each project and each collection of people.  But even when you’ve picked the best methodology, the NBA players will always beat the high school team, as long as the NBA players execute well on the motivational and teamwork aspect that any good methodology incorporates. With methodology we’re more in need of good execution and somehow assembling a talented and experienced team than we are of fresh new ideas.

Ptolemy, Copernicus, Kepler, Newton

Content, however, is another story altogether. I think our current best languages and architectures will look positively medieval from the perspective of a better approach to content. I think there is a possible revolution here, one that can bring about dramatic improvements in productivity, but which requires an entirely new mind-set, as different as Copernicus and Ptolemy, as different as Einstein and Newton.

Having said that, let me also say that the “new” ideas are by no means completely new. Many existing products and projects have exploited important parts of these ideas. What is mostly new here is not any particular programming technique, but an overriding vision and approach that ties various isolated programming techniques into a unified, consistent whole. Kepler’s equations described the motion of planets with accuracy equal to Newton’s – in fact, you can derive one set of equations mathematically from the other; but Newton provided the vision (gravity) and the tools (calculus) that transformed some practical techniques (Kepler’s equations) into the basis of a new view of the world. That’s why physics up to the end of the nineteenth century was quite justifiably characterized as “Newtonian” and not “Keplerian.”

Ptolemy and Newton looked at the same set of objects; Ptolemy picked the closest one, the earth, to serve as the center of thinking, while still incorporating the rest.

His main goal was to describe the motions of what you could see, focused on the matter. Copernicus noted that things got simpler and more accurate if you picked one farther away, the Sun, to serve as the center of things.

Kepler made things better by noticing the curves made by the planets. Newton then took the crucial step: instead of focusing on matter, he focused on energy (gravity in this case), and wrote an equation describing how gravity works,

which creates changes in the location and velocity of matter. In programming, we have clear equivalents of matter and energy: matter is data (whether or not it is persistent), and energy is instructions (lines of code, regardless of the language it’s in). In COBOL, for example, this division is made explicit in the data division and the procedure division. In modern languages the two are more intermixed, but it remains clear whether any particular line describes data or describes an action to take (an if statement, assignment statement, etc.).

Now, in spite of what you may have been taught in school, Ptolemy’s method works. In fact, it would be possible (though not particularly desirable) to update his approach with modern observations and methods, and have it produce results that are nearly identical to those possible today; in fact, only when you get to relativity does Ptolemy’s method break down altogether -- but remember that the effects of relativity are so small, it takes twentieth century instruments to detect them. But no one bothers to do this, because the energy-centered approach developed by Newton and refined by his successors is so much simpler, cleaner and efficient.

Similarly, there is no doubt that the instructions-centric approach to programming works. But what comes out of it is complicated, ugly and inefficient. The Newtonian breakthrough in programming is replacing writing and organizing instructions (and oh, by the way, there’s some data too) as the center of what we do with defining and describing data (and oh, by the way, there are some instructions too). The instruction-centered approach yields large numbers of instructions with a small amount of associated data definitions; the data-centered approach yields large numbers of data definitions and descriptions operated on by a significant body of unchanging, standard instructions and small numbers of instructions specifically written for a particular program. In the instruction-centered approach, we naturally worry about how to organize collections of instructions so that we can write fewer of them, and arrive at concepts like objects, components and class inheritance (a subclass inherits and can override its parent’s methods (instructions)). In the data-centered approach, we naturally worry about how to organize collections of data definitions so that we can have the minimal set of them, and arrive at concepts like meta-data, standard operations (e.g. pre-written meta-data-driven instructions enabling create, query, select, update and delete of a given collection of data) and data inheritance (a child data definition, individual or group, inherits and can override its parent’s definitional attributes). In short, we separate out everything we observe into a small, unchanging core (like Newton's gravity) that produces ever-changing results in a diverse landscape.

We shift our perspective in this way not because it enables us to accomplish something that can’t be accomplished by the current perspective, but because it proves to be cleaner, simpler, more efficient, easier to change, etc.

Instructions, data and maps

Only by understanding the details of this approach can it really be appreciated, but the metaphor of driving instructions and maps is appropriate and should make the core idea clear.

Suppose your job is to drive between two locations, and the source and destination location are always changing. There are two general approaches for giving you directions:

1. Turn-by-turn directions (the instruction-driven, action-oriented approach)
2. A map, with source and destination marked (the data-driven, matter-oriented approach)

The advantage of directions is that they make things easy for the driver (the driver is like the computer in this case). You pick up one step, drive it, then pick up the next, drive it, and so on until the end. You don’t have to think in advance. All you have to do is follow directions, which tell you explicitly what to do, in action-oriented terms (turn here, etc.).

The problem with directions come in a couple of circumstances:

• You have to have a huge number of directions to cover all possible starting points and destinations, and there is a great deal of overlap between sets of directions
• If there is a problem not anticipated by the directions, such as an accident or road construction, you have to guess and get lucky to get around the problem and get back on track.

A map, on the other hand, gives you most of the information you need to generate your own directions between any two given points. The map provides the information; you generate the actions from that information. With a direction-generating program and some parameters like whether to use toll roads, a generic program can generate directions on the fly. With a program like Wayz, real-time changes due to updated traffic information can even be generated.

The Wayz program itself isn't updated very often -- it's mostly the map and road conditions. Same thing with a program written using this approach; the core capabilities are written in instructions, while the details of the input, storage, processing and output are all described in a "map" of the data and what is to be done with it.

Conclusion

It's pretty simple: programming today largely follows the method of Ptolemy, resulting in an explosion of software epi-cycles to get anything done. Attempts to keep things easily changeable sound promising but never work out in practice.

The way forward is to focus instead on what there is and what is to be done with it (data and metadata), with a small amount of Wayz-like code to take any user or data stream from its starting point to its destination with easy changes as needed.

Better Software and Happier Customers with Post-hoc Design

What can you possibly mean by "post-hoc design?" Yes, I know it means "after-the-fact design," using normal English. It's nonsense! First you design something. Then you build it. Period.

Got your attention, have I? I agree that "post-hoc design" sounds like nonsense. I never heard of it or considered it for decades. But then I did. Before long I saw that great programmers used it to create effective high-quality, loved-by-customers software very quickly.

The usual way to build software: design then build

The way to build good software is obviously to think about it first. Who does anything important without having a plan? Start by getting requirements from the best possible source, as detailed as possible. Then consider scale and volume. Then start with architecture and drill down to design.

When experienced people do architecture and design, they know that requirements often "evolve." So it's important to generalize the design anticipating the changes and likely future requirements. Then you make plans and can start building. Test and verify as you drive towards alpha then beta testing. You know the drill. Anything but this general approach is pure amateur-hour.

I did this over and over. Things kept screwing up. The main issue was requirements "evolution," which is something I knew would happen! Some of the changes seemed like they were from left field, and meant that part of my generalized architecture not only failed to anticipate them, but actually made it harder to meet them! Things that I anticipated might happen which I wove into the design never happened. Not only had I wasted the time designing and building, the weren't-needed parts of the design often made it hard for me to build the new things that came along that I had failed to anticipate.

I assumed that the problem was that I didn't spend enough time doing the architecture and design thinking, and I hadn't been smart enough about it. Next time I would work harder and smarter and things would go more smoothly. Never happened. How about requirements? Same thing. The people defining the requirements did the best they could, but were also surprised when things came along, and embarrassed when things they were sure would be important weren't.

After a long time -- decades! -- I finally figured out that the problem was in principle unsolvable. You can't plan for the future in software. Because you can't perfectly predict the future! What you are sure will happen doesn't, and what you never thought about happens.Time spent on anything but doing and learning as you go along is wasted time.

The winning way to build software: Build then Design

Build first. Then and only then do the design for the software you've already built. Sounds totally stupid. That's part of why I throw in some Latin to make it sound exotic: "Post-hoc design," i.e., after-the-fact design.

When you design before you build, you can't possibly know what you're doing. You spend a bunch of time doing things that turn out to be wrong, and making the build harder and longer than it needs to be. When you build in small increments with customer/user input and feedback at each step, keeping the code as simple as possible, you keep everything short and direct. You might even build a whole solution for a customer this way -- purposely NOT thinking about what other customers might need, but driving with lots of hard-coding to exactly what THIS customer needs. Result: the customer watches their solution grow, each step (hopefully) doing something useful, guides it as needed, and gets exactly what they need in the shortest possible time. What's bad about a happy customer?

Of course, if you've got the typical crew of Design-first-then-build programmers, they're going to complain about the demeaning, unprofessional approach they're being forced to take. They might cram in O-O classes and inheritance as a sop to their pride; if they do, they should be caught and chastised! They will grumble about the enormous mountain of "technical debt" being created. Shut up and code! Exactly and only what's needed to make this customer happy!

When the code is shown to another customer, they might love some things, not need some other things and point out some crucial things they need aren't there. Response: the nearly-mutinous programmers grab a copy of the code and start hacking at it, neutering what isn't needed, changing here and adding there. They are NOT permitted to "enhance" the original code, but hack a copy of it to meet the new customer's need. At this point, some of the programmers might discover that they like the feeling of making a customer happy more quickly than ever before.

After doing this a couple times, exactly when is a matter of judgement, it will be time to do the "design" on the software that's already been built. Cynics might call this "paying off tech debt," except it's not. You change the code so that it exactly and only meets the requirements of the design you would have made to build these and only these bodies of code. You take the several separate bodies of code (remember, you did evil copy-and-modify) and create from them a single body of code that can do what any of the versions can do.

When you do this, it's essential that you NOT anticipate future variations -- which will lead to the usual problems of design-first. The pattern for accomplishing this is the elimination of redundancy, i.e., Occamality. When you see copy/modify versions of code, you replace them with a single body of code with the variations handled in the simplest way possible -- for example, putting the variations into a metadata table.

This isn't something that's done just once. You throw in a post-hoc design cycle whenever it makes sense, usually when you have an unwieldy number of similar copies.

As time goes on, an ever-growing fraction of a new user's needs can be met by simple parameter and table settings of the main code line, and an ever-shrinking fraction met by new code.

Post-Hoc Design

Ignoring the pretentious name, post-hoc design is the simplest and most efficient way to build software that makes customers happy while minimizing the overall programming effort. The difference is a great reduction in wasted time designing and building, and in the time to customer satisfaction. Instead of a long requirements gathering and up-front design trying valiantly to get it right for once, resulting in lots of useless code that makes it harder to build what it turns out is actually needed, you hard-code direct to working solutions, and then periodically perform code unification whose purpose is to further shorten the time to satisfaction of new customers. To the extent that a "design" is a structure for code that enables a single body of code to be easily configured to meet diverse needs, doing the design post-hoc assures zero waste and error.

What is the purpose of architecture and design anyway? It is to create a single body of code (with associated parameters and control tables) that meets the needs of many customers with zero changes to the code itself. The usual method is outside-in, gaze into the future. Post-hoc design is inside-out, study what you built to make a few customers happy, and reduce the number of source code copies to zero while reducing the lines of code to a minimum. The goal is post-hoc design is to minimize the time and effort to satisfy the next customer, and that's achieved by making the code Occamal, i.e., eliminating redundancies of all kinds. After all, what makes code hard to change? Finding all the places where something is defined. If everything is defined in exactly one place, once you've found it, change is easy.

Post-hoc design is a process that should continue through the whole life of a body of code. It prioritizes satisfaction of the customer in front of your face. It breaks the usual model of do one thing to build code and another to modify it. In the early days of what would normally be called a code "build," the code works, but only does a subset of what it is likely to end up doing. When customers see subsets of this kind, it's amazing how it impacts their view of their requirements! "I love that. I could start using it today if only this and that were added!" It's called "grow the baby," an amazing way to achieve both speed and quality.

New name for an old idea

All I'm doing with "Post-hoc design" is putting a name and some system around a practice that, while scorned by academia and banned by professional managers, has a long history of producing best-in class results. I'm far from the first person who has noticed the key elements of post-hoc design.

Linus Torvalds (key author of Linux, the world's leading operating system) is clearly down on the whole idea of up-front design:

Don’t ever make the mistake [of thinking] that you can design something better than what you get from ruthless massively parallel trial-and-error with a feedback cycle. That’s giving your intelligence much too much credit.

Gall's Law is a clear statement of the incremental approach:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.

The great computer scientist Donald Knuth, author of the multi-volume Art of Computer Programming, was a master of shifting between assembler language programming and abstract algorithms and back, the key activities of the speed-to-solution and post-hoc abstraction phases of the method I've described here.

People who discover the power and beauty of high-level, abstract ideas often make the mistake of believing that concrete ideas at lower levels are worthless and might as well be forgotten. On the contrary, the best computer scientists are thoroughly grounded in basic concepts of how computers actually work. The essence of computer science is an ability to understand many levels of abstraction simultaneously.

Thanks to Daniel Lemire for alerting me to these quotes.

Conclusion

Post-hoc design is based on the idea that software is only "built" once, and after that always changed. So why not apply the optimal process of changing software from day one? And then alternate between as-fast-as-possible driving to the next milestone, with periodic clean-ups to make fast driving to the next goal optimal? Post-hoc design is a cornerstone of the process of creating happy customers and optimal code.  It also happens to conform to the goals of software architecture. Post-hoc design is like first fighting a battle, and then, once the battle is over and you've won, cleaning and repairing everything, incorporating what you learned from the battle just past so that everything is ready for the next battle. Post-hoc design is the way to win.

 

How to Improve Software Productivity and Quality: Schema Enhancements

Most efforts to improve programmer productivity and software quality fail to generate lasting gains. New languages, new project management and the rest are decades-long disappointments – not that anyone admits failure, of course.

The general approach of software abstraction, i.e., moving program definition from imperative code to declarative metadata, has decades of success to prove its viability. It’s a peculiar fact of software history and Computer Science that the approach is not mainstream. So much the more competitive advantage for hungry teams that want to fight the entrenched software armies and win!

The first step – and it’s a big one! – on the journey to building better software more quickly is to migrate application functionality from lines of code to attributes in central schema (data) definitions.

Data Definitions and Schemas

Every software language has two kinds of statements: statements that define and name data and statements that do things that are related to getting, processing and storing data. Definitions are like a map of what exists. Action statements are like sets of directions for going between places on a map. The map/directions metaphor is key here.

In practice, programmers tend to first create the data definitions and then proceed to spend the vast majority of their time and effort creating and evolving the action statements. If you look at most programs, the vast majority of the lines are “action” lines.

The action lines are endlessly complex, needing books to describe all the kinds of statements, the grammar, the available libraries and frameworks, etc. The data definitions are extremely simple. They first and foremost name a piece of data, and then (usually) give its type, which is one of a small selection of things like integer, character, and floating point (a number that has decimal digits). There are often some grouping and array options that allow you to put data items into a block (like address with street, town and state) and sets (like an array for days in a year).

One of the peculiar elements of software language evolution is whether the data used in a program is defined in a single place or multiple places. You would think – correctly! – that the sensible choice is a single definition. That was the case for the early batch-oriented languages like COBOL, which has a shared copybook library of data definitions. A single definition was a key aspect of the 4-GL languages that fueled their high productivity.

Then the DBMS grew as a standard part of the software toolkit; each DBMS has its own set of data definitions, called a “schema.” Schemas enable each piece of data to have a name, a data type and be part of a grouping (table). That’s pretty much it! Then software began to be developed in layers, like UI, server and database, each with its own data/schema definitions and language. Next came services and distributed applications, each with its own data definitions and often written in different languages. Each of these things need to “talk” with each other, passing and getting back data, with further definitions for the interfaces.

The result of all this was an explosion of data definitions, with what amounts to the same data being defined multiple times in multiple languages and locations in a program.

In terms of maps and directions, this is very much like having many different collections of directions, each of which has exactly and only the parts of the map those directions traverse. Insane!

The BIG First Step towards Productivity and Quality

The first big step towards sanity, with the nice side effect of productivity and quality, is to centralize all of a program’s data definitions in a single place. Eliminate the redundancy!

Yes, it may take a bit of work. The central schema would be stored in a multi-part file in a standardized format, with selectors and generators for each program that shared the schema. Each sub-program (like a UI or service) would generally only use some of the program’s data, and would name the part it used in a header. A translator/generator would then grab the relevant subset of definitions and generate them in the format required for the language of the program – generally not a hard task, and one that in the future should be provided as a widely-available toolset.

Why bother? Make your change in ONE place, and with no further work it’s deployed in ALL relevant places. Quality (no errors, no missing a place to change) and productivity (less work). You just have to bend your head around the "radical" thought that data can be defined outside of a program.

If you're scratching your head and thinking that this approach doesn't fit into the object-oriented paradigm in which data definitions are an integral part of the code that works with them, i.e. a Class, you're right. Only by breaking this death-grip can we eliminate the horrible cancer of redundant data definitions that make bodies of O-O code so hard to write and change. That is the single biggest reason why O-O is bad -- but there are more!

The BIG Next Step towards Productivity and Quality

Depending on your situation, this can be your first step.

Data definitions, as you may know, are pretty sparse. There is a huge amount of information we know about data that we normally express in various languages, often in many places. When we put a field on a screen, we may:

• Set permissions to make it not visible, read-only or editable.
• If the field can be entered, it may be required or optional
• Display a label for the field
• Control the size and format of the field to handle things like selecting from a list of choices or entering a date
• Check the input to make sure it’s valid, and display an error message if it isn’t
• Fields may be grouped for display and be given a label, like an address

Here's the core move: each one of the above bullet items -- and more! -- should be defined as attributes of the data/schema definition. In other words, these things shouldn't be arguments of functions or otherwise part of procedural code. They should be just like the Type attribute of a data definition is, an attribute of the data definition.

This is just in the UI layer. Why not take what’s defined there and apply it as required at the server and database layers – surely you want the same error checking there as well, right?

Another GIANT step forward

Now we get to some fun stuff. You know all that rhetoric about “inheritance” you hear about in the object-oriented world? The stuff that sounds good but never much pans out? In schemas and data definitions, inheritance is simple and … it’s effective! It’s been implemented for a long time in the DBMS concept of domains, but it makes sense to greatly extend it and make it multi-level and multi-parent.

You’ve gone to the trouble of defining the multi-field group of address. There may be variations that have lots in common, like billing and shipping address. Why define each kind of address from scratch? Why not define the common parts once and then say what’s unique about shipping and billing?

Once you’re in the world of inheritance, you start getting some killer quality and productivity. Suppose it’s decades ago and the USPS has decided to add another 4 digits to the zip code. Bummer. If you’re in the enhanced schema world, you just go into the master definition, make the change, and voila! Every use of zip code is now updated.

Schema updating with databases

Every step you take down the road of centralized schema takes some work but delivers serious benefits. So let’s turn to database schema updates.

Everyone who works with a database knows that updating the database schema is a process. Generally you try to make updates backwards compatible. It’s nearly always the case that the database schema change has to be applied to the test version of the database first. Then you update the programs that depend on the new or changed schema elements and test with the database. When it’s OK, you do the same to the production system, updating the production database first before releasing the code that uses it.

Having a centralized schema that encompasses all programs and databases doesn’t change this, but makes it easier – fewer steps with fewer mistakes. First you make the change in the centralized schema. Then it’s a process of generating the data definitions first for the test systems (database and programs) and then to the production system. You may have made just a couple changes to the centralized schema, but because of inheritance and all the data definitions that are generated, you might end up with dozens of changes in your overall system – UI pages, back end services, API calls and definitions and the database schema. Making an omission or mistake on just one of the dozens of changes means a bug that has to be found and fixed.

Conclusion

I’ve only scratched the surface of a huge subject in this post. But in practice, it’s a hill you can climb. Each step yields benefits, and successive steps deliver increasingly large results in terms of productivity and quality. The overall picture should be clear: you are taking a wide variety of data definitions expressed in code in different languages and parts of a system and step by step, collapsing them into a small number of declarative, meta-data attributes of a centralized schema. A simple generator (compile-time or run-time) can turn the centralized information into what’s needed to make the system work.

In doing this, you have removed a great deal of redundancy from your system. You’ve made it easier to change. While rarely looked on as a key thing to strive for, the fact that the vast majority of what we do to software is change it makes non-redundancy the most important measure of goodness that software can have.

What I've described here are just the first steps up the mountain. Near the mountain's top, most of a program's functionality is defined by metadata!

FWIW, the concept I'm explaining here is an OLD one. It's been around and been implemented to varying extents in many successful production systems. It's the core of climbing the tree of abstraction. When and to the extent it's been implemented, the productivity and quality gains have in fact been achieved. Ever hear of the RAILS framework in Ruby, implementing the DRY (Don't Repeat Yourself) concept? A limited version of the same idea. Apple's credit card runs on a system built on these principles today. This approach is practical and proven. But it's orthogonal to the general thoughts about software that are generally taught in Computer Science and practiced in mainstream organizations.

This means that it's a super-power that software ninjas can use to program circles around the lumbering armies of mainstream software development organizations.

The Map for Building Optimal Software

So you want to build optimal software, do you? What’s “optimal” in software, anyway? Is it building the software that’s needed quickly and well, with no bugs? Is it building software that’s easy to enhance, adding new features – including ones you never thought of – with minimal fuss and bother? Is it building software that’s easy to scale endlessly with little trouble? How about all of the above, all at once? Yup, that would be optimal, sure enough.

I’ve described in general terms about optimal software. Here’s a map, a specific example, of how to get there.

Going to new places

Let’s think about going places we haven’t been before, or taking routes that are new to us. What are the options for figuring out how to do it?

If you’re talking with someone who is familiar with the place to which you want to go, you might ask them how to get there. If there are more than a couple of turns, you might try to simplify the directions or write them down. If you’re on your own, you might consult a map. If you’re computer-oriented you might use one of the on-line mapping programs. If you’re driving in a car, you might use a navigation system.

No matter what you do, you’ll end up (let’s say) getting in a car and driving. When you drive, one way or another, you’ll be following directions and making choices of turns. Nothing happens unless you drive and to drive you need directions.

It’s obvious that while you need directions, a fixed set of directions by itself is adequate only if nothing goes wrong – there are no accidents, no construction detours, no recent changes to the roads, etc. The minute there is any problem, you need a map or a live, map-driven navigation system.

What you do when you look at a map, or what a navigation system does, is execute a generic direction-creating algorithm. Either your brain or a computer looks at the map, sees where you are and where you want to go, and figures out the sequence of roads and turns to get you there. Is there an obstruction? The nav system doesn't care -- it just computes a fresh set of directions for you, based on where you are now.

Here's an example. It shows two different routes for me to drive from my home to the wonderfully close source of some of the world's best home-made ice cream: Denville Dairy.

A direction-creating algorithm is a relatively small amount of code that is independent of the map that it uses. You may need to enhance it when different kinds of things are introduced into the map (for example, you decide to support avoiding toll roads or taking ferries), but once you’ve added the capability, it should automatically apply to all the maps you have.

So “where” is the application functionality here? The ability to create directions is in a small amount of abstract code. This code accepts a reference to a map, a starting point, an ending point and perhaps some parameters, like whether to minimize distance or time or avoid toll roads. The actual directions themselves are drawn from the substance of the map. In this case,

• what you’ve got is the map;
• what you do to what you’ve got is create directions.

Isn’t it interesting that most of the time and effort is creating the set of maps, without which there are no directions? And isn’t it interesting that the maps are data?

Maps are:

• Extensive
• Subject to frequent updating and change

An error in a map is bad, but should not “crash the program” – it should just lead to bad results, and should be easily fixed, without going into debuggers, single-stepping and other geeky things.

The direction-creating program is:

• Short
• Infrequently updated
• Unaffected by additional maps
• Unaffected by changes to maps

Best of all, the direction-creating program is unaffected by the driver not following the directions – the program just builds new directions to the destination from wherever the car happens to be.

Errors in this program are crippling, but it’s likely that errors will be found early in the process, after which it won’t change much.

Maps and writing optimal code

So what does this have to do with writing code? Lots.

A map (meta-data) is an efficient, compact, re-purpose-able representation of what the user cares about. It is truly Occamal, without redundancy. Each piece of knowledge you have about the world is represented in the map exactly once.

A reasonably well-written direction-generating program is an efficient, compact representation of strategies for generating routes through the entire universe of possible maps. Each algorithm or strategy for generating directions should be represented in the program exactly once.

Combined, the map and the direction-generating program are a pretty good template/metaphor for an Occamal program. Here’s the pattern:

• You should make everything you possibly can into a “map” (metadata).
• Abstract actions that cannot be reduced to metadata should be coded once, and should be “driven” to the largest extent possible by the metadata.

The map to building optimal software is a simple idea: build the smallest amount of code you possibly can, and put as much information as possible into passive, declarative meta-data. That will get you to your destination safely and in the shortest amount of time without breaking laws.

Lessons for Better Software from Washing Machine Design

The power of Occam-optimality, a.k.a. occamality, can be difficult to appreciate if your head is awash in myriad software buzzwords, and explaining it in software terms can make understanding it challenging to anyone unfamiliar with those terms. So it makes sense at this point to explain the core concept in a different context.

Let’s think about the design of physical things, the kind of things that require discrete manufacturing, have bills of material, and so on, for example a washing machine.

If every section of the washing machine were designed by someone different, each designer may call for a particular screw here, or a particular composition of aluminum there, making his best judgment on what is the best to use. When you broke down the bill of material into a consolidated parts list, you may see two of this kind of screw, one of that kind, and a couple of yet another kind. It may be that the various screws need to be sourced from multiple suppliers. As far as the designer is concerned, the various screws are completely legitimate requirements. Each was chosen to have everything required to get the job done, but only to get the job done. The designer is probably proud, with good justification, of the care and effort he expended in specifying exactly the right screw for the job. When the design is complete, the washing machine will have a list of unique parts and designs and/or instructions for how to construct and/or assemble them.

While the designers may be proud, everyone else’s jobs have been made more difficult and the overall expense of building washing machines has been increased. If instead the designers had been motivated to get together and find a way to use a single screw type that could be sourced from a single supplier, more screws would be bought from a single supplier in greater quantities, reducing both unit and shipping costs, and enabling all assembly points to be supplied from a single pool of screws. Only one type of screw driver would be needed, which would also help keep all the screw drivers in working order during manufacturing. Even the repair manual would be slightly easier to write and would be slightly shorter, and repair people would need only a single type of screw and a single type of driver, which would increase the odds that they would have what they needed to do a repair on a single visit. If the design were done in this way, the overall design would be virtually unchanged, but the parts list would be shorter – there would be a smaller number of unique parts, with each used more frequently.

How about the designers? Instead of starting from scratch when they needed a screw, picking the “right” screw from the universe of available screws, which is large, they would agree up front on a screw that would meet all their anticipated needs closely enough, and then, every time they needed a screw, they would just check to make sure the pre-chosen screw was adequate to the job. This is a simpler job than selecting the “best” screw from a large selection, because you’re just applying a short list of disqualifying characteristics to the pre-selected screw. Moreover, you’re arranging your design as you go along (for example, the thickness of the plates) to match up with the qualities of the screw, making the selected screw very likely to be OK in each case. So the “shortest” (without going too crazy) parts list, the one with the smallest number of unique parts, clearly is the best design.

Notice that the original design, the one with lots of types of screw types, would be defended by its designers as the “best” design, because they understood that the important thing for them to do was to create the “best” solution for each design problem they faced, considered in isolation. Let’s assume they did. But once the designers look at the whole problem, and try to pick the best design for the entire lifecycle of the washing machine, including sourcing, manufacturing and repair, it’s obvious that the different screw selections were incidental aspects of the washing machine design. The design could just as well have been embodied in one that had fewer screw types, and as we now understand, having fewer parts is better.

Suppose there are a couple of established washing machine manufacturers. Suppose you wanted to enter the market with a superior product. If things worked the way they normally do, the earlier products are probably far from occamal. That means it's possible to design a washing machine with just as high quality as the ones on the market, but less expensive to build and easier to service. Everyone would like your product better than the existing ones. Cheaper to build, cheaper and easier to fix in a single visit; what's not to like? This means that, in addition to being a core design principle, occamality helps us understand and predict the evolution of products in a market, all other things being equal.

In a virtually identical way, a program that has the smallest (within reason) number of unique “parts” is the best, for reasons that are very similar to the way that the best physical design has the shortest list of parts. It definitely helps the designers, and it helps everyone else even more. Just like with a washing machine, a program can be built, from the requirements all the way through, to have lots of unique “parts” and custom things, each of which may make sense when considered in isolation. But when you look at the whole software lifecycle, taking a view over the entire program and everything that will eventually be done to it, including later maintenance, change and enhancement, it becomes clear that designing it to contain the smallest number of “unique parts,” so that every “program concept” is defined in exactly one place, yields the best design.

The parable of the screw can help you, your organization and your users avoid being screwed during software design. You’ll get there if you keep this principle in mind: always use the smallest number of unique “parts” that you can, and be willing to adjust your design to make the number even smaller. This may sound easy to you, or it may sound like an essentially trivial exercise in neatness. It is neither. It can be challenging in practice, and the implications of it are actually profound and far-reaching.

William of Occam is the Inventor of the Method for Building Optimal Software

Lots of people would like the credit for establishing, once and for all, the core, immutable principle of optimal software design/architecture – the method for measuring which, among an infinite number of embodiments of software requirements, is provably the best among them. I am one of the crowd that would love to take credit! Alas, I and my competitors have been beaten to the punch, by a mere 700 years or so, by a guy named William of Occam.

As it turns out, the software crowd is, as usual, late to the party: loads of luminaries in other fields have recognized William’s achievement, viz., Occam’s Razor, for the indispensable concept that it is. Various software folks have been nipping around the edges of it, for example with the DRY (don’t repeat yourself) principle that’s gotten some attention. But they haven’t grasped the fundamental, touches-nearly-everything, deep nature of the Razor. Too bad! Crappy software continues to be churned out by the ton by people who are enthralled by false idols, convinced they’re on the righteous path toward software excellence, when in fact they’re doing something else, words for which should not be used in polite company.

William of Occam (ca. 1285 to 1349) was an English logician and Franciscan friar.

He is credited with formulating a principle that is already widely applied to various aspects of computer systems. In those areas of computing to which it has been applied, it reigns supreme – it unquestionably supplies the most fundamental, relevant and optimal understanding and solutions to the relevant problems, so much so that the people involved don’t think about it or question it.

However, there are large areas of computing to which Occam’s razor has not been applied. Worse, it is not even one of the candidates under consideration. As a result, those aspects of computing are fractured, inefficient, unpredictable, and driven by fashion and politics. Even while the practitioners call themselves "computer scientists," which is something that none of them are.

Everyone involved knows that the whole process of specifying, designing, building, testing and supporting software is hopelessly inefficient, unpredictable and error-prone. The leading methodology that concentrates on the process of building software, “project management,” is theoretically bankrupt and in any case has an empirical track record of failure. In terms of the content of good software, there are fierce battles among competing approaches, none of which is anything but a collection of unsupported and unfounded assertions, and which in practice don’t contribute to building good software.

Occam’s razor leads to the principles on which good software may be built, and supplies a single simple, widely applicable theme that, when applied, cuts away (as a razor should) all the inefficiency, inapplicability and generally what we don’t like about software. Once the principle is understood and widely applied, I expect it will become the un-discussed and undisputed standard for how software is built, just as it has in the other areas of computing to which it has been applied.

What is this amazing standard? It’s simple. I mean literally simple. That’s what Occam’s Razor is all about.

Occam’s razor is:

    Entia non sunt multiplicanda praeter necessitatem.

    No more things should be presumed to exist than are absolutely necessary.

Occam’s razor applied to software would be:

    No more software entities should be created than are absolutely necessary.

Another way of expressing this is that all redundancy, of any kind and in any way, should be eliminated from a piece of software. This often requires writing imperative code to implement any and all abstract actions, and to create largely declarative meta-data to specify everything that is specific to an application. The meta-data, of course, should also have no redundancy; this is normally achieved by use of multiple inheritance with override. See this for more.

To be extremely brief, the reason why eliminating redundancy is good is that practically everything that happens to a piece of software is that it is enhanced, altered, bug-fixed or otherwise modified. Creation happens just once; everything that follows is growth and change. Implementing change of any kind to a program takes the least effort with the least risk if everything is defined in exactly one place – once you’ve found the place, you make the change and you’re done. Among the infinite universe of programs solving the same problem that work, the speed, cost and risk of change is the overriding virtue, both in technical and business terms.

Thanks to William of Occam, I can’t claim to have invented the core principle of software, and the way to express the measure of goodness for software. William has me beat by over 700 years. Thanks a lot, Bill! Lots of people have applied Occam’s Razor to lots of tough things with optimal success, even in things closely related to software, such as information theory. Way too late again. About all I can do is mess with his name and make up a term that expresses goodness in software. Here it is: I propose that a piece of software can be measured by its “Occamality.” The more “Occamal” it is, the better it is. And I propose that Occamality is strictly correlated with the extent to which there is no redundancy of any kind in a program, which is almost always strictly correlated with the program being at the high end of the hierarchy of abstraction.

How to Pay Down Technical Debt

When we look at a body of software, how do we judge it? Clearly, there are strong opinions on this subject – but there is no general consensus on the standards by which a body of code should be judged! This is shocking and unacceptable!

For most normal people, i.e., people who aren’t software developers, software is “good” if is doesn’t crash and pretty much does what you want most of the time. What more could anyone want? Well, how about changing the software. This is one of the BIG things that makes software different from pretty much anything else in our experience. How often do you try to make changes to your car in the same sense that you change software? The answer is never.

There are standards for nearly everything in life, certainly for things that are built by engineers and other technical people. How is it that, not only are there NO standards for what constitutes “good” software, but the fact that there are no such standards is not decried, and no one appears to be sheepish or unapologetic about admitting it – because they’re never asked to admit it. The subject never comes up – not only in “polite company” but even in “impolite company,” i.e., among programmers talking quietly among themselves.

“Ya know,” someone might say, “we tell them there’s technical debt. Sounds fancy, right? Anyone ever been asked what it means? They seem to think that WE know what it means, and that’s good enough for them. I wonder how much longer we’re going to be able to get away with it?” Someone else says, “Forever, man. This has been going on since before we were BORN, ya know? Unless word somehow gets out, and how could it, we’ll die without anyone being the wiser. Good thing.”

I wish programmers did have quiet dialog among themselves like the one I made up above. But they don’t, at least that I’ve ever heard. There’s nearly always someone who has strong opinions about what software should look like internally, and of course the current body of software doesn’t measure up. So there is agitation and there are threats about the awful future consequences of failing to rein in the spreading disease – soon! RIGHT AWAY!! – unless the transformation is authorized.

Then the re-writing death march starts off, with various sections of the code slated for repair, renewal or even flat-out re-write. What is the target? It varies. Favorite criticisms include that the code is spaghetti, it’s a monolith, it’s not structured according to some set of object-oriented principles, it’s written in some obviously inferior language, there aren’t components, there are no services, micro or otherwise, and on and on. The solution is obvious to the converted, and value to be achieved by the solution is so obvious that it’s hardly worth stating.

Is there any value to be gained by the clean-up of “technical debt?” Maybe. But often, nothing really changes except a lot of time has been spent with no improvement to the business. No one can point to a proof or a study or a clear historic pattern demonstrating the effort is worth making.

This is a shame, because there is a clear pattern of success for a method of organizing and/or re-organizing code for business benefit.

Let’s start from the business benefit. Of all the ways a piece of software could be written that performs the same function, which is the best in business terms? I’m assuming here roughly the same level of performance, etc.

Tick, tick, tick… Think about it!

When you come right down to it, no one cares about today’s software doing today’s things for today’s customers – so long as it works and performs reasonably well. What matters is simple, and everyone knows it: it’s the next request by an important existing customer, and more important, the next potential customer who is balking about things they need that aren’t there, the time, cost and risk to implement the software, etc. In other words, what matters about today’s software is TOMORROW’s changes, whatever they may be – we can suspect, but we won’t know until we know them!

Software people have talked forever about things like “architecting for change” and various fantasies that mostly go poof when the harsh wind of reality rushes at them.

There are exactly two rarely-discussed aspects of code that position it optimally for the widest possible range of unanticipated changes and the needs of installation:

• A minimum of redundancy in the code – anything you want to change can be changed by going to one place
• To the largest extent possible, particular things the application does are defined in meta-data instead of code, with the result that the code is smaller and more abstract with minimal redundancy -- and that any change can more likely be made by changing meta-data, which is quicker and safer than changing code.
  

That’s it! See this for more.

How to Build Applications that can be Changed Quickly

Nearly everyone who talks about fast and reliable application building seems to talk about the process (Agile etc.) and/or a couple architectural things (object-oriented, micro-services, etc.). From what I can tell, these haven’t led to any break-throughs in application velocity or predictability. However, it’s clear that there are methods for high-velocity application growth. A wide variety of small, motivated groups have re-invented them over the years. I have had the privilege of observing these groups and their methods, and seen the impact when they add or drop velocity-aiding methods.

Here is an overview of some of the methods I’ve seen used. The improvements can range from cutting time in half to whole number factors, things like turning weeks into days or in some cases hours.

Forget requirements, focus on change

The normal requirements process starts with the often non-technical people who have a need. The need can be expressed directly, or through refusal to buy/use a product, which gets the attention of sales and marketing people. There then ensues a process of requirements gathering and refinement. Wire-frame UI’s may be involved. Finally, the requirements are ready for engineering, and away we go on hopefully not too many “sprints.” This is an “outside-in” approach.

Speed is achieved by taking an “inside-out” approach – instead of focusing almost exclusively on requirements, center your efforts on the code and what it can do or easily be changed to do today to go in the direction of the requirements. Take steps to change and add to the existing code to meet the unmet needs. Do this in small steps, getting feedback at each step. When you’re done, go back and “clean up” the code to eliminate redundancies as much as possible, to make the next change as easy as possible.

What this does is ground the changes in the existing code, resulting in the changes and additions being smaller than they otherwise would be. As a additional benefit, it usually results in a greater uniformity of both the code and the interfaces (UI and API) to the code.

Minimize the customization and optimization of the code and UI

When product people think about a new process that has UI, they usually try to do a good job, understanding all the details and nuances involved. The result of this sophistication is often requirements that are finely adapted to the particular function being performed. What’s wrong with this, you may wonder? One problem is that this approach ends up taking longer to built and results in more code. More important is the users – the less someone has to learn to use a new feature, the more quickly and easily they’ll be able to use it. Familiarity is incredibly important. Which means, among other things, that literally using rarely changing templates to generate interfaces will both make any new interface element recognizable and easy to learn, but also lead to the least amount of code changes to get the job done.

This means that the fineness and sophistication of product people can be a problem. The more time they spend on a subject and the more they think and reflect on it, the worse the temptation is likely to get to do something other than just crank out another instance of a familiar pattern.  Resist!

Avoid planning and designing for change and future requirements

When starting a project, accomplished software professionals often insist on a period devoted to design or architecture, in which the overall technical approach to the project is determined. When things go wrong later, inadequate or insufficient up-front design is often cited as the reason. “We’re gong to insist on being able to do it right next time, so this won’t happen again.” But all too often, projects slip and slide regardless of any up-front design work.

Generally, up-front design time is actually counter-productive. The reasons are simple: you spend time “designing” when you could have been building; the result of the design is often building for anticipated change, which takes time; when the anticipated change doesn’t happen and unanticipated change arrives, the code added because of the design almost always makes things harder.

None of this happens when you build incrementally and strive toward uniformity, eliminating code redundancies and moving functionality from code into meta-data.

Wartime software

The methods I have described as wartime software maximize speed and efficiency of building new software that meets business needs. See here for details. Generally, the methods include:

• Eliminate overhead
• Avoid things resembling “project management”
• Optimize for speed instead of expectations
• Use production environments for testing instead of sand boxes
• Use comparison QA; avoid test-driven development, unit testing and test scripts
• Minimize the use of documents, meetings and other internal overhead
• The main architectural value should be speed and scale
• Avoid technical fashions

Move up the mountain of abstraction

When you think about code and methods, the main values should be the elimination of redundancy in both data definitions and code, and migrating functionality from code into meta-data. See this and this.

The other methods described here are important for moving quickly, and are complementary to this one. But moving up Abstraction Mountain is the most powerful of the methods, often yielding qualitative improvements.

Conclusion

This is a short post, and the methods are described briefly. In a different world, each topic here would be at least a course in the Computer Science curriculum, and in a couple cases multiple books. I don't claim anything I've said here is fully described. But every single item has been proven in practice by many groups over many years, in different ways, always resulting in better quality software that meets business needs being built dramatically more quickly than other methods could possibly have achieved.

My goal here is simply to provide a short list of the main methods for fast development in a single place.

The Progression of Abstraction in Software Applications

This post describes a little-known concept for understanding and creating software architecture that small groups use to defeat large, powerful incumbents and nimble competitors. It is one of a small number of powerful, repeating patterns that help us understand and predict the evolution of software. Understanding these patterns can help entrepreneurs direct their efforts; if they do it well, they greatly enhance their chances of success. Understanding the patterns can also help investors choose to invest in groups that are walking a path to success.

Evolution of Applications Towards Abstraction on a Platform

One of these patterns is the stages that applications naturally evolve through on a technology platform. Each step or stage brings a big increase in the power of the software, decreasing the effort and increasing the speed and effectiveness of being deployed to meet customer needs.

A category of software applications may well get “stuck” at a particular stage for a long time, sometimes even decades. During this time, the software may appear to move forward, and of course the marketing people and managers will always put things in the best possible light. But it’s always vulnerable to being supplanted by a next-stage version of the functionality.

While there aren’t clear lines of delineation between the stages, it’s nonetheless useful to understand them roughly as:

• Prototype. A hard-coded body of code.
• Custom Application. Does a job reliably, but most changes require changing source code.
• Basic Product. The code now has parameters, maybe user exits and API’s. Real-life implementations tend to require extensive professional services, and the cost of upgrading to new versions tends to be high.
• Parameterized Product. The level of parameterization is high, with interface layers to many things outside the core code, so that many implementations can be done without changing source code. There may be some meta-data or editable rules.
• Workbench Product. A large portion of the product’s functionality has migrated from code to editable meta-data, so that extensive UI, workflow, interface and functionality changes can be accomplished via some form of workbench, which could be just a text editor. The key is that the details of application functionality are expressed as editable data, meta-data, instead of code. Nonetheless, all fundamental capabilities are expressed in highly abstract code.

As a body of code goes through this sequence of abstraction, it is increasingly able to meet the needs of new customers and changing needs of existing customers, with decreasing amounts of effort, risk and changes to source code. At the same time, the more abstract a program, the more functionality is expressed as data that is not part of the software itself, a.k.a. meta-data, and the more the software implements generic capabilities, as directed by the meta-data.

The pattern applies both to individual bodies of code and to collections of them. It applies to code built internally for an organization and to code that is sold as a product in any way.

I defined the stages above as a convenience; in reality, the categories are rarely hard-and-fast. A body of code could be given a big transformation and leap along the spectrum, or it could take a long series of small steps. One body of code could remain stuck with little change in abstraction, while other bodies of code doing similar things could be ahead, or progress rapidly towards abstraction.

The Driver of Abstraction Evolution

In biological nature, competitive pressures and external change appear to drive evolutionary changes. Similarly, when we look at categories of software, if little changes to make the software change, it doesn’t change – why take the trouble and expense to change software that meets your needs or the needs of your customers?

In reality, someone always seems to want changes to an application. A prospective user would gladly use the software if it did this, that or the other thing. A current user complains about something – it’s too slow, too hard to use, too complicated, or it just doesn’t work for X, Y or Z. How often does a piece of software not have a “roadmap?” If it doesn’t, it’s probably slated for retirement before long. Brand-new software is rare. The vast majority of software effort goes into making changes to an existing piece of software.

How much time and effort is needed to change a particular body of software? That is the key touch-point between techies and business people. This is the point at which the level of abstraction of the application comes into play. Regardless of the level of abstraction of the application, the “change” required either has been anticipated and provided for or it has not.

• If the change has been anticipated, the code can already do the kind of thing the user wants – but not the particular thing. Doing the particular thing requires that a parameter be defined, a configuration file changed, a template created or altered, workflow or rule changes made, or something similar that is NOT part of the program’s source code. This means that the requirement can be met quickly, with little chance of error.
• If the change has not been anticipated, then source code has to be changed in some way to make the change. The changes required may be simple and localized, or complex and extensive – but the source code is changed and a new version of the program is created.

This is what the level of abstraction of a software application is all about: the more abstracted the application, the more ways it can be changed without altering the source code and making a new version of the program.

This is the fundamental driver of applications towards increasing abstraction: as changes have to be made to the application, at some point the technical people may decide to make the change easier to make in the future, and create the appropriate abstraction for the kind of change. This may happen repeatedly. As more complex changes are required, the program may just get more complicated and more expensive to make further changes, or more sophisticated abstractions may be introduced.

While historically it appears that outside forces drive applications towards increasing abstraction, smart programmers can understand the techniques of abstraction and build an application that is appropriately abstract from the beginning. Similarly, the abstracting methods can be applied by smart programmers to existing bodies of code to transform them, just because it's a way to build better code and meet ever-evolving business requirements.

Conclusion

The hierarchy of abstraction in software is one of the most important concepts for understanding a specific piece of software, or a group of related software. Over time, software tends to become more abstract because of competitive and business pressures, or because of smart programmers working to make things better. The more abstract a piece of software is, the more likely that it can respond to business and user demands without modifications to the source code itself, i..e., quickly and with low risk.

The hierarchy of abstraction is certainly a valuable way of understanding the history of software. But it is more valuable as a framework for understanding a given piece of software, and the way to evolve that software to becoming increasingly valuable. It is most valuable to software developers as a framework for understanding software, and helping them to direct their efforts to get the greatest possible business impact with the smallest amount of time and effort.

How to Evaluate Programming Languages

Programmers say language A is "better" than language B. Or, to avoid giving offense, they'll say they "like" language A. Sometimes they get passionate, and get their colleagues to program in their new favorite language.

This doesn't happen often; inertia usually wins. When a change is made, it's usually passion and energy that win the day. If one person cares a WHOLE LOT, and everyone else is, "whatever," then the new language happens.

Sometimes a know-nothing manager (sorry, I'm repeating myself) comes along and asks the justification for the change. The leading argument is normally "the new language is better." In response to the obvious "how is it better?" people try to get away with "It's just better!" If the manager hangs tough and demands rationality, the passionate programmer may lose his cool and insist that the new language is "more productive." This of course is dangerous, because the rational manager (or have I just defined the empty set?) should reply, "OK, I'll measure the results and we'll see." Mr. Passion has now gotten his way, but has screwed over everyone. But it usually doesn't matter -- who measures "programmer productivity" anyway?

But seriously, how should we measure degrees of goodness in programming languages? If there's a common set of yardsticks, I haven't encountered them yet.

The Ability of the Programmer

First, let's handle the most obvious issue: the skill of the programmer. Edgar Allan Poe had a really primitive writing tool: pen (not even ball-point!) and paper. But he still managed to write circles around millions of would-be writers equipped with the best word processing programs that technology has to offer. I've dealt with this issue in detail before, so let's accept the qualification "assuming the programmer is equally skilled and experienced in all cases."

Dimensions of Goodness

Programmers confined to one region of programming (i.e., most programmers) don't often encounter this, but there are multiple dimensions of goodness, and they apply quite differently to different programming demands.

Suppose you're a big fan of object-orientation. Now it's time to write a device driver. Will you judge the goodness of the driver based on the extent to which it's object-oriented? Only if you're totally blindered and stupid. In a driver you want high performance and effective handling of exception conditions. Period. That is the most important dimension of goodness. Of all the programs that meet that condition, you could then rank them on other dimensions, for example readability of the code.

Given that, what's the best language for writing the driver? Our fan of object orientation may love the fact that his favorite language can't do pointer arithmetic and is grudgingly willing to admit that, yes, garbage collection does happen, but with today's fast processors, who cares?

Sorry, dude, you're importing application-centric thinking into the world of systems software. Doesn't work, terrible idea.

It isn't just drivers. There is a universe of programs for which the main dimensions of goodness are the efficiency of resource utilization. For example, sort algorithms are valued on both performance and space utilization (less is better). There is a whole wonderful universe of work devoted to this subject, and Knuth is near the center of that universe.

Consistent with that thinking, Knuth made up his own assembler language, and wrote programs in it to illustrate his algorithms. Knuth clearly felt that minimum space utilization and maximum performance were the primary dimensions of goodness.

The largest Dimension of Goodness

While there are other important special cases, the dimension of goodness most frequently relevant is hard to state simply, but is simple common sense:

The easier it is to create and modify programs, quickly and accurately, the more goodness there is.

• Create. Doesn't happen much, but still important.
• Modify. The most frequent and important act by far.
• Quickly. All other things being equal, less effort and fewer steps is better.
• Accurately. Anything that tempts us to error is bad, anything that helps us to accuracy is good.

That, I propose, is (in most cases) the most important measure of goodness we should use.

Theoretical Answer

Someday I'll get around to publishing my book on Occamality, which asks and answers the question "how good is a computer program?" Until then, here is a super-short summary: among all equally accurate expressions of a given operational requirement, the best one has exactly one place where any given semanic entity is expressed, so that for any given thing you want to change, you need only go to one place to accomplish the change. In other words, the least redundancy. What Shannon's Law is for communications channels, Occamality is for programs. Given the same logic expressed in different languages, the language that enables the least redundancy is the best, the most Occamal.

Historical Answer

By studying a wide variety of programs and programming languages in many fields over many years, it's possible to discern a trend. There is a slow but clear trend towards Occamality, which demonstrates what it is and how it's best expressed. The trend is the result of simple pressures of time and money.

You write a program. Sombody comes along and wants something different, so you change the program. Other people come along and want changes. You get tired of modifying the program, you see that the changes they want aren't to different, so you create parameters that people can change to their heart's content. The parameters grow and multiply, until you've got loads of them, but at least people feel like they're in control and aren't bugging you for changes. Parameters rule.

Then someone wants some real action-like thing just their way. You throw up your hands, give them the source code, and tell them to have fun. Maybe they succeed, maybe not. But eventually, they want your enhancements in their special crappy version of your nice program, and it's a pain. It happens again. You get sick of it, analyze the places they wanted the changes, and make official "user exits." Now they can kill themselves customizing, and it all happens outside your code. Phew.

Things keep evolving, and the use of user exits explode. Idiots keep writing them so that the whole darn system crashes, or screws up the data. At least with parameters, nothing really awful can happen. The light bulb comes on. What if I could have something like parameters (it's data, it can't crash) that could do anything anyone's wanted to do with a user exit? In other words, what if everything my application could do was expressed in really powerful, declarative parameters? Hmmm. The users would be out of my hair for-like-ever.

What I've just described is how a new programming "level" emerges historically. This is what led to operating systems, except that applications are one giant user exit. UNIX is chock full of things like this. This history is the history of SQL in a nutshell -- a powerful, does-everything system at the level of declarative, user-exit-like parameters!

The Answer

In general, the more Occamal the language (and its use), the better it is. More specifically, given a set of languages, the best one has

• semantics that are close to the problem domain
• features that let you eliminate redundancy
• a declarative approach (rather than an imperative one)

Let's go through each of these.

Problem Domain Semantics

A great example of a such a language is the unix utility AWK. It's a language whose purpose is to parse and process strings. Period. You want an accounting system, don't use AWK. You want to generate cute-looking web pages, don't use AWK. But if you've got a stream of text that needs processing, AWK is your friend.

From the enterprise space, ABAP is an interesting example. While it's now the prime language for writing SAP applications, ABAP was originally Allgemeiner Berichts-Aufbereitungs-Prozessor, German for "general report creation processor." In other words, they saw the endless varieties of reporting and created a language for it; then having seen the vast utility of putting customization power in the hands of users, generalized it.

Features that let you eliminate redundancy

This is what subroutines, classes and inheritance are supposed to be all about. And they can help. But more often, creating another program "level" is the most compelling solution, i.e., writing everything that the program might want to do in common, efficient ways, and having the application-specific "language" just select and arrange the base capabilities. This is old news. Most primitively, it's a subroutine library, something that's been around since FORTRAN days. But there's an important, incredibly powerful trick here. In FORTRAN (and in pretty much all classically-organized subroutine libraries), the library sits around passively waiting to be called by the statements in the language. In a domain-specific language, it's the other way round!

A related approach, which has been implemented over and over, is based on noticing that languages are usually designed independent of databases, but frequently used together. The result is monster redundancy! Wouldn't it be a hoot if database access were somehow an integral part of the language! Well, that explains how and why ABAP evolved from a reporting lanaguage (necessarily intimate with the database) into "Advanced Business Application Programming." And it explains why Ruby, when combined with the database-leveraging RAILS framework, is so popular and highly productive.

A declarative approach

In the world of programming, as in life, the world divides between imperative (commands you, tells you how to do something, gives you directions for getting to point B) and declarative (tells you what should be done, identifies point B as the goal). In short, "what" is declarative and "how" is imperative. A core reason for the incredible success of SQL is that it is declarative, as Chris Date has described in minute detail. Declarative also tends to be less redundant and more concise. As well as doesn't crash.

Conclusion

I'm sorry if you're disappointed I didn't name Erlang or whatever your favorite is as the "best programming language," but I insist that it's far more novel and useful to decide on what basis we are to judge "goodness" in programming languages, and in programs. In general and in most domains (with important exceptions, like inside operating systems), non-redundancy is the reigning virtue, and languages that enable it are superior to ones that are not. Non-redundancy is nearly always best achieved with a declarative, problem-domain-centric approach. And it further achieves the common-sense goal of fast, accurate creation and modification of programs.

Occamality is rarely explicitly valued by programmers, but the trend to it is easy to see. There are widespread examples of building domain-specific languages, meta-data and other aspects of Occamlity. Many programmers already act as though Occamality were the primary dimension of goodness -- may they increase in numbers and influence!