In the long-fought war to improve software programming productivity, there have been offensives on many fronts, but precious little genuine progress made. We give our programmers the fanciest, most high-tech equipment imaginable – and it is orders of magnitude faster and more powerful than the equipment available to earlier generations – but this new equipment has made only marginal difference. While relieving programmers of the burden of punch cards helps, the latest generation of programmers are not getting the job done much better than their comparatively low-tech predecessors.
Form and Content
As usual, most efforts to improve the situation focus on one of two general approaches: form or content.
People who focus on form tend to think that the process of getting software written is what’s important. They talk about how one “methodology” is better than another. They think about how people are selected, trained, organized and motivated. As you might imagine, the spectrum of methodologies is a broad one. On one end of the spectrum is the linear approach, which starts from the generation of business requirements for software, and ends with testing, installation and ongoing maintenance. On the other end of the spectrum is the circular, interactive approach, in which a small working program is built right away, and gradually enhanced by programmers who interact closely with the eventual end-users of the program. There are any number of methodologies between these two extremes, each of which claims an ideal combination of predictable linearity with creative interactivity.
People who focus on content tend to think that the language in which programs are written and the structure and organization of programs are what’s important. Naturally, they like design and programming tools that give specific support to their preferred language and/or architecture. There tends to be a broad consensus of support at the leading edge around just a couple of languages and structures, while the majority of programmers struggle with enhancing and maintaining programs originally written according to some earlier generation’s candidate for “best language” or “best architecture.” At the same time, many of those programmers make valiant attempts to renovate older programs so that they more closely resemble the latest design precepts, frequently creating messes. Regardless of the generation, programmers quickly get the idea that making changes to programs is their most time-consuming activity (apart, of course, from never-ending meetings), and so they focus on ways to organize programs to minimize the cost of change. This leads to a desire to build “components” that can be “assembled” into working programs, and naturally to standards for program components to “talk” with each other.
Lots of effort, not much progress
The net effect of all these well-intentioned efforts, on both the form and content sides, has been a little bit of progress and a great deal of churn. Having some agreed-on methodology leads to better predictability and generally better results than having none; it seems that having a beat to which everyone marches in unison, even if it’s not the best beat, leads to better results than having a great beat that many people don’t know or simply refuse to march to. What is the best methodology in any one case depends a great deal on both how much is already known about the problem to be solved and how smart and broadly skilled the participants in the project are. The more qualified the people and the less known about the target, the more appropriate it is to be on the “interactive” end of the spectrum; think highly qualified and trained special forces going after a well-defended target in enemy territory – creativity, teamwork and extraordinary skills are what you need. The more ordinary the participants and with better-understood objectives, the more appropriate being somewhere towards the “linear” end of the spectrum; think of a large army pressing an offensive against a broad front – with so many people, they can’t all be extraordinary, and you want coordinated, linear planning, because too much local initiative will lead to chaos.
As to content, while it is clear that the latest programming languages encourage common-sense concepts like modular organization and exposing clear interfaces, good programmers did that and more decades ago in whatever language they were writing, including assembler, and bad programmers can always find a way to make messes. And even if you use the best tools and interface methods, incredible churn is created by frequent shifts in what is considered to be the best architecture, practically obsoleting prior generations. Probably the single biggest movement over the last several decades has been the gradual effort to take important things about programs that originally could be discovered only by inspecting the source code of the program and making those things available for discovery and use by people and programs, without access to the source code. The first major wave of this movement led to exposure of much of a program’s persistent data, in the form of DBMS schemas; the other major wave of this movement (in many embodiments, from SAA to SOAP to microservices) exposes a program’s interfaces, in the form of callable functions and their parameters. This has been done in the belief that it will enable us to make important changes to some programs without access to or impact on others, and thus approach the dream of programming by assembling fixed components, like building blocks or legos, into completed programs or buildings. The belief in this dream has been affected very little by decades of evidence that legos don’t build things that adults want to live in.
I believe that while considerations of form are incredibly important, and when done inappropriately can drag down or even sink any programming project, there is little theoretical headway to be made by improvements in methodology. I think most of the relevant ideas for good methodology have already been explored from multiple angles, with the exception of how to match methodology to people and project requirements – no one methodology is the best for each project and each collection of people. But even when you’ve picked the best methodology, the NBA players will always beat the high school team, as long as the NBA players execute well on the motivational and teamwork aspect that any good methodology incorporates. With methodology we’re more in need of good execution and somehow assembling a talented and experienced team than we are of fresh new ideas.
Ptolemy, Copernicus, Kepler, Newton
Content, however, is another story altogether. I think our current best languages and architectures will look positively medieval from the perspective of a better approach to content. I think there is a possible revolution here, one that can bring about dramatic improvements in productivity, but which requires an entirely new mind-set, as different as Copernicus and Ptolemy, as different as Einstein and Newton.
Having said that, let me also say that the “new” ideas are by no means completely new. Many existing products and projects have exploited important parts of these ideas. What is mostly new here is not any particular programming technique, but an overriding vision and approach that ties various isolated programming techniques into a unified, consistent whole. Kepler’s equations described the motion of planets with accuracy equal to Newton’s – in fact, you can derive one set of equations mathematically from the other; but Newton provided the vision (gravity) and the tools (calculus) that transformed some practical techniques (Kepler’s equations) into the basis of a new view of the world. That’s why physics up to the end of the nineteenth century was quite justifiably characterized as “Newtonian” and not “Keplerian.”
Ptolemy and Newton looked at the same set of objects; Ptolemy picked the closest one, the earth, to serve as the center of thinking, while still incorporating the rest.
His main goal was to describe the motions of what you could see, focused on the matter. Copernicus noted that things got simpler and more accurate if you picked one farther away, the Sun, to serve as the center of things.
Kepler made things better by noticing the curves made by the planets. Newton then took the crucial step: instead of focusing on matter, he focused on energy (gravity in this case), and wrote an equation describing how gravity works,
which creates changes in the location and velocity of matter. In programming, we have clear equivalents of matter and energy: matter is data (whether or not it is persistent), and energy is instructions (lines of code, regardless of the language it’s in). In COBOL, for example, this division is made explicit in the data division and the procedure division. In modern languages the two are more intermixed, but it remains clear whether any particular line describes data or describes an action to take (an if statement, assignment statement, etc.).
Now, in spite of what you may have been taught in school, Ptolemy’s method works. In fact, it would be possible (though not particularly desirable) to update his approach with modern observations and methods, and have it produce results that are nearly identical to those possible today; in fact, only when you get to relativity does Ptolemy’s method break down altogether -- but remember that the effects of relativity are so small, it takes twentieth century instruments to detect them. But no one bothers to do this, because the energy-centered approach developed by Newton and refined by his successors is so much simpler, cleaner and efficient.
Similarly, there is no doubt that the instructions-centric approach to programming works. But what comes out of it is complicated, ugly and inefficient. The Newtonian breakthrough in programming is replacing writing and organizing instructions (and oh, by the way, there’s some data too) as the center of what we do with defining and describing data (and oh, by the way, there are some instructions too). The instruction-centered approach yields large numbers of instructions with a small amount of associated data definitions; the data-centered approach yields large numbers of data definitions and descriptions operated on by a significant body of unchanging, standard instructions and small numbers of instructions specifically written for a particular program. In the instruction-centered approach, we naturally worry about how to organize collections of instructions so that we can write fewer of them, and arrive at concepts like objects, components and class inheritance (a subclass inherits and can override its parent’s methods (instructions)). In the data-centered approach, we naturally worry about how to organize collections of data definitions so that we can have the minimal set of them, and arrive at concepts like meta-data, standard operations (e.g. pre-written meta-data-driven instructions enabling create, query, select, update and delete of a given collection of data) and data inheritance (a child data definition, individual or group, inherits and can override its parent’s definitional attributes). In short, we separate out everything we observe into a small, unchanging core (like Newton's gravity) that produces ever-changing results in a diverse landscape.
We shift our perspective in this way not because it enables us to accomplish something that can’t be accomplished by the current perspective, but because it proves to be cleaner, simpler, more efficient, easier to change, etc.
Instructions, data and maps
Only by understanding the details of this approach can it really be appreciated, but the metaphor of driving instructions and maps is appropriate and should make the core idea clear.
Suppose your job is to drive between two locations, and the source and destination location are always changing. There are two general approaches for giving you directions:
- Turn-by-turn directions (the instruction-driven, action-oriented approach)
- A map, with source and destination marked (the data-driven, matter-oriented approach)
The advantage of directions is that they make things easy for the driver (the driver is like the computer in this case). You pick up one step, drive it, then pick up the next, drive it, and so on until the end. You don’t have to think in advance. All you have to do is follow directions, which tell you explicitly what to do, in action-oriented terms (turn here, etc.).
The problem with directions come in a couple of circumstances:
- You have to have a huge number of directions to cover all possible starting points and destinations, and there is a great deal of overlap between sets of directions
- If there is a problem not anticipated by the directions, such as an accident or road construction, you have to guess and get lucky to get around the problem and get back on track.
A map, on the other hand, gives you most of the information you need to generate your own directions between any two given points. The map provides the information; you generate the actions from that information. With a direction-generating program and some parameters like whether to use toll roads, a generic program can generate directions on the fly. With a program like Wayz, real-time changes due to updated traffic information can even be generated.
The Wayz program itself isn't updated very often -- it's mostly the map and road conditions. Same thing with a program written using this approach; the core capabilities are written in instructions, while the details of the input, storage, processing and output are all described in a "map" of the data and what is to be done with it.
Conclusion
It's pretty simple: programming today largely follows the method of Ptolemy, resulting in an explosion of software epi-cycles to get anything done. Attempts to keep things easily changeable sound promising but never work out in practice.
The way forward is to focus instead on what there is and what is to be done with it (data and metadata), with a small amount of Wayz-like code to take any user or data stream from its starting point to its destination with easy changes as needed.
Comments