The Black Liszt

Franz Liszt: 210

This week is the 210th anniversary of the birth of Franz Liszt on Oct 22, 1811. I wrote a couple highlights about Liszt for his 200th anniversary. Aside from loving his achievements in life, he was the inspiration for this blog.

Liszt was one of those rare people who was world-class brilliant and also appreciated the valuable work of others, supporting that work in word and deed.

One way he demonstrated this was by championing the work of other composers by transcribing their work and performing it for audiences who might otherwise not have a chance to hear it. He made amazing transcriptions of operas and many other kinds of works. Probably the most impressive transcriptions he made were making piano versions of all nine Beethoven symphonies!

Oddly enough, the symphony transcriptions were largely ignored in the world of music, even by Liszt's pupils. It wasn't until Glenn Gould recorded a couple of them in 1967 that they began to appear on the scene. But they remain obscure, even among classical music lovers.

I thought I knew all the Beethoven symphonies, but hearing a Liszt piano transcription performed was like hearing it for the first time. After a great deal of struggle he finally managed to transcribe the fourth (choral) movement of the ninth symphony. And did it again for duo pianos.

Frederic Chiu has recorded the 5th and 7th Symphonies for Centaur Records. In his program notes he suggests that "Liszt's piano scores must therefore be taken as a sort of gospel in regards to Beethoven's intentions with the Symphonies" because of Liszt's unique perspective, having met Beethoven in person, having heard collaborators and contemporaries of Beethoven perform the Symphonies, having studied and performed the works both as a pianist/transcriber and as a conductor in Weimar. No one in history could claim to have as much exposure, insight and journalistic integrity as regards Beethoven's intentions around the Symphonies.

I personally recommend the recordings by Konstantin Scherbakov. You will be listening to the unique inspiration of Beethoven, further enhanced by the incredible genius of Liszt.

Listening to these transcriptions can pull you out of normal physical space into a dimension filled with drama, beauty and inspiration, one that, unlike the austere, timeless beauty of math, drives through time from a start to a conclusion.

Software Programming Language Evolution: Libraries and Frameworks

We've talked about the major advances in programming languages and the enhancements that brought those languages to a peak of productivity -- a peak which has not been improved on since. Nonetheless there has been an ongoing stream of new programming languages invented, each of which is claimed to be "better" -- with no discussion about what constitutes "goodness" in a language! This is high on the list of causes of the never-ending chaos and confusion that pervades the world of software. While loads of people focus on language, the most important programming tools that make HUGE contributions to the productivity of programmers using them goes largely unremarked, in spite of the fact that they are part of every programmer's day-to-day programming experience. What are these essential, always-used but largely in the background thing? Libraries and frameworks.

Libraries

A library is a collection of subroutines, often grouped by subject area, which perform commonly used functions for programs. Every widely used language has a library that is key to its practical use.

For example, the C language has libraries that contain hundreds of functions for

• input and output, including formatting
• string manipulation and character testing
• math functions
• standard functions, dozens of them
• date and time

There are libraries of functions for controlling displays and input devices and many other things. After all these years and the "fatal" flaw of not being object-oriented (gasp!!), it is currently the #2 language in popularity. While the libraries don't play as important part of its popularity as certain other languages, they are still important.

Python became widely used among people doing analytics not particularly because of the virtues of the language itself, but because it grew one of the richest modern libraries of routines for calculations available. It ended up with great support for statistics and the things you need to do with large number sets including array and matrix manipulation. When doing analytics, the library functions do nearly all the heavy lifting. All that the program has to do is deploy the rich set of functions against the problem in the right way!

A 2021 survey of language popularity confirms the popularity of Python, and the importance of its library in giving it its popularity. Here's a summary:

Have you ever heard of the language R? It's an open-source language which became usable in the year 2000. Chances are if you work in statistics, data analytics, operations research or other areas involving serious data manipulation you know all about it and probably use it. The language itself has valuable features for math programming that most languages don't have, for things like vectors and matrices. More importantly it has an amazing rich library of the R equivalent of libraries, called packages. R packages contain reusable collections of functions and data definitions to perform valuable calculations. When you're trying to do something you're pretty sure someone else has tried to tackle in some way, the first thing you do is look for an appropriate package. Packages are available for most common data science tasks; they do much of the work and nearly all the "heavy lifting" of performing any job for which the package applies. R is a prime example of a language having virtues, but the bulk of the value and productivity coming from the available libraries.

One of the sad turns taken in programming language evolution was driven by the ascendancy of the modern RDBMS. Ironically, the DBMS emerged at about the same time as do-everything language environments dominated the new minicomputer landscape. With a single environment like MUMPS or PICK you could get a whole job done -- user interface, core program, data storage and access, everything! These environments enabled massive productivity gains. See this for more. But the DBMS blasted in, took over, and HAD to be used. So 3-GL's that were less productive than the new environments became even less so by finding cumbersome ways to use DBMS's. The way they did it I explain here. This was a first major step down the degenerate, productivity-killing road of layers. How did 3-GL's pull off the integration? With libraries, of course! For example, Java's JDBC library became a requirement for burdensome, error-prone access to standard databases from within Java, while those who didn't mind the overhead and "impedance mismatch" could use one of the ORM's (Object-Relational Mapping systems) that emerged.

Not all libraries are tightly associated with a language! One such library is redis.io, an open source project started by an Italian developer to build a powerful in-memory key/value datastore and cache. It has taken off and has many more powerful features including queuing. While unknown among non-programmers, it is incredibly popular, widely used and available on the major cloud providers.

Other libraries are directed towards a particular aspect of programming. With the emergence of the web UI, a succession of libraries to ease their creation have emerged.

A UI library that has become dominant is React, created and then open-sourced by Facebook. It is specifically for the Javascript language and greatly enhances the productivity of building and the resulting performance of web UI's. The benefits of javascript or any other language are trivial compared to the productivity gain produced by using React. Unlike most libraries, React comes close to being a framework; it resembles an R package, in that it's a comprehensive solution to the problem of building web UI's. Interestingly, part of the programmer productivity gain is due to the fact that it has a major declarative aspect to its design, like AngularJS (see below).

Bottom line: while the details of software language features have some impact on productivity, the availability and richness of libraries enhances productivity many times over. It's no contest.

Frameworks

Frameworks take the idea of libraries but with an important reversal. Libraries are sets of routines, any of which may be called by a program written in a supported language. The program written in the language is completely in charge. Frameworks, by contrast, provide an environment in which a language can operate. You select exactly one framework to work in. When you play by its rules, you normally enjoy large productivity benefits.

A library is a rich set of resources covering many issues, like a book library. A framework is a selected set of resources enabling rapid work on a given category of issues, like a kitchen for cooking.

One impressive framework is the RAILS framework for the language Ruby. While the Ruby language had become fairly successful, Ruby on Rails (as it was called) established it as a player because many groups discovered that it was many times faster than other tools at building a web application that has a UI and a database. The reason is simple: normally to build such an application you have one expert creating the UI probably using javascript and some library, another one building the server-side business logic, and another one creating the DBMS -- with a great deal of work and trouble relating the database to the server program; see for example JDBC and ORM's above. With RAILS, you defined your database, which automatically defined the names you use inside Ruby to access the data. Similar story with the UI.

The philosophy of RAILS centered on the DRY principle -- Don't Repeat Yourself.

Unlike the endless repetition of the usual layered environment, RAILS took a major step towards the principle of Occamality. Was RAILS original, a real break-through? In its object-oriented environment, yes. In programming in general, no. RAILS is a typical example of the appalling lack of knowledge of history in software. Most of the benefits of RAILS were delivered in a more integrated way many years before by the Powerbuilder environment. I discuss the context here.

What comprehensive frameworks like this REALLY do is attempt to re-create the comprehensive, everything-in-one-place development environments that emerged to enhance programmer productivity beyond what was achievable in 3-GL's, mostly by incorporating data storage and user interface functions, as I explain here. Their emergence and widespread adoption is clear evidence of the power of eliminating the insanity of layers in software.

There are also frameworks that are much narrower in scope, addressing only part of the programming puzzle. While they leave much of an overall programming job alone, they can give a major boost to the narrow area on which they focus. One that is focused exclusively on building web UI's  that caught on and became widely used is AngularJS, a new version of which is simply Angular. This framework is highly declarative, focused more on describing the elements of the UI than the actions required to implement it.

This led to HUGE programmer productivity gains. Why would anyone build a web UI from scratch? Nonetheless, the similarities between the ReactJS library and the AngularJS framework are strong, and both of them powered huge programmer productivity gains that had very little to do with the virtues (or lack thereof) of the associated language, javascript.

Languages vs Libraries and Frameworks

 New languages continue to flow out of the creative minds of groups of programmers who obviously don't have enough to do to keep themselves fully occupied. Each new language tends to be lauded, if only by its creators, with extreme claims of virtue on many dimensions. None of these claims are ever subjected to verification and testing, much less of the rigorous kind. In any case, there is simply no contest between the gains delivered by a rich library or framework and a new programming language.

I'll just give glaring example. One of the main virtues that Object-oriented languages are supposed to have is code reuse. The concept that is bandied about is that good class system is like a set of Lego blocks, enabling new programs to be easily assembled. Mostly it doesn't happen. For re-use, libraries and frameworks are the gold standard. Think of a normal, old-fashioned book library. The whole reason they exist is that people reuse books! It's the same thing for software libraries -- code gets into software libraries because it's re-used often! That's how they got to be called "libraries." Duh.

 

What is technical due diligence for Venture Capital?

I’m Technology Partner in a VC firm, Oak HC/FT. I help look at companies we’re considering investing in, and I help out as needed after investment. Recently I’ve been thinking about what I do, mostly because I’ve encountered multiple cases of technology due diligence performed on companies I know well. To put it plainly, I don’t do it the same way.

The firms whose work I’ve seen are professional, well-regarded groups. Their work fits into a pattern of similar work I’ve seen over many years. Their work can have value.

My Evolution in Tech Due Diligence

When I started performing tech due diligence for the VC firm Oak Investment Partners in 1992, I had no specific idea of how to do it. My background had been almost exclusively on the “other side,” i.e., working and building software, mostly for small, entrepreneurial companies. But I’d had occasion to dive into bodies of software over the years and figure them out, so I guessed I’d be able to figure it out better than any non-programmer could. All I can say is, I went through quite an evolution from there.

The starting point was clear and simple: evaluate the software. There’s a lot of work involved and not many people have the skills to do it well, but there’s no magic. I quickly developed an outline for what an evaluation consists of. It was a few pages long. Here’s the last part:

The earlier parts … well, let’s just say it was comprehensive.

I marched forward doing the work and interacting with the people in the company being evaluated and with the partners in the VC firm. I evolved my views of tech due diligence.

In the late 1990’s, I did an on-site tech diligence of a little company called Inktomi. It was started by a couple guys from U Cal Berkeley, one a young professor and the other a grad student. They were hard at work on a project called NOW – the network of workstations, which was an early attempt to build software that would enable a bunch of workstations connected by fast networking to solve really big computing problems, of the kind normally only so-called super-computers could handle. They built the software, and needed a big, big problem to solve to demonstrate that it worked. There were already full-text databases around that ran on standard computers. Also at the time, the internet was exploding, with no end in sight. There was an emerging need to help people find things on the internet. At the time, the best available solutions were “portals,” of which Yahoo was a leading example. A portal was a densely populated page managed by people who would put up links to the main things people were interested in. But what if you wanted more? Wouldn’t it be great to have a full-text search engine that could index and search the entire internet, even as it grew endlessly?

This is the problem Inktomi took on. No existing search engine could come within a factor of 100 of indexing and searching the whole internet at the time. The boys at Inktomi used their NOW infrastructure to attack the problem, first building a crawling and index-building system, then building the search. It wasn’t easy, but it turns out that it fit right into the NOW approach. If you understood it, you could see that by adding workstations to the NOW, you could scale the index endlessly, no matter how large the internet grew.

I walked into the second-floor warren of offices above stores on Shattuck Ave. in Berkeley, with machines and people crammed in anywhere they would fit. I already had heard the basic idea. I looked over people’s shoulders at screens, and noticed some things about the C code there. I asked questions and discovered they had implemented parallelism not by using standard UNIX multi-tasking, but by a super-low-overhead coding method. I saw the cables and boards, and found out the clever things they had done to minimize inter-workstation latency. And some more stuff.

Before long, and without doing anything like the exhaustive inventory of my original approach to due diligence, I had discovered the originality of their approach and their impressive implementation of it, which would give them a lead over any competitors that would emerge. I became a strong advocate of the deal.They took our money, the largest investment we had made in a company to that date, and ended up making over 100X return.

Of course today, we don’t “inktomi” for things on the internet, we “google.” That’s a story for another time, and happened after Inktomi had attained a public market valuation in excess of $10 Billion.

I tell the story to illustrate one of the fulcrum points of my evolution in performing tech due diligence. The way I did it and the resulting win helped spur me on to doing what the VC really needs, which is to leverage my experience and skills into getting insights into a potential deal that are invisible to the other members of the firm because of their lack of detailed knowledge of software and systems. The insights can range from “makes no difference here” to “OMG negative” to “how could I have known that super-positive.”

What I finally evolved to, and continue to work at, revolves around this simple fact: the VC firm wants to know if this technology, the way it is today and is likely to grow into the future, will make the company into a success and end up making money for us and our investors. That’s it! Everything else is a footnote or appendix.

The vast majority of tech due diligence I see performed by firms who specialize in performing that work, and who are regularly retained by groups doing investments and/or acquisitions, is not about this. What I almost universally see is something like what I started with long ago, with this very important added element: how do the tech, the people, the organization, the process, the deployment and the architectural choices made by the firm compare to widely held industry standards, best practices and regulations? When I look back at my old outline, I see this element entirely missing! Thinking back, I realize that the reason is that I had already decided that those widely held standards, unlike the ones in law and accounting, were a pile of crap. I had already noticed what I later began to systematically study and write about, the difference between the "war time" software practices of winning startups, compared to the widely accepted "peace time" methods.

Conclusion

If you want to acquire a tech-oriented company, you naturally want to perform due diligence. If you’re a big company, you may want to acquire it for its momentum and market edge; Facebook and Google do lots of acquiring for this reason, among others. You want to know how well the target company conforms to the kinds of standards, practices and procedures that your internal organization is held to. But if you’re a smart VC, you want different things from tech diligence. Does the tech give the company an “unfair advantage?” Do they approach tech with a speed-oriented, get-stuff-done attitude? The tech may be 2X better than the incumbents, but could a newcomer pretty easily get 5X? Sounds improbable, but this kind of thing happens all the time. If you’re a VC about to place a bet on an up-and-coming company, that’s the kind of tech input you should want.

Software Programming Language Evolution: the Structured Programming GOTO Witch Hunt

In prior posts I’ve given an overview of the advances in programming languages, described in detail the major advances and defined just what is meant by “high” in the phrase high-level language. I've described the advances in structuring and conditional branching that brought 3-GL’s to a peak of productivity.

The structuring and branching caught the attention of academics. Watch out! What happened next was that a theorem was proved, a movement was declared and named, and a certain indispensable part of any programming language, the GO TO statement, was declared to be a thing only used by bad programmers and should be banned. Here's the story of the nefarious GOTO.

Structures in Programming Languages

I've described how structures were part of the first 3-GL's and how they were soon elaborated to more clearly express the intention of programmers, making code even more productive to write. The very first FORTRAN compiler, delivered in 1957, included primitive versions of conditional branching and loops, two of the foundations of programming structure. It was so powerful that the early users figured it decreased the number of statements needed to achieve a result more than 10 times.

These are the people who actually WRITE PROGRAMS! They want to make it easier and jumped on anything that gave a dramatic improvement.

“Significantly, the increasing popularity of FORTRAN spurred competing computer manufacturers to provide FORTRAN compilers for their machines, so that by 1963 over 40 FORTRAN compilers existed. For these reasons, FORTRAN is considered to be the first widely used cross-platform programming language.”

Before long, the structuring capabilities of the original IF (conditional branching) and DO (controlled looping) statements were enhanced and augmented to something close to their current form. I describe this here. The result was a peak of programmer productivity that has not substantially been increased since, and often been degraded.

The Bohm-Jacopini Theorem

Completely independent of the amazing advances in languages and programming productivity that were taking place, math-oriented non-programmers were hard at work deciding how software should be written. Here is the story in brief:

The structured program theorem, also called the Böhm–Jacopini theorem,^[1][2] is a result in programming language theory. It states that a class of control-flow graphs (historically called flowcharts in this context) can compute any computable function if it combines subprograms in only three specific ways (control structures). These are

1. Executing one subprogram, and then another subprogram (sequence)
2. Executing one of two subprograms according to the value of a boolean expression (selection)
3. Repeatedly executing a subprogram as long as a boolean expression is true (iteration)

The structured chart subject to these constraints may however use additional variables in the form of bits (stored in an extra integer variable in the original proof) in order to keep track of information that the original program represents by the program location. The construction was based on Böhm's programming language P′′.

The theorem forms the basis of structured programming, a programming paradigm which eschews goto commands and exclusively uses subroutines, sequences, selection and iteration.

This theorem got all the academic types involved with computers riled up. The key to good software has been discovered! The fact that math theorems are incomprehensible to the vast majority of people, and the fact that perfectly good computer programs can be written by people who aren't math types didn't concern any of these self-anointed geniuses.

The important thing to note about the theorem is that it was NOT created in order to make programming easier or more productive. It just "proved" that it was "possible" to write a program under the absurd and perverse constraints of the theorem to compute any computable function. Assuming you were willing to use a weird set of bits to store location information in ways that would make any such program unreadable by any normal person. Way to go, guys -- let's go back to the days of writing in all-binary machine language!

The Crisis in Software and its solution

Not long after this, the academic group of Computer Science “experts” formed. They had a conference. They looked at the state of software and declared it to be abysmal. The whole conference was about the "crisis" in software. See this for details.

One of the most prominent of those Computer Scientists was Edsger W. Dijkstra. He looked at the powerful constructs for conditional branching, loops and blocks that had been added to 3-GL's and invented the term "structured programming" to describe them. He related those statements to the wonderful but useless math proof about the minimal requirements for programming a solution to any "computable function." The proof "proved" that such programs could be written without the equivalent of a GOTO statement. BTW, I do not dispute this. He wrote "the influential "Go To Statement Considered Harmful" open letter in 1968."

Among the solutions to the software crisis they proclaimed was strict adherence to the dogma of what Dijkstra called “structured programming,” which prominently declared that the GOTO statement had no place in good programming and should be eliminated.

Does the fact that's it is POSSIBLE to program a solution to any computable function without using GOTO mean that you SHOULD write without using GOTO's? When children go to school, it's POSSIBLE for them to crawl the whole way, without using "walking" at all. Everyone accepts that this is possible. When you're on your feet all sorts of bad things can happen -- you can trip and fall! Most important, you can get the job done without walking ... and therefore you SHOULD eliminate walking for kids getting to school. QED.

This is academia for you – a prime example of how Computer Science works hard to make sure that programs are hard to write, understand and deliver, all in the name of achieving the opposite.

The debate about structured programming

There was no debate about the utility of the conditional branching, controlled looping and block structures that rapidly became part of any productive software language. They were there and programmers used them, then and now. The debate was about "structured programming," which by its academic definition outlawed the use of the GOTO statement. That wasn't all. It also outlawed having more than one exit to a routine, breaks from loops and other productive, transparent and generally useful constructs.

I remember clearly as a programmer in the 1980's having a non-technical manager type coming to me and quizzing me about whether I was following the rigors of structured programming, which was then talked about as the only way to write good code. I don't remember my answer, but since I knew the manager would never go to the trouble of actually -- gasp! -- reading code, my answer probably didn't matter.

The most important thing to know about the leader of the wonderful movement to purify programming is his lack of interest in actually writing code:

Fortunately, there are sane people in the world, including the incomparable Donald Knuth (an academic Computer Scientist who's actually great!) and a number of others.

An alternative viewpoint is presented in Donald Knuth's Structured Programming with go to Statements, which analyzes many common programming tasks and finds that in some of them GOTO is the optimal language construct to use.^[9] In The C Programming Language, Brian Kernighan and Dennis Ritchie warn that goto is "infinitely abusable", but also suggest that it could be used for end-of-function error handlers and for multi-level breaks from loops.^[10] These two patterns can be found in numerous subsequent books on C by other authors;^{[11][12][13][14]} a 2007 introductory textbook notes that the error handling pattern is a way to work around the "lack of built-in exception handling within the C language".^[11] Other programmers, including Linux Kernel designer and coder Linus Torvalds or software engineer and book author Steve McConnell, also object to Dijkstra's point of view, stating that GOTOs can be a useful language feature, improving program speed, size and code clarity, but only when used in a sensible way by a comparably sensible programmer.^[15][16] According to computer science professor John Regehr, in 2013, there were about 100,000 instances of goto in the Linux kernel code.^[17]

Any programmer can make mistakes. Any statement type can be involved in that mistake. For example, I think nearly everyone accepts that cars are a good thing. But over 30,000 people a year DIE in car accidents! So where's the movement to eliminate cars because of this awful outcome! It makes as much sense as outlawing the GOTO because sometimes it's used improperly. Like every other statement type.

Software Programming Language Evolution: Structures, Blocks and Macros

In prior posts I’ve given an overview of the advances in programming languages, described in detail the major advances and defined just what is meant by “high” in the phrase high-level language. In this post I’ll dive into the additional capabilities added to 3-GL’s that brought them to a peak of productivity.

History

Let’s remember what high-level languages are all about: productivity! They are about the amount of work it takes to write the code and how easy code is read.

The first major advance, from machine language to assembler, was largely about eliminating the grim scut-work of taking the statements you wanted to write and making the statements “understandable” to the machine by expressing them in binary. Ugh.

The second major advance, to 3-GL’s like FORTRAN and COBOL, was about eliminating the work of translating from your intention to the assembler statements required to express that intention. A single line of 3-GL code can easily translate into 10 or 20 lines of assembler code. And the 3-GL line of code often comes remarkably close to what you actually want to “say” to the computer, both writing it and reading it.

FORTRAN achieved this goal to an amazing extent.

“with the first FORTRAN compiler delivered in April 1957.^[9]:75 This was the first optimizing compiler, because customers were reluctant to use a high-level programming language unless its compiler could generate code with performance approaching that of hand-coded assembly language.^[16]”

“While the community was skeptical that this new method could possibly outperform hand-coding, it reduced the number of programming statements necessary to operate a machine by a factor of 20, and quickly gained acceptance.”

The reduction in the amount of work was the crucial achievement, but just as important was the fact that each set of FORTRAN statements were understandable, in that they came remarkably close to expressing the programmer’s intent, what the programmer wanted to achieve. No scratching your head when you read the code thinking to yourself “I wonder what he’s trying to say here??” This meant easier to write, fewer errors and easier to read.

Enhancing conditional branching and loops

The very first iteration of FORTRAN was an amazing achievement, but it’s no surprise that it wasn’t perfect. At an individual statement level it was nearly perfect. When reading long groups of statements there were situations where the code was clear, but the intention of the programmer not clearly expressed in the code itself – it had to be inferred from the code.

The first FORTRAN had a couple of the intention-expressing statements: a primitive IF statement and DO loop. Programmers soon realized that more could be done. The next major version was FORTRAN 66, which cleaned up and refined the early attempts at structuring. Along with the appropriate use of in-line comments, it was nearly as clear and intention-expressing as any practical programmer could want.

The final milestone in the march to intention-expressing languages was C.

It’s an amazing language. While FORTRAN was devised by people who wanted to do math/science calculations and COBOL by people who wanted to do business data processing, C was devised to enable computer people to write anything – above all “systems” software, like operating systems, compilers and other tools. In fact, C was used to re-write the first Unix operating system so that it could run on any machine without re-writing. C remains the language that is used to implement the vast majority of systems software to this day.

I bring it up in this context because C added important intention-expressing elements to its language that have remained foundational to this day. It enhanced conditional statements, creating the full IF-THEN-ELSE-ENDIF that has been common ever since. This meant you could say IF <condition is true> THEN <a statement>. The statement would only be executed if the condition was true. You could tack on ELSE <another statement> ENDIF. This isn’t dramatic, but the only other way to express this common thought is with GOTO statement – which certainly can be understood, but takes some figuring. In addition, C added the ability to use a delimited block of statements wherever a single statement could be used. When there are a moderate number of statements in a block, the code is easy to read. When there would be a large number, a good programmer creates a subroutine instead.

C added a couple more handy intention-expressing statements. A prime example is the SWITCH CASE BREAK statement. This is used when you have a number of conditions and something specific to do for each. The SWITCH defines the condition, and CASE <value> BREAK pairs define what to do for each possible <value> of the SWITCH.

Lots of following languages have added more and more statements to handle special cases, but the cost of a more complex language is rarely balanced with the benefit in ease and readability.

The great advance that's been ignored: Macros

C did something about all those special cases that goes far beyond the power of adding new statements to a language, and vastly increases not just the power and readability of the language but more important the speed and accuracy of making changes. This is the macro pre-processor.

I was already very familiar with macros when I first encountered the C language, because they form a key part of a good assembler language – to the extent that an assembler that has such a facility is usually called a macro-assembler. Macros enable you to define blocks of text including argument substitution. They resemble subroutine calls, but they are translated at compile time to code which is then compiled along with the hand-written code. A macro can do something simple like create a symbol for a constant that is widely used in a program, but which may have to be changed. When a change is needed, you just change the macro definition and – poof! – everywhere it’s used it has the new value. It can also do complex things that result in multiple lines of action statements and/or data definitions. It is the most powerful and extensible tool for expressing intention and enabling rapid, low-risk change in the programmer’s toolbox. While the C macro facility isn’t quite as powerful as the best macro-assemblers, it’s a zillion times better than not having one at all, like all the proud but pathetic modern languages that wouldn’t know a macro if it bit them in the shin.

The next 50 years of software language advances

The refrain of people who want to stay up to date with computers is, of course, "What's new?" Everyone knows that computers evolve more quickly than anything else in human existence, by a long shot. The first cell phones appeared not so long ago, and they were "just" cell phones. We all know that now they're astounding miniature computers complete with screens, finger and voice input, cameras and incredible storage and just about any app you can think of. So course we want to know what's new.

The trouble is that all that blindingly-fast progress is in the hardware. Software is just along for the ride! The software languages and statements that you write are 99% the same as in the long-ago times before smart phones. Of course there are different drivers and things you have to call, just as in any hardware environment. But the substance of the software and the lines of code you use to write it are nearly the same. Here is a review of the last 50 years of "advances" in software. Bottom line: it hasn't advanced!

Oh, an insider might argue: what about the huge advances of the object-oriented revolution? What about Google's new powerhouse, Go? Insiders may ooh and ahh, just like people at a fashion catwalk. The come-back is simple: what about programmer productivity? Claims to this effect are sometimes made, but more often its that the wonderful new language protects against stupid programmers making errors or something. There has not been so much as a single effort to measure increase of programmer productivity or quality. There is NO experimental data!! See this for more. Calling what programmers do "Computer Science" is a bad joke. It's anything but a science.

What this means is simple: everyone knows the answer -- claims about improvement would not withstand objective experiments -- and therefore the whole subject is shut down. If you're looking for a decades-old example of "cancel culture," this is it. Don't ask, don't tell.

Conclusion

3-GL's brought software programming to an astounding level of productivity. Using them you could write code quickly. The code you wrote came pretty close to expressing what you wanted the computer to do with minimal wasted effort. Given suitable compilers, the code could run on any machine. Using a 3-GL was at least 10X more productive than what came before for many applications.

A couple language features were incorporated into the very first languages, like conditional branching and controlled looping, that were good first steps. The next few years led early programmers to realize that a few more elaborations of conditional branching and controlled looping would handle the vast majority of practical cases. With those extra language features, code became highly expressive. All subsequent languages have incorporated these advances in some form. Sadly, the even more productive feature of macros has been abandoned, but as we'll see in future posts, their power can be harnessed to an even greater extent in the world of declarative metadata.

Software Components and Layers: Problems with Data

Components and layers are supposed to make software programs easier to create and maintain. They are supposed to make programmers more productive and reduce errors. It’s all lies and propaganda! In fact, the vast majority of software architectures based on components (ranging from tiny objects to large components) and/or layers (UI, server, storage and more) lead to massive duplication and added value-destroying work. The fact that these insane methods continue to be taught in nearly all academic environments and required by nearly all mainstream software development groups is cornerstone evidence that Computer Science not only isn’t a science, it resembles a deranged religious cult.

Components and Layers

It has been natural for a long time to describe the different “chunks” of software that work together to deliver results for users in dimensional terms.

First there’s the vertical dimension, the layers. Software that is “closer” to end users (real people) is thought of as being on “top,” while the closer software is to long-term data storage and farther from end users is thought of as being on the “bottom” of a software system. In software that has a user interface, the different depths are thought of as “layers,” with the user interface the top layer, the server code the middle layer and the storage code the bottom layer. Sometimes a system like this is called a “three tier” system, evolved from a two tier “client/server” system.

Second there’s the horizontal dimension, the components. These are bodies of code that are organized by some principle to break the code up into pieces for various reasons. I've given a detailed description of components. A component may have multiple layers in it or it may operate as a single layer.

Layers are often written in different languages and using different tools. Javascript is prominent in user interface layers, while some stored procedure language is often used in the lowest data access layer. The server-side layer may be written in any of dozens of languages, including java and python. Sometimes there are multiple layers in server-resident software, with a scripting language like PHP or javascript used for server code on the “top,” communicating with the client software.

Components are self-contained bodies of software that communicate with other components by some kind of formal means. Formal components always have data that can only be accessed by the code in the component. The smallest kind of component is an object, as in object-oriented languages. The data members of the object can only be accessed by routines attached to the object, known as methods.  Methods can normally be called by methods of other objects. Larger components are often called microservices or services. These are usually designed so that they could run on separate machines, with calling methods that can span machines, like old RPC’s or more modern RESTful API’s. Sometimes components are designed so that instead of being directly called, they take units of work from a queue or service bus, and send results out in the same way. When a component makes a call, it calls a specific component. When a component puts work onto a queue or bus, it has no knowledge or control over which component takes the work from the queue or bus.

Layer and components are often combined. For example, microservice components often have multiple layers, frequently including a storage layer. With a thorough object-oriented approach, there could be multiple objects inside each layer that makes up a service component.

Why are there layers and components? Software certainly doesn’t have to use these mechanisms to be effective. In fact, major systems and tools have been built and widely deployed that mostly ignore the concepts of layers and components! See this for historic examples.

The core arguments for layers and components typically revolve around limiting the amount of code (components) and the variety of technologies (layers) a programmer has to know about to make the job easier, along with minimizing errors and maximizing teamwork and productivity. I have analyzed these claims here and here.

Data Definitions in Components and Layers

All software consists of both procedures (instructions, actions) and data. Each of those needs to be defined in a program. When data is defined for a procedure to act upon, the data definitions are nearly always specific to and contained within the component and/or layer they’re part of. For the general concept and variations on how instructions and data relate to each other, see this.

Thinking about data that’s used in a component or layer, some of the data will be truly only for use by that component or layer. But nearly any application you can imagine has data that is used by many components and layers. This necessity is painful to object/component purists who go to great lengths to avoid it. But when a piece of data like “application date” is needed, it will nearly always have to be used in multiple layers: user interface, server and database. To be used it must be defined. So it will be defined in each layer, typically a minimum of three times!

When data is defined in software, it always has a name used by procedures to read or change it. It nearly always also has a data type, like character or integer. The way most languages define data that’s it! But there’s more.

• When the “application date” data is shown to the user in the UI layer, it typically also needs a label, some formatting information (month, day, year), error checking (was 32 entered into the day part?) and an error message to be displayed.
• When application date is used in the server layer by some component, some kind of error checking is often needed to protect against disaster if a mistaken value is sent by another component.
• When a field like social security number is used in the storage layer, sanity checks are often applied to make sure, for example, that the SSN entered matches the one previously stored for the particular customer already stored in the database.
• There need to be error codes produced if data is presented that is wrong. When the user makes an error, you can’t use a code, you have to use a readable message, which the user layer might need to look up based on the error code it gets from another component or layer.
• Each language has its own standards for defining data and the attributes associated with it. Someone has to make sure that all the definitions in varying languages match up, and that when changes are made they are made correctly everywhere.
• If the data is sent from one component to another, more definitions have to be made: the procedure that gets the data, puts it into a message and sends it, possibly on a queue; the procedure that gets the message, maybe from a queue and sends the data to the routine that will actually do the processing.
• When data is sent between components, various forms of error checking and error return processing must also be implemented for the data to protect against the problems caused by bad data being passed between components that, for example, might have been implemented and maintained by separate groups. Sometimes this is formalized into "contracts" between data-interchanging components/layers.

So what did we gain by breaking everything up into components and layers? A multiplication of redundant data definitions containing different information, expressed in different ways! What we “gained” by all those components and layers was a profusion of data definitions! The profusion is multiplied by the need for components and layers to pass data among themselves for processing.The profusion can’t be avoided and can only be reduced by introducing further complexity and overhead into the component and layer definitions.

See this for another take on this subject.

I’ve heard the argument that unifying data definitions makes things harder for the specialists that often dominate software organizations. The database specialists are guardians of the database, and make sure everything about it is handled in the right way. The user interface specialists keep the database specialists away from their protected domain, because if they meddled the users wouldn’t enjoy the high quality interfaces they’ve come to expect. There is no doubt you want people to know their stuff. But none of this is really that hard – channeling programmers into narrow specialties is one of the many things that leads to dysfunction. Programmers produce the best results by thoroughly understanding their partners and consumers, which can only be done by spending time working in different roles – for example spending time in sales, customer service and different sub-departments of the programming group.

Data Access in Components and Layers

Now we’ve got data defined in many components and layers. In a truly simple system, data would be defined exactly once, be read into memory and be accessed in the single location in which it resides by whatever code needs it for any reason. If the code needs to interact with the user, perform calculations or store it, the code would simply reference the piece of data in its globally accessible named memory location and have at it. Fast and simple.

If this concept sounds familiar, you may have heard of it in the world of relational DBMS’s. It’s the bedrock concept of having a “normalized” schema definition, in which each unique piece of data is stored in exactly one place. A database that isn’t normalized is asking for trouble, just like the way that customer name and address are frequently stored in different places in various pieces of enterprise software that evolved over time or were jammed together by corporate mergers.

Components and layers performing operations on global data is neither fast nor simple. Suppose for example that you’ve got a new financial transaction that you want to process against a customer’s account. In an object system, the customer account and financial transaction would be different objects. That’s OK, except that the customer account master probably has a field that is the sum of all the transactions that have taken place in the recent time period.

In a sensible system, adding the transaction amount to a customer transaction total in the customer record would probably be a single statement that referenced each piece of data (the transaction amount and the transactions total) directly by name. Simple, fast and understandable.

In a component/object system that’s properly separated, transaction processing might be handled in one component and account master maintenance in another. In that case, highly expensive and non-obvious remote component references would have to be made, or a copy of the transaction placed on an external queue. In an object system, a method of the transaction object would have to be called and then a method of the account master object.

It’s a good thing we’ve got smart, educated Computer Scientists arranging things so that programmers do things the right way and don’t make mistakes, isn’t it?

Conclusion

With the minor exception of temporary local variables, nearly every layer or component you break a body of software into leads to a multiplication of redundant, partly overlapping data definitions expressed in different languages -- each of which has to be 100% in agreement to avoid error. The communication by call or message between layers and components to send and receive data increases the multiplication of data definitions and references. Adding the discipline of error checking is a further multiplication.

Not only does each layer and component multiply the work and the chances of error, each simple-seeming change to a data definition results in a nightmare of redundancy. You have to find each and every place the data is defined, used or error-checked and make the right change in the right language. Components and layers are the enemy of Occamality. Software spends the VAST majority of its life being changed! Increasingly scattered data definitions make the thing that is done 99% of the time to software vastly harder and riskier to do.

The catastrophic effect of bad metaphors in software

Software is invisible. The vast majority of people who talk about it have little or no practical experience writing it. But software is incredibly important and people need to be able to talk about it: creating it, modifying it, installing and using it and making it work as intended. How do you talk about something that's invisible? Answer: you use real-world metaphors that everyone knows and talk in those terms. Given the importance of the process of creating and modifying software, you also use real-world metaphors for the processes, methods and techniques that people use for those things on software.

Given the high cost, long time periods and risky outcomes of software development, real-world metaphors are used as models for how to create what seem like the equivalent things in the invisible world. Software professionals use the same metaphors and think in the same terms as their non-programming counterparts, the more so as advancement in software normally correlates with doing less hands-on programming.

Before long, the metaphors turn into highly detailed, structured things embodied in software tools for software management, and regulations for how software must be built in order to be acceptable for acquisition by governments and major enterprises. The software in medical devices, for example, is highly regulated in this way, everything from requirements to testing and modifications.

These metaphors are great for communication among people involved in software in some way, but disastrous for building fast, efficient software, quickly and effectively. The metaphors are largely to blame for the rolling crisis/disaster of software that was first recognized over 50 years ago.

Small groups of smart, hard-working software nerds sometimes break out of metaphor prison and use techniques that naturally grow out of the experience of programmers who live in the land of software, which, though invisible to the vast majority of people, is vividly visible to them. They are usually simple ideas that come out of the writing real-world software that meets needs and evolves with experience. Top programmers often think of themselves as "lazy," meaning that they are aware of how they spend their time and find ways to avoid doing things that don't advance the cause. They visualize the world of software they're working on, always looking for ways to spend less time writing less code to get more stuff done without errors.

The methods that they use rarely conform to any of the established software methodologies and typically make non-software people uncomfortable. I've observed and described many of those techniques, which as a whole I term "wartime software."

In this post I will identify and summarize some of the widely accepted metaphors that continue to wreak havoc on software development

The factory metaphor

The metaphor of the software factory tends to float around. Wouldn’t it be nice if we could churn out software as reliably and predictably as a factory? We’d put the requirements and other goals and constraints into the factory, and after a series of careful, orderly processing steps, out would come a wonderful piece of software! Unfortunately, factories make copies of things that are already designed and tested. All the effort goes in to making the design of a new car or drug; sometimes the design is no good and has to be discarded. Once you’ve made test versions and made sure they operate the way you planned, you turn the design over to the factory for making lots of identical copies of it. See this for more.

The building metaphor

Buildings and roads are widely used as metaphors for software. Lots of the words are the same: we have requirements, architects, standards that must be met, common design patterns. We have project plans with resource requirements and target dates. We might put the project out to bid. We use specialists for each aspect of the work. We build in phases and have regular reviews by the owners and quality inspectors. At the end we have walk-throughs, which may result in punch lists of defects to be corrected. Then we have the closing and we’re off and running!

It’s a powerful metaphor. It’s better than the factory metaphor because each house is uniquely built by a team, just like each piece of software is uniquely built by a team. The trouble is that a house just sits there. It’s a passive physical object. Software is NOT a physical object; it’s mostly is a set of actions to be taken by a computer, along with some built-in data. See this for more detail.

There's another issue with the metaphor. Buildings are built and then mostly used. Modifications are sometimes made, but they are normally small compared to the effort put into the original building. Software, on the other hand, is mostly changed. Vastly more effort is typically put into changing a body of software over time than was spent with the original building. Most of what most employees of software departments do is make changes. See this for more detail.

The Architect metaphor

The architect metaphor is closely related to the building metaphor. It's just as bad for the same reasons. Here is a detailed description that uses architecture software to illustrate the difference.

Waterfall and Agile

The waterfall method is the universally derided classic method of applying project management to building software. After years of failure with increasingly rigorous and detailed project planning to improve software development outcomes, it was supposed that waterfall just wasn't flexible enough. It was too rigid. More agility to respond to emerging knowledge and conditions would solve the problems. Thus, Agile software development method was invented, purposely built on the metaphor of a nimble-footed engineering team going through the process with lots of re-calibration and obstacle avoidance.

Hah. Nothing has changed except the rhetoric. Waterfall is a method that fixes the requirements and figures out the time to build them. Agile is a method that fixes the time periods and figures out what can be built in each. The difference in practice is that Agile has more overhead, and no better results. See this for more.

Objects in software

Could there be a more inappropriate metaphor? A bundle of data definitions and code is an "object?" A thing? Sorry, programs are NOT things. They're not passive; programs are actions!

Object-orientation started as an obscure design pattern for creating simulation programs, grew into a fad and has evolved into a standard part of software orthodoxy. I remember long ago hearing the sales pitch for the early O-O languages like Smalltalk. The business sales pitch was that it would greatly enhance the speed and quality of software development, while enabling changes to be more easily accomplished. The tech pitch was that by creating "objects" consisting of a definition of a related set of data and by including in that definition all the code that would read and write the data, concerns would be isolated and mistakes avoided by having data manipulated only by "its own" code.

The common metaphor was that objects were like Lego blocks, lending themselves to easily build elaborate structures and making changes to them without collateral damage. By throwing in things like inheritance -- which was always illustrated by physical things like the hierarchy of species, families, etc. -- the Lego effect was even stronger.

As it turned out, O-O has been a giant step backwards for software. While appropriate in some situations, as a rigid structure with no exceptions it imposes difficult and unnatural constraints on how code and data can be related and arranged, making everything more complicated than it needs to be. Lego blocks are great as physical objects, but their software version is a failure.

The walls and defenses metaphor

This one is huge in cybersecurity. Everyone talks about putting up walls to protect our computers, establishing and maintaining an effective periphery. We talk about attackers who want to "penetrate" our security defenses.

A great deal of software security is built according to this metaphor. The fact that software is invisible to everyone who makes important decisions about computer security leads directly to the string of monstrous security disasters that continue to this day. Here is a good way using baseball to understand the impact of the invisibility. Here is a detailed description of the impact of invisibility.

Conclusion

We all think using metaphors. It's not inherently a bad thing. But when the metaphor has a strong real-world grounding that everyone understands, it can lead to disaster when applied to software in ways that don't match the correct understanding of the software itself. This is unfortunately the case for a great deal of software methods and processes, and helps explain the ongoing software disaster that was first recognized over 50 years ago.

 

The Dangerous Drive Towards the Goal of Software Components

Starting fairly early in the years of building software programs, some programs grew large. People found the size to be unwieldy. They looked for ways to organize the software and break it up into pieces to make it more manageable. This effort applied to the lines of code in the program and to the definitions of the data referred to by the code, and to how they were related. How do you handle blocks of data that many routines work with? How about ones whose use is very limited? The background of this issue is explained here.

These questions led to a variety of efforts that continue to this day to break a big body of software code and data definitions into pieces. The obvious approach, which continues to be used today, was simply to use files in directories. But for many people, this simple, practical approach wasn't enough. They wanted to make the walls between subsets of the code and data higher, broader and more rigid, creating what were generally called components. The various efforts to create components vary greatly. They usually create a terminology of their own, terms like “services” and “objects.” Such specialized, formulaic approaches to components are claimed to make software easier to organize, write, modify and debug. In this post I’ll take a stab at explaining the general concept of software components.

Components and kitchens

When you’re young and in your first apartment, you’ve got a kitchen that starts out empty. You’re not flush with cash, so you buy the minimum amount of cooking tools (pots, knives, etc.) and food that you need to get by. You may learn to make dishes that need additional tools, and you may move into a house with a larger kitchen that starts getting filled with the tools and ingredients you need to make your growing repertoire of dishes. You may have been haphazard about your things at the start, but as your collection grows you probably start to organize things. You may put the flour, sugar and rice on the same shelf, and probably put all the pots together. You may put ingredients that are frequently used together in the same place. You probably store all the spices and herbs together. It makes sense – if everything were scattered, you’d have a tough time remembering where each item was. The same logic applies to a home work bench and to clothes.

The same logic applies for the same reason to software! It’s a natural tendency to want to organize things in a way that makes sense – for remembering where they are, and for convenience of access. This natural urge was recognized in the early days of software programming. Given that there aren’t shelves or drawers in that invisible world, most people settled on the term “component” as the word for a related body of software. The idea was always that instead of one giant disorganized block of code, the thing would be divided into a set of components, each of which had software and data that was related.

A software program consists of a set of procedures (the actions) and a set of data definitions (what the procedures act on). Breaking up a large amount of code into related blocks was helped by the early emergence of the subroutine – a block of code that is called on to do something and then returns with a result; kind of like a mixer in a kitchen. The problems that emerged were how to break a large number of subroutines into components (each of which had multiple subroutines), and how to relate the various data definitions to the routine components. This problem resembles the one in the kitchen of organizing tools and ingredients. Do you keep all the tools separate from the ingredients, or do you store ingredients with the tools that are used to process them?

In the world of cooking, this has a simple answer. If all you do is make pancakes, you might store the flour and milk together with the mixing bowls and frying pans you use with them. But no one ever just makes pancakes – duh! And even if you did, you’d better put the milk and butter in the fridge! Ditto with spices. Nearly everyone has a spice drawer or shelf, and spices used for everything from baking to making curries are stored there.  Similarly, you store all the pans together, all the knives, etc.

In software it’s a little tougher, but not a lot. One tough subject is always do you group routines (and data) together by subject/business area or by technology area? The same choice applies to organizing programmers in a department. It makes sense for everyone who deals primarily with user interfaces to work together so they can create uniform results with minimal code. Same thing with back-end or database processing. But over time, the programmers become less responsive to the business and end users; the problem is often solved by having everyone who contributes to a business or kind of user work as a team. Responsiveness skyrockets! Before long, things begin to deteriorate on the tech side, with redundant, inconsistent data put into the database, UI elements that varying between subject areas, confusing users, etc. There’s pressure to go back to tech-centric organization. And so it goes, round and round. Answer: there is no perfect way to organize a group of programmers!. Just as painfully, there is no perfect way to organize a group of procedures and their relationship to the data they work on!!

The drive towards components

There are two major dimensions of making software into components. One dimension is putting routines into groups that are separate from each other. The second dimension is controlling which routines can access which data definitions.

Separating routines into groups can be done in a light-weight way based on convenience, similar to having different workspaces in a kitchen. In most applications there are routines that are pretty isolated from the rest and others that are more widely accessed. Enabling programmers to access any routine at any time and having access to all the source code makes things efficient.

Component-makers decided that programmers couldn't be trusted with such broad powers. They invented restrictions to keep routines strictly separate. While there were earlier versions of this idea, a couple decades ago "services" were created to hold strictly separate groups of routines. An evolved version of that concept is "micro-services." See this for an analysis.

The second major dimension of components is controlling and limiting the relationship between routines and data. The first step was taken very early. It was sensible and remains in near-universal use today: local variables. These are data definitions declared inside a routine for the private use of that routine during its operation. They are like items on a temporary worksheet, discarded when the results are created.

Later steps concerning "global" variables were less innocent. The idea was to strictly separate which routines can access which data definitions. Early implementations of this were light-weight and easily changed. Later versions built the separation into the architecture and code details. For example, each micro-service is ideally supposed to have its own private database and schema, inaccessible to the other services. This impacts how the code is designed and written, increasing the total amount of code, overhead and elapsed time.

Languages that are "object oriented" are an extreme version of the component idea. In O-O languages, each data definition ("class") can be accessed only by an associated set of routines ("methods"). This bizarre reversal of the natural relationship between code and data results in a wide variety of problems.

These ideas of components can be combined, making the overhead and restrictions even worse. People who build micro-services, for example, are likely to use O-O languages and methods to do the work. Obstacles on top of problems.

Components in the kitchen

All this software terminology can sound abstract, but the meaning and consequences can be understood using the kitchen comparison.

What components amount to is having the kitchen be broken up into separate little kitchenettes, each surrounded by windowless walls. There are two openings in each kitchenette's walls, one for inputs and one for outputs. No chef can see or hear any other chef. The only interaction permitted is by sending and receiving packages. Each package has an address of some kind, along with things the sending chef wants the receiving chef to work on. For example, the sending chef may mix together some dry ingredients and send them to another chef who adds liquid and then mixes or kneads the dough as requested. The mixing chef might then put the dough into another package and return it to the sending chef, who might put together another package and send it to the baking chef who controls the oven.

If the components are built as services, the little kitchenettes are built as free-standing buildings. In one version of components (direct RESTful calls), there is a messenger who stands waiting at each chef's output window. When the messenger receives a package, the messenger reads the address, jumps into his delivery van with the package and drives off to the local depot and drops off the package; another driver grabs the package, loads it into his van and delivers it to the receiver's input window. If the kitchenettes are all in the same location the vans and depot are still used -- the idea is that a genius administrator can change the location of the kitchenettes at any time and things will remain the same.

Another version of components is based around an Enterprise Service Bus (ESB). This is similar to the vans and the central office depot except that the packages are all queued up at the central location, which has lots of little storage areas. Instead of a package going right to a recipient, it's sent to one of these little central storage areas. Then, when the chef in a kitchenette is ready for more work he sends a requests to the central office, asking for the next package from a given little storage area. Then a worker grabs the oldest package, gives it to a driver who puts it in his van and delivers the package to the input window of the requesting chef.

If this sounds bizarre and lots of extra work, it's because ... it's bizarre and requires lots of extra work.

The ideal way to organize software

The ideal way to break software into components is actually pretty similar to the way good kitchens are organized. Generally speaking, you start by having the same kinds of things together. You probably store all the pots and pans together, sorted in a way that makes sense depending on how you use them. You probably pick a place that’s near the stove where they’ll probably be used – maybe even hanging on hooks from the ceiling. You probably have a main store of widely used ingredients like salt, but you may periodically put containers of it near where it is most often used. An important principle is that most chefs work at or near their stations – but (it goes without saying) can move anywhere to get anything they need. There aren't walls stopping you, and you don’t bother someone else to get something for you when you can more easily do it yourself.

Exactly the same principle applies whether you are creating an appetizer, an entree, a side dish or a dessert -- you do the same gathering and assembly from the ingredients in the kitchen, but deliver the results on different plates at different times.

In software this means that while routines may be stored in files in different directories for convenience, and that usually your work is confined to a single directory, you go wherever you need to in order to do your job. Same thing with data definitions; you can break them up if it seems to make things more organized, but any routine can access any data definition it needs to. When you’re done, you make a build of the system and try it out. That's what champion/challenger QA is for.

Are you deploying on a system that has separate UI, server and storage? No problem! That's what build scripts (which you would have anyway) are for! This approach makes it easier to migrate from the decades-old DBMS-centric systems design and move towards a document-centered database with auto-replication to a DBMS for reporting, which typically improves both performance and simplicity by many whole-number factors.

Are the chefs in your kitchen having trouble handling all the work in a timely way? In the software world you might think of making a "scalable architecture" with separate little kitchenettes, delivery vans, etc. -- which any sensible person knows adds trouble and work and makes every request take longer from receipt to ultimate delivery. In the world of kitchens (and sensible software people) you might add another mixer or two, install a couple more ovens and hire another chef or two, everyone communicating and working in parallel, and churning out more excellent food quickly, with low overhead.

If you think this sounds simple, you’re right. If you think it sounds simplistic, perhaps you should think about this: any of the artificial, elaborate, rigid prescriptions for organizing software necessarily involves lots of overhead for people and computers in designing, writing, deploying, testing and running software. Each restriction is like telling a chef that under no circumstances is he allowed to access this shelf of ingredients or use that tool when the need arises.

Arguments to the contrary are never backed with facts or real-world experiments – just theory and religious fervor. Instead, you should consider the effective methods of optimizing a body of software, which are centered on Occamality (elimination of redundancy) and increasing abstraction (increasingly using metadata instead of imperative code). Both of these things will directly address the evils that elaborate component methods are supposed to cure but don't.

Fundamental Concept: Software is not a passive thing, it is actions

Even lay people have heard that there are computer languages. They've heard that the languages process data -- they take data in and put it out. They gather there are commands in the language to do stuff to data. So of course software is actions.

In spite of this knowledge, something very different is in the forefront of peoples' thinking about software, both in the industry and not. This different way of thinking results from the fact that software is invisible to nearly everyone, and that in order to think and communicate about it, language is used that assumes software is a thing.

We talk about a body of software, the 2021 version of a piece of software, as though it were a car. Speaking of cars, we talk about how software crashes. We talk about how software has bugs, a phrase that originated from a bug found in a long-ago computer that had broken.

We talk about creating requirements for a new piece of software that someone will architect and the building may be put out to bid. The rest of the software building process sounds remarkably like building a house. We think of a time when the building project is done and we can start using the house.

The trouble is that a house just sits there. It’s a passive physical object. Software is NOT a physical object; it’s mostly is a set of actions to be taken by a computer, along with some built-in data. A piece of software is like a data factory – it takes data in, makes all sorts of changes, sometimes puts some of it in a storage room, and spits out data in all sorts of volumes and formats. A software program responds to each piece of data it gets, e.g. mouse click or data file; software is an actor, while a house is a thing that does nothing. Designing a piece of software is more like writing a play for an improv theater – instead of having a fixed script, the play can go in all sorts of directions depending on the input it gets from the audience. The improv theater script even has to tell the actor that when an audience member uses a swear word, the actor has to respond saying that those words aren’t permitted, would he like to try again. This is NOTHING like designing a house! Here is more detail.

Suppose you use an advanced architecture program and design a house in great detail. Modern programs go down to the level of the framing, the details of the steps, doors and windows, literally every object that will be part of the eventual house. When you're done with such a design, you can give it to the building department for approval and to a construction company to build. If they follow the instructions correctly, you'll get exactly what you saw in the 3-D renderings and the live-action walk-throughs. Here's the killer: the equivalent of such a detailed design for computer software is ... a computer program! Or at least enough detail in pseudo-code, data definitions and screen designs so that coding is fairly mechanical. A detailed design for a computer program is like the results of extensive interactions by the script-writing team for a new episode of a series TV show, with recordings and lots of notes. Chances are a junior script-writer who has been intensely following everything with some participation is assigned the job of producing a clean draft of the script, ready for refinement and editing. That is the program, ready for actors to give it a studio reading, like you would run a test version of the software for the developers.

Software isn't a thing, regardless of how often we talk about it as though it were. It isn't a thing, even though we nearly always use methods like project management developed for building things for organizing its creation. Software is a collection of actions, like an incredibly detailed script for improvisational theater with loads of conditional branching and the ability to bring things up that were talked about before in creative and useful ways.

This matters because how you think influences what you do. If you're thinking about running a marathon while you're climbing a mountain, chances are you'll slip and fall. If you're thinking about baseball plays while you're trying to choreograph a ballet, I suspect you won't win any awards. But at least those are thinking about actions. If you're thinking about the design of a book while you're writing a comedy, I doubt you'll get many laughs.

This fundamental concept is a core aspect of all of programming. The fact that people so often have the wrong things in their heads while they are dealing with software has profound effects.

And ... just as bad, we mostly don't "build" software -- mostly we change it!

The Relationship between Data and Instructions in Software

The relationship between data and instructions is one of those bedrock concepts in software that is somehow never explicitly stated or discussed. While every computer program has instructions (organized as routines or subroutines) and data, the details of how the data is identified, named and accessed vary among programming languages and software architectures. Those differences of detail have major consequences. That’s why understanding the underlying principles is so important.

Programmers argue passionately about the supposed virtues or defects of various  software architectural approaches and languages, but do so largely without reference to the underlying concepts. Only by understanding the basic concepts of data/instruction relationships can you understand the consequences of the differences.

The professional kitchen

One way to understand the relation between instructions (actions) and data is to compare it to something we can all visualize. An appropriate comparison with a program is a professional chef’s kitchen with working cooks. But in this kitchen, the cooks are a bit odd -- only one of them is active at a time. When someone asks them to do something they get active, each performing its own specialty. A cook may ask another cook for help, giving the cook things or directing them to places, and getting the results back. The cooks are like subroutines in that they get called on to do things, often with specific instructions like “medium rare.” The cook processes the “call” by moving around the kitchen to fetch ingredients and tools, brings them to a work area to process the ingredients (data) with the tools, and then delivers the results for further work or to the server who put in the order. The ingredients (data) can be in long-term storage, working storage available to multiple chefs or in a workspace undergoing prep or cooking. In addition, food (data) is passed to chefs and returned by them.

The action starts when a server gives an order to the kitchen for processing.

• This is like calling a program or a subroutine. Subroutines take data as calling parameters, which are like the items from the menu written on the order to the kitchen.

The person who receives the order breaks the work into pieces, giving the pieces to different specialists, each of whom does his work and returns the results. Unlike in a kitchen, only one cook is active at a time. One order might go to the meat chef and another to whoever handles vegetables.

• This is like the first subroutine calling other subroutines, giving each one the specifics of the data it is supposed to process. The meat subroutine would be told the kind and cut of meat, the finish, etc.

In a professional kitchen there is lots of pre-processing done before any order is taken. Chefs go to storage areas and bring back ingredients to their work areas. They may prepare sauces or dough so that everything is mixed in and prepped so that it can be finished in a short amount of time. They put the results of their work in nearby shelves or buckets for easy access later in the shift.

• This is like getting data from storage, processing it and putting the results in what is called static or working storage, which is accessible by many different subroutines.

There is a storage area and refrigerator that stores meat and another that stores vegetables. The vegetable area might have shelves and bins. The cook goes to the storage area and brings the required ingredients back to the cook’s work space. Depending on the recipe, the cook may also fetch some of the partly prepared things like sauces, often prepared by others, to include.

• This is like getting data from long-term storage and from working storage and bringing it to automatic or local variables just for this piece of work.

The storage area could be nearby. It could be a closet with shelves containing big boxes that have jars and containers in it. A cook is in charge of keeping the pantry full. They go off and get needed ingredients and put them in the appropriate storage area as needed. They could also deliver them as requested right to a chef.

• This is like having long-term storage and access to it completely integrated with the language, or having it be a separate service that needs to be called in a special way.

The chef does the work on the ingredients to prepare the result.

• This is like performing manipulations on the data that is in local variables until the desired result has been produced. In the course of this, a chef may need to reach out and grab some ingredient from a nearby shelf.

The chef may need extra space for a large project. He grabs some empty shelves from the storage area and uses them to store things that are in progress, like dough that needs time to rise. later a chef might call out “grab me the next piece of dough” or “I need the dough on the right end of the third shelf.

• This is like taking empty space and using it. Pointers are sometimes used to reference the data, or object ID’s in O-O systems.

The cook delivers the result for plating and delivery.

• This is like producing a return variable. It may also involve writing data to long-term storage or working storage.

I’m not a cooking professional, but I gather that the work in professional kitchens and how they’re organized has evolved towards producing the best results in the least amount of elapsed time and total effort. As much prep work as done before orders are received to minimize the time and work to deliver orders quickly and well. The chefs have organized work and storage spaces to handle original ingredients (meat, spices, flour, etc.) and partly done results (for example, a restaurant can’t wait the 45 minutes it might take to cook brown rice from scratch).

In the next section, this is all described again somewhat more technically. If you’re interested in technology or have a programming background, by all means read it. The main points of this post and the ones that follow can be understood without it.

Instructions and data in a computer program

The essence of a computer program is instructions that the computer executes. Most of the instructions reference data in some way – getting data, manipulating it, storing results. See this for more.

In math, from algebra on up, variables simply appear in equations. In computer software, every variable that appears in a statement must be defined as part of the program. For example, a simple statement like

          X = Y+1

Means “read the value stored in the location whose name is Y, add the number 1 to it, and store the result in the location whose name is X.” Given this meaning, X and Y need to be defined. How and where does this happen? There are several main options:

• Parameters. These form part of the definition of a subroutine. When calling a subroutine, you include the variables you want the subroutine to process. These are each named.
• Return value. In many languages, a called routine can return a value, which is defined as part of the subroutine.
• Automatic or local variables. These are normally defined at the start of a subroutine definition. They are created when the subroutine starts, used by statements of the subroutine and discarded when the subroutine exits.
• Static or working storage variables. These are normally defined separately (outside of) subroutines. They are assigned storage at the start of the whole program (which may have many subroutines), and discarded at the end.
• Allocated variables. Memory for these is allocated by a subroutine call in the course of executing a program. Many instances of such allocated variables may be created, each distinguished by an ID or memory pointer.
• File, database or persisting variables. These are variables that exist independent of any program. They are typically stored in a file system or DBMS. Some software languages support these definitions being included as part of a program, while others do not. See this for more.

There are a couple concepts that apply to many of the places and ways variables can be defined.

• Grouping. Groups of variables can be in an ordered list, sometimes with a nesting hierarchy. This is like the classic Hollerith card: you would have a high-level definition for the whole card and then a list of the variables that would appear on the card.
• There might be subgroups; for example, start-date could be the name of a group consisting of the variables day, month, year.
• Referring to such a variable might look like “year IN start-date IN cust-record” in COBOL, while in other languages it might be year.start-date.cust-record.
• Multiples. Any variable or group can be declared to be an array, for example the variable DAY could be made an array of 365, so there’s one value per day of a year.
• Types or templates. Many languages let you define a template or type for an individual variable or group. When you define a new variable with a new name like Y, you could say it’s a variable of type X, which then uses the attributes of X to define Y.
• Definition scope. Parameters, return values and local variables are always tied to the subroutine of which they are a part. They are “invisible” outside the subroutine. The other variables, depending on the language, may be made “visible” to some or all of a program’s subroutines. Exactly how widely visible data definitions are is the subject of huge dispute, and is at the core of things like components, services and layers.
  

When you look at a statement like X = Y+1, exactly how and where X and Y are defined isn’t mentioned. X could be a parameter, a local variable or defined outside of the subroutine in which the statement appears. Part of the job of the programmer is to name and organize the data definitions in a clean and sensible way.

The variety of data-instruction organization and relationships

Most of the possibilities for defining variables listed above were provided for by the early languages FORTRAN, COBOL and C, each of which remains in widespread use. Not long after these languages were established, variations were introduced. Software languages and architectures were created that selected and arranged the way instructions related to data definition. Programmers and academics decided that some ways of referencing and organizing data were error-prone and introduced restrictions that were intended to reduce the number of errors that programmers made when creating programs. In software architecture, the idea arose that all of a program's subroutines should be organized into separate groups, usually called "components" or "services," each with its own collection of data definitions. The different components call on each other or send messages to ask for help and get results, but can only directly operate on data that is defined as part of the component.

The most extreme variation of instruction/data relationship is a complete reversal of point of view. The view I've described here is "procedural," which means everything is centered around the actor, the chef who does things. The reversal of that point of view is "object-oriented," called O-O, which organizes everything around the data, the acted upon, the ingredients and workspaces in a kitchen. Instead of following the chef around as he gets and operates on ingredients (data), we look at the data, called objects, each of which has little actors assigned to it, mini-chefs, that can send messages for help, but can only work on their own little part of the world. It's hard to imagine!

The basic idea is simple: instead of having a master chef or ones with broad specialties like desserts, there are a host of mini-chefs called "methods," each of  which can only work on a specific small group of ingredients. A master chef has to know so much -- he might make a mistake! By having a mini-chef who is 100% dedicated to dough, and never letting anyone else create the dough, we can protect against bad chefs (programmers) and make sure the dough is always perfect! Hooray! Or at least that's the theory...

Conclusion

Computers take data in, process it and write data out. Inside the computer there are instructions and data. Software languages have evolved to make it easier for programmers to define the data that is read and created and to make it easier to write the lines of code that refer to the data and manipulate it. As bodies of software have grown, people have created ways to organize the data that a computer works on, for example putting definitions for the in-process data of a subroutine inside the subroutine itself or collecting a group of related subroutines into a self-contained group or component with data that only it can work on.

Understanding the basic concepts of instruction/data relationships and how those relationships can be organized and controlled is the key to understanding the plethora of approaches to language and architecture that have been created, and making informed decisions about which language and architecture is best for a given problem. The overall trend is clear: Programming self-declared elites decide that this or that restriction should be placed on which variables can be accessed by which instructions in which way, with the goal of reducing the errors made by normal programming riff-raff. Nearly all such restrictions make things worse!

Fundamental Concept: Software is rarely Built, mostly it is Changed

Nearly everything we hear about the construction of software is that it is built. This is a fundamental misconception that pervades academia and practice. This misconception is a major underlying cause of why software has so many problems getting created in a timely way and functioning with acceptable speed and quality.

The software-is-built misconception is woven into every aspect of software education and practice. You are first taught how to write software programs; all the exercises start with a blank screen or sheet of paper. You’re taught how to test and run them. In software engineering you’re taught how to perform project management, from requirements to deployment. You may learn how to automate software quality testing. Everything about these processes assumes that software is built, like a house or a car.

The idea that software is built isn’t wrong – it just applies to a tiny fraction of the total effort that goes in to a body of software from birth to its final resting place in the great bit-bucket in the sky. The vast majority of the effort goes into changing software – making it do something different than the version you’re working on does. Maybe you’re fixing a bug, maybe you’re adding a feature or adapting to a new environment. Sometimes the changes are extensive, like when you’re changing to a new UI framework. For a typical  body of software, however much effort is put into the first production version, a minimum of 10X and often over 100X of the work goes into changing it.

We don't think of software as ever-changing, ever-growing partly because we think in terms of physical metaphors. We use the same project management techniques that are used for houses that we use for software. While sometimes houses are changed, mostly they're just built and then used for years. It's an extremely rare house that is endlessly extended the way software is. See this for more.

When it comes to enterprise software, the fraction of effort that goes into change sky rockets even more. A good example is the kind of software that runs hospitals and health systems. The software already exists, and it’s already had 100X the effort put into change than the original building. For example, standard software from the leading industry supplier was installed in the NYC hospital system. The original contract -- just to buy and install existing production software -- was for over $300M and 4 years! A couple years later the carefully planned project cost more than doubled. All because of changes. Here are details.

Why does thinking software is built rather than changed matter?

The idea that software is about building and not changing is responsible for most of the processes, procedures and methods that are overwhelmingly common practice either by custom or by regulation. The mismatch between the practices and the fact that software is nearly always changed instead of being created in the first place results in the never-ending nightmare of software disasters that plague us. The never-ending flow of new tools, languages and methods are futile attempts to solve these problems. No one would find the new things attractive if there weren’t a problem to fixed – a problem everyone knows and that no one wants to explicitly discuss!

Example: Project Management, Architecture, Design and Coding

Mostly what happens when you get a job in software is that you join a group that works with a substantial existing body of code. You may be a project manager, a designer, a customizer, any number of things, all of which revolve around the body of code.

Yes, you have to start any project from an understanding of what you want to accomplish. Here's the thing: what you want to get done is in the context of the existing code. What this means is that you have to understand as much of the code as possible to do things. If what you want to do is fairly circumscribed in the existing code, you may get away with ignoring the rest -- which is where comprehensive, nothing-left-out regression testing comes in, which means better make darn sure you didn't break anything.

In this context, lots of skills are relevant. But the overwhelming number one thing is ... knowledge of the ins and outs of the existing code base!

I've worked in a couple places where there were large code bases of this kind. Forget about departmental hierarchy -- the go-to person in every case of doing anything important was the person who best knew that particular aspect or section of the code.

In this context, what's the point of all the fancy project management stuff people think is the cornerstone of effective, predictable work? When you're traveling through an unknown land with lots of mountains, jungles, rivers and swamps, what do you want above all else (besides nice paved roads that don't exist)? You want an experienced GUIDE, someone who's been all over the place and knows the best way to get there from here.

Kind of turns all the project management stuff on its head, doesn't it?

Example: Software QA

Most software QA revolves around writing test scripts of one kind or another that mirror the new code you write: if the code does X, the test should assure that the code does X correctly. This is true whether you're doing test-driven development or whether QA people write scripts. As you build the code, you're supposed to build the test code. When you make a new release, you run integration tests that include running all the individual test scripts.

Everyone with experience knows the problems: there are never enough tests, they don't cover all the conditions, there are omissions and errors in the tests, with the result that new releases put into production often have problems.

The root cause of this is the never-spoken assumption that software is about building, when the reality is that it's mostly changed. What this means is that the focus should be on the existing code and what it does -- do the changes that you made break anything that used to work? When you make changes, you should be thinking champion/challenger like in data science: the new code is the challenger: did it make things better, and did it (probably unintentionally) make anything worse? The best way to do this is to run the same data through the production program and the challenger program in parallel, and have a tool that compares the output. Looking at the output will tell you what you want to know. Even better, running the new code in live-parallel-production, with lots of real customer data going through both versions but only the current champion output going to the customer provides a safe way to see if the customers are OK, while testing out the new functionality on the champion code only to make sure it does what you wanted.

Who needs test scripts, TDD and the rest? They're backwards! See this for more.

Conclusion

Software is literally invisible to the vast majority of people. Therefore, we all talk and think about software in terms of metaphors. That would be OK if the metaphors matched reality; unfortunately, the metaphors don’t match reality, and play a major role in software dysfunction. In real life, we build houses, bridges, and other structures -- we mostly don't change them.

So what are the methods that are appropriate when you recognize that the main thing you do with software is change it instead of building it? I've done my best to abstract the methods I've seen the best people use, calling them (for want of a better term) war-time software. Instead of creating a whole adult human at once, like Frankenstein, it's like starting with a baby that can't do much yet, but is alive. The software effort is one of growing the baby instead of constructing a Frankenstein.

 

The Crisis in Software is over 50 years old

Everyone knows there’s a problem in software. No one likes to talk about it. Waves of new tools and techniques are introduced, hyped and fade away. Are there waves of new tools and techniques in accounting? During interviews, are accountants asked if they practice the currently “hot” methods? Of course not! What about cars? Hah, you might say – look at the wave of new electric cars! Yes, let’s look at them – do electric cars crash more than normal ones? If cars crashed or mis-performed at the same rate as software, we would all be afraid to ride in them!

The existence of a software crisis was first prominently identified in 1968. The results of the crisis were clear at that time, and haven’t changed a bit since then. Nothing – no wonderful new language, paradigm, project management or quality method, architectural technique or anything else has made a dent in the crisis.

The solutions to the crisis have largely been identified and repeatedly proven in practice by small groups of programmers who desperately need to get things done. Their methods are ignored or suppressed by the ruling elites in academia, government and corporations – including Big Tech. Sadly, this is unlikely to change any time soon. Happily, effective methods for building software that are ignored and suppressed provide a way for small, ambitious groups to do new stuff and win!

What is the software crisis?

The term “software crisis” was coined during the 1968 NATO Software Engineering Conference. Many computer science experts attended. There was general agreement that there was a serious problem. Part of the solution was identified by creating a discipline of “computer engineering,” which would structure and formalize the techniques for building software. Papers identifying the problems and solutions were generated as a result of the conference and a similar one held the following year.

The problems are summarized in the Wiki article:

Look familiar?

How do we know there’s a crisis?

When things are really bad, it’s typical for people to avoid talking about it. When there’s a whiff of a solution, though, it’s likely to receive attention. When people engage in something that’s likely to go wrong, most of them will get the best advice and be sure to follow widely accepted practices in order to avoid trouble. If something goes wrong anyway, having followed the standard accepted techniques provides a good defense – I did what people in my position are supposed to do! When hiring people, the hiring organization will often trumpet their use of these authorized and accepted methods to assure avoidance of blame. And finally, when things go wrong, great care is taken to assure that as few people know about the problem as possible.

When you hire someone in physics, do you advertise that the position requires knowledge and acceptance of relativity theory? When you hire a doctor, do you need to make sure that the doctor adheres to the infectious theory of disease?

We know there’s a crisis because, in spite of all the effort to keep it quiet, disasters happen that are so public, widespread and annoying that they get talked about. The trouble is that the highly publicized disasters are just the visible tip of a truly gigantic iceberg that takes up most of the space in the ocean of software.

We know there's a crisis because the academic field devoted to the subject, Computer Science, isn't a "science." Not even close. Check out these.

We know there's a crisis because the famous big tech companies have amazing reputations, like they're all geniuses -- while the facts show that they're pretentious bumblers with astounding salaries.

We know there's a problem because of the fashion-driven nature of the field. Here is a discussion of specific fashions, and see these for examples.

Conclusion

Look again at the list of problem identified over 50 years ago: software that's late, over budget, not what you wanted, poor quality, and sometimes has to be thrown out. This is one of the reasons managers take estimates from programmers and multiply the time by 3X before passing them on up the chain. Higher managers pad even more. People try to avoid publicizing the wonderful thing that's coming real soon because all too often, days before completion, it blows up.

There have been decades of widely-touted solutions to the problem that never solve the problem. Here is a review of 50 years of "progress" in programming languages, for example.

There are solutions to this 50 year problem. Here is one of those solutions. Small groups of programmers, pressured to produce great results in short period of time, re-invent the solutions. There are underlying errors of thought that are the primary factors behind the problem and its incredible persistence. How else can you understand the near-universal resistance to winning methods-- proven in practice! -- to be ignored?

 

Software programming languages: the Declarative Core of Functional Languages

In a prior post I described the long-standing impulse towards creating functional languages in software. Functional languages have been near the leading edge of software fashion since the beginning, while perpetually failing to enter the mainstream. There is nonetheless an insight at the core of functional languages which is highly valuable and probably has played a role in their continuing attractiveness to leading thinkers. When that insight is applied in the right situations in a good way it leads to tremendous practical and business value, and in fact defines a path of advancement for bodies of software written in traditional ways.

This is one of those highly abstract, abstruse issues that seems far removed from practical values. While the subject is indeed abstract and abstruse, the practical implications are far-reaching and result in huge software and business value when intelligently applied.

Declarative and Imperative

A computer program is imperative. It consists of a series of machine language instructions that are executed, one after the other, by the computer's CPU (central processing unit). Each Instruction performs some little action. It may move a piece of data, perform a calculation, compare two pieces of data, jump to another instruction, etc. You can easily imagine yourself doing it. Pick up what's in front of you, take a step forward, if the number over there is greater than zero, jump to this other location, etc. It's tedious! But every program you write in any language ends up being implemented by imperative instructions of this kind. It's how computers work. Period.

Computers operate on data.  Data is passive. Data can be created by a program, after which it's put someplace. The locations where data is held/stored are themselves passive; those locations are declared (named and described) as part of creating the imperative part of a computer program.

Data is WHAT you've got; instructions are HOW you act. WHAT is declarative; HOW is imperative. WHAT is like a map; HOW is like an ordered set of directions for getting from point A to point B on a map.

The push towards declarative

From early in the use of computers, some people saw the incredible detail involved in spelling out the set of exacting instructions required to get the computer to do something. A single bit in the wrong position can cause a computer program to fail, or worse, arrive at the wrong results. This detailed directions approach to programming is wired into how computers work. Is there a better way?

There is in fact no avoiding the imperative nature of the computer's CPU. As high level languages began to be invented that freed programmers from the tedious, error-prone  detail of programming in machine language, some people began to wonder if there were a way to write a low-level program, a program that of necessity would be imperative, that somehow enabled programs to be created in some higher level language that were declarative in nature.

Some people who were involved with early computers, nearly all with a strong background in math, proceeded to create the declarative class of programming languages, the most distinctive members of which are functional programming languages as I described in an earlier post.

The ongoing attempt to create functional languages and use them to solve the same problems for which imperative languages are used has proven to be a fruitless effort. But there are specific problem areas for which a declarative approach is well-suited and yields terrific, practical results -- so long as the declarative approach is implemented by a program written in an imperative language, creating a workbench style of system for the declarations. The current fashion of "low code" and "no code" environments is an attempt to move in that direction. But I'd like to note that there's nothing new in those movements; they're just new names for things that have been done for decades.

The Declarative approach wins: SQL

DBMS's are ubiquitous. By far the dominant language for DBMS is SQL. Data is defined in a relational DBMS by a declarative schema, defined in DDL (data definition language). Data is operated on by a few different kinds of statements such as SELECT and INSERT.

SELECT certainly sounds like a normal imperative keyword in any language, like COBOL's COMPUTE statement. But it's not. It's declarative. A SELECT statement defines WHAT data you want to select from a particular database, but says nothing about HOW to get it.

This is one of the cornerstones of value in a relational DBMS system. A SELECT statement can be complicated, referencing columns in multiple rows of various tables joined in various ways. The process of getting the selected data from the database can be tricky. Without query optimization, a key aspect of a DBMS, a query could take thousands of times longer than a modern DBMS that implements query optimization will take. Furthermore, table definitions can be altered and augmented, and so long as the data referenced in the query still exists, the SELECT statement will continue to do its job.

If all you're doing is grabbing a row from a table (like a record from an indexed file), SQL is nothing but a bunch of overhead, and you'd be better off with simple 3-GL Read statements. But the second things get complicated, your program will require many fewer lines of complex code while being easier to write and maintain if you have SQL at your disposal. A win for the declarative approach to data access, which is why, decades after it was created, SQL is in the mainstream.

The Declarative approach wins: Excel

I don't know many programmers who use Excel. Too bad for them; it's a really useful tool for many purposes, as its widespread continuing use makes clear.

Excel is a normal executable program written in an imperative language, but it implements a declarative approach to working with data. Studying Excel is a good way to understand and appreciate the paradigm.

An Excel worksheet is two dimensional matrix of values (cells). A cell can be empty or have a value of any kind (text, number, currency, etc.) entered. What's important in this context is that you can put a formula into a cell that defines its value. The formula can reference other cells, individually or by range, absolutely or relatively. A simple formula could be the sum of the values in the cells above the cell with the formula. It could be arbitrarily complex. It can have conditionals (if-then-else). Going beyond formulas, you can turn ranges of cells into a wide variety of graphs.

If you're not familiar with them, you should look at Pivot tables, and when you've absorbed them, move on to the optimization libraries that are built in, with even better ones available from third parties. Pivot tables enable you define complex summaries and extractions of ranges of cells. For example, I have an Excel worksheet in which I list each day of the year and the place where I am that day. A simple Pivot table gives me the total of days spent in each location for the year, something that simple formulas could not compute.

The key thing here is that even though there are computations, tests and complex operations taking place, it's all done declaratively. There is no ordering or flow of control. If your spreadsheet is simple, Excel updates sums (for example) when you enter or change a value in a cell. For more complex ones, you just click re-calc, and Excel figures out all the formula dependencies and evaluates them in the right order. This makes Excel a quicker way to get results than programming in any imperative language, assuming the problem you have fits the Excel paradigm.

The Declarative approach wins: React.js

One of the most widely-used frameworks for building UI's is React.js. Last time I looked, the header page of react.js included this:

It says right out that it's declarative! And that makes it easy! I have found a number of places (example, example) that have nice explanations of how it works and why it's good.

The Declarative approach wins: Compilers

The best computer-related course I took in college by far was a graduate course on the theory and construction of compilers. The approach I was taught all those decades ago remains the main method for compiler construction. First you have a lexical parser to turn the text of the program into a string of tokens. Then you have a grammar parser to turn the tokens into a structured graph of semantic objects. Then you have a code generator to turn the objects into an executable program.

The key insight is that each stage in this process is driven by a rich collection of meta-data. The meta-data contains all the details of the input language and its grammar and the output target. A good compiler is really a compiler-compiler, i.e., the imperative code that reads and acts on the lexical definitions, the grammar and the code generation tables. The beauty is that you write the compiler-compiler just once. If you want it to work on a new language, you give it the grammar of the language without changing any of the imperative code in the compiler-compiler! If you want to generate code for a new computer for a language you've already got, all you do is update the code generation tables! Once you've got such a compiler, you can write the compiler in its own language, at the start "boot-strapping" it.

While they didn't exist at the time I took the course, tools to perform these functions, YACC (Yet Another Compiler Compiler) and LEX, were built at Bell Labs by one of the groups of pioneers who created Unix and the C language. I didn't have those tools but used the concepts to build the FORTRAN compiler I wrote in 1973 while at my first post-college job. Since the only tool I had to use was assembler language, taking the meta-data, compiler-compiler approach saved me huge amounts and time and effort compared to hard-coding the compiler without meta-data.

This is meta-data at its best.

The Declarative spectrum

Excel and SQL are 100% all-in on the declarative approach. But it turns out that it doesn't have to be all one way or the other. If you're attacking an appropriate problem domain, you can start with just programming it imperatively, and then as you understand the domain, introduce an increasing amount of declaration into your program, by defining and using increasing amounts of meta-data. This is exactly what I have described as climbing up the tree of abstraction. Each piece of declarative meta-data you introduce reduces the size of the imperative program, and moves information that is likely to be changed or customized into bug-free, easily-changed meta-data.

Conclusion

Functional languages as a total alternative to imperative languages will perpetually be attractive to some programmers, particularly those of a math theory bent. Except in highly limited situations, it won't happen. No one is going to write an operating system in a functional language.

Nonetheless, the declarative approach with its emphasis on declaring facts, attributes and relationships is the powerful core of the functional approach, and can bring simplicity and power to otherwise impossibly complicated imperative programs.

The Revolutionary Champion/Challenger Method of Software QA

Having issues with software quality? Have you tried test-driven development? Do you have test scripts based on test requirements? Do you have a rich sandbox with good test data? What’s your code coverage? How often is a fully tested version released into production with problems that were missed? How often does the new feature work adequately in production but with destructive side effects that integration testing missed? How long and expensive is the pipeline from programmer’s release to working production? Are you proud of being on the forefront of development methods with Agile, Scrum and the rest but still avoid releasing what comes out of each Sprint to production because you can’t risk another disaster?

If any of these apply to you, you may want to consider a decades-old, widely proven in production method of software QA that enables frequent, error-free releases to production with almost no overhead or traditional QA work. It’s not taught, there are no certifications and It’s ignored by mainstream software experts. But it works.

I used to call it “comparison-based QA.” The term is appropriate because the core of the method is comparing results of the production version of the code with a test version. A CTO with a data science background suggested a better term, which I will use henceforth: champion/challenger software QA. It’s like when you’ve got a good model, the champion, and you want to see if a new model, the challenger, yields better results. Or it’s like A::B testing of a consumer UI, when you want to see if a proposed variation works better. In both cases, you’ve done something new and you want to do two things: (1) see if the new thing works like it should, and (2) make sure nothing that used to work is broken. Sounds like feature testing and regression testing, doesn’t it? Yup!

Nobody writes test requirements or feature tests for champion/challenger. You just feed the new thing the same data you fed the old thing and compare the results. You expect differences. If something is better and nothing is worse, you could have a winner – you test with a wider range of data, and if the results hold up, the challenger becomes the new champion and you move on.

Here are the main beauties of champion/challenger:

• The tests are nothing but inputs and saved outputs of real-world processing. Nothing artificial, no carefully crafted "test data."
• You create the code for comparison of output data once. The code/model could get endlessly complex, but the output comparison identifies differences whether it has 10KB or 10GB to compare. No test scripts!
  
  • You can do the comparison with a UI with some extra work, but again one-and-done.
• The comparison pulls out all the differences. You look to see if each difference is something you expected: if everything new you expected is there, and crucially if there are any unexpected differences. If there are, you’ve got a regression failure to fix.
• You can do the comparison in batch with large samples of old data, giving you great coverage.
  
• You can do the comparison live, in a production environment, using the challenger code/model output for comparison and sending only the champion’s output to the user.
  • You can test the changes you've made on the challenger code alone, looking at the challenger code's output, to see if you're happy with the change you made. While you're doing that, the challenger code will simultaneously be running production data, which the comparison with the champion will make sure is still OK.
    
  • After you’ve done enough live parallel production testing, you can be confident that the challenger will do everything the champion did – with real-world data in the production environment -- so you merely turn the switch so that challenger output becomes live and the old champion is retired.
  • You can do this in stages to crank the risk way down, so that the challenger becomes champion for just 10% of the inputs and then gradually turn up the fraction. This guarantees problem-free production releases.

I've written about about the details of how to get this done in a book.

I've written a post explaining the core concepts of the two main aspects of QA and how they're different and best served by different methods.

I've written about the path that led me to writing this and related books and summaries of the books.

I've written about how this method provides crucial fuel enabling startups to surpass tech groups hundreds of times their size along with specific strategies those groups use here and here.

The method is a "secret" -- because the vast majority of industry elites choose to ignore it. Want to succeed against industry giants? Use secret methods like champion/challenger to enable rapid release of quickly evolving code with zero errors in production and minimal QA effort. You'll run circles around the lumbering, stumbling giants and then leave them in your dust.

The IRS Could Have Prevented the Tax Data Leak

If the IRS had wanted to prevent the leak of tax returns recently reported by Propublica, they could have done it. The methods are simple, effective and in use. They just didn’t implement leak prevention methods. Why? The problem isn’t money; the IRS spends billions of dollars a year on computer systems. Will this embarrassment get them to fix things? I’ve read through the “IRS Integrated Modernization Business Plan,” the April 2019 document that describes how the IRS will spend many billions over the next 5 years to “modernize” their computer systems, and nowhere in the document is there a hint that they’ll do anything but spend more money to implement more of the ineffective security systems they already have.

The IRS doesn’t create or invent cybersecurity methods; they try to adhere to all the security regulations, follow the standards and take the advice of agencies that specialize in cybersecurity. These other agencies employ top experts who set the standards that institutions follow to protect their computer systems and confidential data. So what’s going on here? Did the IRS suffer the tax data leak because they failed to implement one of these clear standards? Or is there something missing or wrong with the standards that affects the IRS and all the other organizations that are guided by them? Let’s see.

Cybersecurity is a complex issue. I’ve used the metaphor of a gated community to explain general computer security; while the walls and gates of a gated community tend to be secure and well-maintained, the equivalent in the computer world is a patch-work of incompatible wall sections from different manufacturers which are never built properly and often need fixes to be applied, which the computer managers too often take months to apply if they do the work at all.

It’s possible that a hacker broke into the IRS. But what probably happened is that an IRS employee or contractor with legitimate access to IRS data decided to make a political statement by grabbing the files of ultra-wealthy Americans, smuggling them out of the agency and giving them to Propublica. This is known as an “insider” threat. Here’s the shocker: modern corporate and government cybersecurity standards and regulations fail to prevent or even detect insider threats!

Insiders stealing the data of the company or agency they work for has happened many times. The famous Edward Snowden case is a classic example of an insider stealing secret information and leaking it for publication. Snowden was a contractor who worked at the super-secret NSA (National Security Agency). He saw the surveillance of citizens that was being performed by the agency and didn’t think it was right, so he gathered lots computer files documenting the behavior and sent the files outside the agency for publication.

Snowden did electronically what Daniel Ellsberg did decades ago physically. Ellsberg was a military officer who had helped create reports describing in detail secret operations the US conducted during the Vietnam war. While working at the Top Secret RAND Corporation he gained access to a copy of the reports and walked out the door with them in his briefcase. He gave them to the press, where they were headlined as the Pentagon Papers.

The NSA has a positive reputation for cybersecurity. The cover story in Wired Magazine in June 2013 featured a description of a visit to NSA HQ in Fort Meade with its elaborate security measures. The strong impression given is that an organization that has so many strong walls, locks and cameras must be able to do the equivalent in the invisible world of computers. The timing of the cover story was perfect. Edward Snowden started leaking secret NSA documents in December 2012; the leaked documents were published shortly after the publication of the Wired Magazine issue praising the ultra-security of the NSA.

There are systemic issues that result in most of the successful hacks of governments and large companies which I describe here. What it comes down to is two main factors: the people in charge don’t understand the world of computers; the people in charge take a slow, regulatory approach to security, while the opposition is fast and creative.

For the IRS, the data loss is similar to books being taken from a library without being checked out, and can be fixed using electronic versions of methods that librarians use: check the books anyone walks out with!

Personal tax information is valuable, like the goods sold by high-end retailers. Think about jewelry stores; nearly anyone can go in the store, but all the valuable jewels are closely watched as they are taken out of display cases, tried on and put down. You don’t get away with slipping a diamond into your pocket and walking out of the store. Systems like this can be and have been implemented in the world of computers. I go into more detail here.

Going beyond basic monitoring of the behavior of computer users, it’s possible to translate methods that are in production today for catching credit card fraud to the problem of data leaks. Basically what you do is use machine learning to model everyone’s normal behavior concerning data access. When someone does something that is not normal for them, the model immediately notices and calls software to stop them and raise an alert.

In the case of the IRS the general behavior monitoring behavior could be refined, since IRS employees work on cases that have been assigned to them. The software would look at each file a user accesses and make sure that file is relevant to a case they’re working on; if not, the software would prevent access and raise an alarm. That way an errant employee who tried to pull Warren Buffet’s tax data who wasn’t specifically assigned to the case wouldn’t be allowed to do so. And the person working on Warren Buffet’s case wouldn’t be able to access Elon Musk’s case.

It’s less likely but possible that instead of the bad guy being an employee, it was a hacker who gained access to internal systems using methods similar to the ones that resulted in financial records of 147 million Americans being stolen from Equifax in 2017. I describe that hack here.

If the internal monitoring systems I have described were in place, it would also catch a person who had gotten into the IRS by hacking – the beauty of the method is that you don’t worry about who the actor is – you just worry about what they do, just like in a library or jewelry store.

The cybersecurity problem isn’t limited to giant government bureaucracies with outdated computer systems. It’s widespread, in part because they all follow experts, standards and regulations that ignore the insider threat. I analyzed in detail the various experts who were quoted in articles published by the New York Times about the Wannacry ransomware attacks based on software that had been leaked from the NSA. I found that the experts were simply wrong about the reasons, methods and responses to the attack.

It is ironic that the same government authorities who force everyone to follow ineffective regulations they craft by the ton are spending even more money training young people in their methods. My local community college was conducting training sponsored jointly by the NSA and DHS (the Department of Homeland Security); when I looked into it I found that the experts couldn’t even build functioning, secure websites with accurate information.

I sincerely hope that the ongoing flood of illegal leaks and ransomware attacks will end soon. But so long as the current batch of bureaucrats, regulators and experts are in charge of things, we’re likely to spend ever-increasing amounts of money on cybersecurity with ever-worsening results.

Note: this was originally published at Forbes.

Bitcoin Enables Criminals to Thrive While Promoters Deny the Facts

Many people believe, for good reason, that the cryptocurrency Bitcoin is widely used by criminals. The growing number of firms looking to profit from the use of Bitcoin as a legal investment don’t like being associated with crime. So they decided to form an organization and pay an ex-CIA director to lend his prestige and credibility to a report that distorts the truth and whitewashes the huge and ongoing use of Bitcoin and other cryptocurrencies as key parts of criminal enterprises.

The Crypto Council for Innovation

The IPO of crypto firm Coinbase at a valuation of about $100 billion shines a bright light on the main asset it manages, Bitcoin. Coinbase, along with other crypto firms and major financial firms such as Fidelity, have formed a trade group called Crypto Council for Innovation whose purpose is to promote the benefits and general wonders of crypto and to “encourage the responsible regulation of crypto in a way that unlocks potential and improves lives.” They also will be “…addressing misperceptions and misinformation…”

To that end, the group has sponsored a paper. Here’s their description:

In An Analysis of Bitcoin's Use in Illicit Finance, a study authored by Michael Morell, former Acting Director, Deputy Director and Director of Intelligence at the Central Intelligence Agency (CIA) examines the general assertion that the Bitcoin market is rife with illicit activity. Morell concludes that Bitcoin's use in illicit finance activity is limited and orders of magnitude lower than what has been cited by government officials. Morell's analysis also reveals that the blockchain ledger is a highly effective crime-fighting and intelligence-gathering tool.

The group is already meeting its goal of influencing public opinion. The headlines of articles include:

Embracing Bitcoin is now a matter of national security says former CIA director

Bitcoin is a ‘Boon for Surveillance’ says former CIA Director

 

How an Ex-CIA Director Proved Bitcoin Use in Crime is Declining

Former CIA Director Comes Out in Favor of Bitcoin

The pro-Bitcoin propaganda is working!

There’s the Report and then there are the Facts

Like all propaganda of its kind, the Morell report is all about starting with the conclusion you want – Bitcoin is great, criminals are fleeing from it! – and marshaling an impressive-sounding array of name-brand institutions and experts to say what you want. In this case, since the facts diverge from the desired conclusion so drastically, a good deal of work is required to reach that goal. Here are some of the major things the report did to whitewash the truth.

The report is self-serving

First and foremost, the people who wrote the report were bought and paid to do the job they did. Of course the report included words about how it was “objective.”

Think about it this way: suppose a governor of a state were accused of sexual harassment by multiple women and further accused of hiding actions leading to the deaths of thousands of seniors during the pandemic; how credible would a report be that was commissioned and paid for by that same governor? That’s what we have here.

The “experts” are nearly all anonymous

The report references experts from a wide variety of name-brand institutions who are said to support the report’s conclusions. This is an important subject. Don’t you think the report’s authors could get more than one such expert to go on the record? The one expert they got on the record has been retired for years.

Bitcoin technology is difficult for most people to understand

None of the report authors have any technical expertise. The ex-CIA man spent his early years on energy and East Asia, and then became a manager and communicator – which is what you would expect of someone who now spends lots of time working for media outlets like CBS.

The report shows little understanding of Bitcoin technology

The words used in the second conclusion are one example among many: “The blockchain ledger on which Bitcoin transactions are recorded…” The way this is worded shows a lack of the most basic knowledge of how Bitcoin works, implying that Bitcoin is somehow not part and parcel of what some call the “blockchain ledger,” a phrase made up to describe one of the inextricable parts of the Bitcoin code base. In the normal world, transactions like buying a hose at a hardware store take place; some of those real-world transactions may later be recorded in a ledger; in Bitcoin, there is no difference between the transaction, the Bitcoin and the ledger.

Why do criminals like Bitcoin?

The report claims that criminals are fleeing from Bitcoin. Let’s step back and see what it is about Bitcoin that criminals like. The reason why criminals like Bitcoin and other cryptocurrencies is simple: it’s easy for them to avoid getting caught! It’s even better than wearing a mask when robbing a bank! It’s as anonymous as cash except that it enables the transacting parties to be an ocean away from each other.

How does Bitcoin enable criminals to exchange money secretly?

It is true, as the whitewashing report claims, that the Bitcoin ledger contains a complete record of Bitcoin transactions and is open for viewing to anyone. A knowledgeable person can look at any current Bitcoin holding and trace it back to the prior owner, the owner before, and so on to the transaction that created the Bitcoin. It’s totally transparent!

There’s a little wrinkle, though, that’s the key to everything: the buyer and seller of each Bitcoin transaction are identified solely by the public side of the encryption keys controlled by the transactors. The physical-world identity of the sender and receiver of Bitcoin is not recorded in any way, shape or form! In terms of the Bitcoin ledger itself, everything is 100% anonymous.

The report mostly disputes claims few people make

In the introduction they talk about “…public statements from officials on both sides of the Atlantic who have suggested that Bitcoin is used primarily for illicit activities.” In fact the argument most often made is NOT about the fraction of Bitcoin used for criminal purposes, but about the fact that the nature of Bitcoin makes it DESIRABLE for criminals to use, and that criminals in fact make use of it. At no point do the authors argue against the fact of criminals’ preference for Bitcoin. The authors’ first major conclusion is “The broad generalizations about the use of Bitcoin in illicit finance are significantly overstated.”

The first major section of the report is “Bitcoin’s Use in Illicit Activity is Relatively Limited” The authors don’t deny that criminals like to use Bitcoin. They even admit that it is the currency most often found in Dark Net Markets, i.e., places where illegal substances and objects are bought and sold.

They argue that the fraction of Bitcoin activity performed by criminals is decreasing

As the speculative frenzy for Bitcoin buying and selling continues to grow, this is plausible, but irrelevant to the core observation that criminals like using Bitcoin and that it enables their activity.

Suppose a city suffers 100 murders per year. Then the population of the city greatly increases, but 100 people a year are still murdered. Is the fact that a decreasing fraction of the population is murdered every year something to be celebrated? Would you want to proclaim that your city’s murder rate is going down?

The report they rely on does not support their conclusion

The Chainalysis report they quote does not claim that criminal use is decreasing, but that overall use is growing: “One reason the percentage of criminal activity fell is because overall economic activity nearly tripled between 2019 and 2020”

The also report states: “However, as always, cryptocurrency remains appealing for criminals as well due primarily to its pseudonymous nature and the ease with which it allows users to send funds anywhere in the world instantly, despite its transparent and traceable design.”

The uncertainty of the Chainalysis data about criminal use is ignored

Because of the difficulty of identifying the participants in Bitcoin transactions outside of highly regulated exchanges that enforce standard bank KYC provisions, identifying which are criminal is mostly guesswork.

For example, the 2020 report identified 1.1% of 2019’s transactions to be criminal activity. In the latest 2021 report that number was revised to 2.1%, nearly double! They admit the same uncertainty about their numbers for 2020 saying “we should expect 2020’s reported criminal activity numbers to rise over time as well.”

The authors minimize the extent of the use of Bitcoin by criminals

The revised Chainalysis 2019 number shows “$21.4 billion worth of transfers” by criminals. This is no small amount! It is about the same as the worldwide total amount of money lost to credit card fraud by banks! Vastly more people use credit cards than hold Bitcoin, and the yet the total amount of crime is about the same!

The supposed use of blockchain analysis for fighting crime

The report’s second conclusion finishes with: “Put simply, blockchain analysis is a highly effective crime fighting and intelligence gathering tool.” If this is true, don’t you think the report would have included some juicy examples of crimes that had been foiled or intelligence that was gathered from Bitcoin? The authors fail to give a single example – not one of a crime that has been solved by using “blockchain analysis.” And not one example of intelligence that has been gathered. Do you think this might have something to do with the fact that exactly zero personal identity information is contained in the blockchain? You know, the reason criminals like it?

On the other hand...

Criminals are everywhere. Criminals like money. Criminals have been finding ways to exchange and launder money as long as there has been money. Fully regulated banks that enforce KYC (Know Your Customer) identification standards are still used for criminal purposes. The most recent report (from 2015) from the US Treasury estimates that about $300 billion is laundered every year in the United States. This is in spite of the massive AML (anti-money-laundering) regulations imposed on banks that cause them to produce a flood of required AML reports to the feds, who proceed to catch and stop only a small fraction of the crime. Only this year, after years of increasing regulations and costs with ongoing ineffectiveness, has the relevant agency started to take steps towards measuring effectiveness instead of just requiring “churning out more data that proves to be less than helpful” in actually catching the bad guys.

Conclusion

Bitcoin promoters are anxious to upgrade the reputation of their currency. The report they sponsored for that purpose marshals an impressive array of classic propaganda techniques to convey its misinformation.  Why not just state the facts?

The facts in this case are simple. Criminals make extensive use of our existing financial institutions. They manage to do so in spite of huge, costly efforts of banks, regulators and enforcement agencies, who end up catching only a small fraction of the crime. Criminals were early to jump on the Bitcoin bandwagon because of the anonymous, instant transactions it enabled. Criminals use Bitcoin today and are highly likely to continuing doing so, just as they continue to use existing banking mechanisms and largely escape capture. There is nothing about the “transparency” of Bitcoin that makes it easier to catch bad guys than existing systems.

I hope that the relevant organizations abandon the time-wasting report generation approach they’ve taken to finding financial crime in most areas other than credit card fraud and shift to a more entrepreneurial, results-oriented model with proper incentives to the participants. Here is the idea of the approach in general, and here’s an example of how it’s worked out in credit card fraud.

This was originally published in Forbes.

Gated Communities Help Us Understand the Ongoing Cyber-Security Disaster

There’s a war going on in our computers and networks. It’s a silent, invisible war. It’s fierce and continues to escalate.

The bad guys are winning. They are aggressive, hard-working, learning, inventing and focused on the goal of making money. The large army of the good guys is led by hapless, incompetent, unmotivated bureaucrats with meaningless certifications in this or that, consumed by building an audit trail showing that they’ve followed the ever-growing body of useless regulations so that when the nearly-inevitable security disaster happens, they can prove it wasn’t their fault. It’s clearly not a fair fight.

The security war isn’t like a war between nations. It’s more like a sprawling collection of gated communities infiltrated and attacked by myriad bands of criminal groups who break in, rob valuables and sometimes take hostages for ransom. The communities spend more money every year building walls that are higher and stronger and hiring ever more highly trained security people. Governments have multiple departments whose purpose is to stop the criminals directly and to help the communities better defend themselves.

Every year the money spent to prevent cyber-crime goes up, and every year the amount of illicit goods the criminals make off with goes up. The criminals are almost never caught. The problem is clearly not that the communities aren’t spending enough money. The problem is that the defenders don’t know much about computers and are going through the motions while the attackers are going for the gold, i.e. bitcoin.

Hardly anyone, including certified computer professionals with academic degrees, understands what goes on inside computers. They attempt to secure computers using ineffective methods that sound impressive but which they themselves don’t understand. They take great pains to do fancy-sounding things that sound impressive but make no difference.

I have explained the details of the massive hack of Equifax hacking by comparing Equifax security in the computer world to a car dealership’s security in the physical world. Translating invisible computer events to common sense physical things can help anyone understand what this cybersecurity war is all about. In this article I’ll attempt to explain what computer-style security would look like if it were applied to a gated community.

 

The Gated Community: Defenders and Attackers

The people who build the walls that protect the outer perimeter of the community are proud of their work. In some places they even have walls inside walls!

• If the walls were built like computer “walls” are built, you’d see that the walls are a patchwork of wall segments designed at different times by different vendors using very different materials and designs and are shipped with so many flaws that they need frequent upgrades. The people in charge of wall installation and maintenance rarely stay on top of the never-ending flow of walls patches and corrections and apply them haphazardly, if at all. The result is that all the savvy criminal has to do is jump on a vulnerability the second the manufacturer announces it and probe all the walls. It’s not hard – a large fraction of wall maintainers leave gaping holes unpatched for months or even years. Shazam! The criminal is inside the community.

The entrance to the gated community is a gate with a 24x7 guard checking the ID of everyone who enters against the list of people who are permitted to enter. The guard permits no exceptions.

• The people who live in gated communities want lots of people to come to their homes and do things for them. They call in and have the person added to the list. The service person comes to the gate, shows ID and is allowed to enter. Maybe they go to the house of the person who wanted them, but there’s no one to stop them going to other houses and doing whatever they feel like, just like a criminal who snuck through the flawed outer wall. Computer programs who “knock on the doors” of heavily guarded computers do the same thing, often with stolen ID’s.

The houses in the community are built securely, with locks on their doors and windows, so that even if a criminal manages to get inside the community, they can’t rob the house.

• In the computer world the houses and their exterior walls (servers, operating systems and applications) are built by the same kind of hodge-podge of vendors that build the exterior fences that protect the community. Houses are supplied by a variety of huge vendors using complicated methods and materials. It’s extremely rare that a house is installed without flaws – the builder will usually claim that it’s flawless, but then will come a stream of patches that need to be applied to the house with varying levels of urgency. A diligent clever criminal can go around probing houses for flaws that have yet to be discovered or repaired by the original manufacturer; a lazy criminal can just wait until the flaws are announced and probe specifically for the known flaws, confident that most homeowners won’t bother to apply the corrections.

People in the community want service workers to come to their homes when they’re out to do jobs like cleaning. It’s a huge convenience to have that guard at the gate checking ID to make sure only authorized people are let in. The guard can also loan the authorized person a spare key so they can enter the house and do the work they’ve been asked to do.

• In the computer world the criminal enters with a fake ID and gets a “key” which usually gives you permission to enter many “houses,” where you can do whatever you want – mess things up, steal things, etc. You can even change the “locks” and scramble things up (encrypt them) so badly that the house can’t be used. Imagine that the frying pan is stored under the hats in the coat closet and all the cooking knives scattered inside pieces of clothing in every room. How would you feel about cooking?

As in many high-class gated communities, mail service to the house is provided. Of course, only an authorized mail person is let in the gate in his truck that contains all the mail and packages to be delivered.

• In the computer world, homeowners aren’t careful about opening their mail, and sometimes packages they open contain invisible little criminal robots that immediately scurry out of the homeowner’s sight, pull out their cell phones and start communicating with criminal HQ. They run around the house and send out all the private information about the people who live there, including ID’s. They’re RAT’s (remote-access trojans) and result from what are called phishing attacks in email. Most homeowners aren’t able to resist opening infected emails of this kind. Sometimes the RAT’s make copies of themselves and send them to neighbors’ homes, spreading the problem.

Some security-minded people in the gated community protect themselves by installing cameras and other security devices so they’re alerted any time a person enters their house when they’re not there.

• In the computer world, tracking activities inside the house is extremely rare; detecting unusual activity and sending alerts when it’s detected is almost unheard-of. The criminal computer invader is free to take his time while sending copies of all the valuable information in the house of the gated community to criminal HQ. If the criminal feels like it, he can massively scramble the contents of the house to make it practically unusable, maybe even putting things in locked cabinets. Then he leaves a sign on the entrance door explaining to the homeowner what he’s done and demanding a ransom to return things to normal. When the ransom is paid, nearly always in untraceable bitcoin, the house often isn’t returned to normal after all. Why is anyone surprised?

The guard at the entrance gate makes sure that moving trucks that enter have been authorized by a homeowner, and that when they leave, the homeowner has approved the exit.

• In the computer world, watching what goes on inside the walls is rare. The equivalent of moving trucks can be created and endless streams of them can go through the exit of the guarded gate and no one checks.

Some homeowners are particularly concerned about the valuables inside their houses, and so they keep those valuables in an expensive, thick safe, secure from thieves.

• In the computer world, data you want to protect from being stolen is encrypted “at rest.” But just like you have to take jewelry out of the safe to wear it, you have to unencrypt data to use it. Therefore the programs in the house that make use of valuable data use a method to access the data that unencrypts it automatically after taking it from the “safe” and before giving it to the program that needs it. When the criminal software is in the house, it simply uses those same programs to access any and all data that it wants, loads massive valuable data into moving trucks and sends it out the unguarded, unwatched exit gates of the gated community. The owners frequently don’t find out about the theft until months later.

Conclusion

I’m sorry to say, things in the computer world are just as bad as I have described. Cyber-security experts and regulators do little but build up the mountains of pointless, ineffective procedures and regulations to ever-growing heights, seemingly without questioning the value of their efforts. Why should they? No one else does – their highly paid jobs are secure!

This was originally posted at Forbes.

T

The Colonial Pipeline Cyber-security Disaster in Context

There’s little new about the Colonial pipeline security disaster; nearly everything was business as usual: expensive but ineffective cyber-security systems and people; penetration and massive data stolen secretly; learning about the breach when ransomware popped up and said “pay me;” shutting everything down; taking days to recover; and the government, which is incapable of protecting itself, making solemn statements about “helping” more.

Colonial did manage to stand out from its fellow victims in a couple of ways: unlike many victims, they paid the big ransom; and their shutdown hurt millions of normal people in significant ways.

The other way Colonial stood out is remarkable: it was big news in the media for days, while the many “successful” ransomware attacks that take place each and every day on businesses, governments, schools and hospitals are rarely made public, much less make the news. Cars being unable to get gas in multiple states, even at inflated prices, may have had something to do with it...

Colonial Pipeline in Context

As usual in disasters of this kind, many important details of the attack and the response to it are closely-guarded secrets.

Let's step back and put what we do know about the disaster in context.

Ransomware has been around for a couple decades. It was usually sent to consumer emails with threats of various kinds if a ransom wasn’t paid. The ransom amount was usually under $1,000 and was paid to prepaid cards and other places. Because of the ability to trace the payment, the scam became less profitable and more dangerous for the criminals involved.

Then bitcoin came along. By 2013 exchanges appeared that enabled criminals to receive instant payments while remaining anonymous and untraceable. Why attack consumers for small amounts when you can get into a large institution and demand large amounts that you can receive instantly and anonymously anywhere on the globe? The threat evolved as well. The attacking software, after gaining entry into the target’s internal computer network, would encrypt all the data, making the organization’s computers nonfunctional until the data was unencrypted using a key known only to the criminals. The victim would be stalled in place, unable to function until the ransom was paid or the computers were otherwise restored. The use of ransomware exploded.

While the well-paid but ineffective defenders sloppily applied what they were taught in school and followed the thousands of pages of security-related regulations, the criminals evolved on multiple fronts. Ransomware evolved rapidly during 2020. The criminal attackers started to take a copy of the target’s data (exfiltrate it) before locking it up. The threat evolved to: if you don’t pay us we’ll make all your data public and you’ll be locked up.

Industry expert EMSISoft tells us: “As the year progressed, more and more groups started to exfiltrate data, using the threat of releasing the stolen information as additional leverage to extort payment. At the beginning of 2020, only the Maze group used this tactic. By the end of the year, at least 17 others had adopted it and were publishing stolen data on so-called leak sites.”

Oh, expose the data — how bad can that be? Pretty bad. “The data that was published included Protected Health Information (PHI), sensitive information related to school children, and police records related to ongoing investigations.”

What? These sound like highly regulated hospitals and government organizations, even law enforcement. Isn't the government, which creates and sometimes enforces all these cyber-security regulations able to protect itself? I guess not: “Unfortunately the barrage continued into 2020 with at least 2,254 US governments, healthcare facilities and schools being impacted. The impacted organizations included 113 federal, state and municipal governments and agencies, 560 healthcare facilities, 1,681 schools, colleges and universities.”

Colleges and universities? Aren't these the places that train all these cyber-security people and create the theory and practice they all learn and put into practice, with their fancy degrees? How is it possible that the security experts can't keep themselves secure? And just look at those numbers: more than four per day were hit and hurt!

On the other hand, it's just people's data getting exposed and ransoms being paid, right? Sadly it’s more than embarrassment: “The attacks caused significant, and sometimes life-threatening, disruption: ambulances carrying emergency patients had to be redirected, cancer treatments were delayed, lab test results were inaccessible, hospital employees were furloughed and 911 services were interrupted.”

More than an attack a day took place at healthcare facilities! Have you seen any headlines about that?

Alright, alright, this is all about governments and similar institutions. Commercial companies want to protect their profits and their reputations. They probably handle things much better — they must, because you almost never hear about them. Ummmm, maybe not; again from EMSISoft: “The private sector was hit hard too. Globally, more than 1,300 companies, many US-based, lost data including intellectual property and other sensitive information. Note, this is simply the number of companies which had data published on leak sites and takes no account of the companies which paid to prevent publication. Multiple companies in the US Defense Industrial Base sector also had data stolen, including a contractor which supports the Minuteman III nuclear missile program.”

Read that last sentence again, please. The bad guys are successful in stealing even from defense contractors. And the number above is the tip of the iceberg, because it's just the ones where someone was able to find their data for sale — it doesn't count all the ones who paid up, hushed it up, etc. Somebody did a survey to find out just how widespread the problem was in commercial businesses. Here's the bad news: “according to a study by security firm Sophos, 51 percent of all surveyed businesses were hit by ransomware in 2020.”

The iceberg is indeed huge. We're talking serious money given to criminals. From Pentest Magazine: “By the end of 2019, cybercriminals using ransomware had made off with a reported $11.5 billion in ransom payments. By the end of 2020, that number is projected to reach $20 billion.”

That's "just" the ransom money — much more money is spent recovering from the attack, even if the ransom is paid.

With all that bad stuff going on and the FBI and other agencies devoting huge resources to it, at least some of the bad guys are being caught and punished, right? No. According to EMSISoft “the effective enforcement rate for cybercrime in the US is estimated at only about 0.05%.”

In case you're not feeling math-y, let me help. This means that out of each 2,000 cybercrimes, only one is prosecuted.

Conclusion

The Colonial Pipeline event was extremely rare — not that it happened, since about half of all businesses get hit with ransomware every year — but because it made the news and was widely covered.

The reality is that, largely invisible to the public, there are gangs of criminals roving secretly and largely unchecked through our computer systems and networks stealing valuables and extorting money in huge volumes. Business and government spend increasing amounts of money with ever-growing staffs of highly educated, certified professionals to prevent the on-going pillaging. They are failing. Horribly. The vast majority of the "cures" that are batted about will definitely cause everyone involved to spend more money, and will equally certainly make little difference.

I have discussed the issues and illustrated the problems and solutions but it won't make a difference — all the power and prestige go, as usual, to people who are proven ponderous pontificators to whom the entire realm of software is invisible.

Note: This was originally published in Forbes.

 

Ingredients: Whole Foods Sneaks in Sugar

You really do have to read and understand the ingredients list of any food you buy. Even places like Whole Foods need to be carefully checked. Yes, the same Whole Foods that charges "whole paycheck" prices to supposedly only sell food that's good for you.

The Whole Foods brand Olives

Whole Foods is a grocery store with a huge selection of products produced by others. Whole Foods also has its own brand of products, some with standard packaging and some that are custom-packed. Someone I know went shopping there the other day and bought some Whole Foods olives, intrigued by what olives with "tangerine and chilis" would taste like.

After getting them home, my friend opened up the package and tried a couple of the olives. Yuck, what's this? These taste outrageously sweet! These are supposed to be flavored hot and with citrus! What's going on? At that point she took a closer look at the label:

Surprise! The third ingredient after olives and water is "cane sugar." More sugar than oil, more than vinegar, more than tangerines, more than red pepper and more than salt.

This being Whole Foods, the ingredient of salt isn't just any old salt, it's supposedly "sea salt," which sounds like it's better for you somehow. And the sugar isn't any old sugar, it's "cane sugar" -- oh, right from sugar cane, it's "natural," right?

Sugar is a Big Problem

Most people like sugar, which is part of why deserts can taste so good. The trouble is that people who make food only care that you like it and want to buy more of it; they mostly don't care whether the food is healthy for you to eat. Regardless of what they say. Therefore, sugar of one kind or another is added to an astounding range and variety of foods, including ones that are "supposed" to be sweet.

Even the CDC, the well-funded government organization whose whole purpose is to keep us healthy, agrees that sugar is a problem.

Weight gain and obesity, type 2 diabetes and heart disease. Is that all? In the section on obesity, the CDC admits that obesity itself is a serious problem:

This being the CDC, all they do is admit that obesity is "associated with" those fatal things and nowhere do they spell out the curse of sugar added to foods and the need to read the ingredients. The picture above shows obviously sweet cupcakes, while much of the problem is with foods that aren't obviously sweet.

Getting information about sugar

You may have noticed my use of phrases expressing cynicism about the CDC. Although the statements I've excerpted above are true, they have appeared on the CDC after decades of being studiously ignored by this corrupt agency. This is because the CDC has long practice in putting its authority and the patina of "science" behind what various powerful pressure groups want to advance their own interests.

Sadly, like with so many other things, it's up to each person to look out for him/her self. In this case, it's pretty simple: moderate your ingestion of sugar in any form; avoid buying and eating packaged food that has any form of sugar added to it; for this purpose, trust only the small-print list of ingredients.

Conclusion

You can't depend on the idea that a "health oriented" store will only sell you healthy food. You can't depend on how a food product is labelled. You have to take control of your own health, which means taking control of what you eat, which means taking care about the ingredients of the food you buy. It's not too hard, it doesn't take much extra effort, and the payback in terms of length and quality of life are more than worth it.

Software programming language evolution: Functional Languages

Once computers were invented and started being used, people discovered that writing programs for them was brutally hard. It didn’t take long before some smart folks figured out ways to make it easier, taking two giant steps forward in ease and productivity in quick succession.

After that wonderful start, people kept on inventing new languages, but somehow never made things better – in fact the new languages often made programming harder and less productive, a situation that the experts and exalted Professors studiously ignored. They never even bothered to spell out what makes one language better than another, which you'd think would be at the heart of promoting a new language.

I this post I’ll discuss a major category of new languages that keep getting created, receive passionate kudos, are rarely used but refuse to die: functional languages. In a subsequent post I'll describe the truly good idea that lurks inside the madness of the functional language world.

The core idea: time and timeless, music and math

I learned math as a kid. In high school I took the most accelerated and advanced courses available, finishing with a course in calculus, which I aced while mostly sitting in the back of the classroom teaching myself differential equations from my father’s grad school text book. I liked it. At the same time I got into music of multiple kinds but with a strong preference for classical. The strong connection between music and math has often been noted.

During my junior year of high school I took a course in computer programming, with FORTRAN as the language. Our “teacher,” scrambling to keep a chapter ahead of us in the text book, had arranged machine time for the class on Saturdays at a computer at a nearby rocket firm, Reaction Motors. After lots more programming, by my first year of college, it was clear that math took a distinct second place to both music and software in my heart.

In this post I explain where software comes from and its relation to music. The key concept is that, while you can read a page of music or software or math, math usually exists in a world “outside” of time – at most, time is a dimension like space; think for example of a proof in geometry. Music and software, by contrast, can only be understood as flowing through time. When you look at a page of music or software you naturally start at the beginning and hear/see it flow through time in your mind, just like reading a page of text.

What this means is that there is a fundamental dis-connect between math and computing. I explore this disconnect here. The math types are always trying to turn software into math, for example plodding for decades on a fruitless pursuit for ways to “prove” the correctness of programs.

This is the best way to understand both the failure and the refusal to admit defeat of the math types to develop a practical math-like way to write software, an effort and category of languages that is usually called “functional.”

The history of functional languages

Wikipedia's introduction to functional languages is accurate:

Turing had nothing to do with making software programming practical. That amazing job fell to others, who created the two giant advances in software language development. But since the math types were hanging around computers in the early days, particularly because doing math calculation was one of the earliest tasks to which computers were applied, finding a way to make programming more math-like, essentially eliminating the factor of time from software was a top priority.

I had already done serious programming, including working on a giant FORTRAN project to optimize oil refinery operation, when a friend introduced me to functional programming for the first time. The language was APL. It came a bit closer to being able to handle practical math than many of the other non-functional "functional languages" because of its focus on the matrix, but it was as easy to read and understand as advanced math. I could and can imagine that it would be appropriate for representing certain math transformations in a compact way, but why anyone who wasn't being tortured would agree to use it for general programming remains beyond me -- assuming of course that what you want is to ... radical idea here, be warned! ... get stuff done. As opposed to demonstrate and revel in your ideological and mathematical purity and exalted status in the pantheon of Computer Science greatness.

None of this is any concern to the cultists, who march on inventing an endless stream of "new" functional languages. Among the names you might see if you look into this are  OCaml, Scheme, Haskell, Clojure, Scala. There are even a few that are object-oriented on top of being functional, like Erlang. Articles continue to pop their heads up out of the underground insisting that functional languages really are making inroads in commercial applications, better for getting a job, more effective to use to create your ground-breaking start-up, etc. I predict no end to the flow.

The fact that functional languages are mostly of interest to a cult of fanatical core believers minimizes their widespread, on-going pernicious impact on software development, fueled by the near-universal math orientation of academic and corporate Computer Science. Functional concepts keep getting introduced by language fanatics into otherwise-sane normal languages.

Conclusion

Computer software is rarely driven by purely practical considerations much less objective knowledge and definition of what constitutes "good" in a language. But when an approach like functional programming is pushed that leads to results that are often 10X worse on multiple dimensions than "normal" programming, the problem is so large that people who aren't already committed members of the cult notice the difference and refuse to put up with the nonsense.

In spite of all this, there is an amazing valuable insight at the core of the functional language movement that leads to highly productive, real-world-valuable results when embodied in the right way. I will spell this insight out and illustrate its use in a subsequent post. The heart of the insight is taking a declarative approach instead of an imperative one when and to the extent that is appropriate.

 

Yelp is Big Tech Incompetent and Corrupt

Yelp is a Big Tech company that isn't listed when people talk about "big tech," but it's nonetheless all about tech and it's unarguably big, with well over 100 million unique visitors per month to their site and a valuation in the $ billions.

How good is their software? Are they a company from which non-tech corporate America should poach talent to upgrade their in-house tech? I recently had occasion to spend a little time with Yelp, and found that they are indeed an exemplar of the Big Tech tradition of tech bumbling as I have documented.

Harvard Business School famously teaches according to the case study method. Students are presented the case and challenged with figuring out what to do. Yelp would make an excellent case study. Any of the bright, aggressive b-school students who declared that the "solution" to the case was to fire all the executives and replace the entire product and tech team with talented high school grads, would get at least a B. Anyone who further suggested avoiding hiring anyone with a Computer Science degree would get an A.

Yelp and Local Business

Yelp is an advertising company that features listings of local businesses and user-generated reviews of those businesses.I recently visited my local dentist and received excellent care on the way to getting a crown put on one of my teeth. She knows I'm a computer guy and I've helped her out in the past, so she brought up her concern about a couple of malicious false reviews that were posted to her Yelp page. I'd said I'd check it out. I did. The results were illuminating. I sometimes complain about bumbling big corporate companies and their painful but fixable software problems. Yelp provides a great example of how second-tier Big Tech companies are no better at building software than the super-famous first tier ones are.

Yelp and a Local Dentist Business

I checked out Yelp's page for my dentist, Doctor Vu. It had a couple long negative reviews. So I posted a review of my experience, along with a selfie of me at her office.

I went back after a couple days to check out the page in more detail.

At the top of the page:

I see that she has 5 reviews with an average score of what looks like 3.5.

I scrolled down and found the photo section with the photo I had posted:

I took the selfie in her office holding the coaster I had given her last year that I couldn't resist buying at my favorite store in Cape May. She's had it in her waiting room ever since; I hear it often elicits amusement. It just seemed right to take the picture at the end of a visit to get a crown. :-)

After scrolling through ads and other information I got to the reviews. Mine was first:

I scrolled through the rest, including the two 1-star slams. Something didn't seem right. I went back and carefully counted the reviews. I double-checked. There were 6 reviews! The header at the top said 5! Then I made note of the star ratings. The two super-baddies were 1 star and the other 4 were 5 stars each. That's 22 points divided by 6 reviews = 3.67. Looking carefully at the points at the top, the dividing line is clearly at the top of the star, so Yelp thinks it's 3.5.

But then Yelp said there were only 5 reviews in spite of displaying 6.  Is it 22/5? Nope, that's 4.4. A complete mystery. Maybe I'll figure it out later.

At the end I found this:

What's this about? In faint type 16 other reviews? I check it out. At the top of the page that shows up there's a video I can watch and this long, serious-sounding thing about how Yelp cares and they're smart and they're doing the best for everyone:

Impressive! Finally I scroll down to check out these malicious reviews and here's the first thing I see:

Wow. Billions of data points! That software must be good, right? So I scroll down to see these scurrilous reviews. Imagine my surprise when I see this:

Wow. My review. Again!

At least the mystery of the rating average is solved. Even though my review was shown among the legit reviews -- first, no less -- it appears that their ludicrously bad software had marked it as not recommended and so some piece of software decided there were just 5 reviews. So 3 5-star plus 2 1-star is 17 stars, divided by 5 is 3.4. Close enough to the 3.5 average rating they displayed.

Even better -- I received a congratulatory email from Yelp after posting the review with the picture, saying how great it was and I should post more pictures along with my excellent reviews. My picture is indeed featured. And my review is both first on the list of legit reviews AND first on the list of those not recommended. Score! This is what you get by employing thousands of Silicon Valley's best super-programmers...

Those billions of data points must have somehow confused the Yelpers. And their software.

Finally I read through the rest of the reviews that weren't recommended. There was nothing fake about them.

I visited Dr. Vu again to get actually crowned -- the first visit was just preparation. She told me she had contacted Yelp about the situation. First she learned that the Yelp algorithms were state-of-the-art great, and in any case there was nothing a Yelp employee could do to alter their results. Second she learned that she really should pay Yelp lots of money so she could fix up her page, post pictures, etc. Maybe if she paid something could be done about those bad reviews...

My conclusion from this wasn't hard to figure. Yelp is corrupt (pay to play!). Yelp is incompetent (bad algorithms, double-posting, bad arithmetic). Sensible people should ignore Yelp.

Conclusion

I'm tempted to say that Yelp fits right in with its first tier Big Tech buddies, and of course it does. If you can stand it, check out the summary of their issues on Wikipedia. I did and also went to some sources to make sure it wasn't a Wiki hit-job. I also checked out the disastrous business and personal judgment and behavior of the leaders. Pathetic. Yelp is now on my blocked list of websites.

The Dimension of Software Automation Depth Examples

I have described the concept of automation depth, which goes through natural stages starting with the computer playing a completely supportive role to the person (the recorder stage) and ending with the robot stage in which the person plays a secondary role. I will illustrate these stages with a couple examples that illustrate the surprising pain and trouble of going from one stage to the next. Unlike the progression of software applications from custom through parameterized to workbench, customers tend to resist moving to the next stage of automation for various reasons including the fear of loss of control and power.

I will use companies to illustrate this progression that are known to me personally but not widely known. Software history generally is the history of the winners as written by the winners, like most history. However if you want to guide your own software strategy by taking advantage of the clear patterns of history, it is essential to include examples of companies that are rarely discussed.

Nextpage

NextPage was active from 1999 to around 2010. It had a distributed document management product that was prominent in service industries, for example lawyers and accountants. The product operated at a “recorder” level in terms of automation depth. It wasn’t directive, it just sat there waiting for a document request, and when it got one, responded with a list of applicable documents and lets you view the documents.

They and some of their larger customers saw that, in fact, many of their engagements weren’t so very different. As a high-powered, highly paid lawyer, you may think that every job is unique and your ability to handle the differences very important, but it was obvious to people involved that certain transactions involved very similar documents and workflows. It made sense to attempt to capture and automate the similarities.

With the cooperation of one of the world’s largest law firms, NextPage built a new product on top of the existing distributed document platform; this new product was automation at the “power tool” level. While recognizing that each transaction of a particular type would end up producing a unique set of documents, for example, it operated like a series of factory distribution belts, assuring that all lawyers working on an M&A transaction, for example, worked on the right things in the right order with the right base documents, reviewed in the right order by the right people, etc. etc. Lawyers continued to craft their documents. But instead of treating each (for example) M&A transaction as though it were the first one the firm had ever handled, the new system treated them like a repeatable process, with variations.

But the lawyers didn’t think of it that way at all! They were used to thinking of themselves as powerful, autonomous, self-directing agents. Now they were to be subservient to an assembly line in a factory, having their work and output measured as though they were hourly workers? No way – I’m a professional – there’s no way I’m going to put my head in that yoke!

NextPage had a good position in the market, existing customers, a prestigious lead customer for the product, a strong business case, and tangible results. But the resistance to moving to a “power tool” level of application was stronger than the force NextPage was able to bring to bear at that time, and the effort failed to bear fruit. Anyone who looks at the application knows it’s going to happen, the benefits are definitely there. But anyone with sense would be reluctant to predict even the decade when it would be likely to actually happen.

The pattern of resistance encountered by Nextpage was strongly correlated with the prestige and power of the people whose work would be affected. The same resistance is seen elsewhere; for example, how eager are the highly paid and prestigious category of medical doctors to have their work automated?

Captura/Concur

My VC firm invested in Captura, which is now part of SAP Concur, the leader in on-line expense management systems. These applications basically automate the process of filling out expense reports, getting them checked and approved, and finally paid.

If you think about expense reports, you imagine a form that you fill in with your expenses during a trip. You then attach your receipts, drop it off, and eventually get reimbursed for out-of-pocket expenses. So it seems like it would be a “recorder” type application. In fact several companies were started and eventually failed with products that weren’t much more than recorder type applications, with some automatic routing added on. This kind of application didn’t provide enough value to offset the expense of installing and running it.

Captura went much farther. While parts of this application operate at the “recorder” level, particularly the entry of un-automated expense information, much of it actually operates at the “robot” level, which is why the payback for its use is so high. It has built-in rules for approval levels and documentation requirements and many other things. It knows whether and when and to whom expense reports should be routed for approval from the HR system. It gets feeds from the corporate card system. It kicks out exceptions. It allocates expenses to the right accounts. It feeds directly into the accounts payable system to cut checks. When it needs something it can’t get from a computer, it asks a human for it, and otherwise does its job without help.

Unlike other “power tool” type applications, this application tended not to threaten people who were powerful in the adopting organizations. The most powerful people who used it tended to regard it as saving them time, and everyone got their reimbursements more quickly and accurately. The administration group saw it as saving thankless clerical work, and the accounting executives saw automatic enforcement of all their standards. Still, this application category didn’t really take off until the cost of implementation was reduced to a painless level; in other words, it had to wait for nearly universal access to the web to be practical and affordable. And it worked much better as a secure service than as an enterprise application, because the burden of installation and maintenance on IT added so much friction that only the most motivated potential customers would go for it.

ClickSoftware

ClickSoftware provides a classic example of a “robot” level application. It takes the problem of a field service operation, in which there are calls for service from companies in different locations, with different equipment to be repaired, with varying levels of urgency and service level agreements, with different hours of operation and times to travel to them, and finally with different service technicians who have varying skills and qualifications and constraints about hours of work. Who gets sent to which location to fix which problem in which order?

Without something like Click Software, this problem is solved in a rudimentary way by the service manager, who uses some combination of experience and common sense to work out the best solution. However, for a large service organization, the difference between a good solution and the optimal solution can be worth a great deal of money. Click Software takes all those inputs and produces the optimal solution, and asks the human manager for help when the problem simply can’t be solved; in response, for example, the human may ask a critical resource to work overtime, or might call a customer and see if their request can be deferred.

Click illustrates another characteristic common to robot-type applications. It is typical that “power tool” type applications do not become robot type ones by incremental addition. A robot application requires a whole new approach to the problem, and a level of math programming that is likely to be entirely foreign to the software group that wrote the power tool application.

Tape backup systems

In tape backup, whenever a new platform has come out, tape backup software products tend to emerge in the same order, starting with products that just record what the backup operator does, moving to power products that give the operator hints and suggestions and automate the repetitive operations, and ending with true robot products, which are driven by a set of rules and which request operator intervention at only and exactly those times when human intervention is required. For example, Palindrome and Cheyenne (now part of CA) introduced robot-level backup products in the early 1990’s for the PC server platform, at a time when similar products had been in general use on earlier platforms (e.g. IBM mainframe) for decades.

Having cycled from primitive to robot level automation several times as new platforms came out that had tape backup, the whole category is dying, frozen in time as the latest computer systems use the kind of disks that have become incredible cheap and tape has become obsolete.

Conclusion

It seems inevitable that an industry would move, step by step, towards robot-level automation. The historical fact is that while the incentives for advancement are always there, advancement happens only when an industry is "ready" to move, which can be decades after it is technically possible and economically practical.

Anthem Needs my Feedback and Reveals Deep Problems

I’m glad I have health insurance; I’ve had some health problems, now under control, that were expensive to fix. My insurance company, Anthem, has paid the claims.

Still, I can’t help being impressed by how many actions of this health insurance giant are stupid, wasteful, incompetent and worse. To be clear: I don't claim they're worse or better than the others; they just happen to be the one that I have.

I’ve just had a new stupidity inflicted on me by them that is low on the “it matters” scale, and high on the “a smart high school student could have done better” scale. What's interesting is that while the stupidity itself is minor in the overall scheme of things, it's the result of a serious dysfunction causing widespread trouble and hassle for patients and providers, while adding expense to Anthem. The extra painful thing is that, for anyone with a moderate amount of software knowledge, it could easily be fixed!

When I think about all the big, important, highly paid executives involved in and in charge of this, it does make me wonder whether we’d all be better off if they refused to hire any more people with college degrees for any job, and in particular, management.

The only reason this story is worth telling is that it's not about Anthem; it illustrates how  the techniques taught in business and computer science schools and embodied in endless standards and regulations keep large organizations locked into acting in ways that are both expensive and ineffective.

Summary

The root of the issue is my company negotiating a new plan for its employees. Anthem then sent an email (how modern!) welcoming me and inviting me to go digital with card images and an app. I suspect there were self-congratulatory executive meetings about the wonderful modernity of all this. See this for the inexcusably bumbling reality.

A few weeks ago I went to the dentist, got an exam and cleaning and then scheduled for a crown. My dentist submitted claims as usual and got denied, and I received denial EOB's in the paper mail. I sent my new digital card to the office, and that led to another denial and another paper EOB. Finally the office person was able to get through to a human being at Anthem and somehow resolved the issue.

Then I received a survey from one of Anthem's contracted firms to see how impressed I was by Anthem's new plan signup experience. The survey design and implementation resembled a 1995 paper-based process cluelessly "updated" to use, uhhhh, computers.

The whole mess could have been avoided by having a simple system that took incoming claims and checked them against a database of patients and plans, including plans that had been updated and/or replaced. When a claim arrived for a person with a no-longer-in-effect plan, a simple lookup would enable translating it to the new plan information and everything would take place seamlessly, with ZERO action, trouble or inconvenience to patient or provider. You wouldn't even need to send out a survey to ask how the transition went, because it would have been seamless.

An Email asking for Feedback

Some Anthem exec wanted to get some customer feedback about how their new plan roll-out was going. I got an email. Here’s the lead paragraph:

As you’ll guess from my past blog posts regarding Anthem (a summary with links is below), I clicked, hopeful that they’ve gotten moderately competent, but suspecting that some new sophomoric amateur-hour performance will ensue. (BTW, I’ve always been curious about the use of “sophomoric,” since it is the second year of a four year education program. Shouldn’t a performance that’s worse than “sophomoric” be called “freshmanic” or something?)

Sure enough, I clicked, my browser came up and showed me … a blank screen. Yup, nothing. Nada. I refreshed, tried again, same results. Then I thought, “remember this is Anthem we’re dealing with here; what would someone who dropped out of Computer Engineering 1.01 do? Sure, they’d test their stuff on their favorite browser and declare it working!” I strongly suspected that the kid’s (or highly paid seasoned professional with the skill of a kid who dropped out) browser was today’s most widely used one, Chrome. The next one is Safari, Apple’s browser on the Mac. Behind Safari’s share but still with substantial use are Firefox and Microsoft’s Edge. I copied the URL, brought up Edge and plugged it in. It worked! But it failed totally with my Firefox browser. I visit a large number of websites, and this is the first time in years that I’ve gone to a site that managed a complete face plant on Firefox. It takes a certain kind of perverse skill to pull off a feat like that!

With such a great start, who knows what joys would follow? Could this be a candidate for “freshmanic” status??

The Survey

The very first question puts this survey in the running for un-great-ness. Would you take a look at the never-before-seen way for formatting answers to a multiple choice question? With the major choices ending with “that”?

The second question continues the fun.

After asking me whether I remembered receiving this or that communication and answering “no,” the next question was always an huge graphic reproducing what they sent with the question (in effect) now do you remember? Not once -- several times.

Finally came a request to allow Anthem to follow up with me to help improve their experience. To see what would happen, I said yes.

I then got a form in which I was supposed to enter my name, telephone number, email, and best time to reach me. Right. You already know all this. Anthem gave you my email with an ID. All you need to do is give them my answers with the associated ID. That would make it easy for me. Instead, in typical brain-dead, big-corporate manner, you make me enter all the information you already know AGAIN. So NO, I’m not going to fill it in. So I hit Next. What do I get? You can guess, I bet. Yup!

Lots of red with the same questions in red backgrounds. Of course if this stupidity was supposedly all about protecting my privacy, they could have said something about that, apologizing for the inconvenience of entering information they already have. But no. Just ERROR. And I've already agreed to have my answers and identity shared with Anthem!

I could have just closed the browser, but they provided a button labelled “log off.” Which leads to a screen titled “Logged Off,” informing me of my “success” in logging off. But I haven’t “exited” the survey! They close with “Please close your browser to exit the survey.” OMG! If I don’t close the browser I haven’t “exited” the survey even though I’ve “logged off of this survey.” All these years in the business and I’m only just now hearing a new level of sophistication, about how “exiting” and “logging off” are different. Could I possibly screw things up here?

The Anthem Market Strategy and Insights team

I went back to the original email, and clicked on the link “To find out more about Anthem’s research surveys.”

I found out lots of interesting things like how “we’ll never ask for any personal data in our surveys, like social security or ID numbers.” I guess my name, phone, email and (on another question) age aren’t personal. Who knew?

I found out that the geniuses who put together this survey are just one of a crowd:

Always friendly, always available to answer your questions:

I could go on, but what’s the point. Anthem is incapable of doing the simplest kind of market research on their own, so they turned to a bevy of outsiders who can’t do it either, but Anthem can’t tell the difference between good and bad, so who cares? Except it takes a certain kind of genius to consistently pick the worst on multiple dimensions. I guess that’s why they have a “team” working on it – no mere “department” could do it on their own.

How Anthem could leap decades ahead and get to 2010

Anthem doesn't need to invent a thing. All they have to do is copy methods that are decades ahead of the ones they use -- and are quicker, cheaper and more effective to boot. Here are things they could do:

• Make the transition of employees to a new plan seamless.
  
  • The best thing would be to keep the plan number the same and put in a switch to adjudicate and give plan information according to the date of the switch-over. No new cards or numbers, digital or otherwise!
  • The next best thing would be to issue new numbers but keep a database of employees/patients enrolled in the prior plan, and automatically update things like newly arriving claims with the old numbers to the new ones as needed.
  • If you do one of these you won't need a survey!
• Dump the whole survey thing.
  • Track customer actions in detail like modern web companies do and detect when things don't go as they should. If someone isn't getting what they want, detect it and fix it, and give them an opportunity to tell you what's wrong at the time they experience it.
    
• Dump the whole paper thing.
  • I've opted out of paper every way I can at Anthem. I still get loads of paper mail. Why is it so hard?
• Fire your design department and start over.
  • I get lots of to-the-point communications from Amazon. Copy them!
  • Keep it simple. Test everything in small scale with real people before inflicting it on your customers.

Those are just highlights. There's lots more Anthem could do if their august team of highly experienced professionals would stoop to it. See the next section for some samples.

Past Anthem issues

Just to make it easy for anyone researching giant health insurance company outstanding achievements, here is the list of Anthem issues that I’ve looked into.

Again, I make no claim that Anthem is better or worse than any other insurance company. It’s just the one I experience, so theirs are the stories I tell.

Sometime in 2014 Anthem “lost” tens of millions of patient records. And completely botched telling their customers about it.

I discovered on my own that my data was in the stolen data. Anthem then offered a worse-than-useless plan to “help” me.

Somebody at Anthem decided that I would be cheaper to insure if I acted better and made an expensive botch job of offering a pre-paid card as an incentive.

At one point they decided to send me an email to get me to use a primary care physician they had selected for me. Everything about the experience was a nightmare. A case study in stupidity.

Then I discovered on an offer on Anthem’s website to enable me to pay my patient co-pays. Wonderful idea! Except it doesn’t work. Not just a big screw-up, a huge face-plant.

While I was discovering the joys of Anthem’s inability to help with co-pays, I got and responded to a customer survey. It was a model of how to p*ss customers off, not to mention being decades behind standard practice.

Most recently I discovered that my insurance ID changed and that they botched the change and the wonderful new app that was supposed to replace my card.

Conclusion

Big companies can't build software that works. They can't even do surveys. Hiring people from Big Tech companies doesn't help, because they can't do it either. The good news is, that gives LOTS of room for small, innovative groups that get stuff done that people need, building effective software that works quickly along the way. Hooray!

Experts are Super Smart

It's something I always knew, but I just couldn't make the nagging doubts that would pop up from time to time go away. Finally, at long last, the evidence has arrived: experts really are super smart and expert at what they say and do!

The evidence can be found in this article.

This is an extra relief to me personally, since I've expressed such strong support for the authority of experts in the past, giving examples of their prescient ability to analyze and predict with super-human accuracy.

I need worry no more. The evidence has arrived. Experts truly are everything I had hoped they were!

Ingredients and Truth

What's inside the food I'm eating or the substance I'm putting in my body? As I hope we all know, two of the most important factors in health and sickness are WHAT and HOW MUCH we ingest. You would think that paying attention to what we ingest and its varying impacts, good and bad, would be front and center in research and education. Yes, we're told to have a "healthy" diet. We're told some things about what that means. Sadly, a great deal of what we're told is wrong, and even worse, few people pay attention to the actual SUBSTANCE, i.e. the ingredients, of what we eat.

Even though lists of ingredients are printed on food packages in tiny print, how many people read them and make informed decisions based on what they read? Based on the incredible and growing fraction of people in our society who are overweight, I think it's fair to guess that the number of people who make such judgements for themselves is minuscule and, unlike their bodies, shrinking.

A Lesson about Reading the Ingredients by the Dentist

I went to the dentist last week for a periodic cleaning and exam. At the end the hygienist prepared the little plastic bag she gives patients containing a toothbrush, paste, and floss. They are freebies by the manufacturers hoping to win new customers. The hygienist added an extra box of a different toothpaste, saying that my gums bled quite a bit and I should try this toothpaste that reduces gum bleeding.

OK, I thought. I had a couple minutes waiting for the X-ray and for the dentist to come examine me, so I examined the box.

See the red streak on the bottom, drawing your attention to "Helps prevent bleeding gums?" My next thought was "I wonder how it pulls that off?" Yes, I know, I'm a nerd, but that really was my next thought. So I turned to the ingredients. Here they are:

My curiosity was immediately piqued. Just one active ingredient! And it's fluoride, the thing common to all toothpastes. 0.454%. Pretty exact amount. It's purpose is stated to be "Anticavity, Antigingivitus." Nothing about bleeding!

Let's look at the other box, the one with the "regular" toothpaste. Let's see what's different about the ingredients. Maybe there's a different amount of fluoride?

Wow. Exactly the same active ingredient, the same in every detail. But the purposes! The same animosity towards cavities and gum rot, but somehow in this toothpaste the fluoride has a new, added purpose: fighting hypersensitivity! I wonder if that's accomplished by putting the fluoride through extra training or something to give it a new purpose in life? What's that about?

Let's look at the front of the box:

Things are getting clear. The normal toothpaste is called Sensodyne, supposedly to attack sensitivity. With fluoride. Same active ingredient.

It's clear, number one, that our wonderful regulators don't care how products are described or sold, so long as the universally-ignored ingredients list in tiny print is accurate.

Oh, wait! Maybe I'm being too hasty! I'd better check the so-called "inactive" ingredients. Maybe one of them is the key thing that makes the difference.

Here we are for Sensodyne:

And the corresponding information for the terrific product that helps prevent bleeding:

A careful check shows that the two products have the same, the identical blankety-blank ingredients!

I'm a complete, total pain in the neck, I know, so I mentioned this politely as a question to my dentist and hygienist. They had no explanation. Oh, the hygienist said, it's what the salesperson said.

I've had many different dentists over the years as I've lived in different places. I've always gotten the little bag of freebies at the end. Has there been an uprising in the dental professional community about this decades-long lying and misrepresentation? If you've heard of it, please let me know.

Conclusion

The big companies making toothpaste and other things you put in your mouth depend on their ability to lie about the virtues of what they're selling and get away with it. They know through experience that no one pays attention to ingredients and that the relevant professionals ignore the issue if they're even aware of it.

The example I've given is trivial. It's just toothpaste and whatever you pick is unlikely to hurt you. But things that you put in your mouth and then swallow or breathe in are a whole different matter. The trouble is that exactly the same blatant lying goes on with food products as I've illustrated here with toothpaste. It would stop if the government made it stop. But it won't. It's the government, after all, the same government that continues to dish out misinformation about nutrition.

It will only stop when consumers rise up and act: start paying attention to ingredients and stop buying products with ingredients that are bad for them. Do schools teach this? I would say that schools spend 10X more time on sex education and political indoctrination than they do on teaching kids what ingredients mean and how to select and control what they eat -- except that implies that the amount time spent on that is greater than zero, which it probably isn't in most cases.

I will return to this health-essential issue in future posts.

The Dimension of Software Automation Depth

I have discussed the fundamental concept of automation.

Software automates human effort to varying degrees. In doing so, software emerges that performs this automation to an increasing extent. In this post I'll describe the basic stages through which automation progresses. In later posts I'll give specific examples.

Automation depth

Automation depth can be understood in terms of the extent to which human effort, observation, knowledge and control is replaced by the computer system.

Stages: help a human do something; do something a human would otherwise have to have done; make a decision a human would have made; gather information about all the work that is arriving, in process and completed; make work and resource allocation decisions. There are a few different  ways automation depth can be understood. Here are a couple of them:

• Stages: As depth increases, the human does things faster and has less work to do.
• Automation depth is correlated with moving from an “open loop” system to a “closed loop” system.
• Effort. In the earlier stages the computer augments or replaces the person’s efforts. This is like power steering in a car, in which everything is the same except turning the wheel takes less effort.
• Knowledge. As automation depth increases, less knowledge is in the heads of people and more in the computer.
  
• Control. As automation depth increases, the human loses control and eventually becomes controlled by the computer.

Automation depth has these basic levels:

Recorder	The software essentially tracks what people do and records the results of what they create or decide. The most primitive software of this class does this for a single job task. More advanced software does it for multiple tasks, eventually for a person’s whole job.
Power tool	The software takes high-level instructions from the operator, and does most of the work. In less skilled environments, the power tool will often select and order the work to be done.
Robot	Like an auto-pilot, the software replaces the human for most functions, performing those functions better than a human could, leaving the humans only to set goals and processing rules, and handle exceptions.

Unlike some of the software evolution progressions I've described, iIt is not generally a quick win to get to the next level of automation depth. An industry can stay stuck at a level of automation depth for decades. Normally, the people directly involved with an application strongly resist moving to the next level of automation depth, because they see it as deeply threatening to their autonomy, power or skills.

If a company tries to go to the next level, it is essential that it understand its position, and have the backing to last out the transition and the flexibility to keep trying until it establishes a beach-head from which it can expand. The benefits have got to be dramatic and tangible. Feel-good stuff tends not to work here. Some people are going to get bent out of shape, and managers of managers have got to impose the solution. The only way that typically happens is if things are arranged for their risk of failure to be low and the rewards great, and indisputable when they actually happen.

I will give examples of this progressions in future posts.

Micro-Services: the Forgotten History of Failures

Micro-services are one of today's leading software fashions; see this for more on software fashions in general. I have treated in depth the false claims of micro-services to make applications "scalable," and separately treated in depth the false claim that micro-services enhance programmer productivity.

Where the heck did such a crappy idea that has taken such a large mind-share in the software and management community come from?

I don't know. Who knows why skirts are short or long this year? Or whether shirts are tucked or not tucked? What I do know is that bogus fashion trends infect the software industry, some of them delivering decades of perniciousness. Looking at the spectrum of software fashions, I've noticed similarities among some of them. Some fashions well up, spread and peter out, only to recur again later with minor changes and a new name. The new name appears to be important, so that everyone is fooled into thinking the hot fashion is truly "new," not tarnished by the abject failures of its prior incarnations. I describe a clear example of an old idea with new name here.

The History of Micro-services

Why do we have micro-services? How did this bad idea get to be so popular? Was it someone's recent "great idea?" There are probably people who think it's their idea.

Do micro-services have a history? Of course they do! Nearly every major practice in software has a history of some kind. The number of software practices that are newly invented is truly tiny and going down rapidly. So why do we hear about software advances and the "new thing" so often? The whole field of software ignores its own history in truly radical ways. It even ignores what other software people in other fields and domains are doing.

Nonetheless, anyone who knows a broad range of software practices over a period of decades can easily recognize similarities and patterns -- patterns that surely constitute "history" whether or not the actors recognize the precedents of what they do.

What are Services?

Let's step back and understand what "micro-services" are. By the name, it's obvious that a micro-service is an itty-bitty "service." So what's a service? Once you peel back all the layers, a service is a plain old subroutine call with a bunch of fancy, time-consuming stuff stuck in between the code that calls and the code that is called. See this for my explanation of the fundamental concept of the subroutine.

Subroutines (or functions or methods or whatever) are an essential aspect of programming. Every programming language has its syntax, statements and other things that people get all wrapped up in. Every programming language also has an associated library -- as in function/subroutine library. If you look at a manual for a language and one for the associated library, the library is nearly always larger. The richness of a library is a key factor in the utility of a language. In modern terms, an associated framework serves essentially the same purpose; an example is the RAILS framework for Ruby, without which Ruby would just be one line on the ever-growing list of mostly-forgotten languages.

When you're writing a program, knowing and making great use of the associated subroutine library is essential. But what if you see that you're doing the same kind of thing over and over in your program? It makes sense to create a subroutine to perform that common function as part of the overall application you're building. As time goes on, most well-structured applications include a large portion of nested subroutines.

What if there's a program that can only run on another computer and you want to call it as a subroutine -- give it some data to work on and get the results back? This problem was addressed and solved in many variations decades ago. It's most often called an RPC -- Remote Procedure Call. To make a call to a distant subroutine you implement a version of it locally that looks and acts the same, but does some networking to send the fact of the call and the parameters to a cooperating routine on the distant computer. That routine gets the network call, retrieves the data, and makes a local call to the subroutine. When the return happens, the data is sent back by the same mechanism. It's just like writing a memo and putting in your out-box. The clerk who picks up the outgoing memos delivers the ones that are local, and puts the ones that aren't into an envelope, addresses it and puts the memo with return instructions into the mail. And so on. The calling code and the called code act like they normally do and the RPC makes it happen.

Along comes the internet. People notice that http is supported all over the place. Why not leverage it to implement the mechanics of the RPC? Voila! Now we have SOAP -- Simple Object Access Protocol, which uses http and XML to send and return the data. A subroutine with a SOAP interface was called a "service." This all got standardized roughly 20 years ago. Then along came a simpler version to use that had even more overhead called Restful, which is still in widespread use.

If you need to call a subroutine that can't or shouldn't run on your computer but on some other computer, having some form of RPC is extremely useful. The time penalty can easily amount to a factor of thousands or even millions compared to a normal local subroutine call, but you'll gladly pay the price if there's no other way.

Now let's recall what all this amounts to. A "service" is a subroutine that has a HUGE amount of overhead. There is no reason to ever take on the overhead unless, in the vast majority of cases, the service you want to call is on a different computer than the one doing the calling, a computer that can only be accessed by some kind of networking.

Prior Incarnations of Service Insanity

There have been a couple waves of service fashion mania. One of the waves took place roughly twenty years ago during the internet bubble. People were very concerned to be able to support growth in the use of their applications by unprecedented factors. Expert opinion rapidly agreed that building a "distributed application" was the way to accomplish this. The idea was that the load on the computer would greatly exceed the capacity of even the largest, most capable computer to handle it. To anticipate this, the application should be architected to be able to run on multiple computers that could be changed at will. Instead of a "central" application, the application would be "distributed." This was further elaborated by replacing the simple RPC mechanism with queues called "service buses," as in a bus to route large numbers of service calls from one place to another. The bus itself had to be robust so that messages weren't dropped, so it evolved into an "enterprise service bus," something which exists today. A huge, complex mechanism for accomplishing this became part of java, J2EE (Java 2 Enterprise Edition). See this for the complex, reincarnating history of the transaction monitor.

It turned out that building a "distributed application" was vastly more expensive and time-consuming than just building a plain old application, the kind that today is sneeringly dismissed as being "monolithic." The computational and elapsed time running cost of such an application was also huge. Hardly any applications had loads so large that they needed to be distributed. Even worse, the few applications that ended up with huge loads that required many computers found vastly simpler, more effective ways to get the job done. It's worth noting that articles with titles like "Why were we all fooled into thinking building distributed applications was a good idea" never appeared. The whole subject simply faded away. Here's a description from another angle of the distributed computing mania.

Another incarnation of this nutty idea took place largely inside large enterprises. Many of these places had highly diverse collections of applications running the business, accumulated by acquisition, multiple departments building applications for themselves, etc. Instead of doing the sensible thing of reducing the total number of applications by evolving the best versions to be able to handle more diverse tasks, the idea of a SOA, service-oriented architecture, became the focus. All these applications could be turned into services! Instead of building new applications, people would now build services. Tools were built to manage all these new services. There were directories. There was tracking and control. All sorts of new demands came latching onto the IT budget.

SOA never got to the fever pitch of distributed applications, but it was big. It went the same way -- fading as it turned out to be a lot of time and trouble with little benefit.

Now we come to the present. Remember that the original motivation of the RPC in any form is that a subroutine you want to call is ONLY running on some other computer. An RPC, whatever the form or cost, is better than nothing. Sensible then and sensible now. Then came ways of building programs so that their parts COULD BE run on different computers, if the computational requirements became huge. This also turned out to be rarely needed, and when it was needed, there were always better, faster, cheaper ways than a service approach. Now, with micro-services, we have reincarnated these proven-to-be-bad ideas, claiming that not only will micro-services effortlessly yield that wonderful virtue "scalability," but it will even make programmers more productive. Wrong and wrong.

Conclusion

I like the phrase "oldie but goodie." Sometimes it's applicable. In computing when a hot new tech trend comes along that "everyone" decides is the smart way to go, it rarely is. It most often is an "oldie but baddie" with a new name and new image. You would think that software people were so facts-oriented that they wouldn't be subject to this kind of emotionally-driven, be-part-of-the-group, get-with-the-program-man kind of thinking. But most people want to belong and want increased status. If believing in micro-services is the ticket to membership in the ranks of the software elite, a shocking (to me) number of people are all in.

Why Can't Big Companies Build Software that Works?

Years ago when I encountered big-company software flubs that were screamingly bad I would wonder about them: all those people, all that money, all those MBA-led processes and controls -- how could they screw up basic software things? What's wrong with them?

Now I know more. I know the flubs I saw weren't exceptions. They were and are the rule. I know that big corporate and government organizations are incapable of building reliable, cost-effective software. I have identified many of the contributing causes to this disturbing phenomenon, but there's much that isn't known. What's worse, all of the important people, leaders of academia, government and industry, ignore and/or deny this fact. 

Yet Another Inexcusable, Simple Stumble

I've covered many big organization face-plants in the past. In order to make it clear that the awfulness isn't just crippling software fails but encompasses a broad range of consumer-dissing inconvenience, I'll show some software that "works" but puts customer inconvenience front and center.

I got a bill from my gas utility provider. Utility company billing has been around for awhile, right? Treating your customers decently and getting paid should be high on the list of any company's priorities, right? How long have auto-pay and e-billing been around -- just flew out this year, didn't they? Bad joke. We all know that auto-pay of some kind has been around for decades, and so has some form of e-billing. So how long do you think it should take for a giant, lumbering company to modify their billing software so that auto-pay is handled reasonably well? Common sense says it should only take a couple months, but rich, giant company maybe a year? Two years? Let's say that by any reasonable measure it should have been completely nailed by 2010, over a decade ago.

Let's look at a bill I just got from my local gas utility. It was an e-bill -- hooray!

I read the above. I immediately think -- hey, I've got this account on auto-pay; what's this with Click here to make a payment?? I'm on auto-pay!

I keep reading and next see this:

Great. If -- If -- If I'm enrolled in Auto Pay? What??? You guys don't know if I'm enrolled in auto-pay?? Your software went to my account master record in your database. You extracted my name, billing address, e-mail address and current amount owed. It's hard to imagine that my auto-pay status isn't also there -- after all, you knew enough to send me this e-mail.

Couldn't you possibly, while you were there with your ancient gnarly claws on my account data, also have extracted my auto-pay status and congratulated me for being enrolled? Couldn't you have assured me that this e-mail was for my interest only and was being paid as usual?

Of course you could have. A trivial addition. Instead you make me wonder whether you have somehow dis-enrolled me as a sneaky way of tacking on a late charge? How can I find out? Then you give this:

Yeah, dial that number and wait around. Send an email. Sure. Or go to your generic website and try to pass the obstacle course to getting the information I need. That's what I did. I got through the obstacle course and verified that I'm in auto-pay and in fact owe nothing. Thanks for nothing, South Jersey Gas.

This is such an obvious thing. Maybe they're too small? It turns out that South Jersey Gas is a public company with $1.5 Billion in annual revenue and 1,100 employees. They serve 400,000 customers.

Maybe they're desperately searching for programmers? I looked at their job openings. No programmers. But there is a technical kind-of job that they're seriously looking to fill:

Yup, a complete b.s. job, whose only relevance is to produce reports and do things to make the bureaucrats in government agencies who wouldn't know a line of code if it bit them on the ear stay calm.

Here is the first sentence of the job description, which is followed by paragraphs of similar verbiage:

The role will be responsible for establishing appropriate Critical Access and SOD rule sets for different applications (Workday Financials and HCM, Oracle Customer Care & Billing (CC&B), Maximo and Hyperion), managing IT organizational policies and standards in support of legal and regulatory compliance needs, designing and testing general IT and organizational information security controls and interfacing with Internal Audit and business leaders to ensure that controls are designed and operating effectively. 

Got it? I looked through the rest of the details, and nowhere was "common sense" or "assure at least moderate levels of customer respect" mentioned or even hinted at.

Conclusion

The South Jersey Gas e-mail bill notice is a hardly-worth-mentioning issue given the massive fails that these organizations regularly commit. But when you consider that auto-pay is a good thing for customers and for the gas company and that about 5 minutes of real work is all it would take to fix the problem, and you think back on the important, well-paid job they're advertising for, you realize that the software problem is wired into the system and that everyone whose opinion matters thinks things are fine.

Maybe they should buy software instead of building it? They do buy software. They can't handle that either. See this for a great example with links to more.

 

Fundamental Concepts of Computing: Subroutine Calls

Subroutine calls are glossed over as yet another thing to learn when software is taught or described. The fact is that subroutine calls are one of the most amazing, powerful aspects of computer programming. They are the first step up the mountain of abstraction, the path towards creating software that solves problems with optimal efficiency, effectiveness and ease.

What's a subroutine? In common-sense terms, it's like a habit. I hope you have the habit of brushing your teeth. Calling a subroutine is like deciding you should brush your teeth. The subroutine itself is like the steps you take to brush your teeth: open your medicine chest, get out your toothbrush and toothpaste (probably always in the same order), etc. When you're done (you've rinsed your mouth and put stuff away for the next time) the subroutine is done and it "returns" to whatever called it -- what was I doing when I decided it was time to brush my teeth, you might think, but a computer subroutine always returns to exactly where it was before making the subroutine call.

In computer subroutines this calling behavior can get more elaborate. Subroutines can call (start) other subroutines, which can call yet others, without end. This is called "nested" calling.

The teeth cleaning subroutine ends with (we hope) clean teeth. Computer subroutines can end with (return) lots more. Suppose you're researching political revolutions and are making lots of notes. Then you decide it's important to study the 1917 Russian revolution. You put your general revolution notes to the side and dive into Russia. You start taking notes and then realize how important the Menshiviks and Bolshiviks are. You put the notes you've taken so far on 1917 on top of the general notes and dive into the next subject. When you've gone as deep down the rabbit hole as you can stand, you take the top notes from your stack on the side, and fill in the notes of what you've just learned. Eventually you pick up the next notes from the stack and so on until you're back to your revolution notes. It's also a bit similar to clicking on a web page, and then on that one and then again, and finally hitting the back arrow until you're back on the page where you started. Each new subject you dive into is like calling a subroutine, from which you can make further calls, eventually returning from each to the place you left off, only now with additional knowledge and information.

Just like in real life, when you call a subroutine you pass it information (parameters). The information you pass tells the subroutine exactly what you wish done. When the subroutine is done it can pass information back to the caller. You might pass dirty plates to the dishwasher subroutine, for example, which passes clean plates back when it returns. Or more elaborately you could pass a bunch of ingredients to the bake-a-cake routine and it would return a cake.

Some Technobabble

Computer software people aren't as clear as you'd like them to be with the names they use. The term I'm using here, "subroutine," is out of fashion these days, though still in use. In different software languages or programming communities a subroutine could be called a routine, subprogram, function, method or procedure.

Let's start with the basics of programming languages. Here's an overview of the bedrock on which everything is built, machine language, and the higher-level languages that have been built on that foundation. When you look at the memory of a computer what you see is data. It's all one's and zero's. But some of that data can be understood by the machine to be instructions. An instruction may include a reference to a memory location, an address. The instruction could ask that the data at the given address replace the value in a register, be added to it or other supported functions. There is always a "jump" instruction, which instructs the machine not to execute the next sequential instruction as it normally would, but to execute instead the instruction at the address in the instruction. There are test instructions that normally cause a jump if the specified condition is met.

The Subroutine call

Amongst the rich jumble of instruction types is one that stands out with amazing power: the subroutine call. A subroutine call, sometimes implemented directly in machine language and sometimes in multiple machine language instructions, does a couple things at once.

• It saves the address of the location after the subroutine call somewhere, usually on a stack.
• It puts some parameters someplace, often on a stack.
• It transfers control to the starting address of the subroutine.

The subroutine then:

• Executes its instructions, getting parameters from wherever they were passed to it.
• Returns to the address after the caller as stored on the stack
• Optionally gives a "return value" to the calling code.

Call by value vs. call by reference!

It's a bit esoteric, but there's an important distinction between two basic ways parameters can be passed to a subroutine: pass by value and pass by reference. Some languages can handle both kinds, while others can only handle one. This sounds abstract but it's important and easy to understand. Passing by value is like when you put the ingredients for a cake into a box, pass the box through an opening to another room, where a baker turns the ingredients into a cake and passes the cake (the value) back through the opening. Passing by reference is like when there's a larger kitchen and one of the people working there is a baker. You ask the baker to bake a cake, and the baker goes into the different storage places, gets the ingredients she needs and turns them into a cake on the spot.  In passing by value the subroutine only gets what you give it. In passing by reference, you tell the subroutine (by address pointers) what it should work with. In the peculiar world of object-oriented thinking, the equivalent of calling by reference is sacrilegious. But that's a subject for another time.

Subroutine Power

Subroutines are the first step towards reducing the size and complexity of applications, increasing their level of abstraction, and making them easier to change and enhance with minimal trouble and error. Subroutines are high on the list of fundamental concepts of computing.

The Progression Towards Abstraction on a Software Platform: Examples

I have described the way that once an application appears on a software platform, there is a consistent way the category of applications evolves on that platform. In this post I'll describe examples of this pattern. Briefly, the sequence starts with a prototype, and goes through these stages:

• Custom application
• Basic product
• Parameterized product
• Workbench product

There can be several levels of sophistication at each of these levels. The sequence is important because each step increases the speed and decreases the cost of making a body of code meet a set of customer needs.

Metasys (bought by Optum), Syntra (Clearcross)

These were medium-sized software companies, with revenues of tens of millions of dollars a year, which practically no one would have any reason to know existed today. Both these companies had a “custom application” that they were attempting to upgrade to a “basic product.” Both code bases evolved from large projects that were done on a consulting basis for a particular customer. Because both companies marketed what they had as though they were real products and failed to put the intellectual energy and money into properly upgrading their code bases, they were always coming up short in terms of features in sale situations, and when they were able to get sales, the implementation and customization efforts were harrowing to everyone concerned. The plain fact was, everyone was excited by the vision of how lovely it would be if their code were at the level of “basic product” and built a sales effort as though it were, but in the end, what each had was a “custom application” that had been severely hacked and “marketectured.”

Aurum

Aurum had a “basic product” for the emerging CRM market. At the time I looked at them with my VC firm, they were trying to bring it to the parameterized level. Because of the need for extensive customization in the CRM market, most of their implementations involved source code modifications, which made support and upgrading to new releases a major challenge. We initially chose not to invest because the company failed to recognize this. Later, with new management, the company at least recognized the issue, took steps to correct it and made business arrangements to minimize the impact on customers. We then tried to invest, but didn’t have the opportunity. The company went public and was a good win for its investors, but failed to get its product to the next level, and ended up selling to what at the time was one of the major ERP companies, Baan.

Paysys VisionPLUS

This credit card processing product was a “parameterized product,” and met all the relevant criteria. The company gained ground against the competition before and during the time I was CTO of the company because it fully met the criteria for “parameterized product,” while the competition was somewhere between that and “basic product.” Customers were able to make the product perform many more functions without source code changes due to this fact, an advantage the market appreciated. If the parameters that are made available correspond to the kind of changes you want to make, a parameterized product is ideal, and worlds better than a basic product.

The VisionPLUS product had its origin in two earlier products that were created by the company. The first of these products was CardPac, a fairly parameterized product for processing bank cards, like Visa and MasterCard. The product enjoyed a good deal of success at a time when most such card processing was performed by custom applications, which only the largest institutions could afford to write. Cardpac was sold to smaller banks, and parameters were introduced to reduce the cost of customization, maintenance and support.

A couple of retail institutions approached the company and asked it to modify Cardpac so that it could process retail (closed-loop) cards. After a couple of failed attempts, the company produced a completely new product for this purpose, Vision21. This product enjoyed considerable success among high-end retailers.

Finally, there was very large processor, Household International, that was running multiple copies of both products, separate because they had been customized for a variety of reasons, for example to support methods of credit that were unique to a market (for example “hire-purchase” in South Africa). While Paysys had failed to create a generic bank/retail product when confronted with the generic problem, unifying multiple bodies of related code into a single, highly parameterized code base proved to be a far more tractable problem, particularly with a single important customer who insisted that these variations were the only ones to worry about. The unified product was called VisionPLUS.

The industry quickly rallied to this new product that could be directed at so many different problems with such relative ease. While “parameterized product” may sound like an abstract concept, it translates directly into business advantage compared to more primitive product types, by enabling the product to be customized, installed, upgraded and maintained with less labor, less time and lower risk of error.

Paysys was bought by a major card processor, First Data, when First Data needed to expand into international markets with tough requirements. First Data's code base at the time was written in assembler language and was pretty much at the basic product level with minimal parameterization. Buying the Paysys COBOL code with its parameterized power was a major step forward for them. The number of accounts managed by VisionPLUS expanded by a factor of four under its new owner, to over 600 million.

Pivotal

Pivotal was early in the CRM market with a workbench-type product. Not all markets value this implementation method equally. CRM is one of the markets that values it most highly, because nearly every installation needs to undergo substantial customization. The benefits to customers were the ability to migrate their applications to new platforms as they emerged (e.g., Pivotal automatically converted Windows apps to browser apps) and the ability to extensively alter data, screens and application logic without source code changes. These characteristics took a good deal of effort on the company’s part to explain to customers. Their early efforts were much too technical, and didn’t resonate. Once they reached a level of market penetration, however, reference selling was the key. A potential customer would talk with an existing Pivotal customer who had first installed someone else’s application that seemed similar to Pivotal’s, which in fact it probably was, as it came out of the box. Then the reference would talk about all the time, effort and money to make simple-sounding customizations. Out would go the competing product, and in would come Pivotal. Suddenly customizations were fast, low-effort and low-cost. End of sales effort. Take order.

This implementation worked in a text-book manner for Pivotal, and is typical of the pattern. The company went public in 1999. However, its further growth was greatly limited by the fact that they chose to operate exclusively within the confines of the Microsoft Office product suite.

ERP products

The major ERP products have been at various stages of parameterized and workbench for a number of years. At one point, SAP seemed to be on the way to dominating the field, and armies of consultants were engaged in years-long projects to use SAP’s proprietary language, ABAP4, to modify and customize the base system to meet customer needs. Then along came PeopleSoft, originally just a vendor of HR software, with a complete ERP suite. While the software was not as richly functional out of the box as SAP’s, everyone knew by this point that no one used the software out of the box anyway – you had to wait for years while endless customization took place. What PeopleSoft had that was different was a set of development tools, PeopleTools, which was a generation ahead of SAP’s. SAP’s ABAP4 was at the time your basic 4GL, and you could program nearly anything with it, but you did have to spend a great deal of time programming. The PeopleSoft alternative, while no technical break-through, was still miles ahead of SAP in terms of ease of use and programmer productivity; they actually had a screen painter!

PeopleSoft convinced many buyers to care more about the implementation level of the product than the base functionality, and convinced those buyers that they were farther along the scale to a truly workbench product (although that term was not used) than the alternatives. As a result, they enjoyed years of outstanding growth. They were bought by Oracle for over $10 billion.

Conclusion

While not the only factor in success, companies whose products are higher up the tree of abstraction than their competitors enjoy a clear strategic advantage over their customers. As products evolve on a platform, the ones that appear or migrate to new levels of product abstraction tend to be more successful than ones at a lower level. While the discussion within a company is often about this or that customer or feature, raising the discussion to a new level of abstraction and acting on it can provide a huge competitive boost to a company's fortunes.