The Black Liszt

Barriers to Software Innovation: Radiology 2

Value-creating innovations are rarely the result of a bright new A-HA moment, though an individual may have that experience. A shocking number of innovations are completely predictable, partly because they've already been implemented -- but put back in the vast reservoir of ready-to-use innovations, or implemented in some other domain. This fact is one of the most important patterns of software evolution.

Sometimes the innovation is created, proven and fully deployed in production, like the optimization method Linear Programming, which I describe here. In other cases, like this one, the innovation is built as a functioning prototype with the cooperation of major industry players -- but not deployed.

In a prior post I described how I went to the San Francisco bay area in the summer of 1971 to help a couple of my friends implement a system that would generate a radiology report from a marked-up mark-sense form. We got the system working to the point where it could generate a customizable radiologist's report from one of the form types, the one for the hand. Making it work for all the types of reports would have been easy -- we demonstrated working software, and wrote a comprehensive proposal for building the whole system. It was never built.

True to the nature of software evolution, the idea probably pounded on many doors over the years, always ignored. But about 10 years ago, a pioneering radiologist in Cleveland came up with essentially the same idea. Of course, instead of paper mark-sense forms, the radiologist would click on choices on a screen, and would usually look at the medical image on the computer screen. This enabled the further benefit of reducing the work, and letting doctors easily read images that were taken in various physical locations. Tests showed that doctors using the system were much more productive than those who worked in the traditional way. Finally, they decided that mimicking the radiologist's normal writing style was a negative, and that the field would be improved by having all reports follow a similar format, with content expressed in the same order in the same way. This was actually a detail, because the core semantic observations would be recorded and stored in any case, enabling a leap to a new level of data analytics. It also, by the way, made the report generation system much easier to build than the working prototype we had built decades earlier, which enabled easy customization to mimic each radiologist's style of writing.

The founding radiologist was a doctor, of course, and knew little about software. He did his best to get the software written, got funding, and got the system working. Professional management was hired. My VC group made an investment. Many people saw the potential of the system; it was adopted by a famous hospital system in 2015. But in the end, the company was sold off in pieces.

Nearly 50 years after software was first written that was able to produce medical imaging diagnostic reports quickly and reliably while also populating a coded EMR to enable analytics, the system is sitting in the vast reservoir of un-deployed innovations. It can be built. It saves time. It auto-populates an EMR.

Many people have opined on why this particular venture failed to flourish. It's a classic example of the realities of software innovation and evolution. The reasons for failure were inside the company and outside the company. For the inside reasons, let's just say that the work methods of experienced, professional managers in the software development industry lead to consistently expensive, mediocre results. Nonetheless, the software worked and was in wide production use, delivering the advertised benefits. For the outside reasons, let's say that, well, the conditions weren't quite right just yet for such a transformation of the way doctors work to take place.

The conditions that weren't right just yet for this and uncountable other innovations add up to the walls, high and thick, behind which a reservoir of transformative innovation and "new" software awaits favorable conditions. In other words, the reservoir of innovations wait for that magic combination of software builders who actually know how to build software that works, with a business/social nexus that accepts the innovation instead of the standard no-holds-barred resistance.

Corporations promote what they call innovation. They are busily hiring Chief Innovation Officers, creating innovation incubation centers, hanging posters about the wonders of innovation, etc. etc. They continue to believe the standard-issue garbage that innovation needs to be invented fresh and new.

The reality is that there is a vast reservoir of in-old-vations that are proven and frequently deployed in other domains. All that's needed is to select and implement the best ones. HOWEVER, a Chief Innovation Officer is STILL needed -- to perform the necessary function of identifying and breaking down the human and institutional barriers that have prevented the in-old-vations from being deployed, in many cases preventing roll-out for -- literally! -- decades!!

The Comes-before is not Causation Fallacy in Software Evolution

Practically no one understands software, so why should we expect anyone to understand the broader and in some ways more demanding subject of software evolution? But while rarely (as in, almost never) studied, thoughts about software evolution are often found lurking in the backs of people's minds, and pop out in the things they write.

In biological evolution, putting aside nasty things like DNA and all the stuff it can and can't explain, there is a now-obvious pattern in the fossil record of how species have evolved. There is a clear progression from simple to complex, less capable to more capable and so on. Cold-blooded lizards are pretty neat, but there's little doubt that warm-blooded mammals rule, and that mammals appear in the fossil record after the cold-blooded animals. The picture of this nearly all of us share is that the later creatures evolved from earlier ones, in a more-or-less continuous progression. No leap-frogging is involved -- just a step-by-step process of learning and mostly improving.

A recent example in Wired Magazine about the Apollo 11 computer system illustrates this way of thinking very well. Here's the title:

Her Code Got Humans on the Moon -- and Invented Software Itself

The Apollo code in general and Margaret Hamilton in particular "invented software itself." Really? Let's look at some of the article. Here's a key paragraph:

By mid-1968, more than 400 people were working on Apollo’s software, because software was how the US was going to win the race to the moon. As it turned out, of course, software was going to help the world do so much more. As Hamilton and her colleagues were programming the Apollo spacecraft, they were also hatching what would become a $400 billion industry.

The last clause of the above quote clearly states that the $400 billion software industry that eventually emerged was "hatched" from the Apollo effort. Like so many people, the author thinks that the 400 person effort to build Apollo software somehow made possible, laid the groundwork for, or something all the developments to follow.

What a pile of hogwash. But typical!

My personal initiation into programming was on an IBM 360 model 50 computer that I was able to access on Saturdays during the 1966 school year when I was in high school. I wrote FORTRAN programs, key-punched them into cards, fed them into the card reader, got direct feedback from the computer about my imperfections as a programmer, and finally got results I wanted. While doing this, I was one of many people at many facilities using this software.

The IBM System 360 effort was started about 1960, and was announced in 1964. Gene Amdahl was the main hardware architect, tasked with the then-unprecedented task of creating a family of computers with a wide range of capabilities that could all run the same instruction set, i.e., run the same software.

Operating system software for such a system had never been created, and was a difficult task, particularly because supporting running multiple programs at the same time, again a first, was a requirement. The software was much-delayed, and by the time Fred Brooks took over the effort, more than 1,000 people were working on the project. Brooks later went on to write a classic book inspired by the effort, the Mythical Man-Month.

The System 360 was wildly successful, and fueled the incredible growth of IBM. Its instruction set lives on to this day in the System z mainframe servers. Its use was exploding at the time the Apollo software was still being developed.

What about the Apollo software? It was peculiar from the beginning, and exhibited issues that had already been solved in OS/360 and its programs. While the independent software industry did indeed grow and mature on S/360 computers, in spite of IBM's efforts to keep anyone from buying any software except their own, the Apollo software led to nothing. The now-lauded Margaret Hamilton tried to commercialize the special software methods she had used on the Apollo project, but her efforts failed and led to nothing. There were a couple of later attempts to build and commercialize what we can now see are versions of her ideas, for example in model-driven development, but after a period of hype they went nowhere.

None of this, by the way, is intended in any way to minimize the achievement of the Apollo mission, the pioneering computer system and software that was an integral part of it, or of Margaret Hamilton herself. She led a huge team that had to get software for a novel system and mission working the first time. It's a real achievement! But it's hardly "inventing software itself."

Even if you believe that someone "invented software itself," the inventor was certainly not Ms. Hamilton; even if you believe that a big, early software effort "hatched" today's software industry, it was not the Apollo effort.

So did IBM do it? Is IBM OS/360 the progenitor of today's software industry? Nope! A growing number of groups were designing and building computers, and even more were building software. You can see patterns, but this biological evolutionary idea of a capability being somehow created in DNA, and through inheritance and random improvement leading to subsequent efforts in a consequential way is nothing but the application of an inapplicable metaphor. While patterns can be observed, each piece of software is written from scratch, based on the ideas and experience of the authors. No software-equivalent of DNA, with the obvious exception of the use of shared code.

I doubt this little rant will be sufficient to stamp out the use of bad logic and inapplicable metaphors by people who insist on writing about software, even though they've never seen it, can't read it, and are clueless about what it really is and how it works. But at least writing this has gotten it out of my system, kind of...

The Slow Spread of Linear Programming Illustrates How In-old-vation in Software Evolution Works

There is loads of talk about “innovation.” Lots of people want to do it, lots of people think they’re doing it, consultants run courses in how to be innovative, and large organizations claim to promote innovation and be innovative. The assumption behind most of this “innovation” talk is that a wonderful bright idea that will change the world (or at least your organization or startup) can pop into anyone’s head. It’s new! It’s brilliant! We’re going to win big with this great new idea! See this for example.

When you study software evolution, you get an entirely different picture of software-based “innovation.” Software evolution shows you that new ideas that work are extremely rare. Oh sure, there’s a flood of new ideas popping into people’s heads all the time. Mostly, they’re not new, and the new ones rarely work. The software concepts that make it big are, in the vast majority of cases, clear examples of existing patterns of software evolution, and have in most cases already been implemented in a different context.

I first encountered the mystery of software evolution while in my first job programming software.

I started programming in a course I took starting in 1966 in high school, taught by a math teacher who couldn’t himself program, but had convinced the school to let him teach the course, and had convinced a local company, a pioneering rocket engine company called Reaction Motors, to give us computer time on Saturdays. I had a textbook about FORTRAN, a steady stream of programming assignments and a computer on which to test my programs. It was great! I continued programming the following summer, as part of an NSF math camp I was able to attend. As I was nearing high school graduation the following year, I got lucky; Diane, a high school friend, talked about me with her father, who got me connected with a nearby company, and I landed a job there for the summer before starting at Harvard College in the fall of 1968.

The company was EMSI, Esso Mathematics and Systems, Inc. in Florham Park NJ.

They were a service company, one of the about 100 units of the Standard Oil of NJ (Esso) companies, devoted to applying math and computers to improving every aspect of the company. I was immediately thrown into the group that was developing optimization models for oil refinery operation. Our focus was on the giant refinery in Venezuela.

What we had running was an implementation of a classic OR (Operations Research) algorithm called LP (linear programming), solved via the simplex algorithm first devised by George Danzig in 1948. In this kind of model, there is a goal equation and a set of constraints. The goal equation calculated profits, using hundreds of contributing variables, including prices you could sell things for, and the costs of various inputs. The constraints were greater/less than equations, each essentially describing some tiny aspect of how the refinery worked. What the algorithm did was find the values of the variables that maximized the goal equation (profits) while satisfying all the constraint equations.

The model was constantly being modified to make it more precise and applicable to actual refinery operations. I had a variety of jobs, including writing new code, fixing bugs, etc.

Since prices were such a key part of the LP model, we had a separate program to calculate what they were likely to be in the future, a Monte Carlo model. I also made enhancements and fixed bugs in this body of code.

I was fortunate to be able to hitch a ride to work and back with a PhD who worked there and lived in my town. During the ride I would tap his deep knowledge of things. He put me onto the various journals in which advances in various forms of OR were described, which I dove into. The math was often above my head, but I was motivated to teach it to myself on a rolling, as-needed basis.

I thought this was a really cool way of running things. There were all sorts of controls in the refinery, controls that let you create more airline fuel and less heating oil, or any number of other trade-offs. It was amazing that you could compute the setting of all the control knobs that would produce the best mix of products that the market needed. Why would you ever run any operation any other way?? It would be simply ignorant and stupid.

I finished school, learned more about the way the world worked, and searched high and low for implementations of LP. Anywhere! If they were out there, they were doing a great job of hiding.

I was confused. How could this be? LP was math, doggone it. It yielded a provably optimal way of running a business. It’s proven in real-life production at Esso. Any other way of operating a business was clearly seat-of-the-pants, wet-finger-in-the-wind amateur hour; anyone whose operation was sizable enough to justify the effort would have to use this method if they weren’t plain-and-simple incompetent. But no one seemed to be using it! What’s going on here??

This mystery was on my mind while I participated in one of the periodic AI crazes that has swept the world of with-it people. While I was still in college, Winograd published his MIT SHRDLU research, in which an “intelligent” robot would converse with a human in English about a world consisting entirely and solely of blocks. You could ask SHRDLU to “put the red block on top of the blue block” and it would do it; you could ask it questions about block world, and it would answer. Amazing! Super-practical! While this was happening, I wrote and submitted a thesis about how to structure knowledge inside of an intelligent robot. All of it useless compared to LP and the associated OR techniques.

The craze was AI, and the mania lasted a few more years, generating a steady stream of "promising" results, none of them in the same universe of practicality and benefit as LP or any other OR optimization technique, which continued to be used only in "secret" little islands of astounding efficiency and productivity.

Later in the 1970’s I first applied for a home mortgage. A key part of getting the mortgage was the interview with the loan officer. You had to pass muster to get the mortgage! Another 10 years passed before OR-type models started to be used for credit. When I next applied for a mortgage in 1981, I was interviewed by a loan officer. By my next mortgage in 1987 it was at least partly automated.

When I got into venture capital in the 1990’s, I discovered that high-value repair parts were had recently been optimized in terms of inventory levels and locations. I looked in detail at the pioneering company ProfitLogic, which did inventory and sale optimization for retail stores, answering questions like when should which products be put on what kind of sale, questions that had traditionally been answered by the local marketing “expert,” just as oil refineries had previously been run exclusively by experienced experts.

Only in the late 2010’s did exactly the same LP models start being applied to medical scheduling, to optimize the use of things like infusion centers and operating rooms. Just as oil refineries produced much more value from exactly the same crude oil inputs in 1968 as a result of LP models, so are infusion centers handling 30% more throughput using exactly the same capital and human resources using the very same LP models.

Here’s the mystery: why did it take so blankety-blank long for LP models to be applied??? There has been no theoretical break-through. Yes, the computers are less expensive, but given the scale of the opportunity, that was never the obstacle. WHY??!! The answer is simple: there is no good answer. Except of course for the ever-relevant one of human ignorance, stupidity and sloth.

The example of LP optimization and its agonizingly long roll-out through different applications and industries over more than 50 years – a roll-out that is far from complete! – is a prime example of the reality of computer/software evolution. Among other things, it illustrates the point that many of the most impactful "innovations" are really "in-old-vations," things that are just sitting there, proven in production and waiting for someone to apply them to one of the many domains which they would benefit.

Here are a couple cornerstones of computer software evolution:

• Software evolution resembles biological evolution only a little. Not-very-fit software species thrive in broad areas of application, unchallenged for years or decades, while vastly superior ones rule the roost not far away. The only reason why the superior software species don’t migrate to the attractive new place appears to be human ignorance and inertia.
• Software evolution resembles the much-derided theory of “intelligent design” quite a bit, if you make a slight edit and call it “un-intelligent, un-educated design.” A “superior” (HA!) being does indeed do the designing of the software, in the form of highly paid software professionals.
• When software appears in a new “land” (platform, business domain), it most often starts evolution all over again, first appearing in classic primitive forms, and then slowly re-evolving through stages already traversed in the past in other “lands.” This persistent phenomenon supports the “unintelligent design with blinders on" theory of software evolution.

I will explain and illustrate these points in future posts and a forthcoming book.