The Black Liszt

The Dimension of Software Automation Breadth Examples

One of the major ways software evolves is by increasing along the dimension of automation breadth. A domain can be dominated by products at a given breadth of automation, and suddenly a existing or new competitor starts winning by increasing its breadth of automation, offering its customers more value for less effort and money. It's a classic move and a good way for new entrants to disrupt a market.

One of the most frequently given pieces of advice, including by me, is to “focus,” i.e. basically solve fewer problems, try to satisfy a narrower range of customers, etc. While this advice is applicable more often than not, the natural and recurring progression of products through the spectrum of “automation breadth” makes it clear that, sometimes, when the conditions are right, the winning strategy is to be among the first to increase the breadth of automation that you incorporate into your product.

Example: Athena Health

A clear example of this is shown by the story of AthenaHealth. At the time the founders started the company, a wide variety of products were already available to run physician offices, from small single-office practices to extended medical groups. These products generally ran on inexpensive machines that the practice would keep in some back room, and would support multiple users via a LAN or terminals. Most of the products were sold by license, so that the office had to pay only a modest price for the license, and then annual maintenance.

Along comes little Athenahealth, with a better way of doing things. Athena had a cool new practice management system (PMS). Unlike all PMS’s at the time, it was built using internet technologies, so that it could be operated as a service, with people at the office accessing the system using machines with browsers and internet connections. Athena took care of the computers, relieving the medical office of a burden it basically didn’t want.     

But they ran into a little problem: the people in charge of medical practices are doctors, and doctors really don’t care about PMS’s – they care about patients and medicine. A PMS is a necessary evil, something you should buy for as little as you can get away with and ignore until things get so bad you are forced to buy a new one. Money spent on the PMS is just money out of the doctors’ pockets, as far as they’re concerned. Oh, you have a “better” one, do you, whatever that means? Stop wasting my time.

The folks at Athena noticed that one little thing the PMS does is produce bills and claims, the purpose of which is to get patients and insurance companies to send them money. No claim, no money. Unfortunately, merely producing the claims rarely proves to be sufficient to get the money flowing. People are required to do special things to the claims, provide additional information, harass the payers, etc. This is so specialized and time-consuming that it either consumes the time of a number of people at the office, or is outsourced to a “billing service.”

The Athena folks went on to notice additional important things: (1) the chances of getting paid are a direct reflection of the quality and appropriateness of the information on the claim; (2) the PMS and how it is used is the main source of this information; (3) by actually performing the billing service, you can learn how to produce a better PMS that produces better claims, increasing the effectiveness of the billing service while reducing the cost of running it at the same time. Finally, they found out that there is something the doctors who run medical practices care about other than medicine – surprise, surprise, that something is money.

So Athena introduced an outsourced billing service, but required practices that use it to also use their practice management system – at no additional cost! And they got so good at collecting the money that doctors could essentially get more money and a really cool, state-of-the-art PMS (like they cared…) for free!

This is a nice story for Athena, but the point of telling it here is that it illustrates the principle of product automation breadth evolution. While products are evolving within a “level” of automation breadth (i.e., how many of an organization’s functions it automates), it is normally a good idea to maintain discipline, avoid distractions, and concentrate on automating that function. But at a certain point in the evolution of each product category, pretty much everyone in a space has automated everything within that function, and everyone is reduced to concentrating on sales strategies and niggling little details. At that point, and PMS’s were at that point when Athena came along, it makes sense to do what you’re normally supposed to avoid – look for another function inside the organization to automate, particularly if there are synergies in implementing the two functions within a single framework, as there certainly were in this case.

Example: Bank and Retail Credit Cards

In 1983 a small company called CCS (Credit Card Software, later called Paysys) released a body of COBOL code that would enable a bank to process credit cards. A number of small and regional banks bought copies of the code and ran it successfully. The code was enhanced over the years.

A major retailer, Michael's Jewelers, approached CCS and asked if they could make a version of the bank software that could handle purchases from their stores, including a variety of payment plans and financing options offered by the store that were not supported by bank card software.

The company's programmers quickly gave up on the idea of modifying the bank code to handle the problem. Many aspects of bank card processing, such as the difference between issuing and acquiring, were irrelevant to retail. In addition, the many financing options supported by retailers went far beyond anything banks did. So they borrowed from the bank code to the extent that it helped and ended up creating a separate body of software called Vision 21. Once it was available, it proved to be a big success in the market, and was quickly enhanced by customer demand to include all the options desired by retailers Before long it supported the needs of retailers in other countries as diverse as Japan and South Aftrica.

Finally, there was very large processor, Household International, that was running multiple copies of both products, separate because they had been customized for a variety of reasons, for example to support methods of credit that were unique to a market (for example “hire-purchase” in South Africa). While CCS, now called Paysys, had failed to create a generic bank/retail product when confronted with an example of the generic problem, unifying multiple bodies of related code into a single, highly parameterized code base proved to be a far more tractable problem, particularly with a single important customer who insisted that these variations were the only ones to worry about.

The industry quickly rallied to this new product, called Vision PLUS, that could be directed at so many different problems with such relative ease. For example, it enabled retailers to issue co-branded cards that worked like regular bank cards, except when used in the issuer's retail store, when it acted like a classic store card with features like "90 days same as cash" options that bank cards don't support. While “parameterized product” may sound like an abstract concept, it translates directly into business advantage compared to more primitive product types, by enabling the product to be customized, installed, upgraded and maintained with less labor, less time and lower risk of error.

The company that built Vision PLUS was bought by First Data, a major card processor. The reason is interesting: in spite of having thousands of programmers (Paysys had only dozens), First Data was unable to modify their US-centric code base to handle processing in Japan. At the time of the sale, Vision PLUS was processing about 150 million cards world-wide. The code lives on and currently runs over 600 million cards.

Conclusion

These are two examples of companies growing by broadening their focus -- in a strategic way, driven by just a couple of representative customers. In neither case did they address a whole new market at the beginning, though that was the eventual goal. They took their existing software along with a cooperative customer and met that customer's needs. Athena started with just one specialty in one state with a single payer. Paysys started with a single existing customer. In each case they grew the broadening of their focus a step at a time, making each customer happy as they went. As they grew, word got around in the industry, and they shifted to saying "no" to the vast majority of inquiries in order to maintain the step-wise customer success they were building.

This is a classic pattern of focus broadening that can bring transformative success to companies when handled well.

 

The Dimension of Automation Depth in Information Access

I have described the concept of automation depth, which goes through natural stages starting with the computer playing a completely supportive role to the person (the recorder stage) and ending with the robot stage in which the person plays a secondary role. I have illustrated these stages with a couple examples that illustrate the surprising pain and trouble of going from one stage to the next.

Unlike the progression of software applications from custom through parameterized to workbench, customers tend to resist moving to the next stage of automation for various reasons including the fear of loss of control and power.

Automation depth in Information Access

Each of the patterns of software evolution I've described are general in nature. I’ve tried to give examples to show how the principles are applied. In this section, I’ll show how the entire pattern played out in “information access,” which is the set of facilities for enabling people to find and use computer-based information for business decision making.

Built-in Reporting

“Recorder” is the first stage of the automation depth pattern of software evolution. In the case of information access, early programs were written to record the basic transactions that took place; as part of the recording operation, reports were typically produced, summarizing the operations just performed. For example, all the checks written and deposits made at a bank would be recorded during the day; then, at night, all the daily activity would be posted to the accounts. The posting program would perform all the updates and create reports. The reports would include the changes made, the new status of all the accounts, and whatever else was needed to run the bank.

At this initial stage, the program that does the recording also does the reporting. Reporting is usually thought to be an integral part of the recording process – you do it, and then report on what you did. Why would you have one program doing things, and a whole separate program figuring out and reporting on what the first program did? It makes no sense.

What if you need reports for different purposes? You enhance the core program and the associated reports. What if lots of people want the reports? You build (in the early days) or acquire (as the market matured) a report distribution system, to file the reports and provide them to authorized people as required.

Efficiency was a key consideration. The core transaction processing was “touching” the transactions and the master files; while it was doing this, it could be updating counters and adding to reports as it went along, so that you wouldn’t have to re-process the same data multiple times.

Report Writers

The “power tool” stage of automation depth had two major sub-stages. The first of these was the separation of reporting from transaction processing. Information access was now a key goal in itself, and was so important and done so frequently that specialized tools were built to make it easy, which is always the sign that you’re into the “power tool” phase.

This first generation of power tools were specialized software packages generally called “report writers.” The power tool was directed at the programmer who had to create the report. Originally, the language that was used for transaction processing was also used for generating the report. The most frequent such language was COBOL. The fact that COBOL was cumbersome for this purpose was reflected in the fact that specialized syntax was added to COBOL to ease the task of writing reports. But various clever people saw that by creating a whole new language and software environment, the process of writing reports could be tremendously enhanced and simplified. These people began to think in terms of reporting itself, so naturally they broke the problem into natural pieces: accessing the data you want to report on, processing it (select, sort, sum, etc.), and formatting it for output.

The result of this thinking was a whole industry that itself evolved over time, and played out in multiple environments and took multiple forms. The common denominator was that they were all software tools to enable programmers to produce reports more quickly and effectively than before, and were complete separate from the recorder or transaction processing function.

At the same time, data storage was evolving. The database management system emerged through several generations. This is not the place for that story, which is tangential to the automation depth of information access. What is relevant is that, as the industry generally recognized that information access had moved to the report writer stage of automation, effort was made to create a clean interface between data and the programs that accessed the data for various purposes.

Data Warehouse and OLAP

Report writers were (and are) important power tools – but they’re basically directed at programmers. But programmers are not the ultimate audience for most reports; most reports are for people charged with comprehending the business implications of what is on the report and taking appropriate action in response. And the business users proved to be perennially dissatisfied with the reports they were getting. There was too much information (making it hard to find the important things), not enough information, information organized in confusing ways (so that users would need to walk through multiple reports side-by-side), or information presented in boring ways that made it difficult to grasp the significance of what was on the page. And anytime you wanted something different, it was a big magilla – you’d have to get resources authorized, a programmer assigned, suffer through the work eventually getting done, and by then you’d have twice as many new things that needed getting done.

As a result of these problems, a second wave of power tools emerged, directed at this business user. These eventually were called OLAP tools. The business user (with varying levels of help from those annoying programmers) had his own power tool, giving him direct access to the information. Instead of static reports, you could click on something and find out more about it – right away! But with business users clicking, the underlying data management systems were getting killed, so before long the business users got their own copy of the data, a data warehouse system.

In a sign of things to come, the business users noticed that sometimes, they were just scanning the reports for items of significance, and that it wasn’t hard to spell out exactly what they cared about. So OLAP tools were enhanced to find and highlight items of special significance, for example sales regions where the latest sales trends were lower than projections by a certain margin. This evolved into a whole system of alerts.

Predictive Analytics

OLAP tools are certainly power tools, but the trouble with power tools is that you need power users – people who know the business, can learn to use a versatile tool like OLAP effectively, and can generate actions from the information that help the business. So information access advanced to the final stage in our general pattern, the “robot” stage, in which human decision making is replaced by an automated system. In information access, that stage is often called “predictive analytics,” which is a kind of math modeling.

As areas of business management are better understood, it usually turns out that predictive analytics can do a better, quicker job of analyzing the data, finding the patterns, and generating the actionable decisions than a person ever could. A good example is home mortgage lending, where the vast majority of the decisions today are made using predictive analytics. Many years ago, a person who wanted a home mortgage would make an appointment with a loan officer at a local savings bank and request the loan. The officer would look at your information and make a human judgment about your loan worthiness.

That “power user” system has long since been supplanted by the “robot” system of predictive analytics, where all the known data about any potential borrower is constantly tracked, and credit decisions about that person are made on the basis of the math whenever needed. No human judgment is involved, and in fact would only make the system worse.

Predictive analytics is the same in terms of information utilization as the prior stages, but the emphasis on presenting a powerful, flexible user interface to enable a power user to drive his way to information discovery is replaced by math models that are constantly tuned and updated by the new information that becomes available.

Sometimes the predictive analytics stage is held back because of a lack of vision or initiative on the part of the relevant industry leaders. However, a pre-condition for this approach really working is the availability of all the relevant data in suitable format. For example, while we tend to focus on the math for the automated mortgage loan processing, the math only works because it has access to a nationwide database containing everyone’s financial transactions over a period of many years. A power user with lots of experience, data and human judgment will beat any form of math with inadequate data; however, good math fueled with a comprehensive, relevant data set will beat the best human any time.

Conclusion

All these stages of automation co-exist today. One of the key rules of computing is that old programs rarely die; they just get layered on top of, given new names, and gradually fade into obscurity. There are still posting programs written in assembler language that have built-in reporting. In spite of years of market hype from the OLAP folks, report writing hasn’t gone away; in fact, some older report writers have interesting new interactive capabilities; OLAP and data warehouses are things that some organizations aspire to, while others couldn’t live without them; finally, there are important and growing pockets of business where the decisions are made by predictive analytics, and to produce pretty reports for decision-making purposes (as opposed to bragging about how well the predictive analytics are doing) would be malpractice.

Even though all these stages of automation co-exist in society as a whole, they rarely co-exist in a functional segment of business. Each stage of automation is much more powerful than the prior stage, and it provides tangible, overwhelming advantages to the groups that use it. Therefore, once a business function has advanced to use a new stage of information access automation, there is a “tipping point,” and it tends to become the new standard for doing things among organizations performing that function.

The Dimension of Software Automation Breadth

Computers and software are all about automation. See this for the general principles of automation. When you dive into details and look at lots of examples, patterns emerge. The patterns amount to sequences or stages of automation that emerge over time, with remarkable consistency. When you apply your knowledge of the pattern to the software used in an industry at any given time, you can identify where the software is in the sequence. You can confirm the earlier stage in the sequence and predict with accuracy the stage that will follow. This gives you ability to say what will happen, though not when or by whom.

The patterns of automation play out in multiple dimensions. I have described what may be called the depth of automation, in which software evolves from recording what people do through helping them and eventually to replacing them.

In this post I will describe another dimension, which may be thought of as the breadth of automation. The greater the automation breadth the more functions are incorporated into the automation software and the more highly integrated the functions are with each other.

The Dimension of automation breadth

Automation breadth has these basic levels, independent of the automation depth of the products involved:

Component	Not a stand-alone product, but a component that could be incorporated into many custom applications and/or products
Point product	Implements a single function
Product collection	A group of point products from a single vendor
Product suite	An integrated set of separate products, with meaningful benefits from the integration
Integrated product	A single body of source code that performs a variety of related functions that would otherwise require separate products, with meaningful benefits from the unification
Integrated product with selective outsourcing	A product that is written and delivered in such a way that the using organization can choose to have the vendor staff a number of functions off-site.

Components rarely appear first in historical terms, but they are the beginning of this sequence. Typically, functionality that is very difficult to write or whose requirements change rapidly is separated out and delivered in the form of a component. The quality of the component may become so important that it becomes an industry standard.

Example: The Ocrolus service (disclosure: Oak HC/FT is an investor) is a classic example of a valuable, narrow component. It takes images of documents of nearly any kind, recognizes them, extracts their data and returns the data to the component user. This functionality is so challenging to write, and the consequence of errors so great, that most applications that need the functionality are likely to use the component, which is delivered as a service, rather than write their own.

In a time of rapidly emerging functionality, point products are usually first, because they can be gotten to market quickly. Buyers typically talk in terms of “best of breed,” but have the problem of negotiating and maintaining relationships with multiple vendors, who frequently have conflicting interests. The buyer also has to take responsibility for integrating the various products he has bought. Buyers reasonably worry about whether their typically small vendor will stay in business and continue to invest in their product.

Example: Captura (now part of Concur) provided a point product to enable a company to automate the process of entering, approving and paying expense reports.

Product collections solve some of these problems. There is now a single vendor; the vendor is probably much larger and more likely to stay in business; the products will be maintained and there is no conflict of interest. Once a product collection becomes available in a category, buyers will typically prefer an adequate product from a collection to a superior point product. Product collections can be formed by acquisition.

Example: CA, Computer Associates, is a typical vendor that acquires products in a category and puts them together into a product collection.

It is often desirable to share data among the products in a collection. Frequently, they maintain copies of essentially the same information, have essentially the same security roles defined, etc. Users have to go through considerable time and effort to accomplish this on their own, and then again when there is a new release, and desirable integrations are sometimes not even possible. Product suites solve these problems to a large extent, since a single vendor performs the integration and sells the integration as part of the product collection. Once a product suite becomes available in a category, buyers will typically prefer an adequate product suite to a product collection whose products are superior, because the cost of installation and maintenance is lower and the benefits of integration often outweigh the benefits of individual product features. Building a product suite typically requires source-code level coding and functionality design changes, but the code bases of the products can be separate.

Classic example: Microsoft Office was one of the first product suites for office products. While the benefits of integration are not overwhelming in this functional area, users clearly benefit from having a suite rather than individual products. Once the benefits of a suite became accepted, buyers no longer wanted individual office products.

Recent example: The market for business formation has long been dominated by the point product Legalzoom. A new, rapidly growing product suite called Zenbusiness (disclosure: Oak HC/FT is an investor) is taking classic advantage of moving to the next stage of automation breadth, by offering small business customers not only business formation services, but also services for websites, banking and accounting.

Not all functionality areas benefit from having an integrated product, but for those that do, integrated products are better for vendors (reduced costs due to elimination of redundancy) and for buyers (a simpler, more unified product that is easier to learn and operate, and has deep, fully-automated function integration), and typically win in the marketplace.

Example: Everyone thinks of SAP’s R3 as being the first client/server ERP product, but its real break-through was being the first truly integrated product on which a large enterprise could run its business. Using a single body of code and a common shared DBMS, its many modules each ran different parts of the enterprise business. In spite of the high cost of implementation and operation, the advantages of running your business on a single integrated product were overwhelming. Like most projects to build a unified product, the vendor had deep domain knowledge, it took a long time to get it right, and the early installations were painful.

Once functionality markets reach a level of maturity, the competition becomes intense, and organizations often have to choose areas of distinction, outsourcing the functions they choose not to compete in to a low-cost provider. Products that enable organizations to selectively outsource in this way typically take business from products that must be fully staffed by the buyer.

Example: FDC is the leading processor for credit cards in the US. With a billion cards outstanding, the market is huge and highly competitive. While the technology base of the FDC product is decades old, the functionality it delivers is highly sophisticated. In addition to delivering a completely integrated, multi-department product, FDC offers off-site staffing for the various departmental functions, where the staff sounds on the telephone as though they worked for you.

Conclusion

The dimension of automation breadth helps us understand the evolution of software and the progression that naturally takes place in a given market space. The companies that win are most often the ones that dominate a narrow market segment with a particular product and then broaden the range of software they can sell to a given organization. They typically move step by step according to the progression I have described here.

Software Programming Language Evolution: Libraries and Frameworks

We've talked about the major advances in programming languages and the enhancements that brought those languages to a peak of productivity -- a peak which has not been improved on since. Nonetheless there has been an ongoing stream of new programming languages invented, each of which is claimed to be "better" -- with no discussion about what constitutes "goodness" in a language! This is high on the list of causes of the never-ending chaos and confusion that pervades the world of software. While loads of people focus on language, the most important programming tools that make HUGE contributions to the productivity of programmers using them goes largely unremarked, in spite of the fact that they are part of every programmer's day-to-day programming experience. What are these essential, always-used but largely in the background thing? Libraries and frameworks.

Libraries

A library is a collection of subroutines, often grouped by subject area, which perform commonly used functions for programs. Every widely used language has a library that is key to its practical use.

For example, the C language has libraries that contain hundreds of functions for

• input and output, including formatting
• string manipulation and character testing
• math functions
• standard functions, dozens of them
• date and time

There are libraries of functions for controlling displays and input devices and many other things. After all these years and the "fatal" flaw of not being object-oriented (gasp!!), it is currently the #2 language in popularity. While the libraries don't play as important part of its popularity as certain other languages, they are still important.

Python became widely used among people doing analytics not particularly because of the virtues of the language itself, but because it grew one of the richest modern libraries of routines for calculations available. It ended up with great support for statistics and the things you need to do with large number sets including array and matrix manipulation. When doing analytics, the library functions do nearly all the heavy lifting. All that the program has to do is deploy the rich set of functions against the problem in the right way!

A 2021 survey of language popularity confirms the popularity of Python, and the importance of its library in giving it its popularity. Here's a summary:

Have you ever heard of the language R? It's an open-source language which became usable in the year 2000. Chances are if you work in statistics, data analytics, operations research or other areas involving serious data manipulation you know all about it and probably use it. The language itself has valuable features for math programming that most languages don't have, for things like vectors and matrices. More importantly it has an amazing rich library of the R equivalent of libraries, called packages. R packages contain reusable collections of functions and data definitions to perform valuable calculations. When you're trying to do something you're pretty sure someone else has tried to tackle in some way, the first thing you do is look for an appropriate package. Packages are available for most common data science tasks; they do much of the work and nearly all the "heavy lifting" of performing any job for which the package applies. R is a prime example of a language having virtues, but the bulk of the value and productivity coming from the available libraries.

One of the sad turns taken in programming language evolution was driven by the ascendancy of the modern RDBMS. Ironically, the DBMS emerged at about the same time as do-everything language environments dominated the new minicomputer landscape. With a single environment like MUMPS or PICK you could get a whole job done -- user interface, core program, data storage and access, everything! These environments enabled massive productivity gains. See this for more. But the DBMS blasted in, took over, and HAD to be used. So 3-GL's that were less productive than the new environments became even less so by finding cumbersome ways to use DBMS's. The way they did it I explain here. This was a first major step down the degenerate, productivity-killing road of layers. How did 3-GL's pull off the integration? With libraries, of course! For example, Java's JDBC library became a requirement for burdensome, error-prone access to standard databases from within Java, while those who didn't mind the overhead and "impedance mismatch" could use one of the ORM's (Object-Relational Mapping systems) that emerged.

Not all libraries are tightly associated with a language! One such library is redis.io, an open source project started by an Italian developer to build a powerful in-memory key/value datastore and cache. It has taken off and has many more powerful features including queuing. While unknown among non-programmers, it is incredibly popular, widely used and available on the major cloud providers.

Other libraries are directed towards a particular aspect of programming. With the emergence of the web UI, a succession of libraries to ease their creation have emerged.

A UI library that has become dominant is React, created and then open-sourced by Facebook. It is specifically for the Javascript language and greatly enhances the productivity of building and the resulting performance of web UI's. The benefits of javascript or any other language are trivial compared to the productivity gain produced by using React. Unlike most libraries, React comes close to being a framework; it resembles an R package, in that it's a comprehensive solution to the problem of building web UI's. Interestingly, part of the programmer productivity gain is due to the fact that it has a major declarative aspect to its design, like AngularJS (see below).

Bottom line: while the details of software language features have some impact on productivity, the availability and richness of libraries enhances productivity many times over. It's no contest.

Frameworks

Frameworks take the idea of libraries but with an important reversal. Libraries are sets of routines, any of which may be called by a program written in a supported language. The program written in the language is completely in charge. Frameworks, by contrast, provide an environment in which a language can operate. You select exactly one framework to work in. When you play by its rules, you normally enjoy large productivity benefits.

A library is a rich set of resources covering many issues, like a book library. A framework is a selected set of resources enabling rapid work on a given category of issues, like a kitchen for cooking.

One impressive framework is the RAILS framework for the language Ruby. While the Ruby language had become fairly successful, Ruby on Rails (as it was called) established it as a player because many groups discovered that it was many times faster than other tools at building a web application that has a UI and a database. The reason is simple: normally to build such an application you have one expert creating the UI probably using javascript and some library, another one building the server-side business logic, and another one creating the DBMS -- with a great deal of work and trouble relating the database to the server program; see for example JDBC and ORM's above. With RAILS, you defined your database, which automatically defined the names you use inside Ruby to access the data. Similar story with the UI.

The philosophy of RAILS centered on the DRY principle -- Don't Repeat Yourself.

Unlike the endless repetition of the usual layered environment, RAILS took a major step towards the principle of Occamality. Was RAILS original, a real break-through? In its object-oriented environment, yes. In programming in general, no. RAILS is a typical example of the appalling lack of knowledge of history in software. Most of the benefits of RAILS were delivered in a more integrated way many years before by the Powerbuilder environment. I discuss the context here.

What comprehensive frameworks like this REALLY do is attempt to re-create the comprehensive, everything-in-one-place development environments that emerged to enhance programmer productivity beyond what was achievable in 3-GL's, mostly by incorporating data storage and user interface functions, as I explain here. Their emergence and widespread adoption is clear evidence of the power of eliminating the insanity of layers in software.

There are also frameworks that are much narrower in scope, addressing only part of the programming puzzle. While they leave much of an overall programming job alone, they can give a major boost to the narrow area on which they focus. One that is focused exclusively on building web UI's  that caught on and became widely used is AngularJS, a new version of which is simply Angular. This framework is highly declarative, focused more on describing the elements of the UI than the actions required to implement it.

This led to HUGE programmer productivity gains. Why would anyone build a web UI from scratch? Nonetheless, the similarities between the ReactJS library and the AngularJS framework are strong, and both of them powered huge programmer productivity gains that had very little to do with the virtues (or lack thereof) of the associated language, javascript.

Languages vs Libraries and Frameworks

 New languages continue to flow out of the creative minds of groups of programmers who obviously don't have enough to do to keep themselves fully occupied. Each new language tends to be lauded, if only by its creators, with extreme claims of virtue on many dimensions. None of these claims are ever subjected to verification and testing, much less of the rigorous kind. In any case, there is simply no contest between the gains delivered by a rich library or framework and a new programming language.

I'll just give glaring example. One of the main virtues that Object-oriented languages are supposed to have is code reuse. The concept that is bandied about is that good class system is like a set of Lego blocks, enabling new programs to be easily assembled. Mostly it doesn't happen. For re-use, libraries and frameworks are the gold standard. Think of a normal, old-fashioned book library. The whole reason they exist is that people reuse books! It's the same thing for software libraries -- code gets into software libraries because it's re-used often! That's how they got to be called "libraries." Duh.

 

Software Programming Language Evolution: the Structured Programming GOTO Witch Hunt

In prior posts I’ve given an overview of the advances in programming languages, described in detail the major advances and defined just what is meant by “high” in the phrase high-level language. I've described the advances in structuring and conditional branching that brought 3-GL’s to a peak of productivity.

The structuring and branching caught the attention of academics. Watch out! What happened next was that a theorem was proved, a movement was declared and named, and a certain indispensable part of any programming language, the GO TO statement, was declared to be a thing only used by bad programmers and should be banned. Here's the story of the nefarious GOTO.

Structures in Programming Languages

I've described how structures were part of the first 3-GL's and how they were soon elaborated to more clearly express the intention of programmers, making code even more productive to write. The very first FORTRAN compiler, delivered in 1957, included primitive versions of conditional branching and loops, two of the foundations of programming structure. It was so powerful that the early users figured it decreased the number of statements needed to achieve a result more than 10 times.

These are the people who actually WRITE PROGRAMS! They want to make it easier and jumped on anything that gave a dramatic improvement.

“Significantly, the increasing popularity of FORTRAN spurred competing computer manufacturers to provide FORTRAN compilers for their machines, so that by 1963 over 40 FORTRAN compilers existed. For these reasons, FORTRAN is considered to be the first widely used cross-platform programming language.”

Before long, the structuring capabilities of the original IF (conditional branching) and DO (controlled looping) statements were enhanced and augmented to something close to their current form. I describe this here. The result was a peak of programmer productivity that has not substantially been increased since, and often been degraded.

The Bohm-Jacopini Theorem

Completely independent of the amazing advances in languages and programming productivity that were taking place, math-oriented non-programmers were hard at work deciding how software should be written. Here is the story in brief:

The structured program theorem, also called the Böhm–Jacopini theorem,^[1][2] is a result in programming language theory. It states that a class of control-flow graphs (historically called flowcharts in this context) can compute any computable function if it combines subprograms in only three specific ways (control structures). These are

1. Executing one subprogram, and then another subprogram (sequence)
2. Executing one of two subprograms according to the value of a boolean expression (selection)
3. Repeatedly executing a subprogram as long as a boolean expression is true (iteration)

The structured chart subject to these constraints may however use additional variables in the form of bits (stored in an extra integer variable in the original proof) in order to keep track of information that the original program represents by the program location. The construction was based on Böhm's programming language P′′.

The theorem forms the basis of structured programming, a programming paradigm which eschews goto commands and exclusively uses subroutines, sequences, selection and iteration.

This theorem got all the academic types involved with computers riled up. The key to good software has been discovered! The fact that math theorems are incomprehensible to the vast majority of people, and the fact that perfectly good computer programs can be written by people who aren't math types didn't concern any of these self-anointed geniuses.

The important thing to note about the theorem is that it was NOT created in order to make programming easier or more productive. It just "proved" that it was "possible" to write a program under the absurd and perverse constraints of the theorem to compute any computable function. Assuming you were willing to use a weird set of bits to store location information in ways that would make any such program unreadable by any normal person. Way to go, guys -- let's go back to the days of writing in all-binary machine language!

The Crisis in Software and its solution

Not long after this, the academic group of Computer Science “experts” formed. They had a conference. They looked at the state of software and declared it to be abysmal. The whole conference was about the "crisis" in software. See this for details.

One of the most prominent of those Computer Scientists was Edsger W. Dijkstra. He looked at the powerful constructs for conditional branching, loops and blocks that had been added to 3-GL's and invented the term "structured programming" to describe them. He related those statements to the wonderful but useless math proof about the minimal requirements for programming a solution to any "computable function." The proof "proved" that such programs could be written without the equivalent of a GOTO statement. BTW, I do not dispute this. He wrote "the influential "Go To Statement Considered Harmful" open letter in 1968."

Among the solutions to the software crisis they proclaimed was strict adherence to the dogma of what Dijkstra called “structured programming,” which prominently declared that the GOTO statement had no place in good programming and should be eliminated.

Does the fact that's it is POSSIBLE to program a solution to any computable function without using GOTO mean that you SHOULD write without using GOTO's? When children go to school, it's POSSIBLE for them to crawl the whole way, without using "walking" at all. Everyone accepts that this is possible. When you're on your feet all sorts of bad things can happen -- you can trip and fall! Most important, you can get the job done without walking ... and therefore you SHOULD eliminate walking for kids getting to school. QED.

This is academia for you – a prime example of how Computer Science works hard to make sure that programs are hard to write, understand and deliver, all in the name of achieving the opposite.

The debate about structured programming

There was no debate about the utility of the conditional branching, controlled looping and block structures that rapidly became part of any productive software language. They were there and programmers used them, then and now. The debate was about "structured programming," which by its academic definition outlawed the use of the GOTO statement. That wasn't all. It also outlawed having more than one exit to a routine, breaks from loops and other productive, transparent and generally useful constructs.

I remember clearly as a programmer in the 1980's having a non-technical manager type coming to me and quizzing me about whether I was following the rigors of structured programming, which was then talked about as the only way to write good code. I don't remember my answer, but since I knew the manager would never go to the trouble of actually -- gasp! -- reading code, my answer probably didn't matter.

The most important thing to know about the leader of the wonderful movement to purify programming is his lack of interest in actually writing code:

Fortunately, there are sane people in the world, including the incomparable Donald Knuth (an academic Computer Scientist who's actually great!) and a number of others.

An alternative viewpoint is presented in Donald Knuth's Structured Programming with go to Statements, which analyzes many common programming tasks and finds that in some of them GOTO is the optimal language construct to use.^[9] In The C Programming Language, Brian Kernighan and Dennis Ritchie warn that goto is "infinitely abusable", but also suggest that it could be used for end-of-function error handlers and for multi-level breaks from loops.^[10] These two patterns can be found in numerous subsequent books on C by other authors;^{[11][12][13][14]} a 2007 introductory textbook notes that the error handling pattern is a way to work around the "lack of built-in exception handling within the C language".^[11] Other programmers, including Linux Kernel designer and coder Linus Torvalds or software engineer and book author Steve McConnell, also object to Dijkstra's point of view, stating that GOTOs can be a useful language feature, improving program speed, size and code clarity, but only when used in a sensible way by a comparably sensible programmer.^[15][16] According to computer science professor John Regehr, in 2013, there were about 100,000 instances of goto in the Linux kernel code.^[17]

Any programmer can make mistakes. Any statement type can be involved in that mistake. For example, I think nearly everyone accepts that cars are a good thing. But over 30,000 people a year DIE in car accidents! So where's the movement to eliminate cars because of this awful outcome! It makes as much sense as outlawing the GOTO because sometimes it's used improperly. Like every other statement type.

Software Programming Language Evolution: Structures, Blocks and Macros

In prior posts I’ve given an overview of the advances in programming languages, described in detail the major advances and defined just what is meant by “high” in the phrase high-level language. In this post I’ll dive into the additional capabilities added to 3-GL’s that brought them to a peak of productivity.

History

Let’s remember what high-level languages are all about: productivity! They are about the amount of work it takes to write the code and how easy code is read.

The first major advance, from machine language to assembler, was largely about eliminating the grim scut-work of taking the statements you wanted to write and making the statements “understandable” to the machine by expressing them in binary. Ugh.

The second major advance, to 3-GL’s like FORTRAN and COBOL, was about eliminating the work of translating from your intention to the assembler statements required to express that intention. A single line of 3-GL code can easily translate into 10 or 20 lines of assembler code. And the 3-GL line of code often comes remarkably close to what you actually want to “say” to the computer, both writing it and reading it.

FORTRAN achieved this goal to an amazing extent.

“with the first FORTRAN compiler delivered in April 1957.^[9]:75 This was the first optimizing compiler, because customers were reluctant to use a high-level programming language unless its compiler could generate code with performance approaching that of hand-coded assembly language.^[16]”

“While the community was skeptical that this new method could possibly outperform hand-coding, it reduced the number of programming statements necessary to operate a machine by a factor of 20, and quickly gained acceptance.”

The reduction in the amount of work was the crucial achievement, but just as important was the fact that each set of FORTRAN statements were understandable, in that they came remarkably close to expressing the programmer’s intent, what the programmer wanted to achieve. No scratching your head when you read the code thinking to yourself “I wonder what he’s trying to say here??” This meant easier to write, fewer errors and easier to read.

Enhancing conditional branching and loops

The very first iteration of FORTRAN was an amazing achievement, but it’s no surprise that it wasn’t perfect. At an individual statement level it was nearly perfect. When reading long groups of statements there were situations where the code was clear, but the intention of the programmer not clearly expressed in the code itself – it had to be inferred from the code.

The first FORTRAN had a couple of the intention-expressing statements: a primitive IF statement and DO loop. Programmers soon realized that more could be done. The next major version was FORTRAN 66, which cleaned up and refined the early attempts at structuring. Along with the appropriate use of in-line comments, it was nearly as clear and intention-expressing as any practical programmer could want.

The final milestone in the march to intention-expressing languages was C.

It’s an amazing language. While FORTRAN was devised by people who wanted to do math/science calculations and COBOL by people who wanted to do business data processing, C was devised to enable computer people to write anything – above all “systems” software, like operating systems, compilers and other tools. In fact, C was used to re-write the first Unix operating system so that it could run on any machine without re-writing. C remains the language that is used to implement the vast majority of systems software to this day.

I bring it up in this context because C added important intention-expressing elements to its language that have remained foundational to this day. It enhanced conditional statements, creating the full IF-THEN-ELSE-ENDIF that has been common ever since. This meant you could say IF <condition is true> THEN <a statement>. The statement would only be executed if the condition was true. You could tack on ELSE <another statement> ENDIF. This isn’t dramatic, but the only other way to express this common thought is with GOTO statement – which certainly can be understood, but takes some figuring. In addition, C added the ability to use a delimited block of statements wherever a single statement could be used. When there are a moderate number of statements in a block, the code is easy to read. When there would be a large number, a good programmer creates a subroutine instead.

C added a couple more handy intention-expressing statements. A prime example is the SWITCH CASE BREAK statement. This is used when you have a number of conditions and something specific to do for each. The SWITCH defines the condition, and CASE <value> BREAK pairs define what to do for each possible <value> of the SWITCH.

Lots of following languages have added more and more statements to handle special cases, but the cost of a more complex language is rarely balanced with the benefit in ease and readability.

The great advance that's been ignored: Macros

C did something about all those special cases that goes far beyond the power of adding new statements to a language, and vastly increases not just the power and readability of the language but more important the speed and accuracy of making changes. This is the macro pre-processor.

I was already very familiar with macros when I first encountered the C language, because they form a key part of a good assembler language – to the extent that an assembler that has such a facility is usually called a macro-assembler. Macros enable you to define blocks of text including argument substitution. They resemble subroutine calls, but they are translated at compile time to code which is then compiled along with the hand-written code. A macro can do something simple like create a symbol for a constant that is widely used in a program, but which may have to be changed. When a change is needed, you just change the macro definition and – poof! – everywhere it’s used it has the new value. It can also do complex things that result in multiple lines of action statements and/or data definitions. It is the most powerful and extensible tool for expressing intention and enabling rapid, low-risk change in the programmer’s toolbox. While the C macro facility isn’t quite as powerful as the best macro-assemblers, it’s a zillion times better than not having one at all, like all the proud but pathetic modern languages that wouldn’t know a macro if it bit them in the shin.

The next 50 years of software language advances

The refrain of people who want to stay up to date with computers is, of course, "What's new?" Everyone knows that computers evolve more quickly than anything else in human existence, by a long shot. The first cell phones appeared not so long ago, and they were "just" cell phones. We all know that now they're astounding miniature computers complete with screens, finger and voice input, cameras and incredible storage and just about any app you can think of. So course we want to know what's new.

The trouble is that all that blindingly-fast progress is in the hardware. Software is just along for the ride! The software languages and statements that you write are 99% the same as in the long-ago times before smart phones. Of course there are different drivers and things you have to call, just as in any hardware environment. But the substance of the software and the lines of code you use to write it are nearly the same. Here is a review of the last 50 years of "advances" in software. Bottom line: it hasn't advanced!

Oh, an insider might argue: what about the huge advances of the object-oriented revolution? What about Google's new powerhouse, Go? Insiders may ooh and ahh, just like people at a fashion catwalk. The come-back is simple: what about programmer productivity? Claims to this effect are sometimes made, but more often its that the wonderful new language protects against stupid programmers making errors or something. There has not been so much as a single effort to measure increase of programmer productivity or quality. There is NO experimental data!! See this for more. Calling what programmers do "Computer Science" is a bad joke. It's anything but a science.

What this means is simple: everyone knows the answer -- claims about improvement would not withstand objective experiments -- and therefore the whole subject is shut down. If you're looking for a decades-old example of "cancel culture," this is it. Don't ask, don't tell.

Conclusion

3-GL's brought software programming to an astounding level of productivity. Using them you could write code quickly. The code you wrote came pretty close to expressing what you wanted the computer to do with minimal wasted effort. Given suitable compilers, the code could run on any machine. Using a 3-GL was at least 10X more productive than what came before for many applications.

A couple language features were incorporated into the very first languages, like conditional branching and controlled looping, that were good first steps. The next few years led early programmers to realize that a few more elaborations of conditional branching and controlled looping would handle the vast majority of practical cases. With those extra language features, code became highly expressive. All subsequent languages have incorporated these advances in some form. Sadly, the even more productive feature of macros has been abandoned, but as we'll see in future posts, their power can be harnessed to an even greater extent in the world of declarative metadata.

The Dimension of Software Automation Depth Examples

I have described the concept of automation depth, which goes through natural stages starting with the computer playing a completely supportive role to the person (the recorder stage) and ending with the robot stage in which the person plays a secondary role. I will illustrate these stages with a couple examples that illustrate the surprising pain and trouble of going from one stage to the next. Unlike the progression of software applications from custom through parameterized to workbench, customers tend to resist moving to the next stage of automation for various reasons including the fear of loss of control and power.

I will use companies to illustrate this progression that are known to me personally but not widely known. Software history generally is the history of the winners as written by the winners, like most history. However if you want to guide your own software strategy by taking advantage of the clear patterns of history, it is essential to include examples of companies that are rarely discussed.

Nextpage

NextPage was active from 1999 to around 2010. It had a distributed document management product that was prominent in service industries, for example lawyers and accountants. The product operated at a “recorder” level in terms of automation depth. It wasn’t directive, it just sat there waiting for a document request, and when it got one, responded with a list of applicable documents and lets you view the documents.

They and some of their larger customers saw that, in fact, many of their engagements weren’t so very different. As a high-powered, highly paid lawyer, you may think that every job is unique and your ability to handle the differences very important, but it was obvious to people involved that certain transactions involved very similar documents and workflows. It made sense to attempt to capture and automate the similarities.

With the cooperation of one of the world’s largest law firms, NextPage built a new product on top of the existing distributed document platform; this new product was automation at the “power tool” level. While recognizing that each transaction of a particular type would end up producing a unique set of documents, for example, it operated like a series of factory distribution belts, assuring that all lawyers working on an M&A transaction, for example, worked on the right things in the right order with the right base documents, reviewed in the right order by the right people, etc. etc. Lawyers continued to craft their documents. But instead of treating each (for example) M&A transaction as though it were the first one the firm had ever handled, the new system treated them like a repeatable process, with variations.

But the lawyers didn’t think of it that way at all! They were used to thinking of themselves as powerful, autonomous, self-directing agents. Now they were to be subservient to an assembly line in a factory, having their work and output measured as though they were hourly workers? No way – I’m a professional – there’s no way I’m going to put my head in that yoke!

NextPage had a good position in the market, existing customers, a prestigious lead customer for the product, a strong business case, and tangible results. But the resistance to moving to a “power tool” level of application was stronger than the force NextPage was able to bring to bear at that time, and the effort failed to bear fruit. Anyone who looks at the application knows it’s going to happen, the benefits are definitely there. But anyone with sense would be reluctant to predict even the decade when it would be likely to actually happen.

The pattern of resistance encountered by Nextpage was strongly correlated with the prestige and power of the people whose work would be affected. The same resistance is seen elsewhere; for example, how eager are the highly paid and prestigious category of medical doctors to have their work automated?

Captura/Concur

My VC firm invested in Captura, which is now part of SAP Concur, the leader in on-line expense management systems. These applications basically automate the process of filling out expense reports, getting them checked and approved, and finally paid.

If you think about expense reports, you imagine a form that you fill in with your expenses during a trip. You then attach your receipts, drop it off, and eventually get reimbursed for out-of-pocket expenses. So it seems like it would be a “recorder” type application. In fact several companies were started and eventually failed with products that weren’t much more than recorder type applications, with some automatic routing added on. This kind of application didn’t provide enough value to offset the expense of installing and running it.

Captura went much farther. While parts of this application operate at the “recorder” level, particularly the entry of un-automated expense information, much of it actually operates at the “robot” level, which is why the payback for its use is so high. It has built-in rules for approval levels and documentation requirements and many other things. It knows whether and when and to whom expense reports should be routed for approval from the HR system. It gets feeds from the corporate card system. It kicks out exceptions. It allocates expenses to the right accounts. It feeds directly into the accounts payable system to cut checks. When it needs something it can’t get from a computer, it asks a human for it, and otherwise does its job without help.

Unlike other “power tool” type applications, this application tended not to threaten people who were powerful in the adopting organizations. The most powerful people who used it tended to regard it as saving them time, and everyone got their reimbursements more quickly and accurately. The administration group saw it as saving thankless clerical work, and the accounting executives saw automatic enforcement of all their standards. Still, this application category didn’t really take off until the cost of implementation was reduced to a painless level; in other words, it had to wait for nearly universal access to the web to be practical and affordable. And it worked much better as a secure service than as an enterprise application, because the burden of installation and maintenance on IT added so much friction that only the most motivated potential customers would go for it.

ClickSoftware

ClickSoftware provides a classic example of a “robot” level application. It takes the problem of a field service operation, in which there are calls for service from companies in different locations, with different equipment to be repaired, with varying levels of urgency and service level agreements, with different hours of operation and times to travel to them, and finally with different service technicians who have varying skills and qualifications and constraints about hours of work. Who gets sent to which location to fix which problem in which order?

Without something like Click Software, this problem is solved in a rudimentary way by the service manager, who uses some combination of experience and common sense to work out the best solution. However, for a large service organization, the difference between a good solution and the optimal solution can be worth a great deal of money. Click Software takes all those inputs and produces the optimal solution, and asks the human manager for help when the problem simply can’t be solved; in response, for example, the human may ask a critical resource to work overtime, or might call a customer and see if their request can be deferred.

Click illustrates another characteristic common to robot-type applications. It is typical that “power tool” type applications do not become robot type ones by incremental addition. A robot application requires a whole new approach to the problem, and a level of math programming that is likely to be entirely foreign to the software group that wrote the power tool application.

Tape backup systems

In tape backup, whenever a new platform has come out, tape backup software products tend to emerge in the same order, starting with products that just record what the backup operator does, moving to power products that give the operator hints and suggestions and automate the repetitive operations, and ending with true robot products, which are driven by a set of rules and which request operator intervention at only and exactly those times when human intervention is required. For example, Palindrome and Cheyenne (now part of CA) introduced robot-level backup products in the early 1990’s for the PC server platform, at a time when similar products had been in general use on earlier platforms (e.g. IBM mainframe) for decades.

Having cycled from primitive to robot level automation several times as new platforms came out that had tape backup, the whole category is dying, frozen in time as the latest computer systems use the kind of disks that have become incredible cheap and tape has become obsolete.

Conclusion

It seems inevitable that an industry would move, step by step, towards robot-level automation. The historical fact is that while the incentives for advancement are always there, advancement happens only when an industry is "ready" to move, which can be decades after it is technically possible and economically practical.

The Dimension of Software Automation Depth

I have discussed the fundamental concept of automation.

Software automates human effort to varying degrees. In doing so, software emerges that performs this automation to an increasing extent. In this post I'll describe the basic stages through which automation progresses. In later posts I'll give specific examples.

Automation depth

Automation depth can be understood in terms of the extent to which human effort, observation, knowledge and control is replaced by the computer system.

Stages: help a human do something; do something a human would otherwise have to have done; make a decision a human would have made; gather information about all the work that is arriving, in process and completed; make work and resource allocation decisions. There are a few different  ways automation depth can be understood. Here are a couple of them:

• Stages: As depth increases, the human does things faster and has less work to do.
• Automation depth is correlated with moving from an “open loop” system to a “closed loop” system.
• Effort. In the earlier stages the computer augments or replaces the person’s efforts. This is like power steering in a car, in which everything is the same except turning the wheel takes less effort.
• Knowledge. As automation depth increases, less knowledge is in the heads of people and more in the computer.
  
• Control. As automation depth increases, the human loses control and eventually becomes controlled by the computer.

Automation depth has these basic levels:

Recorder	The software essentially tracks what people do and records the results of what they create or decide. The most primitive software of this class does this for a single job task. More advanced software does it for multiple tasks, eventually for a person’s whole job.
Power tool	The software takes high-level instructions from the operator, and does most of the work. In less skilled environments, the power tool will often select and order the work to be done.
Robot	Like an auto-pilot, the software replaces the human for most functions, performing those functions better than a human could, leaving the humans only to set goals and processing rules, and handle exceptions.

Unlike some of the software evolution progressions I've described, iIt is not generally a quick win to get to the next level of automation depth. An industry can stay stuck at a level of automation depth for decades. Normally, the people directly involved with an application strongly resist moving to the next level of automation depth, because they see it as deeply threatening to their autonomy, power or skills.

If a company tries to go to the next level, it is essential that it understand its position, and have the backing to last out the transition and the flexibility to keep trying until it establishes a beach-head from which it can expand. The benefits have got to be dramatic and tangible. Feel-good stuff tends not to work here. Some people are going to get bent out of shape, and managers of managers have got to impose the solution. The only way that typically happens is if things are arranged for their risk of failure to be low and the rewards great, and indisputable when they actually happen.

I will give examples of this progressions in future posts.

The Progression Towards Abstraction on a Software Platform: Examples

I have described the way that once an application appears on a software platform, there is a consistent way the category of applications evolves on that platform. In this post I'll describe examples of this pattern. Briefly, the sequence starts with a prototype, and goes through these stages:

• Custom application
• Basic product
• Parameterized product
• Workbench product

There can be several levels of sophistication at each of these levels. The sequence is important because each step increases the speed and decreases the cost of making a body of code meet a set of customer needs.

Metasys (bought by Optum), Syntra (Clearcross)

These were medium-sized software companies, with revenues of tens of millions of dollars a year, which practically no one would have any reason to know existed today. Both these companies had a “custom application” that they were attempting to upgrade to a “basic product.” Both code bases evolved from large projects that were done on a consulting basis for a particular customer. Because both companies marketed what they had as though they were real products and failed to put the intellectual energy and money into properly upgrading their code bases, they were always coming up short in terms of features in sale situations, and when they were able to get sales, the implementation and customization efforts were harrowing to everyone concerned. The plain fact was, everyone was excited by the vision of how lovely it would be if their code were at the level of “basic product” and built a sales effort as though it were, but in the end, what each had was a “custom application” that had been severely hacked and “marketectured.”

Aurum

Aurum had a “basic product” for the emerging CRM market. At the time I looked at them with my VC firm, they were trying to bring it to the parameterized level. Because of the need for extensive customization in the CRM market, most of their implementations involved source code modifications, which made support and upgrading to new releases a major challenge. We initially chose not to invest because the company failed to recognize this. Later, with new management, the company at least recognized the issue, took steps to correct it and made business arrangements to minimize the impact on customers. We then tried to invest, but didn’t have the opportunity. The company went public and was a good win for its investors, but failed to get its product to the next level, and ended up selling to what at the time was one of the major ERP companies, Baan.

Paysys VisionPLUS

This credit card processing product was a “parameterized product,” and met all the relevant criteria. The company gained ground against the competition before and during the time I was CTO of the company because it fully met the criteria for “parameterized product,” while the competition was somewhere between that and “basic product.” Customers were able to make the product perform many more functions without source code changes due to this fact, an advantage the market appreciated. If the parameters that are made available correspond to the kind of changes you want to make, a parameterized product is ideal, and worlds better than a basic product.

The VisionPLUS product had its origin in two earlier products that were created by the company. The first of these products was CardPac, a fairly parameterized product for processing bank cards, like Visa and MasterCard. The product enjoyed a good deal of success at a time when most such card processing was performed by custom applications, which only the largest institutions could afford to write. Cardpac was sold to smaller banks, and parameters were introduced to reduce the cost of customization, maintenance and support.

A couple of retail institutions approached the company and asked it to modify Cardpac so that it could process retail (closed-loop) cards. After a couple of failed attempts, the company produced a completely new product for this purpose, Vision21. This product enjoyed considerable success among high-end retailers.

Finally, there was very large processor, Household International, that was running multiple copies of both products, separate because they had been customized for a variety of reasons, for example to support methods of credit that were unique to a market (for example “hire-purchase” in South Africa). While Paysys had failed to create a generic bank/retail product when confronted with the generic problem, unifying multiple bodies of related code into a single, highly parameterized code base proved to be a far more tractable problem, particularly with a single important customer who insisted that these variations were the only ones to worry about. The unified product was called VisionPLUS.

The industry quickly rallied to this new product that could be directed at so many different problems with such relative ease. While “parameterized product” may sound like an abstract concept, it translates directly into business advantage compared to more primitive product types, by enabling the product to be customized, installed, upgraded and maintained with less labor, less time and lower risk of error.

Paysys was bought by a major card processor, First Data, when First Data needed to expand into international markets with tough requirements. First Data's code base at the time was written in assembler language and was pretty much at the basic product level with minimal parameterization. Buying the Paysys COBOL code with its parameterized power was a major step forward for them. The number of accounts managed by VisionPLUS expanded by a factor of four under its new owner, to over 600 million.

Pivotal

Pivotal was early in the CRM market with a workbench-type product. Not all markets value this implementation method equally. CRM is one of the markets that values it most highly, because nearly every installation needs to undergo substantial customization. The benefits to customers were the ability to migrate their applications to new platforms as they emerged (e.g., Pivotal automatically converted Windows apps to browser apps) and the ability to extensively alter data, screens and application logic without source code changes. These characteristics took a good deal of effort on the company’s part to explain to customers. Their early efforts were much too technical, and didn’t resonate. Once they reached a level of market penetration, however, reference selling was the key. A potential customer would talk with an existing Pivotal customer who had first installed someone else’s application that seemed similar to Pivotal’s, which in fact it probably was, as it came out of the box. Then the reference would talk about all the time, effort and money to make simple-sounding customizations. Out would go the competing product, and in would come Pivotal. Suddenly customizations were fast, low-effort and low-cost. End of sales effort. Take order.

This implementation worked in a text-book manner for Pivotal, and is typical of the pattern. The company went public in 1999. However, its further growth was greatly limited by the fact that they chose to operate exclusively within the confines of the Microsoft Office product suite.

ERP products

The major ERP products have been at various stages of parameterized and workbench for a number of years. At one point, SAP seemed to be on the way to dominating the field, and armies of consultants were engaged in years-long projects to use SAP’s proprietary language, ABAP4, to modify and customize the base system to meet customer needs. Then along came PeopleSoft, originally just a vendor of HR software, with a complete ERP suite. While the software was not as richly functional out of the box as SAP’s, everyone knew by this point that no one used the software out of the box anyway – you had to wait for years while endless customization took place. What PeopleSoft had that was different was a set of development tools, PeopleTools, which was a generation ahead of SAP’s. SAP’s ABAP4 was at the time your basic 4GL, and you could program nearly anything with it, but you did have to spend a great deal of time programming. The PeopleSoft alternative, while no technical break-through, was still miles ahead of SAP in terms of ease of use and programmer productivity; they actually had a screen painter!

PeopleSoft convinced many buyers to care more about the implementation level of the product than the base functionality, and convinced those buyers that they were farther along the scale to a truly workbench product (although that term was not used) than the alternatives. As a result, they enjoyed years of outstanding growth. They were bought by Oracle for over $10 billion.

Conclusion

While not the only factor in success, companies whose products are higher up the tree of abstraction than their competitors enjoy a clear strategic advantage over their customers. As products evolve on a platform, the ones that appear or migrate to new levels of product abstraction tend to be more successful than ones at a lower level. While the discussion within a company is often about this or that customer or feature, raising the discussion to a new level of abstraction and acting on it can provide a huge competitive boost to a company's fortunes.

 

Software Evolution: Functionality on a New Platform: Transaction Monitors

This is the fourth in a series of examples to illustrate the way that functionality that had been implemented on an older platform appears on a newer platform.

See this post for a general introduction with example and explanation of this peculiar pattern of software evolution. This earlier post contains an example in security services software, this earlier post describes an example in remote access software and this earlier post describes an example in market research.

Unlike the prior examples, this one is well known and I had no personal involvement.

Example: BEA systems

Old platform	IBM mainframe MVS
Old function	Transaction application execution enhancement: CICS
New platform	UNIX
New function	Essentially the same, with special support for UNIX-oriented applications, databases and UI’s
Outcome	Slow growth at the beginning and some market education required, but became virtually a standard, with just a couple competitors. It was acquired by Oracle in 2008 when it had about $1.5 Billion in revenue.

This is a classic example of the pattern of functionality emerging on a technology platform, and then emerging in pretty much the same way in the same order on other platforms.

IBM mainframes were originally used in a batch processing mode. They had operating systems that were really good at it. Groups of users increasingly wanted on-line access, and they wanted transaction processing control and security. The operating system didn’t provide it. So they built what the operating system thought of as an application, but which special applications that ran with it thought of as a specialized kind of operating system, a “transaction processing monitor.” Over time, the value of the main IBM TP monitor, CICS, became recognized, and it evolved to quasi-operating system status.

UNIX gets developed by a bunch of smart people at Bell Labs, and eventually becomes widely deployed on many kinds of computers. It was originally developed with interactive use in mind, and the early UNIX people would have been offended by the idea that it would need to be augmented by a primitive thing like a TP monitor. But that’s exactly what happened, and the UNIX-oriented TP monitors were built along very similar lines and for similar reasons as CICS!

The original authors of Unix ignored TP monitors when they built it in the 1970's, but AT&T had its own version of a TP monitor that ran on IBM mainframes to control its network. AT&T decided it would be cheaper to use their own computers instead of IBM mainframes to run its control software. Their computers had the Unix operating system, but transaction wouldn't work properly without a transaction monitor. So they set about building their own specifically to run on top of Unix, Tuxedo starting in 1983. It was successful and became widely used.

The founders of BEA were ex-Sun Microsystems employees who were amazingly prescient in seeing this need. But they didn't actually build anything! They started by buying Tuxedo, the Unix-centric TP monitor that had been owned by Novell since 1993. They played with other middleware companies but most importantly bought WebLogic, a TP monitor built specifically for Sun's Java language.

It was the availability of Unix-based TP monitors that helped enable the massive growth of the Unix platform during the internet explosion in the late '90's and early 2000's. I was active during this period, both with CICS applications and internet applications. The whole internet tech world, techies and investors, became convinced that certain things were required to build an application with that all-important characteristic, "scalability." It had to run on Sun computers, with the Sun version of Unix. It had to use the Oracle DBMS. And it had to be written in the Java language using the J2EE (usually spoken as jay-two-double-E) library, intended to ride on ... a TP monitor! Oracle ended up being the winner; it first acquired BEA in 2008 and shortly after acquired Sun.

The key, just as it was for CICS, was the integrity of transactions when many people are accessing a shared set of data. UNIX simply did not provide this capability, and just like on the mainframe, the capability was added on top of the operating system, with an application that the operating system thought of as a normal application, but which the applications that made use of it thought of as a specialized kind of operating system.

You might well think that this whole set of functionality is dead today. Who talks about TP monitors? Many companies showed that you don't need all the complexity and overhead of a TP monitor to build scalable applications.

In reality, it's a clear example of another strong, repeating pattern: bad ideas in software rarely die; they just fade for a bit and then re-emerge with a new name in slightly different form. Here's a good example of the morphing of data warehouse into Big Data.

I won't go into detail here, but while J2EE and classic Unix TP monitors are on life-support, the same idea of distributed applications for scalability lives on today in the form of microservices woven together with an enterprise message bus. Even though, just like 20 years ago, they're not needed for building scalable applications!

Software Programming Language Evolution: 4GL’s and more

Not long after third-generation computer languages (3-GL’s) got established, ever-creative software types started inventing the next generation. In a prior post, I’ve covered two amazing programming environments that were truly an advance. They were both widely used in multiple variations, and programs written using them continue to perform important functions today – for example powering more hospitals than any competing system. But they were pretty much stand-alone unicorns; the academic community ignored them entirely and nearly all the leading figures, experts and trend-setters in software ignored them and looked elsewhere.

Experts “in the know” directed their attention to what came to be called fourth-generation languages (4-GL’s) and object-oriented (O-O) 3-GL’s. These were supposed to be the future of software. Let’s see what happened with 4-GL's.

The background of 4-GL’s

The earlier posts in this series give background that is helpful to understand the following discussion.

In prior posts I’ve given an overview of the advances in programming languages, described in detail the major advances and defined just what is meant by “high” in the phrase high-level language. I’ve described two true advances beyond 3-GL’s. And then there were 4-GL’s, supposedly a whole generation beyond the 3-GL’s. Let’s take a look at them.

The best way to understand 4-GL’s is to look at the context in which they were invented. First, the academic types were busy at work creating languages that essentially ignored how data got into and out of the program. The first of these was Algol, followed by others. The academic community got all excited by this class of languages, but they were ignored by the large community of programmers who had to get things done with computers. That was in the background. In the foreground, modern DBMS’s were invented and commercialized.

4-GL's!

Apparently everywhere new languages sprang to life, created inside, around and on top of DBMS's. It's a revolution, a once-in-a-lifetime opportunity to become a major milestone in software history! My name could be right up there with von Neumann and Turing!

All the major DBMS vendors created their own languages, usually with snappy names like Informix 4GL and Oracle's PL/SQL. How could they fail to respond to this massive opportunity for market expansion?

Brand-new vendors popped up to take advantage of the hunger for DBMS along with the new hardware configuration of client-server computing, in which an application ran on a group of Microsoft Windows PC's, all connected with and sharing a DBMS running on a server. One startup that powered to great commercial success was a company called PowerSoft which created a product called Powerbuilder. The Powerbuilder development environment enabled you to work directly with a DBMS schema and create a program that would interact with a user and the data. The central feature of the system was an interactive component called a DataWindow, which enabled you to visually select data from the database and create a UI for it supporting standard CRUD (create, read, update and delete) functions without writing code. This was a real time-saver.

The 3-GL's Respond

Vendors of 3-GL's couldn't ignore the tumult raging outside their comfy offices. Before long support was added to most COBOL systems to embed SQL statements right in the code. Sounds simple, right? It was anything but. COBOL programs had data definitions which the majority of lines of COBOL code used. The way to handle the mis-match between SQL tables and COBOL record definitions wasn't uniform, but in many cases a single COBOL Read statement was replaced with embedded SQL with additional new COBOL code to map between the DBMS results and the data structures already in the COBOL. Ditto when data was being updated and written. Then there's the little detail that DBMS performance was dramatically worse than simple COBOL ISAM performance, since DBMS's were encumbered with huge amounts of functionality not needed by COBOL programs but which couldn't be circumvented or turned off.

Net result: the 3-GL's were worse off, by quite a bit.

What Happened?

Naturally the programming landscape is dominated by 4-GL's today, right? Or maybe their successors? How could it be otherwise? Just as each new generation of languages represented a massive advance in productivity from the earlier one and became the widely-accepted standard, why wouldn't this happen again?

It didn't happen. 4-GL's are largely of historic interest today, mostly confined to legacy code that no one can be bothered to re-write. Even the systems that genuinely provided a productivity advantage like Powerbuilder faded into stasis, rarely used to build new programs.

There is a great deal to be said about this fact. One of the factors is certainly the rise to dominance of object-oriented orthodoxy, which in spite of supposedly being centered on data definitions (classes) is nonetheless highly code-centric and has NO productivity gain over non-O-O languages. Where have you read that before? Nowhere? Probably the same place you haven't read all the studies showing in great detail how it achieves productivity gains. What can I say? Computer Non-Science reigns supreme.

Conclusion

I won't be writing a follow-up blog post on 5-GL's. Yes, they existed and were the hot thing at the time. I remember vividly all the hand-wringing in the US over the massive effort in Japan with the government funding research into fifth-generation languages. The US would be left in the dust by Japan in software, just like they're beating us in car design and manufacturing! When was the last time you heard about that? Ever?

Computers are objective things. Software either works or it doesn't. Unlike perfume, clothes or novels, it's not a mater of taste or personal preference; it's more like math. So what is it with the mis-match between enthusiasm and reality in software? It would be nice to understand it, but what's most important is to understand that much of what goes on in software is NOT based on objective right-or-wrong things like math but on fashion trends and the equivalent of Instagram influencers. Don't know anything about computer history? If you want to be accepted by the experts and elite, that's a good thing. If you want to get things done, quickly and well, ignore it at your peril.

 

Software Programming Language Evolution: Credit Card Software Examples 1

In prior posts I've discussed the nature of programming languages and their evolution. I have given an overview of the so-called advances in programming languages made in the last 50 years. Most recently I described a couple of major advances beyond the 3-GL's. The purpose of this post is to give a couple real-life examples of how amazing new 4-GL’s and O-O languages have worked out in practice.

I was CTO of a major credit card software company in the late 1990’s. Because of that I had a front-row seat in what turned out to be a rare clinical trial of the power and productivity of the two major new categories of programming languages that were supposed to transform the practice of programming. Of course no one, in academia or elsewhere, has written about this real-world clinical trial or any of the similar ones that have played out over the years.

Bank One and 4-GL's

Bank One, based in Columbus Ohio, was a major force among banks in the 1990’s. They were growing and projected a strong image of innovation. During the 1990’s the notion that applications should be based on a DBMS was becoming standard doctrine, and the companies that valued productivity over Computer Science (and internet) purity were united behind one form or another of 4-GL as the tool of choice to get things done. Together with Anderson Consulting, one of the giant consulting companies at the time, Bank One proceeded on a huge project to re-write all their credit card processing code into a 4-GL.

After spending well north of $50 million (I heard nearly $100 million) and taking over 3 years, the project was quietly shelved, though industry insiders all heard the basic story. No one had an explanation. 4-GL’s are amazing, so much better than ancient things like COBOL – and card processing is just simple arithmetic, right, with a bit of calculating interest charges thrown in. How hard could it be? Harder than a 4-GL wielded by a crack team of one of the country’s top tech consulting firms could pull off with years of time and a giant budget, I guess.

On top of everything else, they had a clear and unambiguous definition of what the 4-GL program needed to do in the form of the existing system. They had test cases and test data. This already eliminates a huge amount of work and uncertainty in building new software. Compared with most software projects, the work was simple: just do what the old program did, using existing data as the test case. This fact isolated the influences on the outcome so that the power and productivity of the 4-GL was the most important factor. Fail.

Word of this should have gotten out. There should have been headlines in industry publications. The burgeoning 4-GL industry should have been shattered. Computer Science professors who actually cared about real things should have swarmed all over and figured out what the inherent limitations of 4-GL's were, whether they could be fixed, or whether the whole idea was nothing but puffery and arm-waving. None of this happened. Do you need to know anything else to conclude that Computer Science is based on less rationality than Anna Wintour and Vogue?

Capital One and Java

Capital One was the card division of a full-service bank that was spun out in 1994, becoming an unusual bank whose only business was to issue credit cards. In just a couple years the internet boom started, and with it enthusiasm for the most prominent object-oriented language for the enterprise, java. Capital One management was driving change in the card world and presumably felt they needed a modern technology underpinning to do it fully. So they authorized a massive project to re-write their entire existing card software from COBOL to Java. I remember reading at the time that they expected incredible flexibility and the power to evolve their business rapidly from the unprecedented power of Java.

The project took a couple years and was funded to the tune of many tens of millions of dollars; the amounts were never made public. As time went on, we heard less about it. Then there was a small ceremony and the project was declared a success, a testimony to the forward-looking executive management and pioneering tech team at the company. Then silence. I poked around with industry friends and discovered that the code had indeed been put into production – but just in Canada, which was a new market for the company at the time, handling a tiny number of cards. Why? It didn’t have anywhere close to the features and processing power that the existing COBOL system had to handle the large US card base. Just couldn't do it and company management decided to stop throwing good money after bad.

Conclusion

Executives and tech teams at major corporations bought into the fantasy that the latest 4-GL's and O-O languages would transform the process of writing software. They put huge amounts of money with the best available teams to reap the benefit for their business. And they failed.

These projects and their horrible outcomes should have made headlines in industry publications and been seared in the minds of academics. Software experts should have changed their tune as a result, or found what went wrong and fixed it. None of this happened. It tells you all you need to know about the power and productivity gains delivered by 4-GL's and object-oriented languages. Nothing has changed in the roughly twenty years since these events took place except for further evidence for the same conclusion piling up and the never-ending ability of industry participants and gurus to ignore the evidence.

 

Software Evolution: User Interface

User interfaces have gone through massive evolution since their first appearance in the 1950's. Lots of people talk about this. But not many separate the main two threads of UI evolution: technical and conceptual.

The technical thread is all about the tools and techniques. Examples of elements in the technical thread are the mouse, function keys, menus, and graphical windowing systems. Advances in the technical thread of UI evolution are created by researchers, systems people and systems makers, both hardware and software. People who build actual UI’s generally have to use the tools they’ve been given.

The conceptual thread of UI evolution is about thoughts in the heads of application builders about what problem they’re trying to solve and how they’re supposed to go about solving it. Application builders are generally taught the base concepts they are supposed to use, and then usually apply those concepts throughout their careers. But all application builders don’t have the same thoughts in their heads. The thoughts they have exhibit a clear progression from less evolved to more evolved. It is interesting that the way application builders think about what job they are supposed to do is almost completely independent of the tools they have, i.e., the technical thread. Yes, they can and do use the tools available to them, but this conceptual thread of UI evolution rides “above” the level of the technical tools.

The evolution of UI on the technical side is widely discussed and understood. As hardware has gotten better and less expensive, the richness of the interaction between computer and human has increased, with the computer able to present more information to the user more quickly, and with immediate reaction on the computer’s part to user requests. For the most part, this is a good thing, although people who think only about user interfaces can make serious product design mistakes when they fail to put the user interface in the broader context of product design. For example, generally speaking, pointing at a choice with a mouse is better than entering a code on a keyboard, and giving users lots of control through a rich user interface is better than giving them no control. However, in situations where there are repetitive tasks and efficiency is very important, the keyboard beats the mouse any day of the week, and in situations where tasks performed by humans can be automated, it is far better to have the computer do it -- quickly, effectively and optimally -- rather than depending on and using the time of a human being, regardless of how wonderful his UI may be. This post goes into detail with examples on this subject.

Conceptual UI evolution, by contrast to evolution on the technical side, is not widely discussed and not generally understood. Understanding this evolution enables you to build superior software by creating software that enables tasks to be accomplished with less human effort and greater accuracy.

UI Concepts

The conceptual level of user interfaces is most easily understood by asking two questions: (1) whose perspective is the primary one in the mind of the application UI builder – the computer’s or the user’s; and (2) to what extent is the user relied upon to operate the software correctly and optimally? The most primitive UI’s “look” at things from the computer’s point of view, and, somewhat paradoxically, rely almost entirely on the user to get optimal results from the computer. The most advanced UI’s “look” at things from the user’s point of view, while at the same time imposing as little burden of intelligence and decision-making as possible on the user.

When you state it this way – a UI should be user-centered and should help the user to be successful – you may well assume that building UI’s in this way would be standard operating procedure, and that building UI’s in any other way would be considered incompetent. Sadly, this is not the case. Like all the patterns I describe in my series on software evolution, most people, companies and even industries tend to be “at” a particular stage of evolution in the subject areas I describe here; companies gain comparative advantage by taking the “next” step in the pattern evolution earlier than others, and exploit it for gain more vigorously than others.

Some of the patterns I've observed in software evolution just tend to repeat themselves historically with minor variations. Other patterns, of which this is an example, don't seem to be as inevitable or time-based. This pattern is much like the pattern of increasing abstraction in software applications, described in detail here. Competitive pressures and smart, ambitious people tend to drive applications to take the next step on the spectrum of goodness.

For UI, the spectrum can be measured. The UI that requires the least time and effort by a user to get a given job done is the best. That's it!

Do UI experts think this way? Is this a foundational part of their training and expertise? Of course not! Just because computers are involved, no one should be under the illusion that we live in a numbers-driven world. For all the talk of numbers, people are more influenced by the culture they're part of, and generally want validation from that culture. Doing something further up the UI optimization curve than is customary in their milieu is nearly always an act of rebellion, and most people just don't do it.

Software Programming Language Evolution: Beyond 3GL’s

In prior posts I’ve given an overview of the advances in programming languages, described in detail the major advances and defined just what is meant by “high” in the phrase high-level language. In this post I’ll dive into the amazing advances made in expanding programmer productivity beyond the basic 3-GL’s. What's most interesting about these advances is that they were huge, market-proven advances, and have subsequently been ignored and abandoned by academia and industry in favor of a productivity-killing combination of tools and technologies.

From the Beginning to 3-GL's

The evolution of programming languages has been different from most kinds of technical evolution. In most tech development, there’s an isolated advance. The advance is then copied by others, sometimes with variations and additions. There follows a growing number of efforts concentrating on some combination of commercialization, enhancement and variation. This resembles biological evolution in that once mammals were “invented” there followed a growing number of varied mammalian species with ever-growing variations and enhancements.

If you glance at the evolution of programming languages, it can easily seem as though the same kind of evolution has taken place. It makes sense: software languages are for computers, and don’t computers get faster, smaller and cheaper at an astounding rate?

Let’s start by reviewing the evolution of programming languages up to what are commonly called 3-GL’s. For details see this.

First generation languages are all the native language of a particular computer, expressed as the computer executes the language: in binary. A program in a 1-GL, normally called machine language, is a big block of 1’s and 0’s. If you understand it, you can break the numbers up into data and instructions, and the instructions into command codes and arguments. Necessary for a machine, but a nightmare for humans.

A program in a 2-GL, normally called assembler language, is a text version of a 1-GL program, with nice additions like labels for locations of instructions and data. 2-GL’s were a night-and-day advance over machine language.

A program in a 3-GL, for example COBOL, FORTRAN or C, is a text language that is independent of the computer that can be translated (compiled or interpreted) to run on any machine. There are statements for defining data and for defining the actions that will be taken on the data. The action statements normally constitute the vast majority of the lines. For many programs, 3-GL’s were 5 to 10 times more productive than assembler language, with the added advantage that they could run on any machine.

We’re done, right? In some sense, we are – there are still vast bodies of production code written in those languages. No later language can create a program that has greater execution speed. But maybe we’re not done. As I’ve described, the “high” of high-level isn’t about the efficiency of the computer; it’s about the efficiency of the human – the time and effort it takes a human program to write a given program using the language. There have been a host of languages invented since the early days of 3-GL’s that claim to do this.

Let’s look at a couple of languages that no one talks about and don’t have a category name, were wildly popular in their day and that live on today, unheralded and ignored. I’ll use two examples.

The Way-beyond-3-GL's: MUMPS

The first of these languages I’ll describe is MUMPS, developed at Mass General Hospital for medical  processing. Have you ever heard of it? I didn’t think so.

In modern terms, MUMPS is definitely a programming language; it has all the capability and statement types that 3-GL’s have. But MUMPS goes way beyond the boundaries of all 3-GL’s to encompass the entire environment needed for building and running a program. Normally with a 3-GL someone needs to pay lots of attention to things “outside” the language to achieve an effective solution, particularly in the areas of data access, storage and manipulation, but also in the operating system. A MUMPS program is inherently multi-user and multi-tasking. It has the ability to reference data without the potential danger of pointers. It has the power and flexibility of modern DBMS technology built in – not just relational DBMS but also key-value stores and array manipulation features that are still missing from most subsequent languages. In other words, you can build a comprehensive software application in a single programming environment without external things like databases, etc.

The result of this wide variety of powerful features all available in one place implemented as an integral part of the language was definitely outside the mainstream of programming languages – but wildly productive. MUMPS had strong uptake in the medical community. For example, the hospital software with dominating market share today is Epic, which was originally written in MUMPS (now called Cache) and remains so today.  An amazing number of other leading medical systems are written in the language, as well as in the financial sector.

Net-net: MUMPS is truly a beyond-3-GL high level language in that the total amount of human effort required to reach a given programming result is much less. Even better, all the skills are normally in a single person, while modern languages require outside skills to achieve a given result, for example a database expert.

The Way-beyond-3-GL's: PICK

PICK is another beyond-3-GL that delivered a huge up-tick in programmer productivity. PICK, like MUMPS, is largely forgotten today. It’s an afterthought in any discussion of programming language history, ignored by academics, and generally erased. The title of its entry in Wikipedia is even wrong – it’s called an operating system! Of course, it is an operating system – AND a database AND a dictionary AND a full-featured programming language AND a query system AND a way to manage users and peripherals -- everything you need to build and deliver working software, all in one place. PICK was a key driving factor in fueling the minicomputer industry during its explosive growth in the 1970’s and 80’s, while also running on mainframes and PC’s.

Wikipedia says: "By the early 1980s observers saw the Pick operating system as a strong competitor to Unix.^[13] BYTE in 1984 stated that "Pick is simple and powerful, and it seems to be efficient and reliable, too ... because it works well as a multiuser system, it's probably the most cost-effective way to use an XT."

PICK was the brainchild of a modest guy named Dick Pick. During the early 1980’s I worked at a small company in the Boston area that attempted to build a competitor to PICK, which seemed to be everywhere at the time. As you might imagine, programmer humor emerged on the subject, including such gems as

If Dick Pick picked a pickle, which pickle would Dick Pick pick?

PICK lived on in many guises and with multiple names. But it has zero mind-share in Computer Science and among most people building new applications today.

Conclusion

All-encompassing programming environments like MUMPS and PICK should have become the dominating successors to the 3-GL languages, particularly as the total effort to develop working systems based on 3-GL’s like Java exploded with the arrival of independent DBMS’s, multi-tier hardware environments and the orthodoxy of achieving scalability via distributed applications. Yet another step on the peculiar path of software evolution.

I remember the frenzy during the internet explosion of the late 1990’s and early 2000’s of money flowing in and the universal view among investors and entrepreneurs about how applications must be written in order to be successful. I encountered this personally when my VC partner introduced me to what appeared to be a promising young medical practice management system company that was having some trouble raising money because investors were concerned that the young programmer doing much of the work and leading the effort wasn’t using java. I interviewed the fellow, Ed Park, and quickly determined that he was guided in his technical decision-making by smart, independent thinking rather than the fashionable orthodoxy. I endorsed investing. The company was Athena Health, which grew to become a public company with a major market share in its field. And BTW achieved linear scalability while avoiding all the productivity-killing methods everyone at the time insisted were needed.

The history of amazing, beyond-3-GL's like MUMPS and PICK that deliver massive programmer productivity gains demonstrates beyond all doubt that software and all its experts are driven by fashion trends instead of objective results, and that Computer Science is a science in name only.

Software Evolution: Functionality on a New Platform: Market Research

This is the third in a series of examples to illustrate the way that functionality that had been implemented on an older platform appears on a newer platform.

See this post for a general introduction with example and explanation of this peculiar pattern of software evolution. This earlier post contains an example in security services software and this earlier post describes an example in remote access software.

This example is known to me personally because my VC firm was an investor, and I was involved with them through the life of the investment. 

Example: Knowledge Networks

Old platform	Telephone, mail, focus groups
Old function	Conducting surveys for market and opinion research
New platform	Internet
New function	Essentially the same, with much greater knowledge of the activities of panel members before and after taking a survey, and the ability to conduct interactive surveys
Outcome	The premise was valid, but the company was ahead of the market. It was acquired in 2011.

Organizations that depend a great deal on the opinions and actions of a large number of people sometimes conduct market research to help them shape the details of a product, an advertising campaign, a political campaign or other relevant effort. This kind of research long pre-dates computers. It normally starts using informal methods for selecting the people to ask and evaluating the results. But then it moves in stages towards increasing amounts of scientific control and analysis in order to reduce costs and improve accuracy.

Market research was already well-established on the prior technology “platform” of the telephone when the internet started to spread quickly in the second half of the 1990’s, when a substantial and growing fraction of the US population got internet access. People in the field were familiar with the issues of selection bias, people without telephones and random digit dialing methods of assuring statistically valid panels. But when the web (the new platform) started spreading quickly, did market research transfer its knowledge, methods and techniques to the new platform? It didn’t (I think you probably guessed the answer), because brand-new people put together the first web-based market research systems. It was quick, easy and had the advantage of being inexpensive – but it was as scientifically primitive as telephone-based surveys were prior to the introduction of statistical methods.

Knowledge Networks was started by a couple of professors with stature in market research in the pre-internet world, with the goal of keeping the cost and speed advantages of the internet, but bringing it up to pre-internet scientific standards.

While there has definitely been a migration of internet-based market research to higher levels of scientific standards, which Knowledge Networks has both led and benefited from, their experience is an example of the danger of getting too far ahead of the pattern. One of the key facts about the “emergence of functionality on a new platform” pattern is that the functionality emerges in the same order as it did on the earlier platforms – but it doesn’t skip steps or leap right to the end! These professors knew that internet-based market research would evolve to greater scientific integrity – and they were right! – but they didn’t fully appreciate that the market would get there in its own sweet time, and that it would insist on dawdling on intermediate steps. By insisting that Knowledge Networks provide only the best, highest-integrity market research methods, up to the standards of the best available on earlier technology platforms, the original leaders of the company caused it to be “out of step” with the market. They were “ahead” of the market, which is a great place to be if you want to be a business “visionary,” but is rarely a good place to be if your goal is to build a substantial business.

I have to say that this is a really hard one to get right in practical situations. I personally was involved with Knowledge Networks at the time some crucial decisions were made, but I didn’t know enough about or appreciate the power of the pattern to help make the company as successful as it could have been. In fact, I was probably part of the problem. The professors who started the company were really smart, and they were on top of all the issues of market research. I knew this, appreciated it, and was excited by the possibilities of benefiting by translating the best practices from traditional market research to the internet. What’s painful is that I also knew in general terms the dangers of being ahead of a market. But that’s exactly what we were, and yet again, for the umpteenth time, I didn’t see it and didn’t call it. Arghhh!

Lesson: being too early is just as bad as being too late.

Software Evolution: Functionality on a New Platform: Remote Access

This is the second in a series of examples to illustrate the way that functionality that had been implemented on an older platform appears on a newer platform.

See this post for a general introduction with example and explanation of this peculiar pattern of software evolution. This earlier post contains an example in security services software.

This example is known to me personally because my VC firm was an investor, and I was involved with them through the life of the investment.

Example: Aventail

Old platform	Dedicated IP-SEC VPN
Old function	Remote access to internal LAN resources
New platform	Web server
New function	Use existing Web infrastructure and https to provide old functionality, enhanced by application-level security, reducing costs and increasing flexibility and security.
Outcome	Some market education required in the early years, but strong position vis-à-vis the competition and good growth. The company was acquired by SonicWall in 2007.

Aventail built functionality for remote access that has been implemented over and over again, each time a new technology platform has emerged. But they rode what was at the time the latest wave (internet protocols and SSL encryption), and so were participating in a growing market.

I remember using teletype paper terminals running at 110 baud in the late 1960’s for remote access to computers. Whenever a new platform would come out, the new technology wouldn’t support remote access, but for some strange reason, people would want it! So, focused entirely on getting something working in the new environment, and either ignoring or simply being ignorant of earlier solutions to the same problem, someone would build a remote access solution. But then inadequacies would be found, and a release two would come out. All in what appears to be ignorance of solutions built on prior platforms, blind to their lessons-learned..

A good example is the identification and access control system for remote access. The system you want to connect to has some system for user ID’s and passwords, and then some method of access control based on user groups. The remote access is normally first built in the simplest possible way, having its own system administration, user identification and access control. As the use of the system grows, this parallel administration is a burden, and so some level of integration with the core security system is then implemented. The pattern is that the separate system is normally built first; the need for integration is “discovered;” the integrated control systems are supplied in a later release.

When you see this pattern of stupidity and ignorance for the first time, you scratch your head. These are programmers and experienced product people! How could they have missed such an obviously valuable feature of the same functionality built on an earlier platform? Well, that's the pattern, as I described in detail in the first post in this series. It's a wonderful pattern -- it enables anyone who understands it to predict the future with great accuracy and precision!

Software Evolution: Functionality on a New Platform: Security Services

A whole book could be devoted to spelling out the “natural” emergence of features on a platform, and identifying and accounting for the minor variations from platform to platform (the emergence sequences don’t repeat exactly for a variety of reasons). However, the similarities are obvious and universal enough that anyone with longitudinal familiarity with a couple of comparable platforms would recognize them.

This is the first in a series of examples to illustrate the way that functionality that had been implemented on an older platform appears on a newer platform. All the examples illustrate the point that, even though the functionality is there for anyone to see on the older platform, working and delivering business value, it only appears at its “proper time” on the newer platform.

See this post for a general introduction with example and explanation of this peculiar pattern of software evolution.

I have mostly selected companies that most readers may not be familiar with, to demonstrate the strength of the pattern and the way it affects all participants in a market, not just the well-known, mainstream companies. Also, I know them personally, and so can tell their stories from personal knowledge, instead of just repeating things I’ve read.

Example: Securant

Old platform	IBM mainframe
Old function	Security services: ACF2, RACF
New platform	Web server
New function	Security services for Web applications were extremely basic, and focused on message encryption. Securant implemented mainframe-class user and application security. The same kind of companies that depended on mainframes for transaction processing were very attracted to Securant’s approach to Web applications.
Outcome	Terrific product and sales traction despite product inadequacies and the company’s internal problems. The company was acquired for a good price by RSA Security in 2001.

Securant was started by a couple of young programmers in their 20’s as a services business. They were hired by a large financial institution that was adding internet/web infrastructure to their IT infrastructure. Naturally, the web applications needed access to the programs and data on the mainframe systems, and those systems were protected by state of the art security systems. The financial company would have preferred to be protected by a security facility for the web just like it did for the mainframe, but none was available. So they agreed with these guys to build one.

The young programmers knew nothing about mainframes, and never bothered learning anything much about them. As far as they were concerned, mainframes were pretty useless things well on the way to becoming obsolete – why become an expert in steam engines in the 1930’s when internal combustion engines are obviously the future? So they focused completely and solely on what they knew, and built a ground-up application that met the business security needs as they understood them from the users, who were also barely familiar with what the mainframe had to offer.

Before long, the one-off services project became a very hot, rapidly growing product company. One of the many ironies in this project is that the father of one of the two founding programmers had run one of the mainframe security companies! But he had been estranged from his son for many years, and neither knew that the other was or had been involved in computer security. The father decided to come back from the retirement that had been funded by the sale of his security company, re-connected with his son, and you can imagine his amazement when he found out what that son had accomplished. The father saw the functionality he had on the mainframe, re-imagined and re-implemented for the new environment.

One of the powerful things about this pattern is that it’s like the tide or the current in a river – at the right point, it just pushes the vendors and functionality in the direction of the “flow,” and everything ends up moving in the same direction. The people involved tend to think they’re inventing things – but what they invent is pretty predictable, because they’re responding to the same kind of needs that the previous bunch of inventors were responding to.

Software Evolution: Functionality on a New Platform

When a new “platform” emerges (UNIX, Windows, Web, Apps), if you look at any application area and see how it evolved on prior platforms, the application’s functionality will emerge on the new platform in roughly the same order, though often on a compressed timescale. The functionality that is relevant depends on the particular application area. This concept applies both to system and application software.

The pattern is: functionality emerges on a new platform in roughly the same order as it emerged on earlier platforms. The timescale of the emergence may be compressed; the important aspect of the pattern isn’t the timing but the order. The pattern means that functional steps are rarely skipped – what was next last time is also next this time. The pattern also means that when someone tries to introduce functionality too soon, before the functionality that preceded it on prior platforms is generally available, the market will not accept it.

While this pattern takes a good deal of knowledge and judgment to notice and apply, I have consistently been impressed by its predictive power. By following the pattern, you can be pretty confident that you’re building proven functionality, and that you’re following a pattern of success.

I have noticed a couple of danger points here. When the company is too aware of the pattern, it is easy for them to “get ahead of themselves” and more importantly ahead of the market, by solving problems that certainly will become important, but problems that the market doesn’t know it has yet. The “same order” part of the pattern is important; building ahead of the market appears to win few business benefits.

On the other hand, without knowledge of the pattern, it is easy to make up all sorts of things you think people might want, and build them, only to find out later that you wasted time and money, because the functionality you built is never part of the accepted function set.

Really great companies who have lots of creative people, the ability to execute, and listen closely to market reactions to what they’re doing, don’t “need” to know about this pattern. However, for the rest of us mortals down here, having a “cheat sheet” to what important features the market will be demanding next can prove awfully helpful.

Basic Example: operating systems

IBM famously created an operating system for their mainframe line of computers, OS/360. It had the capability of running multiple jobs at once. Its multi-tasking abilities grew and became more sophisticated through the 1970’s.

Eventually a transaction monitor, CICS, was written and became a de facto part of the operating system for applications with lots of interactive users. As the operating system was used, it became evident that various access methods for storage and communications needed to be separate from the core, and so clear interfaces were created, and the notion of completely self-contained access methods (for example, a file system) as replaceable units was supported. A strong security system was not part of the early versions of the operating system, and the need for one became critical, and so strong external modules were written and support for security was added to the core. While there was a “main” operating system, alternative operating systems were written for various purposes, and a virtual operating system was written to actually run on the “bare metal.” With VM (the virtual OS), you could devote most of a machine’s resources to the production users, while letting some of the users running on a completely different operating system.

While all this was taking place, people were studying and experimenting in university environments, deciding just what an operating system was and was not, what the best ways to build one were, and so on.

Before very long, mini-computers were invented; these were basically mainframes on the cheap, with all sorts of features and functions missing – but they were cheap. And, since each had a unique instruction set, each minicomputer needed an operating system. Programmers were hired, and those programmers, of course, ignored the mainframe operating systems, and built simple, cheap OS’s to go along with the cheap hardware. Surprise, surprise, those cheap OS’s resembled nothing as much as – the first generation of mainframe operating systems! But people quickly discovered the limitations, just as they had before, and set about making the same set of enhancements that the previous generation of pioneering programmers had made. Within ten years, they had re-invented many of the important mainframe OS concepts, and were on the way to building the rest.

With all this knowledge of operating systems floating around and pretty easily available, what do you suppose happened when people took the early micro-processor chips and made them into micro-computers? Naturally, they understood the state of the art of operating systems theory and practice and adapted an existing OS (which were starting to be built in high level languages) or built one that took all this knowledge and applied it with appropriate adjustments to the new environment, right? Bzzzzt! Of course not!

What the kids who were faced with the task did was start from scratch, not only in terms of code, but also in terms of knowledge. They didn’t stand on the shoulders of giants; they didn’t learn from the experiences of the many that preceded them; they built OS’s as though they were the first programmers who ever tried to do such a thing. And the result was pretty much like early mainframe (even pre-360!) operating systems. There was no serious memory protection or address mapping; there was no real concept of multiple users, multiple levels and types of users, or any real security; no multi-tasking; the access methods were hard-wired in, and so on. The limitations and problems emerged pretty quickly, and so did add-on rubber band and baling wire patches, just like in earlier days.

It’s a good thing that IBM came along at this point, and brought commercial order and education to the emerging microcomputer market. When they came out with the IBM PC, they not only legitimized the market, they had deep history with mainframes and minicomputers. They employed true experts who knew operating systems inside and out. They had a research division, where there were people who could tell you what the operating systems of the future would look like. So it makes sense they would get those experts together, and they would create a small but efficient micro-tasking kernel, common interfaces for installable access methods, and many other appropriate variations on all the modern operating systems concepts. The last thing such a smart, educated and astute major company like IBM would do was make an exclusive deal with a small company that had never built an operating system, who had just bought the rights on the cheap to a half-baked excuse for a primitive first-generation OS, and make that the IBM-blessed … Wait! … that’s what they did do! Arrrgggghhh!

Explanation

One might well ask, how can a pattern like this continue to have predictive power? Why wouldn’t the people who develop on a new platform simply take a little time to examine the relevant applications on the older platforms, and leap to the state of the art? Why wouldn’t customers demand it?

It is hard to know for sure, but I think there are a couple main factors at work, and there is evidence for the relevance of each of the factors.

The first factor is the developers. It is well known that most developers learn a platform and then stick with the platform they’ve learned for an extended period of time, basically as long as they can. The reason is simple: they are experts on the platform they already know, and therefore have prestige and make more money than they would as novices on a platform they’re just learning. I speculate that this is one of the many contributing factors to the rapid migration of ambitious programmers into management, where they can advance without being tied to a platform, at least as much. So who learns the new platforms? With few exceptions, new people. If you’re just entering the industry, you are counseled to learn the hot new languages; you tend to be sensitive to where the demands and rising salaries are. Still, you expect and are paid entry-level wages, along with most other people (except managers). Why should the experienced programmers jump to the new platform? They would have to compete with hard-working young people, their knowledge of the older platform will be considered a liability, and on top of everything else, they’d have to take a pay cut.

The result is that no one working on the new platform has an in-depth, working knowledge of the applications on the older platform, and at least in part because of this, everyone considers knowledge and use of the platform to be vastly more important than knowledge of an old application on an obsolete platform. So they ignore it! As a result, they dive in and attempt to automate the application area “from scratch.” Their attempts are usually quite close to every first generation program for that application on past platforms, because it turns out that the determining factor isn’t the platform, it’s the business problem. They proceed to re-discover, step by step, the reasons why the first generation was inadequate and had to be supplanted by a second generation, etc.

The second factor is the buyers. When a new platform emerges, most buyers simply ignore it. Why pay attention? It’s a toy, there are no good applications, etc. The few buyers who do pay attention tend to be crazy, early adopter types who just love the experience of trying new things. Like the programmers, they also tend to care about the platform more than the application – otherwise, they wouldn’t even consider buying what is typically a seriously immature application on the new platform. But they can only buy what’s being sold, and so they choose among the inadequate applications. Because they don’t care about applications as much as platforms, they don’t even ask for features they know could only be present in the mature applications for older platforms – they press the applications vendors for the “next” obvious cool feature, in the narrow universe of the new platform.

The reason why application evolution repeats itself on the new platform, then, is that nearly everyone involved, builder and buyer, is ignorant of the past and wears what amounts to blinders. It’s as though they are building and buying the application for the first time. Therefore, they respond most strongly to the same business pressures that people involved in computer automation tend to see first, and then next, and so on. It’s as though there’s a “natural” most climb-able path up a mountain, and successive waves of climbers approach the climb from the foot of the mountain, completely ignorant of the experiences of those who came before them, but confronted with the same options they tend to make the same choices, and so everyone takes roughly the same route up the mountain.

Why don’t smart groups who know all this leap-frog their way to the most advanced application functionality? I have seen this happen, and it’s the buyers who tend to rain on this kind of parade. The buyers tend to be familiar with the range of applications in a category, and those applications tend to address a highly overlapping set of problems in ways that vary only slightly. The “far-seeing” builder then comes along with all sorts of solutions to problems that the buyers don’t even know they have! The buyers look around in confusion – why isn’t anyone else talking about this? I kind of understand what you’re talking about, but I feel silly thinking that something’s important when no one else does. I think I’ll wait. And they do. So getting too far ahead of the buyers is just as much of a problem as being too far behind the competition. The result: the application evolution repeats itself on the new platform, in roughly the same order each time, and no “cheating” or “running ahead” is allowed.

This all sounds incredibly common-sense when I read it written down, but I have to admit that this particular piece of common sense is not only uncommon, it took me personally decades and multiple failures to finally get this simple thought into my thick skull. The key thought is this one: you may think you know a person has a problem; the problem might be a severe one, and cost the person a great deal of money and trouble; you may even be entirely right in this judgment. However, if the person in question does not think he has a problem, why should he pay for a solution – in fact, why would he go to the trouble of implementing a solution even if it were free?  Even worse, why would he even waste time talking with you, once he got the idea that you thought he had a problem he didn’t think he had? Why would he listen to a chimney sweeper’s pitch if he lives in a house without chimneys? The air quality in Los Angeles in 1970 was terrible. But there was no market for catalytic converters on automobile exhaust systems at that time. The problem that converters solve existed, and was getting worse. It was obvious for anyone to see. It was even talked about. But in peoples’ minds at the time, having a car that contributed to air pollution was not a problem most people accepted they had (even though we know that, objectively speaking, they did have it).

The Progression of Abstraction in Software Applications

This post describes a little-known concept for understanding and creating software architecture that small groups use to defeat large, powerful incumbents and nimble competitors. It is one of a small number of powerful, repeating patterns that help us understand and predict the evolution of software. Understanding these patterns can help entrepreneurs direct their efforts; if they do it well, they greatly enhance their chances of success. Understanding the patterns can also help investors choose to invest in groups that are walking a path to success.

Evolution of Applications Towards Abstraction on a Platform

One of these patterns is the stages that applications naturally evolve through on a technology platform. Each step or stage brings a big increase in the power of the software, decreasing the effort and increasing the speed and effectiveness of being deployed to meet customer needs.

A category of software applications may well get “stuck” at a particular stage for a long time, sometimes even decades. During this time, the software may appear to move forward, and of course the marketing people and managers will always put things in the best possible light. But it’s always vulnerable to being supplanted by a next-stage version of the functionality.

While there aren’t clear lines of delineation between the stages, it’s nonetheless useful to understand them roughly as:

• Prototype. A hard-coded body of code.
• Custom Application. Does a job reliably, but most changes require changing source code.
• Basic Product. The code now has parameters, maybe user exits and API’s. Real-life implementations tend to require extensive professional services, and the cost of upgrading to new versions tends to be high.
• Parameterized Product. The level of parameterization is high, with interface layers to many things outside the core code, so that many implementations can be done without changing source code. There may be some meta-data or editable rules.
• Workbench Product. A large portion of the product’s functionality has migrated from code to editable meta-data, so that extensive UI, workflow, interface and functionality changes can be accomplished via some form of workbench, which could be just a text editor. The key is that the details of application functionality are expressed as editable data, meta-data, instead of code. Nonetheless, all fundamental capabilities are expressed in highly abstract code.

As a body of code goes through this sequence of abstraction, it is increasingly able to meet the needs of new customers and changing needs of existing customers, with decreasing amounts of effort, risk and changes to source code. At the same time, the more abstract a program, the more functionality is expressed as data that is not part of the software itself, a.k.a. meta-data, and the more the software implements generic capabilities, as directed by the meta-data.

The pattern applies both to individual bodies of code and to collections of them. It applies to code built internally for an organization and to code that is sold as a product in any way.

I defined the stages above as a convenience; in reality, the categories are rarely hard-and-fast. A body of code could be given a big transformation and leap along the spectrum, or it could take a long series of small steps. One body of code could remain stuck with little change in abstraction, while other bodies of code doing similar things could be ahead, or progress rapidly towards abstraction.

The Driver of Abstraction Evolution

In biological nature, competitive pressures and external change appear to drive evolutionary changes. Similarly, when we look at categories of software, if little changes to make the software change, it doesn’t change – why take the trouble and expense to change software that meets your needs or the needs of your customers?

In reality, someone always seems to want changes to an application. A prospective user would gladly use the software if it did this, that or the other thing. A current user complains about something – it’s too slow, too hard to use, too complicated, or it just doesn’t work for X, Y or Z. How often does a piece of software not have a “roadmap?” If it doesn’t, it’s probably slated for retirement before long. Brand-new software is rare. The vast majority of software effort goes into making changes to an existing piece of software.

How much time and effort is needed to change a particular body of software? That is the key touch-point between techies and business people. This is the point at which the level of abstraction of the application comes into play. Regardless of the level of abstraction of the application, the “change” required either has been anticipated and provided for or it has not.

• If the change has been anticipated, the code can already do the kind of thing the user wants – but not the particular thing. Doing the particular thing requires that a parameter be defined, a configuration file changed, a template created or altered, workflow or rule changes made, or something similar that is NOT part of the program’s source code. This means that the requirement can be met quickly, with little chance of error.
• If the change has not been anticipated, then source code has to be changed in some way to make the change. The changes required may be simple and localized, or complex and extensive – but the source code is changed and a new version of the program is created.

This is what the level of abstraction of a software application is all about: the more abstracted the application, the more ways it can be changed without altering the source code and making a new version of the program.

This is the fundamental driver of applications towards increasing abstraction: as changes have to be made to the application, at some point the technical people may decide to make the change easier to make in the future, and create the appropriate abstraction for the kind of change. This may happen repeatedly. As more complex changes are required, the program may just get more complicated and more expensive to make further changes, or more sophisticated abstractions may be introduced.

While historically it appears that outside forces drive applications towards increasing abstraction, smart programmers can understand the techniques of abstraction and build an application that is appropriately abstract from the beginning. Similarly, the abstracting methods can be applied by smart programmers to existing bodies of code to transform them, just because it's a way to build better code and meet ever-evolving business requirements.

Conclusion

The hierarchy of abstraction in software is one of the most important concepts for understanding a specific piece of software, or a group of related software. Over time, software tends to become more abstract because of competitive and business pressures, or because of smart programmers working to make things better. The more abstract a piece of software is, the more likely that it can respond to business and user demands without modifications to the source code itself, i..e., quickly and with low risk.

The hierarchy of abstraction is certainly a valuable way of understanding the history of software. But it is more valuable as a framework for understanding a given piece of software, and the way to evolve that software to becoming increasingly valuable. It is most valuable to software developers as a framework for understanding software, and helping them to direct their efforts to get the greatest possible business impact with the smallest amount of time and effort.

The Fundamentals of Computer Automation

The principles underlying computer automation are clear and strong. They account for most of the automation we see. They tell us clearly what will happen, why it will happen, and what the benefits will be. What the principles do NOT tell us is who will first apply automation to what process in what sector.

Understanding the principles lets you predict the rough order of the sequence of automation.

The principles are extremely simple. Perhaps that's why they're rarely stated and appear not to be taught in schools or understood by practitioners. So much the better for people who want to innovate and win by automating!

The Principles

You can probably guess the principles of automation from the definition of the word: "automate" means to do by machine or computer what a human would otherwise do.

At core, there's really a single simple principle:

• Replace time that a person would have spent with work done by a computer.

The core principle is demonstrated by a before-and-after comparison: the principle requires that human time/value doing whatever activity is being automated, net of the investment in the computer hardware and software, is reduced after the addition of the computer and software. Lots of words, same meaning: apply the computer stuff, and there's less total human time/effort.

Expanding on the replace theme, "replace" means anything that reduces human time to do something:

• Replace time that a human would have spent with work done by computer; REPLACE!
  
• Reduce the amount of time spent by a human; REDUCE!
  
• Arrange the human's time more efficiently, eliminating waste; RE-ARRANGE!
• Decide Who does What work, How and Where they do it; RE-ORGANIZE!

That's it! 

This notion of what software was REALLY about struck me quite early in my career. The thought was simple: the ultimate purpose of most of the software I wrote is to replace humans. The thought made me uncomfortable. But after a while, I connected the thought with all the rest of the mechanization and industrialization in society for hundreds of years. A new machine is valuable only to the extent that it somehow reduces the total amount of human labor to reach a given result, everything taken into account! This was true for the Jacquard Looms to which the luddites violently objected in the early 1800's, and the same principles are at work today with computers, resisted by modern-day luddites.

In other words, computers are the next step in a long sequence of people figuring out how to get stuff done with less effort by themselves and/or other people.

Here are some simple examples:

REPLACE

Replace what humans do with computers doing it is the core principle of automation. E-mail is a super-simple example, since computers automate the entire process of delivering the result of typing to another human being, eliminating the many steps involved in doing it physically. The replacement can be low-level and mechanical, like delivering email, or it can be high-level, like the work of deciding who should do what work in what order.

REDUCE

Electric light is a pre-computer example, since it reduces all the time required to, for example, deal with the oil lamp, and replaces it with flicking a switch. A modern example is spreadsheets, which automate the time people used to spend making calculations with adding machines.

RE-ARRANGE

Introducing chat into customer call centers eliminated huge amounts of wasted waiting time, so that phone operators could handle chat when they weren't on the phone.

RE_ORGANIZE

Routing certain kinds of calls to the optimal customer service person (Who does What work), giving them scripts and hints.

The Principles apply to Manual and Mental work

At the start of automation, the computer and software automate work that it's pretty plain that people are doing. As it gets more advanced, a couple interesting things happen.

1. Once the work is done by computer, it sometimes happens that better ways of getting the work done are possible. This is the same principle we see in mechanical flight: even though all birds have wings that flap, airplane wings don't flap. Instead of flying the ways birds do, engineers have figured out of the core principles of flying and reflected those in airplane design. Similarly, human work is often first translated to computer automation simplistically, and later rebuilt without reference to the human origins.
   
2. Once computers are a significant part of the landscape, computer people figure out valuable things to do that have little to do with what humans have done in the past, though this is uncommon.

The Principles are recursive

At the beginning, getting computers to do things is incredibly labor-intensive. As time goes on, some people look at the work humans do to get computers to do things -- and automate the automation process! Yes, the very programmers whose job is to put people out of work can be put out of work by computers. And on and on.

The Principles applied to human interaction

We can see the principles in action in the sequence of changes that have taken place in normal interactions between human beings.

Prior to automation, the only way humans could communicate was by voice, and by being in each other's physical presence. Here are the major automation steps that have taken place:

• Writing a letter.
  
  • Automation is making and acquiring paper, pen and ink, and transporting letter
    
  • Eliminates the travel time of the sender and/or receiver
• Publishing books, newspapers.
  • Automation is printing and distributing the books or papers
  • Eliminates the travel time of all the receivers
• Telephone
  • Eliminates the travel time for the parties to get together
  • Eliminates the time spent writing and reading, and the delay in transmission of prior methods
    
• Mobile phone
  • Eliminates the need to travel to a phone to make and/or receive calls
• Voice mail
  
  • Eliminates the requirement that the receiver be available when the caller calls, for those cases when it's applicable
    
• E-Mail
  
  • Writing a letter with near-real-time delivery and the ability to read when ready, like letter reading and voice mail.
• Chat
  • Like email, only with a different interface, making it better for real-time, written exchanges, with minor delays acceptable.
• Scripting
  • In customer service centers, reduces training, reduces human error, and reduces the time to produce an optimal answer to a question.
    
• VRU, chatbot
  • A recorded and/or machine-generated party in a conversation.
  • Applicable to all of the above methods.
  • Eliminates a human giving repetitive responses or messages.

This all seems obvious, right? Like we know all this stuff, what's new?

For the steps above that are computer-based, what's interesting is that the technical capabilities were often in place considerably before they were brought into production. The gap was usually not decades-long, like it has been with more esoteric technologies. And in each case, knowing the technology and this fundamental principle could enable you to predict that the automation would be built and deployed.

Conclusion

If you want to predict the future of technology and evolution, study the evolution of software and the fundamental principles that drive its application. The principles are obvious! But they are nonetheless rarely discussed or applied.

 

Laser Disks and Workflow Illustrate the Insane, Fashion-Driven Nature of Software Evolution

Computers are so exact and numbers-based that we tend to think of the people in charge of them as white-jacketed scientists with esoteric knowledge driving towards calculated optimal results, much the same way we imagine that the scientists and engineers calculated the path of the Apollo capsule to the Moon. If the Apollo program were run the way most computer projects are, it would have been a miracle if the capsule made it off the launch pad, much less traced an incredibly exact path from Cape Canaveral to the chosen landing spot on the Moon, a journey of over a quarter million miles.

The reality is that the application of computers and software to automation is painfully slow, usually failing to achieve optimal results by large margins, and often lagging behind what is achievable by decades. The reasons appear to be a semi-random mixture of commercial interests, technology fashion trends, ignorant and risk-averse managers, and technical specialists rooted in the past and wearing blinders. That’s all!

Here’s the story of a typical example.

Microfilm, document imaging and laser disks

Long before computers, microfilm was the preferred medium of librarians and archivists to preserve paper documents for future use. Microfilm was more compact than paper, and lasted much longer. What happened when computers came along is a curious and educational story, a prime example of how computer use and software evolve.

Disclosure: I lived and worked as a participant in this history during the late 1980’s and early 1990’s. The story I will tell is not systematic or comprehensive, but like a traveler’s tale of what he did and saw.

A side show in the long evolution of computer storage is the laser disk. A laser disk has data recorded onto it either as a whole, during manufacturing, or incrementally, as part of a computer system. The laser disk had quite a run in the consumer market in the form of CD’s, and then DVD’s. They were an improvement on existing methods for distributing digital content of all kinds.

In the computer industry, they took the form of WORM (write-once-read-many) drives, and were used to record archival data that should not be updated, but could be read as often as needed. The technology, as often happens, went searching for problems it could solve. In an effort to reduce human labor more than microfilm could, jukeboxes were invented. Each jukebox had a disk reader and many shelves for disks, with a mechanism to load a selected disk into the reader. FileNet, now part of IBM, built one of the first of these systems in the mid-1980’s.

At around the same time that WORM drives became practical, paper document scanners were evolved from systems to capture images of the paper onto microfilm. In each case, light was shown onto paper and focused onto a target; a scanner replaced the film with an optical sensing device.

How could we get people to buy WORM drives and jukeboxes, the creative marketing people wondered? The departments that put documents onto microfilm weren’t going for it – the new systems were much more expensive, microfilm wasn’t accessed often enough to make it worth connecting it to computers, and the records retention people were understandably skeptical that any computer disk would be readable 100 years from now, as microfilm will be.

Adding workflow to document imaging

I have no idea who made the audacious leap, but some creative person/group came up with the idea of doing computer document scanning at the start of the process, when paper arrived at a location for processing, instead of microfilmed at the end for archival purposes. But WHY??? The creative types scratched their heads until they were losing hair rapidly. Why would anyone make responding to loan requests or whatever even longer than it is today by introducing the step of scanning?

GENIUS TIME: LET’S INVENT “WORKFLOW”!!!

Everyone has an image for what “workflow” is. For documents, it’s the path of processing steps from arrival to filing. In larger organizations, there are often multiple departments involved in processing a document, so there are in-boxes and out-boxes and clerks transporting documents from one to the other.

Rubbing their hands and cackling, the scammers worked out their strategy: We’ll scan the documents and convert them to images so that we can move images from place to place electronically instead of piles of paper! We’ll call it “document imaging workflow.” It will deliver massive benefits to the whole organization, to everyone who touches the paper, not just to the tiny group who files and archives at the end – and storing the scanned documents onto WORM drives will do their job too! Hooray! We will leap-frog the old-fashioned, paper-flooded back office into the modern electronic age. What’s not to like?

It was audacious and brilliant. It was a scam that would wither if confronted with just a little common sense or cost accounting. Which, true to the pattern of such fashion-driven trends, it never had to confront!

Implementing workflow

A typical target environment for implementing workflow was large offices with large numbers of documents, often forms of some kind, arriving every day. The forms would go from the mail room to the relevant people’s desks for processing. The person handling the form would often have an IBM 3270 terminal for interacting with a mainframe computer. When the person was done with the form, they would place it in one or more outboxes for further processing at other desks, or for filing. Sometimes documents would go into file cabinets for a period of time, but all would end up being archived, and sometimes microfilmed for permanent storage.

Implementing workflow first of all required scanning the documents as a first step, and storing them immediately on normal computer disks, and ultimately on WORM drives, thought to be the equivalent of microfilm. Once scanned, the documents would be managed by workflow software, which would route them to the appropriate desk in the appropriate department for processing. Every desk needed to have the old computer terminal replaced or augmented with a large, high-resolution image terminal, capable of handling at least the display of the document, and sometimes also the computer screen. The old, slow connections between terminals and mainframe needed to be replaced with a fast local area network, and there needed to be substantial servers for handling the local image handling and workflow routing.

Of course, lots of analysis and programming was involved in addition to the equipment. Workflows had to be analyzed and programmed, and all the management, reporting and exception handling taken care of.

The benefits of workflow

When a movement like document imaging workflow has a head of steam up, no one seems to ask whether it’s a good idea, and if so, how good of a good idea it is – quantitatively. When I was involved, everyone threw around there would be at least a 30% productivity improvement due to workflow, and that would make all the expense and disruption worth it. I never encountered a single study that measured this.

Think about it for a minute. Would looking at an image of a document make you work faster than you would looking at the original paper? Would picking the next paper from the in-box be slower than getting the system to show you the next image? What about the labor of the clerks moving stacks of paper from one person’s outbox to the next one’s inbox? It would certainly be saved, but chances are a single clerk could handle the paper movement for many dozens of people. And remember, there’s the scanning and indexing that’s been added and the massive computer and software infrastructure that has to be justified.

It’s obvious why no one EVER did a cost-benefit analysis of workflow – the back-of-the-envelope version easily shows that it’s a couple zeros away from making sense.

I remember a couple of out-of-it, tech-fashion-disaster know-nothings mildly thinking out loud about how workflow could possible be worth it financially. The immediate response from workflow fashion leaders was upping the ante – don’t you know, the wonderful workflow tool from (for example) FileNet lets you program all sorts of efficiencies for each stage of work; it’s something we really need to dive into once we get the basic thing installed. End of subject!

So who fell for this stuff? Just a who’s-who list of major organizations. Each one that fell for it increased the pressure for the rest to go for it. None of them did the numbers – they just knew that they couldn’t fall too far behind their peers.

Conclusion

It’s hard to think of a field that’s more precise and numbers-oriented than computers. Who has a clue what actually goes on inside those darn things? And the software? There are millions of lines of code; if a single character in any of those lines is wrong, the whole thing can break The impression nearly everyone has, understandably, is of a field that is impossibly precise and unforgiving of error. When the software experts in an organization say that some new technology should be adopted, sensible people just agree, particularly when there’s lots of noise and other major organizations are doing it. It can be even more gratifying to get known as pioneering, as one of the first organizations to jump on an emerging tech trend!

Somehow, the smoke and noise from the software battlefield is so intense that the reality is rarely seen or understood. The reality, as I’ve illustrated here, should result in everyone involved being sent to the blackboard by a stern teacher, and being made to write, over and over: “I pledge to try harder to rise above the level of rank stupidity next time.”

Simple Data Entry Technology Illustrates How In-Old-Vation in Software Evolution Works

Since when is “data entry” (entering data into a computer) a pivotal, innovative technology? When the difference between doing it the normal way and doing it with advanced technologies is a …ten-to-one productivity difference … that’s when.

I’ve described how the Operations Research algorithm of Linear Programming is fifty years into an agonizingly slow roll-out through different applications, from scheduling oil refineries in the 1960’s to scheduling retail sales in the 1990’s, and now scheduling medical infusion centers and operating rooms in the late 2010’s. In each case, laborious and error-prone human scheduling was replaced by the algorithm, with improvements ranging from no less than 10% to over 50%. This is major! Why did such an innovation wait for decades to be applied, and for many applications, is still waiting!!?? This is the mystery of how what’s called “innovation” works in reality, and why it should be called “in-old-vation” instead.

You may think that part of the cause is that LP is an exotic algorithm – even though it’s a standard part of the Operations Research engineering curriculum, most so-called normal people haven’t heard about it. While it appears that even the hosts of people who wax eloquent about AI and ML are clueless about LP, it’s not exactly a secret. So let’s see if obscurity is the reason why LP remains a “future innovation” in many potential applications, by examining the super-plain, ordinary, completely-understandable-by-normal-people case of data entry.

Data Entry

Just as process optimization is done by hand or stupid methods for decades until some genius comes up with the brilliant idea of applying tried-and-tested LP to the problem and dramatically improves it, so is Data Entry widely performed by primitive methods until some “innovator” comes along and applies “Heads-Down Data Entry” (HDDE) methods to the process – and typically gets improvements of 3 to 10X! The only difference is that while LP is taught in Engineering departments and studied by math nerds, Heads-Down-Data-Entry is “just” a collection of common-sense techniques that require no math and no professors to understand or implement. It’s so “common” that it doesn’t even have a generally accepted name – though it’s been implemented in many places and been thoroughly proven in practice. It’s far too humble to merit an academic department – and yet, when applied, has delivered truly massive gains, far higher proportionately than exotic Linear Programming has!

The methods of HDDE were first implemented in places that had huge volumes of repetitive data to be entered into computers. Banks were early users for check entry, and so were the credit card companies who, at the start, had huge volumes of paper charge slips to process. Simple ideas like minimizing keystrokes and eye movement were implemented, and then taking advantage of the eye-to-fingers pipeline, when people noticed that showing clerks the next item to be entered before the current one was complete led to a big jump in speed. Other methods like double-blind techniques were invented, so that entry clerks just entered – whether their work was original or used to check someone else’s work was entirely handled by the Data Entry system.

As soon as scanning and image display became practical, HDDE adopted them. That led to another jump in productivity, enabling large, complex forms to be broken up into pieces, so that a clerk would see the image on a screen of the same piece of data from a whole set of forms instead of entering a whole form from start to finish. No HDDE shop would even consider having the entry clerk think about anything on the form, stuff like “if this field is missing, do this instead of that,” because it would just slow them down.

Finally, there’s ICR (Image Character Recognition), which is having the computer “read” the image instead of a human. This technology has existed for many decades. Once you’ve got HDDE in place, phasing in ICR is a natural, so that the proportion of entry done by humans gradually decreases as the effectiveness of ICR increases.

Remember, applying LP to scheduling might result in a 30% improvement, which in most cases, is major to the point of being revolutionary. What about HDDE? Entering data from a paper form into a computer using primitive methods might get between 1,000 and 1500 KPH (keystrokes per hour). There are lots of stages of improvement, including things I’ve mentioned like eliminating the thinking and breaking up the form, but levels of 10,000 to 15,000 KPH in a professional environment are widely achieved – with superior quality. That’s a minimum of 5X! Typically much more. Of course, as you incorporate ICR into the process, it gets even better, gradually reducing the human factor, so that most fields are entered with no human involvement. At this point, the technology is probably best called ICR+HDDE, though there is no generally accepted term.

Given all this, it would be insane to handle computer data entry by anything other than HDDE methods, right? Welcome to the software industry, where insanity of this kind is the accepted state of affairs. And where almost no one practices the most simple and basic of computer fundamentals, such as counting.

How HDDE gets implemented

I described how Linear Programming went from problem domain to domain, each time acting like an innovation, as indeed it was in that area of application. Once it gets established, it tends to stay. The case of HDDE is different, I think because it’s not a recognized “thing” in the halls of academia, or among the poo-bahs of big business. It’s the kind of thing that no self-respecting Professor of Computer Science would stoop to consider, assuming he ever encountered such a low-status thing – you know, the kind of thing that “merely” makes common sense and, well, works.

HDDE has appeared in competitive, high-volume service businesses, where it has a major role to play in delivering results for the customers of the business. There have been software products that directly support HDDE, so that all you have to do is buy and implement them. It’s neither obscure nor hidden. But it’s never been talked about at conferences as the “coming thing.”

Case Study: HDDE rejected

In the early 1990’s, when document imaging and workflow technology were hot and something people talked about the way they talk about AI/ML and innovation today, the government-backed student loan organization, Sallie Mae, decided to apply the technology to improve the operations at the handful of processing centers they had at the time, employing many thousands of people and processing millions of documents a year. The popular thing to do at the time was to scan documents on receipt, and then send them to the same places the paper was sent, so that workers could process the images of the documents displayed on new big screens instead of paper. The job was basically to type the data into the right places of the software application they used.

Everyone at the time said that converting the documents was important ONLY because it enabled wonderful workflow, the elimination of inboxes and outboxes for paper. And the bits of other stuff you could do, like having a group of people taking from a common inbox instead of each having their own. The common “wisdom” was that you could gain 30% productivity improvement by implementing this marvelous new technology.

I got involved, since I was a recognized expert on document imaging technology at the time, and had personally coded one of the early workflow systems. I figured out and showed in detail that by canning the workflow and implementing HDDE techniques, they could gain a minimum of 5X productivity improvement. No one disputed my thoughts or detailed plan. They just ignored it, and proceeded to implement the standard stuff. I strongly suspect that after considerable time and expense, there were walk-throughs of the Sallie Mae sites showing visitors the big screens and absence of paper – what a big success the project was!

Case Study: ICR-HDDE applied with success

There are two current cases I know of where ICR-HDDE is being applied and winning. Each are classic, narrow service businesses where converting forms to data is the key value of the business, and where the companies buying the service just want fast, accurate data delivery, they don’t care how it’s done. Disclosure: each of these is an Oak HC/FT investment.

At Groundspeed, insurance forms and reports are captured and the relevant data is extracted from them by the most effective relevant means, often involving forms of ICR-HDDE. There is lots of forms recognition from documents and images that are often computer output, with the relevant data appearing at varying places on a page. Nonetheless, Groundspeed is able to deliver the data stream the customer needs, quickly and accurately, The results are so powerful that new levels of analytics are enabled by the newly available stream of structured data.

At Ocrolus, financial documents of all kinds including bank statements and pay stubs are converted to data in a standard format to enable fast and effective operations like giving loans for business and personal use, along with a growing list of other operations that also need good data. An effective combination of ICR-HDDE techniques are applied to get results for companies that need accurate data to make fast decisions.

Conclusion

HDDE is a collection of methods that have been proven in practice for many decades. The technology that is behind it continues to deepen and reduce human effort even more, with the addition of ICR. But it remains a niche technology, ignored by the numerous places that could benefit from it, even more than LP.

The big difference between LP and HDDE is that LP is a formal piece of magic that’s in academia. HDDE is nowhere. In fact, it’s really just an example of classic industrial engineering applied to computer software and the people who use it. Which makes it all the more mysterious that it's largely ignored.

In-old-vation is real. Most “innovations” are minor variations on things long-since proven and demonstrated in practice, but are unimplemented in the many situations that would benefit from them until some mysterious combination of circumstances arises to let them explode into practical reality.

Understanding Software Evolution Helps you Predict the Future of Software

There are patterns in software evolution. Because almost no one studies software history, and even fewer study software evolution, these patterns are almost never discussed.

The patterns are amazing. In some cases, you can be pretty sure that a trend that “everyone” says is going to be the future will fizzle out. In other cases, you can predict with a high degree of certainly that software of a certain definite kind and description will be built – even though it hasn’t been built yet – it hasn’t been built anywhere by anyone, but nonetheless you know it will be built.

The reason I started to study software evolution is that it’s just plain interesting. The patterns that emerge from it are striking and educational. But most important, knowing software evolution can help you predict the future!!

Knowing software evolution can help you predict the future for the simple reason that software does not evolve the way pretty much everyone thinks it does. It is often the case that, when a new software system appears on the market that has a big impact, people who know the patterns were sitting around waiting for it to happen! They knew it was coming!

I remember living in New Haven CT in the early 1970’s. I was a vegetarian. A new restaurant had just opened, a kind I’d never been to, an Indian restaurant. I went, and it changed my culinary life. Indian food – made by vegetarians for vegetarians, healthy and wonderful flavors!

The new restaurant was definitely new in Connecticut, a true innovation. The owners had to go through quite a bit to open it up, get supplies, etc. But was it otherwise innovative? Did the owners invent a single one of the dishes or recipes? No, of course not. Just like with a piece of software, they had to create it from scratch in a new setting, making appropriate adaptations, and so to the locals, it was new. But in reality, it was a newly adapted instance of a proven winner. Software works the same way – a piece of software that is a big new winner, the first of its kind, has most often been proven elsewhere.

The first Indian restaurant in New Haven could have opened years earlier or later than it did, with different people doing the work. There was no way to predict when the Indian restaurant would have opened, or who would have done it. But isn’t it obvious that, in the context of ethnic restaurants enjoying increasing success and the growing Indian population that someone would have opened an Indian restaurant there? Shockingly to most people, software works exactly the same way! In a surprising number of cases, you can’t predict WHO or WHEN, but you can sure as heck predict WHAT will be built and even WHERE (i.e., in what business domain).

Here is a specific example of software that resembles in important ways the example of Indian restaurants. Instead of Indian food, the star is a specific optimization method within the field of Operations Research (OR) called Linear Programming (LP). While most people know there are loads of Indian people in India and elsewhere and they have a unique cuisine, understanding this is helped by the fact that we can picture India on a map of the globe and we all eat some kind of food. How many people have even heard of OR or LP, or have seen and understood software of any kind? To be clear, what I mean by "having seen software" is seeing the source code, not the results of it on a screen. This LP software sits around, widely used in some domains, and completely unknown in others, just like there were no Indian restaurants in New Haven before a certain date.

Knowing this helps you predict the future of software, because in a shocking number of cases, it doesn't need to be invented, but "merely" imported into an area of use where it is not currently in use. That is a big reason to be interested in understanding software evolution.

Barriers to Software Innovation: Radiology 2

Value-creating innovations are rarely the result of a bright new A-HA moment, though an individual may have that experience. A shocking number of innovations are completely predictable, partly because they've already been implemented -- but put back in the vast reservoir of ready-to-use innovations, or implemented in some other domain. This fact is one of the most important patterns of software evolution.

Sometimes the innovation is created, proven and fully deployed in production, like the optimization method Linear Programming, which I describe here. In other cases, like this one, the innovation is built as a functioning prototype with the cooperation of major industry players -- but not deployed.

In a prior post I described how I went to the San Francisco bay area in the summer of 1971 to help a couple of my friends implement a system that would generate a radiology report from a marked-up mark-sense form. We got the system working to the point where it could generate a customizable radiologist's report from one of the form types, the one for the hand. Making it work for all the types of reports would have been easy -- we demonstrated working software, and wrote a comprehensive proposal for building the whole system. It was never built.

True to the nature of software evolution, the idea probably pounded on many doors over the years, always ignored. But about 10 years ago, a pioneering radiologist in Cleveland came up with essentially the same idea. Of course, instead of paper mark-sense forms, the radiologist would click on choices on a screen, and would usually look at the medical image on the computer screen. This enabled the further benefit of reducing the work, and letting doctors easily read images that were taken in various physical locations. Tests showed that doctors using the system were much more productive than those who worked in the traditional way. Finally, they decided that mimicking the radiologist's normal writing style was a negative, and that the field would be improved by having all reports follow a similar format, with content expressed in the same order in the same way. This was actually a detail, because the core semantic observations would be recorded and stored in any case, enabling a leap to a new level of data analytics. It also, by the way, made the report generation system much easier to build than the working prototype we had built decades earlier, which enabled easy customization to mimic each radiologist's style of writing.

The founding radiologist was a doctor, of course, and knew little about software. He did his best to get the software written, got funding, and got the system working. Professional management was hired. My VC group made an investment. Many people saw the potential of the system; it was adopted by a famous hospital system in 2015. But in the end, the company was sold off in pieces.

Nearly 50 years after software was first written that was able to produce medical imaging diagnostic reports quickly and reliably while also populating a coded EMR to enable analytics, the system is sitting in the vast reservoir of un-deployed innovations. It can be built. It saves time. It auto-populates an EMR.

Many people have opined on why this particular venture failed to flourish. It's a classic example of the realities of software innovation and evolution. The reasons for failure were inside the company and outside the company. For the inside reasons, let's just say that the work methods of experienced, professional managers in the software development industry lead to consistently expensive, mediocre results. Nonetheless, the software worked and was in wide production use, delivering the advertised benefits. For the outside reasons, let's say that, well, the conditions weren't quite right just yet for such a transformation of the way doctors work to take place.

The conditions that weren't right just yet for this and uncountable other innovations add up to the walls, high and thick, behind which a reservoir of transformative innovation and "new" software awaits favorable conditions. In other words, the reservoir of innovations wait for that magic combination of software builders who actually know how to build software that works, with a business/social nexus that accepts the innovation instead of the standard no-holds-barred resistance.

Corporations promote what they call innovation. They are busily hiring Chief Innovation Officers, creating innovation incubation centers, hanging posters about the wonders of innovation, etc. etc. They continue to believe the standard-issue garbage that innovation needs to be invented fresh and new.

The reality is that there is a vast reservoir of in-old-vations that are proven and frequently deployed in other domains. All that's needed is to select and implement the best ones. HOWEVER, a Chief Innovation Officer is STILL needed -- to perform the necessary function of identifying and breaking down the human and institutional barriers that have prevented the in-old-vations from being deployed, in many cases preventing roll-out for -- literally! -- decades!!

The Comes-before is not Causation Fallacy in Software Evolution

Practically no one understands software, so why should we expect anyone to understand the broader and in some ways more demanding subject of software evolution? But while rarely (as in, almost never) studied, thoughts about software evolution are often found lurking in the backs of people's minds, and pop out in the things they write.

In biological evolution, putting aside nasty things like DNA and all the stuff it can and can't explain, there is a now-obvious pattern in the fossil record of how species have evolved. There is a clear progression from simple to complex, less capable to more capable and so on. Cold-blooded lizards are pretty neat, but there's little doubt that warm-blooded mammals rule, and that mammals appear in the fossil record after the cold-blooded animals. The picture of this nearly all of us share is that the later creatures evolved from earlier ones, in a more-or-less continuous progression. No leap-frogging is involved -- just a step-by-step process of learning and mostly improving.

A recent example in Wired Magazine about the Apollo 11 computer system illustrates this way of thinking very well. Here's the title:

Her Code Got Humans on the Moon -- and Invented Software Itself

The Apollo code in general and Margaret Hamilton in particular "invented software itself." Really? Let's look at some of the article. Here's a key paragraph:

By mid-1968, more than 400 people were working on Apollo’s software, because software was how the US was going to win the race to the moon. As it turned out, of course, software was going to help the world do so much more. As Hamilton and her colleagues were programming the Apollo spacecraft, they were also hatching what would become a $400 billion industry.

The last clause of the above quote clearly states that the $400 billion software industry that eventually emerged was "hatched" from the Apollo effort. Like so many people, the author thinks that the 400 person effort to build Apollo software somehow made possible, laid the groundwork for, or something all the developments to follow.

What a pile of hogwash. But typical!

My personal initiation into programming was on an IBM 360 model 50 computer that I was able to access on Saturdays during the 1966 school year when I was in high school. I wrote FORTRAN programs, key-punched them into cards, fed them into the card reader, got direct feedback from the computer about my imperfections as a programmer, and finally got results I wanted. While doing this, I was one of many people at many facilities using this software.

The IBM System 360 effort was started about 1960, and was announced in 1964. Gene Amdahl was the main hardware architect, tasked with the then-unprecedented task of creating a family of computers with a wide range of capabilities that could all run the same instruction set, i.e., run the same software.

Operating system software for such a system had never been created, and was a difficult task, particularly because supporting running multiple programs at the same time, again a first, was a requirement. The software was much-delayed, and by the time Fred Brooks took over the effort, more than 1,000 people were working on the project. Brooks later went on to write a classic book inspired by the effort, the Mythical Man-Month.

The System 360 was wildly successful, and fueled the incredible growth of IBM. Its instruction set lives on to this day in the System z mainframe servers. Its use was exploding at the time the Apollo software was still being developed.

What about the Apollo software? It was peculiar from the beginning, and exhibited issues that had already been solved in OS/360 and its programs. While the independent software industry did indeed grow and mature on S/360 computers, in spite of IBM's efforts to keep anyone from buying any software except their own, the Apollo software led to nothing. The now-lauded Margaret Hamilton tried to commercialize the special software methods she had used on the Apollo project, but her efforts failed and led to nothing. There were a couple of later attempts to build and commercialize what we can now see are versions of her ideas, for example in model-driven development, but after a period of hype they went nowhere.

None of this, by the way, is intended in any way to minimize the achievement of the Apollo mission, the pioneering computer system and software that was an integral part of it, or of Margaret Hamilton herself. She led a huge team that had to get software for a novel system and mission working the first time. It's a real achievement! But it's hardly "inventing software itself."

Even if you believe that someone "invented software itself," the inventor was certainly not Ms. Hamilton; even if you believe that a big, early software effort "hatched" today's software industry, it was not the Apollo effort.

So did IBM do it? Is IBM OS/360 the progenitor of today's software industry? Nope! A growing number of groups were designing and building computers, and even more were building software. You can see patterns, but this biological evolutionary idea of a capability being somehow created in DNA, and through inheritance and random improvement leading to subsequent efforts in a consequential way is nothing but the application of an inapplicable metaphor. While patterns can be observed, each piece of software is written from scratch, based on the ideas and experience of the authors. No software-equivalent of DNA, with the obvious exception of the use of shared code.

I doubt this little rant will be sufficient to stamp out the use of bad logic and inapplicable metaphors by people who insist on writing about software, even though they've never seen it, can't read it, and are clueless about what it really is and how it works. But at least writing this has gotten it out of my system, kind of...

The Slow Spread of Linear Programming Illustrates How In-old-vation in Software Evolution Works

There is loads of talk about “innovation.” Lots of people want to do it, lots of people think they’re doing it, consultants run courses in how to be innovative, and large organizations claim to promote innovation and be innovative. The assumption behind most of this “innovation” talk is that a wonderful bright idea that will change the world (or at least your organization or startup) can pop into anyone’s head. It’s new! It’s brilliant! We’re going to win big with this great new idea! See this for example.

When you study software evolution, you get an entirely different picture of software-based “innovation.” Software evolution shows you that new ideas that work are extremely rare. Oh sure, there’s a flood of new ideas popping into people’s heads all the time. Mostly, they’re not new, and the new ones rarely work. The software concepts that make it big are, in the vast majority of cases, clear examples of existing patterns of software evolution, and have in most cases already been implemented in a different context.

I first encountered the mystery of software evolution while in my first job programming software.

I started programming in a course I took starting in 1966 in high school, taught by a math teacher who couldn’t himself program, but had convinced the school to let him teach the course, and had convinced a local company, a pioneering rocket engine company called Reaction Motors, to give us computer time on Saturdays. I had a textbook about FORTRAN, a steady stream of programming assignments and a computer on which to test my programs. It was great! I continued programming the following summer, as part of an NSF math camp I was able to attend. As I was nearing high school graduation the following year, I got lucky; Diane, a high school friend, talked about me with her father, who got me connected with a nearby company, and I landed a job there for the summer before starting at Harvard College in the fall of 1968.

The company was EMSI, Esso Mathematics and Systems, Inc. in Florham Park NJ.

They were a service company, one of the about 100 units of the Standard Oil of NJ (Esso) companies, devoted to applying math and computers to improving every aspect of the company. I was immediately thrown into the group that was developing optimization models for oil refinery operation. Our focus was on the giant refinery in Venezuela.

What we had running was an implementation of a classic OR (Operations Research) algorithm called LP (linear programming), solved via the simplex algorithm first devised by George Danzig in 1948. In this kind of model, there is a goal equation and a set of constraints. The goal equation calculated profits, using hundreds of contributing variables, including prices you could sell things for, and the costs of various inputs. The constraints were greater/less than equations, each essentially describing some tiny aspect of how the refinery worked. What the algorithm did was find the values of the variables that maximized the goal equation (profits) while satisfying all the constraint equations.

The model was constantly being modified to make it more precise and applicable to actual refinery operations. I had a variety of jobs, including writing new code, fixing bugs, etc.

Since prices were such a key part of the LP model, we had a separate program to calculate what they were likely to be in the future, a Monte Carlo model. I also made enhancements and fixed bugs in this body of code.

I was fortunate to be able to hitch a ride to work and back with a PhD who worked there and lived in my town. During the ride I would tap his deep knowledge of things. He put me onto the various journals in which advances in various forms of OR were described, which I dove into. The math was often above my head, but I was motivated to teach it to myself on a rolling, as-needed basis.

I thought this was a really cool way of running things. There were all sorts of controls in the refinery, controls that let you create more airline fuel and less heating oil, or any number of other trade-offs. It was amazing that you could compute the setting of all the control knobs that would produce the best mix of products that the market needed. Why would you ever run any operation any other way?? It would be simply ignorant and stupid.

I finished school, learned more about the way the world worked, and searched high and low for implementations of LP. Anywhere! If they were out there, they were doing a great job of hiding.

I was confused. How could this be? LP was math, doggone it. It yielded a provably optimal way of running a business. It’s proven in real-life production at Esso. Any other way of operating a business was clearly seat-of-the-pants, wet-finger-in-the-wind amateur hour; anyone whose operation was sizable enough to justify the effort would have to use this method if they weren’t plain-and-simple incompetent. But no one seemed to be using it! What’s going on here??

This mystery was on my mind while I participated in one of the periodic AI crazes that has swept the world of with-it people. While I was still in college, Winograd published his MIT SHRDLU research, in which an “intelligent” robot would converse with a human in English about a world consisting entirely and solely of blocks. You could ask SHRDLU to “put the red block on top of the blue block” and it would do it; you could ask it questions about block world, and it would answer. Amazing! Super-practical! While this was happening, I wrote and submitted a thesis about how to structure knowledge inside of an intelligent robot. All of it useless compared to LP and the associated OR techniques.

The craze was AI, and the mania lasted a few more years, generating a steady stream of "promising" results, none of them in the same universe of practicality and benefit as LP or any other OR optimization technique, which continued to be used only in "secret" little islands of astounding efficiency and productivity.

Later in the 1970’s I first applied for a home mortgage. A key part of getting the mortgage was the interview with the loan officer. You had to pass muster to get the mortgage! Another 10 years passed before OR-type models started to be used for credit. When I next applied for a mortgage in 1981, I was interviewed by a loan officer. By my next mortgage in 1987 it was at least partly automated.

When I got into venture capital in the 1990’s, I discovered that high-value repair parts were had recently been optimized in terms of inventory levels and locations. I looked in detail at the pioneering company ProfitLogic, which did inventory and sale optimization for retail stores, answering questions like when should which products be put on what kind of sale, questions that had traditionally been answered by the local marketing “expert,” just as oil refineries had previously been run exclusively by experienced experts.

Only in the late 2010’s did exactly the same LP models start being applied to medical scheduling, to optimize the use of things like infusion centers and operating rooms. Just as oil refineries produced much more value from exactly the same crude oil inputs in 1968 as a result of LP models, so are infusion centers handling 30% more throughput using exactly the same capital and human resources using the very same LP models.

Here’s the mystery: why did it take so blankety-blank long for LP models to be applied??? There has been no theoretical break-through. Yes, the computers are less expensive, but given the scale of the opportunity, that was never the obstacle. WHY??!! The answer is simple: there is no good answer. Except of course for the ever-relevant one of human ignorance, stupidity and sloth.

The example of LP optimization and its agonizingly long roll-out through different applications and industries over more than 50 years – a roll-out that is far from complete! – is a prime example of the reality of computer/software evolution. Among other things, it illustrates the point that many of the most impactful "innovations" are really "in-old-vations," things that are just sitting there, proven in production and waiting for someone to apply them to one of the many domains which they would benefit.

Here are a couple cornerstones of computer software evolution:

• Software evolution resembles biological evolution only a little. Not-very-fit software species thrive in broad areas of application, unchallenged for years or decades, while vastly superior ones rule the roost not far away. The only reason why the superior software species don’t migrate to the attractive new place appears to be human ignorance and inertia.
• Software evolution resembles the much-derided theory of “intelligent design” quite a bit, if you make a slight edit and call it “un-intelligent, un-educated design.” A “superior” (HA!) being does indeed do the designing of the software, in the form of highly paid software professionals.
• When software appears in a new “land” (platform, business domain), it most often starts evolution all over again, first appearing in classic primitive forms, and then slowly re-evolving through stages already traversed in the past in other “lands.” This persistent phenomenon supports the “unintelligent design with blinders on" theory of software evolution.

I will explain and illustrate these points in future posts and a forthcoming book.

Barriers to Software Innovation: Radiology 1

There is a general impression that software innovation in one of its many forms e.g. “Digital Transformation” is marching ahead full steam. There are courses, consultants, posters hanging in common spaces and newly-created Chief Innovation Officer positions.  What’s new? What’s the latest in software?

The reality is that there are large reservoirs of proven, tested and working software innovations ready to be rolled out, but these riches are kept behind the solid walls of dams, with armies of alert guardians ready to leap in and patch any holes through which these valuable innovations may start leaking into practice. Almost no one is aware of the treasure-trove of proven innovations kept dammed up from being piped to the many places that could benefit from them; even the guardians are rarely fully conscious of what they’re doing.

If anyone really wanted to know what was coming in software, all they would have to do is find the dams and peer into the waters they hold back.  In spite of the mighty dams, it sometimes happens that the software finds its way into practice, normally in a flood that blankets a small neighborhood. Sometimes the flood has been held back for decades. There are cases I know of where an innovation was proven 50 years ago, and is still not close to being rolled out.

The dams are built in many ways with many materials. The raw materials appear to include aspects of human nature: ignorance, sloth, greed -- you know, the usual. The really high, solid dams have broad institutional support, in which “everyone” is fine with things as they are, and won’t so much as give the time of day to an amazing innovation that would change many things for the better – except of course for a key interest group.

Here is one of the examples I personally know about. It was one of my introductions to what innovation is all about, and the sad fact that creating a valuable innovation is generally the easy part – the hard part is usually overcoming the human and institutional barriers to deploying it.

Automating Medical Image Reading and Reporting

When a radiologist gets an X-ray, there are two phases of work. The first is to “read” the X-ray and observe anything non-typical that it shows, anything from a broken bone to a tumor. The second is to generate a report of the findings. Most radiologists, then and now, dictate their findings; then someone transcribes the dictated report and sends it as needed. The details of the report can vary depending on the purpose of the X-ray and the kind of person for whom it’s intended.

There has been technology first tested decades ago that appears to show that software is capable of “reading” an X-ray at least as accurately as a human radiologist. I will ignore that work for now, and focus on what should be the less threatening technology, which is translating the doctor's observations to an appropriate report.

While I was in college, I worked on the early ARPAnet with an amazing group of people, one of whom was an MIT student from the San Francisco area who later went on to fame making major advances in integrated circuits, among other things. The summer after we did most of our ARPAnet work, he got involved with a new initiative to transform the way radiologist reports of X-rays were created. He knew that some of my skills in automated language parsing and generation were relevant, so he invited me out to pitch in. I went.

GE, then and now, was a major maker of medical imaging systems. They were seriously experimenting with ways of enhancing their systems to make it easier for radiologists to produce reports of their findings. They created a set of mark sense forms, of the kind widely used at the time for recording the answers to tests, to enable a radiologist to quickly mark his observations of the part of the body in question. Here is the form for a person's guts:

Here is part of the form for the hand, showing how you can mark your observations:

Here is part of the form for the spine, showing how you can customize the report output as needed:

My friends had gotten most of the system together -- all I had to do was build the software that would create the radiologist's report. Because of uncertainty about radiologist's accepting the results, I had to make the report generator easily customizable, so that the radiologist's typical style of writing was created.

Leaving out the details, in a few weeks I created a domain-specific language resembling a generative grammar and rules engine to do the job -- along with the necessary interpreter, all written in PDP-8 assembler language, which was new to me. My friends wrote a clear and compelling report describing our work and included an example of our working software in it. Here was the sample filled-out form:

And here was part of the report that was generated by the software we wrote from the input of that form:

The software worked! And yes, the date on the report, 1971, is the date we did the work.

A major company, prominent in the field, had taken the initiative to design mark-sense forms, incorporating input from many radiologists. A few college kids, in contact with one of GE's leading partners, created a working prototype of customization report-generating software, along with a proposal to bring the project to production.

Just as a side effect, this project would have done something transformative: capture the diagnostic observations of radiologists into fully coded semantic form. This is a form of electronic medical record (EMR) that still doesn't exist, even today! For all the billions of dollars that have been spent on EMR's, supposedly to capture the data that wil fuel the insights that will improve medical care, a great deal of essential medical observation is still recorded only in un-coded, narrative form -- including medical imaging reports!

The bottom line is that this project never got off the ground. Not because the software couldn't be written, but because ... well, you tell me.

See the next post for the continuation of this sad but typical story.

The Evolution of Software

There is strong interest in the latest developments in software. No one wants to be left behind.

At the same time, there is a peculiar (to me) lack of interest in whatever the latest thing grew out of or evolved from. What are the new thing's predecessors? What did that thing grow out of. Have similar things appeared in the past? Are there patterns we can observe here, or are the new software things that explode onto the scene just a meaningless sequence of random events? Are they things that we obsess about while they're here without questioning or wondering where they came from, without asking what else they may resemble in some way? Are they things we just forget about and ignore once they depart the stage of software fads?

The questions have obvious answers. Answers that no one seems to be interested in.

So long as we fail to ask these crucial questions, we will remain ignorant of software in deep ways, and will continue to stumble from fad to fad without improvement or understanding. History is important!

Evolution in Science

If you study different sciences and their history, you notice that, at some point, scientists begin to pay attention to evolution, i.e., change over time of the thing being studied.

The most obvious thing is evolution of an individual item over time. In biology, you have the cycle from birth to death. In geology, you have the change of mountains, rivers, beaches and glaciers. In materials science you have the change of objects as they are subjected to various conditions over time, for example the way that iron rusts.

Evolutionary studies reach a breaking point and get really interesting when you study groups and types and see the patterns of change. In biology, this was the revolution that happened with Darwin and his study of the evolution of species. Something similar happened in geology with plate tectonics.

When you put the patterns together, things get totally amazing and transformative. While no longer fashionable to talk about and study, the parallels between the growth stages of an individual animal and the evolution of species are obvious, though not literal at a detail level.

Studying evolution is an important stage in the evolution of most sciences!

It's long-since past time to study software evolution, as an integral part of software science.

Software Evolution

Software evolves at every level.

At a basic level, software languages evolve. I have given an introduction to this subject here. The evolution of software languages resembles biological evolution in remarkable ways, as species (languages) emerge and evolve in response to new conditions, including other species. Similarly to biology, languages evolve; sometimes many species descend from them; and sometimes a species goes extinct. See this snapshot, for example:

(Credit)

Consistent with the determination of nearly everyone involved with software to ignore its history, this chart and anything like it are ignored, and the evolutionary principles behind it are not studied. When you start to study them, you learn amazing things. Suddenly some of the random change in the world of software languages starts to make sense.

Software languages are the very most basic things about software. Just as interesting and vastly more important are the programs written in software languages. Everyone involved in writing or maintaining a software program (application) just thinks about that program. It definitely occurs to business people what the program can do compared to its immediate competitors, if any. Yes, there is a kind of "survival of the fittest" in software evolution.

What about the "insides" of the program compared to other program insides? Sometimes people are aware of gross differences, like the language the program is written in. But the curiosity normally stops there.

The program "insides" have huge practical consequences. Depending on the details of how a program is written, it can be more or less:

• reliable/buggy
• easy to hack (secure)
• expensive to run/operate
• dangerous/costly to change
• able to respond quickly to changing loads
• speedy or slow

People commonly talk about structure or architecture, but those things are just the tip of the iceberg.

When you dig in, you find the equivalent of islands and continents of software. I've treated this subject here, for example. For example, you may have spent some number of years after getting your CS degree working on website design, and interacted with a community of people similarly engaged. Not everyone uses the same tools or does things the same way, of course, and you're likely to think there's quite a variety of approaches.

Then you go wild, dive into hospital management systems, and your mind is blown. What you thought was the wide variety of software and tools used by website designers turns out to be a whole continent apart from what you find in hospital automation. You learn that the flagship product of the company that has more than a third of the market (Epic) is written in a language you've never even heard of! It's like growing up in New Jersey among what you think is a very wide variety of people and going to Chennai -- a huge city in India you've never heard of, whose population is larger than your whole state, where the main language is one you've never heard of, written in a script you've never before seen.

Even more ignored are the obvious (to those who trouble to look) repeating patterns that can be observed in applications as they are first written and then evolve over time.

The Study of Software Evolution

To put it bluntly, the study of software evolution has barely begun. A few isolated souls, hardy or foolhardy as you like, have dipped their toes into the deep waters of software evolution. Those few have found worlds to explore, oceans whose depths have yet to be plumbed.

This is particularly the case because software is a "the only reality is the present" kind of field, as it now stands. It doesn't have to be this way!

Very few people study the history of math. They don't need to: anyone who learns math starts with math as it existed thousands of years ago. If they do well in math, they get all the way up to ... the late 1600's, when Newton and/or Leibnitz discovered calculus! Each step in math is based on and often depends on the earlier steps.

In some sense, this is also true of software. Except not really.

First, knowing nothing about software, you can dive right into the latest language and tools and use them to make something that works. Anyone think you could do that with differential equations, I mean without already knowing how to count, add and multiply? The difference is that in software, the "accomplished" new-comer really is standing on generations of older software ... of which s/he is blissfully unaware! The new programmer, of course, uses an incredible array of operating systems, drivers, networking, file and database systems, and on and on. These form the "land" on which the new programmer "stands" to work. None of the wonderful new stuff would work without it. The big difference, of course, is that using the underlying software and understanding the underlying software are two rather different things. This makes software totally different from not just math, but most of the other legitimate sciences (leaving out the fake ones like political "science" and the rest).

Second, when you're building a program, chances are excellent that you're not the first person to try building something very much like that program. Lots of details may be different, but even the details are probably not as unique as you might think. Suppose you need a program that sets up an account for a person, and then when the person does stuff, the details are checked with the person's account, and if OK are added to the account, and if not are rejected but recorded. This high-level description applies to a sales operation with customers, a medical office with patients, a manufacturing company with customers, and a bank. Given that programs like this have been built tens of thousands of times, don't you think that the ways of building programs like this would have evolved? That there would be pre-built parts, specialized tools for using the parts, building new ones and gluing them together? Special methods that have evolved and been refined to make sure it's done in the optimal way? Did any of you with any level of degree in Computer Science learn any of that -- other than, of course, some boiler-plate about the virtues of object-oriented languages and maybe some other fashionable stuff? Of course not! Anyone with any sense of fashion and/or status knows that these are not career-advancing subjects. Anything involving software history or comparative study of software projects is career-killing.

Conclusion

Software evolves. But it evolves differently than other things studied by science. Virtually no one studies software history in any way, much less the patterns of evolution that become apparent when you study that history. There have been a couple of attempts to learn why software doesn't get better in the way many other things do, most notably Fred Brooks with his book The Mythical Man-Month and other writings. I have dissected this flawed analysis in my work on Software Project Management and Wartime Software. See this for a start.

I have worked for years trying to identify some notable patterns in software evolution, and will be releasing some of that work in the coming months.

How We Got Chatbots for Mobile Financial Apps

Chatbots for financial and other applications aren’t just a cool new thing. They’re a necessity! They solve the worsening problem of too many options to choose from on shrinking screens, with unhelpful help screens.

Do you access your financial accounts online? If you do, perhaps you’ll remember that the first time you tried to do something, you had the fun of poring over the menu system to find what button to click, sometimes only to reach another screen full of buttons and menus. On the plus side, online financial systems let you get a lot done. On the minus side, jumping through the hoops to actually get them to do what you want can be a long slog. Have you ever gotten frustrated and tried to click on Help? And gotten the long, unhelpful Help stuff? Helped a whole lot, didn’t it?

Today, nice big screens with high resolution, sitting on a desk somewhere, are used less and less. Cute, portable little screens with phone and camera built in are used more and more, partly because it’s in your pocket, right there when you need it. What happens to all those giant screens packed with menu choices? I guess the designers could have reduced the typeface so much that you’d need a magnifying glass to read them, but they bowed to reality and put just a few menu choices on each screen. Nice to read, but the result is that your multi-screen journey to get to the one you want got even longer. Assuming you remember how to get there. And what if you want something more elaborate, like how close am I getting to hitting my budget for restaurants this month? Fuggedabout it.

The designers of mobile apps for financial applications aren’t between a rock and a hard place. It’s worse. They’re stuck way back in a long, narrow cave that floods. What to do?

Echo and Siri have been training us to just ask for things. Who won the game last night? What’s the weather forecast? Even jokes! But banks … now that’s a different matter altogether. Banks are serious things. There’s money involved. Not just any money – MY money!

So what’s a bank app creator supposed to do?

It’s actually pretty simple, because there’s not much choice. If you want the bank app to be, you know, USED by the people who have it, you have to make it USABLE. Period. There’s not enough room for menus, except maybe a couple super-popular buttons. Help files? Waste of time. You’re down to one choice: make it so you can TALK (or chat) to the app, and ask it to do stuff for you.

That’s the logic. Even better, it’s really happening. In real life!

Bank of America is advertising its chatbot Erica heavily in some parts of the country. The reason is simple: they want their customers to be able to USE the BofA app, not be frustrated by it. There’s the chatbot technology provider Kasisto (Disclosure: Oak HC/FT is an investor) used by multiple banks to power human interactions. The bar has now been raised for financial institutions with apps.

Let’s review some history. Years ago, what financial institutions had to have was a website for customers to access their accounts. As smartphones spread, the bar was raised: OK, you’ve got a website, but do you have an app? Not having an app was a reason for customers to move to a place that did. Just as most of the financial institutions were breathing sighs of relief that they’ve caught up, the bar is raised again: you mean we have to make the app USABLE? What’s a chatbot, anyway? The pattern here is clear: technology keeps marching along, some financial institution applies it to their customers’ and their own benefit, and the others scramble to catch up. What’s next?

To find out what’s next, all we have to do is look at the non-financial domains that use chatbots. For example, look at Amazon’s Echo. It started out pretty primitive, but it’s adding capabilities all the time – as Amazon puts it, new “skills.” The bar is already being raised by technology vendors in two specific, technically challenging areas:

It’s one thing to answer a simple question, like what’s my balance? But real chatting as done by people requires that the chatbot take into account complex questions, context and history.

A complex question requires doing some real “thinking.” For example, “How much did I spend on eating out last month?” has loads of complexity. The system has to know that “eating out” means spending money on the merchant category “restaurants.”  It’s got to find all those transactions that took place in the month before today’s month, add them up and give you the answer.

Taking conversation history and context in account is even trickier. Suppose you get the answer to the eating out question. Suppose you now ask “ How about the prior month?” Easy for a human; for a bot, not easy at all! The bot has to figure out that you’re still talking about eating out (restaurants) and that you want the total for a month. It’s also got to figure out that you want not last month, but the one before that. These are the kind of amazing interactions supported by Kasisto.

Chatbots for financial apps are just being rolled out to solve the otherwise unsolvable problem of screen real estate. Even though we’re still in the early roll-out stages, the next battle is clear. Most of today’s chatbots are in elementary school – soon they’ll need to graduate to middle school!

A slightly different version of this post originally appeared in Forbes.

Here's what we can learn from the shift to smart credit card terminals

I’ve been involved in computer software for decades. Lots has changed over that time. One thing that hasn’t changed is the question people most like to ask me. It’s this: “What do you see that’s new and interesting?” It’s a perfectly reasonable question, though one for which I rarely have a ready-made answer.

A question I never hear goes something like this: “What do you see that’s touted as the newest new thing, but is mostly old stuff, and was completely predictable?” Now that’s an interesting question. And the un-helpful but honest answer is “Practically everything that’s touted as a new thing is mostly old stuff, with a little bit of ‘obvious next step’ thrown in for variety.”

Still, there are some unpredictable aspects of the fancy new things: it’s really hard to know WHEN the new thing will happen and WHO will make it happen.

A case in point is … [I’m not sorry about the pun] … the new smart card terminal company Poynt. (Disclosure: my VC fund, Oak HC/FT, is an investor.) I can see eager marketing people at Poynt are raising their hands in the back at this … ahem … point, all anxious to point (groan…) out that Poynt is a pioneer in the market, arguably the inventor of the smart terminal, an amazing device that not only takes card payments, but also rings up items just like a POS terminal and hosts endless numbers of third-party apps. True! I happily concede the point. But I hasten to point out that there are robust competitors in the market, notably including Square and Clover.

The smart terminal is a new thing, and the market is glad to have it, but it’s hardly a NEW new thing, or something where you’d knock your head and say “now who’d-a thought-a that!?”

It’s natural for consumers of technology to look at the new devices and appreciate them for what they are. That’s like being a tourist, driving on a road through the country-side, appreciating the nice new views. That’s nice for the tourist, but are there patterns here, patterns that would enable an educated person to expect something like a smart terminal to appear, and womdering when it would happen?

Yes there are. The main pattern at work here is the rate of change of the underlying hardware. Today’s hardware is something like 1,000 times faster than much larger, more expensive hardware at the turn of the last century, less than 20 years ago. That number may not seem like much, but think of this: the average human walking speed is about 3 mph. The speed of a commercial jet while flying is less than 600 mph, about 200 times faster. Now imagine a human evolving so quickly that the human could walk at the speed of a jet -- and increase in computer speed is 5 times greater than that, in less than 20 years!

What’s the point? Or Poynt? Here it is: there are underlying geology-like forces in the world of computing that make it highly likely that something very much like Poynt would be invented – though as I said, predicting who will do it and when they’ll do it is a whole other thing.

The first step in creating a technology solution to a problem is often building problem-specific hardware. Then the technology evolves, getting faster, cheaper and more capable. Then there’s a tipping point, at which the purpose-specific hardware is replaced by general-purpose hardware, and most of the specific features of the device are implemented in sofware that runs on the general-purpose hardware. Then a new era begins. The general pattern is that special-purpose devices are supplanted by general-purpose ones.

In the case of card processing technology, first we see imprints of cards made on paper, with the physical paper being sent to a central place for processing. Then the big jump to computer technology and networking: a series of increasingly-better charge terminals, specifically made for processing card charges. The terminals evolved from dial-up networking to the internet, and from stand-alone to connected to a point-of-sale system. Wonderful devices!

Now think about cell phones. If you’ve been around for a little while, you remember big phones getting better and smaller and finally evolving into flip phones. Great phones, … but they’re phones. Then came the big shift, to a next generation of phones that were really small, portable, general-purpose cmputers that can run a myriad of applications … with cell phone hardware and software built in. Yes, it’s a phone. But it runs Facebook, email, and any of thousands of applications avialable in the app store. It’s a “smart phone!”

You know this. The reason I’m reminding you of that history is that it’s exactly the transition that card-charging “terminals” are going through right now – as they become “smart terminals,” i.e., small, portable, general-purpose computers that can run a myriad of applications … with card charging hardware and software built in. Yes, it’s a terminal, but a smart one.

How often do you see “I’m just a phone” devices? Flip phones? Yup! The old card-charge terminal will become just as rare a sight in the next couple of years.

So are the new “smart terminals” new? Yes! But hardly unexpected, at least to those who see the clearly repeating patterns of the underlying technology.

A less Poynted version of this post was previously published at Forbes.