The Black Liszt

Software Evolution: User Interface Concepts: Whose Perspective, Who's in Charge??

This post dives more deeply into the issue of Conceptual UI evolution as introduced in this post. Understanding UI conceptual evolution, which in practice is a spectrum, enables you to build applications that have UI's that produce dramatically better results than the competition -- getting more done, more quickly and at lower cost.

Whose perspective?

The least evolved UI concept looks at things completely from the point of view of the computer – what do I (the computer, which is really the person writing the application) need to get my job done? In this concept, the UI job is conceived as the computer getting things from the user, and protecting the computer from the user’s mistakes. This was, at one time, the prevailing concept for user-machine interactions, and remains surprisingly widespread, although few people would admit to thinking this way today.

At the other end of the spectrum, the software designer looks at things completely from the point of view of the human user – what do I (the human user, which is really the person writing the application) need right now and what can I do? In this concept, the UI job is conceived as the human getting things from the computer, directing it to do things, and presenting the user with options, possibilities and help that are as close and immediate as possible to what the user probably is trying to do.

Obviously, the technical side of UI’s has played a role in what’s possible here. In the early days of computers, we were glad to have them, and decks of cards and batch processing were way better than the alternatives. Computer time was rare and valuable; people were cheap by comparison; so it just made sense to look at things from the computer’s point of view.

The equation reversed long ago. Most computers spend most of their time idling, waiting in anxious anticipation for a user to do something, anything, just GET ME OUT OF THIS IDLE LOOP!! Sorry. That was the computer in me breaking out. Normally under control. Sorry.

Now, it’s entirely feasible to construct user interfaces entirely from the human’s point of view.

Who’s in charge?

There are some cases where the purpose of a program (and its UI) is entirely to be at the service of the user, and there are essentially no external constraints or advice to be given to the user. However, in a wide variety of practical cases, there are lots of people whose concerns need to be reflected in the way the computer is used. At one end of this spectrum, the user is in charge. If the user is in charge, and if we want to make sure the user does a certain thing under certain circumstances (think of a customer service call center environment), we give the user extensive training, and all sorts of analysts look at the results, so that certain customers are responded to in certain ways under different circumstances. We monitor the user’s calls, look at what they entered into the computer, and we work on changing what they do using group and individual meetings, training sessions, etc. All our effort is focused on the user, who clearly controls the computer; if we want things to be different, we go to the center of power, the user.

At the other end of this spectrum, the computer is in charge, in the sense that all major decisions and initiatives originate in the software. After basic how-to-use-a-computer type training, the user needs no training – everything you want the user to do is in the software, from what they should work on next to how they should respond to a particular request. Everyone who would have tried to influence the users directly now tries to put their knowledge into the software, which then applies it and delivers instructions to users as appropriate. When this concept is taken to the extreme, the human operator is little more than a complex and expensive media translation device, getting information the computer can’t get directly, and sending information to places the computer can’t send to directly.

So what does this mean in reality? It varies from application to application, but the net effect is always the same – the computer operator needs little training in how to respond to customers under different circumstances, because that information is all in the software. The operator mostly needs to learn how to take his cues and direction from the software, which provides a constant stream of what you might think of as “just in time training.” The user has no way of knowing if what he’s being asked to do or say has been done by many people for years, or is a new instruction just for this unusual situation.

This approach enables a revolution in how organizations respond to their customers. It makes complete personalization possible. It enables you to respond one way to a high-value customer in a situation, and another way to a low-value customer in the identical situation.  It also enables you to make nearly immediate, widespread changes to the way you respond to customers because you have a central place to enter the new “just in time” instructions, and don’t have to go through the painful process of building customer service training materials, training the trainers, getting everyone into classes, only in the end to have inconsistent and incomplete execution of your intentions.

While I’ve discussed this concept in terms of a call center application, exactly the same idea applies to the system interacting with people directly.

The system is really in charge

Let’s understand this way of building a UI with the “system in charge” a little better, since many people are unfamiliar with it.

The first step is to put all the real knowledge about how an operator should respond to which situation in which way into the system, and to enable changes to be made at will. The next step is to change operator/user training so that they understand how to map from the unstructured interactions they have to the choices presented by the system; normally, you have to train them to do this and to know how to respond. Finally, you can provide a set of pre-recorded inputs to the operators and capture their responses, to give them practice in applying their training in real life before they are inflicted on actual people.

Instead of thinking about the UI itself, think about the training that is normally required to get people to use an application, monitor their use of it on an on-going basis, and finally to make changes to the application and how people use it. You can start by thinking of the training as being like a wizard mode of the client, but with a training/case-based spin. Your trainers could build a big branching tree of what people on the other side of the phone can say, and how we should respond. All the content would be supplied by the training/customer service group. This would operate as the default mode of the application, until an operator has “passed,” and optionally beyond.

On one part of the screen could be a list of things customers can say to us. For each item, there would be one or more variations, not identical to the text, that would be recorded. In “pure” training mode, the application would randomly pick one, and the PC would play the recording. The operator would pick the item on the list of potential customer sayings that he felt was closest, and the system would then provide a suggested reply for the operator to give, and (if appropriate) highlight a field and give the operator a directive to interact with that field. This would continue cycling until the transaction was completed, abandoned or otherwise ended.

In “assisted” training mode, the list of customer requests, suggested replies and field highlights would remain, with the customer role being provided by a trainer or by real customers doing real transactions. In this case, a recording could be made of the conversation between operator and customer for additional checking or potentially for dispute resolution.

Obviously, the application needs to be extended to provide this framework, and then to download the content. But the advantage is completely realistic, integrated training. If we make changes to the application, we can automatically throw clients into this training mode to give them “just in time” training on the new features.

For what it’s worth, this is not a new idea. For example, operator training is a big issue in large-scale call center environments. Over the years, the best of those environments have evolved from classroom training to videos, to on-screen help to something like what I've described, which is now the state of the art in large scale call centers and is supported by major vendors of call center software. It’s the best because it’s completely grounded in reality and an extension of the actual software they have to use. The power of the approach shows most clearly in post-training changes and updates. Obviously, with it this integrated, it’s pretty easy to direct a client in training to a training server and check the data he’s entering.

Now that voice bots are becoming available, this approach to building UI's is all the more important and valuable. In any case, optimizing the work of the human is always in order, as spelled out in detail in this post. This post gives a detailed description of the huge project at Sallie Mae in which I played a part in the 1990's. It describes the 10X gains that can be achieved by taking optimizing the UI seriously.The main principles of human optimization in the UI are largely ignored by UI designers, making things many whole-number-factors less efficient than they could be in many cases. Amazing but typical. I've gone into just how and why this "I'd rather be stupid and get crappy results" to building software in general and UI's in particular in this post, in which I also describe the personal evolution that led me to the thoughts.

Software Programming Language Evolution: Credit Card Software Examples 1

In prior posts I've discussed the nature of programming languages and their evolution. I have given an overview of the so-called advances in programming languages made in the last 50 years. Most recently I described a couple of major advances beyond the 3-GL's. The purpose of this post is to give a couple real-life examples of how amazing new 4-GL’s and O-O languages have worked out in practice.

I was CTO of a major credit card software company in the late 1990’s. Because of that I had a front-row seat in what turned out to be a rare clinical trial of the power and productivity of the two major new categories of programming languages that were supposed to transform the practice of programming. Of course no one, in academia or elsewhere, has written about this real-world clinical trial or any of the similar ones that have played out over the years.

Bank One and 4-GL's

Bank One, based in Columbus Ohio, was a major force among banks in the 1990’s. They were growing and projected a strong image of innovation. During the 1990’s the notion that applications should be based on a DBMS was becoming standard doctrine, and the companies that valued productivity over Computer Science (and internet) purity were united behind one form or another of 4-GL as the tool of choice to get things done. Together with Anderson Consulting, one of the giant consulting companies at the time, Bank One proceeded on a huge project to re-write all their credit card processing code into a 4-GL.

After spending well north of $50 million (I heard nearly $100 million) and taking over 3 years, the project was quietly shelved, though industry insiders all heard the basic story. No one had an explanation. 4-GL’s are amazing, so much better than ancient things like COBOL – and card processing is just simple arithmetic, right, with a bit of calculating interest charges thrown in. How hard could it be? Harder than a 4-GL wielded by a crack team of one of the country’s top tech consulting firms could pull off with years of time and a giant budget, I guess.

On top of everything else, they had a clear and unambiguous definition of what the 4-GL program needed to do in the form of the existing system. They had test cases and test data. This already eliminates a huge amount of work and uncertainty in building new software. Compared with most software projects, the work was simple: just do what the old program did, using existing data as the test case. This fact isolated the influences on the outcome so that the power and productivity of the 4-GL was the most important factor. Fail.

Word of this should have gotten out. There should have been headlines in industry publications. The burgeoning 4-GL industry should have been shattered. Computer Science professors who actually cared about real things should have swarmed all over and figured out what the inherent limitations of 4-GL's were, whether they could be fixed, or whether the whole idea was nothing but puffery and arm-waving. None of this happened. Do you need to know anything else to conclude that Computer Science is based on less rationality than Anna Wintour and Vogue?

Capital One and Java

Capital One was the card division of a full-service bank that was spun out in 1994, becoming an unusual bank whose only business was to issue credit cards. In just a couple years the internet boom started, and with it enthusiasm for the most prominent object-oriented language for the enterprise, java. Capital One management was driving change in the card world and presumably felt they needed a modern technology underpinning to do it fully. So they authorized a massive project to re-write their entire existing card software from COBOL to Java. I remember reading at the time that they expected incredible flexibility and the power to evolve their business rapidly from the unprecedented power of Java.

The project took a couple years and was funded to the tune of many tens of millions of dollars; the amounts were never made public. As time went on, we heard less about it. Then there was a small ceremony and the project was declared a success, a testimony to the forward-looking executive management and pioneering tech team at the company. Then silence. I poked around with industry friends and discovered that the code had indeed been put into production – but just in Canada, which was a new market for the company at the time, handling a tiny number of cards. Why? It didn’t have anywhere close to the features and processing power that the existing COBOL system had to handle the large US card base. Just couldn't do it and company management decided to stop throwing good money after bad.

Conclusion

Executives and tech teams at major corporations bought into the fantasy that the latest 4-GL's and O-O languages would transform the process of writing software. They put huge amounts of money with the best available teams to reap the benefit for their business. And they failed.

These projects and their horrible outcomes should have made headlines in industry publications and been seared in the minds of academics. Software experts should have changed their tune as a result, or found what went wrong and fixed it. None of this happened. It tells you all you need to know about the power and productivity gains delivered by 4-GL's and object-oriented languages. Nothing has changed in the roughly twenty years since these events took place except for further evidence for the same conclusion piling up and the never-ending ability of industry participants and gurus to ignore the evidence.

 

We Don't Need Fedcoin We Already Have a National Digital Currency

The cryptocurrency enthusiasts are at it again, with a new name and even more ambitious goals than before: now they want a “national digital currency.” Hurry! The Chinese will beat us to it, and we’ll be left behind!

Somehow, no one in the debate acknowledges the obvious fact that we already HAVE a national digital currency. It’s fast, cheap and secure! It has no issue with regulators, and it’s accepted everywhere. Who knew? It’s called … the US dollar. The wild-eyed “national digital currency” groupies prefer to ignore the fact – yes, it’s a fact – that the US dollar is a digital currency. Instead, they’re convinced it can’t possibly be a good thing, because it’s not based on brand-new, cool, “immutable distributed ledger” blockchain-based cryptocurrency technology. Bzzzt! Wrong.

The national digital currency of the USA

The people who talk about “national digital currency” are obsessively focused on cryptocurrencies. They make believe digital currencies are a recent invention, and that only things that have evolved from Bitcoin meet the description. Nonetheless, by any reasonable definition, here in the good old USA we already have a digital currency. It’s called the US dollar. It’s managed by the Federal Reserve Bank. But that’s not digital, you might say – what about that green stuff in my wallet, and those coins jangling in my pocket or purse?

I agree, we have cash. As of Feb 12, 2020 there was $1.75 trillion worth of paper cash in various denominations in circulation. That’s quite a bit. But it’s far from the whole story. For the rest of the story, we turn to the money supply, the total amount of which is one of the chief responsibilities of the Fed to maintain – and grow and shrink, as needed. There are two main measures of the money supply, M1 and M2. See this for the Fed’s definition. Basically, M2 includes checking and savings bank deposits, money market funds, and similar cash-equivalents. As of December 2019, M2 was $15.434 trillion dollars.

What this means is simple: almost 90% of US dollars have no physical existence – they are purely digital. But this isn’t just for the USA; world-wide, only 8% of currency exists as physical cash!

The US dollar took many steps over more than a century to evolve from physical cash to today’s largely digital currency. First, paper currency wasn’t “real” money – it was a promise by a bank to trade the paper for the equivalent in gold. For example, here’s a $5,000 bill from 1882 that’s a promise to exchange for $5,000 in gold coin on demand:

Bureau of Engraving and Printing color specimen of a $5,000 Gold certificate, Series of 1882

National Museum of American History - Image by Godot13

In practice, no one exchanged these large-dollar notes for gold; they were mostly used by banks and the government to move funds between themselves, a practice which stopped in 1934.

Long before the advent of computers, the gold exchange promise was dropped. Here’s a bill as printed in 1928 that simply declares that it’s $5,000:

1928 Federal Reserve note

National Museum of American History - Image by Godot13

The last high-denomination bills were printed in 1945. Large inter-bank transfers were done without the exchange of cash; tightly controlled procedures were used to transfer “money” between bank ledgers before the advent of computers. In 1969 the large bills were officially discontinued, and the government started destroying them. In 1975, the government started depositing social security payments into recipient’s accounts electronically. By 1990, all money transfers between commercial and central banks were done electronically.

There is no single date when you can say that the dollar became digital. The process of transformation took place step by step, each leading to the next. The early steps took place long before computers; the principle was established and in universal use among banks and the federal reserve already in 1945! The invention and use of computers simply enabled further automation of the digitization of the US dollar, and enabled fully real-time transfers to take place.

What all this adds up to is that the US dollar is a national digital currency, by any reasonable definition, and has been for years. The vast majority of currency value is fully and completely digital, and all large-dollar transactions are completely digital. We also have cards, which are smaller, lighter and more convenient than smartphones, with the added convenience that they don’t crash or run out of power. In addition, we have the added convenience of physical cash, 100% interchangeable with its digital currency equivalent, as we see with ATM’s every day. Cash is convenient for small transactions and for people who don’t have working, powered and connected small computers on their person. The US dollar is indeed a national digital currency, with the added convenience of cards and cash.

What’s a national digital currency?

The vast majority of people know through everyday experience that the US dollar is a national digital currency. But almost no one talks in those terms. When people use that recently-coined term, they usually means something brand-new, a form of cryptocurrency. For example, a recent WSJ article describes a push towards a “national digital currency.” One of the quoted authors waxes eloquent about its virtues, but never really says what it is.

The only way to understand “national digital currency” is to back up and look at the history of where the concept came from. While no one likes to talk about it, the undisputed origin of the concept is a brilliant, well-implemented and widely used body of software called Bitcoin. The concept and every major feature of Bitcoin was designed to operate with no central authority of any kind in charge. Amazing. How can it be that anyone anywhere could declare themselves to be a Bitcoin “bank” (they call them “miners”) and the system works? See this for an explanation. Bitcoin was also designed to give total anonymity to the people who deposit, send and receive Bitcoin, making it a favorite of international criminals around the world.

Before long, Bitcoin competitors appeared, each claiming to add or correct something important in Bitcoin; for example, Ethereum introducing the so-called “smart contract.” Next, people started talking about “blockchain” and the “distributed immutable ledger,” taking out the concept of currency. Supposedly, these technologies would solve long-standing problems involving data that was in many locations. This led to loads of blockchain start-ups and service companies, with giant corporations infected with bad cases of FOMO funding pilots and proofs-of-concept. Major companies like Microsoft and IBM now offer blockchain-as-a-service in their cloud products.

More recently, we have seen highly publicized efforts to legitimize something like an enhanced Ethereum-like currency, most prominently Facebook’s Libra, which has the backing of a large number of name-brand financial institutions.

All this leads up to the newly “coined” notion of a “national digital currency” – let’s have the US government implement it instead of Facebook and its consortium partners!

This is all-too-typical technology mania. We’ve seen it many times. The true believers ignore evidence, ignore existing practice and fervently believe in the world-transforming new technology. Loads of highly-paid executives and government leaders pay obeisance, effectively paying insurance against the remote possibility that the cult delivers real value. There’s a strong lemming effect: don’t be left behind!

Inconvenient facts

People who advocate for a “national digital currency” like to ignore the one we already have, in favor of some variation of the currency beloved by human smugglers, drug lords and international illegal arms traffickers. Like the people at the Digital Currency Initiative at the much-revered Media Lab at MIT. In a recent WSJ article, the director of the lab immediately conceded that with direct deposit of salary and Venmo to split the cost of dinner with friends, it seems like we already have a digital currency. But this isn’t good enough! After all, there are fees, and big banks are involved and sometimes transactions can take days. Ugh. With a real national digital currency, a federal cryptocurrency, payments would be “faster, cheaper and more secure.”

There are just a couple little problems. Here are some highlights:

Cryptocurrency is slow

Crypto-groupies love to talk about the slowest transactions in the multi-trillion dollar US digital dollar system. While large parts of the US digital dollar system execute huge numbers of transfers in seconds, Bitcoin takes on average ten minutes to execute a single transfer. And that’s only if you pay an above-average fee – if you don’t pay much, you could wait for hours for your transaction to process.

Cryptocurrency can’t scale

Depending on the transaction size, Bitcoin can only process between 3 and 7 transactions per second. If there were always transactions waiting to be processed, 24 by 7, at 5 transactions per second Bitcoin could handle at most 158 million transactions per year. By contrast, over 10 billion transactions are performed at just ATM machines every year in the US alone. There were over 110 billion card transactions in the US in 2016. The growth in transactions from 2015 was over 7 billion; the growth in card transactions was about 50 times greater than the maximum capacity of Bitcoin.

Cryptocurrency is expensive for users

Crypto-groupies love to talk about the high fees for doing certain dollar transactions, ignoring the immense transaction flow of cheap and easy transactions like direct deposit and ACH, which operate at huge volumes. They don’t talk much about the costs of running cryptocurrency. They’re smart to ignore the subject. Today’s Bitcoin transactions are costly, and the second you try to correct the various problems (speed, scalability, security), the costs skyrocket.

Cryptocurrency is expensive to operate

Hardly anyone uses Bitcoin, and the volumes are tiny compared to the dollar. Nonetheless, Bitcoin is incredibly, mind-blowingly expensive to operate. Even at today’s minuscule volumes, Bitcoin computer processing consumes about the same amount of electricity as the whole country of Switzerland!

Cryptocurrency loss is permanent

If you lose your checkbook, your credit or bank card or anything else, you’re OK; you contact the bank and they fix it. By contrast, if you lose your cryptocurrency key (a string of numbers), there is literally no way to recover your money. About 20% of all Bitcoin are believed to be lost, something like $20 billion!! If you lose your key, whoever gets it can take all your Bitcoin, unlike with for example a lost card, where you call the bank, report the lost card, and avoid losing any money.

Cryptocurrency is horribly insecure

The crypto folks love the fact that everyone imagines that “crypto” means “can’t be cracked.” So they avoid the subject. The fact is, crypto banks are robbed and every Bitcoin stolen all too often. Nearly a million bitcoins have been lost in this way, a loss at today’s prices of roughly $10 billion!! Even the core defense of Bitcoin has now been cracked.

No proposed crypto alternative to Bitcoin solves the problems

To the outside, crypto people are all about ignoring the problems and promoting wonderfulness. Among themselves, the relatively sane advocates recognize the problems and try to solve them, with endless variations being rolled out. In doing so, they either make the problems worse or destroy what little value there is in cryptocurrency. One of the leading ideas is to make a private blockchain, which is a pathetic joke. For example, Microsoft and Intel spell out many problems by way of selling their ineffective solution, and the Facebook Libra coalition takes the “solve it by making it worse” approach to new lows.

The strengths of the US dollar digital currency

The whiners will whine about what’s wrong with today’s US dollar. Is it really chock-full of problems, as the crypto-groupies like to say? Let’s do something rare: focus on the positive. First and foremost, let’s remember that the dollar has worked for a couple centuries now, and along the way transformed itself from physical to about 90% digital, all without breaking! In addition, it has benefited from tremendous private-sector innovation. Here are some highlights of the fastest, cheapest and most secure currency ever created:

Physical cash is great. When I’m in the city and someone gets my car for me from the garage, I like to give a tip. It’s easy: I pull out my wallet and hand over bills. Anything fully digital would require electronics and would be a pain.

Cards are great. When I pull into a gas station in New Jersey, where gas is pumped for you, I open the window, say “fill with regular, please” and hand over a card. When it’s done, I get the card and a receipt and drive off. Easier than cash because no change. This is fully digital. Today. And, at my great local gas station, they often clean my windows, so I get to hand the guy a couple bucks as a tip. Painless.

Cardless is great. I call for an Uber from the app. When the car arrives, we each check each other’s identities and away we go. On arrival, I get out. That’s it!

Wiring money for a house closing is great. I call USAA, my bank, who verifies my identity and gets it done in minutes. No going to a branch, certified checks, etc. The phone call is a good thing – it reduces the chance of fraud to near-zero, unlike the fraud-riven crypto world.

P2P apps are great. There are zero-cost, instant transfer apps like Venmo, CashApp and Zelle. These are used by over a hundred million of people, and they work. Today. How could crypto in any form be better? Actually, it would be worse. See this.

What about those awful transactions that supposedly take days? Yup, there are some. It’s called a step-by-step, no errors or crashes permitted transition to real-time. Transactions are already 100% digital, and with ACH (like electronic checks) very low cost. The version of ACH in the UK is already same-day, and ACH in the US is in the middle of a transition to same-day and real-time.

What about international payments? I guess the crypto-groupies are out of touch with what’s going on here in the real world. For personal use, credit cards are already accepted nearly everywhere, with everyone involved getting or paying in their own currency. The big complaint of the crypto people is international business transactions, involving lots of time, transfers and fees. That was true. Which is why a handful of amazing new companies have emerged and are addressing the issue. A couple of them are operating at scale and in production today.

Currency Cloud, for example, has a brilliant solution. A company that has suppliers in many countries gets the suppliers to give Currency Cloud their preferred local bank accounts. Currency Cloud itself maintains local accounts for itself in all the countries it supports. The buyer sends a payment directive to Currency Cloud, who then does a local transfer of the requested amount from its account in the target country to the vendor in that country. As the network grows, each supported country has a larger number of companies both sending and receiving payments, so that a growing number of transfers can be done completely locally – only the net payment imbalance between countries needs to be settled by Currency Cloud between its own accounts, which it optimizes for minimum cost. This is 100% digital, low cost, real-time, and operating at scale. Today.

For smaller business and individuals there are services exploding onto the scene for international payments. For example, Rapyd (disclosure: my VC fund is an investor) enables people without bank accounts to buy, sell and get paid for work in over 100 countries at over 2 million access points, where they either get or give local currency to complete international digital transactions. For example, you could be a driver for Uber and get paid, even though you have no card or bank account.

Conclusion

Get over it, crypto-fanatics and blockchain groupies. Yes, the Bitcoin technology is an impressive achievement, and highly useful to the criminal class. But it makes any real-world currency problem you can think of worse, and completely ignores the patent reality, which is that the wonderful “future” of a national digital currency is something we have today – the US dollar!

Note: this post first appeared at Forbes.

Could Blockchain Help Fix My Car That Was Destroyed by a Tree Branch?

My car was safely parked in my driveway. A large branch broke off of a tree that had recently been checked by an arborist and declared healthy. Ignoring the arborist’s expert opinion, the branch broke off and fell anyway. My formerly sound, two-year-old car was towed to a repair shop, an estimate for repairs made, and my insurance company declared it not worth fixing. Totaled.

But this shocking event had a couple good outcomes. The first was that I ended up leasing a nice new car. The second outcome was some education that is hard to come by, and has serious implications – I learned how valuable Blockchain technology would be in helping to coordinate the information and efforts of my car insurance company, the repair shop, and the car rental company that supplied me with a car until I could get a new one.

Blockchain is the immutable distributed ledger technology, a kind of distributed database that powers Bitcoin and other cryptocurrencies, whose promise is actively being pursued in many industries. What Blockchain is all about is enabling countless independent parties with independent computer systems to interact with each other in a fast, secure way, sharing information to reach a mutually desired outcome. That’s exactly what we have here, with loads of insurance companies, a number of car rental chains, and untold thousands of car repair shops – all of whom need to share information and coordinate their efforts to help the consumer with the damaged car. Perfect for Blockchain!

Think about the situation. Insurance companies are all about long documents with fine print, and long times on hold waiting to talk with someone who often can’t, in the end, do much but promise to send a form in the mail. You’ve probably driven by loads of auto repair shops. Which can handle the repair your car needs? How much will they charge? Will insurance pay for it? And then I’ll be without a car. Renting a car at the airport is one thing, but locally? How do I pick a company and get there. At the end I’ve got to deal with picking up my repaired car and returning the rental. Will insurance pay? It’s all yuck, yuck, yuck. Getting my car smashed is one thing, but this makes a bad situation worse.

Imagine what a Blockchain-fueled application could do – it could eliminate the paperwork and calls, get the insurance company talking with the repair shops and car rental companies. Blockchain would enable electronic “paperwork” to be exchanged safely and securely. The insurance company could arrange for a local repair shop that can handle my car to do the repair – and pay them directly! They could dig up a local car rental company, and arrange for me to be picked up and dropped off at the end – and pay for the car directly! If things took longer than planned, all parties could communicate directly and just get it done. It would be a true distributed transaction application, minus the Bitcoin but with the transactions I care about now – getting my car fixed!

 

I know I’ve expressed doubt about blockchain and cryptocurrencies in the past, while admiring their power. This could be the inflection point for me – a real, practical, everyday nightmare that would be transformed by Blockchain! Maybe I could even dive in and lead making it happen; wouldn’t that be ironic?

Enough of living in fantasy-land – I’ve got a car that needs fixing. With dreams of a future Blockchain-fueled revolution in the back of my mind, imagine my shock as I went through the process, and found that everybody seemed to know everything! My insurance company knew a local repair shop to use, and contacted them for me. They also contacted a local branch of Enterprise Rent-A-Car, who sent someone out to pick me up. Then I found out that Enterprise knew where my car had been towed, and was ready to pick up their car from there when I went for it. Then I found out that my insurance company was paying the car repair shop directly, and paying Enterprise directly. Then when the estimate came in and my car was declared a total loss, things were taken care of until I could get a new car – which my insurance company also helped with.

What’s going on here? Have they already implemented Blockchain?!

I started asking some questions. It turns out that the nightmare of coordination and paperwork flying around was noticed decades ago. In 1994, Enterprise created the Automated Rental Management System (ARMS^®) “to help insurance companies simplify the cumbersome process of managing replacement rental cars for policyholders.” By the early 2000’s, it was already widely used.

Things progressed over the years. As of 2017, “hundreds of insurance companies and thousands of collision repair centers use Enterprise’s value-added system, which processes millions of transactions every year.”

This sounds good, but there must be a catch. This could be some centralized, expensive enterprise system that locks everyone in. Well, maybe not:

Central control? “ABS’ approach, on the other hand, enables collision repair centers, insurance companies and fleet owners to remain in control of their data for the long term – a high priority since vehicle technology and associated repair processes are changing rapidly.”

What about data format standards, the tough thing for Blockchain? “The ABS system helps protect insurance companies, collision repair centers and fleet owners by converting their information from EMS (Estimate Management Standard) to a more secure protocol, BMS (Business Message Suite).”

I’ve learned important things about Blockchain from this experience. I’ve learned that a huge problem in car repair, insurance and rental involving many disparate parties, has already been solved and is in production, used by industry giants and thousands of local businesses. This is just the kind of problem whose solution “everyone” says Blockchain “enables.” It’s in production today. It has evolved with technology,  No Blockchain needed. So why is it exactly that Blockchain is the key missing ingredient for solving distributed data, sharing and interaction problems of this kind?

Note: this post first appeared at Forbes.

 

The Bronte Sisters and Software

Who would have thought that the amazing, pioneering and tragic Bronte sisters could demonstrate important things about software programming languages? Not me, until I started thinking about it. I realized that their achievement has a close parallel to what great programmers do: they don’t invent a new language, they use an existing language to express new things, thoughts that were in their heads but which hadn’t before been published.

The Sisters

I hope you’ve at least heard of these ladies, and even better read a couple of their wonderful novels, among which are Charlotte’s Jane Eyre, Emily’s Wuthering Heights and Anne’s The Tenant of Wildfell Hall.

Their novels were very successful. Originally published with a man’s name listed as author, their success continued after their real identities were revealed, which demonstrates that the success of their work was solely due to the quality and originality of their work There have been movies and numerous references to them and their work in other media.

If you haven’t already spent some time enjoying their work, I hope you will.

In all the talk about the Bronte’s, no one bothers to mention the perfectly obvious fact that they used the English language of their day to write their novels. English didn’t hold them back. Nor did English “help” them. Their originality was completely in the way they used the English language.

The Bronte’s and Software Languages

The obvious response to the above is … duhhh, the reason no one mentions their use of unaltered, un-enhanced English is that nearly all novelists do the same.

Now let’s turn to programming. Most programmers, like novelists, just use the language they've been given to get the job done. Most programmers, like most who attempt to write a novel, do pedestrian work.

Unlike novelists, there is a subset of programmers who obsess about which is the “best” language as measured by various scales. Programmers who consider themselves to be a cut above the rest fiercely criticize this or that tiny detail of whatever language is in their cross-hairs this morning. If their ire runs at peak level for a while, may even invent a new language. Why? Their amazing new language will “prevent” programmers from making this or that kind of error – like sure, when has that ever happened – or somehow raise who ever uses it to new levels of power and productivity and quality. Not. Never happened. Baseless assertions and propaganda.

Was something important “added” or “corrected” in the English language that enabled the Bronte’s to do what they did? Nope.

This leads to a thought that is blasphemous to the self-appointed elite of software: the software language you use is almost irrelevant, of course with some exceptions; what's important is what you write in the language you're using. Just like with novels.

Languages and science

Hold on there just a sec! Novels are fiction, meant to entertain. Completely different subject. Software is like math -- it's pure and exact, devoid of messy things like the emotions and nuances of human interaction that novels are full of.

True enough. First I would say, try having a discussion about the differences between programming languages with one of the software elite who obsess about the subject. See how much "cool, calm and collected" you get; every time I've tried having a rational discussion on this subject over the years voices have gone up notches at a time, and passion has been slopping all over the place.

Perhaps we can be enlightened by the subject that's been raised over the years of the best language for science and/or math. There are even books on the subject!

Let's take a quick look at a bit of evidence:

Skipping over loads of details, what you quickly find is that, not long after Galileo broke with tradition and wrote in his normal speaking language (Italian) instead of Latin, scientists tended to write in whatever language they used in everyday life. Chemistry was dominated by the German language in the 1800's not because German was somehow better for chemistry (which didn't stop some people arguing that it was), but because most of the productive chemists happened to be most comfortable writing in German, mostly because they spoke German. A few years ago a couple Norwegian scientists were award the Nobel prize. They probably spoke Norwegian in the lab, but if they wanted to be read, they had to write in a widely-read language: English. Not because English was "better" for science, but just more widespread at that point.

In all these cases, the language happened to be used for expressing the thoughts, facts and concepts -- which were independent of the language used!

Just like it is in ... software programming!

Conclusion

With a few important exceptions, the language you use to write a program is like the language you use to write a scientific paper or a novel. The language used is not the most important thing. By far. The most important thing is what you have to say in whatever language you end up using.

Software Evolution: User Interface

User interfaces have gone through massive evolution since their first appearance in the 1950's. Lots of people talk about this. But not many separate the main two threads of UI evolution: technical and conceptual.

The technical thread is all about the tools and techniques. Examples of elements in the technical thread are the mouse, function keys, menus, and graphical windowing systems. Advances in the technical thread of UI evolution are created by researchers, systems people and systems makers, both hardware and software. People who build actual UI’s generally have to use the tools they’ve been given.

The conceptual thread of UI evolution is about thoughts in the heads of application builders about what problem they’re trying to solve and how they’re supposed to go about solving it. Application builders are generally taught the base concepts they are supposed to use, and then usually apply those concepts throughout their careers. But all application builders don’t have the same thoughts in their heads. The thoughts they have exhibit a clear progression from less evolved to more evolved. It is interesting that the way application builders think about what job they are supposed to do is almost completely independent of the tools they have, i.e., the technical thread. Yes, they can and do use the tools available to them, but this conceptual thread of UI evolution rides “above” the level of the technical tools.

The evolution of UI on the technical side is widely discussed and understood. As hardware has gotten better and less expensive, the richness of the interaction between computer and human has increased, with the computer able to present more information to the user more quickly, and with immediate reaction on the computer’s part to user requests. For the most part, this is a good thing, although people who think only about user interfaces can make serious product design mistakes when they fail to put the user interface in the broader context of product design. For example, generally speaking, pointing at a choice with a mouse is better than entering a code on a keyboard, and giving users lots of control through a rich user interface is better than giving them no control. However, in situations where there are repetitive tasks and efficiency is very important, the keyboard beats the mouse any day of the week, and in situations where tasks performed by humans can be automated, it is far better to have the computer do it -- quickly, effectively and optimally -- rather than depending on and using the time of a human being, regardless of how wonderful his UI may be. This post goes into detail with examples on this subject.

Conceptual UI evolution, by contrast to evolution on the technical side, is not widely discussed and not generally understood. Understanding this evolution enables you to build superior software by creating software that enables tasks to be accomplished with less human effort and greater accuracy.

UI Concepts

The conceptual level of user interfaces is most easily understood by asking two questions: (1) whose perspective is the primary one in the mind of the application UI builder – the computer’s or the user’s; and (2) to what extent is the user relied upon to operate the software correctly and optimally? The most primitive UI’s “look” at things from the computer’s point of view, and, somewhat paradoxically, rely almost entirely on the user to get optimal results from the computer. The most advanced UI’s “look” at things from the user’s point of view, while at the same time imposing as little burden of intelligence and decision-making as possible on the user.

When you state it this way – a UI should be user-centered and should help the user to be successful – you may well assume that building UI’s in this way would be standard operating procedure, and that building UI’s in any other way would be considered incompetent. Sadly, this is not the case. Like all the patterns I describe in my series on software evolution, most people, companies and even industries tend to be “at” a particular stage of evolution in the subject areas I describe here; companies gain comparative advantage by taking the “next” step in the pattern evolution earlier than others, and exploit it for gain more vigorously than others.

Some of the patterns I've observed in software evolution just tend to repeat themselves historically with minor variations. Other patterns, of which this is an example, don't seem to be as inevitable or time-based. This pattern is much like the pattern of increasing abstraction in software applications, described in detail here. Competitive pressures and smart, ambitious people tend to drive applications to take the next step on the spectrum of goodness.

For UI, the spectrum can be measured. The UI that requires the least time and effort by a user to get a given job done is the best. That's it!

Do UI experts think this way? Is this a foundational part of their training and expertise? Of course not! Just because computers are involved, no one should be under the illusion that we live in a numbers-driven world. For all the talk of numbers, people are more influenced by the culture they're part of, and generally want validation from that culture. Doing something further up the UI optimization curve than is customary in their milieu is nearly always an act of rebellion, and most people just don't do it.

How to build a secure, auditable voting system

 I am a computer and software guy with experience in building systems, networks and security in multiple industries. I know only what I’ve read about the voting systems in place in the US. The basic information that’s widely available is enough to make it clear that today’s voting systems were designed using circa 1990 principles without regard to the security methods that were used by the best systems even at that time. In the light of modern systems and security design they might as well have large, blinking neon signs on them reading “Cheat Me.” This has NOTHING to do with the 2020 election, just as building a secure bank vault has nothing to do with whose money it holds. We ALL care about having safe, secure and accurate voting.

A Modern Voting System

It’s possible to build a voting system that is transparent, fully auditable and extremely difficult to cheat. Each vote would not just be recorded but also logged simultaneously in multiple places, locally and in the cloud and available for public and private review within seconds. This by itself would eliminate huge amounts of potential fraud by getting humans and local physical devices out of the counting loop and above all by ending the secrecy of today’s proprietary closed systems.

Some elements of a secure voting system vary for in-person voting and mail-in voting but ballot counting is the same:

• Custom voting equipment of any kind should be eliminated and replaced with COTS (Commercial Off-The-Shelf) equipment.
• The only local operations that should be performed are voter identification and vote capture
• Paper ballots (and envelopes, if relevant) should be scanned and the encrypted images sent securely, in real time, to multiple locations in the Cloud.
• Assure that each paper has a unique ID and include full log information, things like date/time, source IP, etc.
• All processing possible should be done in the cloud for both in-person and mail voting:
• in multiple places and clouds, with results compared
• fully logged real time in multiple places
• Converting images to votes should be done distributed in parallel
• The knowledge of whether a vote was made and who it was for should be kept separate
• The cloud tallied vote made by a person should be shown to them seconds after they vote in person
• Tallies are updated in real time by voting location including rejections with reason codes, all public
• While some on-site software is unavoidable, it should be minimal and become open source
• All places where voting takes place or ballots are counted should be under total video surveillance, streamed to multiple places, using COTS equipment
• Each jurisdiction enters the voting data uniquely under their control. For example, a state would enter information about the candidates who qualified for the ballot for governor, the US senators from their state, etc. The same would be done at the County and local levels, for example a town would add town council candidates, school board candidates, etc. In each case they would enter all the information that would appear on an electronic or paper ballot and more.
• Each jurisdiction would enter additional voting requirements such as what kind of ID is required, whether signatures are required and/or checked for mail ballots, etc. All such parameters are public and changes are logged.
• The parameters made public actually control the software operation in addition to being publicly readable documents, leaving no room for secretly interfering with the operation of the software.

Scanning and Processing a Ballot Overview

The process of scanning and processing a ballot would be nearly identical whether voting in person or by mail. This by itself is a big step ahead in efficiency and security. This means that a person fills out the same ballot in the same way whether at an in-person center or voting by mail. They see how they voted by looking at where they marked the paper. The only thing done locally by electronics during either in-person voting or mailed ballot processing is scanning the ballot (like taking a picture of it) and sending the unprocessed, encrypted image to multiple secure storage locations in the cloud, along with logging this in the cloud. Once ballot images are sent, stored and logged they can’t be altered or discarded. This reduces to a bare minimum the exposure to error or fraud at the processing location.

The fact that images are converted into votes and tallied within seconds by distributed software working in multiple clouds comparing their results and then showing them to the voter further reduces fraud – each voter can see whether their vote has been tallied correctly.

Various jurisdictions have methods to prevent someone voting more than once today. This is a big subject. Briefly, the key is assigning each qualified voter an ID. The vast majority of places already do this in some way, which is how they can mail ballots to registered voters and make sure no one votes in duplicate. A uniform system of voter ID’s would be created, each linked to whatever local method is in use to minimize disruption. The ID’s would be printed on all ballots mailed and affixed to ballots made available to people voting in-person. The ID’s would be scanned as part of the ballot and processed by software in the cloud immediately. Because this is done within seconds of ballot presentation, attempts at duplicate voting in any way would be caught and prevented from being included in vote totals.

Ballot Processing Center Setup

Most of a ballot processing center would be the same for in-person voting or for processing mail-in votes. In fact, centers could be used for both purposes, even at the same time.

Each center would have a unique name displayed near its entrance. There would an appropriate number of workstations, perhaps tables with chairs, for processing ballots. Each workstation would have a unique name prominently displayed on a sign, for example the name of the location and the number of the workstation. Each workstation would be equipped with a modern high resolution color scanner, preferably duplex (scans front and back in one pass). If for any reason duplex scanning is not practical or available, the operator would scan in two steps, front and back. But duplex is preferable because it eliminates a step and reduces the chance of operator error. There are a variety of COTS scanners with the quality and resolution to handle the job. The scanner would be connected to a simple off-the-shelf laptop computer with a tiny amount of custom software whose only job would be to control the scanner and copy the scanned images to multiple locations in the cloud, with each step logged multiple times. The laptop’s camera would be live and also streamed to the cloud.

In order to minimize the possibility of hacking, the computers used would be recent, bare, off-the-shelf models. Since all the leading laptop operating systems are huge, incredibly complicated bodies of software that have security holes and need regular patches, the software used on it would a small amount of custom compiled C code that would be made open source to enable the software community to find and correct errors and security issues. During room setup the physical machine would be connected to the internet; the custom control code would be downloaded, wipe the machine clean and make itself the only software on the machine. An operator would point the machine’s camera to the ID sign on the workstation, which the software would read, register itself to the cloud and display on its screen. It would then communicate with the cloud control software that would cause other tests to be made. There’s lots of detail here that is beyond the scope of this brief description, but the point is that the opportunities for local shenanigans or errors are brought to near zero.

Each room used for handling ballots would be equipped with multiple modern security video cameras streaming to the cloud to assure that nothing improper was done during ballot processing. Old-style proprietary video security systems would not be used. The video stream would be made available at least to appointed observers and perhaps also to the public.

Voting in Person

A voter checks in much like today, presenting ID as required. If they are deemed qualified to vote they are given a paper ballot with their unique voter ID affixed to it. They go to a private desk area to fill out the ballot. They take the ballot to an available scanning workstation and feed their ballot to the machine. After the ballot is read it is put in a discard bin.

If the voter chooses they can pause at the workstation for a couple seconds and wait for the results of processing their vote to appear on the screen of the laptop at the workstation. When the image of their ballot has been securely stored in the cloud and their votes extracted from the image, validated and tallied, the candidates and issues they voted for are displayed on the screen. The voter knows that not only have they voted but that their votes have been recorded and correctly tallied without delay, and with no further steps required. They can then leave the workstation, pick up their “I Voted” sticker if they like and leave the place of voting.

Additional security would be provided by asking each voter to not just look at the things they voted for displayed on the screen as recorded and tallied, but also to validate that the votes recorded for them do indeed match the votes they think they made. This would be done by giving the user two screen choices: "votes are correct" and "votes aren't correct."

Exactly how re-voting is handled must be done carefully, because it's an opportunity to introduce fraud.  One way to handle it would be for the user to touch the votes on the screen they want changed and once the voter is satisfied to click to submit the corrected vote. I suggest a paper-based method be used to assure that this potential door for hacking to enter remains closed. For example, the voter would take one of a stack of blank "vote correction" sheets, sign it, copy a number onto it that is displayed on the screen and scan the sheet to prove their intention to make a correction. The frequency and distribution of corrections should be monitored closely in real-time to detect bad things happening.

Voting by Mail

Mail-in ballots that arrive by any method would be immediately processed in the same kind of room as used for in-person voting – it could be the same room! There is no reason why the two kinds of voting couldn’t be done at the same time.

The mail-in ballot processing operator would:

• pick up the top envelope from the stack of unprocessed ballots, the in-box
• show the envelope to the camera being sure to avoid obscuring the unique code on the envelope
• This image provides a double-check that the ballot was processed
• scan the envelope (duplex, front and back, in one step)
• remove the contents of the envelope and place the envelope into a processed box, the out-box
• if the contents is a secrecy envelope, scan it (duplex, front and back), remove the ballot and place the envelope in the out-box
  
• unfold the ballot if required and scan it
• place the scanned ballot in the out-box.
• Repeat until the in-box is empty.

Each log entry would contain a date/time stamp, location GPS and name and workstation ID, links to the corresponding images that were scanned and to the place in the video log that captured the scanning process. There would be software running in multiple cloud locations that would process each log entry as it was written and make counts and summaries publicly available via API and web pages. The same software would produce real-time roll-ups so that anyone could follow the progress of ballot registration.

Many states have systems to enable mail-in voters to see whether and when their ballots have been received and then whether they’ve been accepted. Each ballot was printed with a code uniquely assigned to a voter. As soon as the log entry was written for the images the fact that the ballot with the voter’s ID was received would be sent by the cloud software to the state system for updating that voter’s ballot status to “received.” After complete processing, which would normally just be seconds later, the status would be updated as appropriate to “accepted” or “rejected.” The system would provide the state the reason the ballot was rejected, which could include duplicate ballot, lack of signature, ballot not in the right envelope (ID mis-match), etc.

Turning the Ballot Images into Votes and Totaling Them

Ballot images are captured and stored securely in the cloud along with detailed logs in the same way for in-person and mail-in ballots. The only differences are that additional images are stored for mail-ins – the envelopes, both outer and security.

The crucial process of “reading” the images would be performed by software running in the cloud. Multiple copies would be run on different servers controlled by different entities. The output of each major result from the software would be sent by that software to its siblings, all of whom would have to declare that the results matched in order to proceed. They would then “report” their results to multiple queues and  log files. This method of operating copies in parallel and comparing results is an established method of assuring against malicious actors and assuring fault tolerance.

In addition, the software would not be in a single block. The images would be broken up and the different parts processed by different pieces of software. This is a modern approach to building distributed software that is usually called micro-services. It is used to build highly scalable, secure and fault-tolerant systems out of components (called “services”), each of which has a small, isolated job to do. Using this method, the software service that decides whether a signature is present or whether a box or circle has been filled in to indicate a vote has no way of knowing who or what the vote is for, and therefore no way to slant the results. In the unlikely event of a successful hack of one or two pieces of software, it wouldn’t be able to hack the results.

To process the images the software uses modern OCR (Optical Character Recognition) algorithms. OCR is used on both the preprinted text and whatever the voter entered on the ballot. OCR is a mature technology, with software libraries widely available and deployed in production today. OCR is used by bank apps to enable customers to electronically deposit checks using just an image of the check taken by smartphone. Higher quality versions are in broad use today that enable people to scan and submit bank statements, pay stubs, ID’s, and many other document types in order to apply for a loan on-line, become an Uber driver and a host of other purposes. OCR software is no less accurate than human readers who type what they read, and arguably more accurate.

It’s important that the image processing be done in small steps so that no one piece of software processes a whole ballot. This serves multiple purposes including keeping a voter’s votes secret and maximizing the defense against hacking. Once an image was scanned, there are proven techniques for accomplishing this that are faster, more accurate and secure than the processing that is done today for both in-person and mail-in ballots by humans and local machines. Here are the highlights:

• Every image is stored with strong encryption in deep cloud storage with multiple backups.
• Paper ballots today do not normally have the voter’s name on them. The name appears only on the containing envelope. This is  a good practice and should be maintained.
• The image of the ballot itself would be processed by program that would use OCR to “read” all the printed text much like a person would.
• The OCR would pick out each candidate name and issue description and identify the area on the image in which a voter is supposed to fill in a circle in order to vote.
• The same program would create a unique ID for each snippet of the ballot image that the voter could have filled in and write each little image to a new file along with the identifying information, put it on a queue and create a log entry.
• Multiple copies of separate “vote recognition” programs would  be constantly reading the queues and reading the vote snippets. They would evaluate each snippet for whether it had been filled in or not according to uniform standards – without having any information about where the vote was made, which candidate the image was associated with or who the voter was. Each program would then write its results to a queue and log file itself. This file would contain the vote recognition program’s unique ID, the unique ID of the snippet and its judgment of whether or not it had been filled in.
• Separate “vote collector” programs would read the queues of the “vote recognition” programs to gather all the votes in a single ballot together. These would be written to a queue and log of their own.
• The first ballot-reading program would read the collected vote queue, use its data to see which vote was for which candidate as read by it from the ballot and write the final vote tally into a multi-copy log file. The most important data in each log entry is the list of candidates who received a vote. The unique ID of the image would also be in the log entry, linking them to make a completely transparent audit.
• It is essential that this step be performed in parallel by multiple copies of the software running in completely separate clouds and the results only released when all the copies reach consensus.
• If there are enough software copies, say a dozen, then if all but one report the same results, the exception is logged and discarded.
• If something goes wrong with a cloud location or service, so long as most of the services copies are alive the process would be unimpeded.
• Finally, vote tally programs would read the vote logs in real time and update vote totals in real time for anyone to see.
• Each individual in-person vote would in addition be immediately returned to the voting site and specific workstation for display to the person who voted.

The steps described above provide the general idea and specifics for ballots. Some fraction of the votes will be mail-in. They will be marked as being mail-in during the scanning process and will consist of more images; for a typical vote there would be six images, two for each of two envelopes and one ballot. Depending on the requirements set the system would:

• OCR and check the envelope.
• Record the ID on it and assure that the person hasn’t already voted.
• OCR and check the inner envelope.
• Apply the same ID check and assuring that it matches the containing envelope.
• Apply any signature requirements (see below)
• Process the ballot as already described, checking the ID for match.

Using modern technology the entire process just described for either in-person or mail-in should take place in seconds.

Suppose 200 million people voted and all the votes arrived in a 10 hour period, which they wouldn’t. This is 555 votes per second. Suppose that just a hundred machines were used; many times that are available in each of the major cloud services. This would mean that each of the 100 machines would need to handle roughly 5 votes per second. Even with all the parallel and additional processing and checking, this is a laughably trivial load for modern machines and networks to handle. The system would be self-managing like any good micro-service mesh and automatically scale up the servers if and when needed.

This is not a software specification. I’ve written many such documents and am well aware that there are a number of conditions and details I have not addressed. This is an overview to give a sense of the overall approach.

Checking Signatures

I have purposely left the issue of signatures to the end. I don’t want to address the question of to what extent they should be required and checked.

These are the main elements of a solution that can be applied to whatever extent is decided. First seeing whether there is a signature:

• The program that handles the image (probably not the ballot) that can have a signature would perform OCR on the image to identify and extract the portion of the overall image that could contain a signature, much like the ballot processing program extracts the areas on the ballot that the voter could have filled in.
• Much like the process described for seeing if a voting circle has been filled in, a separate program receives the signature image and decides whether there is a signature. The several such programs that look at this image assure that they concur and the results are logged and sent back.
• This information is then read by an additional signed-vote program. It takes the input from the signature-page-reading program and the is-there-a-signature program, and combines it with the input from the ballot reading program, creating the log that the vote tally programs read. This enables them to create separate talles of valid and invalid votes.

If signature matching is also required, additional steps must be performed. In short they are:

• The voter rolls with signatures should be scanned in advance of voting.
• The same physical equipment should be used as for mail-in ballot processing
• The software should get the name and address of the voter, the ID of the voter as used by the relevant authority and an image of the signature.
• Unfortunately, the exact method of doing this may vary by jurisdiction. I don’t know enough about current practice to handle this issue with confidence.
• When ballots are mailed to voters, the ID’s placed on the mailed documents should be put into a secure online table to enable the signature images as scanned during registration to be matched with signatures made on voting documents.
• During vote counting the same process to extract signature images as described about should be followed. The process to determine where a non-blank signature exists should also be followed.
• If the signature doesn’t exist at all, the vote is invalid and should be handled as above.
• If the signature exists, there are two ways to handle it, which could be done in any combination.
• The automated method would be done entirely by software. Probably using a machine language method called convolutional deep learning, neural networks would be trained with huge samples of real-life signatures with multiple signatures from the same person. For example check archives could be used for this purpose.
• The widely used human-in-the-loop method would show workers pairs of signatures on the same screen with no other information. One would be the original signature and the other would be the signature on the mail-in ballot. The worker would enter one key for “match” and another key for “no match.” No other information would be provided. The system would assure that the humans who saw the signatures lived far away from the signers, but in any case the checkers would only see each pair of images for 5 seconds or less.
• Each pair of signatures could be presented to multiple human and/or automated checkers and the results compared.
• This is a huge subject, but elaborations of these basic procedures would produce results that were no worse than today’s physical methods, with very low probability of bias entering into the process.

The methods I’ve described here can be applied to other things the voter may be required to write on an envelope, including for example a date.

Software Controls and Parameters

All software has controls and parameters to adapt to local requirements and conditions. In primitive systems like today’s proprietary machines, each machine is set up by a local systems administrator, who can set and change the machine’s parameters and controls at any time.

In this system, all controls and parameters for all software are contained in a centralized system of tightly controlled and logged editable metadata to control all operations of the system instead of typical administrative control and parameters. This is a key aspect of making a diverse set of micro-services fully controlled and coordinated, while conforming to the requirements and conditions of each jurisdiction. The metadata would be organized in a hierarchy with inheritance, so that rules set at a state level would automatically inherit down and control the relevant aspect of domains within its jurisdiction. The hierarchy would establish a parent/child relationship between blocks of metadata so that counters such as voter and candidate vote totals would automatically roll up. There could be multiple hierarchies enabling for example a voting location to belong to just one town, but the town to belong separately to a county and a congressional district.

The metadata would control exactly which images constituted a mail-in vote, the tests to be applied for validity, the reason codes used for rejection, etc. This is an important aspect of making the system operation fully transparent – the metadata could be used to generate a human-readable document on the web anyone could read.

The controls for creating and editing the metadata are crucial. There would be a CRUD (Create Read Update Delete) matrix between each permission group and each metadata block instance, for example a state. A person who belonged to the permission group for a particular state with C permission would be able to enter and edit candidates and issues to vote for. Since this is done only once per election and the data is so small, it’s likely that such high-level permissions would be restricted to a couple of centralized people with security similar to that for launching attack rockets. Local security would be for creating voting locations and stations. Things like whether signatures are required would be made at the appropriate controlling jurisdiction level. In any case all changes would be made in collaboration with a central group including verbal interaction with multiple people to prevent hacking of any kind.

In all cases setting and changing parameters is highly infrequent but dangerous, which is why gaining access is made burdensome and the results fully public. Changes would be halted at an agreed time prior to an election and before early voting if any.

Because all control parameters and their settings are handled in this way with public viewing of the settings, there is no need to do any software administration and update for any reason, which makes it possible for the software source code itself to be made available for public inspection.

Building the System

The system described here can be built quickly and inexpensively if the appropriate skills and entrepreneurial methods are used. Disaster, delays and ballooning expense would result from a typical corporate/governmental RFP process with endless meetings, reviews and input from “experts.”

Separate teams of a couple people each with appropriate skills could write each of the components/services described here. The toughest skills to get are the currently rare knowledge of bare-machine programming; therefore a preliminary step of running the software on a standard operating system could be used to get a working prototype. There are a few infrastructure things that would need to be agreed to, for example the exact methods for making calls among the mesh of services and otherwise coordinating their activities. It would be best if common tools like redis were used for reliable, super-fast queuing were agreed to and used when appropriate. The metadata control system would need to be built by a single team, but there would not be a much code involved. Its API would be accessed by all software services, probably just once at the start of operation.

The system could first be deployed in small scale with purely local elections for things like home-school associations. Cooperating government entities could make boxes of ballots from past elections available to try out the software.

Conclusion

One of the key benefits of the modern method of voting I’ve described is that it eliminates nearly all of the human setup and administration that is currently performed by thousands of people at vote-processing locations. It also eliminates the many thousands of error-prone human steps that are required to process votes, including things like USB drives that are currently moved by administrators from voting machines to web-connected laptop computers.

While there are lots of details I haven’t filled in, nothing I’ve described here should be foreign to software people on the front lines. Systems much like it are in production at scale in multiple industries. The wide-scale logging, parallel processing and comparison of results are standard methods for assuring that a fault of any kind, malicious or just random, doesn’t cause problems. While everyone, including me, has concerns about hacking, it’s well-known that the worst hacks are typically inside jobs, and taking people out of most of the processing goes a long way to increasing security. The chances of success would be greatly increased by making the software be a kind of open source so that anyone can point to vulnerabilities. For example, open-source Linux software runs something like 90% of the world’s web servers; it’s the gold standard for open and auditable while also being more secure than anything produced by a closed software group.

If a system like this were in use, everyone would be able to be confident that insiders of any variety weren’t using their power over the process to skew the results; except for the identity of the people voting, every step of every vote would be open to inspection by anyone in near-real-time.

Everyone should be able to support bringing voting up to modern standards.

Software Programming Language Evolution: Beyond 3GL’s

In prior posts I’ve given an overview of the advances in programming languages, described in detail the major advances and defined just what is meant by “high” in the phrase high-level language. In this post I’ll dive into the amazing advances made in expanding programmer productivity beyond the basic 3-GL’s. What's most interesting about these advances is that they were huge, market-proven advances, and have subsequently been ignored and abandoned by academia and industry in favor of a productivity-killing combination of tools and technologies.

From the Beginning to 3-GL's

The evolution of programming languages has been different from most kinds of technical evolution. In most tech development, there’s an isolated advance. The advance is then copied by others, sometimes with variations and additions. There follows a growing number of efforts concentrating on some combination of commercialization, enhancement and variation. This resembles biological evolution in that once mammals were “invented” there followed a growing number of varied mammalian species with ever-growing variations and enhancements.

If you glance at the evolution of programming languages, it can easily seem as though the same kind of evolution has taken place. It makes sense: software languages are for computers, and don’t computers get faster, smaller and cheaper at an astounding rate?

Let’s start by reviewing the evolution of programming languages up to what are commonly called 3-GL’s. For details see this.

First generation languages are all the native language of a particular computer, expressed as the computer executes the language: in binary. A program in a 1-GL, normally called machine language, is a big block of 1’s and 0’s. If you understand it, you can break the numbers up into data and instructions, and the instructions into command codes and arguments. Necessary for a machine, but a nightmare for humans.

A program in a 2-GL, normally called assembler language, is a text version of a 1-GL program, with nice additions like labels for locations of instructions and data. 2-GL’s were a night-and-day advance over machine language.

A program in a 3-GL, for example COBOL, FORTRAN or C, is a text language that is independent of the computer that can be translated (compiled or interpreted) to run on any machine. There are statements for defining data and for defining the actions that will be taken on the data. The action statements normally constitute the vast majority of the lines. For many programs, 3-GL’s were 5 to 10 times more productive than assembler language, with the added advantage that they could run on any machine.

We’re done, right? In some sense, we are – there are still vast bodies of production code written in those languages. No later language can create a program that has greater execution speed. But maybe we’re not done. As I’ve described, the “high” of high-level isn’t about the efficiency of the computer; it’s about the efficiency of the human – the time and effort it takes a human program to write a given program using the language. There have been a host of languages invented since the early days of 3-GL’s that claim to do this.

Let’s look at a couple of languages that no one talks about and don’t have a category name, were wildly popular in their day and that live on today, unheralded and ignored. I’ll use two examples.

The Way-beyond-3-GL's: MUMPS

The first of these languages I’ll describe is MUMPS, developed at Mass General Hospital for medical  processing. Have you ever heard of it? I didn’t think so.

In modern terms, MUMPS is definitely a programming language; it has all the capability and statement types that 3-GL’s have. But MUMPS goes way beyond the boundaries of all 3-GL’s to encompass the entire environment needed for building and running a program. Normally with a 3-GL someone needs to pay lots of attention to things “outside” the language to achieve an effective solution, particularly in the areas of data access, storage and manipulation, but also in the operating system. A MUMPS program is inherently multi-user and multi-tasking. It has the ability to reference data without the potential danger of pointers. It has the power and flexibility of modern DBMS technology built in – not just relational DBMS but also key-value stores and array manipulation features that are still missing from most subsequent languages. In other words, you can build a comprehensive software application in a single programming environment without external things like databases, etc.

The result of this wide variety of powerful features all available in one place implemented as an integral part of the language was definitely outside the mainstream of programming languages – but wildly productive. MUMPS had strong uptake in the medical community. For example, the hospital software with dominating market share today is Epic, which was originally written in MUMPS (now called Cache) and remains so today.  An amazing number of other leading medical systems are written in the language, as well as in the financial sector.

Net-net: MUMPS is truly a beyond-3-GL high level language in that the total amount of human effort required to reach a given programming result is much less. Even better, all the skills are normally in a single person, while modern languages require outside skills to achieve a given result, for example a database expert.

The Way-beyond-3-GL's: PICK

PICK is another beyond-3-GL that delivered a huge up-tick in programmer productivity. PICK, like MUMPS, is largely forgotten today. It’s an afterthought in any discussion of programming language history, ignored by academics, and generally erased. The title of its entry in Wikipedia is even wrong – it’s called an operating system! Of course, it is an operating system – AND a database AND a dictionary AND a full-featured programming language AND a query system AND a way to manage users and peripherals -- everything you need to build and deliver working software, all in one place. PICK was a key driving factor in fueling the minicomputer industry during its explosive growth in the 1970’s and 80’s, while also running on mainframes and PC’s.

Wikipedia says: "By the early 1980s observers saw the Pick operating system as a strong competitor to Unix.^[13] BYTE in 1984 stated that "Pick is simple and powerful, and it seems to be efficient and reliable, too ... because it works well as a multiuser system, it's probably the most cost-effective way to use an XT."

PICK was the brainchild of a modest guy named Dick Pick. During the early 1980’s I worked at a small company in the Boston area that attempted to build a competitor to PICK, which seemed to be everywhere at the time. As you might imagine, programmer humor emerged on the subject, including such gems as

If Dick Pick picked a pickle, which pickle would Dick Pick pick?

PICK lived on in many guises and with multiple names. But it has zero mind-share in Computer Science and among most people building new applications today.

Conclusion

All-encompassing programming environments like MUMPS and PICK should have become the dominating successors to the 3-GL languages, particularly as the total effort to develop working systems based on 3-GL’s like Java exploded with the arrival of independent DBMS’s, multi-tier hardware environments and the orthodoxy of achieving scalability via distributed applications. Yet another step on the peculiar path of software evolution.

I remember the frenzy during the internet explosion of the late 1990’s and early 2000’s of money flowing in and the universal view among investors and entrepreneurs about how applications must be written in order to be successful. I encountered this personally when my VC partner introduced me to what appeared to be a promising young medical practice management system company that was having some trouble raising money because investors were concerned that the young programmer doing much of the work and leading the effort wasn’t using java. I interviewed the fellow, Ed Park, and quickly determined that he was guided in his technical decision-making by smart, independent thinking rather than the fashionable orthodoxy. I endorsed investing. The company was Athena Health, which grew to become a public company with a major market share in its field. And BTW achieved linear scalability while avoiding all the productivity-killing methods everyone at the time insisted were needed.

The history of amazing, beyond-3-GL's like MUMPS and PICK that deliver massive programmer productivity gains demonstrates beyond all doubt that software and all its experts are driven by fashion trends instead of objective results, and that Computer Science is a science in name only.

Software Evolution: Functionality on a New Platform: Market Research

This is the third in a series of examples to illustrate the way that functionality that had been implemented on an older platform appears on a newer platform.

See this post for a general introduction with example and explanation of this peculiar pattern of software evolution. This earlier post contains an example in security services software and this earlier post describes an example in remote access software.

This example is known to me personally because my VC firm was an investor, and I was involved with them through the life of the investment. 

Example: Knowledge Networks

Old platform	Telephone, mail, focus groups
Old function	Conducting surveys for market and opinion research
New platform	Internet
New function	Essentially the same, with much greater knowledge of the activities of panel members before and after taking a survey, and the ability to conduct interactive surveys
Outcome	The premise was valid, but the company was ahead of the market. It was acquired in 2011.

Organizations that depend a great deal on the opinions and actions of a large number of people sometimes conduct market research to help them shape the details of a product, an advertising campaign, a political campaign or other relevant effort. This kind of research long pre-dates computers. It normally starts using informal methods for selecting the people to ask and evaluating the results. But then it moves in stages towards increasing amounts of scientific control and analysis in order to reduce costs and improve accuracy.

Market research was already well-established on the prior technology “platform” of the telephone when the internet started to spread quickly in the second half of the 1990’s, when a substantial and growing fraction of the US population got internet access. People in the field were familiar with the issues of selection bias, people without telephones and random digit dialing methods of assuring statistically valid panels. But when the web (the new platform) started spreading quickly, did market research transfer its knowledge, methods and techniques to the new platform? It didn’t (I think you probably guessed the answer), because brand-new people put together the first web-based market research systems. It was quick, easy and had the advantage of being inexpensive – but it was as scientifically primitive as telephone-based surveys were prior to the introduction of statistical methods.

Knowledge Networks was started by a couple of professors with stature in market research in the pre-internet world, with the goal of keeping the cost and speed advantages of the internet, but bringing it up to pre-internet scientific standards.

While there has definitely been a migration of internet-based market research to higher levels of scientific standards, which Knowledge Networks has both led and benefited from, their experience is an example of the danger of getting too far ahead of the pattern. One of the key facts about the “emergence of functionality on a new platform” pattern is that the functionality emerges in the same order as it did on the earlier platforms – but it doesn’t skip steps or leap right to the end! These professors knew that internet-based market research would evolve to greater scientific integrity – and they were right! – but they didn’t fully appreciate that the market would get there in its own sweet time, and that it would insist on dawdling on intermediate steps. By insisting that Knowledge Networks provide only the best, highest-integrity market research methods, up to the standards of the best available on earlier technology platforms, the original leaders of the company caused it to be “out of step” with the market. They were “ahead” of the market, which is a great place to be if you want to be a business “visionary,” but is rarely a good place to be if your goal is to build a substantial business.

I have to say that this is a really hard one to get right in practical situations. I personally was involved with Knowledge Networks at the time some crucial decisions were made, but I didn’t know enough about or appreciate the power of the pattern to help make the company as successful as it could have been. In fact, I was probably part of the problem. The professors who started the company were really smart, and they were on top of all the issues of market research. I knew this, appreciated it, and was excited by the possibilities of benefiting by translating the best practices from traditional market research to the internet. What’s painful is that I also knew in general terms the dangers of being ahead of a market. But that’s exactly what we were, and yet again, for the umpteenth time, I didn’t see it and didn’t call it. Arghhh!

Lesson: being too early is just as bad as being too late.

Software Evolution: Functionality on a New Platform: Remote Access

This is the second in a series of examples to illustrate the way that functionality that had been implemented on an older platform appears on a newer platform.

See this post for a general introduction with example and explanation of this peculiar pattern of software evolution. This earlier post contains an example in security services software.

This example is known to me personally because my VC firm was an investor, and I was involved with them through the life of the investment.

Example: Aventail

Old platform	Dedicated IP-SEC VPN
Old function	Remote access to internal LAN resources
New platform	Web server
New function	Use existing Web infrastructure and https to provide old functionality, enhanced by application-level security, reducing costs and increasing flexibility and security.
Outcome	Some market education required in the early years, but strong position vis-à-vis the competition and good growth. The company was acquired by SonicWall in 2007.

Aventail built functionality for remote access that has been implemented over and over again, each time a new technology platform has emerged. But they rode what was at the time the latest wave (internet protocols and SSL encryption), and so were participating in a growing market.

I remember using teletype paper terminals running at 110 baud in the late 1960’s for remote access to computers. Whenever a new platform would come out, the new technology wouldn’t support remote access, but for some strange reason, people would want it! So, focused entirely on getting something working in the new environment, and either ignoring or simply being ignorant of earlier solutions to the same problem, someone would build a remote access solution. But then inadequacies would be found, and a release two would come out. All in what appears to be ignorance of solutions built on prior platforms, blind to their lessons-learned..

A good example is the identification and access control system for remote access. The system you want to connect to has some system for user ID’s and passwords, and then some method of access control based on user groups. The remote access is normally first built in the simplest possible way, having its own system administration, user identification and access control. As the use of the system grows, this parallel administration is a burden, and so some level of integration with the core security system is then implemented. The pattern is that the separate system is normally built first; the need for integration is “discovered;” the integrated control systems are supplied in a later release.

When you see this pattern of stupidity and ignorance for the first time, you scratch your head. These are programmers and experienced product people! How could they have missed such an obviously valuable feature of the same functionality built on an earlier platform? Well, that's the pattern, as I described in detail in the first post in this series. It's a wonderful pattern -- it enables anyone who understands it to predict the future with great accuracy and precision!

Software Evolution: Functionality on a New Platform: Security Services

A whole book could be devoted to spelling out the “natural” emergence of features on a platform, and identifying and accounting for the minor variations from platform to platform (the emergence sequences don’t repeat exactly for a variety of reasons). However, the similarities are obvious and universal enough that anyone with longitudinal familiarity with a couple of comparable platforms would recognize them.

This is the first in a series of examples to illustrate the way that functionality that had been implemented on an older platform appears on a newer platform. All the examples illustrate the point that, even though the functionality is there for anyone to see on the older platform, working and delivering business value, it only appears at its “proper time” on the newer platform.

See this post for a general introduction with example and explanation of this peculiar pattern of software evolution.

I have mostly selected companies that most readers may not be familiar with, to demonstrate the strength of the pattern and the way it affects all participants in a market, not just the well-known, mainstream companies. Also, I know them personally, and so can tell their stories from personal knowledge, instead of just repeating things I’ve read.

Example: Securant

Old platform	IBM mainframe
Old function	Security services: ACF2, RACF
New platform	Web server
New function	Security services for Web applications were extremely basic, and focused on message encryption. Securant implemented mainframe-class user and application security. The same kind of companies that depended on mainframes for transaction processing were very attracted to Securant’s approach to Web applications.
Outcome	Terrific product and sales traction despite product inadequacies and the company’s internal problems. The company was acquired for a good price by RSA Security in 2001.

Securant was started by a couple of young programmers in their 20’s as a services business. They were hired by a large financial institution that was adding internet/web infrastructure to their IT infrastructure. Naturally, the web applications needed access to the programs and data on the mainframe systems, and those systems were protected by state of the art security systems. The financial company would have preferred to be protected by a security facility for the web just like it did for the mainframe, but none was available. So they agreed with these guys to build one.

The young programmers knew nothing about mainframes, and never bothered learning anything much about them. As far as they were concerned, mainframes were pretty useless things well on the way to becoming obsolete – why become an expert in steam engines in the 1930’s when internal combustion engines are obviously the future? So they focused completely and solely on what they knew, and built a ground-up application that met the business security needs as they understood them from the users, who were also barely familiar with what the mainframe had to offer.

Before long, the one-off services project became a very hot, rapidly growing product company. One of the many ironies in this project is that the father of one of the two founding programmers had run one of the mainframe security companies! But he had been estranged from his son for many years, and neither knew that the other was or had been involved in computer security. The father decided to come back from the retirement that had been funded by the sale of his security company, re-connected with his son, and you can imagine his amazement when he found out what that son had accomplished. The father saw the functionality he had on the mainframe, re-imagined and re-implemented for the new environment.

One of the powerful things about this pattern is that it’s like the tide or the current in a river – at the right point, it just pushes the vendors and functionality in the direction of the “flow,” and everything ends up moving in the same direction. The people involved tend to think they’re inventing things – but what they invent is pretty predictable, because they’re responding to the same kind of needs that the previous bunch of inventors were responding to.

Software Evolution: Functionality on a New Platform

When a new “platform” emerges (UNIX, Windows, Web, Apps), if you look at any application area and see how it evolved on prior platforms, the application’s functionality will emerge on the new platform in roughly the same order, though often on a compressed timescale. The functionality that is relevant depends on the particular application area. This concept applies both to system and application software.

The pattern is: functionality emerges on a new platform in roughly the same order as it emerged on earlier platforms. The timescale of the emergence may be compressed; the important aspect of the pattern isn’t the timing but the order. The pattern means that functional steps are rarely skipped – what was next last time is also next this time. The pattern also means that when someone tries to introduce functionality too soon, before the functionality that preceded it on prior platforms is generally available, the market will not accept it.

While this pattern takes a good deal of knowledge and judgment to notice and apply, I have consistently been impressed by its predictive power. By following the pattern, you can be pretty confident that you’re building proven functionality, and that you’re following a pattern of success.

I have noticed a couple of danger points here. When the company is too aware of the pattern, it is easy for them to “get ahead of themselves” and more importantly ahead of the market, by solving problems that certainly will become important, but problems that the market doesn’t know it has yet. The “same order” part of the pattern is important; building ahead of the market appears to win few business benefits.

On the other hand, without knowledge of the pattern, it is easy to make up all sorts of things you think people might want, and build them, only to find out later that you wasted time and money, because the functionality you built is never part of the accepted function set.

Really great companies who have lots of creative people, the ability to execute, and listen closely to market reactions to what they’re doing, don’t “need” to know about this pattern. However, for the rest of us mortals down here, having a “cheat sheet” to what important features the market will be demanding next can prove awfully helpful.

Basic Example: operating systems

IBM famously created an operating system for their mainframe line of computers, OS/360. It had the capability of running multiple jobs at once. Its multi-tasking abilities grew and became more sophisticated through the 1970’s.

Eventually a transaction monitor, CICS, was written and became a de facto part of the operating system for applications with lots of interactive users. As the operating system was used, it became evident that various access methods for storage and communications needed to be separate from the core, and so clear interfaces were created, and the notion of completely self-contained access methods (for example, a file system) as replaceable units was supported. A strong security system was not part of the early versions of the operating system, and the need for one became critical, and so strong external modules were written and support for security was added to the core. While there was a “main” operating system, alternative operating systems were written for various purposes, and a virtual operating system was written to actually run on the “bare metal.” With VM (the virtual OS), you could devote most of a machine’s resources to the production users, while letting some of the users running on a completely different operating system.

While all this was taking place, people were studying and experimenting in university environments, deciding just what an operating system was and was not, what the best ways to build one were, and so on.

Before very long, mini-computers were invented; these were basically mainframes on the cheap, with all sorts of features and functions missing – but they were cheap. And, since each had a unique instruction set, each minicomputer needed an operating system. Programmers were hired, and those programmers, of course, ignored the mainframe operating systems, and built simple, cheap OS’s to go along with the cheap hardware. Surprise, surprise, those cheap OS’s resembled nothing as much as – the first generation of mainframe operating systems! But people quickly discovered the limitations, just as they had before, and set about making the same set of enhancements that the previous generation of pioneering programmers had made. Within ten years, they had re-invented many of the important mainframe OS concepts, and were on the way to building the rest.

With all this knowledge of operating systems floating around and pretty easily available, what do you suppose happened when people took the early micro-processor chips and made them into micro-computers? Naturally, they understood the state of the art of operating systems theory and practice and adapted an existing OS (which were starting to be built in high level languages) or built one that took all this knowledge and applied it with appropriate adjustments to the new environment, right? Bzzzzt! Of course not!

What the kids who were faced with the task did was start from scratch, not only in terms of code, but also in terms of knowledge. They didn’t stand on the shoulders of giants; they didn’t learn from the experiences of the many that preceded them; they built OS’s as though they were the first programmers who ever tried to do such a thing. And the result was pretty much like early mainframe (even pre-360!) operating systems. There was no serious memory protection or address mapping; there was no real concept of multiple users, multiple levels and types of users, or any real security; no multi-tasking; the access methods were hard-wired in, and so on. The limitations and problems emerged pretty quickly, and so did add-on rubber band and baling wire patches, just like in earlier days.

It’s a good thing that IBM came along at this point, and brought commercial order and education to the emerging microcomputer market. When they came out with the IBM PC, they not only legitimized the market, they had deep history with mainframes and minicomputers. They employed true experts who knew operating systems inside and out. They had a research division, where there were people who could tell you what the operating systems of the future would look like. So it makes sense they would get those experts together, and they would create a small but efficient micro-tasking kernel, common interfaces for installable access methods, and many other appropriate variations on all the modern operating systems concepts. The last thing such a smart, educated and astute major company like IBM would do was make an exclusive deal with a small company that had never built an operating system, who had just bought the rights on the cheap to a half-baked excuse for a primitive first-generation OS, and make that the IBM-blessed … Wait! … that’s what they did do! Arrrgggghhh!

Explanation

One might well ask, how can a pattern like this continue to have predictive power? Why wouldn’t the people who develop on a new platform simply take a little time to examine the relevant applications on the older platforms, and leap to the state of the art? Why wouldn’t customers demand it?

It is hard to know for sure, but I think there are a couple main factors at work, and there is evidence for the relevance of each of the factors.

The first factor is the developers. It is well known that most developers learn a platform and then stick with the platform they’ve learned for an extended period of time, basically as long as they can. The reason is simple: they are experts on the platform they already know, and therefore have prestige and make more money than they would as novices on a platform they’re just learning. I speculate that this is one of the many contributing factors to the rapid migration of ambitious programmers into management, where they can advance without being tied to a platform, at least as much. So who learns the new platforms? With few exceptions, new people. If you’re just entering the industry, you are counseled to learn the hot new languages; you tend to be sensitive to where the demands and rising salaries are. Still, you expect and are paid entry-level wages, along with most other people (except managers). Why should the experienced programmers jump to the new platform? They would have to compete with hard-working young people, their knowledge of the older platform will be considered a liability, and on top of everything else, they’d have to take a pay cut.

The result is that no one working on the new platform has an in-depth, working knowledge of the applications on the older platform, and at least in part because of this, everyone considers knowledge and use of the platform to be vastly more important than knowledge of an old application on an obsolete platform. So they ignore it! As a result, they dive in and attempt to automate the application area “from scratch.” Their attempts are usually quite close to every first generation program for that application on past platforms, because it turns out that the determining factor isn’t the platform, it’s the business problem. They proceed to re-discover, step by step, the reasons why the first generation was inadequate and had to be supplanted by a second generation, etc.

The second factor is the buyers. When a new platform emerges, most buyers simply ignore it. Why pay attention? It’s a toy, there are no good applications, etc. The few buyers who do pay attention tend to be crazy, early adopter types who just love the experience of trying new things. Like the programmers, they also tend to care about the platform more than the application – otherwise, they wouldn’t even consider buying what is typically a seriously immature application on the new platform. But they can only buy what’s being sold, and so they choose among the inadequate applications. Because they don’t care about applications as much as platforms, they don’t even ask for features they know could only be present in the mature applications for older platforms – they press the applications vendors for the “next” obvious cool feature, in the narrow universe of the new platform.

The reason why application evolution repeats itself on the new platform, then, is that nearly everyone involved, builder and buyer, is ignorant of the past and wears what amounts to blinders. It’s as though they are building and buying the application for the first time. Therefore, they respond most strongly to the same business pressures that people involved in computer automation tend to see first, and then next, and so on. It’s as though there’s a “natural” most climb-able path up a mountain, and successive waves of climbers approach the climb from the foot of the mountain, completely ignorant of the experiences of those who came before them, but confronted with the same options they tend to make the same choices, and so everyone takes roughly the same route up the mountain.

Why don’t smart groups who know all this leap-frog their way to the most advanced application functionality? I have seen this happen, and it’s the buyers who tend to rain on this kind of parade. The buyers tend to be familiar with the range of applications in a category, and those applications tend to address a highly overlapping set of problems in ways that vary only slightly. The “far-seeing” builder then comes along with all sorts of solutions to problems that the buyers don’t even know they have! The buyers look around in confusion – why isn’t anyone else talking about this? I kind of understand what you’re talking about, but I feel silly thinking that something’s important when no one else does. I think I’ll wait. And they do. So getting too far ahead of the buyers is just as much of a problem as being too far behind the competition. The result: the application evolution repeats itself on the new platform, in roughly the same order each time, and no “cheating” or “running ahead” is allowed.

This all sounds incredibly common-sense when I read it written down, but I have to admit that this particular piece of common sense is not only uncommon, it took me personally decades and multiple failures to finally get this simple thought into my thick skull. The key thought is this one: you may think you know a person has a problem; the problem might be a severe one, and cost the person a great deal of money and trouble; you may even be entirely right in this judgment. However, if the person in question does not think he has a problem, why should he pay for a solution – in fact, why would he go to the trouble of implementing a solution even if it were free?  Even worse, why would he even waste time talking with you, once he got the idea that you thought he had a problem he didn’t think he had? Why would he listen to a chimney sweeper’s pitch if he lives in a house without chimneys? The air quality in Los Angeles in 1970 was terrible. But there was no market for catalytic converters on automobile exhaust systems at that time. The problem that converters solve existed, and was getting worse. It was obvious for anyone to see. It was even talked about. But in peoples’ minds at the time, having a car that contributed to air pollution was not a problem most people accepted they had (even though we know that, objectively speaking, they did have it).

What is high about a high level language in software?

I’ve talked in detail about the supposed progress in computer software languages, and explained the two major advances that have been made. All modern software languages are called “high-level languages.” As is typical in the non-science of Computer Science, no one bothers to define exactly what is meant by “high.” This is hilarious, since a single character being missing or out of place can cause a giant piece of software to crash – if there’s anything in this world in which precision is important, it’s in programming computers. But somehow the academic field and the vast majority of the practitioners don’t bother with little details like defining precisely what is “high” about a high-level language.

There was no such lack of precision by the person who invented the first widely used high-level language. John Backus first proposed FORTRAN to his superiors at IBM in late 1953.  A draft specification was completed in 1954 and the first working compiler was delivered in early 1957. Wikipedia has it clear, simple and correct:

While the community was skeptical that this new method could possibly outperform hand-coding, it reduced the number of programming statements necessary to operate a machine by a factor of 20, and quickly gained acceptance. John Backus said during a 1979 interview with Think, the IBM employee magazine, "Much of my work has come from being lazy. I didn't like writing programs, and so, when I was working on the IBM 701, writing programs for computing missile trajectories, I started work on a programming system to make it easier to write programs."

According to the creator of the first successful high level language (echoed by his buddies and collaborators), “high level” in the context of software languages means “takes fewer statements and less work” to write a given program. That’s it! My blog post on the giant advances in software programming languages explains and illustrates this in detail.

The whole point of FORTRAN was to make writing numeric calculation programs quicker and easier. It succeeded. But it didn’t enable ANY assembler language program to be written in it. As I explain here, the invention of COBOL filled the gaps in FORTRAN for business programming, and C filled the gaps for systems programming.

How did the HLL’s achieve their productivity?

The early HLL’s were centered on wonderful, time-saving statements like assignments and if-then-else that both saved time writing and increased readability. In addition, the early creators were thoroughly grounded in the fact that the whole point of software was … to read data, do some tests and calculations and produce more data, a.k.a. results. Each of the amazing new languages therefore included statements to define the data to be read and written, and other statements to perform the actions of reading and writing. COBOL, for example, had and has two major parts to each program:

• Data Division, in which all the data to be used by the program is defined. When data is used by multiple programs, the definitions are typically stored in one or more separate files and copied by reference into the Data Division of any program that uses them. These are usually called “copy books.”
• Procedure Division, in which all the action statements of the program are defined. These include Read and Write statements, each of which includes a reference to the data definitions to be read or written.

This way of thinking about things was obvious to the early programmers. They had data; they had to make some calculations based on it and make new data. Job one was define the data. Job two was, with reference to the data defined, perform tests and calculations. For example, reading in deposits and withdrawals, and updating current balances.

After the Great Start...

Of course, things were not sweetness and light from thence forth. Huge amounts of noise and all the attention of software language people were generated by minor variations on the basic theme of high level languages. No one EVER argued or measured how many fewer statements or reduction of work it took. I guess a little birdie deep inside most of the participants would chirp “don’t go there” whenever one of the language fashionistas was tempted to actually measure what difference their new language resulted in.

I’m not going into the claims of virtue made during the last 50 years of pointless language invention work in detail, any more than I would go into the contents of garbage trucks to count and measure the differences in what people throw out. In the end, it’s all garbage. I’ll just mention these:

There’s a large group of inventors who claim that use of their new wonder-language will prevent programmers from making as many errors. This is high on the list of people who make programs HARDER to write by eliminating pointers. Any studies or measurements? Anyone notice even anecdotally a big uplift in software quality? Anyone?

Advocates of the O-O cult like to talk about how objects keep things private and prevent bugs. They suppress all talk of the hoops that programmers have to jump through to make programs conform to O-O dogma, with the resulting increase of lines of code, effort and bugs.

Conclusion

The starting years of computer software language invention were incredibly productive. The original high level languages achieved the vast majority of the "height," i.e., reduction in lines of code and effort to write a given program, that could possibly be achieved. The subsequent history of language invention includes a couple minor improvements and filling a small but important gap in coverage (in systems software). But mostly it's a story of wandering around in the wilderness spiced up by things being made worse, as I'll show in subsequent posts.

If You Care About Good Software You Should Care About Documentation

I recently posted what I thought would be my least-read post ever. It's about the subject that has by far the greatest spread by far between "something tech people say is important" and "something tech people avoid thinking about or doing."

There are many reasons why this is the case. High on the list is the fact that documentation occupies "below zero" status on the list of things programmers and their managers actually care about. I'm also guilty of this. Look at this detailed post I wrote about the hierarchy of status in programming and you'll see that documentation is nowhere mentioned!

Here's a metaphor to help you understand the importance of documentation. Suppose an important election is on the horizon and for various reasons an unprecedented number of people are likely to vote by mail instead on in person. Suppose that there's a large region in which USPS mailboxes have been installed in scattered places. Suppose that a security failure has been discovered in the design of the mailboxes that makes it quick and easy for a trouble-maker to open it and remove all the mail, including of course any ballots. Word of this vulnerability seems likely to leak out, tempting activists to raid mailboxes where ballots  cast for the party they want to lose are likely to be found. It's a bug! It has to be fixed immediately!

In the real world, there are probably lists of the locations of all such public mailboxes. Even if there aren't, there are USPS employees who visit the boxes regularly and know where they are. Failing everything else, the public can be asked to register the location of any boxes they know about. No problem.

In the wonderful world of software, things are entirely different. The mailboxes and everything around them are invisible to normal people. Members of the public don't "go there." USPS employees don't go there. The original contractor who installed each box at various times may have been required to provide documentation of his work, but the original documentation was incomplete and full of errors, and was never updated for subsequent additions and changes. What's worse, the boxes are darned hard to find. Practically no one has the right kind of eyes and training to even be able to correctly recognize a box when driving slowly down a road looking for them.

If you're a modern, with-it programmer you might be thinking to yourself at this point "hah! That wouldn't happen with my code -- I use modern micro-services, so I'd just have to go to the right service." Uh huh. What if the bug had to do with some data that was defined incorrectly? Do you have a DBMS? Do you have a UI? Does every piece of data really appear and get used exactly once in exactly once place? Is it really so easy to not only find each instance of the data but all the downstream uses and consequences of the data that are impacted by the error?

Regardless of how your code is organized, finding and fixing a bug can take a depressing amount of time, and part of the time is often the result of not having comprehensive, accurate low-level documentation.

In the relatively simple and visible-to-everyone real world of paper ballots and mailboxes, we know there are errors and faults, some of them extensive. Bad things can and do still happen. Because of the simplicity and visibility of the normal world the faults are often noticed quickly, like when bad ballots are sent. In the incredibly complex and visible-to-few world of software, the faults can go for long periods without even being noticed, and when they finally surface it can take the tiny number of super-specialists with the right training, vision and persistence a long time to find and fix the bugs lurking in the vast spaces of the largely invisible, undocumented oceans of software we all depend on.

Documentation is unlikely to improve in the world of software because practically no one, despite the sounds that may come from their mouths, really cares. The good news is that better approaches to building software than are fashionable today go a long way to minimizing the trouble. The more we move towards Occamality, the more we'll get there. The reason is simple: if everything in a program is in exactly one place instead of scattered redundantly all over the place, at least you'll know that there's just one fierce monster bug out there and not an army of clones.

The Universally Ignored Disaster of Software Documentation

Nearly everyone knows what “click bait” is – an article title that strongly tempts readers to click and read the article. Part of being a good writer/editor these days is developing a facility for click bait.

My proposal for the most click unbaitable or click repulsion title is anything that includes the words “software documentation.” Is there a more boring subject on the planet? Even to software people? Or I should say especially to software people?

It’s too bad, because software documentation is a genuinely important subject. Every ounce of effort put into it is not just an ounce of wasted effort, but many ounces of waste by side effect. Software documentation is the subject of near-universal complaints, but few dispute its irreplaceable importance in the cycle of “responsible” software development, from requirements all the way through testing and support. We should be talking about software documentation – and documenting (heh…) its odious nature.

Software Documentation

Documentation is the backbone – or ball-and-chain – of software from start to finish. The time, effort and resulting bulk of documentation normally vastly exceeds that of the software itself. The documentation is supposed to define both generally and precisely WHAT is to be built, HOW it is to be built and tested, and then what was actually built.

If you are in a normal corporate environment, this all ends up being HUGE. If you’re in a regulated environment, it’s so big it’s amazing anything gets done. Here’s a little snippet of just some of the documentation required for medical devices by the FDA.

As you can see, you even need a document that has the plan for documents! See this for more detail and explanation.

Documenting the problems with documentation is an immense job, and given that no one cares about documentation one way or the other, I’m not going to do it. But here are a few of the lowlights.

• Requirements documentation is supposed to be the foundation of a software project, defining in MBA-understandable terms the what, why and how of the software effort to be undertaken.
• In reality, most requirements documents fall somewhere in the range between useless and ignored.
• Requirements documents are kind of like the plan to discover a new sea route to Asia in the Europe of 1490, which was basically using a lot of words to say “sail west.”
• Requirements documents are never updated to reflect what actually happened, beyond brief memos saying triumphant things like “we sailed the ocean blue and hit land in 1492!”
• Architecture documents are typically produced by people who won’t end up writing the code.
• Architecture documents resemble someone on a roof top in Kathmandu pointing mostly east and a bit north, waving and jumping around, saying excitedly with lots of words – go there and keep climbing up and up and up, and you’ll get to the peak of Mount Everest!
• Often the climb-Everest plan contains huge amounts of gruesome detail -- all written by someone who's never been to base camp, much less climbed.  
  
• The people who write the code may or may not pay lip service to the document. They certainly won’t waste time updating it with what actually happened.
• More detailed design documents may be written by people who sit in the same general area as the people who write the code.
• In the end you have to make the code sort of work, and there are time pressures. Never has a manager ever said “and I want the updated version of the design document BEFORE you release the working code.”
• Everyone cares about quality, of course. So there are test documents of all kinds, even more in a test-driven-development environment.
• Is automated testing ever actually comprehensive in the real world? Has any manager insisted that updated test plans be created before the automated test is revised? I mean, of course, a manager who keeps his job?
• Code itself needs to be documented. How are programmers who need to dive in and fix things and/or make changes supposed to figure things out without it?
• Every programmer complains about the lack of relevant, accurate documentation when they dive in to a new code base to get something done. That same complaining programmer, of course, takes extra time at the end to carefully document what he’s done and the things he learned along the way. Sure.
• One of the reasons why having code in multiple layers and scattered among loads of little microservices is so attractive to modern-thinking people who think of themselves as Computer Scientists (or at least Engineers) is that when you’re looking for something about the handling of the data you’re trying to change, the code (and related documentation – hah!) is in lots of different places with no good way to figure out what’s where – or even what language it’s in! I bet you never knew the perversity behind this modern thinking, did you?
  

Responses to the Documentation Disaster

It’s been a long time since I’ve seen a new response to the ongoing disaster that is documentation. Here are some of the typical responses.

• Real programmers like to complain about documentation. Regularly.
• They complain when documentation they think should be available to them doesn’t exist.
• When it exists, they complain about how much time they wasted in a failed attempt to get anything useful from it.
• When someone tells them they have to write documentation, they have a list of compelling reasons why higher priorities won’t let them.
• When they lose and have to write, they complain about what a waste of time it is, and how useless their work will be after the next round of major changes to the software.
• Ambitious programmers learn to promote documentation
• Most important promotion decisions are made by people who are completely clueless about software
• One of the traits most highly valued by these ignorant managers is the ability to “communicate.”
• Communication with the clueless normally means having reports, project plans and other documents with enough plain language to seem relevant with lots of acronyms to convey credibility.
• Ambitious programmers emphasize adherence to standards to convey the appearance of being risk-averse; standards tend to include piles of documentation
• Since it’s all a charade in the end, ambitious programmers learn to take short cuts and make it clear to underlings that document content quality, accuracy and completeness are weights that can be shed in the race to completion.
• Has any top manager ever done more than glance at the thick pile of project documentation, assured themselves that it was appropriately thick, and nodded "good?"
• Managers nearly always have documentation on every project plan – otherwise they’d look like they didn’t know what they were doing.
• Experienced managers know it’s a sham, and that it’s documentation that will be squeezed out when time is running short.
• Managers working on projects that fall under government regulation know they might have to pass audits, and so make sure to produce voluminous documents with all the right titles, headings and sections.
• Since auditors don’t actually know anything about software, all they do is make sure the documents look like the sort that normally passes, and grade accordingly.

In the end, real programmers know that there is no such thing as good documentation. The only thing that matters is the code. Of course, “good” code is supposed to have comments embedded in it. Some programmers do this. But the second the code is changed...

Conclusion

 All this craziness stems from the fact that software is invisible. When things are visible, we're not documentation-crazed; for example, If you're having a house built, when the project was underway would you ask to see the updated plans or would you visit the job site? Along with project management and other standard aspects of software development, documentation is one of the reasons why software takes 10 to 100 times more effort to write than it could take. There is a simple solution. For better or worse, the solution is never taught, but is re-discovered on a regular basis by small groups who write great software quickly and well. Typically the groups who discover the solution have little time and less money, but are motivated and desperate to build a good solution quickly. So they avoid project management and don’t write more than a couple pages of what-are-we-doing type documentation at the start – and don’t write anything else. Really! NO documentation!

This makes perfect sense.  If you were running away from a crazed killer, would you stop to make sure your tie was properly tied?

For more constructive suggestions about how to build software better, see my books, particularly Software Business and Wartime Software.

Elizebeth Smith Friedman: The Cancelled Heroine of Cryptography

The unheralded Elizebeth Smith Friedman is a textbook example of the vast gulf that too often separates achievement in a field from getting credit for the achievement.

She was a true pioneer of cryptography and code-breaking, leading multiple efforts against the international criminal mob and the Axis in World War II. Unlike most people called “leaders,” she was actually the best at what she did, personally cracking “uncrackable” codes and personally pioneering new methods. She was a leader in the true sense: the manager/boss of the long distance runners AND the runner far in front of everyone else who gets there first AND helps all her fellow runners speed up.

Technical History Issues

Making an advance in technology is hard. Not many people try to do it, and a tiny fraction of those who try seem to succeed. Many of those apparent successes burn out – they weren’t advances after all. Sometimes a true advance, for various reasons, is never adopted. When it is adopted, there is often a race to claim credit for the advance. The race isn’t so much a race as it is a no-holds-barred war. To win the war, you usually need the support of loads of people who have no idea what the advance is about. These ignorant people create history, along with its winners and ignored achievers.

Is this cynical? Yes. Is it an accurate description of what happens? In all too many cases, sadly yes. Here are examples of from the war for credit for inventing the computer.

In most cases, technical invention isn’t like a giant comet streaking to earth and creating a big boom. It’s more like a sequence of parallel, overlapping efforts to solve a problem or make something better. Often an advance is made by more than one person or group without involvement with the other. What in retrospect is described as the big advance is usually a step forward, one of many, building on earlier work. Sometimes the advance isn’t an advance so much as a commercialization. Matt Ridley describes this with many examples in his mostly excellent book on Innovation. Elizebeth stands out as being a true innovator on multiple dimensions.

Elizebeth Smith Friedman

Getting to the truth about inventors and technology innovation is a problem in general. In the case of Ms. Friedman, the problem was made worse by the credit-taking actions of government leaders. The truth has only emerged recently with the release of previously concealed documents and the ending of secrecy periods.

Here are some highlights of her career:

In the 1930s, Elizebeth Smith Friedman became America’s and indeed the world’s best-known codebreaker. She inflicted severe damage on the interests of organized crime and at times needed to be protected by bodyguards. The evidence she gave in criminal trials describing how she cracked encrypted messages passing between mobsters made her a newspaper sensation.

Later, during World War 2, she broke coded messages sent on Germany’s Enigma machines. These messages revealed a plot by the Argentinian government to help Germany replace South American governments with Nazis, giving Germany bases from which to attack America. Her discoveries allowed the western allies to thwart the Argentinian and German plans.

Elizebeth Smith Friedman’s wartime codebreaking work was so secret that she was forbidden to mention it in public. She died many years before government archives were brought to light showing what she had done. During and after World War 2, J. Edgar Hoover and the FBI took the credit for work Elizebeth and her U.S. Coast Guard team had carried out.

Her whole story is fascinating. Among other things she is a wonderful example of the power of bureaucracies (education, government and corporate) to control and often suppress outstanding talent, and how sometimes, when the bureaucracy is desperate for results, it will break its own rules to achieve a goal – and then claim credit. It is particularly striking in Elizebeth’s case because she was NOT trained in math or STEM of any kind; her fascination was with literature and philosophy.

There is a book about her life which is includes previously classified and/or ignored documents about her career:

If you’re at all interested in people like Alan Turing and Grace Hopper and/or computing history, it’s worth reading. She was completely “unqualified” to do what she did – and she became the best at it, for example cracking the German Enigma machine without the huge staff and machines at Bletchley Park that have become famous.

Likewise, I had never heard of her husband William Friedman, who was also accomplished as a code-breaker both by himself and working with Elizebeth. Here are a couple links that give some highlights, though I still recommend the book.

The giant advances in software programming languages

There have been two gigantic advances in programming languages, each contributing huge advances in programmer productivity. Since then, thousands of times more attention has been given to essentially trivial changes than has been given to the two giant advances. I described these 50 years of non-advances here. In this post I’ll go back in history to describe the two giant advances that were made in the early days of computing, back in the 50’s, and I’ll tack on the one important corrective that’s been made since then.

From machine language to assembler

Every machine has one and only one “machine language.” This is a stream of binary data that the machine’s processor “understands” as instructions. Each instruction causes the machine to perform a single action on a single piece of data.

I learned to read and write machine language for a couple different machines. It’s not easy. From early on, machine language was usually shown as hexadecimal instead of raw binary, so that instead of looking at “0100101000101111” you see the hexadecimal equivalent: 4C2F.  A big plus but you’re looking at a hexadecimal pile of data. A nice editor will at least break it up into lines of code, but you still have to read the hex for the instruction, registers and storage locations

Moving from machine language to assembler language was HUGE. Night and day in terms of readability and programmer productivity. Assembler language is a body of text that closely corresponds to the machine language. Each line of text typically corresponds to exactly one machine language instruction. Assembler language transformed programming. If you understand any machine language, you can easily write and read the corresponding assembly language. For example, here's the binary of a simple instruction:

Here's the equivalent in hexadecimal:

And here it is in assembler language, including a comment:

Just as important as making instructions readable was using text labels for locations of data and addresses of other instructions. In machine language, an address is expressed as a “real” address, for example as a hexadecimal number.  If you inserted a command after a jump command and before its destination, you would have to change the address in the jump. You can see that with lots of  jumps this would quickly become a nightmare. By labeling program lines and letting the assembler generate the “real” addresses, the problem disappears.

From assembler to high-level languages

Assembler made writing code in the language of the machine incredibly easier. It was a huge advance. But people quickly noticed two big problems. The solution to both problems was the second giant advance in software programming, high-level languages.

The first problem was that writing in assembler language, while worlds easier than machine language, still required lots of busy-work. For example, adding B and C together, dividing by 2 and storing the result in A should be pretty easy, right? In nearly any high-level language it looks something like this:

                A = (B+C)/2

Putting aside messy details like data type, this would be the following in pseudo-assembler language:

                Load B into Register 1

                Load C into Register 2

                Add Register 2 to Register 1

                Load 2 into Register 2

                Divide Register 1 by Register 2

                Store Register 1 into A

Yes, you can quickly become proficient in reading and writing assembler, but wouldn’t the 1 line version be easier to write and understand than the 6 line version?

Expressions were the core advance of high level languages. Expressions turned multi-line statement blocks that were tedious and error-prone into simple, common-sense things. Even better, the use of expressions didn’t just help calculations – it helped many things. While assembler language has the equivalent of conditional branches, many such conditionals also involve expressions, for example, the following easy-to-read IF statement with an expression

                If ((B+C)/2 > D) THEN … ELSE …

Would turn into many lines of assembler – tedious to write, taking effort to read.

The second problem emerged as more kinds of computers were produced, each with its own machine and assembler language. A guy might spend months writing something in assembler and then want to share with a friend at another place using a different type of computer. But the friend’s computer was different – it had a different assembler/machine language! The whole program would have to be re-written. Total bummer!

Wouldn’t it be nice to write a program in some language that was like the 1 line version above, and that could be translated to run on any machine??

Easier to write by a good 5X, easy to read and can run on any machine, present or future. Hmmmm… There must be something wrong, this sounds too good to be true.

The concept of high level languages was good and wasn’t too good to be true. Everyone agreed, and it wasn’t long before FORTRAN and similar compute-oriented languages arose. Even performance-obsessed data nerds wrote programs in FORTRAN rather than assembler because FORTRAN compiled into machine language was just as fast, and sometimes even faster because of optimizations those clever compiler guys started inventing! And the programs could run anywhere that had a FORTRAN compiler!

The accounting and business people looked at what was happening over in heavy-compute land and grew more and more jealous. They tried FORTRAN but it just wasn’t suitable for handling records and transactions, not to mention having nice support for financial data. So business processing languages got invented, and COBOL emerged as the best of the bunch.

If you look at a compute statement in FORTRAN and COBOL, they’re nearly identical. But in the early days, FORTRAN had no support for the money data type and no good way to representing blocks of financial data that are core to anything involving accounting.

Once they got going, people just kept on inventing new languages for all sorts of reasons. The obvious deficiencies of FORTRAN and COBOL were fixed. Languages could handle intense calculations and business data processing about as well. But back around 1970, 50 years ago, there remained a gap. That gap was that an important category of programs could not be written in any of the high-level languages. Essential programs like operating systems, device drivers, compilers and other “systems” software continued to be written in the assembler language of the machine for which it was intended. There were attempts to fill this gap, but they failed. Then a couple super-nerds at Bell Labs, building on an earlier good shot at solving the problem, invented the language C.

They rewrote the UNIX kernel in C, and the rest is history – C became the high-level language of choice for writing systems software and still holds that place.

Conclusion

Two giant advances were made in programming languages. Each of these happened early in computing history, in the 1950’s. The first giant advance was the biggest: with assembler language, machine language was reasonable to write, read and change. The second giant advance was the category of high level languages. Two minor variants on the concept, COBOL and FORTRAN, established the value of having a language that enabled expressions and could be compiled to run efficiently on any machine. In spite of the on-going tumult of language invention that has taken place since, the fact that huge bodies of working code written in those two languages continue to perform important jobs makes it clear that they embodied the giant advance that was high level languages. The only substantial omission in those languages – the ability to directly reference computer memory – was filled by the invention of the language C.

Most of what’s happened since in programming languages is little but sound and fury. For a review of those non-advances see this.

I do appreciate some of the subtle changes that have been made in language design. Some modern languages really are better – but mostly not because of the details of the language! I’ll treat this issue in a future post. Meanwhile, it’s important to understand history and appreciate the giant steps forward that high level languages as a category represent.

Software Programming Languages: 50 years of Progress

I was inspired to examine 50 years of progress in computer languages by an article in the Harvard alumni magazine written by one of the leading figures in Harvard computer activity, Harry Lewis. Lewis graduated from Harvard College the year I entered, and he stayed on for a graduate degree while I was a student there. He says:

Thirty veterans of Harvard’s Aiken Computation Lab reunited on January 19, 2020, some 50 years after each of us had a role in creating today’s networked, information-rich, artificially intelligent world. Rip van Winkles who had never fallen asleep, we gathered to make sense of what had evolved from our experience as undergraduates and Ph.D. candidates during the decade 1966-1975. One thing was clear: we hadn’t understood how the work we were doing would change the world.

I interacted with many of the people and projects he describes in the period he focuses on. For this post I decided to focus just on one of his major topics, programming languages.

50 years of progress

You might think a book would be required to adequately describe the advances that have been made in programming languages in the last 50 years. In fact, many experts seem to think so. Programming languages continue to advance and improve, even with everything that’s been achieved in the last 50 years – and more!

There is no doubt that many programming languages have been introduced in decades past. The pace of new language introduction does not seem to have slowed. But has progress been made? Yes – if you count dead-ends, bad ideas, going in circles and trivial variations as progress.

While describing the true progress in programming languages is easy, explaining why all the widely-touted “advances” aren’t in fact progress would indeed take a book. Which someone other than me is welcome to write. It would be boring! What could you expect from a gruesomely detailed exposition that amounts to Idea 1 is stupid because …; Idea 2 isn’t any better because …; idea 3 just another attempt with slightly varied syntax to go after the same fruitless goal as Idea 4, decades before …

I gave a talk that included this subject to a large group of programmers in India 25 years ago. Here is the slide illustrating the mighty progress of languages as illustrated by what COBOL calls a Compute statement:

I hope that gives you a basic idea of the massive advances that had been achieved by the mid 1990’s.

The same progress has continued unabated until the present. Google is promoting the language Go, often called Golang. Go must be one terrific language if the geniuses at Google are behind it, right? Here’s that same simple assignment statement I illustrated decades ago, updated to show how Google has made it even better:

A:=B*C+D

This illustrates the exquisite judgment expressed in new language design: The assignment statement has the colon before the equals sign, like PL/1, but leaves off the ending semicolon. Genius!

Uncommon common sense about programming languages

The inventors and promoters of new programming languages nearly always talk about how the wonderful new language is "better" than all other languages. Frequently mentioned are that the new language helps programmers be more productive and commit fewer errors.

In my experience and subjective judgment, all such claims are false and simply not proven out in practice. More important is the fact that the inventors/promoters of new language have no real arguments beyond their own enthusiasm to back their claims: there is NO evidence-backed science to support ANY of the claims that a given language is superior on some measure than others. There isn't even any substantial agreement on the appropriate measure of goodness, much less an evidence-based hierarchy of languages against the measure of goodness!

That should be the end of it. Once you realize how devoid of evidence, experimentation and science the whole endeavor is, it becomes clear that anyone promoting the virtues of a new language is either an ignorant buffoon or self-promoting cult leader. Or some of each.

Following are some observations to help put this into context.

I do understand how the enthusiasm for a new language can catch hold. Being a programmer requires a powerful set of blinders and the ability to focus on an incredibly narrow field for long periods of time. It's essential to being a good programmer! But you have to step back and look at your enthusiasm objectively. Above all you have to have objective measures to compare your wonderful new language to the bad old ones. This is almost never done in software, though it's commonplace in other fields. Would Marie Curie have been a better scientist had she used German instead of French as her primary language at work? See this for more.

There is a hierarchy of skills in software. Knowing a programming language is an important part of the hierarchy but it's near the bottom of the stack! Think about learning a new human language as an adult, ignoring for the moment the huge advantage you have being proficient in a language. You start with basic vocabulary and grammar. How long and hard is it to get the accent a native speaker has? Now think about vocabulary, the basic subset you've learned vs. the huge set of the native speaker. Think about using the basics of the language to get things done compared to using the language as an excellent writer or speaker. Now think about idiomatic expressions, of which any native speaker has a huge set -- without them, you're not fluent. Finally, think about a profession like law or medicine. Those proficient are using the base language, but with a vocabulary,, knowledge and sophistication that take years to learn. These things are like the subroutine library that is a key part of any language though not strictly speaking part of the language itself, and the professional specialty is like networking or database. Yes, the core language is a necessary foundation, but most of the real skill is in the rest. See this for more depth on this important subject.

It turns out that the "goodness" of a software effort isn't really dependent on the language used! When you dig into what really matters, it turns out that factors outside the strict confines of language design are overwhelming important, though details that new-language enthusiasts tend to ignore can play a role. See this for details.

One more key factor in the explosion of languages that are touted as advances but are really just a variation on the same old thing is the core human behavior of creating a language among people who work together on the same things for long periods of time. But no one has found that any of these languages are superior in some real way than others -- they're just the languages that happen to have evolved, each with some unique vocabulary reflecting unique concerns, like the words that people who live in the arctic have for different kinds of snow and ice. This is a well-known behavior, and why shouldn't it also be seen when people "speak" computer languages together? See this for examples and details.

When you lay out the spectrum of languages, you can see some patterns. The most obvious of the patterns I've observed is the extent to which data is defined as part of the language itself or outside the language proper. Data can be fully accessed and manipulated in all cases -- the difference is the extent to which the data definitions are made part of the language itself, or as part of a core library attached to the language. Language mavens are passionate about each of the variations, and convinced that theirs is best! See this for details.

What among other things enables large groups of people to hail the latest "advance" in computer languages without being laughed out of town is the shocking ignorance and lack of interest in computer and software history everyone involves exhibits. With the least bit of software history knowledge, no one would have the nerve to invent a new language, much less brag about it. See this for more.

What does this mean?

If you're a techie, you should get a grip, some perspective and knowledge of history and science. It will help you see, for example, that the fact that Marie Curie's native language was Polish and that she worked largely among French speakers was incidental to her great mind and achievements. It will help you see that Shakespeare wouldn't have been a much better playwright if only he had written in Latin (or whatever), or would have been much worse had he written in Norse (or whatever).

If you're a manager, you should respect the enthusiasm of your employees but understand the fact that their enthusiasm is baseless. I saw a software re-write into a new language using the Neo4j database. Everyone was pumped. Disaster. I have seen a couple cases in which huge enthusiasm was behind a less radical choice. The results were no better in any way than using a more standard language -- the engineers worked harder than they would have because of their enthusiasm, but were less productive because they were less skilled in the new environment. And of course, new hires were hampered with the same problem.

Conclusion

Computer Science is a profoundly unscientific field. The fashion-driven nature of it is exhibited starkly in the decades-long march of revolutionary new languages that keep getting invented -- and having no impact on anything that matters.

If you want to be the best possible programmer, it's incumbent on you to use a sensible, mainstream language and spend your effort producing excellent results -- including bringing the whole notion of excellence to new levels. The language you use will not be the limiting factor -- what's important is the way you use the language, just as it was for Marie Curie.

Computing in 1968: Gender, Cards, etc.

I grew up in Denville, a little town in northern New Jersey. I had a friend whose father knew someone who worked at what was then called EMSI, Esso Mathematics and Systems Inc., housed in a big complex on Park Ave. in Florham Park, NJ. I don’t think I even was interviewed, but I ended up with a job there the summer after graduating from the local public high school in 1968!

EMSI was a service company, one of over 100 companies in the Standard Oil of NJ group of companies. The purpose of the company was to apply advanced math and computer techniques to make the various Esso companies work better. The company had a large computer center on-site, a large room full of card keypunch machines, and extensive offices.

There was a strict division of work by sex. There were secretaries, all women. The men, all dressed in white shirts and ties, were the math and software design people. Software design was done using graph paper and plastic sheets that had flowchart cut-outs in it, so you could make uniform branch triangles, etc.  Here’s a picture of what it looked like (not mine, I never had or used one):

The programming was all done by women, who wrote lines of code onto coding paper, with lines and columns that matched up with Hollerith cards. This was supposed to be a mechanical affair, driven by the graphical designs. Of course it was more than that. When a sheet was ready, it would be sent for keypunching, all done by women using keypunch machines. I think we used the newer IBM 026 machines. The programmers, all women, would get the cards and sort them in a rigid box for storing and transporting to the machine room, where male machine operators would stack them into card readers for loading into the computer. Often there were errors, of course, so the machine would print the error listing on wide, 132 column paper using a mechanical printer. The paper had a string of square holes along each side that fit into gears in the printer. The printers were “line printers,” printing a single line of characters and numbers on the paper. Once the line was printed, the gears would advance the paper one line, and the next one would be printed. The pages were all joined end-to-end, fed to the printer from a big box sitting beneath it. When a print job was complete, the last page would be advanced out. An operator would tear the print jobs apart from each other and put them on a table for people to pick up, along with any related cards to the programmer. The printout could be a list of error messages from the compiler, maybe along with a printout of the program itself. You could ask for the assembler language version of the program to be printed along with the original source code in FORTRAN, which was important when debugging. If the program compiled properly, i.e., had no syntax errors, it would be run. It would either run to completion and print out the desired results, or it would crash, in which case a listing would be printed including a memory dump, which was a hexadecimal listing of the contents of the memory at and around the cause of the crash. I spent many totally absorbed hours poring over and deciphering memory dumps.

The physical embodiment of a program was an ordered set of cards. You would carry the cards for a program around in a cardboard box, all lined up in a neat stack. The box was as wide and high as a card, and at least a foot long, long enough to carry 2,000 cards.  Every once in a while someone carrying a box of cards would stumble and drop the box, scattering the cards on the floor. A fun time would then ensue, figuring out the right order of the cards and putting them back. This is one of the reasons that the last handful of columns in many card decks contained the sequential number of the card in the program, or set of data, or whatever; without such numbers, putting the deck back together would take even longer. I dropped a deck once. It’s the kind of thing that, once you did it, you took great care never to do again, quite apart from the people around you in the hallway laughing at you.

I don’t know how it happened, but I was the exception – a tall, skinny, male high school grad who was bright but entirely without qualifications. Everyone seemed amused by my existence. Perhaps that’s why I got away with things. It’s probably also why I kept getting asked to do things that cut through the otherwise strict divisions of labor. For example, I skipped the usual two-step of getting programs written, the design part by tie-wearing men using plastic guides to make neat-looking diagrams of program operation and flow, followed by women “translating” the design into actual lines of code. What a waste of time, I thought. Just work out what you wanted to do in your head, and then write the code. Before long I skipped the write-the-code-on-paper part, and just typed it into a vacant keypunch machine in the sea of lady keypunchers.

Looking back on the experience, I realize how lucky I was, and how unusual the experience I had working for a large division of a giant company. I now know that there must have been projects with goals and timelines that had to be met. People were judged based on whether they hit the pre-determined goals. While I was stuck into that environment, I was there as an un-planned-for extra resource who fit into none of the pre-defined categories. People threw me bits of work to keep me busy, without ever telling me how and when it had to be done. Each task involved something I had to learn, often a lot I had to learn, so I just tore into it. When I was done, I presented it to the person who gave me the work, much like a dog who dashed off to fetch a ball someone had thrown, perhaps into a lake. I imagine I looked as eager and pleased with myself as a dog, ready to dash after another ball. I never focused on the learning; I just drove towards the ball, overcoming obstacles (mostly things I didn’t know, sometimes the strict hierarchies and divisions of labor) as quickly as I could along the way. It was a formative experience, and one that was at odds with the normal group working experience. I knew I was lucky to have the job, and my appreciation for how lucky I was has grown over the years.

This is an edited excerpt from the draft of my Computer-related Memoir.

Job requirements for software engineers should stop requiring CS

I've read thousands of job requirements for computer programmers over the years, and written or edited quite a number. I’ve interacted with hundreds of software groups and seen the results of their work. I’ve spent a couple decades cranking out code myself in multiple domains. There are a couple near-universal problems with job requirements that, if changed, would improve the quality of software groups and their productivity.

Of course, it’s not just the job requirements and what the hiring people do – it’s also the managers, from the CEO down. They also have to not just support but champion the changes I describe. If they do, everyone will enjoy better results from the software team.

In this post, I'm going to concentrate on just one of the issues: academic degrees

A near-universal job requirement is an academic CS degree. When analytics, ML or AI is involved, the requirement is often “upgraded” to a Master’s or PhD.

There are many capable programmers who have degrees of this kind. Often it doesn’t seem to hurt or hold them back much. But all too often it does! The more specialized the training, for example in project management or quality, the more likely the “education” the person has received is an active impediment to getting good work done.

Here’s the simple, raw, brutal fact: Getting a degree in Computer Science does NOT train you to become an effective programmer in the real world. All too often, the degree results in the person performing worse than someone with self-training and apprentice experience.

That is the fact. Surprised?

Let’s do a little compare and contrast with medicine. Yes, I know that an MD is a graduate degree, while CS is often undergrad. Medical training has evolved to what it is now after decades and centuries of finding out what it really takes to help make people healthy. By contrast, CS degrees just started being granted 50 years or so ago, and are far from even trying to figure out what kind of training helps create good programmers.

First let’s look at the training and testing:

• You don’t even get into med school without taking the MCAT, a challenging test that takes over 7 hours that few do well on.
• Once you’re in med school you take a four year course to get your MD.
• The first two years are academic, including hands-on labs. Then you take the USMLE-1. If you don’t pass this challenging test you’re out. End of med school.
• The second two years are clinical! You’re in hospitals, clinics and offices seeing patients under supervision. And you’re graded. And then you take the USMLE-2, which is harder than part 1 and has lots of clinical stuff. If you fail, you’re not an MD.
• To practice, even as a general practitioner, you have to apply and be accepted into a Residency. Depending on specialty, this can be 3-7 years of mostly hands-on practice, under close supervision.
• During your first year, you have to take and pass the USMLE-3. Fail and you’re out.
• During your last year you have to take and pass the test specific to your specialty. Fail and you’re out.

Here’s the equivalent of the training and testing in CS:

• There is NO equivalent in CS. No entry testing. No exit testing. Just grades on courses determined by professors who usually pass everyone.

A little compare and contrast between medicine and CS:

• Medicine is taught by doctors who practice medicine
• CS is taught by professors, most of whom have never practiced programming in the real world.
• A large part of medical training is working with real patients with real problems, under the supervision of practicing doctors.
• CS is primarily classroom teaching with textbooks and homework exercises. You have to write programs as exercises, but it’s completely artificial. There is nothing apprentice-like or truly clinical about it.
• Medical training is led by doctors who are incented to produce great doctors.
• CS training is led by academic PhD’s with no real-world experience who are incented to publish papers read by people like them.
• Medical journals publish essential information for practicing doctors, giving advances and new discoveries.
• CS journals are read by the tiny group of academics who publish in them. Practicing programmers pay no attention for good reason.
• Bad doctors are fired for incompetence and barred from practicing.
• CS graduates are rarely fired for incompetence. If CS graduates can’t program well, they usually shift into using their non-skills in “management.”
• In medicine, best practices are increasingly codified. You rapidly fall under scrutiny for deviating.
• CS grads seek out and follow fashions that are the software equivalent of blood-letting, enthusiastically promoting them and getting them adopted with disastrous results.
• Hospitals are compared with each other in terms of results. It’s not hard to find which are the best hospitals.
• Groups of CS grads make it impossible to make comparisons between groups, with the result that huge groups produce major disasters at great expense, while tiny groups of effective programmers perform 10X or more better.

All this doesn’t make things uniformly wonderful in medicine. But it goes a long way towards explaining why software is so bad. It’s awful. The awfulness is so widespread that it’s rarely talked about!. If bridges fell down at 1/100 the rate that software projects fail, there would be a revolt! Instead, everyone in the industry just sighs and says that’s the way things are.

You think things are great in software? Check out a couple of these:

https://www.blackliszt.com/2015/09/software-quality-at-big-companies-united-hp-and-google.html

https://www.blackliszt.com/2014/12/fb.html

https://www.blackliszt.com/2014/02/lessons-for-software-from-the-history-of-scurvy.html

The fact is, CS training leads to horrible results because Computer “Science” is roughly at the same level as medicine was when bleeding patients was the rule. See this:

https://www.blackliszt.com/2019/11/computer-science-is-propaganda-and-computer-engineering-is-a-distant-goal.html

Conclusion

There are lots of things you can do to improve the results of hiring software programmers and managers. Here's how the usual interview process goes; here is specific advice about interviewing. There is a whole pile of advice in my book on software people. If all you did was drop the CS degree requirement, you would have taken a big step forward in quality improvement.

Why is a Monolithic Software Architecture Evil?

Why is a monolithic software architecture evil? Simple. There is no need to explain “why,” because monolithic is not evil. Or even plain old bad. In fact it’s probably better than all the alternatives in most cases. Here’s the story.

The Cool, Modern Programmers Explain

The new, modern, with-it software people come in and look at your existing code base. While admitting that it works, they declare it DOA. They say DOA, implying “dead on arrival.” But since the software apparently works, it can’t be “dead,” except in the eyes of the cool kids, as in “you’re dead to me.” So it must be “disgusting,” “decrepit,” “disreputable,” or something even worse.

Why is it DOA (whatever that means)? Simple: … get ready … it’s monolithic!! Horrors! Or even better: quelle horreur!!

Suppose you don’t immediately grimace, say the equivalent of OMG, and otherwise express horror at the thought of a code base that’s … monolithic!! … running your business. Suppose instead you maintain your composure and ask in even, measured tones: “Why is that bad?” Depending on the maturity level of the tech team involved, the response could range from “OK, boomer,” to a moderate “are you serious, haven’t you been reading,” all the way up to a big sigh, followed by “OK, let me explain. First of all, if an application is monolithic, it’s so ancient it might as well be written in COBOL or something people who are mostly dead now wrote while they were sort of alive. But whatever the language, monolithic applications don’t scale! You want your business to be able to grow, right? Well that means the application has to be able to scale, and monolithic applications can’t scale. What you need instead is a micro-services architecture, which is the proven model for scalability. With micro-services, you can run as many copies of each service on as many servers as you need, supporting endless scaling. Even better, each micro-service is its own set of code. That means you can have separate teams work on each micro-service. That means each team feels like they own the code, which makes them more productive. They’re not constantly stepping on the other teams’ toes, running into them, making changes that break other teams’ work and having their own code broken by who-knows-who else? With monolithic, nobody owns anything and it’s a big free-for-all, which just gets worse as you add teams. So you see, not only can’t the software scale when it’s a monolith, the team can’t scale either! The more people you add, the worse it gets! That’s why everything has to stop and we have to implement a micro-service architecture. There’s not a moment to lose!”

After that, what can a self-respecting manager do except bow to the wisdom and energy of the new generation of tech experts, and let them have at it? All it means is re-writing all the code, so how bad can it be?

One of the many signs that “computer science” does little to even pretend to be a science is the fact that this kind of twaddle is allowed to continue polluting the software ecosphere. You would think that some exalted professor somewhere would dissect this and reveal it for the errant nonsense it is. But no.

Some Common Sense

In the absence of a complete take-down, here are a few thoughts to help people with common sense resist the crowd of lemmings rushing towards the cliff of micro-services.

For starters, let’s acknowledge that modern processor technology has simply unbelievable power and throughput. Handling millions of events per second is the norm. The only barrier to extreme throughput and transaction handling is almost always the limits of secondary systems such as storage.

Without getting into too many details, modern DBMS technology running on fairly normal storage can easily handle thousands of transactions per second. This isn’t anything special – look up the numbers for RDS on Amazon’s AWS for example. Tens of thousands of transactions per second with dynamic scaling and fault tolerance are easily within the capacity of the AWS Aurora RDBMS;  with the key-value DynamoDB database, well over 100,000 operations per second are supported.

Keeping it simple, suppose you need to handle a very large stream of transactions – say for example 30 million per hour. That’s a lot, right? Simple arithmetic tells you that’s less than ten thousand transactions per second, which itself is well within the capacity of common, non-fancy database technology. What applications come even close to needing that kind of capacity?

OK, what if something extreme happens? What if you need more, and that somehow it’s your code that’s the barrier. Here the micro-services groupies have it right – to expand throughput, the right approach is sometimes to spin up another copy of the code on another machine. And another and another if needed. I talk about how to scale with a shared-nothing architecture here. Why is this only possible if the code has been re-written into tiny little slivers of the whole, micro-services?

The micro-service adherent might puke at the thought of making copies of the whole HUGE body of code. Do the numbers. Do you have a million lines of code? Probably not, but suppose each line of take 100 bytes, which would be a lot. That’s 100 MB of code. I’m writing this on a laptop machine that’s a couple years old. It has 8GB of RAM in it. That’s 80 times as large as the space required for the million lines of code, which is probably WAY more than your system has. Oh you have ten million lines? It’s still 8 times larger. No problem. And best of all, no need to rewrite your code to take advantage of running it on as many processors as you care to allocate to it.

I can see the stubborn micro-services cultist shaking his head and pointing out that micro-services isn’t only about splitting up the code into separate little services, but making each service have its own database. Hah! With each service having its own database, everything is separate, and there are truly no limits to growth!

The cultist is clearly pulling for a “mere” tens of thousands of transactions a second not being nearly enough. Think of examples. One might be supporting the entire voting population of California voting using an online system at nearly the same time. There are fewer than 20 million registered voters in that state. Fewer than 60% vote, usually much less. Suppose for sake of argument that voter turnout was 100% and that they all voted within a single hour, a preposterous assumption. A monolithic voting application running on a single machine with a single database would be able to handle the entire load with capacity to spare. Of course in practice you’d have active-active versions deployed in multiple data centers to assure nothing bad happened if something failed, but you’d have that no matter what.

Suppose somehow you needed even more scaling than that. Do you need micro-services then?

First of all, there are simple, proven solutions to scaling that don’t involve the trauma of re-writing your application to micro-services.

The simplest one is a technique that is applicable in the vast majority of cases called database sharding. This is where you make multiple copies of not just your code but also the database, with each database having a unique subset of the data. The exact way to shard varies depending on the structure of the data, but for example could be by the state of the mailing address of the customer, or by the last digit of the account, or something similarly simple. In addition, most sharding systems also have a central copy of the database for system-wide variables and totals, which usually requires a couple simple code changes.

Sharding is keeping the entire database schema in each copy of the code, but arranging things so that each copy has a subset of all the data. Micro-services, in contrast, usually involve creating a separate database schema for each micro-service, and attempting to arrange things so that the code in the service has ALL the tables it needs in its subset of the overall schema, and ONLY the tables it needs. In practice, this is impossible to achieve. The result is that micro-services end up calling each other to get the fields it doesn’t store locally and to update them as well. This results in a maze of inter-service calling, with the attendant errors and killing of elapsed time. If all the code and the entire schema were in one place, this wouldn’t be needed.

I am far from the only person who has noticed issues like this. There was even a list of problems in Wikipedia last time I looked.

Making sure your application is scalable and then scaling it doesn’t arise often, but when it does you should definitely be ready for it. The answer to the question of how best to architect a software application to be scalable from day one is simple: assure that it’s monolithic! Architect your application so it’s not database centric – this has been a reasonable approach for at least a decade, think it might be worth a look-see? If you do have a RDBMS, design your database schema to enable sharding should it be needed in the future. Make sure each software team “owns” a portion of the code; if you work towards eliminating redundancy and have a meta-data-centric attitude, you’ll have few issues with team conflict and overlap.

Do yourself and your team and your customers and your investors a BIG favor: stubbornly resist to siren call to join the fashion-forward micro-services crowd. Everything will be better. And finally, when you use the term “monolithic,” use it with pride. It is indeed something to guard, preserve and be pleased with.

The Map for Building Optimal Software

So you want to build optimal software, do you? What’s “optimal” in software, anyway? Is it building the software that’s needed quickly and well, with no bugs? Is it building software that’s easy to enhance, adding new features – including ones you never thought of – with minimal fuss and bother? Is it building software that’s easy to scale endlessly with little trouble? How about all of the above, all at once? Yup, that would be optimal, sure enough.

I’ve described in general terms about optimal software. Here’s a map, a specific example, of how to get there.

Going to new places

Let’s think about going places we haven’t been before, or taking routes that are new to us. What are the options for figuring out how to do it?

If you’re talking with someone who is familiar with the place to which you want to go, you might ask them how to get there. If there are more than a couple of turns, you might try to simplify the directions or write them down. If you’re on your own, you might consult a map. If you’re computer-oriented you might use one of the on-line mapping programs. If you’re driving in a car, you might use a navigation system.

No matter what you do, you’ll end up (let’s say) getting in a car and driving. When you drive, one way or another, you’ll be following directions and making choices of turns. Nothing happens unless you drive and to drive you need directions.

It’s obvious that while you need directions, a fixed set of directions by itself is adequate only if nothing goes wrong – there are no accidents, no construction detours, no recent changes to the roads, etc. The minute there is any problem, you need a map or a live, map-driven navigation system.

What you do when you look at a map, or what a navigation system does, is execute a generic direction-creating algorithm. Either your brain or a computer looks at the map, sees where you are and where you want to go, and figures out the sequence of roads and turns to get you there. Is there an obstruction? The nav system doesn't care -- it just computes a fresh set of directions for you, based on where you are now.

Here's an example. It shows two different routes for me to drive from my home to the wonderfully close source of some of the world's best home-made ice cream: Denville Dairy.

A direction-creating algorithm is a relatively small amount of code that is independent of the map that it uses. You may need to enhance it when different kinds of things are introduced into the map (for example, you decide to support avoiding toll roads or taking ferries), but once you’ve added the capability, it should automatically apply to all the maps you have.

So “where” is the application functionality here? The ability to create directions is in a small amount of abstract code. This code accepts a reference to a map, a starting point, an ending point and perhaps some parameters, like whether to minimize distance or time or avoid toll roads. The actual directions themselves are drawn from the substance of the map. In this case,

• what you’ve got is the map;
• what you do to what you’ve got is create directions.

Isn’t it interesting that most of the time and effort is creating the set of maps, without which there are no directions? And isn’t it interesting that the maps are data?

Maps are:

• Extensive
• Subject to frequent updating and change

An error in a map is bad, but should not “crash the program” – it should just lead to bad results, and should be easily fixed, without going into debuggers, single-stepping and other geeky things.

The direction-creating program is:

• Short
• Infrequently updated
• Unaffected by additional maps
• Unaffected by changes to maps

Best of all, the direction-creating program is unaffected by the driver not following the directions – the program just builds new directions to the destination from wherever the car happens to be.

Errors in this program are crippling, but it’s likely that errors will be found early in the process, after which it won’t change much.

Maps and writing optimal code

So what does this have to do with writing code? Lots.

A map (meta-data) is an efficient, compact, re-purpose-able representation of what the user cares about. It is truly Occamal, without redundancy. Each piece of knowledge you have about the world is represented in the map exactly once.

A reasonably well-written direction-generating program is an efficient, compact representation of strategies for generating routes through the entire universe of possible maps. Each algorithm or strategy for generating directions should be represented in the program exactly once.

Combined, the map and the direction-generating program are a pretty good template/metaphor for an Occamal program. Here’s the pattern:

• You should make everything you possibly can into a “map” (metadata).
• Abstract actions that cannot be reduced to metadata should be coded once, and should be “driven” to the largest extent possible by the metadata.

The map to building optimal software is a simple idea: build the smallest amount of code you possibly can, and put as much information as possible into passive, declarative meta-data. That will get you to your destination safely and in the shortest amount of time without breaking laws.

Lessons for Better Software from Washing Machine Design

The power of Occam-optimality, a.k.a. occamality, can be difficult to appreciate if your head is awash in myriad software buzzwords, and explaining it in software terms can make understanding it challenging to anyone unfamiliar with those terms. So it makes sense at this point to explain the core concept in a different context.

Let’s think about the design of physical things, the kind of things that require discrete manufacturing, have bills of material, and so on, for example a washing machine.

If every section of the washing machine were designed by someone different, each designer may call for a particular screw here, or a particular composition of aluminum there, making his best judgment on what is the best to use. When you broke down the bill of material into a consolidated parts list, you may see two of this kind of screw, one of that kind, and a couple of yet another kind. It may be that the various screws need to be sourced from multiple suppliers. As far as the designer is concerned, the various screws are completely legitimate requirements. Each was chosen to have everything required to get the job done, but only to get the job done. The designer is probably proud, with good justification, of the care and effort he expended in specifying exactly the right screw for the job. When the design is complete, the washing machine will have a list of unique parts and designs and/or instructions for how to construct and/or assemble them.

While the designers may be proud, everyone else’s jobs have been made more difficult and the overall expense of building washing machines has been increased. If instead the designers had been motivated to get together and find a way to use a single screw type that could be sourced from a single supplier, more screws would be bought from a single supplier in greater quantities, reducing both unit and shipping costs, and enabling all assembly points to be supplied from a single pool of screws. Only one type of screw driver would be needed, which would also help keep all the screw drivers in working order during manufacturing. Even the repair manual would be slightly easier to write and would be slightly shorter, and repair people would need only a single type of screw and a single type of driver, which would increase the odds that they would have what they needed to do a repair on a single visit. If the design were done in this way, the overall design would be virtually unchanged, but the parts list would be shorter – there would be a smaller number of unique parts, with each used more frequently.

How about the designers? Instead of starting from scratch when they needed a screw, picking the “right” screw from the universe of available screws, which is large, they would agree up front on a screw that would meet all their anticipated needs closely enough, and then, every time they needed a screw, they would just check to make sure the pre-chosen screw was adequate to the job. This is a simpler job than selecting the “best” screw from a large selection, because you’re just applying a short list of disqualifying characteristics to the pre-selected screw. Moreover, you’re arranging your design as you go along (for example, the thickness of the plates) to match up with the qualities of the screw, making the selected screw very likely to be OK in each case. So the “shortest” (without going too crazy) parts list, the one with the smallest number of unique parts, clearly is the best design.

Notice that the original design, the one with lots of types of screw types, would be defended by its designers as the “best” design, because they understood that the important thing for them to do was to create the “best” solution for each design problem they faced, considered in isolation. Let’s assume they did. But once the designers look at the whole problem, and try to pick the best design for the entire lifecycle of the washing machine, including sourcing, manufacturing and repair, it’s obvious that the different screw selections were incidental aspects of the washing machine design. The design could just as well have been embodied in one that had fewer screw types, and as we now understand, having fewer parts is better.

Suppose there are a couple of established washing machine manufacturers. Suppose you wanted to enter the market with a superior product. If things worked the way they normally do, the earlier products are probably far from occamal. That means it's possible to design a washing machine with just as high quality as the ones on the market, but less expensive to build and easier to service. Everyone would like your product better than the existing ones. Cheaper to build, cheaper and easier to fix in a single visit; what's not to like? This means that, in addition to being a core design principle, occamality helps us understand and predict the evolution of products in a market, all other things being equal.

In a virtually identical way, a program that has the smallest (within reason) number of unique “parts” is the best, for reasons that are very similar to the way that the best physical design has the shortest list of parts. It definitely helps the designers, and it helps everyone else even more. Just like with a washing machine, a program can be built, from the requirements all the way through, to have lots of unique “parts” and custom things, each of which may make sense when considered in isolation. But when you look at the whole software lifecycle, taking a view over the entire program and everything that will eventually be done to it, including later maintenance, change and enhancement, it becomes clear that designing it to contain the smallest number of “unique parts,” so that every “program concept” is defined in exactly one place, yields the best design.

The parable of the screw can help you, your organization and your users avoid being screwed during software design. You’ll get there if you keep this principle in mind: always use the smallest number of unique “parts” that you can, and be willing to adjust your design to make the number even smaller. This may sound easy to you, or it may sound like an essentially trivial exercise in neatness. It is neither. It can be challenging in practice, and the implications of it are actually profound and far-reaching.

Experts vs Innovation New Book

Experts and anointed authorities of various kinds, both academic and commercial, have been the front lines of resistance to innovation for centuries, up to the present. They are the firewall keeping what they consider to be rogue ideas outside the protected environments they oversee, protecting them from bad influences that their naïve but innocent charges might inadvertently adopt. It’s a good thing they’re on the job – otherwise things would be chaos and nothing would get done!

This pattern is raised to a new level when the subject isn’t some specific business domain like healthcare, but the process of innovation itself. As you may have noticed, many organizations now have the expensive modern equivalent of “suggestion boxes,” departments devoted to fostering innovation in the organization, led by a Chief Innovation Officer. Government officials have gotten into the game, establishing centers for innovation, and “incubators” for startups. Eager not to be left behind, academia has jumped into the game, with august professors doing what they do best: pronouncing truths and activities designed to promulgate them.

There was a time in my relatively innocent past when I was willing to give the experts a pass. Hey, how can you know everything? I know I don’t!  They’re probably just trying to keep their organizations from being taken down by harebrained ideas, and sometimes they fail to recognize a true innovation when it appears! I no longer believe that pleasant, forgiving fiction. They’re also not evil geniuses immediately recognizing a juicy innovation when it sniffs around the door, and stamping it out before it can start making changes. The truth is far worse: the vast, vast majority of them wouldn’t know a practical, effective innovation if it came up to them and slapped them in the face! See this and this for more.

The bulk of front-line experts act this way to “protect” their organizations against scary change. They go to great lengths on multiple dimensions to assure that nothing upsets the status quo. Here is detail about the measures they take to prevent innovation, and simple methods to overcome the measures – which the experts are universally ignoring.

But the elite of the experts are experts in entrepreneurship – people who are “expert” in enabling people who want to create and lead innovation to make it happen. These experts on innovation are like experts on being expert. We are definitely still in the middle of an “innovation” bubble, with everyone acting like it’s a new thing. In fact, it’s looking like less of a bubble these days, and looking more like something that’s going to stick around. Here’s some information about the bubble and how we got here.

I’ve seen so much of this over so many years, and been so disgusted with the useless wisdom of experts, that I’ve put together some preliminary thoughts, drawn from experience and observation, about how innovation happens. See this.

Imagine my surprise when I read an article about entrepreneurs that actually made sense! I'd never heard of the guy, Carl Schramm.  It appears from the article that he knows a bunch of stuff about entrepreneurs and innovation that matches up pretty well with my observations, but is actually backed by … real data! OMG! One of my initial shocks was learning that he was a real professor at a real university, even with a PhD, but that … he had worked as an entrepreneur! How is that possible? How did he manage to slip by the super-strict requirements that prevent anyone who actually knows something from experience becoming a Professor?? Maybe it helped that he had been the head of the Kaufman Foundation, the largest foundation devoted to the study of entrepreneurs and innovation, and that instead of just doling out grants, he did real studies to gather real data. What an idea. You think maybe the people who run Departments of Computer Science could get inspired? Sorry, forgive me, you caught me dreaming the impossible dream there...

I’m not finished reading the book yet, but here it is:

As you might guess from the title, he talks a lot in the beginning about that near-universal requirement for getting innovation funded, the dread business plan. He trashes it. Vigorously and effectively. No, he doesn’t trash this or that business plan, he trashes the very idea that business plans are both essential and good. He’s right! For exactly the same reason (he doesn’t say this, I’m saying it) that project management in software development is not just brain-dead, but a positive impediment to getting good software done!

If you are ready to learn about a different and better way to be an entrepreneur, check out this book.

Here's How the FDA Can Reduce Medical Device Costs While Improving Healthcare

The FDA wants to keep us safe. They want the drugs we take to be what they’re supposed to be, and they want the medical equipment used on us to be safe and without fault or error. We all want that!

However, the way they choose to achieve the goal for the software that is an essential part of most medical devices is deeply flawed, and leads to huge expense with only a small number of companies willing and able to follow the FDA’s regulatory regimen for software. The net result is medical equipment and software (which is increasingly a key component of medical equipment – think MRI machines) that is wildly expensive and uses seriously outdated technology.

There is a simple, easily understandable reason for this horrible state of affairs, which the grandees of the FDA refuse to acknowledge or even understand. The root of the problem is that they don’t understand software. Which doesn’t stop any of them from being certain they can regulate it. Because of this inexcusable ignorance, they take the regulatory approach developed over many years for drugs and manufacturing and apply it with only cosmetic changes to software. Safety is safety, they probably say to themselves. We’ve proven our approach for drugs; why start from scratch for software?

The mistake made by the FDA, along with nearly all the hard-charging graduates of MBA programs, law schools and bureaucrats everywhere is in thinking that the process of manufacturing is like the process of building software, except not as visible or physical.

In manufacturing, you have a factory with raw material arriving, being processed in a series of steps, with quality checks along the way, and emerging as finished goods at the end. The important thing is to check the quality of the raw materials as they enter and the results of each step of processing to make sure it’s up to snuff. At the end all you need is a cursory quality check, because so long as everything is done right along the way, the result is probably good.

Similarly, they think, in software you have a set of requirements, with lines of code and software components being produced along the way, with careful unit testing being performed at each stage, and more tests as the components are woven together. The end product is subject to more testing, but the important testing has already been done. The idea is that quality is designed in.

This method for building software is exactly what, in gruesome detail, the FDA requires. It’s spelled out in highly detailed regulations. Sounds good, right? Why would anyone want crappy software, particularly when it comes to our health?

The trouble is that this whole way is thinking is based on a blatantly false analogy.

What they think is that the manufacturing process of converting raw materials to finished goods is just like the process of creating lines of code and combining them into a finished software product. People even talk about “software factories,” and how important it is to churn out quality code, on time and on budget. Still sounds good, right?

Here’s the problem: a factory that produces finished goods, whether they’re drugs or cars, is making copies of a design that’s already been created and tested. In the drug development process a drug is created and validated through testing. All that’s done in the drug factory is assure that the copies that are made of the already-designed-and-proven drug are exactly and only what the drug creators intended.

Designing and building a new piece of software is like the drug development process, not the drug manufacturing process. The software is created for the very first time, with changes made along the way. The software equivalent of a drug factory is trivial: it’s taking a piece of executable software and making a copy of it. There is a universal software “factory” that works on all software: the copy utility. It’s what happens when you go to the Apple or Google software store and download the software you want. The download makes a bit-for-bit-100%-accurate copy of the original software for your use. That is software “manufacturing!” There’s even a universal quality check – a checksum is always incorporated in the original prior to copying that the receiver can check. The checksum tells you whether the copy is perfect, just like all the drug manufacturing quality checks do, only with software, it’s easy. Yes, because software is different. Here is more about software factories.

What the FDA regulations do is specify and control in gruesome, expensive and time-wasting detail the process of building the very first, original copy of the software – like creating the drug in the first place. This is a complete and total waste. The methods and processes of building software are constantly evolving, with the most innovative companies, the ones that actually create new software, at the forefront. These companies have small, focused teams who crank out great software, and do it quickly. They use what can be called “wartime” methods of building software.

The FDA should scrap its mountain of software regulations and replace them with a simple set of regulations that achieve the same goal, more effectively. I describe this in detail here. The new regulations amount to something like “We don’t care how you build your software, but its your responsibility to assure that the software performs its stated job each and every time, without fail. If the software has errors that cause medical harm, you are responsible for the damage it causes, and you may be barred from supplying software to the medical market in the future.”

This of course, shifts the entire burden onto the software creators – as it should. Inspections are no longer required. Employment at the FDA should go down, but of course, since it’s the government, it probably won’t.

Changing the medical software regulations in this way will unleash a wave of innovative, low-cost medical software. It will be as though runners were required to carry 100 pound backpacks and walk on stilts; as soon as they can dump the pack and use real running shoes, just watch them set records! They will be a race with each other to see who can cross the finish line in style the fastest, with no stumbling along the way.

William of Occam is the Inventor of the Method for Building Optimal Software

Lots of people would like the credit for establishing, once and for all, the core, immutable principle of optimal software design/architecture – the method for measuring which, among an infinite number of embodiments of software requirements, is provably the best among them. I am one of the crowd that would love to take credit! Alas, I and my competitors have been beaten to the punch, by a mere 700 years or so, by a guy named William of Occam.

As it turns out, the software crowd is, as usual, late to the party: loads of luminaries in other fields have recognized William’s achievement, viz., Occam’s Razor, for the indispensable concept that it is. Various software folks have been nipping around the edges of it, for example with the DRY (don’t repeat yourself) principle that’s gotten some attention. But they haven’t grasped the fundamental, touches-nearly-everything, deep nature of the Razor. Too bad! Crappy software continues to be churned out by the ton by people who are enthralled by false idols, convinced they’re on the righteous path toward software excellence, when in fact they’re doing something else, words for which should not be used in polite company.

William of Occam (ca. 1285 to 1349) was an English logician and Franciscan friar.

He is credited with formulating a principle that is already widely applied to various aspects of computer systems. In those areas of computing to which it has been applied, it reigns supreme – it unquestionably supplies the most fundamental, relevant and optimal understanding and solutions to the relevant problems, so much so that the people involved don’t think about it or question it.

However, there are large areas of computing to which Occam’s razor has not been applied. Worse, it is not even one of the candidates under consideration. As a result, those aspects of computing are fractured, inefficient, unpredictable, and driven by fashion and politics. Even while the practitioners call themselves "computer scientists," which is something that none of them are.

Everyone involved knows that the whole process of specifying, designing, building, testing and supporting software is hopelessly inefficient, unpredictable and error-prone. The leading methodology that concentrates on the process of building software, “project management,” is theoretically bankrupt and in any case has an empirical track record of failure. In terms of the content of good software, there are fierce battles among competing approaches, none of which is anything but a collection of unsupported and unfounded assertions, and which in practice don’t contribute to building good software.

Occam’s razor leads to the principles on which good software may be built, and supplies a single simple, widely applicable theme that, when applied, cuts away (as a razor should) all the inefficiency, inapplicability and generally what we don’t like about software. Once the principle is understood and widely applied, I expect it will become the un-discussed and undisputed standard for how software is built, just as it has in the other areas of computing to which it has been applied.

What is this amazing standard? It’s simple. I mean literally simple. That’s what Occam’s Razor is all about.

Occam’s razor is:

    Entia non sunt multiplicanda praeter necessitatem.

    No more things should be presumed to exist than are absolutely necessary.

Occam’s razor applied to software would be:

    No more software entities should be created than are absolutely necessary.

Another way of expressing this is that all redundancy, of any kind and in any way, should be eliminated from a piece of software. This often requires writing imperative code to implement any and all abstract actions, and to create largely declarative meta-data to specify everything that is specific to an application. The meta-data, of course, should also have no redundancy; this is normally achieved by use of multiple inheritance with override. See this for more.

To be extremely brief, the reason why eliminating redundancy is good is that practically everything that happens to a piece of software is that it is enhanced, altered, bug-fixed or otherwise modified. Creation happens just once; everything that follows is growth and change. Implementing change of any kind to a program takes the least effort with the least risk if everything is defined in exactly one place – once you’ve found the place, you make the change and you’re done. Among the infinite universe of programs solving the same problem that work, the speed, cost and risk of change is the overriding virtue, both in technical and business terms.

Thanks to William of Occam, I can’t claim to have invented the core principle of software, and the way to express the measure of goodness for software. William has me beat by over 700 years. Thanks a lot, Bill! Lots of people have applied Occam’s Razor to lots of tough things with optimal success, even in things closely related to software, such as information theory. Way too late again. About all I can do is mess with his name and make up a term that expresses goodness in software. Here it is: I propose that a piece of software can be measured by its “Occamality.” The more “Occamal” it is, the better it is. And I propose that Occamality is strictly correlated with the extent to which there is no redundancy of any kind in a program, which is almost always strictly correlated with the program being at the high end of the hierarchy of abstraction.

How to Pay Down Technical Debt

When we look at a body of software, how do we judge it? Clearly, there are strong opinions on this subject – but there is no general consensus on the standards by which a body of code should be judged! This is shocking and unacceptable!

For most normal people, i.e., people who aren’t software developers, software is “good” if is doesn’t crash and pretty much does what you want most of the time. What more could anyone want? Well, how about changing the software. This is one of the BIG things that makes software different from pretty much anything else in our experience. How often do you try to make changes to your car in the same sense that you change software? The answer is never.

There are standards for nearly everything in life, certainly for things that are built by engineers and other technical people. How is it that, not only are there NO standards for what constitutes “good” software, but the fact that there are no such standards is not decried, and no one appears to be sheepish or unapologetic about admitting it – because they’re never asked to admit it. The subject never comes up – not only in “polite company” but even in “impolite company,” i.e., among programmers talking quietly among themselves.

“Ya know,” someone might say, “we tell them there’s technical debt. Sounds fancy, right? Anyone ever been asked what it means? They seem to think that WE know what it means, and that’s good enough for them. I wonder how much longer we’re going to be able to get away with it?” Someone else says, “Forever, man. This has been going on since before we were BORN, ya know? Unless word somehow gets out, and how could it, we’ll die without anyone being the wiser. Good thing.”

I wish programmers did have quiet dialog among themselves like the one I made up above. But they don’t, at least that I’ve ever heard. There’s nearly always someone who has strong opinions about what software should look like internally, and of course the current body of software doesn’t measure up. So there is agitation and there are threats about the awful future consequences of failing to rein in the spreading disease – soon! RIGHT AWAY!! – unless the transformation is authorized.

Then the re-writing death march starts off, with various sections of the code slated for repair, renewal or even flat-out re-write. What is the target? It varies. Favorite criticisms include that the code is spaghetti, it’s a monolith, it’s not structured according to some set of object-oriented principles, it’s written in some obviously inferior language, there aren’t components, there are no services, micro or otherwise, and on and on. The solution is obvious to the converted, and value to be achieved by the solution is so obvious that it’s hardly worth stating.

Is there any value to be gained by the clean-up of “technical debt?” Maybe. But often, nothing really changes except a lot of time has been spent with no improvement to the business. No one can point to a proof or a study or a clear historic pattern demonstrating the effort is worth making.

This is a shame, because there is a clear pattern of success for a method of organizing and/or re-organizing code for business benefit.

Let’s start from the business benefit. Of all the ways a piece of software could be written that performs the same function, which is the best in business terms? I’m assuming here roughly the same level of performance, etc.

Tick, tick, tick… Think about it!

When you come right down to it, no one cares about today’s software doing today’s things for today’s customers – so long as it works and performs reasonably well. What matters is simple, and everyone knows it: it’s the next request by an important existing customer, and more important, the next potential customer who is balking about things they need that aren’t there, the time, cost and risk to implement the software, etc. In other words, what matters about today’s software is TOMORROW’s changes, whatever they may be – we can suspect, but we won’t know until we know them!

Software people have talked forever about things like “architecting for change” and various fantasies that mostly go poof when the harsh wind of reality rushes at them.

There are exactly two rarely-discussed aspects of code that position it optimally for the widest possible range of unanticipated changes and the needs of installation:

• A minimum of redundancy in the code – anything you want to change can be changed by going to one place
• To the largest extent possible, particular things the application does are defined in meta-data instead of code, with the result that the code is smaller and more abstract with minimal redundancy -- and that any change can more likely be made by changing meta-data, which is quicker and safer than changing code.
  

That’s it! See this for more.

Why the Equifax Hacking Disaster Wouldn't Happen at a Car Dealership

The 2017 Equifax data breach is in the news again because of the recent indictment of four Chinese government hackers for committing the break-in. How did they pull off such a feat? From China? As it turns out, Equifax’s defenses were so pathetic that a couple of bright nerd wise guys anywhere could have done it.

By contrast, the Equifax hacking could not possibly happen at a car dealership – unless the dealership were run by government and corporate cybersecurity experts. Understanding why that’s so tells you everything you need to know about the expertise of the experts.

The 2017 Equifax hack

First, a quick – VERY quick – review of the Equifax hack. Equifax has websites. One of them is just for people who want to dispute their credit rating. This particular site was run on a computer with software that had a serious flaw, allowing a skilled person to get past the normal consumer web pages and run other programs on the computer. The flaw had a fix that could easily have been installed, but because of a series of bungling and delays, the fix wasn’t installed for months.

The flawed software on the Equifax site was widely used. The fact of the flaw and the fix for it were publicly available. Anyone could have read about it and fired off a search for websites on which the patch correcting the flaw had not been installed. Someone found the Equifax site had the issue, exploited the flaw, and ran a couple programs without doing much else.

More than a month later, the flaw still unpatched, someone got into the Equifax server again and started “looking around.” They found an unencrypted file with names and passwords. They used the information to log in and got access to a series of databases that had Equifax customer information. For 76 days, they used the databases like any authorized user would, and issued queries that returned a great deal of customer information, which they stored in files. They encrypted the files and used standard programs to ship the files out of Equifax, presumably to themselves.

Hacking the Equifax-run car dealership

Suppose someone had tried the equivalent of the Equifax hack at a car dealership. The hacker would have physically walked in the service entrance, along with the other existing customers. The hacker knew that many doors had a security flaw in their lock that had not been fixed. The exact nature of the flaw had been publicized, and any reasonably skilled person who knew about doors and locks could exploit it. Once inside the customer service area of the dealership, the hacker looked for the “employees only” door, and quickly saw that it had not been fixed. The hacker walked up to the door, fiddled with it for a couple minutes, opened it and walked in. In a real car dealership, the employees would have immediately noticed a strange person and challenged him, politely showing him out. In the Equifax-run car dealership, the intruder is ignored.

Once in the employee section of the dealership, the hacker wanders around, poking into lots of things. Finally he walks into the finance department and wanders some more, again unchallenged. He notices a row of file cabinets against the wall, and figures there must be valuable information in there about customers and cars, with all sorts of details like names, addresses, driver’s licenses and who knows what else. But he sees the cabinets are locked. Darn! So he looks around some more and spots some keys sitting on someone’s desk. Even though it’s daytime and people are working at the desks, on the phone, etc., no one says a word when the intruder picks up the keys, walks over to the file cabinets, and tries one key after another until finding the one that works. He opens the cabinet, takes out a handful of folders, walks over to the copy machine, and makes a copy of every document in the handful of files. He then goes back to the file cabinet, returns the originals, puts the keys back on the desk, and walks out of the building the way he came, holding a big pile of copied pages in his arms. Unchallenged.

The next day, the hacker returns to the Equifax car dealership and goes through the same drill – goes into the employee-only section, then the finance department, uses the keys to open another drawer, makes copies and walks out, all without a single one of the many employees working there saying a thing. He does this day after day for 76 days.

Finally, someone notices that there’s supposed to be a guard by the employee entrance checking everyone – there always used to be one! So a guard is put there again. The guard notices that the daily visitor carrying lots of paper doesn’t look like everyone else – he’s carrying big piles of paper! The guard doesn’t stop the visitor, but reports to his boss; meanwhile, the visitor notices the guard, figures the jig is up, and stops coming.

More than a month later, the Equifax-run car dealership’s bosses finally let the word out that they had been hacked.

The Equifax breach vs. the car dealership

You already know that nothing even vaguely like the Equifax breach could have taken place in a car dealership. In the dealership, people have common sense and are dealing with physical things, while at Equifax, everything important happens on computers that are in locked rooms, with software that is invisible to nearly everyone, doing things that most experts barely understand, managed by people who have no real knowledge of software using management methods taught in business schools by professors who are ignorant of software, and following rules and regulations written by lawyers and bureaucrats. What can you expect but madness and chaos??

What went wrong at Equifax

Equifax’s cybersecurity methods followed regulations and industry-standard practices, with all required certifications. The result of these methods, when executed perfectly, is security that is worse than that of retail stores or libraries. In the end, Equifax made exactly two mistakes – but each mistake mattered because of further bungling and the failure of ordinary follow-up checks.

• The web server patch, the start of the trouble. The patch was announced on March 7, 2017.
• What they did: On March 9, administrators were told to apply the patch. They didn’t. On March 15 a scan was run that was supposed to detect unpatched systems. It didn’t work. It wasn’t run again or fixed. Result: the patch wasn’t applied until August or later.
• The bad traffic detection box. A great deal of web traffic is encrypted. Equifax had installed a box at the “edge” of their system to stop all incoming encrypted traffic, unencrypt it, make sure nothing “funny” is going on, report suspicious activity, and re-encrypt and send along each message. It’s like a traffic stop for incoming traffic.
• What they did: encryption works with keys, and keys expire periodically. The key used by the box expired 10 months before the hack started. Equifax failed to renew the key, and the traffic-stop box was set up to let all traffic through without checking unless it had a valid key. No one noticed until July 29, 2017, when the certificate was finally renewed. When the traffic stops started again, an administrator noticed suspicious activity, and sounded alarms bells.

I’m going to add this fact, just because I find it amusing: the Chief Security Officer was a music major, Susan Mauldin. She had degrees from U Georgia in music composition. Her background was scrubbed from the internet as soon as the scandal broke. Of course, you don’t need a college diploma to be excellent at software, IMHO. But in this case…

What car dealership management would do at Equifax

People who run car dealerships aren’t usually thought of as geniuses, but part of succeeding in business is protecting your customer records, and everyone involved takes care to do that and do it well. All that was required for Equifax to have avoided being hacked was to do the software equivalent of what every sensible car dealer employee would do. Obviously, such common sense is beyond the ken of the high-paid professionals and certified experts at large places like Equifax. Here are the main things that would be done if car dealer people replaced Equifax management.

• Recall/repair notice.
• The second a car dealer hears about a recall/repair issue for cars, they jump on it. Similarly, just exercising common sense, if they learn that the restricted entry system to the employees-only area has a fixable flaw, they would get it fixed immediately. No excuses. Car dealer management would have seen to it that the flaw was patched, and checked.
• Guard duty.
• At the car dealership, there is extra security around the finance department, which holds all customer data. During working hours, there is always someone alert to the door, challenging anyone entering who shouldn’t be there. Off-hours, there are deadbolts on the metal doors, along with an alarm system that has motion detectors and cameras, which gives an immediate alert on any sign of trouble. If the alarm system drops its constant, real-time communication with the monitored center, electronic alerts are sent, so the problem can be immediately fixed. The room and its data are NEVER left unprotected.
• With car dealership management, the encryption key on the traffic checker wouldn’t have been allowed to expire. Instead of setting up the traffic checker to stop checking unless it had a valid key, car management would have made it set off alarms so it could be fixed right away.
• When car dealership management learned that only incoming traffic was being checked, they would have pitched a fit. What? You’re just letting anyone waltz out with anything?? They would have installed a system to stop and check outbound data traffic before letting it out.
• Keys
• Car dealership management would never allow the file cabinet keys (user names and passwords) to be lying around for anyone to pick up.
• Employee behavior monitoring.
• Car dealership management would make sure anyone opening a file cabinet was a person authorized to do so, and that their actions were reasonable. When they found out that anyone could open a drawer and pull out and copy ALL the files in that drawer, they would have been enraged. They would have immediately put a software system in place to ring a bell and prevent anyone from taking more than a single file.
• While at Equifax the only real checks were with people as they were coming in, after which they could do anything without being checked, car dealership management and employees know that everyone has roles and acts in certain ways – and that everyone is responsible for noticing unusual behavior and questioning it. With car dealership people in charge, the software equivalent of such monitoring for “normal” behavior would have been implemented and strictly enforced, with immediate shut-down of a user if they stepped outside the bounds of their normal actions.
• Car dealership management understands that audits need to be done, and that auditors need broad access to customer files. This is the ONLY time mass access to customer files would be permitted, and ONLY under the watchful, suspicious eyes of multiple dealership managers, who would ASSURE that all files would be replaced WITHOUT BEING COPIED.

Expert recommendations for Equifax

The “experts” have had a wide-ranging set of advice for Equifax. Equifax has spent over $1.4 billion dollars making largely useless “improvements” to its security. I haven’t read ANYWHERE recommendations of the kind of changes any sensible car dealership would make, as described above.

Here are some of the leading recommendations of what Equifax should do to improve their cybersecurity:

• Change management reporting, processes and procedures.
• I love this one. It’s a commonly-recommended “cure” for cybersecurity ills.
• Encrypt all customer data
• This is a favorite, and widely recommended. It is USELESS. It would NOT have prevented the Equifax hack!! Why? Easy: once the hackers were in and were using employee user names, they just issued SQL queries against the database. If the data on the disk, the database un-encrypts the disk blocks, processes the SQL query, and returns UN-encrypted data. Otherwise, the data can’t be used!
• Encrypting data on disk is like having locked, strong metal file cabinets. But authorized people still need access. Therefore, file cabinets have drawers and keys. When you open the drawer, the data is easily accessed. Encrypting data “at rest,” as they say, protects only against the hackers who somehow get close to the cabinet and drill into it from the side or bottom. Who would do that? It’s easier to steal the keys or break the lock!
• Create more silos
• The nice-sounding theory is that breaking everything into silos would limit a break into just one silo. But far from being a solution, silos were actually part of the problem at Equifax – applying the delayed software patch required writing memos and asking multiple people to do things, when in a uniform environment, a single script could have updated everything.
• Change the reporting structure
• Because when you change who reports to whom, everything changes, right? In Equifax’s case, the Chief Security Officer reported to the Chief Legal Officer, while the Chief Information Officer reported to the CEO. “That’s the cause of the breach!” shrieked an amazing number of pundits!
• What there should probably be is two Security Officers:
• A CSRO, Chief Security Regulation Officer, who reports to the Chief Legal Officer. This person is in charge of the massive, ever-changing, lawyer-created body of regulations that are supposed to assure Security. There are severe penalties if you fail to conform to the regulations, which require loads of reports, processes, documents, etc. But they have little to do with real security.
  
• A CRSO, Chief Real Security Officer, who reports to the Chief Technology Officer. This person is in charge of making sure that real security is performed, in spite of the regulations.

Conclusion

Computer software and systems are hard to understand, a problem made worse by the fact that they’re literally invisible. You'd think that would be OK, since for example even fewer people understand quantum physics and we’re OK in physics. The trouble is, the people doing physics really do understand it, while the people doing software in general, and cybersecurity in particular, are faking it – without even suspecting, in spite of the mountain of evidence to the contrary, that they’re faking it.

The result is that government agencies, powerful consultants and weighty experts recommend more of the same medicine that created the problem, without a shred of recognition that it was their own rank stupidity that caused the problems to begin with.

For more perspective on this subject, see these posts.

Note: this post first appeared at Forbes.

How to Build Applications that can be Changed Quickly

Nearly everyone who talks about fast and reliable application building seems to talk about the process (Agile etc.) and/or a couple architectural things (object-oriented, micro-services, etc.). From what I can tell, these haven’t led to any break-throughs in application velocity or predictability. However, it’s clear that there are methods for high-velocity application growth. A wide variety of small, motivated groups have re-invented them over the years. I have had the privilege of observing these groups and their methods, and seen the impact when they add or drop velocity-aiding methods.

Here is an overview of some of the methods I’ve seen used. The improvements can range from cutting time in half to whole number factors, things like turning weeks into days or in some cases hours.

Forget requirements, focus on change

The normal requirements process starts with the often non-technical people who have a need. The need can be expressed directly, or through refusal to buy/use a product, which gets the attention of sales and marketing people. There then ensues a process of requirements gathering and refinement. Wire-frame UI’s may be involved. Finally, the requirements are ready for engineering, and away we go on hopefully not too many “sprints.” This is an “outside-in” approach.

Speed is achieved by taking an “inside-out” approach – instead of focusing almost exclusively on requirements, center your efforts on the code and what it can do or easily be changed to do today to go in the direction of the requirements. Take steps to change and add to the existing code to meet the unmet needs. Do this in small steps, getting feedback at each step. When you’re done, go back and “clean up” the code to eliminate redundancies as much as possible, to make the next change as easy as possible.

What this does is ground the changes in the existing code, resulting in the changes and additions being smaller than they otherwise would be. As a additional benefit, it usually results in a greater uniformity of both the code and the interfaces (UI and API) to the code.

Minimize the customization and optimization of the code and UI

When product people think about a new process that has UI, they usually try to do a good job, understanding all the details and nuances involved. The result of this sophistication is often requirements that are finely adapted to the particular function being performed. What’s wrong with this, you may wonder? One problem is that this approach ends up taking longer to built and results in more code. More important is the users – the less someone has to learn to use a new feature, the more quickly and easily they’ll be able to use it. Familiarity is incredibly important. Which means, among other things, that literally using rarely changing templates to generate interfaces will both make any new interface element recognizable and easy to learn, but also lead to the least amount of code changes to get the job done.

This means that the fineness and sophistication of product people can be a problem. The more time they spend on a subject and the more they think and reflect on it, the worse the temptation is likely to get to do something other than just crank out another instance of a familiar pattern.  Resist!

Avoid planning and designing for change and future requirements

When starting a project, accomplished software professionals often insist on a period devoted to design or architecture, in which the overall technical approach to the project is determined. When things go wrong later, inadequate or insufficient up-front design is often cited as the reason. “We’re gong to insist on being able to do it right next time, so this won’t happen again.” But all too often, projects slip and slide regardless of any up-front design work.

Generally, up-front design time is actually counter-productive. The reasons are simple: you spend time “designing” when you could have been building; the result of the design is often building for anticipated change, which takes time; when the anticipated change doesn’t happen and unanticipated change arrives, the code added because of the design almost always makes things harder.

None of this happens when you build incrementally and strive toward uniformity, eliminating code redundancies and moving functionality from code into meta-data.

Wartime software

The methods I have described as wartime software maximize speed and efficiency of building new software that meets business needs. See here for details. Generally, the methods include:

• Eliminate overhead
• Avoid things resembling “project management”
• Optimize for speed instead of expectations
• Use production environments for testing instead of sand boxes
• Use comparison QA; avoid test-driven development, unit testing and test scripts
• Minimize the use of documents, meetings and other internal overhead
• The main architectural value should be speed and scale
• Avoid technical fashions

Move up the mountain of abstraction

When you think about code and methods, the main values should be the elimination of redundancy in both data definitions and code, and migrating functionality from code into meta-data. See this and this.

The other methods described here are important for moving quickly, and are complementary to this one. But moving up Abstraction Mountain is the most powerful of the methods, often yielding qualitative improvements.

Conclusion

This is a short post, and the methods are described briefly. In a different world, each topic here would be at least a course in the Computer Science curriculum, and in a couple cases multiple books. I don't claim anything I've said here is fully described. But every single item has been proven in practice by many groups over many years, in different ways, always resulting in better quality software that meets business needs being built dramatically more quickly than other methods could possibly have achieved.

My goal here is simply to provide a short list of the main methods for fast development in a single place.

The Iowa Caucus Software Problem Exemplifies Widespread Software Management Problems

Everyone who’s even vaguely in touch with the headlines knows there was a problem getting results from the Iowa caucuses – a problem blamed on a software app built to automate reporting voting results. No one, including the maker of the app, disputes there was a serious software problem.

Most of the commentary has focused on steps that should have been taken to assure that the app worked better than it did. The Wall Street Journal quoted various “experts” who stated – authoritatively, no doubt – that “testing could have prevented” the problems.

What’s not being said is the emperor-has-no-clothes “secret” – our society is irrationally obsessed with software, and acts convinced that getting new software to perform some function that is now getting done with older software or without software will make things better, be worth the trouble and have no downside. Otherwise sensible and experienced people continue to ignore the ongoing, rolling disaster of software building and deployment, and blissfully welcome new software nightmares, as though there’s no chance that anything could go wrong. The Iowa Democrat caucus face-plant is just the latest of a decades-long parade of software disasters. To make it explicit: I'm using this highly visible event as an occasion to illustrate a general, widespread problem; it has nothing to do with Democrats or politics.

The 2016 Caucus Software

Both Democrats and Republicans hold Caucuses in Iowa. Each party sets its own rules and changes those rules as it sees fit. They even do the voting a bit differently. Nonetheless, the parties cooperated to have a single piece of software that would service all their needs. They jointly announced this in the Jan 31, 2016 Des Moines Register:

They also had phone backup in case of software problems. The net result is that the Democrat Party chairman announced final results:

The software wasn’t perfect, of course – software never is. But it got the job done:

The 2020 Caucus Software

In the summer of 2019, the Iowa Democrats decided that, instead of upgrading the software that worked for them last time, they would pay for brand-new software to be built by a tiny, brand-new firm run by a non-programming History major who’d had a role in the tech of the 2016 campaign for Hillary Clinton. New firm. New software. Has to work out of the box, with over 1,700 people using it for the first time. No problem, right? That’s what the chair of the DNC proclaimed just hours before the caucuses opened.

Many people at the caucuses had trouble using the software, and in the end, it just didn’t work – it wasn’t able to load the results it captured from the people who were finally able to use it into the DNC reporting database. An analysis shows the kind of amateur-hour flaws you'd expect from the kind of group that built it.

Of course they forgot or ignored, like most people do, the train wreck of so many high-visibility new software unveilings, illustrated well by the ACA software releases of 2013 such as the one in Oregon I discuss here.

The Curse of Technology

Technology can be a wonderful thing. I’m grateful for the good things it delivers. But even relatively simple, physical technology can have massive problems. Did you know that 3,613,732 people have died in motor vehicle accidents in the US alone between 1899 and 2013? In 2016, 37,461 people died, as the carnage continues unabated. Just to put that number in context, the total number of US military deaths in all wars starting with the Revolution is well under half that number.

Does this surprise you? Here’s an example: during the entire period of the Vietnam war, the US had a total of 47,424 military deaths. During EACH year in the period 1966 to 1973, there were more US traffic deaths than the total for the war, with a high of 53,543 traffic deaths in 1969. You know which category of numbers people paid attention to.

The problems with computer and software technology massively dwarf those of physical technology, by huge factors.

Physical bank and stage coach robbers make for good visuals and press. One of the most famous was Willie Sutton, who spent decades in jail for having gotten about $2 million from all his hold-ups over many years. Let’s introduce computers and software to make things better! Sure. Banks lose over $10 Billion a year due to computer-driven credit card fraud alone! Banks vs. banking software? No contest.

Loads of people in the industry are promoting the latest software fad, for crypto-currency, like Bitcoin. It’s supposed to be super-secure. Sure. It turns out to be even worse, as the mounting software failures and losses demonstrate.

What all of this means is that once a piece of software does what it’s supposed to do, in production and at scale, no sensible person even thinks about replacing it with a whole new application. The reality is that most of our bedrock software systems took years to build, are often decades-old – and continue to work solidly and reliably.

An excellent example of this is the body of COBOL code produced by the small software company Paysys. When I was CTO of the company, in the late 1990’s, it ran over 150 million credit cards, including for Citibank, GE Capital and other name brands. The company was bought by the world’s largest credit card processor, First Data, and an incrementally evolved version of the same code now runs over 600 million cards, more than any other piece of software. The code is nearly 30 years old! It’s written in “obsolete” COBOL! There have multiple major attempts to re-create it using “modern” technology, sometimes with efforts taking years and costing tens of millions of dollars. Failures, every one.

The reality is that software is hard to build and get right, even today, after decades of time to figure it out. If we were as good at building bridges as we are at software, no one would dare drive over one. The big tech companies, for all their glittering reputations, are lousy software builders. When they try to build something new, they usually take years and end up failing – which is one of the reasons they acquire so many software companies, companies that in many cases have somehow managed to build software that they couldn’t. Here are details about Facebook’s ineptitude. If it's so hard for tech giants, chock full of super-bright nerds, why should anyone expect brand-new software written by a tiny group of amateurs to perform well the first time, with no testing or training, and in a national spotlight?

Building Software is different than building other technologies

People make decisions about software that, if it were any other technology, they would never make. In fact, they would consider those decisions to be beyond stupid.

Suppose you needed to get 1,700 people to an important public event from scattered places. While they didn’t all have to arrive at exactly the same, they had to be there within a couple hour window. Without fail. What would you do? You might consider asking a couple of established limo companies with fleets of reliable cars and experienced drivers to get it done. If a couple people insisted on driving themselves, you’d make sure their cars were OK and they had a backup plan. You might arrange for some car-pooling, or bring some people to a central spot and arrange for an experienced bus service with professional drivers. You might do any number of things.

Let’s further suppose you’d done something like this once, but when you did it again, you wanted to do some important things differently. Suppose you needed to make two trips on the same day. You would probably make some changes to what you did last time, while adjusting for whatever didn’t work perfectly before, right?

There are lots of things you wouldn’t even think of doing. Would you do any of the following?  Design, build and distribute brand-new vehicles for doing the transporting? Contract with a tiny, new company with no real track record to design the vehicles from scratch, not even modifying an existing model that works? Distribute the vehicles with no real test-rides? Give them to each of the 1,700 people to drive themselves? Given that the vehicles are new, not built according to standards, expect the new drivers to drive themselves with no training and no help? When the people have trouble driving, have no help available? When the vehicles break down, have no back-up plan to get the job done? When the disaster drags on for days, still be unable to fix it?

No, I don’t think any person would consider making these kinds of decisions with physical-world technology. The very thought would be ludicrous. But this is exactly what happened with the 2020 Iowa Caucus software! Endorsed and supported from the top of the DNC, and not questioned by any of the Iowa leaders!

Conclusion

There is a problem with how we think about and build software. A big problem. It’s nearly universal, and shared by normal people and supposed software experts. What happened at the 2020 Iowa Caucus technology is exceptional only in that it was so highly visible, which is why I use it as an example. The insane decision-making process that took place there takes place every day, from the way normal people and organizations work with software to the way tech giants and software so-called experts work.

The small groups of people who build software effectively and well are largely ignored by experienced managers and organizations. But these are the ones who are most likely to create amazing new bodies of software that change industries, before their software is absorbed and subsumed into the larger organizations who try to avoid breaking it too badly.

The mainstream thinking about software, both by managers and by software experts, is badly flawed. Things won't get better until fundamental things are changed.