The Black Liszt

Software programming languages: the Declarative Core of Functional Languages

In a prior post I described the long-standing impulse towards creating functional languages in software. Functional languages have been near the leading edge of software fashion since the beginning, while perpetually failing to enter the mainstream. There is nonetheless an insight at the core of functional languages which is highly valuable and probably has played a role in their continuing attractiveness to leading thinkers. When that insight is applied in the right situations in a good way it leads to tremendous practical and business value, and in fact defines a path of advancement for bodies of software written in traditional ways.

This is one of those highly abstract, abstruse issues that seems far removed from practical values. While the subject is indeed abstract and abstruse, the practical implications are far-reaching and result in huge software and business value when intelligently applied.

Declarative and Imperative

A computer program is imperative. It consists of a series of machine language instructions that are executed, one after the other, by the computer's CPU (central processing unit). Each Instruction performs some little action. It may move a piece of data, perform a calculation, compare two pieces of data, jump to another instruction, etc. You can easily imagine yourself doing it. Pick up what's in front of you, take a step forward, if the number over there is greater than zero, jump to this other location, etc. It's tedious! But every program you write in any language ends up being implemented by imperative instructions of this kind. It's how computers work. Period.

Computers operate on data.  Data is passive. Data can be created by a program, after which it's put someplace. The locations where data is held/stored are themselves passive; those locations are declared (named and described) as part of creating the imperative part of a computer program.

Data is WHAT you've got; instructions are HOW you act. WHAT is declarative; HOW is imperative. WHAT is like a map; HOW is like an ordered set of directions for getting from point A to point B on a map.

The push towards declarative

From early in the use of computers, some people saw the incredible detail involved in spelling out the set of exacting instructions required to get the computer to do something. A single bit in the wrong position can cause a computer program to fail, or worse, arrive at the wrong results. This detailed directions approach to programming is wired into how computers work. Is there a better way?

There is in fact no avoiding the imperative nature of the computer's CPU. As high level languages began to be invented that freed programmers from the tedious, error-prone  detail of programming in machine language, some people began to wonder if there were a way to write a low-level program, a program that of necessity would be imperative, that somehow enabled programs to be created in some higher level language that were declarative in nature.

Some people who were involved with early computers, nearly all with a strong background in math, proceeded to create the declarative class of programming languages, the most distinctive members of which are functional programming languages as I described in an earlier post.

The ongoing attempt to create functional languages and use them to solve the same problems for which imperative languages are used has proven to be a fruitless effort. But there are specific problem areas for which a declarative approach is well-suited and yields terrific, practical results -- so long as the declarative approach is implemented by a program written in an imperative language, creating a workbench style of system for the declarations. The current fashion of "low code" and "no code" environments is an attempt to move in that direction. But I'd like to note that there's nothing new in those movements; they're just new names for things that have been done for decades.

The Declarative approach wins: SQL

DBMS's are ubiquitous. By far the dominant language for DBMS is SQL. Data is defined in a relational DBMS by a declarative schema, defined in DDL (data definition language). Data is operated on by a few different kinds of statements such as SELECT and INSERT.

SELECT certainly sounds like a normal imperative keyword in any language, like COBOL's COMPUTE statement. But it's not. It's declarative. A SELECT statement defines WHAT data you want to select from a particular database, but says nothing about HOW to get it.

This is one of the cornerstones of value in a relational DBMS system. A SELECT statement can be complicated, referencing columns in multiple rows of various tables joined in various ways. The process of getting the selected data from the database can be tricky. Without query optimization, a key aspect of a DBMS, a query could take thousands of times longer than a modern DBMS that implements query optimization will take. Furthermore, table definitions can be altered and augmented, and so long as the data referenced in the query still exists, the SELECT statement will continue to do its job.

If all you're doing is grabbing a row from a table (like a record from an indexed file), SQL is nothing but a bunch of overhead, and you'd be better off with simple 3-GL Read statements. But the second things get complicated, your program will require many fewer lines of complex code while being easier to write and maintain if you have SQL at your disposal. A win for the declarative approach to data access, which is why, decades after it was created, SQL is in the mainstream.

The Declarative approach wins: Excel

I don't know many programmers who use Excel. Too bad for them; it's a really useful tool for many purposes, as its widespread continuing use makes clear.

Excel is a normal executable program written in an imperative language, but it implements a declarative approach to working with data. Studying Excel is a good way to understand and appreciate the paradigm.

An Excel worksheet is two dimensional matrix of values (cells). A cell can be empty or have a value of any kind (text, number, currency, etc.) entered. What's important in this context is that you can put a formula into a cell that defines its value. The formula can reference other cells, individually or by range, absolutely or relatively. A simple formula could be the sum of the values in the cells above the cell with the formula. It could be arbitrarily complex. It can have conditionals (if-then-else). Going beyond formulas, you can turn ranges of cells into a wide variety of graphs.

If you're not familiar with them, you should look at Pivot tables, and when you've absorbed them, move on to the optimization libraries that are built in, with even better ones available from third parties. Pivot tables enable you define complex summaries and extractions of ranges of cells. For example, I have an Excel worksheet in which I list each day of the year and the place where I am that day. A simple Pivot table gives me the total of days spent in each location for the year, something that simple formulas could not compute.

The key thing here is that even though there are computations, tests and complex operations taking place, it's all done declaratively. There is no ordering or flow of control. If your spreadsheet is simple, Excel updates sums (for example) when you enter or change a value in a cell. For more complex ones, you just click re-calc, and Excel figures out all the formula dependencies and evaluates them in the right order. This makes Excel a quicker way to get results than programming in any imperative language, assuming the problem you have fits the Excel paradigm.

The Declarative approach wins: React.js

One of the most widely-used frameworks for building UI's is React.js. Last time I looked, the header page of react.js included this:

It says right out that it's declarative! And that makes it easy! I have found a number of places (example, example) that have nice explanations of how it works and why it's good.

The Declarative approach wins: Compilers

The best computer-related course I took in college by far was a graduate course on the theory and construction of compilers. The approach I was taught all those decades ago remains the main method for compiler construction. First you have a lexical parser to turn the text of the program into a string of tokens. Then you have a grammar parser to turn the tokens into a structured graph of semantic objects. Then you have a code generator to turn the objects into an executable program.

The key insight is that each stage in this process is driven by a rich collection of meta-data. The meta-data contains all the details of the input language and its grammar and the output target. A good compiler is really a compiler-compiler, i.e., the imperative code that reads and acts on the lexical definitions, the grammar and the code generation tables. The beauty is that you write the compiler-compiler just once. If you want it to work on a new language, you give it the grammar of the language without changing any of the imperative code in the compiler-compiler! If you want to generate code for a new computer for a language you've already got, all you do is update the code generation tables! Once you've got such a compiler, you can write the compiler in its own language, at the start "boot-strapping" it.

While they didn't exist at the time I took the course, tools to perform these functions, YACC (Yet Another Compiler Compiler) and LEX, were built at Bell Labs by one of the groups of pioneers who created Unix and the C language. I didn't have those tools but used the concepts to build the FORTRAN compiler I wrote in 1973 while at my first post-college job. Since the only tool I had to use was assembler language, taking the meta-data, compiler-compiler approach saved me huge amounts and time and effort compared to hard-coding the compiler without meta-data.

This is meta-data at its best.

The Declarative spectrum

Excel and SQL are 100% all-in on the declarative approach. But it turns out that it doesn't have to be all one way or the other. If you're attacking an appropriate problem domain, you can start with just programming it imperatively, and then as you understand the domain, introduce an increasing amount of declaration into your program, by defining and using increasing amounts of meta-data. This is exactly what I have described as climbing up the tree of abstraction. Each piece of declarative meta-data you introduce reduces the size of the imperative program, and moves information that is likely to be changed or customized into bug-free, easily-changed meta-data.

Conclusion

Functional languages as a total alternative to imperative languages will perpetually be attractive to some programmers, particularly those of a math theory bent. Except in highly limited situations, it won't happen. No one is going to write an operating system in a functional language.

Nonetheless, the declarative approach with its emphasis on declaring facts, attributes and relationships is the powerful core of the functional approach, and can bring simplicity and power to otherwise impossibly complicated imperative programs.

The Revolutionary Champion/Challenger Method of Software QA

Having issues with software quality? Have you tried test-driven development? Do you have test scripts based on test requirements? Do you have a rich sandbox with good test data? What’s your code coverage? How often is a fully tested version released into production with problems that were missed? How often does the new feature work adequately in production but with destructive side effects that integration testing missed? How long and expensive is the pipeline from programmer’s release to working production? Are you proud of being on the forefront of development methods with Agile, Scrum and the rest but still avoid releasing what comes out of each Sprint to production because you can’t risk another disaster?

If any of these apply to you, you may want to consider a decades-old, widely proven in production method of software QA that enables frequent, error-free releases to production with almost no overhead or traditional QA work. It’s not taught, there are no certifications and It’s ignored by mainstream software experts. But it works.

I used to call it “comparison-based QA.” The term is appropriate because the core of the method is comparing results of the production version of the code with a test version. A CTO with a data science background suggested a better term, which I will use henceforth: champion/challenger software QA. It’s like when you’ve got a good model, the champion, and you want to see if a new model, the challenger, yields better results. Or it’s like A::B testing of a consumer UI, when you want to see if a proposed variation works better. In both cases, you’ve done something new and you want to do two things: (1) see if the new thing works like it should, and (2) make sure nothing that used to work is broken. Sounds like feature testing and regression testing, doesn’t it? Yup!

Nobody writes test requirements or feature tests for champion/challenger. You just feed the new thing the same data you fed the old thing and compare the results. You expect differences. If something is better and nothing is worse, you could have a winner – you test with a wider range of data, and if the results hold up, the challenger becomes the new champion and you move on.

Here are the main beauties of champion/challenger:

• The tests are nothing but inputs and saved outputs of real-world processing. Nothing artificial, no carefully crafted "test data."
• You create the code for comparison of output data once. The code/model could get endlessly complex, but the output comparison identifies differences whether it has 10KB or 10GB to compare. No test scripts!
  
  • You can do the comparison with a UI with some extra work, but again one-and-done.
• The comparison pulls out all the differences. You look to see if each difference is something you expected: if everything new you expected is there, and crucially if there are any unexpected differences. If there are, you’ve got a regression failure to fix.
• You can do the comparison in batch with large samples of old data, giving you great coverage.
  
• You can do the comparison live, in a production environment, using the challenger code/model output for comparison and sending only the champion’s output to the user.
  • You can test the changes you've made on the challenger code alone, looking at the challenger code's output, to see if you're happy with the change you made. While you're doing that, the challenger code will simultaneously be running production data, which the comparison will make sure is still OK.
    
  • After you’ve done enough live parallel production testing, you can be confident that the challenger will do everything the champion did – with real-world data in the production environment -- so you merely turn the switch so that challenger output becomes live and the old champion is retired.
  • You can do this in stages to crank the risk way down, so that the challenger becomes champion for just 10% of the inputs and then gradually turn up the fraction. This guarantees problem-free production releases.

I've written about about the details of how to get this done in a book.

I've written a post explaining the core concepts of the two main aspects of QA and how they're different and best served by different methods.

I've written about the path that led me to writing this and related books and summaries of the books.

I've written about how this method provides crucial fuel enabling startups to surpass tech groups hundreds of times their size along with specific strategies those groups use here and here.

The method is a "secret" -- because the vast majority of industry elites choose to ignore it. Want to succeed against industry giants? Use secret methods like champion/challenger to enable rapid release of quickly evolving code with zero errors in production and minimal QA effort. You'll run circles around the lumbering, stumbling giants and then leave them in your dust.

The IRS Could Have Prevented the Tax Data Leak

If the IRS had wanted to prevent the leak of tax returns recently reported by Propublica, they could have done it. The methods are simple, effective and in use. They just didn’t implement leak prevention methods. Why? The problem isn’t money; the IRS spends billions of dollars a year on computer systems. Will this embarrassment get them to fix things? I’ve read through the “IRS Integrated Modernization Business Plan,” the April 2019 document that describes how the IRS will spend many billions over the next 5 years to “modernize” their computer systems, and nowhere in the document is there a hint that they’ll do anything but spend more money to implement more of the ineffective security systems they already have.

The IRS doesn’t create or invent cybersecurity methods; they try to adhere to all the security regulations, follow the standards and take the advice of agencies that specialize in cybersecurity. These other agencies employ top experts who set the standards that institutions follow to protect their computer systems and confidential data. So what’s going on here? Did the IRS suffer the tax data leak because they failed to implement one of these clear standards? Or is there something missing or wrong with the standards that affects the IRS and all the other organizations that are guided by them? Let’s see.

Cybersecurity is a complex issue. I’ve used the metaphor of a gated community to explain general computer security; while the walls and gates of a gated community tend to be secure and well-maintained, the equivalent in the computer world is a patch-work of incompatible wall sections from different manufacturers which are never built properly and often need fixes to be applied, which the computer managers too often take months to apply if they do the work at all.

It’s possible that a hacker broke into the IRS. But what probably happened is that an IRS employee or contractor with legitimate access to IRS data decided to make a political statement by grabbing the files of ultra-wealthy Americans, smuggling them out of the agency and giving them to Propublica. This is known as an “insider” threat. Here’s the shocker: modern corporate and government cybersecurity standards and regulations fail to prevent or even detect insider threats!

Insiders stealing the data of the company or agency they work for has happened many times. The famous Edward Snowden case is a classic example of an insider stealing secret information and leaking it for publication. Snowden was a contractor who worked at the super-secret NSA (National Security Agency). He saw the surveillance of citizens that was being performed by the agency and didn’t think it was right, so he gathered lots computer files documenting the behavior and sent the files outside the agency for publication.

Snowden did electronically what Daniel Ellsberg did decades ago physically. Ellsberg was a military officer who had helped create reports describing in detail secret operations the US conducted during the Vietnam war. While working at the Top Secret RAND Corporation he gained access to a copy of the reports and walked out the door with them in his briefcase. He gave them to the press, where they were headlined as the Pentagon Papers.

The NSA has a positive reputation for cybersecurity. The cover story in Wired Magazine in June 2013 featured a description of a visit to NSA HQ in Fort Meade with its elaborate security measures. The strong impression given is that an organization that has so many strong walls, locks and cameras must be able to do the equivalent in the invisible world of computers. The timing of the cover story was perfect. Edward Snowden started leaking secret NSA documents in December 2012; the leaked documents were published shortly after the publication of the Wired Magazine issue praising the ultra-security of the NSA.

There are systemic issues that result in most of the successful hacks of governments and large companies which I describe here. What it comes down to is two main factors: the people in charge don’t understand the world of computers; the people in charge take a slow, regulatory approach to security, while the opposition is fast and creative.

For the IRS, the data loss is similar to books being taken from a library without being checked out, and can be fixed using electronic versions of methods that librarians use: check the books anyone walks out with!

Personal tax information is valuable, like the goods sold by high-end retailers. Think about jewelry stores; nearly anyone can go in the store, but all the valuable jewels are closely watched as they are taken out of display cases, tried on and put down. You don’t get away with slipping a diamond into your pocket and walking out of the store. Systems like this can be and have been implemented in the world of computers. I go into more detail here.

Going beyond basic monitoring of the behavior of computer users, it’s possible to translate methods that are in production today for catching credit card fraud to the problem of data leaks. Basically what you do is use machine learning to model everyone’s normal behavior concerning data access. When someone does something that is not normal for them, the model immediately notices and calls software to stop them and raise an alert.

In the case of the IRS the general behavior monitoring behavior could be refined, since IRS employees work on cases that have been assigned to them. The software would look at each file a user accesses and make sure that file is relevant to a case they’re working on; if not, the software would prevent access and raise an alarm. That way an errant employee who tried to pull Warren Buffet’s tax data who wasn’t specifically assigned to the case wouldn’t be allowed to do so. And the person working on Warren Buffet’s case wouldn’t be able to access Elon Musk’s case.

It’s less likely but possible that instead of the bad guy being an employee, it was a hacker who gained access to internal systems using methods similar to the ones that resulted in financial records of 147 million Americans being stolen from Equifax in 2017. I describe that hack here.

If the internal monitoring systems I have described were in place, it would also catch a person who had gotten into the IRS by hacking – the beauty of the method is that you don’t worry about who the actor is – you just worry about what they do, just like in a library or jewelry store.

The cybersecurity problem isn’t limited to giant government bureaucracies with outdated computer systems. It’s widespread, in part because they all follow experts, standards and regulations that ignore the insider threat. I analyzed in detail the various experts who were quoted in articles published by the New York Times about the Wannacry ransomware attacks based on software that had been leaked from the NSA. I found that the experts were simply wrong about the reasons, methods and responses to the attack.

It is ironic that the same government authorities who force everyone to follow ineffective regulations they craft by the ton are spending even more money training young people in their methods. My local community college was conducting training sponsored jointly by the NSA and DHS (the Department of Homeland Security); when I looked into it I found that the experts couldn’t even build functioning, secure websites with accurate information.

I sincerely hope that the ongoing flood of illegal leaks and ransomware attacks will end soon. But so long as the current batch of bureaucrats, regulators and experts are in charge of things, we’re likely to spend ever-increasing amounts of money on cybersecurity with ever-worsening results.

Note: this was originally published at Forbes.

Bitcoin Enables Criminals to Thrive While Promoters Deny the Facts

Many people believe, for good reason, that the cryptocurrency Bitcoin is widely used by criminals. The growing number of firms looking to profit from the use of Bitcoin as a legal investment don’t like being associated with crime. So they decided to form an organization and pay an ex-CIA director to lend his prestige and credibility to a report that distorts the truth and whitewashes the huge and ongoing use of Bitcoin and other cryptocurrencies as key parts of criminal enterprises.

The Crypto Council for Innovation

The IPO of crypto firm Coinbase at a valuation of about $100 billion shines a bright light on the main asset it manages, Bitcoin. Coinbase, along with other crypto firms and major financial firms such as Fidelity, have formed a trade group called Crypto Council for Innovation whose purpose is to promote the benefits and general wonders of crypto and to “encourage the responsible regulation of crypto in a way that unlocks potential and improves lives.” They also will be “…addressing misperceptions and misinformation…”

To that end, the group has sponsored a paper. Here’s their description:

In An Analysis of Bitcoin's Use in Illicit Finance, a study authored by Michael Morell, former Acting Director, Deputy Director and Director of Intelligence at the Central Intelligence Agency (CIA) examines the general assertion that the Bitcoin market is rife with illicit activity. Morell concludes that Bitcoin's use in illicit finance activity is limited and orders of magnitude lower than what has been cited by government officials. Morell's analysis also reveals that the blockchain ledger is a highly effective crime-fighting and intelligence-gathering tool.

The group is already meeting its goal of influencing public opinion. The headlines of articles include:

Embracing Bitcoin is now a matter of national security says former CIA director

Bitcoin is a ‘Boon for Surveillance’ says former CIA Director

 

How an Ex-CIA Director Proved Bitcoin Use in Crime is Declining

Former CIA Director Comes Out in Favor of Bitcoin

The pro-Bitcoin propaganda is working!

There’s the Report and then there are the Facts

Like all propaganda of its kind, the Morell report is all about starting with the conclusion you want – Bitcoin is great, criminals are fleeing from it! – and marshaling an impressive-sounding array of name-brand institutions and experts to say what you want. In this case, since the facts diverge from the desired conclusion so drastically, a good deal of work is required to reach that goal. Here are some of the major things the report did to whitewash the truth.

The report is self-serving

First and foremost, the people who wrote the report were bought and paid to do the job they did. Of course the report included words about how it was “objective.”

Think about it this way: suppose a governor of a state were accused of sexual harassment by multiple women and further accused of hiding actions leading to the deaths of thousands of seniors during the pandemic; how credible would a report be that was commissioned and paid for by that same governor? That’s what we have here.

The “experts” are nearly all anonymous

The report references experts from a wide variety of name-brand institutions who are said to support the report’s conclusions. This is an important subject. Don’t you think the report’s authors could get more than one such expert to go on the record? The one expert they got on the record has been retired for years.

Bitcoin technology is difficult for most people to understand

None of the report authors have any technical expertise. The ex-CIA man spent his early years on energy and East Asia, and then became a manager and communicator – which is what you would expect of someone who now spends lots of time working for media outlets like CBS.

The report shows little understanding of Bitcoin technology

The words used in the second conclusion are one example among many: “The blockchain ledger on which Bitcoin transactions are recorded…” The way this is worded shows a lack of the most basic knowledge of how Bitcoin works, implying that Bitcoin is somehow not part and parcel of what some call the “blockchain ledger,” a phrase made up to describe one of the inextricable parts of the Bitcoin code base. In the normal world, transactions like buying a hose at a hardware store take place; some of those real-world transactions may later be recorded in a ledger; in Bitcoin, there is no difference between the transaction, the Bitcoin and the ledger.

Why do criminals like Bitcoin?

The report claims that criminals are fleeing from Bitcoin. Let’s step back and see what it is about Bitcoin that criminals like. The reason why criminals like Bitcoin and other cryptocurrencies is simple: it’s easy for them to avoid getting caught! It’s even better than wearing a mask when robbing a bank! It’s as anonymous as cash except that it enables the transacting parties to be an ocean away from each other.

How does Bitcoin enable criminals to exchange money secretly?

It is true, as the whitewashing report claims, that the Bitcoin ledger contains a complete record of Bitcoin transactions and is open for viewing to anyone. A knowledgeable person can look at any current Bitcoin holding and trace it back to the prior owner, the owner before, and so on to the transaction that created the Bitcoin. It’s totally transparent!

There’s a little wrinkle, though, that’s the key to everything: the buyer and seller of each Bitcoin transaction are identified solely by the public side of the encryption keys controlled by the transactors. The physical-world identity of the sender and receiver of Bitcoin is not recorded in any way, shape or form! In terms of the Bitcoin ledger itself, everything is 100% anonymous.

The report mostly disputes claims few people make

In the introduction they talk about “…public statements from officials on both sides of the Atlantic who have suggested that Bitcoin is used primarily for illicit activities.” In fact the argument most often made is NOT about the fraction of Bitcoin used for criminal purposes, but about the fact that the nature of Bitcoin makes it DESIRABLE for criminals to use, and that criminals in fact make use of it. At no point do the authors argue against the fact of criminals’ preference for Bitcoin. The authors’ first major conclusion is “The broad generalizations about the use of Bitcoin in illicit finance are significantly overstated.”

The first major section of the report is “Bitcoin’s Use in Illicit Activity is Relatively Limited” The authors don’t deny that criminals like to use Bitcoin. They even admit that it is the currency most often found in Dark Net Markets, i.e., places where illegal substances and objects are bought and sold.

They argue that the fraction of Bitcoin activity performed by criminals is decreasing

As the speculative frenzy for Bitcoin buying and selling continues to grow, this is plausible, but irrelevant to the core observation that criminals like using Bitcoin and that it enables their activity.

Suppose a city suffers 100 murders per year. Then the population of the city greatly increases, but 100 people a year are still murdered. Is the fact that a decreasing fraction of the population is murdered every year something to be celebrated? Would you want to proclaim that your city’s murder rate is going down?

The report they rely on does not support their conclusion

The Chainalysis report they quote does not claim that criminal use is decreasing, but that overall use is growing: “One reason the percentage of criminal activity fell is because overall economic activity nearly tripled between 2019 and 2020”

The also report states: “However, as always, cryptocurrency remains appealing for criminals as well due primarily to its pseudonymous nature and the ease with which it allows users to send funds anywhere in the world instantly, despite its transparent and traceable design.”

The uncertainty of the Chainalysis data about criminal use is ignored

Because of the difficulty of identifying the participants in Bitcoin transactions outside of highly regulated exchanges that enforce standard bank KYC provisions, identifying which are criminal is mostly guesswork.

For example, the 2020 report identified 1.1% of 2019’s transactions to be criminal activity. In the latest 2021 report that number was revised to 2.1%, nearly double! They admit the same uncertainty about their numbers for 2020 saying “we should expect 2020’s reported criminal activity numbers to rise over time as well.”

The authors minimize the extent of the use of Bitcoin by criminals

The revised Chainalysis 2019 number shows “$21.4 billion worth of transfers” by criminals. This is no small amount! It is about the same as the worldwide total amount of money lost to credit card fraud by banks! Vastly more people use credit cards than hold Bitcoin, and the yet the total amount of crime is about the same!

The supposed use of blockchain analysis for fighting crime

The report’s second conclusion finishes with: “Put simply, blockchain analysis is a highly effective crime fighting and intelligence gathering tool.” If this is true, don’t you think the report would have included some juicy examples of crimes that had been foiled or intelligence that was gathered from Bitcoin? The authors fail to give a single example – not one of a crime that has been solved by using “blockchain analysis.” And not one example of intelligence that has been gathered. Do you think this might have something to do with the fact that exactly zero personal identity information is contained in the blockchain? You know, the reason criminals like it?

On the other hand...

Criminals are everywhere. Criminals like money. Criminals have been finding ways to exchange and launder money as long as there has been money. Fully regulated banks that enforce KYC (Know Your Customer) identification standards are still used for criminal purposes. The most recent report (from 2015) from the US Treasury estimates that about $300 billion is laundered every year in the United States. This is in spite of the massive AML (anti-money-laundering) regulations imposed on banks that cause them to produce a flood of required AML reports to the feds, who proceed to catch and stop only a small fraction of the crime. Only this year, after years of increasing regulations and costs with ongoing ineffectiveness, has the relevant agency started to take steps towards measuring effectiveness instead of just requiring “churning out more data that proves to be less than helpful” in actually catching the bad guys.

Conclusion

Bitcoin promoters are anxious to upgrade the reputation of their currency. The report they sponsored for that purpose marshals an impressive array of classic propaganda techniques to convey its misinformation.  Why not just state the facts?

The facts in this case are simple. Criminals make extensive use of our existing financial institutions. They manage to do so in spite of huge, costly efforts of banks, regulators and enforcement agencies, who end up catching only a small fraction of the crime. Criminals were early to jump on the Bitcoin bandwagon because of the anonymous, instant transactions it enabled. Criminals use Bitcoin today and are highly likely to continuing doing so, just as they continue to use existing banking mechanisms and largely escape capture. There is nothing about the “transparency” of Bitcoin that makes it easier to catch bad guys than existing systems.

I hope that the relevant organizations abandon the time-wasting report generation approach they’ve taken to finding financial crime in most areas other than credit card fraud and shift to a more entrepreneurial, results-oriented model with proper incentives to the participants. Here is the idea of the approach in general, and here’s an example of how it’s worked out in credit card fraud.

This was originally published in Forbes.

Gated Communities Help Us Understand the Ongoing Cyber-Security Disaster

There’s a war going on in our computers and networks. It’s a silent, invisible war. It’s fierce and continues to escalate.

The bad guys are winning. They are aggressive, hard-working, learning, inventing and focused on the goal of making money. The large army of the good guys is led by hapless, incompetent, unmotivated bureaucrats with meaningless certifications in this or that, consumed by building an audit trail showing that they’ve followed the ever-growing body of useless regulations so that when the nearly-inevitable security disaster happens, they can prove it wasn’t their fault. It’s clearly not a fair fight.

The security war isn’t like a war between nations. It’s more like a sprawling collection of gated communities infiltrated and attacked by myriad bands of criminal groups who break in, rob valuables and sometimes take hostages for ransom. The communities spend more money every year building walls that are higher and stronger and hiring ever more highly trained security people. Governments have multiple departments whose purpose is to stop the criminals directly and to help the communities better defend themselves.

Every year the money spent to prevent cyber-crime goes up, and every year the amount of illicit goods the criminals make off with goes up. The criminals are almost never caught. The problem is clearly not that the communities aren’t spending enough money. The problem is that the defenders don’t know much about computers and are going through the motions while the attackers are going for the gold, i.e. bitcoin.

Hardly anyone, including certified computer professionals with academic degrees, understands what goes on inside computers. They attempt to secure computers using ineffective methods that sound impressive but which they themselves don’t understand. They take great pains to do fancy-sounding things that sound impressive but make no difference.

I have explained the details of the massive hack of Equifax hacking by comparing Equifax security in the computer world to a car dealership’s security in the physical world. Translating invisible computer events to common sense physical things can help anyone understand what this cybersecurity war is all about. In this article I’ll attempt to explain what computer-style security would look like if it were applied to a gated community.

 

The Gated Community: Defenders and Attackers

The people who build the walls that protect the outer perimeter of the community are proud of their work. In some places they even have walls inside walls!

• If the walls were built like computer “walls” are built, you’d see that the walls are a patchwork of wall segments designed at different times by different vendors using very different materials and designs and are shipped with so many flaws that they need frequent upgrades. The people in charge of wall installation and maintenance rarely stay on top of the never-ending flow of walls patches and corrections and apply them haphazardly, if at all. The result is that all the savvy criminal has to do is jump on a vulnerability the second the manufacturer announces it and probe all the walls. It’s not hard – a large fraction of wall maintainers leave gaping holes unpatched for months or even years. Shazam! The criminal is inside the community.

The entrance to the gated community is a gate with a 24x7 guard checking the ID of everyone who enters against the list of people who are permitted to enter. The guard permits no exceptions.

• The people who live in gated communities want lots of people to come to their homes and do things for them. They call in and have the person added to the list. The service person comes to the gate, shows ID and is allowed to enter. Maybe they go to the house of the person who wanted them, but there’s no one to stop them going to other houses and doing whatever they feel like, just like a criminal who snuck through the flawed outer wall. Computer programs who “knock on the doors” of heavily guarded computers do the same thing, often with stolen ID’s.

The houses in the community are built securely, with locks on their doors and windows, so that even if a criminal manages to get inside the community, they can’t rob the house.

• In the computer world the houses and their exterior walls (servers, operating systems and applications) are built by the same kind of hodge-podge of vendors that build the exterior fences that protect the community. Houses are supplied by a variety of huge vendors using complicated methods and materials. It’s extremely rare that a house is installed without flaws – the builder will usually claim that it’s flawless, but then will come a stream of patches that need to be applied to the house with varying levels of urgency. A diligent clever criminal can go around probing houses for flaws that have yet to be discovered or repaired by the original manufacturer; a lazy criminal can just wait until the flaws are announced and probe specifically for the known flaws, confident that most homeowners won’t bother to apply the corrections.

People in the community want service workers to come to their homes when they’re out to do jobs like cleaning. It’s a huge convenience to have that guard at the gate checking ID to make sure only authorized people are let in. The guard can also loan the authorized person a spare key so they can enter the house and do the work they’ve been asked to do.

• In the computer world the criminal enters with a fake ID and gets a “key” which usually gives you permission to enter many “houses,” where you can do whatever you want – mess things up, steal things, etc. You can even change the “locks” and scramble things up (encrypt them) so badly that the house can’t be used. Imagine that the frying pan is stored under the hats in the coat closet and all the cooking knives scattered inside pieces of clothing in every room. How would you feel about cooking?

As in many high-class gated communities, mail service to the house is provided. Of course, only an authorized mail person is let in the gate in his truck that contains all the mail and packages to be delivered.

• In the computer world, homeowners aren’t careful about opening their mail, and sometimes packages they open contain invisible little criminal robots that immediately scurry out of the homeowner’s sight, pull out their cell phones and start communicating with criminal HQ. They run around the house and send out all the private information about the people who live there, including ID’s. They’re RAT’s (remote-access trojans) and result from what are called phishing attacks in email. Most homeowners aren’t able to resist opening infected emails of this kind. Sometimes the RAT’s make copies of themselves and send them to neighbors’ homes, spreading the problem.

Some security-minded people in the gated community protect themselves by installing cameras and other security devices so they’re alerted any time a person enters their house when they’re not there.

• In the computer world, tracking activities inside the house is extremely rare; detecting unusual activity and sending alerts when it’s detected is almost unheard-of. The criminal computer invader is free to take his time while sending copies of all the valuable information in the house of the gated community to criminal HQ. If the criminal feels like it, he can massively scramble the contents of the house to make it practically unusable, maybe even putting things in locked cabinets. Then he leaves a sign on the entrance door explaining to the homeowner what he’s done and demanding a ransom to return things to normal. When the ransom is paid, nearly always in untraceable bitcoin, the house often isn’t returned to normal after all. Why is anyone surprised?

The guard at the entrance gate makes sure that moving trucks that enter have been authorized by a homeowner, and that when they leave, the homeowner has approved the exit.

• In the computer world, watching what goes on inside the walls is rare. The equivalent of moving trucks can be created and endless streams of them can go through the exit of the guarded gate and no one checks.

Some homeowners are particularly concerned about the valuables inside their houses, and so they keep those valuables in an expensive, thick safe, secure from thieves.

• In the computer world, data you want to protect from being stolen is encrypted “at rest.” But just like you have to take jewelry out of the safe to wear it, you have to unencrypt data to use it. Therefore the programs in the house that make use of valuable data use a method to access the data that unencrypts it automatically after taking it from the “safe” and before giving it to the program that needs it. When the criminal software is in the house, it simply uses those same programs to access any and all data that it wants, loads massive valuable data into moving trucks and sends it out the unguarded, unwatched exit gates of the gated community. The owners frequently don’t find out about the theft until months later.

Conclusion

I’m sorry to say, things in the computer world are just as bad as I have described. Cyber-security experts and regulators do little but build up the mountains of pointless, ineffective procedures and regulations to ever-growing heights, seemingly without questioning the value of their efforts. Why should they? No one else does – their highly paid jobs are secure!

This was originally posted at Forbes.

T

The Colonial Pipeline Cyber-security Disaster in Context

There’s little new about the Colonial pipeline security disaster; nearly everything was business as usual: expensive but ineffective cyber-security systems and people; penetration and massive data stolen secretly; learning about the breach when ransomware popped up and said “pay me;” shutting everything down; taking days to recover; and the government, which is incapable of protecting itself, making solemn statements about “helping” more.

Colonial did manage to stand out from its fellow victims in a couple of ways: unlike many victims, they paid the big ransom; and their shutdown hurt millions of normal people in significant ways.

The other way Colonial stood out is remarkable: it was big news in the media for days, while the many “successful” ransomware attacks that take place each and every day on businesses, governments, schools and hospitals are rarely made public, much less make the news. Cars being unable to get gas in multiple states, even at inflated prices, may have had something to do with it...

Colonial Pipeline in Context

As usual in disasters of this kind, many important details of the attack and the response to it are closely-guarded secrets.

Let's step back and put what we do know about the disaster in context.

Ransomware has been around for a couple decades. It was usually sent to consumer emails with threats of various kinds if a ransom wasn’t paid. The ransom amount was usually under $1,000 and was paid to prepaid cards and other places. Because of the ability to trace the payment, the scam became less profitable and more dangerous for the criminals involved.

Then bitcoin came along. By 2013 exchanges appeared that enabled criminals to receive instant payments while remaining anonymous and untraceable. Why attack consumers for small amounts when you can get into a large institution and demand large amounts that you can receive instantly and anonymously anywhere on the globe? The threat evolved as well. The attacking software, after gaining entry into the target’s internal computer network, would encrypt all the data, making the organization’s computers nonfunctional until the data was unencrypted using a key known only to the criminals. The victim would be stalled in place, unable to function until the ransom was paid or the computers were otherwise restored. The use of ransomware exploded.

While the well-paid but ineffective defenders sloppily applied what they were taught in school and followed the thousands of pages of security-related regulations, the criminals evolved on multiple fronts. Ransomware evolved rapidly during 2020. The criminal attackers started to take a copy of the target’s data (exfiltrate it) before locking it up. The threat evolved to: if you don’t pay us we’ll make all your data public and you’ll be locked up.

Industry expert EMSISoft tells us: “As the year progressed, more and more groups started to exfiltrate data, using the threat of releasing the stolen information as additional leverage to extort payment. At the beginning of 2020, only the Maze group used this tactic. By the end of the year, at least 17 others had adopted it and were publishing stolen data on so-called leak sites.”

Oh, expose the data — how bad can that be? Pretty bad. “The data that was published included Protected Health Information (PHI), sensitive information related to school children, and police records related to ongoing investigations.”

What? These sound like highly regulated hospitals and government organizations, even law enforcement. Isn't the government, which creates and sometimes enforces all these cyber-security regulations able to protect itself? I guess not: “Unfortunately the barrage continued into 2020 with at least 2,254 US governments, healthcare facilities and schools being impacted. The impacted organizations included 113 federal, state and municipal governments and agencies, 560 healthcare facilities, 1,681 schools, colleges and universities.”

Colleges and universities? Aren't these the places that train all these cyber-security people and create the theory and practice they all learn and put into practice, with their fancy degrees? How is it possible that the security experts can't keep themselves secure? And just look at those numbers: more than four per day were hit and hurt!

On the other hand, it's just people's data getting exposed and ransoms being paid, right? Sadly it’s more than embarrassment: “The attacks caused significant, and sometimes life-threatening, disruption: ambulances carrying emergency patients had to be redirected, cancer treatments were delayed, lab test results were inaccessible, hospital employees were furloughed and 911 services were interrupted.”

More than an attack a day took place at healthcare facilities! Have you seen any headlines about that?

Alright, alright, this is all about governments and similar institutions. Commercial companies want to protect their profits and their reputations. They probably handle things much better — they must, because you almost never hear about them. Ummmm, maybe not; again from EMSISoft: “The private sector was hit hard too. Globally, more than 1,300 companies, many US-based, lost data including intellectual property and other sensitive information. Note, this is simply the number of companies which had data published on leak sites and takes no account of the companies which paid to prevent publication. Multiple companies in the US Defense Industrial Base sector also had data stolen, including a contractor which supports the Minuteman III nuclear missile program.”

Read that last sentence again, please. The bad guys are successful in stealing even from defense contractors. And the number above is the tip of the iceberg, because it's just the ones where someone was able to find their data for sale — it doesn't count all the ones who paid up, hushed it up, etc. Somebody did a survey to find out just how widespread the problem was in commercial businesses. Here's the bad news: “according to a study by security firm Sophos, 51 percent of all surveyed businesses were hit by ransomware in 2020.”

The iceberg is indeed huge. We're talking serious money given to criminals. From Pentest Magazine: “By the end of 2019, cybercriminals using ransomware had made off with a reported $11.5 billion in ransom payments. By the end of 2020, that number is projected to reach $20 billion.”

That's "just" the ransom money — much more money is spent recovering from the attack, even if the ransom is paid.

With all that bad stuff going on and the FBI and other agencies devoting huge resources to it, at least some of the bad guys are being caught and punished, right? No. According to EMSISoft “the effective enforcement rate for cybercrime in the US is estimated at only about 0.05%.”

In case you're not feeling math-y, let me help. This means that out of each 2,000 cybercrimes, only one is prosecuted.

Conclusion

The Colonial Pipeline event was extremely rare — not that it happened, since about half of all businesses get hit with ransomware every year — but because it made the news and was widely covered.

The reality is that, largely invisible to the public, there are gangs of criminals roving secretly and largely unchecked through our computer systems and networks stealing valuables and extorting money in huge volumes. Business and government spend increasing amounts of money with ever-growing staffs of highly educated, certified professionals to prevent the on-going pillaging. They are failing. Horribly. The vast majority of the "cures" that are batted about will definitely cause everyone involved to spend more money, and will equally certainly make little difference.

I have discussed the issues and illustrated the problems and solutions but it won't make a difference — all the power and prestige go, as usual, to people who are proven ponderous pontificators to whom the entire realm of software is invisible.

Note: This was originally published in Forbes.

 

Ingredients: Whole Foods Sneaks in Sugar

You really do have to read and understand the ingredients list of any food you buy. Even places like Whole Foods need to be carefully checked. Yes, the same Whole Foods that charges "whole paycheck" prices to supposedly only sell food that's good for you.

The Whole Foods brand Olives

Whole Foods is a grocery store with a huge selection of products produced by others. Whole Foods also has its own brand of products, some with standard packaging and some that are custom-packed. Someone I know went shopping there the other day and bought some Whole Foods olives, intrigued by what olives with "tangerine and chilis" would taste like.

After getting them home, my friend opened up the package and tried a couple of the olives. Yuck, what's this? These taste outrageously sweet! These are supposed to be flavored hot and with citrus! What's going on? At that point she took a closer look at the label:

Surprise! The third ingredient after olives and water is "cane sugar." More sugar than oil, more than vinegar, more than tangerines, more than red pepper and more than salt.

This being Whole Foods, the ingredient of salt isn't just any old salt, it's supposedly "sea salt," which sounds like it's better for you somehow. And the sugar isn't any old sugar, it's "cane sugar" -- oh, right from sugar cane, it's "natural," right?

Sugar is a Big Problem

Most people like sugar, which is part of why deserts can taste so good. The trouble is that people who make food only care that you like it and want to buy more of it; they mostly don't care whether the food is healthy for you to eat. Regardless of what they say. Therefore, sugar of one kind or another is added to an astounding range and variety of foods, including ones that are "supposed" to be sweet.

Even the CDC, the well-funded government organization whose whole purpose is to keep us healthy, agrees that sugar is a problem.

Weight gain and obesity, type 2 diabetes and heart disease. Is that all? In the section on obesity, the CDC admits that obesity itself is a serious problem:

This being the CDC, all they do is admit that obesity is "associated with" those fatal things and nowhere do they spell out the curse of sugar added to foods and the need to read the ingredients. The picture above shows obviously sweet cupcakes, while much of the problem is with foods that aren't obviously sweet.

Getting information about sugar

You may have noticed my use of phrases expressing cynicism about the CDC. Although the statements I've excerpted above are true, they have appeared on the CDC after decades of being studiously ignored by this corrupt agency. This is because the CDC has long practice in putting its authority and the patina of "science" behind what various powerful pressure groups want to advance their own interests.

Sadly, like with so many other things, it's up to each person to look out for him/her self. In this case, it's pretty simple: moderate your ingestion of sugar in any form; avoid buying and eating packaged food that has any form of sugar added to it; for this purpose, trust only the small-print list of ingredients.

Conclusion

You can't depend on the idea that a "health oriented" store will only sell you healthy food. You can't depend on how a food product is labelled. You have to take control of your own health, which means taking control of what you eat, which means taking care about the ingredients of the food you buy. It's not too hard, it doesn't take much extra effort, and the payback in terms of length and quality of life are more than worth it.

Software programming language evolution: Functional Languages

Once computers were invented and started being used, people discovered that writing programs for them was brutally hard. It didn’t take long before some smart folks figured out ways to make it easier, taking two giant steps forward in ease and productivity in quick succession.

After that wonderful start, people kept on inventing new languages, but somehow never made things better – in fact the new languages often made programming harder and less productive, a situation that the experts and exalted Professors studiously ignored. They never even bothered to spell out what makes one language better than another, which you'd think would be at the heart of promoting a new language.

I this post I’ll discuss a major category of new languages that keep getting created, receive passionate kudos, are rarely used but refuse to die: functional languages. In a subsequent post I'll describe the truly good idea that lurks inside the madness of the functional language world.

The core idea: time and timeless, music and math

I learned math as a kid. In high school I took the most accelerated and advanced courses available, finishing with a course in calculus, which I aced while mostly sitting in the back of the classroom teaching myself differential equations from my father’s grad school text book. I liked it. At the same time I got into music of multiple kinds but with a strong preference for classical. The strong connection between music and math has often been noted.

During my junior year of high school I took a course in computer programming, with FORTRAN as the language. Our “teacher,” scrambling to keep a chapter ahead of us in the text book, had arranged machine time for the class on Saturdays at a computer at a nearby rocket firm, Reaction Motors. After lots more programming, by my first year of college, it was clear that math took a distinct second place to both music and software in my heart.

In this post I explain where software comes from and its relation to music. The key concept is that, while you can read a page of music or software or math, math usually exists in a world “outside” of time – at most, time is a dimension like space; think for example of a proof in geometry. Music and software, by contrast, can only be understood as flowing through time. When you look at a page of music or software you naturally start at the beginning and hear/see it flow through time in your mind, just like reading a page of text.

What this means is that there is a fundamental dis-connect between math and computing. I explore this disconnect here. The math types are always trying to turn software into math, for example plodding for decades on a fruitless pursuit for ways to “prove” the correctness of programs.

This is the best way to understand both the failure and the refusal to admit defeat of the math types to develop a practical math-like way to write software, an effort and category of languages that is usually called “functional.”

The history of functional languages

Wikipedia's introduction to functional languages is accurate:

Turing had nothing to do with making software programming practical. That amazing job fell to others, who created the two giant advances in software language development. But since the math types were hanging around computers in the early days, particularly because doing math calculation was one of the earliest tasks to which computers were applied, finding a way to make programming more math-like, essentially eliminating the factor of time from software was a top priority.

I had already done serious programming, including working on a giant FORTRAN project to optimize oil refinery operation, when a friend introduced me to functional programming for the first time. The language was APL. It came a bit closer to being able to handle practical math than many of the other non-functional "functional languages" because of its focus on the matrix, but it was as easy to read and understand as advanced math. I could and can imagine that it would be appropriate for representing certain math transformations in a compact way, but why anyone who wasn't being tortured would agree to use it for general programming remains beyond me -- assuming of course that what you want is to ... radical idea here, be warned! ... get stuff done. As opposed to demonstrate and revel in your ideological and mathematical purity and exalted status in the pantheon of Computer Science greatness.

None of this is any concern to the cultists, who march on inventing an endless stream of "new" functional languages. Among the names you might see if you look into this are  OCaml, Scheme, Haskell, Clojure, Scala. There are even a few that are object-oriented on top of being functional, like Erlang. Articles continue to pop their heads up out of the underground insisting that functional languages really are making inroads in commercial applications, better for getting a job, more effective to use to create your ground-breaking start-up, etc. I predict no end to the flow.

The fact that functional languages are mostly of interest to a cult of fanatical core believers minimizes their widespread, on-going pernicious impact on software development, fueled by the near-universal math orientation of academic and corporate Computer Science. Functional concepts keep getting introduced by language fanatics into otherwise-sane normal languages.

Conclusion

Computer software is rarely driven by purely practical considerations much less objective knowledge and definition of what constitutes "good" in a language. But when an approach like functional programming is pushed that leads to results that are often 10X worse on multiple dimensions than "normal" programming, the problem is so large that people who aren't already committed members of the cult notice the difference and refuse to put up with the nonsense.

In spite of all this, there is an amazing valuable insight at the core of the functional language movement that leads to highly productive, real-world-valuable results when embodied in the right way. I will spell this insight out and illustrate its use in a subsequent post. The heart of the insight is taking a declarative approach instead of an imperative one when and to the extent that is appropriate.

 

Yelp is Big Tech Incompetent and Corrupt

Yelp is a Big Tech company that isn't listed when people talk about "big tech," but it's nonetheless all about tech and it's unarguably big, with well over 100 million unique visitors per month to their site and a valuation in the $ billions.

How good is their software? Are they a company from which non-tech corporate America should poach talent to upgrade their in-house tech? I recently had occasion to spend a little time with Yelp, and found that they are indeed an exemplar of the Big Tech tradition of tech bumbling as I have documented.

Harvard Business School famously teaches according to the case study method. Students are presented the case and challenged with figuring out what to do. Yelp would make an excellent case study. Any of the bright, aggressive b-school students who declared that the "solution" to the case was to fire all the executives and replace the entire product and tech team with talented high school grads, would get at least a B. Anyone who further suggested avoiding hiring anyone with a Computer Science degree would get an A.

Yelp and Local Business

Yelp is an advertising company that features listings of local businesses and user-generated reviews of those businesses.I recently visited my local dentist and received excellent care on the way to getting a crown put on one of my teeth. She knows I'm a computer guy and I've helped her out in the past, so she brought up her concern about a couple of malicious false reviews that were posted to her Yelp page. I'd said I'd check it out. I did. The results were illuminating. I sometimes complain about bumbling big corporate companies and their painful but fixable software problems. Yelp provides a great example of how second-tier Big Tech companies are no better at building software than the super-famous first tier ones are.

Yelp and a Local Dentist Business

I checked out Yelp's page for my dentist, Doctor Vu. It had a couple long negative reviews. So I posted a review of my experience, along with a selfie of me at her office.

I went back after a couple days to check out the page in more detail.

At the top of the page:

I see that she has 5 reviews with an average score of what looks like 3.5.

I scrolled down and found the photo section with the photo I had posted:

I took the selfie in her office holding the coaster I had given her last year that I couldn't resist buying at my favorite store in Cape May. She's had it in her waiting room ever since; I hear it often elicits amusement. It just seemed right to take the picture at the end of a visit to get a crown. :-)

After scrolling through ads and other information I got to the reviews. Mine was first:

I scrolled through the rest, including the two 1-star slams. Something didn't seem right. I went back and carefully counted the reviews. I double-checked. There were 6 reviews! The header at the top said 5! Then I made note of the star ratings. The two super-baddies were 1 star and the other 4 were 5 stars each. That's 22 points divided by 6 reviews = 3.67. Looking carefully at the points at the top, the dividing line is clearly at the top of the star, so Yelp thinks it's 3.5.

But then Yelp said there were only 5 reviews in spite of displaying 6.  Is it 22/5? Nope, that's 4.4. A complete mystery. Maybe I'll figure it out later.

At the end I found this:

What's this about? In faint type 16 other reviews? I check it out. At the top of the page that shows up there's a video I can watch and this long, serious-sounding thing about how Yelp cares and they're smart and they're doing the best for everyone:

Impressive! Finally I scroll down to check out these malicious reviews and here's the first thing I see:

Wow. Billions of data points! That software must be good, right? So I scroll down to see these scurrilous reviews. Imagine my surprise when I see this:

Wow. My review. Again!

At least the mystery of the rating average is solved. Even though my review was shown among the legit reviews -- first, no less -- it appears that their ludicrously bad software had marked it as not recommended and so some piece of software decided there were just 5 reviews. So 3 5-star plus 2 1-star is 17 stars, divided by 5 is 3.4. Close enough to the 3.5 average rating they displayed.

Even better -- I received a congratulatory email from Yelp after posting the review with the picture, saying how great it was and I should post more pictures along with my excellent reviews. My picture is indeed featured. And my review is both first on the list of legit reviews AND first on the list of those not recommended. Score! This is what you get by employing thousands of Silicon Valley's best super-programmers...

Those billions of data points must have somehow confused the Yelpers. And their software.

Finally I read through the rest of the reviews that weren't recommended. There was nothing fake about them.

I visited Dr. Vu again to get actually crowned -- the first visit was just preparation. She told me she had contacted Yelp about the situation. First she learned that the Yelp algorithms were state-of-the-art great, and in any case there was nothing a Yelp employee could do to alter their results. Second she learned that she really should pay Yelp lots of money so she could fix up her page, post pictures, etc. Maybe if she paid something could be done about those bad reviews...

My conclusion from this wasn't hard to figure. Yelp is corrupt (pay to play!). Yelp is incompetent (bad algorithms, double-posting, bad arithmetic). Sensible people should ignore Yelp.

Conclusion

I'm tempted to say that Yelp fits right in with its first tier Big Tech buddies, and of course it does. If you can stand it, check out the summary of their issues on Wikipedia. I did and also went to some sources to make sure it wasn't a Wiki hit-job. I also checked out the disastrous business and personal judgment and behavior of the leaders. Pathetic. Yelp is now on my blocked list of websites.

The Dimension of Software Automation Depth Examples

I have described the concept of automation depth, which goes through natural stages starting with the computer playing a completely supportive role to the person (the recorder stage) and ending with the robot stage in which the person plays a secondary role. I will illustrate these stages with a couple examples that illustrate the surprising pain and trouble of going from one stage to the next. Unlike the progression of software applications from custom through parameterized to workbench, customers tend to resist moving to the next stage of automation for various reasons including the fear of loss of control and power.

I will use companies to illustrate this progression that are known to me personally but not widely known. Software history generally is the history of the winners as written by the winners, like most history. However if you want to guide your own software strategy by taking advantage of the clear patterns of history, it is essential to include examples of companies that are rarely discussed.

Nextpage

NextPage was active from 1999 to around 2010. It had a distributed document management product that was prominent in service industries, for example lawyers and accountants. The product operated at a “recorder” level in terms of automation depth. It wasn’t directive, it just sat there waiting for a document request, and when it got one, responded with a list of applicable documents and lets you view the documents.

They and some of their larger customers saw that, in fact, many of their engagements weren’t so very different. As a high-powered, highly paid lawyer, you may think that every job is unique and your ability to handle the differences very important, but it was obvious to people involved that certain transactions involved very similar documents and workflows. It made sense to attempt to capture and automate the similarities.

With the cooperation of one of the world’s largest law firms, NextPage built a new product on top of the existing distributed document platform; this new product was automation at the “power tool” level. While recognizing that each transaction of a particular type would end up producing a unique set of documents, for example, it operated like a series of factory distribution belts, assuring that all lawyers working on an M&A transaction, for example, worked on the right things in the right order with the right base documents, reviewed in the right order by the right people, etc. etc. Lawyers continued to craft their documents. But instead of treating each (for example) M&A transaction as though it were the first one the firm had ever handled, the new system treated them like a repeatable process, with variations.

But the lawyers didn’t think of it that way at all! They were used to thinking of themselves as powerful, autonomous, self-directing agents. Now they were to be subservient to an assembly line in a factory, having their work and output measured as though they were hourly workers? No way – I’m a professional – there’s no way I’m going to put my head in that yoke!

NextPage had a good position in the market, existing customers, a prestigious lead customer for the product, a strong business case, and tangible results. But the resistance to moving to a “power tool” level of application was stronger than the force NextPage was able to bring to bear at that time, and the effort failed to bear fruit. Anyone who looks at the application knows it’s going to happen, the benefits are definitely there. But anyone with sense would be reluctant to predict even the decade when it would be likely to actually happen.

The pattern of resistance encountered by Nextpage was strongly correlated with the prestige and power of the people whose work would be affected. The same resistance is seen elsewhere; for example, how eager are the highly paid and prestigious category of medical doctors to have their work automated?

Captura/Concur

My VC firm invested in Captura, which is now part of SAP Concur, the leader in on-line expense management systems. These applications basically automate the process of filling out expense reports, getting them checked and approved, and finally paid.

If you think about expense reports, you imagine a form that you fill in with your expenses during a trip. You then attach your receipts, drop it off, and eventually get reimbursed for out-of-pocket expenses. So it seems like it would be a “recorder” type application. In fact several companies were started and eventually failed with products that weren’t much more than recorder type applications, with some automatic routing added on. This kind of application didn’t provide enough value to offset the expense of installing and running it.

Captura went much farther. While parts of this application operate at the “recorder” level, particularly the entry of un-automated expense information, much of it actually operates at the “robot” level, which is why the payback for its use is so high. It has built-in rules for approval levels and documentation requirements and many other things. It knows whether and when and to whom expense reports should be routed for approval from the HR system. It gets feeds from the corporate card system. It kicks out exceptions. It allocates expenses to the right accounts. It feeds directly into the accounts payable system to cut checks. When it needs something it can’t get from a computer, it asks a human for it, and otherwise does its job without help.

Unlike other “power tool” type applications, this application tended not to threaten people who were powerful in the adopting organizations. The most powerful people who used it tended to regard it as saving them time, and everyone got their reimbursements more quickly and accurately. The administration group saw it as saving thankless clerical work, and the accounting executives saw automatic enforcement of all their standards. Still, this application category didn’t really take off until the cost of implementation was reduced to a painless level; in other words, it had to wait for nearly universal access to the web to be practical and affordable. And it worked much better as a secure service than as an enterprise application, because the burden of installation and maintenance on IT added so much friction that only the most motivated potential customers would go for it.

ClickSoftware

ClickSoftware provides a classic example of a “robot” level application. It takes the problem of a field service operation, in which there are calls for service from companies in different locations, with different equipment to be repaired, with varying levels of urgency and service level agreements, with different hours of operation and times to travel to them, and finally with different service technicians who have varying skills and qualifications and constraints about hours of work. Who gets sent to which location to fix which problem in which order?

Without something like Click Software, this problem is solved in a rudimentary way by the service manager, who uses some combination of experience and common sense to work out the best solution. However, for a large service organization, the difference between a good solution and the optimal solution can be worth a great deal of money. Click Software takes all those inputs and produces the optimal solution, and asks the human manager for help when the problem simply can’t be solved; in response, for example, the human may ask a critical resource to work overtime, or might call a customer and see if their request can be deferred.

Click illustrates another characteristic common to robot-type applications. It is typical that “power tool” type applications do not become robot type ones by incremental addition. A robot application requires a whole new approach to the problem, and a level of math programming that is likely to be entirely foreign to the software group that wrote the power tool application.

Tape backup systems

In tape backup, whenever a new platform has come out, tape backup software products tend to emerge in the same order, starting with products that just record what the backup operator does, moving to power products that give the operator hints and suggestions and automate the repetitive operations, and ending with true robot products, which are driven by a set of rules and which request operator intervention at only and exactly those times when human intervention is required. For example, Palindrome and Cheyenne (now part of CA) introduced robot-level backup products in the early 1990’s for the PC server platform, at a time when similar products had been in general use on earlier platforms (e.g. IBM mainframe) for decades.

Having cycled from primitive to robot level automation several times as new platforms came out that had tape backup, the whole category is dying, frozen in time as the latest computer systems use the kind of disks that have become incredible cheap and tape has become obsolete.

Conclusion

It seems inevitable that an industry would move, step by step, towards robot-level automation. The historical fact is that while the incentives for advancement are always there, advancement happens only when an industry is "ready" to move, which can be decades after it is technically possible and economically practical.

Anthem Needs my Feedback and Reveals Deep Problems

I’m glad I have health insurance; I’ve had some health problems, now under control, that were expensive to fix. My insurance company, Anthem, has paid the claims.

Still, I can’t help being impressed by how many actions of this health insurance giant are stupid, wasteful, incompetent and worse. To be clear: I don't claim they're worse or better than the others; they just happen to be the one that I have.

I’ve just had a new stupidity inflicted on me by them that is low on the “it matters” scale, and high on the “a smart high school student could have done better” scale. What's interesting is that while the stupidity itself is minor in the overall scheme of things, it's the result of a serious dysfunction causing widespread trouble and hassle for patients and providers, while adding expense to Anthem. The extra painful thing is that, for anyone with a moderate amount of software knowledge, it could easily be fixed!

When I think about all the big, important, highly paid executives involved in and in charge of this, it does make me wonder whether we’d all be better off if they refused to hire any more people with college degrees for any job, and in particular, management.

The only reason this story is worth telling is that it's not about Anthem; it illustrates how  the techniques taught in business and computer science schools and embodied in endless standards and regulations keep large organizations locked into acting in ways that are both expensive and ineffective.

Summary

The root of the issue is my company negotiating a new plan for its employees. Anthem then sent an email (how modern!) welcoming me and inviting me to go digital with card images and an app. I suspect there were self-congratulatory executive meetings about the wonderful modernity of all this. See this for the inexcusably bumbling reality.

A few weeks ago I went to the dentist, got an exam and cleaning and then scheduled for a crown. My dentist submitted claims as usual and got denied, and I received denial EOB's in the paper mail. I sent my new digital card to the office, and that led to another denial and another paper EOB. Finally the office person was able to get through to a human being at Anthem and somehow resolved the issue.

Then I received a survey from one of Anthem's contracted firms to see how impressed I was by Anthem's new plan signup experience. The survey design and implementation resembled a 1995 paper-based process cluelessly "updated" to use, uhhhh, computers.

The whole mess could have been avoided by having a simple system that took incoming claims and checked them against a database of patients and plans, including plans that had been updated and/or replaced. When a claim arrived for a person with a no-longer-in-effect plan, a simple lookup would enable translating it to the new plan information and everything would take place seamlessly, with ZERO action, trouble or inconvenience to patient or provider. You wouldn't even need to send out a survey to ask how the transition went, because it would have been seamless.

An Email asking for Feedback

Some Anthem exec wanted to get some customer feedback about how their new plan roll-out was going. I got an email. Here’s the lead paragraph:

As you’ll guess from my past blog posts regarding Anthem (a summary with links is below), I clicked, hopeful that they’ve gotten moderately competent, but suspecting that some new sophomoric amateur-hour performance will ensue. (BTW, I’ve always been curious about the use of “sophomoric,” since it is the second year of a four year education program. Shouldn’t a performance that’s worse than “sophomoric” be called “freshmanic” or something?)

Sure enough, I clicked, my browser came up and showed me … a blank screen. Yup, nothing. Nada. I refreshed, tried again, same results. Then I thought, “remember this is Anthem we’re dealing with here; what would someone who dropped out of Computer Engineering 1.01 do? Sure, they’d test their stuff on their favorite browser and declare it working!” I strongly suspected that the kid’s (or highly paid seasoned professional with the skill of a kid who dropped out) browser was today’s most widely used one, Chrome. The next one is Safari, Apple’s browser on the Mac. Behind Safari’s share but still with substantial use are Firefox and Microsoft’s Edge. I copied the URL, brought up Edge and plugged it in. It worked! But it failed totally with my Firefox browser. I visit a large number of websites, and this is the first time in years that I’ve gone to a site that managed a complete face plant on Firefox. It takes a certain kind of perverse skill to pull off a feat like that!

With such a great start, who knows what joys would follow? Could this be a candidate for “freshmanic” status??

The Survey

The very first question puts this survey in the running for un-great-ness. Would you take a look at the never-before-seen way for formatting answers to a multiple choice question? With the major choices ending with “that”?

The second question continues the fun.

After asking me whether I remembered receiving this or that communication and answering “no,” the next question was always an huge graphic reproducing what they sent with the question (in effect) now do you remember? Not once -- several times.

Finally came a request to allow Anthem to follow up with me to help improve their experience. To see what would happen, I said yes.

I then got a form in which I was supposed to enter my name, telephone number, email, and best time to reach me. Right. You already know all this. Anthem gave you my email with an ID. All you need to do is give them my answers with the associated ID. That would make it easy for me. Instead, in typical brain-dead, big-corporate manner, you make me enter all the information you already know AGAIN. So NO, I’m not going to fill it in. So I hit Next. What do I get? You can guess, I bet. Yup!

Lots of red with the same questions in red backgrounds. Of course if this stupidity was supposedly all about protecting my privacy, they could have said something about that, apologizing for the inconvenience of entering information they already have. But no. Just ERROR. And I've already agreed to have my answers and identity shared with Anthem!

I could have just closed the browser, but they provided a button labelled “log off.” Which leads to a screen titled “Logged Off,” informing me of my “success” in logging off. But I haven’t “exited” the survey! They close with “Please close your browser to exit the survey.” OMG! If I don’t close the browser I haven’t “exited” the survey even though I’ve “logged off of this survey.” All these years in the business and I’m only just now hearing a new level of sophistication, about how “exiting” and “logging off” are different. Could I possibly screw things up here?

The Anthem Market Strategy and Insights team

I went back to the original email, and clicked on the link “To find out more about Anthem’s research surveys.”

I found out lots of interesting things like how “we’ll never ask for any personal data in our surveys, like social security or ID numbers.” I guess my name, phone, email and (on another question) age aren’t personal. Who knew?

I found out that the geniuses who put together this survey are just one of a crowd:

Always friendly, always available to answer your questions:

I could go on, but what’s the point. Anthem is incapable of doing the simplest kind of market research on their own, so they turned to a bevy of outsiders who can’t do it either, but Anthem can’t tell the difference between good and bad, so who cares? Except it takes a certain kind of genius to consistently pick the worst on multiple dimensions. I guess that’s why they have a “team” working on it – no mere “department” could do it on their own.

How Anthem could leap decades ahead and get to 2010

Anthem doesn't need to invent a thing. All they have to do is copy methods that are decades ahead of the ones they use -- and are quicker, cheaper and more effective to boot. Here are things they could do:

• Make the transition of employees to a new plan seamless.
  
  • The best thing would be to keep the plan number the same and put in a switch to adjudicate and give plan information according to the date of the switch-over. No new cards or numbers, digital or otherwise!
  • The next best thing would be to issue new numbers but keep a database of employees/patients enrolled in the prior plan, and automatically update things like newly arriving claims with the old numbers to the new ones as needed.
  • If you do one of these you won't need a survey!
• Dump the whole survey thing.
  • Track customer actions in detail like modern web companies do and detect when things don't go as they should. If someone isn't getting what they want, detect it and fix it, and give them an opportunity to tell you what's wrong at the time they experience it.
    
• Dump the whole paper thing.
  • I've opted out of paper every way I can at Anthem. I still get loads of paper mail. Why is it so hard?
• Fire your design department and start over.
  • I get lots of to-the-point communications from Amazon. Copy them!
  • Keep it simple. Test everything in small scale with real people before inflicting it on your customers.

Those are just highlights. There's lots more Anthem could do if their august team of highly experienced professionals would stoop to it. See the next section for some samples.

Past Anthem issues

Just to make it easy for anyone researching giant health insurance company outstanding achievements, here is the list of Anthem issues that I’ve looked into.

Again, I make no claim that Anthem is better or worse than any other insurance company. It’s just the one I experience, so theirs are the stories I tell.

Sometime in 2014 Anthem “lost” tens of millions of patient records. And completely botched telling their customers about it.

I discovered on my own that my data was in the stolen data. Anthem then offered a worse-than-useless plan to “help” me.

Somebody at Anthem decided that I would be cheaper to insure if I acted better and made an expensive botch job of offering a pre-paid card as an incentive.

At one point they decided to send me an email to get me to use a primary care physician they had selected for me. Everything about the experience was a nightmare. A case study in stupidity.

Then I discovered on an offer on Anthem’s website to enable me to pay my patient co-pays. Wonderful idea! Except it doesn’t work. Not just a big screw-up, a huge face-plant.

While I was discovering the joys of Anthem’s inability to help with co-pays, I got and responded to a customer survey. It was a model of how to p*ss customers off, not to mention being decades behind standard practice.

Most recently I discovered that my insurance ID changed and that they botched the change and the wonderful new app that was supposed to replace my card.

Conclusion

Big companies can't build software that works. They can't even do surveys. Hiring people from Big Tech companies doesn't help, because they can't do it either. The good news is, that gives LOTS of room for small, innovative groups that get stuff done that people need, building effective software that works quickly along the way. Hooray!

Experts are Super Smart

It's something I always knew, but I just couldn't make the nagging doubts that would pop up from time to time go away. Finally, at long last, the evidence has arrived: experts really are super smart and expert at what they say and do!

The evidence can be found in this article.

This is an extra relief to me personally, since I've expressed such strong support for the authority of experts in the past, giving examples of their prescient ability to analyze and predict with super-human accuracy.

I need worry no more. The evidence has arrived. Experts truly are everything I had hoped they were!

Ingredients and Truth

What's inside the food I'm eating or the substance I'm putting in my body? As I hope we all know, two of the most important factors in health and sickness are WHAT and HOW MUCH we ingest. You would think that paying attention to what we ingest and its varying impacts, good and bad, would be front and center in research and education. Yes, we're told to have a "healthy" diet. We're told some things about what that means. Sadly, a great deal of what we're told is wrong, and even worse, few people pay attention to the actual SUBSTANCE, i.e. the ingredients, of what we eat.

Even though lists of ingredients are printed on food packages in tiny print, how many people read them and make informed decisions based on what they read? Based on the incredible and growing fraction of people in our society who are overweight, I think it's fair to guess that the number of people who make such judgements for themselves is minuscule and, unlike their bodies, shrinking.

A Lesson about Reading the Ingredients by the Dentist

I went to the dentist last week for a periodic cleaning and exam. At the end the hygienist prepared the little plastic bag she gives patients containing a toothbrush, paste, and floss. They are freebies by the manufacturers hoping to win new customers. The hygienist added an extra box of a different toothpaste, saying that my gums bled quite a bit and I should try this toothpaste that reduces gum bleeding.

OK, I thought. I had a couple minutes waiting for the X-ray and for the dentist to come examine me, so I examined the box.

See the red streak on the bottom, drawing your attention to "Helps prevent bleeding gums?" My next thought was "I wonder how it pulls that off?" Yes, I know, I'm a nerd, but that really was my next thought. So I turned to the ingredients. Here they are:

My curiosity was immediately piqued. Just one active ingredient! And it's fluoride, the thing common to all toothpastes. 0.454%. Pretty exact amount. It's purpose is stated to be "Anticavity, Antigingivitus." Nothing about bleeding!

Let's look at the other box, the one with the "regular" toothpaste. Let's see what's different about the ingredients. Maybe there's a different amount of fluoride?

Wow. Exactly the same active ingredient, the same in every detail. But the purposes! The same animosity towards cavities and gum rot, but somehow in this toothpaste the fluoride has a new, added purpose: fighting hypersensitivity! I wonder if that's accomplished by putting the fluoride through extra training or something to give it a new purpose in life? What's that about?

Let's look at the front of the box:

Things are getting clear. The normal toothpaste is called Sensodyne, supposedly to attack sensitivity. With fluoride. Same active ingredient.

It's clear, number one, that our wonderful regulators don't care how products are described or sold, so long as the universally-ignored ingredients list in tiny print is accurate.

Oh, wait! Maybe I'm being too hasty! I'd better check the so-called "inactive" ingredients. Maybe one of them is the key thing that makes the difference.

Here we are for Sensodyne:

And the corresponding information for the terrific product that helps prevent bleeding:

A careful check shows that the two products have the same, the identical blankety-blank ingredients!

I'm a complete, total pain in the neck, I know, so I mentioned this politely as a question to my dentist and hygienist. They had no explanation. Oh, the hygienist said, it's what the salesperson said.

I've had many different dentists over the years as I've lived in different places. I've always gotten the little bag of freebies at the end. Has there been an uprising in the dental professional community about this decades-long lying and misrepresentation? If you've heard of it, please let me know.

Conclusion

The big companies making toothpaste and other things you put in your mouth depend on their ability to lie about the virtues of what they're selling and get away with it. They know through experience that no one pays attention to ingredients and that the relevant professionals ignore the issue if they're even aware of it.

The example I've given is trivial. It's just toothpaste and whatever you pick is unlikely to hurt you. But things that you put in your mouth and then swallow or breathe in are a whole different matter. The trouble is that exactly the same blatant lying goes on with food products as I've illustrated here with toothpaste. It would stop if the government made it stop. But it won't. It's the government, after all, the same government that continues to dish out misinformation about nutrition.

It will only stop when consumers rise up and act: start paying attention to ingredients and stop buying products with ingredients that are bad for them. Do schools teach this? I would say that schools spend 10X more time on sex education and political indoctrination than they do on teaching kids what ingredients mean and how to select and control what they eat -- except that implies that the amount time spent on that is greater than zero, which it probably isn't in most cases.

I will return to this health-essential issue in future posts.

The Dimension of Software Automation Depth

I have discussed the fundamental concept of automation.

Software automates human effort to varying degrees. In doing so, software emerges that performs this automation to an increasing extent. In this post I'll describe the basic stages through which automation progresses. In later posts I'll give specific examples.

Automation depth

Automation depth can be understood in terms of the extent to which human effort, observation, knowledge and control is replaced by the computer system.

Stages: help a human do something; do something a human would otherwise have to have done; make a decision a human would have made; gather information about all the work that is arriving, in process and completed; make work and resource allocation decisions. There are a few different  ways automation depth can be understood. Here are a couple of them:

• Stages: As depth increases, the human does things faster and has less work to do.
• Automation depth is correlated with moving from an “open loop” system to a “closed loop” system.
• Effort. In the earlier stages the computer augments or replaces the person’s efforts. This is like power steering in a car, in which everything is the same except turning the wheel takes less effort.
• Knowledge. As automation depth increases, less knowledge is in the heads of people and more in the computer.
  
• Control. As automation depth increases, the human loses control and eventually becomes controlled by the computer.

Automation depth has these basic levels:

Recorder	The software essentially tracks what people do and records the results of what they create or decide. The most primitive software of this class does this for a single job task. More advanced software does it for multiple tasks, eventually for a person’s whole job.
Power tool	The software takes high-level instructions from the operator, and does most of the work. In less skilled environments, the power tool will often select and order the work to be done.
Robot	Like an auto-pilot, the software replaces the human for most functions, performing those functions better than a human could, leaving the humans only to set goals and processing rules, and handle exceptions.

Unlike some of the software evolution progressions I've described, iIt is not generally a quick win to get to the next level of automation depth. An industry can stay stuck at a level of automation depth for decades. Normally, the people directly involved with an application strongly resist moving to the next level of automation depth, because they see it as deeply threatening to their autonomy, power or skills.

If a company tries to go to the next level, it is essential that it understand its position, and have the backing to last out the transition and the flexibility to keep trying until it establishes a beach-head from which it can expand. The benefits have got to be dramatic and tangible. Feel-good stuff tends not to work here. Some people are going to get bent out of shape, and managers of managers have got to impose the solution. The only way that typically happens is if things are arranged for their risk of failure to be low and the rewards great, and indisputable when they actually happen.

I will give examples of this progressions in future posts.

Micro-Services: the Forgotten History of Failures

Micro-services are one of today's leading software fashions; see this for more on software fashions in general. I have treated in depth the false claims of micro-services to make applications "scalable," and separately treated in depth the false claim that micro-services enhance programmer productivity.

Where the heck did such a crappy idea that has taken such a large mind-share in the software and management community come from?

I don't know. Who knows why skirts are short or long this year? Or whether shirts are tucked or not tucked? What I do know is that bogus fashion trends infect the software industry, some of them delivering decades of perniciousness. Looking at the spectrum of software fashions, I've noticed similarities among some of them. Some fashions well up, spread and peter out, only to recur again later with minor changes and a new name. The new name appears to be important, so that everyone is fooled into thinking the hot fashion is truly "new," not tarnished by the abject failures of its prior incarnations. I describe a clear example of an old idea with new name here.

The History of Micro-services

Why do we have micro-services? How did this bad idea get to be so popular? Was it someone's recent "great idea?" There are probably people who think it's their idea.

Do micro-services have a history? Of course they do! Nearly every major practice in software has a history of some kind. The number of software practices that are newly invented is truly tiny and going down rapidly. So why do we hear about software advances and the "new thing" so often? The whole field of software ignores its own history in truly radical ways. It even ignores what other software people in other fields and domains are doing.

Nonetheless, anyone who knows a broad range of software practices over a period of decades can easily recognize similarities and patterns -- patterns that surely constitute "history" whether or not the actors recognize the precedents of what they do.

What are Services?

Let's step back and understand what "micro-services" are. By the name, it's obvious that a micro-service is an itty-bitty "service." So what's a service? Once you peel back all the layers, a service is a plain old subroutine call with a bunch of fancy, time-consuming stuff stuck in between the code that calls and the code that is called. See this for my explanation of the fundamental concept of the subroutine.

Subroutines (or functions or methods or whatever) are an essential aspect of programming. Every programming language has its syntax, statements and other things that people get all wrapped up in. Every programming language also has an associated library -- as in function/subroutine library. If you look at a manual for a language and one for the associated library, the library is nearly always larger. The richness of a library is a key factor in the utility of a language. In modern terms, an associated framework serves essentially the same purpose; an example is the RAILS framework for Ruby, without which Ruby would just be one line on the ever-growing list of mostly-forgotten languages.

When you're writing a program, knowing and making great use of the associated subroutine library is essential. But what if you see that you're doing the same kind of thing over and over in your program? It makes sense to create a subroutine to perform that common function as part of the overall application you're building. As time goes on, most well-structured applications include a large portion of nested subroutines.

What if there's a program that can only run on another computer and you want to call it as a subroutine -- give it some data to work on and get the results back? This problem was addressed and solved in many variations decades ago. It's most often called an RPC -- Remote Procedure Call. To make a call to a distant subroutine you implement a version of it locally that looks and acts the same, but does some networking to send the fact of the call and the parameters to a cooperating routine on the distant computer. That routine gets the network call, retrieves the data, and makes a local call to the subroutine. When the return happens, the data is sent back by the same mechanism. It's just like writing a memo and putting in your out-box. The clerk who picks up the outgoing memos delivers the ones that are local, and puts the ones that aren't into an envelope, addresses it and puts the memo with return instructions into the mail. And so on. The calling code and the called code act like they normally do and the RPC makes it happen.

Along comes the internet. People notice that http is supported all over the place. Why not leverage it to implement the mechanics of the RPC? Voila! Now we have SOAP -- Simple Object Access Protocol, which uses http and XML to send and return the data. A subroutine with a SOAP interface was called a "service." This all got standardized roughly 20 years ago. Then along came a simpler version to use that had even more overhead called Restful, which is still in widespread use.

If you need to call a subroutine that can't or shouldn't run on your computer but on some other computer, having some form of RPC is extremely useful. The time penalty can easily amount to a factor of thousands or even millions compared to a normal local subroutine call, but you'll gladly pay the price if there's no other way.

Now let's recall what all this amounts to. A "service" is a subroutine that has a HUGE amount of overhead. There is no reason to ever take on the overhead unless, in the vast majority of cases, the service you want to call is on a different computer than the one doing the calling, a computer that can only be accessed by some kind of networking.

Prior Incarnations of Service Insanity

There have been a couple waves of service fashion mania. One of the waves took place roughly twenty years ago during the internet bubble. People were very concerned to be able to support growth in the use of their applications by unprecedented factors. Expert opinion rapidly agreed that building a "distributed application" was the way to accomplish this. The idea was that the load on the computer would greatly exceed the capacity of even the largest, most capable computer to handle it. To anticipate this, the application should be architected to be able to run on multiple computers that could be changed at will. Instead of a "central" application, the application would be "distributed." This was further elaborated by replacing the simple RPC mechanism with queues called "service buses," as in a bus to route large numbers of service calls from one place to another. The bus itself had to be robust so that messages weren't dropped, so it evolved into an "enterprise service bus," something which exists today. A huge, complex mechanism for accomplishing this became part of java, J2EE (Java 2 Enterprise Edition). See this for the complex, reincarnating history of the transaction monitor.

It turned out that building a "distributed application" was vastly more expensive and time-consuming than just building a plain old application, the kind that today is sneeringly dismissed as being "monolithic." The computational and elapsed time running cost of such an application was also huge. Hardly any applications had loads so large that they needed to be distributed. Even worse, the few applications that ended up with huge loads that required many computers found vastly simpler, more effective ways to get the job done. It's worth noting that articles with titles like "Why were we all fooled into thinking building distributed applications was a good idea" never appeared. The whole subject simply faded away. Here's a description from another angle of the distributed computing mania.

Another incarnation of this nutty idea took place largely inside large enterprises. Many of these places had highly diverse collections of applications running the business, accumulated by acquisition, multiple departments building applications for themselves, etc. Instead of doing the sensible thing of reducing the total number of applications by evolving the best versions to be able to handle more diverse tasks, the idea of a SOA, service-oriented architecture, became the focus. All these applications could be turned into services! Instead of building new applications, people would now build services. Tools were built to manage all these new services. There were directories. There was tracking and control. All sorts of new demands came latching onto the IT budget.

SOA never got to the fever pitch of distributed applications, but it was big. It went the same way -- fading as it turned out to be a lot of time and trouble with little benefit.

Now we come to the present. Remember that the original motivation of the RPC in any form is that a subroutine you want to call is ONLY running on some other computer. An RPC, whatever the form or cost, is better than nothing. Sensible then and sensible now. Then came ways of building programs so that their parts COULD BE run on different computers, if the computational requirements became huge. This also turned out to be rarely needed, and when it was needed, there were always better, faster, cheaper ways than a service approach. Now, with micro-services, we have reincarnated these proven-to-be-bad ideas, claiming that not only will micro-services effortlessly yield that wonderful virtue "scalability," but it will even make programmers more productive. Wrong and wrong.

Conclusion

I like the phrase "oldie but goodie." Sometimes it's applicable. In computing when a hot new tech trend comes along that "everyone" decides is the smart way to go, it rarely is. It most often is an "oldie but baddie" with a new name and new image. You would think that software people were so facts-oriented that they wouldn't be subject to this kind of emotionally-driven, be-part-of-the-group, get-with-the-program-man kind of thinking. But most people want to belong and want increased status. If believing in micro-services is the ticket to membership in the ranks of the software elite, a shocking (to me) number of people are all in.

Why Can't Big Companies Build Software that Works?

Years ago when I encountered big-company software flubs that were screamingly bad I would wonder about them: all those people, all that money, all those MBA-led processes and controls -- how could they screw up basic software things? What's wrong with them?

Now I know more. I know the flubs I saw weren't exceptions. They were and are the rule. I know that big corporate and government organizations are incapable of building reliable, cost-effective software. I have identified many of the contributing causes to this disturbing phenomenon, but there's much that isn't known. What's worse, all of the important people, leaders of academia, government and industry, ignore and/or deny this fact. 

Yet Another Inexcusable, Simple Stumble

I've covered many big organization face-plants in the past. In order to make it clear that the awfulness isn't just crippling software fails but encompasses a broad range of consumer-dissing inconvenience, I'll show some software that "works" but puts customer inconvenience front and center.

I got a bill from my gas utility provider. Utility company billing has been around for awhile, right? Treating your customers decently and getting paid should be high on the list of any company's priorities, right? How long have auto-pay and e-billing been around -- just flew out this year, didn't they? Bad joke. We all know that auto-pay of some kind has been around for decades, and so has some form of e-billing. So how long do you think it should take for a giant, lumbering company to modify their billing software so that auto-pay is handled reasonably well? Common sense says it should only take a couple months, but rich, giant company maybe a year? Two years? Let's say that by any reasonable measure it should have been completely nailed by 2010, over a decade ago.

Let's look at a bill I just got from my local gas utility. It was an e-bill -- hooray!

I read the above. I immediately think -- hey, I've got this account on auto-pay; what's this with Click here to make a payment?? I'm on auto-pay!

I keep reading and next see this:

Great. If -- If -- If I'm enrolled in Auto Pay? What??? You guys don't know if I'm enrolled in auto-pay?? Your software went to my account master record in your database. You extracted my name, billing address, e-mail address and current amount owed. It's hard to imagine that my auto-pay status isn't also there -- after all, you knew enough to send me this e-mail.

Couldn't you possibly, while you were there with your ancient gnarly claws on my account data, also have extracted my auto-pay status and congratulated me for being enrolled? Couldn't you have assured me that this e-mail was for my interest only and was being paid as usual?

Of course you could have. A trivial addition. Instead you make me wonder whether you have somehow dis-enrolled me as a sneaky way of tacking on a late charge? How can I find out? Then you give this:

Yeah, dial that number and wait around. Send an email. Sure. Or go to your generic website and try to pass the obstacle course to getting the information I need. That's what I did. I got through the obstacle course and verified that I'm in auto-pay and in fact owe nothing. Thanks for nothing, South Jersey Gas.

This is such an obvious thing. Maybe they're too small? It turns out that South Jersey Gas is a public company with $1.5 Billion in annual revenue and 1,100 employees. They serve 400,000 customers.

Maybe they're desperately searching for programmers? I looked at their job openings. No programmers. But there is a technical kind-of job that they're seriously looking to fill:

Yup, a complete b.s. job, whose only relevance is to produce reports and do things to make the bureaucrats in government agencies who wouldn't know a line of code if it bit them on the ear stay calm.

Here is the first sentence of the job description, which is followed by paragraphs of similar verbiage:

The role will be responsible for establishing appropriate Critical Access and SOD rule sets for different applications (Workday Financials and HCM, Oracle Customer Care & Billing (CC&B), Maximo and Hyperion), managing IT organizational policies and standards in support of legal and regulatory compliance needs, designing and testing general IT and organizational information security controls and interfacing with Internal Audit and business leaders to ensure that controls are designed and operating effectively. 

Got it? I looked through the rest of the details, and nowhere was "common sense" or "assure at least moderate levels of customer respect" mentioned or even hinted at.

Conclusion

The South Jersey Gas e-mail bill notice is a hardly-worth-mentioning issue given the massive fails that these organizations regularly commit. But when you consider that auto-pay is a good thing for customers and for the gas company and that about 5 minutes of real work is all it would take to fix the problem, and you think back on the important, well-paid job they're advertising for, you realize that the software problem is wired into the system and that everyone whose opinion matters thinks things are fine.

Maybe they should buy software instead of building it? They do buy software. They can't handle that either. See this for a great example with links to more.

 

Fundamental Concepts of Computing: Subroutine Calls

Subroutine calls are glossed over as yet another thing to learn when software is taught or described. The fact is that subroutine calls are one of the most amazing, powerful aspects of computer programming. They are the first step up the mountain of abstraction, the path towards creating software that solves problems with optimal efficiency, effectiveness and ease.

What's a subroutine? In common-sense terms, it's like a habit. I hope you have the habit of brushing your teeth. Calling a subroutine is like deciding you should brush your teeth. The subroutine itself is like the steps you take to brush your teeth: open your medicine chest, get out your toothbrush and toothpaste (probably always in the same order), etc. When you're done (you've rinsed your mouth and put stuff away for the next time) the subroutine is done and it "returns" to whatever called it -- what was I doing when I decided it was time to brush my teeth, you might think, but a computer subroutine always returns to exactly where it was before making the subroutine call.

In computer subroutines this calling behavior can get more elaborate. Subroutines can call (start) other subroutines, which can call yet others, without end. This is called "nested" calling.

The teeth cleaning subroutine ends with (we hope) clean teeth. Computer subroutines can end with (return) lots more. Suppose you're researching political revolutions and are making lots of notes. Then you decide it's important to study the 1917 Russian revolution. You put your general revolution notes to the side and dive into Russia. You start taking notes and then realize how important the Menshiviks and Bolshiviks are. You put the notes you've taken so far on 1917 on top of the general notes and dive into the next subject. When you've gone as deep down the rabbit hole as you can stand, you take the top notes from your stack on the side, and fill in the notes of what you've just learned. Eventually you pick up the next notes from the stack and so on until you're back to your revolution notes. It's also a bit similar to clicking on a web page, and then on that one and then again, and finally hitting the back arrow until you're back on the page where you started. Each new subject you dive into is like calling a subroutine, from which you can make further calls, eventually returning from each to the place you left off, only now with additional knowledge and information.

Just like in real life, when you call a subroutine you pass it information (parameters). The information you pass tells the subroutine exactly what you wish done. When the subroutine is done it can pass information back to the caller. You might pass dirty plates to the dishwasher subroutine, for example, which passes clean plates back when it returns. Or more elaborately you could pass a bunch of ingredients to the bake-a-cake routine and it would return a cake.

Some Technobabble

Computer software people aren't as clear as you'd like them to be with the names they use. The term I'm using here, "subroutine," is out of fashion these days, though still in use. In different software languages or programming communities a subroutine could be called a routine, subprogram, function, method or procedure.

Let's start with the basics of programming languages. Here's an overview of the bedrock on which everything is built, machine language, and the higher-level languages that have been built on that foundation. When you look at the memory of a computer what you see is data. It's all one's and zero's. But some of that data can be understood by the machine to be instructions. An instruction may include a reference to a memory location, an address. The instruction could ask that the data at the given address replace the value in a register, be added to it or other supported functions. There is always a "jump" instruction, which instructs the machine not to execute the next sequential instruction as it normally would, but to execute instead the instruction at the address in the instruction. There are test instructions that normally cause a jump if the specified condition is met.

The Subroutine call

Amongst the rich jumble of instruction types is one that stands out with amazing power: the subroutine call. A subroutine call, sometimes implemented directly in machine language and sometimes in multiple machine language instructions, does a couple things at once.

• It saves the address of the location after the subroutine call somewhere, usually on a stack.
• It puts some parameters someplace, often on a stack.
• It transfers control to the starting address of the subroutine.

The subroutine then:

• Executes its instructions, getting parameters from wherever they were passed to it.
• Returns to the address after the caller as stored on the stack
• Optionally gives a "return value" to the calling code.

Call by value vs. call by reference!

It's a bit esoteric, but there's an important distinction between two basic ways parameters can be passed to a subroutine: pass by value and pass by reference. Some languages can handle both kinds, while others can only handle one. This sounds abstract but it's important and easy to understand. Passing by value is like when you put the ingredients for a cake into a box, pass the box through an opening to another room, where a baker turns the ingredients into a cake and passes the cake (the value) back through the opening. Passing by reference is like when there's a larger kitchen and one of the people working there is a baker. You ask the baker to bake a cake, and the baker goes into the different storage places, gets the ingredients she needs and turns them into a cake on the spot.  In passing by value the subroutine only gets what you give it. In passing by reference, you tell the subroutine (by address pointers) what it should work with. In the peculiar world of object-oriented thinking, the equivalent of calling by reference is sacrilegious. But that's a subject for another time.

Subroutine Power

Subroutines are the first step towards reducing the size and complexity of applications, increasing their level of abstraction, and making them easier to change and enhance with minimal trouble and error. Subroutines are high on the list of fundamental concepts of computing.

The Progression Towards Abstraction on a Software Platform: Examples

I have described the way that once an application appears on a software platform, there is a consistent way the category of applications evolves on that platform. In this post I'll describe examples of this pattern. Briefly, the sequence starts with a prototype, and goes through these stages:

• Custom application
• Basic product
• Parameterized product
• Workbench product

There can be several levels of sophistication at each of these levels. The sequence is important because each step increases the speed and decreases the cost of making a body of code meet a set of customer needs.

Metasys (bought by Optum), Syntra (Clearcross)

These were medium-sized software companies, with revenues of tens of millions of dollars a year, which practically no one would have any reason to know existed today. Both these companies had a “custom application” that they were attempting to upgrade to a “basic product.” Both code bases evolved from large projects that were done on a consulting basis for a particular customer. Because both companies marketed what they had as though they were real products and failed to put the intellectual energy and money into properly upgrading their code bases, they were always coming up short in terms of features in sale situations, and when they were able to get sales, the implementation and customization efforts were harrowing to everyone concerned. The plain fact was, everyone was excited by the vision of how lovely it would be if their code were at the level of “basic product” and built a sales effort as though it were, but in the end, what each had was a “custom application” that had been severely hacked and “marketectured.”

Aurum

Aurum had a “basic product” for the emerging CRM market. At the time I looked at them with my VC firm, they were trying to bring it to the parameterized level. Because of the need for extensive customization in the CRM market, most of their implementations involved source code modifications, which made support and upgrading to new releases a major challenge. We initially chose not to invest because the company failed to recognize this. Later, with new management, the company at least recognized the issue, took steps to correct it and made business arrangements to minimize the impact on customers. We then tried to invest, but didn’t have the opportunity. The company went public and was a good win for its investors, but failed to get its product to the next level, and ended up selling to what at the time was one of the major ERP companies, Baan.

Paysys VisionPLUS

This credit card processing product was a “parameterized product,” and met all the relevant criteria. The company gained ground against the competition before and during the time I was CTO of the company because it fully met the criteria for “parameterized product,” while the competition was somewhere between that and “basic product.” Customers were able to make the product perform many more functions without source code changes due to this fact, an advantage the market appreciated. If the parameters that are made available correspond to the kind of changes you want to make, a parameterized product is ideal, and worlds better than a basic product.

The VisionPLUS product had its origin in two earlier products that were created by the company. The first of these products was CardPac, a fairly parameterized product for processing bank cards, like Visa and MasterCard. The product enjoyed a good deal of success at a time when most such card processing was performed by custom applications, which only the largest institutions could afford to write. Cardpac was sold to smaller banks, and parameters were introduced to reduce the cost of customization, maintenance and support.

A couple of retail institutions approached the company and asked it to modify Cardpac so that it could process retail (closed-loop) cards. After a couple of failed attempts, the company produced a completely new product for this purpose, Vision21. This product enjoyed considerable success among high-end retailers.

Finally, there was very large processor, Household International, that was running multiple copies of both products, separate because they had been customized for a variety of reasons, for example to support methods of credit that were unique to a market (for example “hire-purchase” in South Africa). While Paysys had failed to create a generic bank/retail product when confronted with the generic problem, unifying multiple bodies of related code into a single, highly parameterized code base proved to be a far more tractable problem, particularly with a single important customer who insisted that these variations were the only ones to worry about. The unified product was called VisionPLUS.

The industry quickly rallied to this new product that could be directed at so many different problems with such relative ease. While “parameterized product” may sound like an abstract concept, it translates directly into business advantage compared to more primitive product types, by enabling the product to be customized, installed, upgraded and maintained with less labor, less time and lower risk of error.

Paysys was bought by a major card processor, First Data, when First Data needed to expand into international markets with tough requirements. First Data's code base at the time was written in assembler language and was pretty much at the basic product level with minimal parameterization. Buying the Paysys COBOL code with its parameterized power was a major step forward for them. The number of accounts managed by VisionPLUS expanded by a factor of four under its new owner, to over 600 million.

Pivotal

Pivotal was early in the CRM market with a workbench-type product. Not all markets value this implementation method equally. CRM is one of the markets that values it most highly, because nearly every installation needs to undergo substantial customization. The benefits to customers were the ability to migrate their applications to new platforms as they emerged (e.g., Pivotal automatically converted Windows apps to browser apps) and the ability to extensively alter data, screens and application logic without source code changes. These characteristics took a good deal of effort on the company’s part to explain to customers. Their early efforts were much too technical, and didn’t resonate. Once they reached a level of market penetration, however, reference selling was the key. A potential customer would talk with an existing Pivotal customer who had first installed someone else’s application that seemed similar to Pivotal’s, which in fact it probably was, as it came out of the box. Then the reference would talk about all the time, effort and money to make simple-sounding customizations. Out would go the competing product, and in would come Pivotal. Suddenly customizations were fast, low-effort and low-cost. End of sales effort. Take order.

This implementation worked in a text-book manner for Pivotal, and is typical of the pattern. The company went public in 1999. However, its further growth was greatly limited by the fact that they chose to operate exclusively within the confines of the Microsoft Office product suite.

ERP products

The major ERP products have been at various stages of parameterized and workbench for a number of years. At one point, SAP seemed to be on the way to dominating the field, and armies of consultants were engaged in years-long projects to use SAP’s proprietary language, ABAP4, to modify and customize the base system to meet customer needs. Then along came PeopleSoft, originally just a vendor of HR software, with a complete ERP suite. While the software was not as richly functional out of the box as SAP’s, everyone knew by this point that no one used the software out of the box anyway – you had to wait for years while endless customization took place. What PeopleSoft had that was different was a set of development tools, PeopleTools, which was a generation ahead of SAP’s. SAP’s ABAP4 was at the time your basic 4GL, and you could program nearly anything with it, but you did have to spend a great deal of time programming. The PeopleSoft alternative, while no technical break-through, was still miles ahead of SAP in terms of ease of use and programmer productivity; they actually had a screen painter!

PeopleSoft convinced many buyers to care more about the implementation level of the product than the base functionality, and convinced those buyers that they were farther along the scale to a truly workbench product (although that term was not used) than the alternatives. As a result, they enjoyed years of outstanding growth. They were bought by Oracle for over $10 billion.

Conclusion

While not the only factor in success, companies whose products are higher up the tree of abstraction than their competitors enjoy a clear strategic advantage over their customers. As products evolve on a platform, the ones that appear or migrate to new levels of product abstraction tend to be more successful than ones at a lower level. While the discussion within a company is often about this or that customer or feature, raising the discussion to a new level of abstraction and acting on it can provide a huge competitive boost to a company's fortunes.

 

How Micro-Services Boost Programmer Productivity

There's a simple way to understand the impact of micro-services on programmer productivity: they make it worse. Much worse. How can that be?? Aren't monolithic architectures awful nightmares making applications unable to scale and causing a drain on programmer productivity? No. No. NO! Does this mean that every body of monolithic code is wonderful, supporting endless scaling and optimal programmer productivity? Of course not. Most of them have problems on many dimensions. But the always-wrenching transition to micro-services makes things worse in nearly all cases. Including reducing programmer productivity.

Micro-services for Productivity

Micro-services are often one of the first buzzwords appearing on the resumes of fashion-forward CTO's. They are one of today's leading software fashions; see this for more on software fashions. I have treated in depth the false claims of micro-services to make applications "scalable."

I recently discussed the issue with a couple CTO's using micro-services who admitted that their applications will never need to get up to hundreds of transactions a second, a tiny fraction of the capacity of modern cloud DBMS's. In each case, they have fallen back on the assertion that micro-services are great for programmer productivity, enabling their teams to move in fast, independent groups with minimal cross-team disruption.

The logic behind this assertion has a couple major aspects. The basic assertion that small teams, each concentrating on a single subject area, are more productive than large, amorphous teams is obviously correct. This is an old idea.  It has nothing to do with micro-services. It takes a fair amount of effort for members of a team to develop low-friction ways of working together, and it also takes time to understand a set of requirements and a body of existing code. Why not leverage that investment, keeping the team intact and working on the same or similar subject areas? Of course you should! No-brainer!

Here's the fulcrum point: given that it's good to have a small team "owning" a given set of functionality and code, what's the best way to accomplish this? The assertion of those supporting micro-services is that the best way is to break the code into separate, completely distinct pieces, and to make each piece a separate, independent executable, deployed in its own container, and interacting with other services using some kind of service bus, queuing mechanism or web API interface shared by the other services. The theory is that you can deploy as many copies of the executables as you want and change each one independently of the others, resulting in great scalability and team independence. In most cases, each micro-service even has its own data store for maximum independence.

I covered the bogus argument about scalability here. You can deploy all the copies of a monolith that you want to. Having separate DBMS's introduces an insane amount of extra work, since there's no way (with rare exceptions) each service database would be truly independent of the others, and sending around the data controlled by the other services at least triples the work. The original service has to get the data and not only store it locally, but send it to at least one other service, which then has to receive it and store it. That's 4X the work to build in the first place and 4X again every time you need to make changes. And sending things from one service to another is thousands of times slower than simply making a local call.

Now we're down to productivity. Surely having the group concentrate on its own body of code is a plus, isn't it? YES! It is! But how does having a separation of concerns in a large body of code somehow require that the code devoted to a particular subject be built and deployed as its own executable???

Let's look at a "monolithic" body of code. Getting into detail, it amounts to a hierarchy of directories containing files. Some of the files will have shared routines (classes or whatever) and others won't be shared. For most groups, going to micro-services means taking those directories and converting them to separate directories, each built and deployed by the team that owns it. Anyone who's tried this and/or knows a large body of code well knows that there's a range of independence, with some routines being clearly separable, some being clearly shared and others some of each.

A sensible person would look at this set of directories as a large group of routines organized into files and directories as it was built. They would see that as changes were made and things evolved, the code and the organization got messy. There were bits of logic that were scattered all over the place that should be put in one place. There were things that were variations on a theme that should be coded once, with the variations handled in parameters or metadata. There were good routines that ended up in the wrong place. The sensible person realizes that not only is this messy, but it makes things harder to find and change, and makes it more likely that something will break when you change it.

The sensible person who sees redundancy and sloppy organization wants to fix it. While you can go crazy about this, the phrase "paying down technical debt" is a reasonable one. For my take on this tricky subject, see this. Here's a simple way to understand the process and value, and here's a more far-reaching explanation of the general principle of Occamality.

The sensible person now has things organized well, with most of the redundancy squeezed out. Why is this important? Simple. What do you mostly do to code? Change it. When you look for the thing that needs changing, where would you like it to be? In ONE PLACE. Not many places in different variations. Concerns about basically the same thing should be in a single set of code. You can build it, deploy it and test it easily.

What's the additional value of taking related code and putting it in its own directory tree, with its own build and deployment? None! First, it's extra work to do it. Second, there are always relationships between the "separate" bodies of code -- that's why, when they're separate services, you've got enterprise service buses, cross-service calls, etc. Extra work! And HUGE amounts of performance overhead. Even worse, if you start out with a micro-services approach instead of converting to one, your separate teams will certainly confront similar problems and code solutions to them independently, creating exactly the kind of similar-but-different code cancer that sensible people try to avoid and/or eliminate!

Separate teams with separate executables also have extra trouble testing. No extra testing trouble you say? Because you do test-driven development and have nice automated tests for everything? I'm sorry to have to be the one to tell you, but if you really do all this obsolete stuff, your productivity is worse by at least a factor of two than if you used modern comparison-based testing. Not to mention that your quality as delivered is worse. See this and this.

Bottom line: separation of concerns in the code is a good thing. Among other things, it enables small groups to mostly work on just part of the code without being an expert in everything. Each group will largely be working on separate stuff, except when there are overlaps. All of this has nothing to do with separate deployment of the code blocks as micro-services.  Adding micro-services to the code clean-up is a LOT of extra work that furthermore requires MORE code changes, an added burden to testing and ZERO productivity benefit.

Conclusion

The claim that small teams working closely together on a sensible subset of a larger code base is a productivity-enhancing way to organize things is true. Old news. The claim that code related to a subject should be in one place also makes sense, kind of the way that pots and pans are kept in the kitchen where they're used instead of in bedroom closets. Genius! Going to the extreme of making believe that a single program should be broken into separate little independent programs that communicate with each other and that the teams and programs share nothing but burdensome communications methods is a productivity killer. It's as though instead of being rooms in a house, places for a family to cook, eat, sleep and relax each had its own building, requiring travel outside to get from one to the other. Anyone want to go back to the days of outhouses? That's what micro-services are, applied to all the rooms of a house.

Software Evolution: Functionality on a New Platform: Transaction Monitors

This is the fourth in a series of examples to illustrate the way that functionality that had been implemented on an older platform appears on a newer platform.

See this post for a general introduction with example and explanation of this peculiar pattern of software evolution. This earlier post contains an example in security services software, this earlier post describes an example in remote access software and this earlier post describes an example in market research.

Unlike the prior examples, this one is well known and I had no personal involvement.

Example: BEA systems

Old platform	IBM mainframe MVS
Old function	Transaction application execution enhancement: CICS
New platform	UNIX
New function	Essentially the same, with special support for UNIX-oriented applications, databases and UI’s
Outcome	Slow growth at the beginning and some market education required, but became virtually a standard, with just a couple competitors. It was acquired by Oracle in 2008 when it had about $1.5 Billion in revenue.

This is a classic example of the pattern of functionality emerging on a technology platform, and then emerging in pretty much the same way in the same order on other platforms.

IBM mainframes were originally used in a batch processing mode. They had operating systems that were really good at it. Groups of users increasingly wanted on-line access, and they wanted transaction processing control and security. The operating system didn’t provide it. So they built what the operating system thought of as an application, but which special applications that ran with it thought of as a specialized kind of operating system, a “transaction processing monitor.” Over time, the value of the main IBM TP monitor, CICS, became recognized, and it evolved to quasi-operating system status.

UNIX gets developed by a bunch of smart people at Bell Labs, and eventually becomes widely deployed on many kinds of computers. It was originally developed with interactive use in mind, and the early UNIX people would have been offended by the idea that it would need to be augmented by a primitive thing like a TP monitor. But that’s exactly what happened, and the UNIX-oriented TP monitors were built along very similar lines and for similar reasons as CICS!

The original authors of Unix ignored TP monitors when they built it in the 1970's, but AT&T had its own version of a TP monitor that ran on IBM mainframes to control its network. AT&T decided it would be cheaper to use their own computers instead of IBM mainframes to run its control software. Their computers had the Unix operating system, but transaction wouldn't work properly without a transaction monitor. So they set about building their own specifically to run on top of Unix, Tuxedo starting in 1983. It was successful and became widely used.

The founders of BEA were ex-Sun Microsystems employees who were amazingly prescient in seeing this need. But they didn't actually build anything! They started by buying Tuxedo, the Unix-centric TP monitor that had been owned by Novell since 1993. They played with other middleware companies but most importantly bought WebLogic, a TP monitor built specifically for Sun's Java language.

It was the availability of Unix-based TP monitors that helped enable the massive growth of the Unix platform during the internet explosion in the late '90's and early 2000's. I was active during this period, both with CICS applications and internet applications. The whole internet tech world, techies and investors, became convinced that certain things were required to build an application with that all-important characteristic, "scalability." It had to run on Sun computers, with the Sun version of Unix. It had to use the Oracle DBMS. And it had to be written in the Java language using the J2EE (usually spoken as jay-two-double-E) library, intended to ride on ... a TP monitor! Oracle ended up being the winner; it first acquired BEA in 2008 and shortly after acquired Sun.

The key, just as it was for CICS, was the integrity of transactions when many people are accessing a shared set of data. UNIX simply did not provide this capability, and just like on the mainframe, the capability was added on top of the operating system, with an application that the operating system thought of as a normal application, but which the applications that made use of it thought of as a specialized kind of operating system.

You might well think that this whole set of functionality is dead today. Who talks about TP monitors? Many companies showed that you don't need all the complexity and overhead of a TP monitor to build scalable applications.

In reality, it's a clear example of another strong, repeating pattern: bad ideas in software rarely die; they just fade for a bit and then re-emerge with a new name in slightly different form. Here's a good example of the morphing of data warehouse into Big Data.

I won't go into detail here, but while J2EE and classic Unix TP monitors are on life-support, the same idea of distributed applications for scalability lives on today in the form of microservices woven together with an enterprise message bus. Even though, just like 20 years ago, they're not needed for building scalable applications!

Why Can't Big Companies Build or even Buy Software that Works?

Many large companies depend on software. They often have staffs of thousands of people using the best methods and project management techniques, supported by professional HR departments and run by serious, experienced software management professionals. They can afford to pay up so that they get the best people. Why is it, then, that after all these years, they still can't build software that works?

Some of these giants recognize that they can't build software. So they buy it instead! Surely with careful judging and the ability to pay for the best, they can at least slap their logo on top-grade software, right? Sadly, the facts lead us to respond ... not right.

What company doesn't want to be part of the digital revolution and have an app? If you're a major health insurance company, why wouldn't you replace old-fashioned insurance cards with something always up-to-date that comes on an app?

As an Anthem customer, I can see that they've gotten with the program. I got this email from them:

An app, huh? Why is it called Sydney? First, let's keep it simple. They say I can now download a digital version of my ID card, so let's try that first.

I clicked on the link, which brought me to the main Anthem login. I logged in. What I expected was normal website behavior, a deep link to the right page, but  having to login before getting there. This "exotic" technique, standard practice for over a decade with websites that care about their users, was beyond the professionals at Anthem. After logging in, I got to my profile. Where's my digital card?? I guess it's one of their intelligence and mental status tests, where they count the clicks and the time it takes for you to get where you're going.

Hoping to succeed, I scrolled down in the Profile section and hit gold. I saw this:

That wasn't too hard! Mobile ID cards! Let's see.

Nothing about seeing it, printing it or emailing it. Just an option to turn off getting a physical card in the mail, and a casual mention (with no link, of course) to "our Engage mobile app." What happened to Sydney??

I thought I had gotten through the usual Anthem obstacle course in record time. Nope. Dead End. There are a lot of people these days screaming about how bad disinformation is and how it needs to be stopped. Hey, guys, over here....!

Back to the home page. Look at all the menus. Check all the drop-down lists. Under "My Plans" there's something called "ID Cards." Bingo! An image of our cards, front and back, with options to print, email, etc. as promised!

Nothing about an app, Engage, Sydney or anything else.

Alright, Anthem, I've had enough of your website. Let me go to the Play Store and check out Sydney. Here's what they say it is:

Sounds pretty good, right? What can it do? Let's see:

Seems like it can do HUGE amounts of stuff!  Let's keep going.

OK, I've got it. Maybe "Engage" is something Anthem's own army of programmers built. Maybe it was crap and management decided to buy some best-of-breed software. Makes sense. Perhaps some of the hundreds of programmers no longer working on Engage can be assigned to update the website and make it kinda sorta accurate and usable, you think?

No doubt Anthem management exercised great care to assure that CareMarket did a great job and was giving them a proven app that customers loved so that when it went out named Sydney, Anthem's reputation would go up. Let's see the reviews:

Over 2,600 reviews. That line by the "1" rating is pretty darn long. Looks longer than 2 to 5 added up. I guess Anthem had trouble threatening enough of their employees into giving 5 star reviews to get the job done, right?

Let's sample a couple of reviews. Here's the top one when I looked:

"This is the worst app I've ever encountered." Error messages. Failed searches. There's a response from the vendor:

Hey guys, she already gave you "a brief description." Do you test your software? Give it to normal people to try before inflicting it on your innocent, unsuspecting customers? Skimming down, I see that pretty much the same response is given to every each tale of woe. Pathetic.

Here's a sample of other reviews:

Get the general drift...?

This app has been downloaded 500,000 times!! The pain and frustration Anthem is causing is hard to fathom. Why is anyone at Anthem involved with Sydney still employed there? Silly question. Did anyone lose their job after the giant hack at Anthem and the catastrophically bad response to it that I've described?

Maybe they should hire people from the big tech companies to do stuff like this. Those people really know how to build great software! Uhhhh, not so much. Here is specifically about Facebook's app. For more see this and this and this.

This big-company software effort is bad beyond belief. I can't comprehend how it is that they pay people big bucks and come out with stuff like this. From what I can tell, though, governments are in close competition for the "prize" of doing the worst job of building and managing software. It's like there's a competition. See this and this.

The whole world is up in arms about the pandemic. Big powerful people and organizations are taking it seriously and making changes with the intention of fixing the problem. When it comes to the software pandemic, however, everyone just whistles and waltzes along like there's no problem. Everyone just expects and accepts awfulness, acting like it's just how life is.

It doesn't have to be this way.

Software Programming Language Evolution: Credit Card Software Examples 2

The vast majority of production software applications undergo a process of continuous modification to meet the evolving needs of the business. Sometimes organizations get fed up with the time and cost of modifying an application decide to replace it entirely. The replacement is most often a brand-new application.

When this happens, nearly everyone agrees that a different programming language should be used for building the new version. After all, everyone knows that programming languages have advanced tremendously since the about-to-be-replaced application was written, so it just makes sense to pick a modern language and get the job done. It occurs to exactly no one that advances in science and even writing novels are regularly achieved by using the same languages that are already in wide use. See this for details.

In a prior post I described a couple of such huge efforts in the credit card industry to take advantage of the latest software technology, specifically 4-GL's and Java. The results didn't make headlines, but news of both multi-year, multi-tens-of-million-dollar disasters spread through industry insiders, of which I was a member at the time.

At around the same time two major card processing companies faced the same challenges and made very different decisions about how to go forward. They did NOT choose the latest-and-greatest programming languages, but went with choices most people in technology thought were obsolete. The result? In sharp contrast with the cool kids who went with powerful modern languages, both projects succeeded. Here are the stories.

Total Systems Services (TSYS) and moving to COBOL

TSYS began inside a little bank in Columbus Georgia as a group writing software to process credit cards in 1959. In 1974 it began processing cards for other banks, went public in the 1980’s and by the 1990’s was a major force in card processing services. Partly due to its early origins and some highly efficient early programmers, its processing software was largely written in IBM assembler code – in the 1990’s!

Executives at the bank decided to modernize the software. I have no inside information on the reasons for their decision. It could have been as simple as a desire to have their software not tied to a particular machine. In any case they authorized a major, multi-year project to rewrite what had become a huge body of production software. Talk about risky! One of the amazing things is that they decided to close their ears to the near-universal acclaim being given to modern software languages and methods and take the relatively low-risk path of re-writing the entire body of assembler language into … COBOL – the very language that was derided and that others were going to great lengths to get out of!!

TSYS put a big team on the job, took a couple years to get it done, and around the end of the 1990’s moved all their production from their old body of IBM assembler code to the new COBOL system with no disruptions in service. An impressive achievement, to put it mildly. The fact that they knew the requirements because of their existing working system written in assembler language played a big role. But the two failed efforts I describe here had the same advantage! The point here is that the 3-GL COBOL is HUGELY more productive than 2-GL assembler language, so the rewrite made sense, in sharp contrast to the failed efforts.

Paysys and COBOL

When I joined Paysys in the mid-1990’s it had two major products: CardPac (processing for credit cards issued by banks) and Vision21 (processing for credit cards issued by retailers, supporting multiple, extensive credit plans on a single account). The company created a first unification of the two products into what became the industry’s leading systems for processing cards, VisionPLUS. The project was completed and put into production while I was there. There were over 5 million lines of COBOL code in the final product.

The COBOL code was unique in being able to handle an unprecedented range of requirements, including a large number of installations outside the US for Citibank, Amex and GE Capital, and in Japan. It handled over 150 million cards across multiple installations at that time.

The head of First Data decided to buy the company at the end of the year 2000, mostly because his existing code base, written in assembler language, couldn’t be made to meet international requirements. The COBOL code First Data bought is now the core of their processing, handling over 600 million cards, far more than any other body of software. Migrating from assembler language to a proven body of COBOL code was a big winner. Twenty years later the code continues to be a winner as its growth by hundreds of millions of accounts demonstrates.

Conclusion

Why should anyone care about this ancient history? Easy: to this day, status in software is conferred by being involved with cool new languages that are oh-so-much-better than prior ones. For example if you're involved in super-cool blockchain, no one with a brain in their head would even suggest using an existing language to implement what are called smart contracts. You need something new and "safe." Sure. The fact is, there have been no significant advances in programming languages in the last fifty years. Nothing but rhetoric and baseless claims.

Software Programming Language Evolution: 4GL’s and more

Not long after third-generation computer languages (3-GL’s) got established, ever-creative software types started inventing the next generation. In a prior post, I’ve covered two amazing programming environments that were truly an advance. They were both widely used in multiple variations, and programs written using them continue to perform important functions today – for example powering more hospitals than any competing system. But they were pretty much stand-alone unicorns; the academic community ignored them entirely and nearly all the leading figures, experts and trend-setters in software ignored them and looked elsewhere.

Experts “in the know” directed their attention to what came to be called fourth-generation languages (4-GL’s) and object-oriented (O-O) 3-GL’s. These were supposed to be the future of software. Let’s see what happened with 4-GL's.

The background of 4-GL’s

The earlier posts in this series give background that is helpful to understand the following discussion.

In prior posts I’ve given an overview of the advances in programming languages, described in detail the major advances and defined just what is meant by “high” in the phrase high-level language. I’ve described two true advances beyond 3-GL’s. And then there were 4-GL’s, supposedly a whole generation beyond the 3-GL’s. Let’s take a look at them.

The best way to understand 4-GL’s is to look at the context in which they were invented. First, the academic types were busy at work creating languages that essentially ignored how data got into and out of the program. The first of these was Algol, followed by others. The academic community got all excited by this class of languages, but they were ignored by the large community of programmers who had to get things done with computers. That was in the background. In the foreground, modern DBMS’s were invented and commercialized.

4-GL's!

Apparently everywhere new languages sprang to life, created inside, around and on top of DBMS's. It's a revolution, a once-in-a-lifetime opportunity to become a major milestone in software history! My name could be right up there with von Neumann and Turing!

All the major DBMS vendors created their own languages, usually with snappy names like Informix 4GL and Oracle's PL/SQL. How could they fail to respond to this massive opportunity for market expansion?

Brand-new vendors popped up to take advantage of the hunger for DBMS along with the new hardware configuration of client-server computing, in which an application ran on a group of Microsoft Windows PC's, all connected with and sharing a DBMS running on a server. One startup that powered to great commercial success was a company called PowerSoft which created a product called Powerbuilder. The Powerbuilder development environment enabled you to work directly with a DBMS schema and create a program that would interact with a user and the data. The central feature of the system was an interactive component called a DataWindow, which enabled you to visually select data from the database and create a UI for it supporting standard CRUD (create, read, update and delete) functions without writing code. This was a real time-saver.

The 3-GL's Respond

Vendors of 3-GL's couldn't ignore the tumult raging outside their comfy offices. Before long support was added to most COBOL systems to embed SQL statements right in the code. Sounds simple, right? It was anything but. COBOL programs had data definitions which the majority of lines of COBOL code used. The way to handle the mis-match between SQL tables and COBOL record definitions wasn't uniform, but in many cases a single COBOL Read statement was replaced with embedded SQL with additional new COBOL code to map between the DBMS results and the data structures already in the COBOL. Ditto when data was being updated and written. Then there's the little detail that DBMS performance was dramatically worse than simple COBOL ISAM performance, since DBMS's were encumbered with huge amounts of functionality not needed by COBOL programs but which couldn't be circumvented or turned off.

Net result: the 3-GL's were worse off, by quite a bit.

What Happened?

Naturally the programming landscape is dominated by 4-GL's today, right? Or maybe their successors? How could it be otherwise? Just as each new generation of languages represented a massive advance in productivity from the earlier one and became the widely-accepted standard, why wouldn't this happen again?

It didn't happen. 4-GL's are largely of historic interest today, mostly confined to legacy code that no one can be bothered to re-write. Even the systems that genuinely provided a productivity advantage like Powerbuilder faded into stasis, rarely used to build new programs.

There is a great deal to be said about this fact. One of the factors is certainly the rise to dominance of object-oriented orthodoxy, which in spite of supposedly being centered on data definitions (classes) is nonetheless highly code-centric and has NO productivity gain over non-O-O languages. Where have you read that before? Nowhere? Probably the same place you haven't read all the studies showing in great detail how it achieves productivity gains. What can I say? Computer Non-Science reigns supreme.

Conclusion

I won't be writing a follow-up blog post on 5-GL's. Yes, they existed and were the hot thing at the time. I remember vividly all the hand-wringing in the US over the massive effort in Japan with the government funding research into fifth-generation languages. The US would be left in the dust by Japan in software, just like they're beating us in car design and manufacturing! When was the last time you heard about that? Ever?

Computers are objective things. Software either works or it doesn't. Unlike perfume, clothes or novels, it's not a mater of taste or personal preference; it's more like math. So what is it with the mis-match between enthusiasm and reality in software? It would be nice to understand it, but what's most important is to understand that much of what goes on in software is NOT based on objective right-or-wrong things like math but on fashion trends and the equivalent of Instagram influencers. Don't know anything about computer history? If you want to be accepted by the experts and elite, that's a good thing. If you want to get things done, quickly and well, ignore it at your peril.

 

Software Programming Language Evolution: Impact of the DBMS Revolution

The invention and widespread acceptance of the modern database management system (DBMS) has had a dramatic impact on the evolution and use of programming languages. It's part of the landscape today. People just accept it and no one seems to talk about the years of disruption, huge costs and dramatic changes it has caused.

The DBMS Blasts on to the Scene

In the 1980’s the modern relational database management system, DBMS, blasted onto the scene. Started by an IBM research scientist, E.F. Codd and popularized by his collaborator Chris Dodd, The Structured Query Language System, SQL, changed the landscape of programming. Completely apart from normal procedural programming, you had a system in which data could be defined using a Data Definition Language, DDL, and then created, updated and read using SQL  The data definitions were stored in a nice, clean format called a schema. Best of all, the new system gave hope to all the people who wanted access to data who couldn’t get through the log jam of getting custom report programs written in a language the analysts didn’t want to be bothered to learn. SQL hid most of the ugly details because of its declarative approach.

SQL was more than just giving access to data. There was a command to insert data into the DBMS, INSERT, then make changes to data, UPDATE, and even to send data to the great bit-bucket in the sky, DELETE. The system even came with transaction processing controls, so that you could perform a deduction from one user's account and an addition to another user's account and assure that either they both happened or neither did. Best of all the system did comprehensive logging, making a permanent record of who did what changes to which data and when.  Complete and self-contained!

This impressive functionality led to a problem. With users demonstrating outside the offices of the programming department, things were getting rowdy. The chants would go something like this:

Leader: What do we want?

Shouting crowd: OUR DATA!

Leader: When do we want it?

Shouting crowd: NOW!

Everyone wanted a DBMS. They wanted access to their data without having to go through the agony of coming on bended knee to the programming department to get reports written at some point in the distant future.

The response of languages: what could have happened

It wouldn't have been difficult for languages to give the DBMS demonstrators what they wanted with little disruption. One possibility was changing a language so that it produced a stream of data changes for the DBMS in addition to its existing data changes. A second possibility was changing a language so that its existing data statements would be applied directly to a DBMS. Either of these alternatives would have supplied a non-disruptive entry of DBMS technology into the computing world. But that's not what happened.

I personally implemented one of these non-disruptive approaches to DBMS integration in the mid-1980's and it worked. Here's the story:

I was hired by EnMasse Computer, one of several upstart companies trying to build computers based on the powerful new class of microprocessors that were then emerging. EnMasse focused on the commercial market which was at the time dominated by minicomputers made by companies like DEC, Data General and Prime. Having a DBMS was considered essential by new buyers in this market, but most existing applications were written in languages without DBMS support. One of my jobs was to figure out how to address the need. I was told to focus on COBOL.

This was a big problem because the way data was represented in COBOL didn't map well into relational database structures. What I did was get a copy of the source code of our chosen DBMS, Informix, and modify it so it could directly accept COBOL data structures and data types. I then went into the runtime system and modified it to send native COBOL read, write, update and delete commands directly to the data store, bypassing all the heavy-weight DBMS overhead. This was tested and proven with existing COBOL programs. It worked and the COBOL programs ran with undiminished speed. The net result was that unmodified COBOL used a DBMS for all its data, enabling business users full access to all the data without programming.

I did all the work to make this happen personally, with some help from an assistant.

I thought this was an obvious solution to the problem that everyone would take. It turns out that EnMasse failed as a business and that no one else took the simple approach that was best for everyone.

The response of languages: what did happen

What actually happened in real life was a huge investment was made with widespread disruption. Instead of burying the conversion, massive efforts were taken to modify -- practically re-write -- programs written in COBOL and other languages so that instead of using their native I/O commands they used SQL commands instead, with the added trouble of mapping and converting all the incompatible data structures. More effort went into modifying a single program for this purpose than I put into making the changes at the systems level to make the issue go away. What's worse is that, because of the massive overhead imposed by DBMS's for data manipulation using their commands instead of native methods performance was degraded by large factors.

While the ever-increasing speed of computers mitigated the impact of the performance penalty, in many cases it was still too much. In the late 1990's after massive increases in computer power and speed, program creators were using the stored procedure languages supplied by database vendors to enable business logic to run entirely inside the DBMS instead of bouncing back and forth between the DBMS and the application. While this addressed the performance issue of using DBMS for storage, it introduced having the logic of a business application written in two entirely different languages running on two different machines, usually with different ways of representing the data. Nightmare.

The rise of OLAP and ETL

One of the many ironies of the developments I've described is that people eventually noticed that the way data was organized for sensible computer program use was VERY different from the best ways to organize it for reporting and analysis. The terms that emerged were OLTP, On-line Transaction Processing, and OLAP, On-Line Analytical Processing. In OLTP, it's best to have data organized in what's called normalized form, in which each piece of data is stored exactly once in one place. This makes it so that a program doesn't have to do lots of work when, for example, a person changes their phone number; the program just goes to the one and only place phone number is stored and makes the change. OLAP is a different story because there's no need to update data that's already been created -- just add new data.

There were also practical details, like the fact that data was stored and manipulated by multiple programs, many of which had overlapping data contents -- for example a bank that has a program to handle checking accounts and a separate program for CD's, even though a single customer could have both. This led to the rise of a special use of DBMS technology called a Data Warehouse, which was supposed to hold a copy of all a system's data. A technology called ETL, Extract Transform and Load, emerged to grab the data from wherever it was first created, convert it as needed and stored it into a centralized place for analysis and reporting.

Given the fact that you really don't want people sending SQL statements to active transaction systems that could easily drag down performance and all the factors above, it turns out that the push to make normal programs run on top of DBMS systems was a monstrous waste of time. One that continues to this day!

Conclusion

Nearly all programmers today assume that production programs should be written using a DBMS. While alternatives like noSQL and key-value stores  have emerged, they don't have widespread use. Since the data structures used by programs are often very different than those used by the DBMS, a variety of work-arounds have been devised such as ORM's (Object-Relational Mappers), each of which has a variety of performance and labor-intensive issues. The invention and near-universal use of relational DBMS in software programming is a rarely recognized disaster with ongoing consequences.

 

Software Evolution: User Interface Concepts: Whose Perspective, Who's in Charge??

This post dives more deeply into the issue of Conceptual UI evolution as introduced in this post. Understanding UI conceptual evolution, which in practice is a spectrum, enables you to build applications that have UI's that produce dramatically better results than the competition -- getting more done, more quickly and at lower cost.

Whose perspective?

The least evolved UI concept looks at things completely from the point of view of the computer – what do I (the computer, which is really the person writing the application) need to get my job done? In this concept, the UI job is conceived as the computer getting things from the user, and protecting the computer from the user’s mistakes. This was, at one time, the prevailing concept for user-machine interactions, and remains surprisingly widespread, although few people would admit to thinking this way today.

At the other end of the spectrum, the software designer looks at things completely from the point of view of the human user – what do I (the human user, which is really the person writing the application) need right now and what can I do? In this concept, the UI job is conceived as the human getting things from the computer, directing it to do things, and presenting the user with options, possibilities and help that are as close and immediate as possible to what the user probably is trying to do.

Obviously, the technical side of UI’s has played a role in what’s possible here. In the early days of computers, we were glad to have them, and decks of cards and batch processing were way better than the alternatives. Computer time was rare and valuable; people were cheap by comparison; so it just made sense to look at things from the computer’s point of view.

The equation reversed long ago. Most computers spend most of their time idling, waiting in anxious anticipation for a user to do something, anything, just GET ME OUT OF THIS IDLE LOOP!! Sorry. That was the computer in me breaking out. Normally under control. Sorry.

Now, it’s entirely feasible to construct user interfaces entirely from the human’s point of view.

Who’s in charge?

There are some cases where the purpose of a program (and its UI) is entirely to be at the service of the user, and there are essentially no external constraints or advice to be given to the user. However, in a wide variety of practical cases, there are lots of people whose concerns need to be reflected in the way the computer is used. At one end of this spectrum, the user is in charge. If the user is in charge, and if we want to make sure the user does a certain thing under certain circumstances (think of a customer service call center environment), we give the user extensive training, and all sorts of analysts look at the results, so that certain customers are responded to in certain ways under different circumstances. We monitor the user’s calls, look at what they entered into the computer, and we work on changing what they do using group and individual meetings, training sessions, etc. All our effort is focused on the user, who clearly controls the computer; if we want things to be different, we go to the center of power, the user.

At the other end of this spectrum, the computer is in charge, in the sense that all major decisions and initiatives originate in the software. After basic how-to-use-a-computer type training, the user needs no training – everything you want the user to do is in the software, from what they should work on next to how they should respond to a particular request. Everyone who would have tried to influence the users directly now tries to put their knowledge into the software, which then applies it and delivers instructions to users as appropriate. When this concept is taken to the extreme, the human operator is little more than a complex and expensive media translation device, getting information the computer can’t get directly, and sending information to places the computer can’t send to directly.

So what does this mean in reality? It varies from application to application, but the net effect is always the same – the computer operator needs little training in how to respond to customers under different circumstances, because that information is all in the software. The operator mostly needs to learn how to take his cues and direction from the software, which provides a constant stream of what you might think of as “just in time training.” The user has no way of knowing if what he’s being asked to do or say has been done by many people for years, or is a new instruction just for this unusual situation.

This approach enables a revolution in how organizations respond to their customers. It makes complete personalization possible. It enables you to respond one way to a high-value customer in a situation, and another way to a low-value customer in the identical situation.  It also enables you to make nearly immediate, widespread changes to the way you respond to customers because you have a central place to enter the new “just in time” instructions, and don’t have to go through the painful process of building customer service training materials, training the trainers, getting everyone into classes, only in the end to have inconsistent and incomplete execution of your intentions.

While I’ve discussed this concept in terms of a call center application, exactly the same idea applies to the system interacting with people directly.

The system is really in charge

Let’s understand this way of building a UI with the “system in charge” a little better, since many people are unfamiliar with it.

The first step is to put all the real knowledge about how an operator should respond to which situation in which way into the system, and to enable changes to be made at will. The next step is to change operator/user training so that they understand how to map from the unstructured interactions they have to the choices presented by the system; normally, you have to train them to do this and to know how to respond. Finally, you can provide a set of pre-recorded inputs to the operators and capture their responses, to give them practice in applying their training in real life before they are inflicted on actual people.

Instead of thinking about the UI itself, think about the training that is normally required to get people to use an application, monitor their use of it on an on-going basis, and finally to make changes to the application and how people use it. You can start by thinking of the training as being like a wizard mode of the client, but with a training/case-based spin. Your trainers could build a big branching tree of what people on the other side of the phone can say, and how we should respond. All the content would be supplied by the training/customer service group. This would operate as the default mode of the application, until an operator has “passed,” and optionally beyond.

On one part of the screen could be a list of things customers can say to us. For each item, there would be one or more variations, not identical to the text, that would be recorded. In “pure” training mode, the application would randomly pick one, and the PC would play the recording. The operator would pick the item on the list of potential customer sayings that he felt was closest, and the system would then provide a suggested reply for the operator to give, and (if appropriate) highlight a field and give the operator a directive to interact with that field. This would continue cycling until the transaction was completed, abandoned or otherwise ended.

In “assisted” training mode, the list of customer requests, suggested replies and field highlights would remain, with the customer role being provided by a trainer or by real customers doing real transactions. In this case, a recording could be made of the conversation between operator and customer for additional checking or potentially for dispute resolution.

Obviously, the application needs to be extended to provide this framework, and then to download the content. But the advantage is completely realistic, integrated training. If we make changes to the application, we can automatically throw clients into this training mode to give them “just in time” training on the new features.

For what it’s worth, this is not a new idea. For example, operator training is a big issue in large-scale call center environments. Over the years, the best of those environments have evolved from classroom training to videos, to on-screen help to something like what I've described, which is now the state of the art in large scale call centers and is supported by major vendors of call center software. It’s the best because it’s completely grounded in reality and an extension of the actual software they have to use. The power of the approach shows most clearly in post-training changes and updates. Obviously, with it this integrated, it’s pretty easy to direct a client in training to a training server and check the data he’s entering.

Now that voice bots are becoming available, this approach to building UI's is all the more important and valuable. In any case, optimizing the work of the human is always in order, as spelled out in detail in this post. This post gives a detailed description of the huge project at Sallie Mae in which I played a part in the 1990's. It describes the 10X gains that can be achieved by taking optimizing the UI seriously.The main principles of human optimization in the UI are largely ignored by UI designers, making things many whole-number-factors less efficient than they could be in many cases. Amazing but typical. I've gone into just how and why this "I'd rather be stupid and get crappy results" to building software in general and UI's in particular in this post, in which I also describe the personal evolution that led me to the thoughts.

Software Programming Language Evolution: Credit Card Software Examples 1

In prior posts I've discussed the nature of programming languages and their evolution. I have given an overview of the so-called advances in programming languages made in the last 50 years. Most recently I described a couple of major advances beyond the 3-GL's. The purpose of this post is to give a couple real-life examples of how amazing new 4-GL’s and O-O languages have worked out in practice.

I was CTO of a major credit card software company in the late 1990’s. Because of that I had a front-row seat in what turned out to be a rare clinical trial of the power and productivity of the two major new categories of programming languages that were supposed to transform the practice of programming. Of course no one, in academia or elsewhere, has written about this real-world clinical trial or any of the similar ones that have played out over the years.

Bank One and 4-GL's

Bank One, based in Columbus Ohio, was a major force among banks in the 1990’s. They were growing and projected a strong image of innovation. During the 1990’s the notion that applications should be based on a DBMS was becoming standard doctrine, and the companies that valued productivity over Computer Science (and internet) purity were united behind one form or another of 4-GL as the tool of choice to get things done. Together with Anderson Consulting, one of the giant consulting companies at the time, Bank One proceeded on a huge project to re-write all their credit card processing code into a 4-GL.

After spending well north of $50 million (I heard nearly $100 million) and taking over 3 years, the project was quietly shelved, though industry insiders all heard the basic story. No one had an explanation. 4-GL’s are amazing, so much better than ancient things like COBOL – and card processing is just simple arithmetic, right, with a bit of calculating interest charges thrown in. How hard could it be? Harder than a 4-GL wielded by a crack team of one of the country’s top tech consulting firms could pull off with years of time and a giant budget, I guess.

On top of everything else, they had a clear and unambiguous definition of what the 4-GL program needed to do in the form of the existing system. They had test cases and test data. This already eliminates a huge amount of work and uncertainty in building new software. Compared with most software projects, the work was simple: just do what the old program did, using existing data as the test case. This fact isolated the influences on the outcome so that the power and productivity of the 4-GL was the most important factor. Fail.

Word of this should have gotten out. There should have been headlines in industry publications. The burgeoning 4-GL industry should have been shattered. Computer Science professors who actually cared about real things should have swarmed all over and figured out what the inherent limitations of 4-GL's were, whether they could be fixed, or whether the whole idea was nothing but puffery and arm-waving. None of this happened. Do you need to know anything else to conclude that Computer Science is based on less rationality than Anna Wintour and Vogue?

Capital One and Java

Capital One was the card division of a full-service bank that was spun out in 1994, becoming an unusual bank whose only business was to issue credit cards. In just a couple years the internet boom started, and with it enthusiasm for the most prominent object-oriented language for the enterprise, java. Capital One management was driving change in the card world and presumably felt they needed a modern technology underpinning to do it fully. So they authorized a massive project to re-write their entire existing card software from COBOL to Java. I remember reading at the time that they expected incredible flexibility and the power to evolve their business rapidly from the unprecedented power of Java.

The project took a couple years and was funded to the tune of many tens of millions of dollars; the amounts were never made public. As time went on, we heard less about it. Then there was a small ceremony and the project was declared a success, a testimony to the forward-looking executive management and pioneering tech team at the company. Then silence. I poked around with industry friends and discovered that the code had indeed been put into production – but just in Canada, which was a new market for the company at the time, handling a tiny number of cards. Why? It didn’t have anywhere close to the features and processing power that the existing COBOL system had to handle the large US card base. Just couldn't do it and company management decided to stop throwing good money after bad.

Conclusion

Executives and tech teams at major corporations bought into the fantasy that the latest 4-GL's and O-O languages would transform the process of writing software. They put huge amounts of money with the best available teams to reap the benefit for their business. And they failed.

These projects and their horrible outcomes should have made headlines in industry publications and been seared in the minds of academics. Software experts should have changed their tune as a result, or found what went wrong and fixed it. None of this happened. It tells you all you need to know about the power and productivity gains delivered by 4-GL's and object-oriented languages. Nothing has changed in the roughly twenty years since these events took place except for further evidence for the same conclusion piling up and the never-ending ability of industry participants and gurus to ignore the evidence.

 

We Don't Need Fedcoin We Already Have a National Digital Currency

The cryptocurrency enthusiasts are at it again, with a new name and even more ambitious goals than before: now they want a “national digital currency.” Hurry! The Chinese will beat us to it, and we’ll be left behind!

Somehow, no one in the debate acknowledges the obvious fact that we already HAVE a national digital currency. It’s fast, cheap and secure! It has no issue with regulators, and it’s accepted everywhere. Who knew? It’s called … the US dollar. The wild-eyed “national digital currency” groupies prefer to ignore the fact – yes, it’s a fact – that the US dollar is a digital currency. Instead, they’re convinced it can’t possibly be a good thing, because it’s not based on brand-new, cool, “immutable distributed ledger” blockchain-based cryptocurrency technology. Bzzzt! Wrong.

The national digital currency of the USA

The people who talk about “national digital currency” are obsessively focused on cryptocurrencies. They make believe digital currencies are a recent invention, and that only things that have evolved from Bitcoin meet the description. Nonetheless, by any reasonable definition, here in the good old USA we already have a digital currency. It’s called the US dollar. It’s managed by the Federal Reserve Bank. But that’s not digital, you might say – what about that green stuff in my wallet, and those coins jangling in my pocket or purse?

I agree, we have cash. As of Feb 12, 2020 there was $1.75 trillion worth of paper cash in various denominations in circulation. That’s quite a bit. But it’s far from the whole story. For the rest of the story, we turn to the money supply, the total amount of which is one of the chief responsibilities of the Fed to maintain – and grow and shrink, as needed. There are two main measures of the money supply, M1 and M2. See this for the Fed’s definition. Basically, M2 includes checking and savings bank deposits, money market funds, and similar cash-equivalents. As of December 2019, M2 was $15.434 trillion dollars.

What this means is simple: almost 90% of US dollars have no physical existence – they are purely digital. But this isn’t just for the USA; world-wide, only 8% of currency exists as physical cash!

The US dollar took many steps over more than a century to evolve from physical cash to today’s largely digital currency. First, paper currency wasn’t “real” money – it was a promise by a bank to trade the paper for the equivalent in gold. For example, here’s a $5,000 bill from 1882 that’s a promise to exchange for $5,000 in gold coin on demand:

Bureau of Engraving and Printing color specimen of a $5,000 Gold certificate, Series of 1882

National Museum of American History - Image by Godot13

In practice, no one exchanged these large-dollar notes for gold; they were mostly used by banks and the government to move funds between themselves, a practice which stopped in 1934.

Long before the advent of computers, the gold exchange promise was dropped. Here’s a bill as printed in 1928 that simply declares that it’s $5,000:

1928 Federal Reserve note

National Museum of American History - Image by Godot13

The last high-denomination bills were printed in 1945. Large inter-bank transfers were done without the exchange of cash; tightly controlled procedures were used to transfer “money” between bank ledgers before the advent of computers. In 1969 the large bills were officially discontinued, and the government started destroying them. In 1975, the government started depositing social security payments into recipient’s accounts electronically. By 1990, all money transfers between commercial and central banks were done electronically.

There is no single date when you can say that the dollar became digital. The process of transformation took place step by step, each leading to the next. The early steps took place long before computers; the principle was established and in universal use among banks and the federal reserve already in 1945! The invention and use of computers simply enabled further automation of the digitization of the US dollar, and enabled fully real-time transfers to take place.

What all this adds up to is that the US dollar is a national digital currency, by any reasonable definition, and has been for years. The vast majority of currency value is fully and completely digital, and all large-dollar transactions are completely digital. We also have cards, which are smaller, lighter and more convenient than smartphones, with the added convenience that they don’t crash or run out of power. In addition, we have the added convenience of physical cash, 100% interchangeable with its digital currency equivalent, as we see with ATM’s every day. Cash is convenient for small transactions and for people who don’t have working, powered and connected small computers on their person. The US dollar is indeed a national digital currency, with the added convenience of cards and cash.

What’s a national digital currency?

The vast majority of people know through everyday experience that the US dollar is a national digital currency. But almost no one talks in those terms. When people use that recently-coined term, they usually means something brand-new, a form of cryptocurrency. For example, a recent WSJ article describes a push towards a “national digital currency.” One of the quoted authors waxes eloquent about its virtues, but never really says what it is.

The only way to understand “national digital currency” is to back up and look at the history of where the concept came from. While no one likes to talk about it, the undisputed origin of the concept is a brilliant, well-implemented and widely used body of software called Bitcoin. The concept and every major feature of Bitcoin was designed to operate with no central authority of any kind in charge. Amazing. How can it be that anyone anywhere could declare themselves to be a Bitcoin “bank” (they call them “miners”) and the system works? See this for an explanation. Bitcoin was also designed to give total anonymity to the people who deposit, send and receive Bitcoin, making it a favorite of international criminals around the world.

Before long, Bitcoin competitors appeared, each claiming to add or correct something important in Bitcoin; for example, Ethereum introducing the so-called “smart contract.” Next, people started talking about “blockchain” and the “distributed immutable ledger,” taking out the concept of currency. Supposedly, these technologies would solve long-standing problems involving data that was in many locations. This led to loads of blockchain start-ups and service companies, with giant corporations infected with bad cases of FOMO funding pilots and proofs-of-concept. Major companies like Microsoft and IBM now offer blockchain-as-a-service in their cloud products.

More recently, we have seen highly publicized efforts to legitimize something like an enhanced Ethereum-like currency, most prominently Facebook’s Libra, which has the backing of a large number of name-brand financial institutions.

All this leads up to the newly “coined” notion of a “national digital currency” – let’s have the US government implement it instead of Facebook and its consortium partners!

This is all-too-typical technology mania. We’ve seen it many times. The true believers ignore evidence, ignore existing practice and fervently believe in the world-transforming new technology. Loads of highly-paid executives and government leaders pay obeisance, effectively paying insurance against the remote possibility that the cult delivers real value. There’s a strong lemming effect: don’t be left behind!

Inconvenient facts

People who advocate for a “national digital currency” like to ignore the one we already have, in favor of some variation of the currency beloved by human smugglers, drug lords and international illegal arms traffickers. Like the people at the Digital Currency Initiative at the much-revered Media Lab at MIT. In a recent WSJ article, the director of the lab immediately conceded that with direct deposit of salary and Venmo to split the cost of dinner with friends, it seems like we already have a digital currency. But this isn’t good enough! After all, there are fees, and big banks are involved and sometimes transactions can take days. Ugh. With a real national digital currency, a federal cryptocurrency, payments would be “faster, cheaper and more secure.”

There are just a couple little problems. Here are some highlights:

Cryptocurrency is slow

Crypto-groupies love to talk about the slowest transactions in the multi-trillion dollar US digital dollar system. While large parts of the US digital dollar system execute huge numbers of transfers in seconds, Bitcoin takes on average ten minutes to execute a single transfer. And that’s only if you pay an above-average fee – if you don’t pay much, you could wait for hours for your transaction to process.

Cryptocurrency can’t scale

Depending on the transaction size, Bitcoin can only process between 3 and 7 transactions per second. If there were always transactions waiting to be processed, 24 by 7, at 5 transactions per second Bitcoin could handle at most 158 million transactions per year. By contrast, over 10 billion transactions are performed at just ATM machines every year in the US alone. There were over 110 billion card transactions in the US in 2016. The growth in transactions from 2015 was over 7 billion; the growth in card transactions was about 50 times greater than the maximum capacity of Bitcoin.

Cryptocurrency is expensive for users

Crypto-groupies love to talk about the high fees for doing certain dollar transactions, ignoring the immense transaction flow of cheap and easy transactions like direct deposit and ACH, which operate at huge volumes. They don’t talk much about the costs of running cryptocurrency. They’re smart to ignore the subject. Today’s Bitcoin transactions are costly, and the second you try to correct the various problems (speed, scalability, security), the costs skyrocket.

Cryptocurrency is expensive to operate

Hardly anyone uses Bitcoin, and the volumes are tiny compared to the dollar. Nonetheless, Bitcoin is incredibly, mind-blowingly expensive to operate. Even at today’s minuscule volumes, Bitcoin computer processing consumes about the same amount of electricity as the whole country of Switzerland!

Cryptocurrency loss is permanent

If you lose your checkbook, your credit or bank card or anything else, you’re OK; you contact the bank and they fix it. By contrast, if you lose your cryptocurrency key (a string of numbers), there is literally no way to recover your money. About 20% of all Bitcoin are believed to be lost, something like $20 billion!! If you lose your key, whoever gets it can take all your Bitcoin, unlike with for example a lost card, where you call the bank, report the lost card, and avoid losing any money.

Cryptocurrency is horribly insecure

The crypto folks love the fact that everyone imagines that “crypto” means “can’t be cracked.” So they avoid the subject. The fact is, crypto banks are robbed and every Bitcoin stolen all too often. Nearly a million bitcoins have been lost in this way, a loss at today’s prices of roughly $10 billion!! Even the core defense of Bitcoin has now been cracked.

No proposed crypto alternative to Bitcoin solves the problems

To the outside, crypto people are all about ignoring the problems and promoting wonderfulness. Among themselves, the relatively sane advocates recognize the problems and try to solve them, with endless variations being rolled out. In doing so, they either make the problems worse or destroy what little value there is in cryptocurrency. One of the leading ideas is to make a private blockchain, which is a pathetic joke. For example, Microsoft and Intel spell out many problems by way of selling their ineffective solution, and the Facebook Libra coalition takes the “solve it by making it worse” approach to new lows.

The strengths of the US dollar digital currency

The whiners will whine about what’s wrong with today’s US dollar. Is it really chock-full of problems, as the crypto-groupies like to say? Let’s do something rare: focus on the positive. First and foremost, let’s remember that the dollar has worked for a couple centuries now, and along the way transformed itself from physical to about 90% digital, all without breaking! In addition, it has benefited from tremendous private-sector innovation. Here are some highlights of the fastest, cheapest and most secure currency ever created:

Physical cash is great. When I’m in the city and someone gets my car for me from the garage, I like to give a tip. It’s easy: I pull out my wallet and hand over bills. Anything fully digital would require electronics and would be a pain.

Cards are great. When I pull into a gas station in New Jersey, where gas is pumped for you, I open the window, say “fill with regular, please” and hand over a card. When it’s done, I get the card and a receipt and drive off. Easier than cash because no change. This is fully digital. Today. And, at my great local gas station, they often clean my windows, so I get to hand the guy a couple bucks as a tip. Painless.

Cardless is great. I call for an Uber from the app. When the car arrives, we each check each other’s identities and away we go. On arrival, I get out. That’s it!

Wiring money for a house closing is great. I call USAA, my bank, who verifies my identity and gets it done in minutes. No going to a branch, certified checks, etc. The phone call is a good thing – it reduces the chance of fraud to near-zero, unlike the fraud-riven crypto world.

P2P apps are great. There are zero-cost, instant transfer apps like Venmo, CashApp and Zelle. These are used by over a hundred million of people, and they work. Today. How could crypto in any form be better? Actually, it would be worse. See this.

What about those awful transactions that supposedly take days? Yup, there are some. It’s called a step-by-step, no errors or crashes permitted transition to real-time. Transactions are already 100% digital, and with ACH (like electronic checks) very low cost. The version of ACH in the UK is already same-day, and ACH in the US is in the middle of a transition to same-day and real-time.

What about international payments? I guess the crypto-groupies are out of touch with what’s going on here in the real world. For personal use, credit cards are already accepted nearly everywhere, with everyone involved getting or paying in their own currency. The big complaint of the crypto people is international business transactions, involving lots of time, transfers and fees. That was true. Which is why a handful of amazing new companies have emerged and are addressing the issue. A couple of them are operating at scale and in production today.

Currency Cloud, for example, has a brilliant solution. A company that has suppliers in many countries gets the suppliers to give Currency Cloud their preferred local bank accounts. Currency Cloud itself maintains local accounts for itself in all the countries it supports. The buyer sends a payment directive to Currency Cloud, who then does a local transfer of the requested amount from its account in the target country to the vendor in that country. As the network grows, each supported country has a larger number of companies both sending and receiving payments, so that a growing number of transfers can be done completely locally – only the net payment imbalance between countries needs to be settled by Currency Cloud between its own accounts, which it optimizes for minimum cost. This is 100% digital, low cost, real-time, and operating at scale. Today.

For smaller business and individuals there are services exploding onto the scene for international payments. For example, Rapyd (disclosure: my VC fund is an investor) enables people without bank accounts to buy, sell and get paid for work in over 100 countries at over 2 million access points, where they either get or give local currency to complete international digital transactions. For example, you could be a driver for Uber and get paid, even though you have no card or bank account.

Conclusion

Get over it, crypto-fanatics and blockchain groupies. Yes, the Bitcoin technology is an impressive achievement, and highly useful to the criminal class. But it makes any real-world currency problem you can think of worse, and completely ignores the patent reality, which is that the wonderful “future” of a national digital currency is something we have today – the US dollar!

Note: this post first appeared at Forbes.

Could Blockchain Help Fix My Car That Was Destroyed by a Tree Branch?

My car was safely parked in my driveway. A large branch broke off of a tree that had recently been checked by an arborist and declared healthy. Ignoring the arborist’s expert opinion, the branch broke off and fell anyway. My formerly sound, two-year-old car was towed to a repair shop, an estimate for repairs made, and my insurance company declared it not worth fixing. Totaled.

But this shocking event had a couple good outcomes. The first was that I ended up leasing a nice new car. The second outcome was some education that is hard to come by, and has serious implications – I learned how valuable Blockchain technology would be in helping to coordinate the information and efforts of my car insurance company, the repair shop, and the car rental company that supplied me with a car until I could get a new one.

Blockchain is the immutable distributed ledger technology, a kind of distributed database that powers Bitcoin and other cryptocurrencies, whose promise is actively being pursued in many industries. What Blockchain is all about is enabling countless independent parties with independent computer systems to interact with each other in a fast, secure way, sharing information to reach a mutually desired outcome. That’s exactly what we have here, with loads of insurance companies, a number of car rental chains, and untold thousands of car repair shops – all of whom need to share information and coordinate their efforts to help the consumer with the damaged car. Perfect for Blockchain!

Think about the situation. Insurance companies are all about long documents with fine print, and long times on hold waiting to talk with someone who often can’t, in the end, do much but promise to send a form in the mail. You’ve probably driven by loads of auto repair shops. Which can handle the repair your car needs? How much will they charge? Will insurance pay for it? And then I’ll be without a car. Renting a car at the airport is one thing, but locally? How do I pick a company and get there. At the end I’ve got to deal with picking up my repaired car and returning the rental. Will insurance pay? It’s all yuck, yuck, yuck. Getting my car smashed is one thing, but this makes a bad situation worse.

Imagine what a Blockchain-fueled application could do – it could eliminate the paperwork and calls, get the insurance company talking with the repair shops and car rental companies. Blockchain would enable electronic “paperwork” to be exchanged safely and securely. The insurance company could arrange for a local repair shop that can handle my car to do the repair – and pay them directly! They could dig up a local car rental company, and arrange for me to be picked up and dropped off at the end – and pay for the car directly! If things took longer than planned, all parties could communicate directly and just get it done. It would be a true distributed transaction application, minus the Bitcoin but with the transactions I care about now – getting my car fixed!

 

I know I’ve expressed doubt about blockchain and cryptocurrencies in the past, while admiring their power. This could be the inflection point for me – a real, practical, everyday nightmare that would be transformed by Blockchain! Maybe I could even dive in and lead making it happen; wouldn’t that be ironic?

Enough of living in fantasy-land – I’ve got a car that needs fixing. With dreams of a future Blockchain-fueled revolution in the back of my mind, imagine my shock as I went through the process, and found that everybody seemed to know everything! My insurance company knew a local repair shop to use, and contacted them for me. They also contacted a local branch of Enterprise Rent-A-Car, who sent someone out to pick me up. Then I found out that Enterprise knew where my car had been towed, and was ready to pick up their car from there when I went for it. Then I found out that my insurance company was paying the car repair shop directly, and paying Enterprise directly. Then when the estimate came in and my car was declared a total loss, things were taken care of until I could get a new car – which my insurance company also helped with.

What’s going on here? Have they already implemented Blockchain?!

I started asking some questions. It turns out that the nightmare of coordination and paperwork flying around was noticed decades ago. In 1994, Enterprise created the Automated Rental Management System (ARMS^®) “to help insurance companies simplify the cumbersome process of managing replacement rental cars for policyholders.” By the early 2000’s, it was already widely used.

Things progressed over the years. As of 2017, “hundreds of insurance companies and thousands of collision repair centers use Enterprise’s value-added system, which processes millions of transactions every year.”

This sounds good, but there must be a catch. This could be some centralized, expensive enterprise system that locks everyone in. Well, maybe not:

Central control? “ABS’ approach, on the other hand, enables collision repair centers, insurance companies and fleet owners to remain in control of their data for the long term – a high priority since vehicle technology and associated repair processes are changing rapidly.”

What about data format standards, the tough thing for Blockchain? “The ABS system helps protect insurance companies, collision repair centers and fleet owners by converting their information from EMS (Estimate Management Standard) to a more secure protocol, BMS (Business Message Suite).”

I’ve learned important things about Blockchain from this experience. I’ve learned that a huge problem in car repair, insurance and rental involving many disparate parties, has already been solved and is in production, used by industry giants and thousands of local businesses. This is just the kind of problem whose solution “everyone” says Blockchain “enables.” It’s in production today. It has evolved with technology,  No Blockchain needed. So why is it exactly that Blockchain is the key missing ingredient for solving distributed data, sharing and interaction problems of this kind?

Note: this post first appeared at Forbes.

 

The Bronte Sisters and Software

Who would have thought that the amazing, pioneering and tragic Bronte sisters could demonstrate important things about software programming languages? Not me, until I started thinking about it. I realized that their achievement has a close parallel to what great programmers do: they don’t invent a new language, they use an existing language to express new things, thoughts that were in their heads but which hadn’t before been published.

The Sisters

I hope you’ve at least heard of these ladies, and even better read a couple of their wonderful novels, among which are Charlotte’s Jane Eyre, Emily’s Wuthering Heights and Anne’s The Tenant of Wildfell Hall.

Their novels were very successful. Originally published with a man’s name listed as author, their success continued after their real identities were revealed, which demonstrates that the success of their work was solely due to the quality and originality of their work There have been movies and numerous references to them and their work in other media.

If you haven’t already spent some time enjoying their work, I hope you will.

In all the talk about the Bronte’s, no one bothers to mention the perfectly obvious fact that they used the English language of their day to write their novels. English didn’t hold them back. Nor did English “help” them. Their originality was completely in the way they used the English language.

The Bronte’s and Software Languages

The obvious response to the above is … duhhh, the reason no one mentions their use of unaltered, un-enhanced English is that nearly all novelists do the same.

Now let’s turn to programming. Most programmers, like novelists, just use the language they've been given to get the job done. Most programmers, like most who attempt to write a novel, do pedestrian work.

Unlike novelists, there is a subset of programmers who obsess about which is the “best” language as measured by various scales. Programmers who consider themselves to be a cut above the rest fiercely criticize this or that tiny detail of whatever language is in their cross-hairs this morning. If their ire runs at peak level for a while, may even invent a new language. Why? Their amazing new language will “prevent” programmers from making this or that kind of error – like sure, when has that ever happened – or somehow raise who ever uses it to new levels of power and productivity and quality. Not. Never happened. Baseless assertions and propaganda.

Was something important “added” or “corrected” in the English language that enabled the Bronte’s to do what they did? Nope.

This leads to a thought that is blasphemous to the self-appointed elite of software: the software language you use is almost irrelevant, of course with some exceptions; what's important is what you write in the language you're using. Just like with novels.

Languages and science

Hold on there just a sec! Novels are fiction, meant to entertain. Completely different subject. Software is like math -- it's pure and exact, devoid of messy things like the emotions and nuances of human interaction that novels are full of.

True enough. First I would say, try having a discussion about the differences between programming languages with one of the software elite who obsess about the subject. See how much "cool, calm and collected" you get; every time I've tried having a rational discussion on this subject over the years voices have gone up notches at a time, and passion has been slopping all over the place.

Perhaps we can be enlightened by the subject that's been raised over the years of the best language for science and/or math. There are even books on the subject!

Let's take a quick look at a bit of evidence:

Skipping over loads of details, what you quickly find is that, not long after Galileo broke with tradition and wrote in his normal speaking language (Italian) instead of Latin, scientists tended to write in whatever language they used in everyday life. Chemistry was dominated by the German language in the 1800's not because German was somehow better for chemistry (which didn't stop some people arguing that it was), but because most of the productive chemists happened to be most comfortable writing in German, mostly because they spoke German. A few years ago a couple Norwegian scientists were award the Nobel prize. They probably spoke Norwegian in the lab, but if they wanted to be read, they had to write in a widely-read language: English. Not because English was "better" for science, but just more widespread at that point.

In all these cases, the language happened to be used for expressing the thoughts, facts and concepts -- which were independent of the language used!

Just like it is in ... software programming!

Conclusion

With a few important exceptions, the language you use to write a program is like the language you use to write a scientific paper or a novel. The language used is not the most important thing. By far. The most important thing is what you have to say in whatever language you end up using.

Software Evolution: User Interface

User interfaces have gone through massive evolution since their first appearance in the 1950's. Lots of people talk about this. But not many separate the main two threads of UI evolution: technical and conceptual.

The technical thread is all about the tools and techniques. Examples of elements in the technical thread are the mouse, function keys, menus, and graphical windowing systems. Advances in the technical thread of UI evolution are created by researchers, systems people and systems makers, both hardware and software. People who build actual UI’s generally have to use the tools they’ve been given.

The conceptual thread of UI evolution is about thoughts in the heads of application builders about what problem they’re trying to solve and how they’re supposed to go about solving it. Application builders are generally taught the base concepts they are supposed to use, and then usually apply those concepts throughout their careers. But all application builders don’t have the same thoughts in their heads. The thoughts they have exhibit a clear progression from less evolved to more evolved. It is interesting that the way application builders think about what job they are supposed to do is almost completely independent of the tools they have, i.e., the technical thread. Yes, they can and do use the tools available to them, but this conceptual thread of UI evolution rides “above” the level of the technical tools.

The evolution of UI on the technical side is widely discussed and understood. As hardware has gotten better and less expensive, the richness of the interaction between computer and human has increased, with the computer able to present more information to the user more quickly, and with immediate reaction on the computer’s part to user requests. For the most part, this is a good thing, although people who think only about user interfaces can make serious product design mistakes when they fail to put the user interface in the broader context of product design. For example, generally speaking, pointing at a choice with a mouse is better than entering a code on a keyboard, and giving users lots of control through a rich user interface is better than giving them no control. However, in situations where there are repetitive tasks and efficiency is very important, the keyboard beats the mouse any day of the week, and in situations where tasks performed by humans can be automated, it is far better to have the computer do it -- quickly, effectively and optimally -- rather than depending on and using the time of a human being, regardless of how wonderful his UI may be. This post goes into detail with examples on this subject.

Conceptual UI evolution, by contrast to evolution on the technical side, is not widely discussed and not generally understood. Understanding this evolution enables you to build superior software by creating software that enables tasks to be accomplished with less human effort and greater accuracy.

UI Concepts

The conceptual level of user interfaces is most easily understood by asking two questions: (1) whose perspective is the primary one in the mind of the application UI builder – the computer’s or the user’s; and (2) to what extent is the user relied upon to operate the software correctly and optimally? The most primitive UI’s “look” at things from the computer’s point of view, and, somewhat paradoxically, rely almost entirely on the user to get optimal results from the computer. The most advanced UI’s “look” at things from the user’s point of view, while at the same time imposing as little burden of intelligence and decision-making as possible on the user.

When you state it this way – a UI should be user-centered and should help the user to be successful – you may well assume that building UI’s in this way would be standard operating procedure, and that building UI’s in any other way would be considered incompetent. Sadly, this is not the case. Like all the patterns I describe in my series on software evolution, most people, companies and even industries tend to be “at” a particular stage of evolution in the subject areas I describe here; companies gain comparative advantage by taking the “next” step in the pattern evolution earlier than others, and exploit it for gain more vigorously than others.

Some of the patterns I've observed in software evolution just tend to repeat themselves historically with minor variations. Other patterns, of which this is an example, don't seem to be as inevitable or time-based. This pattern is much like the pattern of increasing abstraction in software applications, described in detail here. Competitive pressures and smart, ambitious people tend to drive applications to take the next step on the spectrum of goodness.

For UI, the spectrum can be measured. The UI that requires the least time and effort by a user to get a given job done is the best. That's it!

Do UI experts think this way? Is this a foundational part of their training and expertise? Of course not! Just because computers are involved, no one should be under the illusion that we live in a numbers-driven world. For all the talk of numbers, people are more influenced by the culture they're part of, and generally want validation from that culture. Doing something further up the UI optimization curve than is customary in their milieu is nearly always an act of rebellion, and most people just don't do it.