The Black Liszt

Getting Results from ML and AI 5: Fintech Fraud

While the success patterns laid out in the prior posts in this series may seem clear in the abstract, applying them in practice can be hard, because nearly everyone who thinks or talks about AI (these sets over overlap very little, sadly) takes a different approach.

http://www.blackliszt.com/2018/03/getting-results-from-ml-and-ai-1.html

http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-2.html

http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-3-closed-loop.html

I previously discussed the application of the principles to healthcare, with specific examples: https://www.blackliszt.com/2018/08/getting-results-from-ml-and-ai-4-healthcare-examples.html

In this post, I'll show how things can play out with a stellar example in fintech anti-fraud.

The Use of ML in Fraud Detection

Credit card fraud is a difficult, ever-evolving problem, not unlike cyber-security in general. In most cases, real money is involved, billions of dollars of it -- over $20 Billion world-wide as of a few years ago!

Robert Hecht-Nielsen was a pioneer in machine learning in the 1980's. He started HNC software to apply ML to a variety of problems. He was particularly fond of Neural Networks. After a few years of not doing well, he encountered a visionary executive of Household Finance, a large company that provided credit, including cards, to sub-prime customers. The HFC executive convinced Hecht-Nielsen to concentrate on card fraud, which was a large and growing problem for the card industry in general, and HFC in particular.

With HFC's support, HNC built the first effective card fraud detection system. It centered around humans analyzing recent instances of card fraud, building a detector for each particular kind, and training neural nets to combine all the detectors to be applied to each new transaction as it came through.

Since the model's effectiveness depended on having an ever-growing library of fraud detectors, HFC supported HNC creating a network model, in which each participating card issuer would contribute the details of each new case of fraud it encountered. HNC analysts would add detectors, train new models and give updated fraud detecting models to each participating company. It quickly became the industry standard, since all the participants benefited from the experience of all the members. HNC went public in the mid-1990's and later merged with FICO.

Enter Feedzai

How could any new entrant possibly compete against the data monopoly and neural networks of FICO/HNC? Somebody new wouldn't have access to the incredible database of card fraud. Feedzai, backed by Oak HC/FT, had some things that were more important.

First of all, it helped that the Feedzai founders were broadly expert in machine learning and other analytic methods. It took them a while to focus on "just" card fraud. Being a general expert and then deciding to focus on a narrow problem is just like HNC, and is a general success pattern. But HNC got frozen and dabbled in many other domains, failing to continue to innovate in their core product.

Once Feedzai got engaged with a card issuer, they noticed that the FICO solution was so slow that it couldn't detect whether an authorization was fraudulent in real time -- only after it was already approved. Result: every fraudster got at least one free! Beating FICO would require real-time analytics. Next, Feedzai noticed that FICO was trying to see if each transaction was BAD, and that seeing if each transaction was NOT GOOD was just as good -- even better, since you don't care how inventive the bad guys are, just that they're not acting like the real cardholder would act. That meant you could train your models on a single card issuer's customer base, ignoring the rest of the world. Cool! Finally, that meant you could use modern, transparent ML algorithms, and dispense entirely with the time-consuming and error-prone FICO method of modeling each type of fraud. And that meant that your first couple of appropriately skeptical customers could try you out, side-by-side, against the incumbent FICO, and see who won the catch-the-fraud race. Hooray, Feedzai won!

It's important to note that Feedzai also won because they did all the foundational steps described in the earlier posts in this series correctly, including the all-important know-your-data and closed-loop steps.

FICO/HNC moved on from their success in card fraud, thinking they had an absolute lock on the market -- which they did, for many years, but only because someone like Feedzai didn't go after them sooner. Feedzai isn't making the FICO/HNC mistake; they are continuing to build on their lead, both deepening and extending it. It's a big subject, but here are a couple of the highlights:

• Feedzai has built a tool that does much of the work highly skilled engineers used to have to do when supporting a new client. The original tool detects fraud; the new tool automates building a fraud detection tool. The speed and quality that results is unprecedented.
• While rarely discussed, every practical detection system has a set of human-created rules. Feedzai is the first company to automate the evaluation of the rules, and decide which should be changed or dropped. This is huge.
• Feedzai is now, step by step, introducing their technology to adjacent areas such as AML. The incentives created by the heavy regulation in AML are perverse, and has led to bloated costs in banks with no real improvement in AML detection. Feedzai's novel approach is actually making headway in spite of the regulations, and not just doing normal reg-tech automation.

Conclusion

Being an excellent generalist is often a good way to become a superb specialist. But it's rare to find a group of people who are truly up to snuff in the abstruse technologies of AI/ML, but also able to dive into the ugly details of data and real-life cases to make a system that beats the real-world incumbents. As we know from FICO/HNC and many other cases, the pattern is then to declare victory and move on. Feedzai is an object lesson in how to go deeper and deeper, continuing to innovate and automate. They are an excellent example of the principles described in the earlier posts in this series.

Cybersecurity is Almost Impossible to Achieve -- Here's Why

As with most computer software issues, cybersecurity is badly misunderstood by the vast majority of people, including, sadly but as usual, most computer professionals. The result of this is that the vast majority of people have wrong ideas about the source and methods of security breaches and how they can be prevented. Unfortunately, sometimes these wrong ideas have major consequences.

A Typical Phishing Attack

Phishing is a kind of attack on a user by a bad guy. The bad guy sends the target an email that contains something to tempt the target to click on it. Clicking on the hyperlink is like a fish biting the bait on the hook -- you're caught!

I've been the target of a slew of phishing attacks in the last couple of months. Here is a typical email I've received:

 

What a scary email! It's to me, from my company's email administrator, and I'd better do something about it!

An amazing number of recipients of such an email would have clicked on the red "Resolve Errors Now" box. I wouldn't because:

• I know that my email admin person's domain is not "team-admin.net"
• There are multiple suspicious minor grammatical errors.
• Why would the final line be a copyright?

And many other reasons.

I went to the trouble of digging a bit. I exposed the URL at the hyperlink I was supposed to click and copied it. Here is the place I would have gone had I done what the phisher intended me to do:

Yup, it's a one-time domain set up by the phisher with the information about who I was so that the right next hacking steps could be executed -- that's the string after the question mark. This is not an amateur phisher!

There are more where that came from.

The email above was hardly an isolated example. Just yesterday I received an email from someone I knew at another VC firm, asking me to click to download an important document. Half an hour later, I got a legit email from the firm's admin saying sorry, the sender's account was hacked, please don't click on the email, and if you did, you may want to change all your passwords.

Here is another I recently received, apparently from DHL instead of apparently from my email administrator:

Yes, the place they want you to click is totally bogus -- and bonus points for doing an exceptional job mimicking DHL!

Cybersecurity Experts Know all about Phishing Attacks!

Sure they do. It's on everybody's list, and they all put "solutions" in place. Solutions that don't work.

A colleague of mine has been a top tech executive in major financial institutions. He told me how none of the anti-phishing solutions work because too many valid emails end up in the "spam" folder, and/or too many phishing attacks are passed through as OK. If users know there's a phishing filter in place, they tend to assume that every email in their inbox is OK, and click away, making the problem worse.

The executive tried phishing education for users. He ran several different kinds of education, and sent test phishing emails to people as tests of the education. Net result: no form of education improved the practical willingness/ability of users to notice and avoid clicking on phish attacks. Conclusion: highly paid office workers are too stupid to be educated on this subject.

The Place of Phishing Attacks in Cybersecurity

Regardless of your familiarity with computer security, I suspect you've got the image nearly everyone has about it -- an image that is like a medieval castle with high walls and ponderous, thick gates that are heavily guarded. It's all about those evil bad guys roaming around out there, always probing for ways to get around the gate or over the wall to get inside and wreak havoc.

This is a fun image, and accurately reflects something which is part of the truth. Phishing attacks, along with insiders who have become bad guys, are an important part of the rest of the truth. While there are "solutions" to phishing, they're mostly badly-performing filters that dump loads of emails into a "spam" folder, many of which are perfectly legit. Making you manually go through it anyway. What's the point?

Phishing and bent insiders are the poor step-children of cybersecurity, always acknowledged as being part of the family, but surviving on hand-me-downs and left-overs as best they can. This in spite of the fact that they continue to be the source of the most flagrant "breaches" of security we know about! High-end retail stores and even libraries have had working solutions to this kind of problem in place and working since before computers were invented, but there's something about being a fancy-shmancy computer science cybersecurity expert that prevents such solutions from being considered.  Among other things, check out:

https://www.blackliszt.com/2017/05/computer-security-problems-solutions.html

Conclusion

The image of bad guys cackling in distant, dark rooms lit with screens, as they manage increasingly devious attacks against the walls and defenses of the innocent computer systems of the world continues to cause regulators, cybersecurity experts and the other grandees of computer administration to ignore the real threats -- the ones that work, the ones that actually lead to failures of computer security. It's hard to grasp the full extent of combined stupidity, self-satisfaction and blindness that leads the ongoing non-response to bad actors recruiting insiders to do their work for them.

 

Take-aways from the 2019 Edition of the Money 2020 Conference

The 2019 US edition of Money 20/20 is a wrap. All of us who attended are recovering – and digesting.

Money 20/20 is now an established event – even though the first one was just seven years ago, in 2012! You might think it would be getting stodgy and repetitive by now, with bosses sending their underlings. Not so! Among the over 7,000 people attending were an amazing number of top, big-company executives – along with hundreds of startups, investors, and companies ranging from emerging to established. All the tech firms you’d expect also attended.

So what happened? Anything new? Big themes? Here are a couple:

Entrepreneurs

The startup energy was amazing, partly because there were literally hundreds of startups in attendance, many in little booths, many participating in the events and meetups that were organized for them and the larger groups – investors, partners and buyers – who want to know what’s going on. There’s no way I’m going to pick a couple as examples of the hundreds, but as a test, I’d suggest you come up with a couple ideas you think are novel, walk next year’s show, and see how novel your ideas really are. Unless you’re incredibly better than most, you’ll find a startup with your idea – and if not, maybe it’s your time to start a company!

The Feedzai Financial Crime Summit

We’re glad to be investors in Feedzai, and amazed at the sub-conference they’ve run the last couple of years. It’s too bad Agatha Christie couldn’t attend this year to help solve financial crimes, though the stellar crowd recruited by Feedzai did a fine job of it. The top person from Microsoft described how he spends nearly $1 Billion a year on cyber-security, with competing teams of hackers and defenders, and much else.

With so much of money being electronic and transactions being conducted on phones and computers, the silent, remote hacker is indeed the modern equivalent of Willie Sutton – whose answer to why he robbed banks was simple – because that’s where the money is. Now that the money is in computers and transmitted by electronic networks, that’s where the money is, and there was a great deal to be said about this ongoing battle between good guys and bad guys – and the rapidly escalating arms race that makes the outcome of any one battle so uncertain.

It’s clear that this topic will continue to grow in importance in coming years.

Cryptocurrency and blockchain

There was a whole track devoted to this hot topic, and some time on the big stage. In particular, David Marcus, the leader of Facebook’s Libra cryptocurrency effort, had the center seat of the giant center stage to himself, answering softball questions from a moderator. His smooth explanations acknowledged the widespread attention and questioning the Libra effort has engendered, while giving comforting answers and putting a calm, confident face on the effort.

There was quite a contrast between all the sessions devoted to this subject and what you saw on the show floor, which wasn’t much. Given that Money 20/20 is a show about payments, and given that cryptocurrency is, after all, a kind of currency, it’s natural there should be widespread interest in the topic, fueled by a generous portion of FOMO. There are lots of efforts and noise on the subject, but still not much evidence of real action. Everyone follows and will continue to follow the high-profile Facebook Libra effort with interest.

New Technologies: AI and ML

AI and ML continue to be the buzzwords nearly everyone wants to be associated with, in one way or another. This was particularly remarkable when you walked the show floor, with a high fraction of the booths claiming leadership in these subjects. Given the arcane nature of the underlying technology and the widely diverse technologies that are reasonably described as AI or ML – combined with the fact that most peoples’ eyes quickly glaze over when a nerdish person tries to explain what makes one of the dozens of machine learning algorithms different from another – it’s clear that the marketing person has an impossible task, and most of the show attendees have a challenge figuring out which approach is better than the others.

Nonetheless, this is one of those cases where the hype is clearly justified. For example, Feedzai is catching more of the bad guys than its direct competitors because its technology is just plain better, even though the others arguably also use AI/ML.

Eventually, people will figure out that this has happened before. It’s like decades ago, when a few leading companies claimed to be better than the competition because … they used computers! Yes, those incredibly advanced things that no one really understands. It wasn’t long before everyone first claimed to be using them, and then, eventually, everyone still in business was actually using computers! This year’s Money 20/20 clearly demonstrated that we’re still in the intermediate stage, in which the claims to be using AI/ML are widespread, and the reality is still in catch-up mode. We’ll know we’re really there when it’s just assumed – if you’re still in business, you must be using AI/ML; the only question is, who does it better. We’re getting there!

Lending, payments, banking

The show contained an amazing demonstration of continuing innovation in making it easier for people and businesses to send money, get money and borrow money. The days of checking that you’re properly dressed and groomed for your important meeting with the bank lending officer are long gone. Even the days of filling out forms, not just on paper, but also on the computer. As ever-growing portions of your financial history are stored and available in various online repositories, the job is mostly making sure you really are who you say you are. Once that’s settled, a couple simple actions on your phone are increasingly sufficient to get, send and borrow money, cheaply and quickly.

We saw an array of companies from point products to comprehensive, one-stop services, coming ever-closer to making every aspect of banking as quick and painless as messaging on your phone. The competition is fierce. It’s enough to make you wonder why the big banks survive. But all the big-name players are in the game and were at the show as well, with their own approach to making transactions as easy and efficient as possible for consumers and merchants. While everyone likes convenience, change isn’t particularly convenient; the big institutions have most of the customers and aren’t going to give them up without a fight. So they’re offering their own brands of ease and convenience to their customers. It was all on display at the show, and it’s clear that the war for the customer is far from over.

Regulation

As one of the most regulated industries, payments is doing an amazing job of innovating while avoiding the long arm of the bureaucratic enforcers. This is partly because regulation technology is advancing, as we saw at the show. But it’s got a ways to go. Automating regulation is a bit like teenagers eating vegetables – they respond to parental demands, but it’s not the first choice. As regulators continue to ratchet up the stakes and catch up to modern technologies, this is a field that continues to grow in importance at Money 20/20, both in the speakers and the exhibits.

Money 20/20

Next year will be 2020. Will this show be re-named Money 20/30? Or will moving towards 20/20 vision be the metaphor that justifies sticking with the name? Regardless of the name, the show will go on. It’s the one place you can go and see everyone who is anyone, and everyone who wants to be someone, all in one place in a short period of time. We’re already looking forward to next year.

Facebook’s Libra is not a Cryptocurrency

“Everyone” says that Facebook’s Libra is a cryptocurrency. Long before Libra had been imagined, Bitcoin pioneered and established the brand new world of cryptocurrency. Bitcoin created the category, and has always been its leading exemplar. The white paper by the still-unknown Bitcoin creator and inventor spelled out his design goals and the main aspects of Bitcoin that supported those goals. Once you read and understand what cryptocurrency is, it becomes very clear that, whatever Libra may be, it is NOT a cryptocurrency. To claim that it’s a cryptocurrency is like claiming that a locked desk drawer is a bank vault – yes they both have keys and are supposed to keep things safe, but other than that…

Satoshi, the brilliant creator of Bitcoin, designed a currency that involves cryptography. If you want to be extremely loose, you could say that Libra is the same thing, because it’s also a currency that somehow involves cryptography. But that’s like saying that the thing you use to “buy” properties and hotels in the board game Monopoly is “money.” Try depositing some of it at an ATM and see how far you get.  Let’s explore the basics of what makes a cryptocurrency the way Bitcoin is a cryptocurrency.

First and foremost, there’s the concept that in Bitcoin, no one is in charge. How can you possibly make a computer system that works, does lots of computing, keeping lots of financial transactions and makes sure everyone’s account balance is correct … without anyone being in charge?? These things are hard to do when someone IS in charge! There’s quite a bit involved in making this happen, as I illustrate here, but here are some of the key points:

• Anyone who wants to can sign up to be a “miner,” who are the folks that make Bitcoin work.
• A miner has to put money into buying fast computers, running the mining software, and connecting with all the other miners to share work.
• Miners get new transactions that Bitcoin users want to perform and “make them happen.”
• This means that miners race each other to solve complex problems involving cryptography, the net result of which is a new page (block) of transactions that have been vetted, and “locked” by crypto-key.
• Every piece of work a miner does is paid for by newly-minted Bitcoin – the miners are paid with Bitcoin!
• Miners are highly incented to do the work and do it right, because they want to get lots of Bitcoin, and they want Bitcoin to continue to be viable.
• Miners come and go as they see fit – no one “approves” them, literally no one’s in charge.
• Miners can be anywhere, in any country.

Big corporations and regulators don’t like the unsupervised free-for-all of Bitcoin. They like to control things. And that’s exactly why Bitcoin was invented – to escape the control of a central authority but still have a system that works. It’s a brilliant concept, and Bitcoin’s success shows that it works.

Along comes Facebook and Libra. Facebook is ambitious. They keep trying to invent new things. They mostly fail when they build things themselves, so they buy companies instead. Facebook would LOVE to buy Bitcoin – but it’s not for sale, because no one owns it – darn! They’re forced to try to build it. But being a big corporation, they just can’t stop themselves from building their version of Bitcoin in a style that makes them comfortable – violating every single core principle of Bitcoin – the original cryptocurrency – along the way!

Here’s what Facebook is doing with Libra:

• In Bitcoin, literally no one is in charge. With Libra, Facebook is designing and building it. Facebook is in charge and owns it.
• Facebook has gone to considerable lengths to create the illusion that it’s not in charge with this fake Swiss-based consortium of prestigious companies that supposedly control things. Either way, some combination of big name-brand companies are in charge, which is pretty far from Bitcoin’s really-truly NO ONE is in charge.
• Just like Facebook owns and controls all the computers that run Facebook, Libra will own and control all the computers that run Libra in a private data center. To all the corporate computer types, this is a good thing, but it totally and completely violates a core principle of Bitcoin, leaving it open to the same kind of insider corruption that all such places are rife with. It’s also a silly idea, as explained here. Microsoft and Intel explain the issues here.
• One of the less pleasant side effects of Bitcoin’s miners and what they do with cryptography is the fact that “proof of work” takes time. It’s a cornerstone of getting all these strangers to play nice and do good things, but it takes a number of minutes to complete a transaction. To Facebook, this is unacceptable. So they’ve blithely discarded the key cryptographic cornerstone of Bitcoin, and replaced it with some light-weight encryption, so they can still say they’re a “cryptocurrency,” even though they’re not.

There’s more to be said, but that should be sufficient to make the basic point that Libra is a cryptocurrency the same way my cousin, who is sometimes allowed to sing in bars, is an opera singer. My cousin likes to think she is, and I’m nice to her. But she’s never so much as attended a performance at the Metropolitan Opera in New York, much less appeared on stage in front of an audience. Similarly, Facebook’s Libra likes to think it’s a cryptocurrency even better than the original, Bitcoin, but it swore off the core principles of Bitcoin from the start, and doesn’t deserve to be called by the same terminology.

Note: This was originally posted at Forbes.

Computer Science is Propaganda and Computer Engineering is a Distant Goal

To call what is taught in the “computer science” departments of universities a “science” is a mind-game to get everyone involved to believe that what is taught meets the normal criteria for being a “science.” It doesn’t come close. Well, you might say, some of those departments are more humbly and accurately called “computer engineering.” True. At some point in the distant future, what is taught in computer engineering might rise to the level of what is taught in, say mechanical or electrical engineering. Until that goal is in sight, it would be more accurate to call the classes something like “Fads, fashions and sects in computer software practice.”

Physics, Chemistry and Biology

I hope we can all agree that physics, chemistry and biology are sciences. It wasn’t always that way! They have only gotten to be sciences after long struggles. Physics started emerging about 400 years ago, chemistry a couple hundred years ago, and biology just in the last 150 years or so. In each case, they are studies of reality, with generally accepted statements of how that reality works, as shown by many experiments. In each case, they advance by someone making a hypothesis that can be dis-proven by experiment. If the hypothesis is supported by experiment, more tests are done by many people to refine it, and then it becomes part of the accepted science. Sometimes a new hypothesis contradicts something that’s accepted, but more often refines it – for example, the best way to understand most of Einstein’s work is that it refined Newton’s for special, highly unusual cases. In the vast majority of cases, you can safely use Newton’s laws of motion without being concerned about relativity theory.

How does Computer Science stand up to these paragons of science?

In physics, you learn the rules of matter, energy and motion. In chemistry, you learn first of all, the periodic table of elements, and then all the molecules into which they assemble themselves and how they interact. In biology, you learn about all the living things, from viruses, through plants and animals. What’s the equivalent in computer science? You can say, “oh, it’s the science of computers;” except that physics, chemistry and biology are things that exist in the world. Humans didn’t create them. Computers are 100% human creations, no less than spoons and baseballs. There is no science of spoons – there is a bit of history and a range of modern methods and styles of making them, just like computers. Until we decide that it’s OK to create an academic department of Tablecloth Science, we should be able to agree that there is no such thing as Computer “Science.”

How did it happen that loads of departments with courses about obviously bogus Computer “Science” come to be?

The innocent explanation is that, in the early days, computers were closely related to math, and were seen basically as giant programmable calculators. In fact, the word “computer” was originally the name of a person, almost always female, who “computed” the answers to math problems. While math isn’t a “science” in the normal sense of the word, it is a kind of pre-existing non-physical reality that, in ways and for reasons that have never been explained, pervades and underlies everything about us and the world we live in. It’s the grounding of all of science. Computers were often studied by the math people in academia, and the precision naturally associated with computers seems to justify the association with science.

But in the end, computers are just fancy machines that people design and build to do stuff. How would you feel about “Refrigerator Science?” Refrigerators are great, and I like what they do. The advances in Refrigerator Science since we used to call them “ice boxes” is amazing. Uhhh, maybe not.

The less innocent explanation is that everyone involved realized that no way is there such a thing as computer science, if we’re at all serious about the term “science.” But there’s no doubt that it makes everyone involved feel better about themselves, so as propaganda, “computer science” gets an A+.

OK, OK, We’ll call it Computer Engineering

Many places do call it Computer Engineering. Is that any better? Think about mechanical engineering, for example. Let’s look at my favorite example of building bridges. I’ve gone into huge detail about the differences between bridge-building in peace and war and how it applies to software.

Let’s step back and compare the normal peace-time bridge engineering project with the normal computer software one. At the outset, they seem pretty similar. They start with requirements, then an overall design, then build, testing and inspection and finally production. In fact, many engineering-type methods are used in software, including project management.

For bridges, it works out pretty well. Bridges get built and work, 24 by 7, year after year, with routine maintenance. There are exceptions, but they're rare. Software? Not so much. Just to hit some highlights, the problems include:

• Bridge-building project management methods are sound and generally produce reasonable results. Software project management methods don’t work
• Bridges are generally built in-place, so installation is an integral part of the build process. Installing standard software is a nightmare
• Once built, bridges work. Software QA is broken
• Bridges just keep working. Software is full of horrible bugs
• No one stealthily sneaks onto bridges and steals the cars that are on it. Software is full of horrible security holes
• Bridge-building is driven by solid, proven engineering practice, backed by scientific principles. Software is driven by fashion instead of sound practice

Given all this, I guess we can still call it Computer Engineering. But to be fair, we should have it on probation, and call it “Computer Sadly-Deficient Engineering” until it rises to the level of mediocrity -- which it may, with some luck, someday achieve.

Conclusion

Computer Science is not a "science." Not close. To call it science is propaganda, fraud, ignorance, whatever. Computer Engineering is another matter. Computer Engineering, both as taught and as practiced, is horrible. The methods, largely lifted from other disciplines, just don't produce good results most of the time. There isn't even a movement that recognizes this and agitates to fix it!

Happily, the solutions -- good computer engineering practices -- exist and have been proven in practice, many times over by different groups in different times and places. I have discussed what I know of these methods extensively in this blog and in books. Top programmers discover large parts of them on their own, and use them to achieve stellar results -- outside the purview of corporate types, regulators and bureaucrats. For entrepreneurs versed in these methods, it's a serious competitive advantage, while the crippled, lumbering giants of software shuffle along.

Microsoft And Intel Detail The Deep-Seated Problems With Blockchain

Both Microsoft and Intel are big supporters of blockchain. They think it's going to be "bigger than the internet," contributing trillions of dollars to the economy before long. At the same time, they spell out the overwhelming obstacles blockchain must overcome to reach this pinnacle of achievement. Guess what, surprise surprise, the special version of blockchain created by Intel and Microsoft is indispensable to solving the problems and achieving success!

You can see their deep thinking here and here. Before diving in, I'd like to point out that the custom, private blockchain they advocate is a contradiction in terms, as I illustrate here -- even if they implement what they claim perfectly, it will still be a joke.

Here are a few of the little obstacles that blockchain has to overcome before becoming acceptable for enterprise use, according to Microsoft and/or Intel:

• Performance: Normal blockchain performance is a few transactions per second. "Reed said the trusted execution environment of Intel SGX enables Coco to deliver a novel consensus mechanism that can deliver up to 1600 transactions per second..."
• Confidentiality:Normally, everything on a blockchain is public information.  "Microsoft uses Intel Software Guard Extensions (Intel SGX) to protect the Coco Framework. Reed said the trusted execution environment of Intel SGX ...  helps Coco transactions remain confidential among blockchain participants."
• Governance: With a normal public blockchain, no one is in charge. This doesn't come close to meeting enterprise requirements. Microsoft's private blockchain enables classic management, access controls and all the rest.
• Processing power:Intel says "Public cryptocurrency blockchains require huge amounts of energy to verify transactions through node consensus. Analysts have estimated a single bitcoin transaction can require as much energy as the average American home uses in a week."

The other big vendors, like IBM with its team of 1,500 people working on its Blockchain effort, have similar stories about what's wrong with Blockchain and why you should use theirs. When you add it all up, it does make you wonder about this revolutionary new technology, and exactly why important new initiatives should depend on this brand-new, largely untested code that obviously was not built with practical, enterprise use in mind.

This was originally posted at Forbes.

Software Professionals Would Rather Be Fashionable Than Achieve 10X Productivity Gains

I’ve been involved in a large number of pioneering software efforts, and I’ve lived through many tech fashion seasons. It’s very clear that the vast majority of experienced software professionals and managers would rather follow current technology fashion than actually making things better. Because of the pathetic state of software knowledge and analytics, not only do they get away with throwing away truly massive gains for their organization, no one in power has a clue what they’ve done! This happens over and over and over and over again. A few people may know there’s a vastly better approach available – one that’s proven and risk-free – but no one hears them. The siren song of the hot new thing wins most every time.

I have been involved in many such farces in one way or another. Here’s the story of one of them from many years ago. See this and this for general background of the technologies and patterns. It's important to remember: I'm not saying that Sallie Mae was a disaster when everyone else was fine. Sallie Mae was a normal, mainstream project at the time, and therefore an excellent example of the overall insane pattern.

The Workflow project at Sallie Mae

In the early 1990’s, document imaging and workflow technology were hot; they were something people talked about the way they talk about AI/ML and innovation today.

Sallie Mae, then as now, was the government-backed student loan organization. In the early 1990’s they decided to upgrade their computer-based processes to something that was modern in order to make things more efficient. At the time they had over 10 million paper documents per year arriving at their processing centers. They had six of them at the time, each employing thousands of people. Even a 10% productivity improvement would have been big at that scale! Sallie Mae managers had read about the new document imaging and workflow technology that was rapidly growing at the time, and set up a task force of professionals to study it, create an RFP and get it implemented.

The task force knew they weren’t experts at this technology, so they sought out people to help them. Among other things, they attended the AIIM industry association’s annual convention, at which I happened to be a featured speaker due, among things, to my book they had just published. Here is the forward to the book, written by AIIM:

I had also led the technology at one of the industry’s tech vendors, had personally written their workflow software, been involved with major buyers including Amex, etc. So they approached me and asked me to help them out. I agreed. They also contracted with a small consulting firm to join in.

I dove in to analyze the situation. Sallie Mae had loads of IT people who were concerned about the new technology. The existing technology was basically IBM mainframes with many thousands of 3270 terminals on people’s desks. Implementing the new technology would involve buying expensive workstations with large image displays for each worker. The IT people worried about the LAN that would be needed to connect the workers to the image storage system, so I created a spreadsheet to enable them to calculate the transmission speeds that would be needed. While the other consultants were busily drawing up diagrams of paper movement so that workflow could be implemented, I dove into the details of what actually happened at a couple representative workstations.

I observed some people at one of the processing centers doing the work of processing the form. I timed the work and carefully observed the screens. I read the manual that was used to train the workers for the job. I arrived at a pretty radical (for the time) hypothesis: the vast majority of the work was logic-enhanced data entry. If the humans did no thinking but just entered the data, they could work 10X faster. The logic spelled out in the training manual could all be programmed on the data as entered, with a tiny number of exceptions kicked out for human handling.

If this could really be achieved, it had consequences for Sallie Mae. Instead of a massive per-employee technology investment with the work remaining roughly unchanged, a small fraction of the employees could do the same work in a completely new way. I figured I better make sure I was right before I started rattling cages and maybe embarrassing myself.

Logic-enhanced heads-down data entry at Sallie Mae

I knew that everyone assumed that implementing workflow was the goal, with productivity improvements assumed to be 30% after massive investment and disruption. This was broadly accepted, and no one involved was interested in challenging it or testing it. If there was any chance of them taking a new approach, I knew I better have the numbers.

The main things I discovered were:

• Most people entered 1,000 to 1,500 KPH (keystrokes per hour).
• Roughly 20% of those keystrokes were overhead, i.e., navigating to the next field on the mainframe screen, often navigating to a whole new screen. The mainframe screens followed how the data was organized inside the computer, not at all how it appeared on the paper forms.
• The workers weren't bad typists -- they were obviously doing some thinking between fields, essentially applying the special rules they were taught in training to guide them what to enter where.

I refreshed my knowledge of practical image-enabled heads-down data entry (HDDE), See this for an explanation and details. I laid out some plans.

• Image-enabled HDDE could reasonably achieve 12,000 KPH, as it was doing at multiple high-volume operations. Moreover, the navigation overhead could be brought down to a couple percent.
• The training manual was amazingly close to a natural language description of a program. If this field on the form has that, then enter X in Y mainframe field, and so on.
• The current method interwove the entry of the form with the rules. In the new system, all the data would be entered exactly as it was on the paper form, and then a new set of software would apply the rules and enter the data into the mainframe system. When the rules changed, instead of everyone taking a training class, the software would just need to be changed.
• Some forms in the existing method were sent to a customer service person for exception handling. Much of that would still happen, but the numbers were low.
• I knew that Image character recognition could handle some fraction of the data entry work. But it was a new technology, and I figured a 10X productivity improvement and avoiding a massive technology investment for workflow was more than enough to make the case.

I wrote up my results. Since “re-engineering” was a popular concept at the time, and since “zero-based budgeting” was talked about a fair amount, I wrote a paper and called my approach “zero-based re-engineering” to explain the approach and link it to fashionable things.

My involvement with Sallie Mae ended, quickly and quietly. No one objected to my proposal. No one questioned my analysis or numbers. There were some vague comments thanking me for my creative, detailed work. But I was invited to no more meetings, got no more assignments, and that was that.

What happened was that the workflow-at-Sallie-Mae train had long since left the station. My work was nothing but a distraction, with the potential of becoming an embarrassment if things went badly. I had demonstrated that I wasn’t a “team player,” and in most organizations, there’s little worse than that.

Conclusion

Like most history, the workflow project at Sallie Mae in the 1990’s doesn’t matter at all. It’s history! Its value is that it’s a typical example of a recurring pattern in software implementation and evolution. Here is a more recent example. Sallie Mae was not unusual at the time -- they were absolutely typical! Yes, they had an "insider" recommend a simple, radically better way of doing things and rejected it -- but that was the overwhelming mainstream pattern. By recognizing the pattern for what it is, the few who care can learn from it, recognize the pattern when it’s taking place, and maybe even do things better.

Most organizations will almost always follow the tech fashion rather than achieve dramatic gains using proven but unfashionable technologies. Many entrepreneurs are part of the problem, since they need to sell into buyers who value fashion over real, objective, provable value. The best entrepreneurs find a way to fit in with buyers’ hunger for fashion, while also delivering real value. Sometimes they cloak the value in super-attractive, flimsy outerwear. That's OK. They’re entrepreneurs – they find a way!

Surprising Bugs at Amazon Show the Fragile Foundation of AI

I promise I didn't plan it this way, but when I looked on Amazon for a product to help me deal with an infestation of bugs, I encountered a major ... yes, bug. I described the bug in detail here, and at the time thought it might be isolated -- after all, in my all-too-extensive use of Amazon, I had never before encountered such a bug.

Not long after, as the issue continued to "bug" me, I went onto Amazon again, and found more interesting bugs. This could be just some scamming or corruption. I'm taking the trouble to describe what I discovered because it's exactly the kind of data that modern systems are based on, and by playing with the data, people with bad intentions can cause the systems that depend on that data to come out with the results the bad people want. It's a way of causing destruction or stealing that's indirect, effective, and likely to grow as evil-doers catch onto the technique.

The new bug

I went into Amazon looking for a pest repeller again.

This one looked pretty close to the first product I saw -- same vendor, same product appearance, nearly the same language. But the price was different, and the number of reviews, while huge by pest repeller standards, was about half that of the original listing.

I went on, and the questions & answers looked like they were real, and for the product itself. Maybe Amazon has fixed things!

Let's scroll down more and see.

Whoops!

Still an astounding number of positive reviews, but look at the categories and things mentioned. Children's books again, this time with religion and a magic tree.

But the pictures are more creative. Look at the second one above -- clearly snitched from some product far, far away from children's books! One of the leading reviews makes absolutely clear that some serious messing with Amazon's data has taken place:

The only way this is about pest repellers is if you consider the devil to be a pest. No, I don't think so.

Implications

This is a bug. And some serious messing with data inside Amazon's systems. And a GAPING hole in Amazon's security apparatus, showing us yet again that effective cybersecurity is NOT about getting certifications and passing tests devised by government lawyers and bureaucrats.

The direct implications of this are pretty small to the average person. But if best-of-breed Amazon can be hacked this way, what about the labs where data is collected and stored concerning our health? What if, instead of just stealing the data to make a couple bucks selling it to scammers, someone decides to mess with the data to achieve an outcome, like happened here? In cases like this, the data is manipulated to get buyers to see a product based on the large number of positive reviews. The same thing could be done for a drug, a therapy or a health center! Going deeper, the data could be manipulated to impact "knowledge" of the kind that AI/ML discovers and implements. Even more deviously, it could have a devastating impact on "personalized medicine," the most data-driven of all, with a screwed-with set of data that says that a certain innocent-sounding treatment would be just right for someone like you -- except, for someone like you, it's precisely personalized disaster.

In order for super-charged AI/ML engines to actually do good things, we have to make absolutely sure that their data foundations and underpinning are sound, secure, and un-manipulated. Sadly, this is not a "what-if" issue. For details, see this. The assurances of the usual serious-sounding, credentialed experts with stentorian voices and/or soothing manners are not enough. Not even close.

Surprising Bug at Amazon About a Bug Repeller

I have found ample reason to mock the golden-glowed tech reputations of most of the tech giants, in addition to the supposed tech prowess of organizations such as the NSA. There is good reason to believe that old-style libraries are more secure.

I recently stumbled upon a rather glaring bug or result of hacking at Amazon -- and found that Amazon provides no way I could find to report the problem.

The problem was glaring and amusing -- a whole set of over 3,000 reviews of a book attached to a pest repelling product. As I'll describe in a future post, this isn't just a bug concerning bug repellers -- it's the tip of an iceberg about the foundations of AI/ML.

Here's how I found the bug bug.

Stumbling on the bug at Amazon

I was looking for a product to scare away some pests that have found their way into my house. I gave a close look at the following product:

I immediately noticed and was impressed by the large number of reviews, and how favorable they trended. What a popular product! I've got to look at this one. I scrolled down:

Things are still looking good. More details:

Uh-oh! Right after we see that it has 3,051 reviews with an average rating of 4.8, which is super-good, we see that this pest repeller is #2,695 ... in Books!! Something is seriously wrong. Scrolling down, we see some really weird answers to questions:

A couple of the people answering clearly tried to tell Amazon there was a problem.

This next one blew me away -- over 3,000 reviews for a pest repeller??!!

Now we know for sure something's seriously wrong -- a pest repeller is not Children's Literature!

Here is the nail in the coffin, the pictures of the product and a list of the phrases that frequently appear:

 

And here's one of the glowing reviews. It brought tears to the reviewer's eyes, and not because of the strong odor generated by the pest repeller:

 

What's going on, Amazon? I don't usually get to mock you. It's usually the other tech giants that receive my sarcasm, places like Google, Facebook, Twitter, Apple and Microsoft. Amazon has done a reasonable job maintaining quality and growing into new areas. This is such a peculiar bug -- it's as though some disgruntled employee were messing with things to show his displeasure with the world.

As we'll see in a following post, this is not an isolated bug, and it has implications that go far beyond Amazon -- implications concerning the glorious future promised by all the AI/ML enthusiasts.

Laser Disks and Workflow Illustrate the Insane, Fashion-Driven Nature of Software Evolution

Computers are so exact and numbers-based that we tend to think of the people in charge of them as white-jacketed scientists with esoteric knowledge driving towards calculated optimal results, much the same way we imagine that the scientists and engineers calculated the path of the Apollo capsule to the Moon. If the Apollo program were run the way most computer projects are, it would have been a miracle if the capsule made it off the launch pad, much less traced an incredibly exact path from Cape Canaveral to the chosen landing spot on the Moon, a journey of over a quarter million miles.

The reality is that the application of computers and software to automation is painfully slow, usually failing to achieve optimal results by large margins, and often lagging behind what is achievable by decades. The reasons appear to be a semi-random mixture of commercial interests, technology fashion trends, ignorant and risk-averse managers, and technical specialists rooted in the past and wearing blinders. That’s all!

Here’s the story of a typical example.

Microfilm, document imaging and laser disks

Long before computers, microfilm was the preferred medium of librarians and archivists to preserve paper documents for future use. Microfilm was more compact than paper, and lasted much longer. What happened when computers came along is a curious and educational story, a prime example of how computer use and software evolve.

Disclosure: I lived and worked as a participant in this history during the late 1980’s and early 1990’s. The story I will tell is not systematic or comprehensive, but like a traveler’s tale of what he did and saw.

A side show in the long evolution of computer storage is the laser disk. A laser disk has data recorded onto it either as a whole, during manufacturing, or incrementally, as part of a computer system. The laser disk had quite a run in the consumer market in the form of CD’s, and then DVD’s. They were an improvement on existing methods for distributing digital content of all kinds.

In the computer industry, they took the form of WORM (write-once-read-many) drives, and were used to record archival data that should not be updated, but could be read as often as needed. The technology, as often happens, went searching for problems it could solve. In an effort to reduce human labor more than microfilm could, jukeboxes were invented. Each jukebox had a disk reader and many shelves for disks, with a mechanism to load a selected disk into the reader. FileNet, now part of IBM, built one of the first of these systems in the mid-1980’s.

At around the same time that WORM drives became practical, paper document scanners were evolved from systems to capture images of the paper onto microfilm. In each case, light was shown onto paper and focused onto a target; a scanner replaced the film with an optical sensing device.

How could we get people to buy WORM drives and jukeboxes, the creative marketing people wondered? The departments that put documents onto microfilm weren’t going for it – the new systems were much more expensive, microfilm wasn’t accessed often enough to make it worth connecting it to computers, and the records retention people were understandably skeptical that any computer disk would be readable 100 years from now, as microfilm will be.

Adding workflow to document imaging

I have no idea who made the audacious leap, but some creative person/group came up with the idea of doing computer document scanning at the start of the process, when paper arrived at a location for processing, instead of microfilmed at the end for archival purposes. But WHY??? The creative types scratched their heads until they were losing hair rapidly. Why would anyone make responding to loan requests or whatever even longer than it is today by introducing the step of scanning?

GENIUS TIME: LET’S INVENT “WORKFLOW”!!!

Everyone has an image for what “workflow” is. For documents, it’s the path of processing steps from arrival to filing. In larger organizations, there are often multiple departments involved in processing a document, so there are in-boxes and out-boxes and clerks transporting documents from one to the other.

Rubbing their hands and cackling, the scammers worked out their strategy: We’ll scan the documents and convert them to images so that we can move images from place to place electronically instead of piles of paper! We’ll call it “document imaging workflow.” It will deliver massive benefits to the whole organization, to everyone who touches the paper, not just to the tiny group who files and archives at the end – and storing the scanned documents onto WORM drives will do their job too! Hooray! We will leap-frog the old-fashioned, paper-flooded back office into the modern electronic age. What’s not to like?

It was audacious and brilliant. It was a scam that would wither if confronted with just a little common sense or cost accounting. Which, true to the pattern of such fashion-driven trends, it never had to confront!

Implementing workflow

A typical target environment for implementing workflow was large offices with large numbers of documents, often forms of some kind, arriving every day. The forms would go from the mail room to the relevant people’s desks for processing. The person handling the form would often have an IBM 3270 terminal for interacting with a mainframe computer. When the person was done with the form, they would place it in one or more outboxes for further processing at other desks, or for filing. Sometimes documents would go into file cabinets for a period of time, but all would end up being archived, and sometimes microfilmed for permanent storage.

Implementing workflow first of all required scanning the documents as a first step, and storing them immediately on normal computer disks, and ultimately on WORM drives, thought to be the equivalent of microfilm. Once scanned, the documents would be managed by workflow software, which would route them to the appropriate desk in the appropriate department for processing. Every desk needed to have the old computer terminal replaced or augmented with a large, high-resolution image terminal, capable of handling at least the display of the document, and sometimes also the computer screen. The old, slow connections between terminals and mainframe needed to be replaced with a fast local area network, and there needed to be substantial servers for handling the local image handling and workflow routing.

Of course, lots of analysis and programming was involved in addition to the equipment. Workflows had to be analyzed and programmed, and all the management, reporting and exception handling taken care of.

The benefits of workflow

When a movement like document imaging workflow has a head of steam up, no one seems to ask whether it’s a good idea, and if so, how good of a good idea it is – quantitatively. When I was involved, everyone threw around there would be at least a 30% productivity improvement due to workflow, and that would make all the expense and disruption worth it. I never encountered a single study that measured this.

Think about it for a minute. Would looking at an image of a document make you work faster than you would looking at the original paper? Would picking the next paper from the in-box be slower than getting the system to show you the next image? What about the labor of the clerks moving stacks of paper from one person’s outbox to the next one’s inbox? It would certainly be saved, but chances are a single clerk could handle the paper movement for many dozens of people. And remember, there’s the scanning and indexing that’s been added and the massive computer and software infrastructure that has to be justified.

It’s obvious why no one EVER did a cost-benefit analysis of workflow – the back-of-the-envelope version easily shows that it’s a couple zeros away from making sense.

I remember a couple of out-of-it, tech-fashion-disaster know-nothings mildly thinking out loud about how workflow could possible be worth it financially. The immediate response from workflow fashion leaders was upping the ante – don’t you know, the wonderful workflow tool from (for example) FileNet lets you program all sorts of efficiencies for each stage of work; it’s something we really need to dive into once we get the basic thing installed. End of subject!

So who fell for this stuff? Just a who’s-who list of major organizations. Each one that fell for it increased the pressure for the rest to go for it. None of them did the numbers – they just knew that they couldn’t fall too far behind their peers.

Conclusion

It’s hard to think of a field that’s more precise and numbers-oriented than computers. Who has a clue what actually goes on inside those darn things? And the software? There are millions of lines of code; if a single character in any of those lines is wrong, the whole thing can break The impression nearly everyone has, understandably, is of a field that is impossibly precise and unforgiving of error. When the software experts in an organization say that some new technology should be adopted, sensible people just agree, particularly when there’s lots of noise and other major organizations are doing it. It can be even more gratifying to get known as pioneering, as one of the first organizations to jump on an emerging tech trend!

Somehow, the smoke and noise from the software battlefield is so intense that the reality is rarely seen or understood. The reality, as I’ve illustrated here, should result in everyone involved being sent to the blackboard by a stern teacher, and being made to write, over and over: “I pledge to try harder to rise above the level of rank stupidity next time.”

Simple Data Entry Technology Illustrates How In-Old-Vation in Software Evolution Works

Since when is “data entry” (entering data into a computer) a pivotal, innovative technology? When the difference between doing it the normal way and doing it with advanced technologies is a …ten-to-one productivity difference … that’s when.

I’ve described how the Operations Research algorithm of Linear Programming is fifty years into an agonizingly slow roll-out through different applications, from scheduling oil refineries in the 1960’s to scheduling retail sales in the 1990’s, and now scheduling medical infusion centers and operating rooms in the late 2010’s. In each case, laborious and error-prone human scheduling was replaced by the algorithm, with improvements ranging from no less than 10% to over 50%. This is major! Why did such an innovation wait for decades to be applied, and for many applications, is still waiting!!?? This is the mystery of how what’s called “innovation” works in reality, and why it should be called “in-old-vation” instead.

You may think that part of the cause is that LP is an exotic algorithm – even though it’s a standard part of the Operations Research engineering curriculum, most so-called normal people haven’t heard about it. While it appears that even the hosts of people who wax eloquent about AI and ML are clueless about LP, it’s not exactly a secret. So let’s see if obscurity is the reason why LP remains a “future innovation” in many potential applications, by examining the super-plain, ordinary, completely-understandable-by-normal-people case of data entry.

Data Entry

Just as process optimization is done by hand or stupid methods for decades until some genius comes up with the brilliant idea of applying tried-and-tested LP to the problem and dramatically improves it, so is Data Entry widely performed by primitive methods until some “innovator” comes along and applies “Heads-Down Data Entry” (HDDE) methods to the process – and typically gets improvements of 3 to 10X! The only difference is that while LP is taught in Engineering departments and studied by math nerds, Heads-Down-Data-Entry is “just” a collection of common-sense techniques that require no math and no professors to understand or implement. It’s so “common” that it doesn’t even have a generally accepted name – though it’s been implemented in many places and been thoroughly proven in practice. It’s far too humble to merit an academic department – and yet, when applied, has delivered truly massive gains, far higher proportionately than exotic Linear Programming has!

The methods of HDDE were first implemented in places that had huge volumes of repetitive data to be entered into computers. Banks were early users for check entry, and so were the credit card companies who, at the start, had huge volumes of paper charge slips to process. Simple ideas like minimizing keystrokes and eye movement were implemented, and then taking advantage of the eye-to-fingers pipeline, when people noticed that showing clerks the next item to be entered before the current one was complete led to a big jump in speed. Other methods like double-blind techniques were invented, so that entry clerks just entered – whether their work was original or used to check someone else’s work was entirely handled by the Data Entry system.

As soon as scanning and image display became practical, HDDE adopted them. That led to another jump in productivity, enabling large, complex forms to be broken up into pieces, so that a clerk would see the image on a screen of the same piece of data from a whole set of forms instead of entering a whole form from start to finish. No HDDE shop would even consider having the entry clerk think about anything on the form, stuff like “if this field is missing, do this instead of that,” because it would just slow them down.

Finally, there’s ICR (Image Character Recognition), which is having the computer “read” the image instead of a human. This technology has existed for many decades. Once you’ve got HDDE in place, phasing in ICR is a natural, so that the proportion of entry done by humans gradually decreases as the effectiveness of ICR increases.

Remember, applying LP to scheduling might result in a 30% improvement, which in most cases, is major to the point of being revolutionary. What about HDDE? Entering data from a paper form into a computer using primitive methods might get between 1,000 and 1500 KPH (keystrokes per hour). There are lots of stages of improvement, including things I’ve mentioned like eliminating the thinking and breaking up the form, but levels of 10,000 to 15,000 KPH in a professional environment are widely achieved – with superior quality. That’s a minimum of 5X! Typically much more. Of course, as you incorporate ICR into the process, it gets even better, gradually reducing the human factor, so that most fields are entered with no human involvement. At this point, the technology is probably best called ICR+HDDE, though there is no generally accepted term.

Given all this, it would be insane to handle computer data entry by anything other than HDDE methods, right? Welcome to the software industry, where insanity of this kind is the accepted state of affairs. And where almost no one practices the most simple and basic of computer fundamentals, such as counting.

How HDDE gets implemented

I described how Linear Programming went from problem domain to domain, each time acting like an innovation, as indeed it was in that area of application. Once it gets established, it tends to stay. The case of HDDE is different, I think because it’s not a recognized “thing” in the halls of academia, or among the poo-bahs of big business. It’s the kind of thing that no self-respecting Professor of Computer Science would stoop to consider, assuming he ever encountered such a low-status thing – you know, the kind of thing that “merely” makes common sense and, well, works.

HDDE has appeared in competitive, high-volume service businesses, where it has a major role to play in delivering results for the customers of the business. There have been software products that directly support HDDE, so that all you have to do is buy and implement them. It’s neither obscure nor hidden. But it’s never been talked about at conferences as the “coming thing.”

Case Study: HDDE rejected

In the early 1990’s, when document imaging and workflow technology were hot and something people talked about the way they talk about AI/ML and innovation today, the government-backed student loan organization, Sallie Mae, decided to apply the technology to improve the operations at the handful of processing centers they had at the time, employing many thousands of people and processing millions of documents a year. The popular thing to do at the time was to scan documents on receipt, and then send them to the same places the paper was sent, so that workers could process the images of the documents displayed on new big screens instead of paper. The job was basically to type the data into the right places of the software application they used.

Everyone at the time said that converting the documents was important ONLY because it enabled wonderful workflow, the elimination of inboxes and outboxes for paper. And the bits of other stuff you could do, like having a group of people taking from a common inbox instead of each having their own. The common “wisdom” was that you could gain 30% productivity improvement by implementing this marvelous new technology.

I got involved, since I was a recognized expert on document imaging technology at the time, and had personally coded one of the early workflow systems. I figured out and showed in detail that by canning the workflow and implementing HDDE techniques, they could gain a minimum of 5X productivity improvement. No one disputed my thoughts or detailed plan. They just ignored it, and proceeded to implement the standard stuff. I strongly suspect that after considerable time and expense, there were walk-throughs of the Sallie Mae sites showing visitors the big screens and absence of paper – what a big success the project was!

Case Study: ICR-HDDE applied with success

There are two current cases I know of where ICR-HDDE is being applied and winning. Each are classic, narrow service businesses where converting forms to data is the key value of the business, and where the companies buying the service just want fast, accurate data delivery, they don’t care how it’s done. Disclosure: each of these is an Oak HC/FT investment.

At Groundspeed, insurance forms and reports are captured and the relevant data is extracted from them by the most effective relevant means, often involving forms of ICR-HDDE. There is lots of forms recognition from documents and images that are often computer output, with the relevant data appearing at varying places on a page. Nonetheless, Groundspeed is able to deliver the data stream the customer needs, quickly and accurately, The results are so powerful that new levels of analytics are enabled by the newly available stream of structured data.

At Ocrolus, financial documents of all kinds including bank statements and pay stubs are converted to data in a standard format to enable fast and effective operations like giving loans for business and personal use, along with a growing list of other operations that also need good data. An effective combination of ICR-HDDE techniques are applied to get results for companies that need accurate data to make fast decisions.

Conclusion

HDDE is a collection of methods that have been proven in practice for many decades. The technology that is behind it continues to deepen and reduce human effort even more, with the addition of ICR. But it remains a niche technology, ignored by the numerous places that could benefit from it, even more than LP.

The big difference between LP and HDDE is that LP is a formal piece of magic that’s in academia. HDDE is nowhere. In fact, it’s really just an example of classic industrial engineering applied to computer software and the people who use it. Which makes it all the more mysterious that it's largely ignored.

In-old-vation is real. Most “innovations” are minor variations on things long-since proven and demonstrated in practice, but are unimplemented in the many situations that would benefit from them until some mysterious combination of circumstances arises to let them explode into practical reality.

Facebook's Libra Cryptocurrency and the P2P Apps Venmo and CashApp

Facebook is working hard on building a brand-new cryptocurrency system called Libra, sort of like Bitcoin and Ethereum, except it will be much better, at least according to Facebook.

With all the talk about Libra, cryptocurrency, regulation and the rest, no one seems to wonder about what existing solutions normal people will be using to solve the problems for which Libra is suited. This isn’t strange at all actually – in all you’ve read about Facebook’s Libra, how much have you read about the pressing problems it will solve, the unmet needs it will address – right? Mostly what you read about is how Libra will solve all sorts of problems that today’s crypto-currency systems have, how many partners they have and how wonderful it will be.

Libra will be an infrastructure “out there” somewhere, with lots of important people and organizations making sure it’s wonderful. But in practical terms, most people will use it via an e-wallet, an app that they install on their smart phones. That’s where a name that hasn’t appeared a great deal pops up: Calibra. Calibra is a newly-formed Facebook subsidiary that will be the e-wallet for Libra. It will “integrate with” Facebook’s WhatApp and Messenger, giving it incredible consumer reach.

You can read all about what it will do, but it’s basically an e-wallet for holding e-cash and providing basic functions like sending and receiving e-cash to and from another e-wallet. Except for the little “detail” that instead of sending real money, you’re sending the Libra cryptocurrency, and will have to go to an additional step to move dollars in a bank account to and from your Calibra wallet, converting to or from dollars along the way.

Putting aside the fancy new terms of cryptocurrency and the rest, does using a phone to make person-to-person payments remind you of anything? How about Venmo, the P2P e-wallet used by over 51 million people, which is now part of PayPal? How about Cash App, the rapidly growing P2P e-wallet installed in about 60 million phones?

These are proven consumer applications which have already gone through numerous upgrades and feature additions, used at least weekly by tens of millions of people.

Facebook has incredible reach, and billions of cash in the bank gotten by selling your private information to advertisers. They will certainly make a lot of noise. How does Facebook's proposed system compare to Venmo and Cash App?

• Venmo and Cash App just use dollars. Simple. Facebook will use the newly invented Libra, which needs to “work,” something Facebook isn’t good at doing.
• If you split a bill and need to send $7.30 to your friend, you just do it with Venmo and Cash App. With Facebook, you’ll have to convert it to Libra, send that, and have it be converted back. Hopefully the exchange rate won’t move too much.
• Venmo and Cash App support free P2P payments. The Calibra website claims it will be “low cost,” but they have yet to say what the cost will be; after all, there is a HUGE cryptocurrency infrastructure to support, none of which is needed by the existing cash apps.
• It’s easy to imagine Facebook will find a way to sell the information about your transactions to the highest bidder, or somehow find a way to "monetize" what you do with your money. It’s what they do!
• Libra and Calibra will work for international payments! With exchanges from local to Libra to foreign currency, two exchanges instead of one. That’s certainly something Cash App and Venmo don’t do today, and will appeal to some fraction of a hundredths of a percent of the market. Except for the proven massive cryptocurrency uses for money laundering and international crime, who will have yet another channel to support their illicit activities.

Facebook's Libra is getting all the attention any giant corporation could want, including some attention I suspect they'd rather not have, from regulators. But in the end, will they be able to make this massive software effort work? Will it do anything consumers want better than existing apps like Venmo and Cash App that are in widespread use? There is good reason to be skeptical.

This was originally posted at Forbes.

Understanding Software Evolution Helps you Predict the Future of Software

There are patterns in software evolution. Because almost no one studies software history, and even fewer study software evolution, these patterns are almost never discussed.

The patterns are amazing. In some cases, you can be pretty sure that a trend that “everyone” says is going to be the future will fizzle out. In other cases, you can predict with a high degree of certainly that software of a certain definite kind and description will be built – even though it hasn’t been built yet – it hasn’t been built anywhere by anyone, but nonetheless you know it will be built.

The reason I started to study software evolution is that it’s just plain interesting. The patterns that emerge from it are striking and educational. But most important, knowing software evolution can help you predict the future!!

Knowing software evolution can help you predict the future for the simple reason that software does not evolve the way pretty much everyone thinks it does. It is often the case that, when a new software system appears on the market that has a big impact, people who know the patterns were sitting around waiting for it to happen! They knew it was coming!

I remember living in New Haven CT in the early 1970’s. I was a vegetarian. A new restaurant had just opened, a kind I’d never been to, an Indian restaurant. I went, and it changed my culinary life. Indian food – made by vegetarians for vegetarians, healthy and wonderful flavors!

The new restaurant was definitely new in Connecticut, a true innovation. The owners had to go through quite a bit to open it up, get supplies, etc. But was it otherwise innovative? Did the owners invent a single one of the dishes or recipes? No, of course not. Just like with a piece of software, they had to create it from scratch in a new setting, making appropriate adaptations, and so to the locals, it was new. But in reality, it was a newly adapted instance of a proven winner. Software works the same way – a piece of software that is a big new winner, the first of its kind, has most often been proven elsewhere.

The first Indian restaurant in New Haven could have opened years earlier or later than it did, with different people doing the work. There was no way to predict when the Indian restaurant would have opened, or who would have done it. But isn’t it obvious that, in the context of ethnic restaurants enjoying increasing success and the growing Indian population that someone would have opened an Indian restaurant there? Shockingly to most people, software works exactly the same way! In a surprising number of cases, you can’t predict WHO or WHEN, but you can sure as heck predict WHAT will be built and even WHERE (i.e., in what business domain).

Here is a specific example of software that resembles in important ways the example of Indian restaurants. Instead of Indian food, the star is a specific optimization method within the field of Operations Research (OR) called Linear Programming (LP). While most people know there are loads of Indian people in India and elsewhere and they have a unique cuisine, understanding this is helped by the fact that we can picture India on a map of the globe and we all eat some kind of food. How many people have even heard of OR or LP, or have seen and understood software of any kind? To be clear, what I mean by "having seen software" is seeing the source code, not the results of it on a screen. This LP software sits around, widely used in some domains, and completely unknown in others, just like there were no Indian restaurants in New Haven before a certain date.

Knowing this helps you predict the future of software, because in a shocking number of cases, it doesn't need to be invented, but "merely" imported into an area of use where it is not currently in use. That is a big reason to be interested in understanding software evolution.

Barriers to Software Innovation: Radiology 2

Value-creating innovations are rarely the result of a bright new A-HA moment, though an individual may have that experience. A shocking number of innovations are completely predictable, partly because they've already been implemented -- but put back in the vast reservoir of ready-to-use innovations, or implemented in some other domain. This fact is one of the most important patterns of software evolution.

Sometimes the innovation is created, proven and fully deployed in production, like the optimization method Linear Programming, which I describe here. In other cases, like this one, the innovation is built as a functioning prototype with the cooperation of major industry players -- but not deployed.

In a prior post I described how I went to the San Francisco bay area in the summer of 1971 to help a couple of my friends implement a system that would generate a radiology report from a marked-up mark-sense form. We got the system working to the point where it could generate a customizable radiologist's report from one of the form types, the one for the hand. Making it work for all the types of reports would have been easy -- we demonstrated working software, and wrote a comprehensive proposal for building the whole system. It was never built.

True to the nature of software evolution, the idea probably pounded on many doors over the years, always ignored. But about 10 years ago, a pioneering radiologist in Cleveland came up with essentially the same idea. Of course, instead of paper mark-sense forms, the radiologist would click on choices on a screen, and would usually look at the medical image on the computer screen. This enabled the further benefit of reducing the work, and letting doctors easily read images that were taken in various physical locations. Tests showed that doctors using the system were much more productive than those who worked in the traditional way. Finally, they decided that mimicking the radiologist's normal writing style was a negative, and that the field would be improved by having all reports follow a similar format, with content expressed in the same order in the same way. This was actually a detail, because the core semantic observations would be recorded and stored in any case, enabling a leap to a new level of data analytics. It also, by the way, made the report generation system much easier to build than the working prototype we had built decades earlier, which enabled easy customization to mimic each radiologist's style of writing.

The founding radiologist was a doctor, of course, and knew little about software. He did his best to get the software written, got funding, and got the system working. Professional management was hired. My VC group made an investment. Many people saw the potential of the system; it was adopted by a famous hospital system in 2015. But in the end, the company was sold off in pieces.

Nearly 50 years after software was first written that was able to produce medical imaging diagnostic reports quickly and reliably while also populating a coded EMR to enable analytics, the system is sitting in the vast reservoir of un-deployed innovations. It can be built. It saves time. It auto-populates an EMR.

Many people have opined on why this particular venture failed to flourish. It's a classic example of the realities of software innovation and evolution. The reasons for failure were inside the company and outside the company. For the inside reasons, let's just say that the work methods of experienced, professional managers in the software development industry lead to consistently expensive, mediocre results. Nonetheless, the software worked and was in wide production use, delivering the advertised benefits. For the outside reasons, let's say that, well, the conditions weren't quite right just yet for such a transformation of the way doctors work to take place.

The conditions that weren't right just yet for this and uncountable other innovations add up to the walls, high and thick, behind which a reservoir of transformative innovation and "new" software awaits favorable conditions. In other words, the reservoir of innovations wait for that magic combination of software builders who actually know how to build software that works, with a business/social nexus that accepts the innovation instead of the standard no-holds-barred resistance.

Corporations promote what they call innovation. They are busily hiring Chief Innovation Officers, creating innovation incubation centers, hanging posters about the wonders of innovation, etc. etc. They continue to believe the standard-issue garbage that innovation needs to be invented fresh and new.

The reality is that there is a vast reservoir of in-old-vations that are proven and frequently deployed in other domains. All that's needed is to select and implement the best ones. HOWEVER, a Chief Innovation Officer is STILL needed -- to perform the necessary function of identifying and breaking down the human and institutional barriers that have prevented the in-old-vations from being deployed, in many cases preventing roll-out for -- literally! -- decades!!

The Comes-before is not Causation Fallacy in Software Evolution

Practically no one understands software, so why should we expect anyone to understand the broader and in some ways more demanding subject of software evolution? But while rarely (as in, almost never) studied, thoughts about software evolution are often found lurking in the backs of people's minds, and pop out in the things they write.

In biological evolution, putting aside nasty things like DNA and all the stuff it can and can't explain, there is a now-obvious pattern in the fossil record of how species have evolved. There is a clear progression from simple to complex, less capable to more capable and so on. Cold-blooded lizards are pretty neat, but there's little doubt that warm-blooded mammals rule, and that mammals appear in the fossil record after the cold-blooded animals. The picture of this nearly all of us share is that the later creatures evolved from earlier ones, in a more-or-less continuous progression. No leap-frogging is involved -- just a step-by-step process of learning and mostly improving.

A recent example in Wired Magazine about the Apollo 11 computer system illustrates this way of thinking very well. Here's the title:

Her Code Got Humans on the Moon -- and Invented Software Itself

The Apollo code in general and Margaret Hamilton in particular "invented software itself." Really? Let's look at some of the article. Here's a key paragraph:

By mid-1968, more than 400 people were working on Apollo’s software, because software was how the US was going to win the race to the moon. As it turned out, of course, software was going to help the world do so much more. As Hamilton and her colleagues were programming the Apollo spacecraft, they were also hatching what would become a $400 billion industry.

The last clause of the above quote clearly states that the $400 billion software industry that eventually emerged was "hatched" from the Apollo effort. Like so many people, the author thinks that the 400 person effort to build Apollo software somehow made possible, laid the groundwork for, or something all the developments to follow.

What a pile of hogwash. But typical!

My personal initiation into programming was on an IBM 360 model 50 computer that I was able to access on Saturdays during the 1966 school year when I was in high school. I wrote FORTRAN programs, key-punched them into cards, fed them into the card reader, got direct feedback from the computer about my imperfections as a programmer, and finally got results I wanted. While doing this, I was one of many people at many facilities using this software.

The IBM System 360 effort was started about 1960, and was announced in 1964. Gene Amdahl was the main hardware architect, tasked with the then-unprecedented task of creating a family of computers with a wide range of capabilities that could all run the same instruction set, i.e., run the same software.

Operating system software for such a system had never been created, and was a difficult task, particularly because supporting running multiple programs at the same time, again a first, was a requirement. The software was much-delayed, and by the time Fred Brooks took over the effort, more than 1,000 people were working on the project. Brooks later went on to write a classic book inspired by the effort, the Mythical Man-Month.

The System 360 was wildly successful, and fueled the incredible growth of IBM. Its instruction set lives on to this day in the System z mainframe servers. Its use was exploding at the time the Apollo software was still being developed.

What about the Apollo software? It was peculiar from the beginning, and exhibited issues that had already been solved in OS/360 and its programs. While the independent software industry did indeed grow and mature on S/360 computers, in spite of IBM's efforts to keep anyone from buying any software except their own, the Apollo software led to nothing. The now-lauded Margaret Hamilton tried to commercialize the special software methods she had used on the Apollo project, but her efforts failed and led to nothing. There were a couple of later attempts to build and commercialize what we can now see are versions of her ideas, for example in model-driven development, but after a period of hype they went nowhere.

None of this, by the way, is intended in any way to minimize the achievement of the Apollo mission, the pioneering computer system and software that was an integral part of it, or of Margaret Hamilton herself. She led a huge team that had to get software for a novel system and mission working the first time. It's a real achievement! But it's hardly "inventing software itself."

Even if you believe that someone "invented software itself," the inventor was certainly not Ms. Hamilton; even if you believe that a big, early software effort "hatched" today's software industry, it was not the Apollo effort.

So did IBM do it? Is IBM OS/360 the progenitor of today's software industry? Nope! A growing number of groups were designing and building computers, and even more were building software. You can see patterns, but this biological evolutionary idea of a capability being somehow created in DNA, and through inheritance and random improvement leading to subsequent efforts in a consequential way is nothing but the application of an inapplicable metaphor. While patterns can be observed, each piece of software is written from scratch, based on the ideas and experience of the authors. No software-equivalent of DNA, with the obvious exception of the use of shared code.

I doubt this little rant will be sufficient to stamp out the use of bad logic and inapplicable metaphors by people who insist on writing about software, even though they've never seen it, can't read it, and are clueless about what it really is and how it works. But at least writing this has gotten it out of my system, kind of...

The Slow Spread of Linear Programming Illustrates How In-old-vation in Software Evolution Works

There is loads of talk about “innovation.” Lots of people want to do it, lots of people think they’re doing it, consultants run courses in how to be innovative, and large organizations claim to promote innovation and be innovative. The assumption behind most of this “innovation” talk is that a wonderful bright idea that will change the world (or at least your organization or startup) can pop into anyone’s head. It’s new! It’s brilliant! We’re going to win big with this great new idea! See this for example.

When you study software evolution, you get an entirely different picture of software-based “innovation.” Software evolution shows you that new ideas that work are extremely rare. Oh sure, there’s a flood of new ideas popping into people’s heads all the time. Mostly, they’re not new, and the new ones rarely work. The software concepts that make it big are, in the vast majority of cases, clear examples of existing patterns of software evolution, and have in most cases already been implemented in a different context.

I first encountered the mystery of software evolution while in my first job programming software.

I started programming in a course I took starting in 1966 in high school, taught by a math teacher who couldn’t himself program, but had convinced the school to let him teach the course, and had convinced a local company, a pioneering rocket engine company called Reaction Motors, to give us computer time on Saturdays. I had a textbook about FORTRAN, a steady stream of programming assignments and a computer on which to test my programs. It was great! I continued programming the following summer, as part of an NSF math camp I was able to attend. As I was nearing high school graduation the following year, I got lucky; Diane, a high school friend, talked about me with her father, who got me connected with a nearby company, and I landed a job there for the summer before starting at Harvard College in the fall of 1968.

The company was EMSI, Esso Mathematics and Systems, Inc. in Florham Park NJ.

They were a service company, one of the about 100 units of the Standard Oil of NJ (Esso) companies, devoted to applying math and computers to improving every aspect of the company. I was immediately thrown into the group that was developing optimization models for oil refinery operation. Our focus was on the giant refinery in Venezuela.

What we had running was an implementation of a classic OR (Operations Research) algorithm called LP (linear programming), solved via the simplex algorithm first devised by George Danzig in 1948. In this kind of model, there is a goal equation and a set of constraints. The goal equation calculated profits, using hundreds of contributing variables, including prices you could sell things for, and the costs of various inputs. The constraints were greater/less than equations, each essentially describing some tiny aspect of how the refinery worked. What the algorithm did was find the values of the variables that maximized the goal equation (profits) while satisfying all the constraint equations.

The model was constantly being modified to make it more precise and applicable to actual refinery operations. I had a variety of jobs, including writing new code, fixing bugs, etc.

Since prices were such a key part of the LP model, we had a separate program to calculate what they were likely to be in the future, a Monte Carlo model. I also made enhancements and fixed bugs in this body of code.

I was fortunate to be able to hitch a ride to work and back with a PhD who worked there and lived in my town. During the ride I would tap his deep knowledge of things. He put me onto the various journals in which advances in various forms of OR were described, which I dove into. The math was often above my head, but I was motivated to teach it to myself on a rolling, as-needed basis.

I thought this was a really cool way of running things. There were all sorts of controls in the refinery, controls that let you create more airline fuel and less heating oil, or any number of other trade-offs. It was amazing that you could compute the setting of all the control knobs that would produce the best mix of products that the market needed. Why would you ever run any operation any other way?? It would be simply ignorant and stupid.

I finished school, learned more about the way the world worked, and searched high and low for implementations of LP. Anywhere! If they were out there, they were doing a great job of hiding.

I was confused. How could this be? LP was math, doggone it. It yielded a provably optimal way of running a business. It’s proven in real-life production at Esso. Any other way of operating a business was clearly seat-of-the-pants, wet-finger-in-the-wind amateur hour; anyone whose operation was sizable enough to justify the effort would have to use this method if they weren’t plain-and-simple incompetent. But no one seemed to be using it! What’s going on here??

This mystery was on my mind while I participated in one of the periodic AI crazes that has swept the world of with-it people. While I was still in college, Winograd published his MIT SHRDLU research, in which an “intelligent” robot would converse with a human in English about a world consisting entirely and solely of blocks. You could ask SHRDLU to “put the red block on top of the blue block” and it would do it; you could ask it questions about block world, and it would answer. Amazing! Super-practical! While this was happening, I wrote and submitted a thesis about how to structure knowledge inside of an intelligent robot. All of it useless compared to LP and the associated OR techniques.

The craze was AI, and the mania lasted a few more years, generating a steady stream of "promising" results, none of them in the same universe of practicality and benefit as LP or any other OR optimization technique, which continued to be used only in "secret" little islands of astounding efficiency and productivity.

Later in the 1970’s I first applied for a home mortgage. A key part of getting the mortgage was the interview with the loan officer. You had to pass muster to get the mortgage! Another 10 years passed before OR-type models started to be used for credit. When I next applied for a mortgage in 1981, I was interviewed by a loan officer. By my next mortgage in 1987 it was at least partly automated.

When I got into venture capital in the 1990’s, I discovered that high-value repair parts were had recently been optimized in terms of inventory levels and locations. I looked in detail at the pioneering company ProfitLogic, which did inventory and sale optimization for retail stores, answering questions like when should which products be put on what kind of sale, questions that had traditionally been answered by the local marketing “expert,” just as oil refineries had previously been run exclusively by experienced experts.

Only in the late 2010’s did exactly the same LP models start being applied to medical scheduling, to optimize the use of things like infusion centers and operating rooms. Just as oil refineries produced much more value from exactly the same crude oil inputs in 1968 as a result of LP models, so are infusion centers handling 30% more throughput using exactly the same capital and human resources using the very same LP models.

Here’s the mystery: why did it take so blankety-blank long for LP models to be applied??? There has been no theoretical break-through. Yes, the computers are less expensive, but given the scale of the opportunity, that was never the obstacle. WHY??!! The answer is simple: there is no good answer. Except of course for the ever-relevant one of human ignorance, stupidity and sloth.

The example of LP optimization and its agonizingly long roll-out through different applications and industries over more than 50 years – a roll-out that is far from complete! – is a prime example of the reality of computer/software evolution. Among other things, it illustrates the point that many of the most impactful "innovations" are really "in-old-vations," things that are just sitting there, proven in production and waiting for someone to apply them to one of the many domains which they would benefit.

Here are a couple cornerstones of computer software evolution:

• Software evolution resembles biological evolution only a little. Not-very-fit software species thrive in broad areas of application, unchallenged for years or decades, while vastly superior ones rule the roost not far away. The only reason why the superior software species don’t migrate to the attractive new place appears to be human ignorance and inertia.
• Software evolution resembles the much-derided theory of “intelligent design” quite a bit, if you make a slight edit and call it “un-intelligent, un-educated design.” A “superior” (HA!) being does indeed do the designing of the software, in the form of highly paid software professionals.
• When software appears in a new “land” (platform, business domain), it most often starts evolution all over again, first appearing in classic primitive forms, and then slowly re-evolving through stages already traversed in the past in other “lands.” This persistent phenomenon supports the “unintelligent design with blinders on" theory of software evolution.

I will explain and illustrate these points in future posts and a forthcoming book.

Barriers to Software Innovation: Radiology 1

There is a general impression that software innovation in one of its many forms e.g. “Digital Transformation” is marching ahead full steam. There are courses, consultants, posters hanging in common spaces and newly-created Chief Innovation Officer positions.  What’s new? What’s the latest in software?

The reality is that there are large reservoirs of proven, tested and working software innovations ready to be rolled out, but these riches are kept behind the solid walls of dams, with armies of alert guardians ready to leap in and patch any holes through which these valuable innovations may start leaking into practice. Almost no one is aware of the treasure-trove of proven innovations kept dammed up from being piped to the many places that could benefit from them; even the guardians are rarely fully conscious of what they’re doing.

If anyone really wanted to know what was coming in software, all they would have to do is find the dams and peer into the waters they hold back.  In spite of the mighty dams, it sometimes happens that the software finds its way into practice, normally in a flood that blankets a small neighborhood. Sometimes the flood has been held back for decades. There are cases I know of where an innovation was proven 50 years ago, and is still not close to being rolled out.

The dams are built in many ways with many materials. The raw materials appear to include aspects of human nature: ignorance, sloth, greed -- you know, the usual. The really high, solid dams have broad institutional support, in which “everyone” is fine with things as they are, and won’t so much as give the time of day to an amazing innovation that would change many things for the better – except of course for a key interest group.

Here is one of the examples I personally know about. It was one of my introductions to what innovation is all about, and the sad fact that creating a valuable innovation is generally the easy part – the hard part is usually overcoming the human and institutional barriers to deploying it.

Automating Medical Image Reading and Reporting

When a radiologist gets an X-ray, there are two phases of work. The first is to “read” the X-ray and observe anything non-typical that it shows, anything from a broken bone to a tumor. The second is to generate a report of the findings. Most radiologists, then and now, dictate their findings; then someone transcribes the dictated report and sends it as needed. The details of the report can vary depending on the purpose of the X-ray and the kind of person for whom it’s intended.

There has been technology first tested decades ago that appears to show that software is capable of “reading” an X-ray at least as accurately as a human radiologist. I will ignore that work for now, and focus on what should be the less threatening technology, which is translating the doctor's observations to an appropriate report.

While I was in college, I worked on the early ARPAnet with an amazing group of people, one of whom was an MIT student from the San Francisco area who later went on to fame making major advances in integrated circuits, among other things. The summer after we did most of our ARPAnet work, he got involved with a new initiative to transform the way radiologist reports of X-rays were created. He knew that some of my skills in automated language parsing and generation were relevant, so he invited me out to pitch in. I went.

GE, then and now, was a major maker of medical imaging systems. They were seriously experimenting with ways of enhancing their systems to make it easier for radiologists to produce reports of their findings. They created a set of mark sense forms, of the kind widely used at the time for recording the answers to tests, to enable a radiologist to quickly mark his observations of the part of the body in question. Here is the form for a person's guts:

Here is part of the form for the hand, showing how you can mark your observations:

Here is part of the form for the spine, showing how you can customize the report output as needed:

My friends had gotten most of the system together -- all I had to do was build the software that would create the radiologist's report. Because of uncertainty about radiologist's accepting the results, I had to make the report generator easily customizable, so that the radiologist's typical style of writing was created.

Leaving out the details, in a few weeks I created a domain-specific language resembling a generative grammar and rules engine to do the job -- along with the necessary interpreter, all written in PDP-8 assembler language, which was new to me. My friends wrote a clear and compelling report describing our work and included an example of our working software in it. Here was the sample filled-out form:

And here was part of the report that was generated by the software we wrote from the input of that form:

The software worked! And yes, the date on the report, 1971, is the date we did the work.

A major company, prominent in the field, had taken the initiative to design mark-sense forms, incorporating input from many radiologists. A few college kids, in contact with one of GE's leading partners, created a working prototype of customization report-generating software, along with a proposal to bring the project to production.

Just as a side effect, this project would have done something transformative: capture the diagnostic observations of radiologists into fully coded semantic form. This is a form of electronic medical record (EMR) that still doesn't exist, even today! For all the billions of dollars that have been spent on EMR's, supposedly to capture the data that wil fuel the insights that will improve medical care, a great deal of essential medical observation is still recorded only in un-coded, narrative form -- including medical imaging reports!

The bottom line is that this project never got off the ground. Not because the software couldn't be written, but because ... well, you tell me.

See the next post for the continuation of this sad but typical story.

Facebook's Libra Crypto-currency Introduces a Brand-new Smart Contract Language

Facebook’s Libra faces the daunting task of pulling off the flawless world-wide launch sometime next year of a new cryptocurrency based on new code. In taking on this task, they are hoping to pull off a first in software history: a major body of new code that works out of the gate. I assess the odds of this working here. At the same time, they have upped the stakes by also introducing a brand-new smart contract framework based on a brand-new language. Good luck!

Smart contracts are a way of extending and customizing a blockchain. Outsiders might imagine that the Bitcoin competitor Ethereum emerged from the pack because its name is somehow cooler than Bitcoin, but insiders know that an important factor was its pioneering incorporation of the first widely known implementation of smart contracts. Here is my explanation of smart contracts.

There’s just one little problem: however cool Ethereum’s smart contracts may be, in practice a majority of smart contracts have bugs and security holes, as a study of tens of thousands of them has shown. Even worse, smart contracts are part of the “immutable ledger” that is supposed to make things secure. Except when there are bugs and security holes, it doesn’t.

Facebook has quietly recognized that smart contracts are needed to make the primitive blockchain database even marginally practical, but that most smart contracts aren’t even modestly intelligent. How are they going to fix this problem?

One of the wonderful things about the steady stream of blockchain and cryptocurrency initiatives by internet and corporate giants is that they tend to tell us, in plain and simple language, the fatal flaws of the whole block-whoey business. Of course, they don’t put it that way. They know that they’ve created a dramatically improved system of blockchain (or whatever) – and as soon as you fully appreciate how bad the standard-issue stuff is, you’ll insist on buying their new, dramatically improved version. Microsoft and Intel have done us all this favor in explaining the wonders of their proprietary version of blockchain, as I described here.

Facebook has followed in this clear pattern. They actually spell out, in no uncertain terms, that existing smart contract implementations are dangerous things, riddled with bugs and filled with security holes. But it’s nearly impossible to build a marginally usable cryptocurrency system of the kind Facebook wants without them.

Facebook is proud of its solution: a new software language called Move. Yes, a language called “Move.”

I’ve spent a little time checking out the new language. The developers are generally right about the deficiencies they are addressing, effectively endorsing the view that existing smart contracts are hopelessly flawed. They are smart and have put forward credible solutions to the problems. It’s just possible that, after a few years and after the software has gone open-source, the new system will turn out to be an improvement on the old one. But before deciding that, let’s do something programmers avoid doing: take a quick look at history.

Software history is chock full of programming languages, each of which was invented to improve on or fix problems with earlier languages. Most new languages are supposed to make programming faster and more flexible, with fewer errors of any kind. After more than half a century of effort with thousands of new languages, how has that worked out? See this for details. Sorry, humans are creative types, and are capable of making mistakes in any medium at all. While Germans may be deeply certain that the German language is more clear and precise and superior for expressing truths than French, citizens of France are remarkably articulate about how this is not the case – while at the same time demonstrating that the French language is no better.

The team at Facebook has done us all the great service of making widely known the otherwise ignored deep flaws in Smart Contracts, while almost certainly increasing the chances of things going horribly wrong with Libra while introducing a well-intentioned but, well, new language, claiming against decades of experience with thousands of languages that this one will really bring human programmers to the land of perfection.

I’ve got a bridge. It’s cheap – wanna buy it?

This was originally posted at Forbes.

Facebook's Libra Crypto-currency is unlikely to Work - Here's Why

There is a great deal of buzz about Facebook’s new cryptocurrency Libra. There is even a trickle of technical information about it surfacing. No one seems to be talking about the deep-seated technical reasons the new system will crash and burn. Sadly for Libra, there isn’t just one such fatal flaw! Here I’ll describe one of them.

The core reasons that FB’s Libra will fail are:

1. it’s a large body of new code
2. new code is always riddled with bugs, no matter how hard the developers try
3. Unlike the code big companies like FB are used to, bugs are really hard to hide in this application

It’s a large body of new code

The hype machine for crypto, Bitcoin, Ethereum, ICO’s, Blockchain and the rest has been running at full speed for a few years now. Leaders in every industry  are infected with intense FOMO (fear of missing out), and are committing to projects left and right. With all the blockchain projects going on for years now, it’s understandable that most people would think that this code must be solid and tested by now. There’s just one little problem: there are thousands of bodies of code, with new ones emerging all the time as groups get excited about fixing the glaring problems in older implementation of the concepts; little groups like Microsoft and Intel. These aren’t just tweaks – we’re talking major new bodies of code here.

Think about transportation machines as diverse as propeller planes and powered skateboards. Yes, they both get you from point A to point B, but they’re quite a bit different from each other. The code that Libra plans to use is brand new in every way – even the central concepts of how blocks are built and chained are radically different than the proven-in-production methods used by Bitcoin. That’s like saying “we’re using proven engine technology to build our new car – except that its engine won’t use gas, diesel or electricity for power – it will be better!”

 

New code is riddled with bugs

People who write code make mistakes. Lots of them. All the time. There are a host of methods in varying degrees of use to prevent or catch such mistakes, things ranging from test-driven-development to extensive code reviews. None of them work. No, they don't work for Facebook either.

Yes, there are some bodies of code that are remarkably reliable. Linux is a great example – it powers over half of all the web servers in the world! Linux performs a function that was thoroughly understood when it was written, and is an open-source project written in a solid language – C – and led by a true coding genius. Its quality was achieved over years of top-notch leadership with thousands of talented contributing programmers and millions of installations. Libra is at the opposite end of the spectrum. It’s brand new, and it’s supposed to work flawlessly keeping track of financial assets from day one. The chances it will perform without flaws from the beginning are essentially nil.

What’s worse, the internet giants have an unbroken track record of releasing code that’s riddled with errors. Yes, Facebook is partnering with lots of corporate giants – and those giants are equally accomplished at releasing an unbroken stream of software horror shows.

Facebook ignores the issue of its inability to produce software that works and satisfies users, much less have a solution for it that it will apply to Libra.

The tech giants usually hide their bugs

The much-lauded software geniuses at Facebook, Twitter, Google and the rest are convinced that they are as good as programmers get. But their efforts have a track record of failure. More important than the actual failures is the fact that their applications are ones in which hiding errors is built into the applications! When you enter a search query, how do you know whether the results are accurate, so long as you get a list of vaguely relevant results? When you pull up Facebook and see the newsfeed, are the entries always the right ones? Are all the entries that should be there in fact there? How would you know when they’re not?

Contrast this with your credit card. You get a statement. You can be sure the bank has included every transaction you’ve made, so they can make you pay for it. Most people at least scan the transactions to see if there’s one you didn’t make, so you can call the card company and get it removed, so you don’t have to pay for whatever the criminal bought using your card! The typical internet method of hiding errors just isn’t going to work here – and Facebook doesn’t even acknowledge the issue, much less have a way to solve it.

Conclusion

Facebook, like the other internet giants, is incapable of building code that works, even after extensive testing and use by millions of users. Corporate giants and the government are no better. Facebook’s usual method of tricking users into not seeing its errors won’t work here. Facebook and its partners are rushing to somehow leverage Bitcoin’s “success” at funding the international illegal drug and human trafficking trade, flouting anti-money-laundering regulations and providing a platform for what amounts to massive illegal gambling. Of course, they talk about being charitable to the "unbanked" and other noble goals, while continuing to enrich themselves in a way the so-called "robber barons" could only envy. They are likely to fail at this mission because of their inability to write software that works.

Final note: the standard pronunciation “Libra” is Lee-brah, like the astrological sign. Because of its deeply flawed design, which Facebook and its partners try to cover up, I prefer to pronounce Libra as “lie-brah” – because it’s based on lies.

This is cross-posted from the original on Forbes.

The Evolution of Software

There is strong interest in the latest developments in software. No one wants to be left behind.

At the same time, there is a peculiar (to me) lack of interest in whatever the latest thing grew out of or evolved from. What are the new thing's predecessors? What did that thing grow out of. Have similar things appeared in the past? Are there patterns we can observe here, or are the new software things that explode onto the scene just a meaningless sequence of random events? Are they things that we obsess about while they're here without questioning or wondering where they came from, without asking what else they may resemble in some way? Are they things we just forget about and ignore once they depart the stage of software fads?

The questions have obvious answers. Answers that no one seems to be interested in.

So long as we fail to ask these crucial questions, we will remain ignorant of software in deep ways, and will continue to stumble from fad to fad without improvement or understanding. History is important!

Evolution in Science

If you study different sciences and their history, you notice that, at some point, scientists begin to pay attention to evolution, i.e., change over time of the thing being studied.

The most obvious thing is evolution of an individual item over time. In biology, you have the cycle from birth to death. In geology, you have the change of mountains, rivers, beaches and glaciers. In materials science you have the change of objects as they are subjected to various conditions over time, for example the way that iron rusts.

Evolutionary studies reach a breaking point and get really interesting when you study groups and types and see the patterns of change. In biology, this was the revolution that happened with Darwin and his study of the evolution of species. Something similar happened in geology with plate tectonics.

When you put the patterns together, things get totally amazing and transformative. While no longer fashionable to talk about and study, the parallels between the growth stages of an individual animal and the evolution of species are obvious, though not literal at a detail level.

Studying evolution is an important stage in the evolution of most sciences!

It's long-since past time to study software evolution, as an integral part of software science.

Software Evolution

Software evolves at every level.

At a basic level, software languages evolve. I have given an introduction to this subject here. The evolution of software languages resembles biological evolution in remarkable ways, as species (languages) emerge and evolve in response to new conditions, including other species. Similarly to biology, languages evolve; sometimes many species descend from them; and sometimes a species goes extinct. See this snapshot, for example:

(Credit)

Consistent with the determination of nearly everyone involved with software to ignore its history, this chart and anything like it are ignored, and the evolutionary principles behind it are not studied. When you start to study them, you learn amazing things. Suddenly some of the random change in the world of software languages starts to make sense.

Software languages are the very most basic things about software. Just as interesting and vastly more important are the programs written in software languages. Everyone involved in writing or maintaining a software program (application) just thinks about that program. It definitely occurs to business people what the program can do compared to its immediate competitors, if any. Yes, there is a kind of "survival of the fittest" in software evolution.

What about the "insides" of the program compared to other program insides? Sometimes people are aware of gross differences, like the language the program is written in. But the curiosity normally stops there.

The program "insides" have huge practical consequences. Depending on the details of how a program is written, it can be more or less:

• reliable/buggy
• easy to hack (secure)
• expensive to run/operate
• dangerous/costly to change
• able to respond quickly to changing loads
• speedy or slow

People commonly talk about structure or architecture, but those things are just the tip of the iceberg.

When you dig in, you find the equivalent of islands and continents of software. I've treated this subject here, for example. For example, you may have spent some number of years after getting your CS degree working on website design, and interacted with a community of people similarly engaged. Not everyone uses the same tools or does things the same way, of course, and you're likely to think there's quite a variety of approaches.

Then you go wild, dive into hospital management systems, and your mind is blown. What you thought was the wide variety of software and tools used by website designers turns out to be a whole continent apart from what you find in hospital automation. You learn that the flagship product of the company that has more than a third of the market (Epic) is written in a language you've never even heard of! It's like growing up in New Jersey among what you think is a very wide variety of people and going to Chennai -- a huge city in India you've never heard of, whose population is larger than your whole state, where the main language is one you've never heard of, written in a script you've never before seen.

Even more ignored are the obvious (to those who trouble to look) repeating patterns that can be observed in applications as they are first written and then evolve over time.

The Study of Software Evolution

To put it bluntly, the study of software evolution has barely begun. A few isolated souls, hardy or foolhardy as you like, have dipped their toes into the deep waters of software evolution. Those few have found worlds to explore, oceans whose depths have yet to be plumbed.

This is particularly the case because software is a "the only reality is the present" kind of field, as it now stands. It doesn't have to be this way!

Very few people study the history of math. They don't need to: anyone who learns math starts with math as it existed thousands of years ago. If they do well in math, they get all the way up to ... the late 1600's, when Newton and/or Leibnitz discovered calculus! Each step in math is based on and often depends on the earlier steps.

In some sense, this is also true of software. Except not really.

First, knowing nothing about software, you can dive right into the latest language and tools and use them to make something that works. Anyone think you could do that with differential equations, I mean without already knowing how to count, add and multiply? The difference is that in software, the "accomplished" new-comer really is standing on generations of older software ... of which s/he is blissfully unaware! The new programmer, of course, uses an incredible array of operating systems, drivers, networking, file and database systems, and on and on. These form the "land" on which the new programmer "stands" to work. None of the wonderful new stuff would work without it. The big difference, of course, is that using the underlying software and understanding the underlying software are two rather different things. This makes software totally different from not just math, but most of the other legitimate sciences (leaving out the fake ones like political "science" and the rest).

Second, when you're building a program, chances are excellent that you're not the first person to try building something very much like that program. Lots of details may be different, but even the details are probably not as unique as you might think. Suppose you need a program that sets up an account for a person, and then when the person does stuff, the details are checked with the person's account, and if OK are added to the account, and if not are rejected but recorded. This high-level description applies to a sales operation with customers, a medical office with patients, a manufacturing company with customers, and a bank. Given that programs like this have been built tens of thousands of times, don't you think that the ways of building programs like this would have evolved? That there would be pre-built parts, specialized tools for using the parts, building new ones and gluing them together? Special methods that have evolved and been refined to make sure it's done in the optimal way? Did any of you with any level of degree in Computer Science learn any of that -- other than, of course, some boiler-plate about the virtues of object-oriented languages and maybe some other fashionable stuff? Of course not! Anyone with any sense of fashion and/or status knows that these are not career-advancing subjects. Anything involving software history or comparative study of software projects is career-killing.

Conclusion

Software evolves. But it evolves differently than other things studied by science. Virtually no one studies software history in any way, much less the patterns of evolution that become apparent when you study that history. There have been a couple of attempts to learn why software doesn't get better in the way many other things do, most notably Fred Brooks with his book The Mythical Man-Month and other writings. I have dissected this flawed analysis in my work on Software Project Management and Wartime Software. See this for a start.

I have worked for years trying to identify some notable patterns in software evolution, and will be releasing some of that work in the coming months.

The Aftermath of the Net Neutrality Disaster

After many years of fighting against entrenched corporate interests, the FCC finally extended its regulatory authority to the internet, instituting the so-called "net neutrality" rule in 2015. Finally, the internet would remain free and open.

Then a new administration came in, with new leadership at the FCC. There was a move to repeal the common-sense principle of "net neutrality!" Why would anyone think that letting giant corporations pick and choose what we can see on the internet was a good idea, making things they didn't like too expensive or simply blocked -- crazy! It's kind of like repealing the laws against murder, hoping that people would just not do it!

Some leading Democrats made it clear what would happen if net neutrality were repealed:

 

The senate was being careful here -- what about ISP's blocking access to sites they don't want you to see? Senate leaders were outspoken about the danger:

People were upset. There were demonstrations at more than 700 Verizon stores nationwide:

Protestors even spread out through the new FCC chairman's neighborhood, demonstrated in front of his house, took pictures of the inside of his house through his windows, and made it clear that this man, Ajit Pai, was leading the movement to destroy the internet as we know it.

After a vote near the end of 2017 to repeal this important protection, the repeal finally took effect in June 2018, about a year ago. That's when the disaster of corporate control of the internet, access limitations, slower speeds and no more innovation started. Articles were written in the NY Times and elsewhere spelling out what would happen: "repealing these onerous rules “would be the final pillow in (the internet’s) face” (The New York Times), would cause “erosion of the biggest free-speech platform the world has ever known” (ACLU), and would be “end of the internet as we know it” (CNN)."

It's now about a year into the disaster. I started looking into just how bad it's been and was struck by how hard it is to find the news stories about all the sites that have been blocked, all the students relegated to slow lanes and prevented from watching their favorite video streams, and all the price hikes. Have the giant corporations really been so scarily effective at suppressing the consequences of their self-serving actions?

I dug in and found this:

Last year, average internet download speeds shot up almost 36%, and upload speeds climbed 22%, according to internet speed-test company Ookla in its latest U.S. broadband report.

There are more users than ever. More videos to watch. More content to consume. More commerce being conducted. No sites are being blocked. No one is complaining that their service is being throttled. And more people are gaining access to broadband.

During President Trump’s first year in office, in fact, the number of people without access to a broadband connection dropped by 18%.

Meanwhile, the 5G era approaches, which will increase speeds — for mobile and fixed broadband — exponentially while injecting more competition among providers.  Elon Musk’s SpaceX has started launching mini-satellites for his Starlink initiative, which will provide broadband internet to subscribers wherever they are on the planet. 

Oops. Maybe the disaster isn't a disaster at all. Maybe it's even a good thing, hard as that may be to understand. Here's what I wrote about it when repealing net neutrality was being discussed, and here's the long post I wrote on the subject when the campaign to institute net neutrality was underway.

p.s. That earlier post on net neutrality touched a few nerves. A few programmers at companies I worked with got all hot and bothered, and refused to interact with someone so completely unethical that they could harbor such awful thoughts. Well, now we know how it all turned out, and what the repeal-net-neutrality disaster amounted to...

Adventures with Health System Software: Customer Feedback

If you want a cheap laugh, go to the Mount Sinai medical system website and hope they ask you to complete an opinion survey. It’s stupid and ridiculous, deserving lots of snark. But try not to think about what it means or the underlying reality, or you might get kinda depressed. Like I did, because I went to the website because I needed to get something done! I needed a phone number. Sounds simple, right? Until you understand I had already talked with someone at Mount Sinai, and that person gave me the wrong number. But I really needed the number -- I needed to make an appointment for a medical test that is crucial to my health. After a great deal of searching, I finally found what appeared to be the right number. Except it wasn't, as I found when I called.

This was part of my epic struggle to schedule an appointment -- something that I do with a couple clicks for my favorite restaurants, my cat at the animal hospital, or ... yes, my primary care provider. But at that powerhouse medical institution, Mount Sinai? Only the best people who really, really, really want an appointment are graciously granted one. See this for the story.

In this post, I'll confine myself to glancing at the carefully constructed Mount Sinai website and the extraordinary steps they are taking to assure that it is the best it can be. It's clear they're in a race for the top with the health insurance companies on this subject, see this.

Major companies that build websites have a problem, a problem they share with lots of companies that build software. The executives in charge are required to say that they care about quality, and do everything in their power to track and improve it, along with important metrics involving customer satisfaction. They take concrete steps to measure quality, using the best firms out there to help them.

There's just a little problem: they can't get it done.

The Mount Sinai website

I recently encountered a typical example of hopeless executive incompetence while trying to get a simple phone number to schedule a visit to Mount Sinai Hospital in NYC – scheduling that any institution whose software had successfully made the wrenching transition to the 2,000’s would have made long ago. I tell the story of the scheduling adventure here.

It was a long slog to get my MRI appointment made, including a number of calls and emails. You might think that when a window popped up near the end of my ultimately unsuccessful trek through the Mount Sinai website to extract a simple phone number that I would ignore it. After all, the website is a carefully-crafted, attractive-looking piece of useless fluff, impressive perhaps to the important people who are shown images of it in a Powerpoint presentation during some meeting, but in fact annoying, error-filled and generally useless to real people. Silly me: here I am thinking that the hoi-polloi, the real people who have health issues, are the relevant people here – when in reality, it’s the executives, jockeying for ever-growing power, prestige and money among themselves.

If you’ve read any of my other posts on software quality, you may suspect that I’m a glutton for punishment. Your suspicions are correct. So I agreed to take the survey. When I left the site, I expected the survey to pop up, but it didn’t. After all, the request told me, in no uncertain terms, “it will pop up when you leave the site.” OK, I thought, your loss, not mine. But darn! The survey I recently got from my health insurance company was so juicy!! I would have loved to see who wins the race for most dysfunctional survey between a major provider and a major payer!

It turns out, I just needed to wait. Before long, the survey arrived in an email:

I was a bit surprised to get the request in this way, but OK, they’ve obviously got all my information, so fine. As usual, I hover over the link to make sure it’s legit. The URL was portal.gsight.net with some codes after. I quickly discovered this was a domain owned by the company that sent me the email, Greystone.net. Hmmm, who are they?

Wow, a whole company devoted to healthcare marketing and the internet! They must be really good! I wonder if they know about the web?

It appears they know about the Web. And look at this:

It’s a whole process to make the website great! Smart folks, those people at Mount Sinai, turning to professional specialists to figure out how well their website is serving their customers! Though I can only assume that Greystone has only recently been engaged, since the Mount Sinai website is, after all, a pretty-looking pile of stinking crap…

So let’s dig into this expert opinion survey. Click. Here’s where I land:

OMG!!! My jaw has hit the floor so heavily, I’ll probably be scarred for life. I wonder if I can sue Greystone to cover the costs of plastic surgery for my deformed, floor-mangled jaw??

Why is my jaw hurting? Because the link these consummate professionals sent me was to a completely generic landing page! There is this thing known as “deep linking,” in which the custom URL you click brings you right to the place in question. It’s widely used. The landing page knows who you are and why you’re there. I guess the folks at Greystone hired a bunch of interns for this project, ones who hadn’t gotten to that chapter in the “Internet Linking for Dummies” book. And no one with the slightest bit of experience, like the average internet user, had tried it out.

After I gave the right answer, I was thrown into a completely generic survey about the website, utterly uninformed about who I was or any smidgen of knowledge about my site visit – putting the lie to the user tracking they supposedly do. Had they done elementary user tracking, they would have known who I was and which pages of the site I had visited. But no, they decided to ask completely generic questions.

Is this hard to do? Nope. For example, my bank, USAA, notices when I go to the “wire transfer” section of their website and then call them. A recorded voice says something like “I see you’ve recently visited the money transfer section of the USAA website; would you like to wire money today, David?” If I answer “yes,” they transfer me to the relevant department. Not hard! Maybe the Greystone.net interns will eventually get to that chapter.

The survey itself was endless, irrelevant awfulness.

Here’s an example of why the survey was awful:

If they had tracked me and made the survey specific, they would have known that I hadn’t filled out a form. Instead, they present me with a question about form-filling, and then require an answer. Most of the questions were like this. By the way, this question was about ... the twentieth question -- all of them response is required. See the progress bar that says 25%? At some point, it jumped to that, and then, question after question, it didn't changed.

Then at the end, I was invited to give some input. Which I did in calm language, mentioning that it might be nice if the phone numbers on the site were, you know, correct numbers. Of course, since they hadn't deep-linked, they had no way of contacting me to get further information.

Just to be sure I wasn't completely nuts, I went onto the Mount Sinai website again. I got lucky -- an invitation to complete an opinion survey popped up again. I carefully chose "take it after leaving the site," and this time it worked, though in a remarkably clunky way, indicating that whoever built the code had flunked Javascript 1.01. So I get the survey, and was blown away by seeing the very same "how did you get here" question I got from the email link. Any competent web programmer could know how I got there, by looking at exactly how the original URL was invoked. Clearly, performing this elementary task was beyond the collective genius of Greystone and Mount Sinai.

Then as I went farther into the brain-dead survey, I discovered that it just didn't work. Look at this:

Look at the percent completion bar just below the black line under the Mt Sinai logo -- it's still at zero, even though I'm many questions into it, as you can see by the scroll bar on the right. Programming and QA 1.01. Fail. And of course, it was a survey designed so that no normal person would march all the way through to completion.

Discussion

There’s a concept in math and computing, and also in real life, called “recursion,” or sometimes self-reference. It’s a simple concept; it’s been around for literally thousands of years, as we know from fun statements ancient Greeks made involving lying Cretans. In this case, it applies to the question of the quality checkers: who checks the quality of the quality checkers?

The answer is evidently “no one.” The most basic principles in surveys, common sense but also proven by experience, are “keep it short” and “Make every question matter.” We know these are the relevant principles because everyone who hasn’t failed the “survey 1.01” course knows that the most important metric to measure is drop-out rate. Of the people you invite, how many accept? Of those who accept, how far in the survey do they get before dropping out? What’s the completion rate? Any tracking along these lines would have shown minuscule completion rates. I’d love to have a recording of the executive meeting at Mount Sinai in which the survey results were presented, to see whether the issue was even raised.

But beyond that, let me ask: when was the last time you got a survey from Google? Or Amazon? Never, right? Another thing: have you read even a little bit about opinion polling, about how it's long-since been proven that people give one answer when asked, but then act differently? What people who are moderately educated web professionals know is that surveys are useless! That's why folks who know a little about websites watch what you do! If there's a lot of information on the site, they make it search-based, with lots of suggestions. They look for drop-outs.

Yes, I've made fun of how badly the survey was constructed and executed. It was the electronic equivalent of a paper survey from 50 years ago. Which makes sense, because the Mount Sinai website is the electronic equivalent of  a glossy brochure from 50 years ago. That's the killer observation. Mount Sinai could make a huge advance by leaping forward to the state of the art of roughly 20 years ago. The very fact that they're using obsolete technology like surveys -- and on top of it doing it incompetently -- shows that they are clueless. It's the equivalent of using a steam-powered car instead of an oil-powered one, and being unable to run the steam-powered car competently. The right response here isn't build a better survey -- it's use modern customer feedback techniques.

Conclusion

Well, it’s a wash. The hospital system opinion survey was pretty different from the health insurance one, but they each exemplified unique ways of being bad. I wonder how many dimensions of badness there are? The institutions I’ve had the pleasure of experiencing are clearly on the leaderboard of those most likely to get to the maximum. Neither of them has a clue about decades-old methods that are vastly superior for getting customer feedback than surveys, however well-constructed those surveys might be.

Postscript

Learning about the excellent survey work conducted by Greystone.net on behalf of Mount Sinai had an added dimension of amusement for me because I grew up near an institution named Greystone. Or more formally, Greystone Park Psychiatric Hospital.

It was, as my mother the R.N. called it when I was growing up, a ‘Looney bin.” If someone said something she thought was dumb, she would say “Did you just escape from Greystone?”

Recurring Software Fashion Nightmares

Computer software is plagued by nightmares. The nightmares vary.

Sometimes they are fundamentally sound ideas that are pursued in the wrong way, in the wrong circumstances or at the wrong time. Therefore they fail – but usually come back, sometimes with a different name or emphasis.

Sometimes they’re just plain bad ideas, but sound good and are cleverly promoted, and sound like they may be relevant to solving widely acknowledged problems. Except they just don’t work. Sometimes these fundamentally bad ideas resurface, sometimes with a new name.

Sometimes they’re a good idea for an extremely narrow problem that is wildly applied beyond its area of applicability.

The worst nightmares are the ones that slowly evolve, take hold, and become widely accepted as part of what “good programmers” believe and do. Some of these even become part of what is ludicrously called “computer science.” Foundation stones such as these, accepted and taught without proof, evidence or even serious analysis, make it clear to any objective observer that computer “science” is anything but scientific, and that computer “engineering” is a joke. If the bridges and buildings designed by mechanical engineers collapsed with anything close to the frequency that software systems malfunction, the bums and frauds in charge would have long since been thrown out. But since bad software that breaks is so widespread, and since after all it “merely” causes endless delays and trouble, but doesn’t dump cars and trucks into the river, people just accept it as how things are. Sad.

Following is a brief review of sample nightmares in each of these categories. Some of what I describe as nightmares are fervently believed by many people. Some have staked their professional lives on them. I doubt any of the true adherents will be moved by my descriptions – why should they be? It was never about evidence or proof for the faithful to start with, so why should things change now?

Sound Ideas – but not here and now

• Machine learning, AI, OR methods, most analytics.
  • These things are wonderful. I love them. When properly applied in the right way by competent people against the right problem at the right time, amazing things can be accomplished. But those simple-sounding conditions are rarely met, and as a result, most efforts to apply these amazing techniques fail. See this.
    

Bad ideas cleverly promoted

• Microservices.
  • Microservices are currently what modern, transformative CTO’s shove down the throats of their innocent organizations, promising great things, including wonderful productivity and an end to that horror of horrors, the “monolithic code base.” While rarely admitted, this is little but a re-incarnation of the joke of Extreme Programming from a couple decades ago, with slightly modified rhetoric. A wide variety of virtues are supposed to come from micro-services, above all scalability and productivity, but it’s all little but arm-waving. If software valued evidence even a little, micro-services would be widely accepted as the bad joke it is.
• Big Data
  • There's nothing wrong in principle with collecting data and analyzing it. But in practice, the whole big data effort is little but an attempt to apply inapplicable methods to random collections of data, hoping to achieving generic but unspecified benefits. See here.
    

Great ideas for a narrow problem

• Blockchain.
  • My favorite example of an excellent solution to a problem that has been applied way beyond its area of legitimate application is blockchain. Bitcoin was a brilliant solution to the problem of having a currency that works with no one in charge. How do you get people to “guard the vault” and not turn into crooks? The way the mining is designed with its incentives and distributed consensus scheme brilliantly solves the problem. However, the second you start having loads of crypto-currencies, things get weaker. Once you introduce “smart contracts,” there’s a fly in the ointment. Take away the currency and make it blockchain and you’ve got a real disaster. Then “improve” it and make it private and you’ve got yourself a full-scale contradiction in terms that is spectacularly useless. See here.

Bad ideas that are part of the Computer Science and Engineering canon

• Object-orientation.
  • Languages can have varying ways of expressed data structures and relating to them. A particular variant on this issue called “object-orientation” emerged decades ago. After enjoying a heyday of evangelism during the first internet bubble, O-O languages are taught nearly universally in academia to the exclusion of all others, and adherence to the O-O faith is widely taken to be a sign of how serious a CS person you are. For some strange reason, no one seems to mention the abject failure of O-O databases. And no one talks about serious opposing views, such as that of Linus Torvalds, the creator/leader of the linux open source movement, which powers 90% of the world's web servers. Programmers who value results have long since moved on, but academia and industry as a whole remains committed to the false god of O-O-ism.
• Most project management methods.
  • I’ve written a great deal about this, including a book. Whether it’s the modern fashion of Agile with its various add-ons such as SCRUM, project management methods, ripped from other disciplines and applied mindlessly to software programming, have been an unmitigated disaster. The response? It ranges from “you’re not doing it right,” to “you need to use cool new method X.” Yet, project management is a profession, with certifications and course on it taught in otherwise respectable schools. A true nightmare. See these.
    
• Most software QA methods.
  • All software professionals accept what they’ve been taught, which is that some kind of software automation is an essential part of building software that works and keeping it working. Except the methods are horrible, wasteful, full of holes and rarely make things much better. See this.
    

Conclusion

I make no claim that the list of items I’ve discussed here is comprehensive. There are things I could have included that I have left out. But this is a start. Furthermore, if just half the items I’ve listed were tossed and replaced with things that actually, you know, worked, much better software would be produced more quickly than it is today. I don’t call for perfection – but some progress would sure be nice!

Blockchain has been Unchained and Unblocked and it's Broken

Blockchain promoters and enthusiasts continue to blithely stroll along the yellow blockchain road to the golden city where immutable distributed ledgers make decades-long technology problems fade away, like the wicked witch. None them publicly acknowledges or seems to notice the hurricanes and earthquakes that are increasing in frequency and intensity.

In total, hackers have stolen nearly $2 billion worth of cryptocurrency since the beginning of 2017, mostly from exchanges, and that’s just what has been revealed publicly.

That’s no big deal, I guess. I describe these little security problems here and here.

Someone who’s technically sophisticated could argue, following the logic I described here, that the security problem wasn’t in Blockchain itself. The problem was in wallets and exchanges, which are software that sits “on top of” blockchain, making it easier to use. It’s the same kind of security breach that can happen with any software, and has little to do with the inherent security of the software itself, but is mostly due to the layers of software built on top. This is true! One does wonder why Blockchain is so wonderful, then, if in practical use, its supposed greater security is so easily circumvented.

Is it really more secure than a regular DBMS, putting aside all those flaky higher layers of software? That’s what everyone involved declares. The most open and honest of the Blockchain-ista’s will grudgingly admit that a nearly impossible 51% attack could cause a bit of a problem with the heart of the system, the keep of the blockchain castle.

Sadly, the nearly impossible attack has happened. And not with some obscure little crypto-currency no one has ever heard of, but with Ethereum Classic, one of the premier systems, and the home of that transformative invention, the Smart Contract.

An attacker had somehow gained control of more than half of the network’s computing power and was using it to rewrite the transaction history. That made it possible to spend the same cryptocurrency more than once—known as “double spends.” The attacker was spotted pulling this off to the tune of $1.1 million.

To anyone with a shred of common sense, this is a fatal event. It demonstrates that Blockchain’s security has a fatal flaw, even when running in its optimal environment, with public miners.

The big companies promoting private blockchains, should they deign to pay attention, will immediately come back with strong statements about how that kind of attack could only take place in a public blockchain, and couldn’t possible happen with a highly secure, controlled environment they provide with their private blockchain. Sure. That’s like saying that those guys who stole lots of money in the open, in a big public space where everyone could see them, couldn’t possibly get into the single secret room and rob the bank vault in complete privacy. Security in closed computer system managed by big companies who follow all the security regulations and pass audits is abysmal. Ever hear of Edward Snowden? Chelsea Manning? Others? Check out the facts a bit, and then come back to me and explain how it is that the unbroken stream of security breaches of the best systems run by the best military and corporate bureaucracies is going to suddenly stop when the software at the core is Blockchain.

The sad fact is, libraries are more secure than computer systems. Including when Blockchain is involved.

This post first appeared at Forbes.

What’s Wrong with Medical Scheduling and Why it Matters

It’s easier by far to make a reservation at a restaurant than for a medical test. I guess having a meal at a restaurant is far more important than getting an MRI, given that the restaurant people have made the process simple and convenient and effective for all concerned. Getting a medical test must be rare and unimportant, since no one has bothered to make it work well. Sure. I guess that lie is more comfortable than the other possibility, which is that medical system administrators and software providers are too incompetent, lazy or unmotivated  to make things moderately convenient and up-to-date for their employees and customers.

Why this is an important issue

If you’ve gotten this far, you’re already an unusual reader. The vast majority of the leaders, innovators, experts and generally high-prestige people in healthcare would have tuned out and moved on as soon as they saw “medical scheduling.” Their guts tell them “medical scheduling isn’t important.” In terms of career growth and industry prestige, their guts are serving them well.  However, in terms of making a real difference that will positively impact most people involved in medicine, their guts are misleading them.

Medicine has a real prestige problem. I’ve described the medical innovation spectrum, in which the exotic, future-oriented end, like AI and Blockchain, gets most of the money, conferences, attention and career-advancing opportunities. The more you move on the spectrum to simpler, proven, non-exotic things that can make a difference here and now for lots of people, the more you’re moving to the back office or basement, where poorly-paid, invisible people while away their time and whine to no meaningful audience. Here are details and examples.

Scheduling is one of those proven winners that remains largely unimplemented in major medical organizations. The fact that scattered medical groups have implemented it beautifully shows there are no technical barriers.

I can hear it now. Scheduling. Sure. What award am I going to win by implementing something restaurants do? I’m breaking new ground in personalized medicine while I’m curing cancer on the side! Away with your trivial scheduling talk!

I get it. There’s just a little problem. As I’ve detailed here and here and here and here and more, time and money loss and serious medical issues are caused by nuts-and-bolts problems in the medical system. If just some of them were improved, the funding for your precious futuristic projects could be increased! And, by the way, loads of patients would be better off on multiple dimensions, including not dying prematurely! You know, little things.

As an example of the impact, consider just this aspect of scheduling: automated follow-up (yes, it’s part of scheduling). Failure of follow-up problems:

“The impact on patient outcomes included missed cancer diagnoses.” Journal of General Internal Medicine.

“In fact, almost a quarter of all medical errors occurring in outpatient settings can be attributed to poor follow-up of abnormal test results and are believed to represent 25% of malpractice lawsuits involving failures or delays in diagnosis.” AACC

The Scheduling problem

There are an amazing number of dedicated, skilled, hard-working professionals in medicine. There are patients who have health issues who are often grateful for the service and care provided by those professionals. But both groups, providers and patients, are burdened by ancient, dysfunctional and incredibly expensive processes and computer systems that make things that should be quick and easy into something that is cumbersome and time-consuming, often yielding poor results. Everyone involved feels trapped and needlessly harassed. What’s going on here?

This is the horrible general scheduling problem in health care. To illustrate the issues in a concrete way, I’ll focus on scheduling a test as a follow-on to a procedure, and use my own experience as an example – an example that, sadly, is business-as-usual in this world.

A Cat Scheduling Example

Before getting into scheduling a medical test for me, let’s see what happened when I had to schedule a test for someone more important than I am – at least she seems to think she is – my cat, Priss. The comparison between getting tests for Priss and me will be instructive.

I have a cat, Priss. Priss is pretty chill; here she is thinking deep thoughts:

I take Priss to a local animal hospital, where she is well cared for. I get regular notes and emails telling me when something medical needs to be done for Priss. For example, here’s the header of an email I received last year:

Note that the email arrived near the end of August, 2018.

Here’s the top part of the mail:

Note that there’s a picture of a cat in the email – they know who they’re dealing with! They know Priss’s name, and they know she’s a cat, and not one of those friendly but face-licking, slobbering dogs who are incapable of using a civilized thing like a litter box.

Here’s the rest of the email:

They make it hard to miss the phone number or the online scheduling system they support to make an appointment, don’t they? They also spell out exactly what needs to be done. And darn, the email was sent EXACTLY one month, to the day, before the service was due. Please note: this is THEM taking the initiative to follow up, TELLING ME when something needs to be scheduled.

If this were a human hospital we were talking about, I would say “to make a long story short …” but because this is an efficient, modern animal hospital, I get to say “to tell it like it is …” all I had to do was click on the email, buzz through their easy-to-use on-line scheduling system, make an appointment that worked for them and for Priss, get a confirmation email back right away, and I was done. That’s it!

On top of it all, the people there are great, and care for Priss really well. There was a problem with her fecal test -- results back in a single day -- and the treatment was immediate and effective, no more than a day's delay for any step.

An MRI Scheduling Example

Now we turn to hospital scheduling for humans, specifically for me at the world-class hospital system, Mount Sinai. I’ve had some medical issues that have required MRI testing. See this for a description of the issue, with details about the testing experience and the conclusions that can reasonably be drawn from it.

I had  30 days of radiation therapy in early 2018. I was supposed to have MRI’s to see the results of the radiation six months and a year after the treatment. This is standard practice. I know from experience that the grandees at Mount Sinai Medical System haven’t gotten around to sending a team to the Hudson Animal Hospital to learn from their cat scheduling system, so they can put a multi-year project plan in place to implement something similar for their human patients. In a human medical world that worked as well as my local little animal hospital, I would have gotten a reminder a month before my MRI should have been taken.

Being the patient patient that I am, I waited until after the anniversary itself. Then I sent a friendly reminder to my doctor:

He replied the next morning with a perfectly reasonable response – given that he had to do the work that a vet wouldn’t have had to do.

His prompt reply was Feb 4. I heard nothing for weeks, so I finally reached out again three weeks later:

Again, the doctor in charge of the radiation center, doing clerical work well and promptly, work that he shouldn't have had to do, responded right away:

What happened next? More fun wasting everyone’s time. Remember, with a reasonable on-line system like restaurants and vets have, none of this would be needed!

A couple days later I got a voice mail, with a friendly person giving me the date and time for which my MRI has been scheduled – of course, without consulting me! I guess I must be sitting around suffering, anxiously waiting for the first possible moment at which I can get my MRI, at which point I’ll drop everything and arrive two hours early. Or not. Even better, this call isn’t from the MRI center, it’s from my doctor’s office, where someone has made an appointment for me, a couple days after the most recent request, weeks after the original one. So I return the call, explain I can’t make the appointed time, and is there any way I could talk with the MRI center and make an appointment myself? Well, she tries to be accommodating, and says it’s OK, but I also need to make a follow-up appointment with the doctor by calling her. I think I can manage this.

I call the number she gives me, which was the wrong number. I consult the web, and it takes a typically long time to find the center and its phone number for scheduling on the website. Mount Sinai management, it goes without saying, is totally on top of customer feedback and quality management. So they pop up an opinion survey. What happened with the survey is a lesson all by itself in the profound dysfunction of our medical systems in general, and Mount Sinai in particular. I will treat this event in a separate post.

I call what the website says is the right number, and keeping sarcasm – however warranted it may be – to a minimum here, after a journey through automated VRU’s and other wrong numbers, I eventually get to the person who can schedule me. Sure enough, she finds the order for the MRI in her system, and makes an appointment that actually works for me. Phew! I then, as instructed, call back the main doctor’s office and schedule a visit with him to go over the results, as I had been instructed to do. Please note this visit with the doctor I scheduled; it will start its own little trail of incompetence and waste.

After way too much effort on everyone’s part, I thought things were finally set. Silly me. I got this call a few days before my MRI appointment:

After a couple of attempts, I get through. I’m told that the room where the MRI machine is suddenly needs to have major work done, work that apparently couldn’t be scheduled in advance, making the machine itself unavailable. So pick another time. We work something out. I then call the doctor’s office and re-schedule my visit with him, so that it’s suitably after the new date of the test.

On the morning of the date of my original doctor visit, the one I called and re-scheduled, I get a voicemail saying in effect, “you’re coming into today as scheduled, right?” I admit that the nasty thought of ignoring the call crossed my mind, but my better self took control. I called and got through after a couple tries and explained the re-scheduling. The person verified that the new appointment had in fact been made, but that “someone else” must have failed to cancel the old one. Done. Except that it wasn't really "done." See the associated customer service post to see what happened!

A week later something came up and I had to move the appointment with the doctor. I called. After explaining what I wanted, the person said, didn’t you just miss a scheduled appointment with the doctor? Nope. Apparently the last person I talked with, whom I told about their mistake, failed to correct it. Again.

Do you use a calendaring/scheduling system, for example the one in Microsoft Outlook? Have you noticed that you can click on an event and change the date/time, so that it moves to the new time slot? Of course, you can delete the old one and make a new one from scratch if you really want to, but why would you? The evidence seems to point to the possibility that such a feature, which is standard on modern calendar systems, doesn’t exist in the paragon of modern software used by the leading medical system, Mount Sinai.

Conclusion

Follow-up of events, both one-time and multiple recurring, is a standard feature of modern scheduling systems. Self-scheduling by software is a widespread feature. It’s widely used, and benefits everyone involved. It’s not as though self-scheduling by software is particularly difficult for medicine. For example, the primary care office I use, OneMedical, has a convenient system that takes account of the length of visit you need, and gives you choices of providers and locations so you can pick what works best for you.  Once scheduling is on-line, it is amenable to serious optimization techniques, which have been deployed with great success resulting in substantial savings and efficiencies in things like infusion centers and operating rooms. This is not possible with the primitive human-phone-based systems in widespread use.

It’s painfully obvious that the important people at medical centers prefer to spend time doing “important” things at the fancy future end of the innovation spectrum, rather than lowering themselves to implement practical, here-and-now improvements that benefit everyone. When will this change?

 

Software is a Pre-Scientific Discipline

For a wide variety of human-understandable reasons, software is perceived as a science. In academia, it's taught in the Computer Science department, often part of the math department. What could be more precise and scientific than that?

Whatever its pretensions, software is anything but scientific. It's mostly driven by fashions and fads, led by "experts" who promote theories that sound good when described -- but which entirely lack any form of scientific process, testing or evidence.

Software diseases will continue to severely hamper our computer systems until we wake from our long, pre-scientific sleep. We will know we're making progress when software practices at least to the level medicine had achieved 150 years ago. See these examples: the history of scurvy; the history of bloodletting; Yahoo and Hadoop.

The Evolution of Science

Science is not one thing, although the core principles of hypotheses, testing and evidence are always the same. Sciences don't pop up full-grown from nothing. Each field of human endeavor evolves towards being a science (or not) at its own time and pace. There is typically a long gestation period during which resistance is deep and widespread. Physics is a classic example.

Even when an area of science is well-established, the temptation to simply declare and assert the truth of something without careful proof remains strong and happens all too often. Science is something that human beings do, so it's never "perfect." It's also never "done." The resistance of the entire physics establishment to the now-accepted fact that time changes as the speed of an object approaches the speed of light, and that light has no mass but nonetheless exists and travels at a measurable rate, are classic examples of the fits-and-starts evolution of even the well-established science of physics.

It was a long, hard slog for medicine to emerge from its pre-scientific state. While there is loads of room for improvement, as I have often pointed out in this blog, medicine has clearly and explicitly embraced scientific discipline in the large majority of its practices, of course with the occasional embarrassing slippage.

The non-evolution of software towards science

Let's compare the emergence of powered flight as a science to the methods for building software projects. Here are the key points:

• Powered flight was widely recognized as important more than 100 years ago. There were widely accepted experts, and the entire establishment gave them money and support. After a couple spectacular failures by the experts, the obscure people who actually figured out how to create a heavier-than-air flying machine got it done, and their methods were soon universally followed. Here is the story in more detail.
• Building effective software is widely recognized as important today. There are widely accepted experts, whose methods are taught in schools, practiced in all major institutions and mandated by government regulations. After spectacular, repeated failures, everyone says "oh, that's software, what can you do," and moves on. Meanwhile, sometimes obscure people build amazing new software quickly and well most often using unauthorized methods. Their software is widely used and their companies acquired by the big organizations who can't build anything. Nothing changes. Here is an example and details.
  

To take another example, while there are many ways that the science of medical drug development could be improved, there is little doubt that it is a scientific venture. In terms of science, in spite of its problems, limitations and inefficiencies, drug development is probably a hundred years ahead of "computer science" in general and software development in particular. See this for a comparison.

If software were a science

Think of a list of established principles in software -- if software were in fact a science, the things that would be like the basic equations of non-relativistic motion. What's on the list? I suspect it includes things like: Object-oriented programming, comprehensive test automation, architecting for scalability.

Now think of a list of hot new methods or techniques in software, things that are widely accepted but early in widespread adoption. The list may include, depending on the circles in which you travel, things like micro-services, the Clojure language, Agile methodology with SCRUM, test-driven development.

Which of the items on either list -- your version of the lists, not mine -- went through anything like this process:

There was a bad problem that accepted methods weren't solving. People hypothesized an underlying cause and/or a cure, tested it first on a small scale and then on a larger scale. The evidence was overwhelming in A:B comparison that the cure was effective, so it became accepted.

Or,

There were observations that didn't fit existing theories, data that wasn't explained, or discrepancies that couldn't be accounted for. Someone came up with a theory that made sense out of the rogue data. Others formulated the theory exactly and conducted careful experiments; the results were made public, and maybe there was a period of refinement. Finally, the new theory was accepted, because it was experimentally proven to account for all the measured data, something the old theory could not do.

Anyone with reasonable software experience knows the answer to these simple questions: software doesn't work that way! Not even a little bit! Instead, new practices are invented, promoted and sometimes accepted into common practice. In no case is there a scientific vetting process! People just accept the theory because

• it makes sense to them, or
• it's what they've been taught, or
• it's required by the mandated practice of the group in which they work
• it somehow advances their career or enhances their prestige

Sometimes, happily, software fads just fade away for as little reason as they started. A fairly recent example is pair programming, which I describe and examine here.

In the face of this evidence you may swallow hard and admit that software may not be a science, but it is an established discipline with standards and processes that are widely accepted, as for example you can see in the FDA software regulations. Sadly, that makes it even worse, if it's possible to imagine that. The standards and processes that constitute modern software practice are taken from other fields and jammed onto software. They don't fit and they don't work.

Conclusion

No testing. No hypothesis with controlled experiment. No evidence. No process that resembles either medicine (we know there's a problem, we have a possible solution, let's prove it works before using it widely) or physics (we have this data we can't explain, let's propose a theory that accounts for it and run experiments that will prove it right or wrong) or anything else.

You can say that building software is part of "computer science" until you're blue in the face. You can require CS degrees for your new hires. But the evidence is that software is, without question, pre-scientific.

We need to at least start building towards a true Science of Software.

 

Due Diligence: Financial, Legal and Software

When you are going to acquire or make a major investment in a company, it makes sense to perform due diligence. Not only does it make sense, it's a fiduciary responsibility! Due diligence on the company's financials are a no-brainer. Same thing with legal. Wouldn't want to be surprised by a landmine in one of the contracts, would you? If there is software involved, most people perform technical due diligence. In each case, you hire an expert who combs through the relevant portion of the company. The expert produces a report, at minimum highlighting any deficiencies.

They're all pretty similar, right? They're abstruse subjects involving deep expertise where there can be landmines and hidden messes that can have great consequences. That's why you have due diligence.

Yup, that's the standard view, Due Diligence 1.01. It's even all reasonable ... except for Venture Capital investing in tech companies.

Due Diligence for computer and software technology

The vast majority of tech due diligence involves fairly large companies with large staffs. When you're looking to do a deal with a fairly large tech company as the target, it's business as usual. You want to make sure everything is done to industry standards. There are probably security and other regulations. Are they in compliance? Are they following industry-standard development methods, or is there some kind of willy-nilly, out-of-control stuff going on? Do they actually have a QA process that assures that new software releases won't go belly-up and hurt the company and its customers?

This sounds pretty much like legal and financial diligence. Because it is!

Tech Due Diligence for VC's

How can a VC responsibly invest in a young software-based startup without understanding the software? Most investors aren’t programmers, after all, and can’t “see” the software. No big deal! VC’s usually aren’t lawyers or accountants either, after all, and there’s a standard solution: hire an expert to perform due diligence. Hire lawyers for all the legal things, accountants to pour over the books, and technology due diligence people to handle the tech. Now that wasn’t much of a problem, was it?

There's just one little problem: the standard methods for evaluating software lead to terrible results. This shouldn’t be surprising, because the standard methods for building software lead to terrible results! If the industry’s best minds can’t figure out how to build software effectively, why would anyone think that these best minds would be any better at evaluating software and the organizations that create and maintain it?

You won’t find these industry-leading groups and thinkers wringing their hands or wailing lamentations at their on-going failure – that would be bad for business! So instead, awfulness is accepted as just how things are. So when experts evaluate software groups, the typical process is to compare them to how the vast majority of groups “get things done” – kinda, sorta, eventually, with failures and massive security breaches attributed to some combination of life and the existence of evil criminal masterminds. If the group fails to sing the standard songs and say how wonderful the software dances that are currently in fashion make them feel, then questions are raised.

Large corporate and government organizations know and accept all this. They take comfort in striving to attain industry-standard “best practices,” and give presentations about how great their direction is. They may put signs up on walls, or hang banners making vague but strong assertions about “innovation” stuff involving the future.

What if you’re a small organization, a start-up or not much more advanced? For the VC point of view, this makes little difference. They tend to have due diligence providers with solid methods and a track record the partners of the VC firm are comfortable with. This fact of existence leads to a few major behaviors among entrepreneurs:

• Avoid any VC that wastes your time putting you through what you know to be irrelevant evaluations
  
• Hold your nose and go through with it, while trying to gull the evaluating firm into thinking you're more industry-conforming than you are, and to the extent that you're not, you're planning to fix it all up real soon
• Really change things so that you match industry-standard evaluations, often by hiring a tech leader who's "been there done that."

Many VC's pride themselves on their judgment of character, and won't take the tech due diligence really seriously anyway -- experience has taught them that it's mostly a pile of irrelevant gobbledy-gook anyway. They place their bets on the company leaders, whether they have the vision, drive and smarts to create a winning company, or at least to achieve lift-off.

This is a real problem, but it logically reflects a larger problem in the tech industry, one that I've tried to articulate in a variety of books and blog posts: there is a huge gap between how the vast majority of software organizations build software and how the most effective small ones build it, not unlike the difference between army draftees just out of basic training and a team of elite Rangers. They both fight, but if made to fight each other, it wouldn't be a "fair" fight. Same thing with software, only even more so.

The vast armies of software engineers, armed with their impotent and irrelevant Computer Science degrees, resemble draftees out of basic training more than anyone would like to admit. There are so few software Rangers that most of the plodders rarely encounter one. Rangers certainly don't hang out by the water cooler hoping to chat with one of the sloggers -- a real Ranger wouldn't want to enter the building, much less be employed there.

The very best tech due diligence for VC's attempts to differentiate between these cases, which aren't totally distinct:

1. The company develops software following industry standards.  If what they're doing is amazing and pretty much already baked that could be OK.
2. The company isn't following standard norms or any other norms. It's just disorganized. This is a problem. There's a risk in whether it can be fixed. Know what you're dealing with is essential.
3. The group is not following industry norms, but is achieving rapid progress with good quality using non-standard methods -- not anything-goes wild-west, but non-standard techniques that are actually methods. There is a wide range here, which must be judged.

The trouble of course is that standard, off-the-shelf tech due diligence conflates cases 2 and 3 above. Truly effective tech diligence for VC looks for case 3, and judges to what extent the company achieves it.

Conclusion

Performing effective tech diligence is pretty similar to financial and legal diligence -- unless you're investing in a break-the-rules, innovative startup or young company and you're a VC. In that case, you're best off judging them by different standards than you would normally use. In finance and legal, you're looking to find things that are unusual and diverge from norms -- because they could be bad and hurt you. In tech, you're looking for things that strongly differ from standards and norms -- specifically methods that trump those standards and norms. You don't want a group that fails to live up to standard army discipline. You want a group that chooses to do things differently -- because they do them better, WAY better. That's what makes winners, and that's what VC's want to invest in.

An Immutable Distributed Ledger is now in Large-Scale Production!

First there was Bitcoin, friend of criminals, speculators and tech geeks everywhere. It’s grown amazingly. Then there were alternatives to Bitcoin, often sharing much of the same code, but with different and incompatible tokens. One of those Bitcoin alternatives, Ethereum, introduced the concept of Smart Contracts, which I discuss here. Now, increasing attention is being paid to “blockchain,” said to be the foundation on which crypto-currencies like Bitcoin and Ethereum are built. Large corporations are taking up the charge, places like IBM and Microsoft, and leaders in various industries have projects going to prove out the technology. While the terminology isn’t uniform, it’s easy to see that earlier terms with unsavory associations are being abandoned in favor of more generic terminology, names like blockchain,  “Immutable Distributed Ledger” technology, or just “distributed ledger.”

Once you start talking about the technology in generic terms, what are the chances of this actually working? At scale? In practical reality? Lots of people in the community of blockchain enthusiasts have expressed concern about this. Legitimate concern. The question naturally arises, but appears not to have been asked, has something like this been built before? Something that could legitimately be called an immutable distributed ledger?

The answer is a simple “yes.”

This amazing system, which is one of several in production today, has over 2 billion accounts and over 40 million participating agents. It moves over $10 trillion per year, processing over 150 million transactions a day, and can handle over 50,000 transaction messages per second.

Let’s dive into the technology a bit:

• It’s immutable. Once a transaction gets in the system, it can’t be altered or removed. Despite the volumes mentioned above, spread over more than 200 countries, there are no instances of processed transactions being altered.
• It’s distributed. Computers all over the world are involved. It doesn’t matter where you are: this system enables you to send or receive currency. Even better, currency conversions are built into the system! The distribution has been built in part to enable reliability. While any one computer in the system can fail, the system as a whole has never gone down, and transactions have never been lost.
• It’s a ledger. It’s a gigantic ledger of currency going into and out of accounts. The ledger balances to the penny every single day.
• Consumers don’t lose their money! In spite of all the volume, When bad stuff somehow happens, consumers lose nothing. Nada.

OK, OK, I’ll stop the charade. As you’ve probably guessed, the Distributed Ledger technology I’m describing here is in fact VISA. I’m playing this game because reading about how enthusiasts talk about blockchain, I wonder how many of them know about credit card internals? If they did, they would see that all the goals they have for blockchain have already been achieved in credit cards. And more!

The key assumption at the core of the blockchain craze is that blockchain is an amazing new technology, a breakthrough that enables all sorts of long-intractable problems to be solved. These virtues are primarily the fact that it’s immutable, it’s distributed and it’s a ledger. Sorry, guys, you don’t need blockchain to build a system that has those attributes. It’s already been built using plain old database technology and secure networking.

Yes, I played a game when describing the immutable distributed ledger technology that’s already in massive production, knowing everyone would think I was talking about blockchain. But the blockchain groupies are playing a more serious game, convincing themselves and others that blockchain is uniquely able to do things that haven’t been done before, and could never be done without this amazing new invention. Just to be clear, I’m not saying that VISA technology can be applied to the many problems to which blockchain is being applied. I’m simply saying that if you think you have a problem for which an immutable distributed ledger technology is the best solution – that problem can be solved without blockchain, more quickly and with a lot less effort. VISA is just one of many full-scale, in-production examples.

It's sad that the Blockchain mania is so powerful that no self-respecting executive can risk ignoring it. So all the big banks and even, yes, VISA itself have blockchain based projects underway, nearly all of them involving widespread use of the future tense. I am open to the possibility that some of them may even be deployed in some way, once the people implementing them get realistic and relegate the blockchain technology itself to a tiny, marginal aspect of the project's code base, and realize that everything they're doing could be done better and faster without the distraction of largely irrelevant blockchain. But meanwhile, it's great for reputational enhancement and attention-getting!

A version of this post was published at Forbes.

How We Got Chatbots for Mobile Financial Apps

Chatbots for financial and other applications aren’t just a cool new thing. They’re a necessity! They solve the worsening problem of too many options to choose from on shrinking screens, with unhelpful help screens.

Do you access your financial accounts online? If you do, perhaps you’ll remember that the first time you tried to do something, you had the fun of poring over the menu system to find what button to click, sometimes only to reach another screen full of buttons and menus. On the plus side, online financial systems let you get a lot done. On the minus side, jumping through the hoops to actually get them to do what you want can be a long slog. Have you ever gotten frustrated and tried to click on Help? And gotten the long, unhelpful Help stuff? Helped a whole lot, didn’t it?

Today, nice big screens with high resolution, sitting on a desk somewhere, are used less and less. Cute, portable little screens with phone and camera built in are used more and more, partly because it’s in your pocket, right there when you need it. What happens to all those giant screens packed with menu choices? I guess the designers could have reduced the typeface so much that you’d need a magnifying glass to read them, but they bowed to reality and put just a few menu choices on each screen. Nice to read, but the result is that your multi-screen journey to get to the one you want got even longer. Assuming you remember how to get there. And what if you want something more elaborate, like how close am I getting to hitting my budget for restaurants this month? Fuggedabout it.

The designers of mobile apps for financial applications aren’t between a rock and a hard place. It’s worse. They’re stuck way back in a long, narrow cave that floods. What to do?

Echo and Siri have been training us to just ask for things. Who won the game last night? What’s the weather forecast? Even jokes! But banks … now that’s a different matter altogether. Banks are serious things. There’s money involved. Not just any money – MY money!

So what’s a bank app creator supposed to do?

It’s actually pretty simple, because there’s not much choice. If you want the bank app to be, you know, USED by the people who have it, you have to make it USABLE. Period. There’s not enough room for menus, except maybe a couple super-popular buttons. Help files? Waste of time. You’re down to one choice: make it so you can TALK (or chat) to the app, and ask it to do stuff for you.

That’s the logic. Even better, it’s really happening. In real life!

Bank of America is advertising its chatbot Erica heavily in some parts of the country. The reason is simple: they want their customers to be able to USE the BofA app, not be frustrated by it. There’s the chatbot technology provider Kasisto (Disclosure: Oak HC/FT is an investor) used by multiple banks to power human interactions. The bar has now been raised for financial institutions with apps.

Let’s review some history. Years ago, what financial institutions had to have was a website for customers to access their accounts. As smartphones spread, the bar was raised: OK, you’ve got a website, but do you have an app? Not having an app was a reason for customers to move to a place that did. Just as most of the financial institutions were breathing sighs of relief that they’ve caught up, the bar is raised again: you mean we have to make the app USABLE? What’s a chatbot, anyway? The pattern here is clear: technology keeps marching along, some financial institution applies it to their customers’ and their own benefit, and the others scramble to catch up. What’s next?

To find out what’s next, all we have to do is look at the non-financial domains that use chatbots. For example, look at Amazon’s Echo. It started out pretty primitive, but it’s adding capabilities all the time – as Amazon puts it, new “skills.” The bar is already being raised by technology vendors in two specific, technically challenging areas:

It’s one thing to answer a simple question, like what’s my balance? But real chatting as done by people requires that the chatbot take into account complex questions, context and history.

A complex question requires doing some real “thinking.” For example, “How much did I spend on eating out last month?” has loads of complexity. The system has to know that “eating out” means spending money on the merchant category “restaurants.”  It’s got to find all those transactions that took place in the month before today’s month, add them up and give you the answer.

Taking conversation history and context in account is even trickier. Suppose you get the answer to the eating out question. Suppose you now ask “ How about the prior month?” Easy for a human; for a bot, not easy at all! The bot has to figure out that you’re still talking about eating out (restaurants) and that you want the total for a month. It’s also got to figure out that you want not last month, but the one before that. These are the kind of amazing interactions supported by Kasisto.

Chatbots for financial apps are just being rolled out to solve the otherwise unsolvable problem of screen real estate. Even though we’re still in the early roll-out stages, the next battle is clear. Most of today’s chatbots are in elementary school – soon they’ll need to graduate to middle school!

A slightly different version of this post originally appeared in Forbes.

How Can an Immutable Distributed Ledger have Assets Lost or Stolen?

As I’ve described, cryptocurrency losses started long ago and have kept on mounting over the years. Most recently, the largest cryptocurrency exchange in Canada, QuadrigaCX,  has experienced a little problem that has resulted in what appears to be a permanent loss of over $130 million for its many customers.

The losses don’t seem to make any sense. My account is stored in a ledger that’s immutable – it can’t be changed because it’s locked down by unbreakable cryptography. It can’t be lost because the ledger is distributed, so even if a few computers are lost, there are still loads of computers with a copy of my unchangeable account balances in them. So how can it be that I can lose my money? To further increase security, Bitcoin and the other crypto-currencies don’t identify me by my name, but by a really loooong string of letters and numbers. And that’s just the name – to make any changes, someone would need access to my private key. Remember, “private key” isn’t just some password that’s easy to remember and maybe someone could guess – it’s a long number that's part of a proven-unbreakable cryptography scheme, totally guaranteeing that no one but me can access my accounts? It’s impossible for anyone to break through these layers of security, that math guys everywhere calmly assert just can’t be cracked. So what are these losses about???

The only reason this is mysterious is because the vast majority of stuff we read about Bitcoin, blockchain, distributed ledger technology and the rest is produced by adherents to the new cult. They’re promoters, not evaluators. Let’s start with encryption. Loads of things are encrypted – it’s hardly something that started with Bitcoin. Most sessions with websites are encrypted, as you can see by the https in the URL bar in the browser. I hope you only use encrypted WiFi. Email is encrypted. All databases run by moderately competent professionals are encrypted. And so on. Are websites, email systems and corporate servers hacked in spite of it? You know the answer. People can wave their arms and babble with passion about the power of unbreakable encryption all day long, but bad things happen in spite of it. Is it because encryption is in fact breakable? Nope! It’s because even the biggest castle with the most unbreakable walls has to have doors that are easy to open and close, though which properly authorized people and goods go in and out. Bad guys waltz right into the castle with unbreakable walls through the same easy-to-open doors that all the residents and servants of the castle use! How are those doors locked and guarded? Using absolutely proven, tested, tried-and-true methods such as … user names and passwords! And sometimes even PIN codes on top of it!!

Oh, no, everyone but everyone talks about how safe and immutable the distributed ledger is – how is it possible for it to be broken, and why are ancient, obsolete things like usernames and passwords involved? It’s simple, really. Say you’re an experienced break-and-entry person. You walk down a block at night. Most of the houses you see have fences, big locked doors and lights on. One of the houses is dark, and its front door is swinging free. Which house do you pick? Everyone scans for opportunities and goes for the weakest point. So no one even tries to break the cryptography – it really IS secure. But what about the place where the long ID and the no-way-can-I-remember-it private key are held? AHA! We’ve gotten to the weak point of the system: the wallet.

Wallets are just pieces of software that run on your phone, your computer or even online. There are dozens of wallets available. They all claim to store your Bitcoin securely, and maybe they do – except they need to allow a normal human being to actually use them, so they need to be opened by someone a normal human being can handle, like a user name and password! The super-secret but impossible-to-remember private key is stored in the wallet, among other things. Without the private key, you have nothing. If you lose or break the phone that has the wallet, you are screwed. And of course if your phone or computer is hacked, someone else will immediately transfer your Bitcoin out of your account into theirs. There is no recourse. None. Think that never happens? It’s been going on for years, see this. A 20-year-old college kid was just sentenced to 10 years in prison for stealing more than $5 million by phone hacking, see this. That was a first-ever case – usually, no one is caught.

Some people, concerned about the security of their wallets, have turned to professionally-managed exchanges, which both store crypto-currency and enable it to be converted and transferred in and out. An example of such a company is Quadriga, which was the largest such exchange in Canada last year. Now it has made various court filings, and is unable to return its customer’s roughly $200 million in assets, see this. Why? Because those assets were held in a wallet on the now-deceased founder’s laptop computer, and no one can get into the computer – its contents are encrypted and the access information was last seen somewhere in what was the founder’s brain.

Yes, the distributed ledger is secure and immutable. Yes, the cryptography used is in fact unbreakable. Most descriptions stop there, leading you to think you’re protected against loss or theft. As I hope is now clear, Bitcoin and its numerous offshoots do indeed involve some amazing technology, as I’ve described here. But that doesn’t mean that it’s any more secure or protected against loss than any other software. Because of its immaturity, it is proving to be more vulnerable to loss and theft than “normal” money. And the non-monetary blockchain solutions are just as vulnerable.

This post originally appeared on Forbes.