The Black Liszt

The Evolution of Software

There is strong interest in the latest developments in software. No one wants to be left behind.

At the same time, there is a peculiar (to me) lack of interest in whatever the latest thing grew out of or evolved from. What are the new thing's predecessors? What did that thing grow out of. Have similar things appeared in the past? Are there patterns we can observe here, or are the new software things that explode onto the scene just a meaningless sequence of random events? Are they things that we obsess about while they're here without questioning or wondering where they came from, without asking what else they may resemble in some way? Are they things we just forget about and ignore once they depart the stage of software fads?

The questions have obvious answers. Answers that no one seems to be interested in.

So long as we fail to ask these crucial questions, we will remain ignorant of software in deep ways, and will continue to stumble from fad to fad without improvement or understanding. History is important!

Evolution in Science

If you study different sciences and their history, you notice that, at some point, scientists begin to pay attention to evolution, i.e., change over time of the thing being studied.

The most obvious thing is evolution of an individual item over time. In biology, you have the cycle from birth to death. In geology, you have the change of mountains, rivers, beaches and glaciers. In materials science you have the change of objects as they are subjected to various conditions over time, for example the way that iron rusts.

Evolutionary studies reach a breaking point and get really interesting when you study groups and types and see the patterns of change. In biology, this was the revolution that happened with Darwin and his study of the evolution of species. Something similar happened in geology with plate tectonics.

When you put the patterns together, things get totally amazing and transformative. While no longer fashionable to talk about and study, the parallels between the growth stages of an individual animal and the evolution of species are obvious, though not literal at a detail level.

Studying evolution is an important stage in the evolution of most sciences!

It's long-since past time to study software evolution, as an integral part of software science.

Software Evolution

Software evolves at every level.

At a basic level, software languages evolve. I have given an introduction to this subject here. The evolution of software languages resembles biological evolution in remarkable ways, as species (languages) emerge and evolve in response to new conditions, including other species. Similarly to biology, languages evolve; sometimes many species descend from them; and sometimes a species goes extinct. See this snapshot, for example:

(Credit)

Consistent with the determination of nearly everyone involved with software to ignore its history, this chart and anything like it are ignored, and the evolutionary principles behind it are not studied. When you start to study them, you learn amazing things. Suddenly some of the random change in the world of software languages starts to make sense.

Software languages are the very most basic things about software. Just as interesting and vastly more important are the programs written in software languages. Everyone involved in writing or maintaining a software program (application) just thinks about that program. It definitely occurs to business people what the program can do compared to its immediate competitors, if any. Yes, there is a kind of "survival of the fittest" in software evolution.

What about the "insides" of the program compared to other program insides? Sometimes people are aware of gross differences, like the language the program is written in. But the curiosity normally stops there.

The program "insides" have huge practical consequences. Depending on the details of how a program is written, it can be more or less:

• reliable/buggy
• easy to hack (secure)
• expensive to run/operate
• dangerous/costly to change
• able to respond quickly to changing loads
• speedy or slow

People commonly talk about structure or architecture, but those things are just the tip of the iceberg.

When you dig in, you find the equivalent of islands and continents of software. I've treated this subject here, for example. For example, you may have spent some number of years after getting your CS degree working on website design, and interacted with a community of people similarly engaged. Not everyone uses the same tools or does things the same way, of course, and you're likely to think there's quite a variety of approaches.

Then you go wild, dive into hospital management systems, and your mind is blown. What you thought was the wide variety of software and tools used by website designers turns out to be a whole continent apart from what you find in hospital automation. You learn that the flagship product of the company that has more than a third of the market (Epic) is written in a language you've never even heard of! It's like growing up in New Jersey among what you think is a very wide variety of people and going to Chennai -- a huge city in India you've never heard of, whose population is larger than your whole state, where the main language is one you've never heard of, written in a script you've never before seen.

Even more ignored are the obvious (to those who trouble to look) repeating patterns that can be observed in applications as they are first written and then evolve over time.

The Study of Software Evolution

To put it bluntly, the study of software evolution has barely begun. A few isolated souls, hardy or foolhardy as you like, have dipped their toes into the deep waters of software evolution. Those few have found worlds to explore, oceans whose depths have yet to be plumbed.

This is particularly the case because software is a "the only reality is the present" kind of field, as it now stands. It doesn't have to be this way!

Very few people study the history of math. They don't need to: anyone who learns math starts with math as it existed thousands of years ago. If they do well in math, they get all the way up to ... the late 1600's, when Newton and/or Leibnitz discovered calculus! Each step in math is based on and often depends on the earlier steps.

In some sense, this is also true of software. Except not really.

First, knowing nothing about software, you can dive right into the latest language and tools and use them to make something that works. Anyone think you could do that with differential equations, I mean without already knowing how to count, add and multiply? The difference is that in software, the "accomplished" new-comer really is standing on generations of older software ... of which s/he is blissfully unaware! The new programmer, of course, uses an incredible array of operating systems, drivers, networking, file and database systems, and on and on. These form the "land" on which the new programmer "stands" to work. None of the wonderful new stuff would work without it. The big difference, of course, is that using the underlying software and understanding the underlying software are two rather different things. This makes software totally different from not just math, but most of the other legitimate sciences (leaving out the fake ones like political "science" and the rest).

Second, when you're building a program, chances are excellent that you're not the first person to try building something very much like that program. Lots of details may be different, but even the details are probably not as unique as you might think. Suppose you need a program that sets up an account for a person, and then when the person does stuff, the details are checked with the person's account, and if OK are added to the account, and if not are rejected but recorded. This high-level description applies to a sales operation with customers, a medical office with patients, a manufacturing company with customers, and a bank. Given that programs like this have been built tens of thousands of times, don't you think that the ways of building programs like this would have evolved? That there would be pre-built parts, specialized tools for using the parts, building new ones and gluing them together? Special methods that have evolved and been refined to make sure it's done in the optimal way? Did any of you with any level of degree in Computer Science learn any of that -- other than, of course, some boiler-plate about the virtues of object-oriented languages and maybe some other fashionable stuff? Of course not! Anyone with any sense of fashion and/or status knows that these are not career-advancing subjects. Anything involving software history or comparative study of software projects is career-killing.

Conclusion

Software evolves. But it evolves differently than other things studied by science. Virtually no one studies software history in any way, much less the patterns of evolution that become apparent when you study that history. There have been a couple of attempts to learn why software doesn't get better in the way many other things do, most notably Fred Brooks with his book The Mythical Man-Month and other writings. I have dissected this flawed analysis in my work on Software Project Management and Wartime Software. See this for a start.

I have worked for years trying to identify some notable patterns in software evolution, and will be releasing some of that work in the coming months.

The Aftermath of the Net Neutrality Disaster

After many years of fighting against entrenched corporate interests, the FCC finally extended its regulatory authority to the internet, instituting the so-called "net neutrality" rule in 2015. Finally, the internet would remain free and open.

Then a new administration came in, with new leadership at the FCC. There was a move to repeal the common-sense principle of "net neutrality!" Why would anyone think that letting giant corporations pick and choose what we can see on the internet was a good idea, making things they didn't like too expensive or simply blocked -- crazy! It's kind of like repealing the laws against murder, hoping that people would just not do it!

Some leading Democrats made it clear what would happen if net neutrality were repealed:

 

The senate was being careful here -- what about ISP's blocking access to sites they don't want you to see? Senate leaders were outspoken about the danger:

People were upset. There were demonstrations at more than 700 Verizon stores nationwide:

Protestors even spread out through the new FCC chairman's neighborhood, demonstrated in front of his house, took pictures of the inside of his house through his windows, and made it clear that this man, Ajit Pai, was leading the movement to destroy the internet as we know it.

After a vote near the end of 2017 to repeal this important protection, the repeal finally took effect in June 2018, about a year ago. That's when the disaster of corporate control of the internet, access limitations, slower speeds and no more innovation started. Articles were written in the NY Times and elsewhere spelling out what would happen: "repealing these onerous rules “would be the final pillow in (the internet’s) face” (The New York Times), would cause “erosion of the biggest free-speech platform the world has ever known” (ACLU), and would be “end of the internet as we know it” (CNN)."

It's now about a year into the disaster. I started looking into just how bad it's been and was struck by how hard it is to find the news stories about all the sites that have been blocked, all the students relegated to slow lanes and prevented from watching their favorite video streams, and all the price hikes. Have the giant corporations really been so scarily effective at suppressing the consequences of their self-serving actions?

I dug in and found this:

Last year, average internet download speeds shot up almost 36%, and upload speeds climbed 22%, according to internet speed-test company Ookla in its latest U.S. broadband report.

There are more users than ever. More videos to watch. More content to consume. More commerce being conducted. No sites are being blocked. No one is complaining that their service is being throttled. And more people are gaining access to broadband.

During President Trump’s first year in office, in fact, the number of people without access to a broadband connection dropped by 18%.

Meanwhile, the 5G era approaches, which will increase speeds — for mobile and fixed broadband — exponentially while injecting more competition among providers.  Elon Musk’s SpaceX has started launching mini-satellites for his Starlink initiative, which will provide broadband internet to subscribers wherever they are on the planet. 

Oops. Maybe the disaster isn't a disaster at all. Maybe it's even a good thing, hard as that may be to understand. Here's what I wrote about it when repealing net neutrality was being discussed, and here's the long post I wrote on the subject when the campaign to institute net neutrality was underway.

p.s. That earlier post on net neutrality touched a few nerves. A few programmers at companies I worked with got all hot and bothered, and refused to interact with someone so completely unethical that they could harbor such awful thoughts. Well, now we know how it all turned out, and what the repeal-net-neutrality disaster amounted to...

Adventures with Health System Software: Customer Feedback

If you want a cheap laugh, go to the Mount Sinai medical system website and hope they ask you to complete an opinion survey. It’s stupid and ridiculous, deserving lots of snark. But try not to think about what it means or the underlying reality, or you might get kinda depressed. Like I did, because I went to the website because I needed to get something done! I needed a phone number. Sounds simple, right? Until you understand I had already talked with someone at Mount Sinai, and that person gave me the wrong number. But I really needed the number -- I needed to make an appointment for a medical test that is crucial to my health. After a great deal of searching, I finally found what appeared to be the right number. Except it wasn't, as I found when I called.

This was part of my epic struggle to schedule an appointment -- something that I do with a couple clicks for my favorite restaurants, my cat at the animal hospital, or ... yes, my primary care provider. But at that powerhouse medical institution, Mount Sinai? Only the best people who really, really, really want an appointment are graciously granted one. See this for the story.

In this post, I'll confine myself to glancing at the carefully constructed Mount Sinai website and the extraordinary steps they are taking to assure that it is the best it can be. It's clear they're in a race for the top with the health insurance companies on this subject, see this.

Major companies that build websites have a problem, a problem they share with lots of companies that build software. The executives in charge are required to say that they care about quality, and do everything in their power to track and improve it, along with important metrics involving customer satisfaction. They take concrete steps to measure quality, using the best firms out there to help them.

There's just a little problem: they can't get it done.

The Mount Sinai website

I recently encountered a typical example of hopeless executive incompetence while trying to get a simple phone number to schedule a visit to Mount Sinai Hospital in NYC – scheduling that any institution whose software had successfully made the wrenching transition to the 2,000’s would have made long ago. I tell the story of the scheduling adventure here.

It was a long slog to get my MRI appointment made, including a number of calls and emails. You might think that when a window popped up near the end of my ultimately unsuccessful trek through the Mount Sinai website to extract a simple phone number that I would ignore it. After all, the website is a carefully-crafted, attractive-looking piece of useless fluff, impressive perhaps to the important people who are shown images of it in a Powerpoint presentation during some meeting, but in fact annoying, error-filled and generally useless to real people. Silly me: here I am thinking that the hoi-polloi, the real people who have health issues, are the relevant people here – when in reality, it’s the executives, jockeying for ever-growing power, prestige and money among themselves.

If you’ve read any of my other posts on software quality, you may suspect that I’m a glutton for punishment. Your suspicions are correct. So I agreed to take the survey. When I left the site, I expected the survey to pop up, but it didn’t. After all, the request told me, in no uncertain terms, “it will pop up when you leave the site.” OK, I thought, your loss, not mine. But darn! The survey I recently got from my health insurance company was so juicy!! I would have loved to see who wins the race for most dysfunctional survey between a major provider and a major payer!

It turns out, I just needed to wait. Before long, the survey arrived in an email:

I was a bit surprised to get the request in this way, but OK, they’ve obviously got all my information, so fine. As usual, I hover over the link to make sure it’s legit. The URL was portal.gsight.net with some codes after. I quickly discovered this was a domain owned by the company that sent me the email, Greystone.net. Hmmm, who are they?

Wow, a whole company devoted to healthcare marketing and the internet! They must be really good! I wonder if they know about the web?

It appears they know about the Web. And look at this:

It’s a whole process to make the website great! Smart folks, those people at Mount Sinai, turning to professional specialists to figure out how well their website is serving their customers! Though I can only assume that Greystone has only recently been engaged, since the Mount Sinai website is, after all, a pretty-looking pile of stinking crap…

So let’s dig into this expert opinion survey. Click. Here’s where I land:

OMG!!! My jaw has hit the floor so heavily, I’ll probably be scarred for life. I wonder if I can sue Greystone to cover the costs of plastic surgery for my deformed, floor-mangled jaw??

Why is my jaw hurting? Because the link these consummate professionals sent me was to a completely generic landing page! There is this thing known as “deep linking,” in which the custom URL you click brings you right to the place in question. It’s widely used. The landing page knows who you are and why you’re there. I guess the folks at Greystone hired a bunch of interns for this project, ones who hadn’t gotten to that chapter in the “Internet Linking for Dummies” book. And no one with the slightest bit of experience, like the average internet user, had tried it out.

After I gave the right answer, I was thrown into a completely generic survey about the website, utterly uninformed about who I was or any smidgen of knowledge about my site visit – putting the lie to the user tracking they supposedly do. Had they done elementary user tracking, they would have known who I was and which pages of the site I had visited. But no, they decided to ask completely generic questions.

Is this hard to do? Nope. For example, my bank, USAA, notices when I go to the “wire transfer” section of their website and then call them. A recorded voice says something like “I see you’ve recently visited the money transfer section of the USAA website; would you like to wire money today, David?” If I answer “yes,” they transfer me to the relevant department. Not hard! Maybe the Greystone.net interns will eventually get to that chapter.

The survey itself was endless, irrelevant awfulness.

Here’s an example of why the survey was awful:

If they had tracked me and made the survey specific, they would have known that I hadn’t filled out a form. Instead, they present me with a question about form-filling, and then require an answer. Most of the questions were like this. By the way, this question was about ... the twentieth question -- all of them response is required. See the progress bar that says 25%? At some point, it jumped to that, and then, question after question, it didn't changed.

Then at the end, I was invited to give some input. Which I did in calm language, mentioning that it might be nice if the phone numbers on the site were, you know, correct numbers. Of course, since they hadn't deep-linked, they had no way of contacting me to get further information.

Just to be sure I wasn't completely nuts, I went onto the Mount Sinai website again. I got lucky -- an invitation to complete an opinion survey popped up again. I carefully chose "take it after leaving the site," and this time it worked, though in a remarkably clunky way, indicating that whoever built the code had flunked Javascript 1.01. So I get the survey, and was blown away by seeing the very same "how did you get here" question I got from the email link. Any competent web programmer could know how I got there, by looking at exactly how the original URL was invoked. Clearly, performing this elementary task was beyond the collective genius of Greystone and Mount Sinai.

Then as I went farther into the brain-dead survey, I discovered that it just didn't work. Look at this:

Look at the percent completion bar just below the black line under the Mt Sinai logo -- it's still at zero, even though I'm many questions into it, as you can see by the scroll bar on the right. Programming and QA 1.01. Fail. And of course, it was a survey designed so that no normal person would march all the way through to completion.

Discussion

There’s a concept in math and computing, and also in real life, called “recursion,” or sometimes self-reference. It’s a simple concept; it’s been around for literally thousands of years, as we know from fun statements ancient Greeks made involving lying Cretans. In this case, it applies to the question of the quality checkers: who checks the quality of the quality checkers?

The answer is evidently “no one.” The most basic principles in surveys, common sense but also proven by experience, are “keep it short” and “Make every question matter.” We know these are the relevant principles because everyone who hasn’t failed the “survey 1.01” course knows that the most important metric to measure is drop-out rate. Of the people you invite, how many accept? Of those who accept, how far in the survey do they get before dropping out? What’s the completion rate? Any tracking along these lines would have shown minuscule completion rates. I’d love to have a recording of the executive meeting at Mount Sinai in which the survey results were presented, to see whether the issue was even raised.

But beyond that, let me ask: when was the last time you got a survey from Google? Or Amazon? Never, right? Another thing: have you read even a little bit about opinion polling, about how it's long-since been proven that people give one answer when asked, but then act differently? What people who are moderately educated web professionals know is that surveys are useless! That's why folks who know a little about websites watch what you do! If there's a lot of information on the site, they make it search-based, with lots of suggestions. They look for drop-outs.

Yes, I've made fun of how badly the survey was constructed and executed. It was the electronic equivalent of a paper survey from 50 years ago. Which makes sense, because the Mount Sinai website is the electronic equivalent of  a glossy brochure from 50 years ago. That's the killer observation. Mount Sinai could make a huge advance by leaping forward to the state of the art of roughly 20 years ago. The very fact that they're using obsolete technology like surveys -- and on top of it doing it incompetently -- shows that they are clueless. It's the equivalent of using a steam-powered car instead of an oil-powered one, and being unable to run the steam-powered car competently. The right response here isn't build a better survey -- it's use modern customer feedback techniques.

Conclusion

Well, it’s a wash. The hospital system opinion survey was pretty different from the health insurance one, but they each exemplified unique ways of being bad. I wonder how many dimensions of badness there are? The institutions I’ve had the pleasure of experiencing are clearly on the leaderboard of those most likely to get to the maximum. Neither of them has a clue about decades-old methods that are vastly superior for getting customer feedback than surveys, however well-constructed those surveys might be.

Postscript

Learning about the excellent survey work conducted by Greystone.net on behalf of Mount Sinai had an added dimension of amusement for me because I grew up near an institution named Greystone. Or more formally, Greystone Park Psychiatric Hospital.

It was, as my mother the R.N. called it when I was growing up, a ‘Looney bin.” If someone said something she thought was dumb, she would say “Did you just escape from Greystone?”

Recurring Software Fashion Nightmares

Computer software is plagued by nightmares. The nightmares vary.

Sometimes they are fundamentally sound ideas that are pursued in the wrong way, in the wrong circumstances or at the wrong time. Therefore they fail – but usually come back, sometimes with a different name or emphasis.

Sometimes they’re just plain bad ideas, but sound good and are cleverly promoted, and sound like they may be relevant to solving widely acknowledged problems. Except they just don’t work. Sometimes these fundamentally bad ideas resurface, sometimes with a new name.

Sometimes they’re a good idea for an extremely narrow problem that is wildly applied beyond its area of applicability.

The worst nightmares are the ones that slowly evolve, take hold, and become widely accepted as part of what “good programmers” believe and do. Some of these even become part of what is ludicrously called “computer science.” Foundation stones such as these, accepted and taught without proof, evidence or even serious analysis, make it clear to any objective observer that computer “science” is anything but scientific, and that computer “engineering” is a joke. If the bridges and buildings designed by mechanical engineers collapsed with anything close to the frequency that software systems malfunction, the bums and frauds in charge would have long since been thrown out. But since bad software that breaks is so widespread, and since after all it “merely” causes endless delays and trouble, but doesn’t dump cars and trucks into the river, people just accept it as how things are. Sad.

Following is a brief review of sample nightmares in each of these categories. Some of what I describe as nightmares are fervently believed by many people. Some have staked their professional lives on them. I doubt any of the true adherents will be moved by my descriptions – why should they be? It was never about evidence or proof for the faithful to start with, so why should things change now?

Sound Ideas – but not here and now

• Machine learning, AI, OR methods, most analytics.
  • These things are wonderful. I love them. When properly applied in the right way by competent people against the right problem at the right time, amazing things can be accomplished. But those simple-sounding conditions are rarely met, and as a result, most efforts to apply these amazing techniques fail. See this.
    

Bad ideas cleverly promoted

• Microservices.
  • Microservices are currently what modern, transformative CTO’s shove down the throats of their innocent organizations, promising great things, including wonderful productivity and an end to that horror of horrors, the “monolithic code base.” While rarely admitted, this is little but a re-incarnation of the joke of Extreme Programming from a couple decades ago, with slightly modified rhetoric. A wide variety of virtues are supposed to come from micro-services, above all scalability and productivity, but it’s all little but arm-waving. If software valued evidence even a little, micro-services would be widely accepted as the bad joke it is.
• Big Data
  • There's nothing wrong in principle with collecting data and analyzing it. But in practice, the whole big data effort is little but an attempt to apply inapplicable methods to random collections of data, hoping to achieving generic but unspecified benefits. See here.
    

Great ideas for a narrow problem

• Blockchain.
  • My favorite example of an excellent solution to a problem that has been applied way beyond its area of legitimate application is blockchain. Bitcoin was a brilliant solution to the problem of having a currency that works with no one in charge. How do you get people to “guard the vault” and not turn into crooks? The way the mining is designed with its incentives and distributed consensus scheme brilliantly solves the problem. However, the second you start having loads of crypto-currencies, things get weaker. Once you introduce “smart contracts,” there’s a fly in the ointment. Take away the currency and make it blockchain and you’ve got a real disaster. Then “improve” it and make it private and you’ve got yourself a full-scale contradiction in terms that is spectacularly useless. See here.

Bad ideas that are part of the Computer Science and Engineering canon

• Object-orientation.
  • Languages can have varying ways of expressed data structures and relating to them. A particular variant on this issue called “object-orientation” emerged decades ago. After enjoying a heyday of evangelism during the first internet bubble, O-O languages are taught nearly universally in academia to the exclusion of all others, and adherence to the O-O faith is widely taken to be a sign of how serious a CS person you are. For some strange reason, no one seems to mention the abject failure of O-O databases. And no one talks about serious opposing views, such as that of Linus Torvalds, the creator/leader of the linux open source movement, which powers 90% of the world's web servers. Programmers who value results have long since moved on, but academia and industry as a whole remains committed to the false god of O-O-ism.
• Most project management methods.
  • I’ve written a great deal about this, including a book. Whether it’s the modern fashion of Agile with its various add-ons such as SCRUM, project management methods, ripped from other disciplines and applied mindlessly to software programming, have been an unmitigated disaster. The response? It ranges from “you’re not doing it right,” to “you need to use cool new method X.” Yet, project management is a profession, with certifications and course on it taught in otherwise respectable schools. A true nightmare. See these.
    
• Most software QA methods.
  • All software professionals accept what they’ve been taught, which is that some kind of software automation is an essential part of building software that works and keeping it working. Except the methods are horrible, wasteful, full of holes and rarely make things much better. See this.
    

Conclusion

I make no claim that the list of items I’ve discussed here is comprehensive. There are things I could have included that I have left out. But this is a start. Furthermore, if just half the items I’ve listed were tossed and replaced with things that actually, you know, worked, much better software would be produced more quickly than it is today. I don’t call for perfection – but some progress would sure be nice!

Blockchain has been Unchained and Unblocked and it's Broken

Blockchain promoters and enthusiasts continue to blithely stroll along the yellow blockchain road to the golden city where immutable distributed ledgers make decades-long technology problems fade away, like the wicked witch. None them publicly acknowledges or seems to notice the hurricanes and earthquakes that are increasing in frequency and intensity.

In total, hackers have stolen nearly $2 billion worth of cryptocurrency since the beginning of 2017, mostly from exchanges, and that’s just what has been revealed publicly.

That’s no big deal, I guess. I describe these little security problems here and here.

Someone who’s technically sophisticated could argue, following the logic I described here, that the security problem wasn’t in Blockchain itself. The problem was in wallets and exchanges, which are software that sits “on top of” blockchain, making it easier to use. It’s the same kind of security breach that can happen with any software, and has little to do with the inherent security of the software itself, but is mostly due to the layers of software built on top. This is true! One does wonder why Blockchain is so wonderful, then, if in practical use, its supposed greater security is so easily circumvented.

Is it really more secure than a regular DBMS, putting aside all those flaky higher layers of software? That’s what everyone involved declares. The most open and honest of the Blockchain-ista’s will grudgingly admit that a nearly impossible 51% attack could cause a bit of a problem with the heart of the system, the keep of the blockchain castle.

Sadly, the nearly impossible attack has happened. And not with some obscure little crypto-currency no one has ever heard of, but with Ethereum Classic, one of the premier systems, and the home of that transformative invention, the Smart Contract.

An attacker had somehow gained control of more than half of the network’s computing power and was using it to rewrite the transaction history. That made it possible to spend the same cryptocurrency more than once—known as “double spends.” The attacker was spotted pulling this off to the tune of $1.1 million.

To anyone with a shred of common sense, this is a fatal event. It demonstrates that Blockchain’s security has a fatal flaw, even when running in its optimal environment, with public miners.

The big companies promoting private blockchains, should they deign to pay attention, will immediately come back with strong statements about how that kind of attack could only take place in a public blockchain, and couldn’t possible happen with a highly secure, controlled environment they provide with their private blockchain. Sure. That’s like saying that those guys who stole lots of money in the open, in a big public space where everyone could see them, couldn’t possibly get into the single secret room and rob the bank vault in complete privacy. Security in closed computer system managed by big companies who follow all the security regulations and pass audits is abysmal. Ever hear of Edward Snowden? Chelsea Manning? Others? Check out the facts a bit, and then come back to me and explain how it is that the unbroken stream of security breaches of the best systems run by the best military and corporate bureaucracies is going to suddenly stop when the software at the core is Blockchain.

The sad fact is, libraries are more secure than computer systems. Including when Blockchain is involved.

This post first appeared at Forbes.

What’s Wrong with Medical Scheduling and Why it Matters

It’s easier by far to make a reservation at a restaurant than for a medical test. I guess having a meal at a restaurant is far more important than getting an MRI, given that the restaurant people have made the process simple and convenient and effective for all concerned. Getting a medical test must be rare and unimportant, since no one has bothered to make it work well. Sure. I guess that lie is more comfortable than the other possibility, which is that medical system administrators and software providers are too incompetent, lazy or unmotivated  to make things moderately convenient and up-to-date for their employees and customers.

Why this is an important issue

If you’ve gotten this far, you’re already an unusual reader. The vast majority of the leaders, innovators, experts and generally high-prestige people in healthcare would have tuned out and moved on as soon as they saw “medical scheduling.” Their guts tell them “medical scheduling isn’t important.” In terms of career growth and industry prestige, their guts are serving them well.  However, in terms of making a real difference that will positively impact most people involved in medicine, their guts are misleading them.

Medicine has a real prestige problem. I’ve described the medical innovation spectrum, in which the exotic, future-oriented end, like AI and Blockchain, gets most of the money, conferences, attention and career-advancing opportunities. The more you move on the spectrum to simpler, proven, non-exotic things that can make a difference here and now for lots of people, the more you’re moving to the back office or basement, where poorly-paid, invisible people while away their time and whine to no meaningful audience. Here are details and examples.

Scheduling is one of those proven winners that remains largely unimplemented in major medical organizations. The fact that scattered medical groups have implemented it beautifully shows there are no technical barriers.

I can hear it now. Scheduling. Sure. What award am I going to win by implementing something restaurants do? I’m breaking new ground in personalized medicine while I’m curing cancer on the side! Away with your trivial scheduling talk!

I get it. There’s just a little problem. As I’ve detailed here and here and here and here and more, time and money loss and serious medical issues are caused by nuts-and-bolts problems in the medical system. If just some of them were improved, the funding for your precious futuristic projects could be increased! And, by the way, loads of patients would be better off on multiple dimensions, including not dying prematurely! You know, little things.

As an example of the impact, consider just this aspect of scheduling: automated follow-up (yes, it’s part of scheduling). Failure of follow-up problems:

“The impact on patient outcomes included missed cancer diagnoses.” Journal of General Internal Medicine.

“In fact, almost a quarter of all medical errors occurring in outpatient settings can be attributed to poor follow-up of abnormal test results and are believed to represent 25% of malpractice lawsuits involving failures or delays in diagnosis.” AACC

The Scheduling problem

There are an amazing number of dedicated, skilled, hard-working professionals in medicine. There are patients who have health issues who are often grateful for the service and care provided by those professionals. But both groups, providers and patients, are burdened by ancient, dysfunctional and incredibly expensive processes and computer systems that make things that should be quick and easy into something that is cumbersome and time-consuming, often yielding poor results. Everyone involved feels trapped and needlessly harassed. What’s going on here?

This is the horrible general scheduling problem in health care. To illustrate the issues in a concrete way, I’ll focus on scheduling a test as a follow-on to a procedure, and use my own experience as an example – an example that, sadly, is business-as-usual in this world.

A Cat Scheduling Example

Before getting into scheduling a medical test for me, let’s see what happened when I had to schedule a test for someone more important than I am – at least she seems to think she is – my cat, Priss. The comparison between getting tests for Priss and me will be instructive.

I have a cat, Priss. Priss is pretty chill; here she is thinking deep thoughts:

I take Priss to a local animal hospital, where she is well cared for. I get regular notes and emails telling me when something medical needs to be done for Priss. For example, here’s the header of an email I received last year:

Note that the email arrived near the end of August, 2018.

Here’s the top part of the mail:

Note that there’s a picture of a cat in the email – they know who they’re dealing with! They know Priss’s name, and they know she’s a cat, and not one of those friendly but face-licking, slobbering dogs who are incapable of using a civilized thing like a litter box.

Here’s the rest of the email:

They make it hard to miss the phone number or the online scheduling system they support to make an appointment, don’t they? They also spell out exactly what needs to be done. And darn, the email was sent EXACTLY one month, to the day, before the service was due. Please note: this is THEM taking the initiative to follow up, TELLING ME when something needs to be scheduled.

If this were a human hospital we were talking about, I would say “to make a long story short …” but because this is an efficient, modern animal hospital, I get to say “to tell it like it is …” all I had to do was click on the email, buzz through their easy-to-use on-line scheduling system, make an appointment that worked for them and for Priss, get a confirmation email back right away, and I was done. That’s it!

On top of it all, the people there are great, and care for Priss really well. There was a problem with her fecal test -- results back in a single day -- and the treatment was immediate and effective, no more than a day's delay for any step.

An MRI Scheduling Example

Now we turn to hospital scheduling for humans, specifically for me at the world-class hospital system, Mount Sinai. I’ve had some medical issues that have required MRI testing. See this for a description of the issue, with details about the testing experience and the conclusions that can reasonably be drawn from it.

I had  30 days of radiation therapy in early 2018. I was supposed to have MRI’s to see the results of the radiation six months and a year after the treatment. This is standard practice. I know from experience that the grandees at Mount Sinai Medical System haven’t gotten around to sending a team to the Hudson Animal Hospital to learn from their cat scheduling system, so they can put a multi-year project plan in place to implement something similar for their human patients. In a human medical world that worked as well as my local little animal hospital, I would have gotten a reminder a month before my MRI should have been taken.

Being the patient patient that I am, I waited until after the anniversary itself. Then I sent a friendly reminder to my doctor:

He replied the next morning with a perfectly reasonable response – given that he had to do the work that a vet wouldn’t have had to do.

His prompt reply was Feb 4. I heard nothing for weeks, so I finally reached out again three weeks later:

Again, the doctor in charge of the radiation center, doing clerical work well and promptly, work that he shouldn't have had to do, responded right away:

What happened next? More fun wasting everyone’s time. Remember, with a reasonable on-line system like restaurants and vets have, none of this would be needed!

A couple days later I got a voice mail, with a friendly person giving me the date and time for which my MRI has been scheduled – of course, without consulting me! I guess I must be sitting around suffering, anxiously waiting for the first possible moment at which I can get my MRI, at which point I’ll drop everything and arrive two hours early. Or not. Even better, this call isn’t from the MRI center, it’s from my doctor’s office, where someone has made an appointment for me, a couple days after the most recent request, weeks after the original one. So I return the call, explain I can’t make the appointed time, and is there any way I could talk with the MRI center and make an appointment myself? Well, she tries to be accommodating, and says it’s OK, but I also need to make a follow-up appointment with the doctor by calling her. I think I can manage this.

I call the number she gives me, which was the wrong number. I consult the web, and it takes a typically long time to find the center and its phone number for scheduling on the website. Mount Sinai management, it goes without saying, is totally on top of customer feedback and quality management. So they pop up an opinion survey. What happened with the survey is a lesson all by itself in the profound dysfunction of our medical systems in general, and Mount Sinai in particular. I will treat this event in a separate post.

I call what the website says is the right number, and keeping sarcasm – however warranted it may be – to a minimum here, after a journey through automated VRU’s and other wrong numbers, I eventually get to the person who can schedule me. Sure enough, she finds the order for the MRI in her system, and makes an appointment that actually works for me. Phew! I then, as instructed, call back the main doctor’s office and schedule a visit with him to go over the results, as I had been instructed to do. Please note this visit with the doctor I scheduled; it will start its own little trail of incompetence and waste.

After way too much effort on everyone’s part, I thought things were finally set. Silly me. I got this call a few days before my MRI appointment:

After a couple of attempts, I get through. I’m told that the room where the MRI machine is suddenly needs to have major work done, work that apparently couldn’t be scheduled in advance, making the machine itself unavailable. So pick another time. We work something out. I then call the doctor’s office and re-schedule my visit with him, so that it’s suitably after the new date of the test.

On the morning of the date of my original doctor visit, the one I called and re-scheduled, I get a voicemail saying in effect, “you’re coming into today as scheduled, right?” I admit that the nasty thought of ignoring the call crossed my mind, but my better self took control. I called and got through after a couple tries and explained the re-scheduling. The person verified that the new appointment had in fact been made, but that “someone else” must have failed to cancel the old one. Done. Except that it wasn't really "done." See the associated customer service post to see what happened!

A week later something came up and I had to move the appointment with the doctor. I called. After explaining what I wanted, the person said, didn’t you just miss a scheduled appointment with the doctor? Nope. Apparently the last person I talked with, whom I told about their mistake, failed to correct it. Again.

Do you use a calendaring/scheduling system, for example the one in Microsoft Outlook? Have you noticed that you can click on an event and change the date/time, so that it moves to the new time slot? Of course, you can delete the old one and make a new one from scratch if you really want to, but why would you? The evidence seems to point to the possibility that such a feature, which is standard on modern calendar systems, doesn’t exist in the paragon of modern software used by the leading medical system, Mount Sinai.

Conclusion

Follow-up of events, both one-time and multiple recurring, is a standard feature of modern scheduling systems. Self-scheduling by software is a widespread feature. It’s widely used, and benefits everyone involved. It’s not as though self-scheduling by software is particularly difficult for medicine. For example, the primary care office I use, OneMedical, has a convenient system that takes account of the length of visit you need, and gives you choices of providers and locations so you can pick what works best for you.  Once scheduling is on-line, it is amenable to serious optimization techniques, which have been deployed with great success resulting in substantial savings and efficiencies in things like infusion centers and operating rooms. This is not possible with the primitive human-phone-based systems in widespread use.

It’s painfully obvious that the important people at medical centers prefer to spend time doing “important” things at the fancy future end of the innovation spectrum, rather than lowering themselves to implement practical, here-and-now improvements that benefit everyone. When will this change?

 

Software is a Pre-Scientific Discipline

For a wide variety of human-understandable reasons, software is perceived as a science. In academia, it's taught in the Computer Science department, often part of the math department. What could be more precise and scientific than that?

Whatever its pretensions, software is anything but scientific. It's mostly driven by fashions and fads, led by "experts" who promote theories that sound good when described -- but which entirely lack any form of scientific process, testing or evidence.

Software diseases will continue to severely hamper our computer systems until we wake from our long, pre-scientific sleep. We will know we're making progress when software practices at least to the level medicine had achieved 150 years ago. See these examples: the history of scurvy; the history of bloodletting; Yahoo and Hadoop.

The Evolution of Science

Science is not one thing, although the core principles of hypotheses, testing and evidence are always the same. Sciences don't pop up full-grown from nothing. Each field of human endeavor evolves towards being a science (or not) at its own time and pace. There is typically a long gestation period during which resistance is deep and widespread. Physics is a classic example.

Even when an area of science is well-established, the temptation to simply declare and assert the truth of something without careful proof remains strong and happens all too often. Science is something that human beings do, so it's never "perfect." It's also never "done." The resistance of the entire physics establishment to the now-accepted fact that time changes as the speed of an object approaches the speed of light, and that light has no mass but nonetheless exists and travels at a measurable rate, are classic examples of the fits-and-starts evolution of even the well-established science of physics.

It was a long, hard slog for medicine to emerge from its pre-scientific state. While there is loads of room for improvement, as I have often pointed out in this blog, medicine has clearly and explicitly embraced scientific discipline in the large majority of its practices, of course with the occasional embarrassing slippage.

The non-evolution of software towards science

Let's compare the emergence of powered flight as a science to the methods for building software projects. Here are the key points:

• Powered flight was widely recognized as important more than 100 years ago. There were widely accepted experts, and the entire establishment gave them money and support. After a couple spectacular failures by the experts, the obscure people who actually figured out how to create a heavier-than-air flying machine got it done, and their methods were soon universally followed. Here is the story in more detail.
• Building effective software is widely recognized as important today. There are widely accepted experts, whose methods are taught in schools, practiced in all major institutions and mandated by government regulations. After spectacular, repeated failures, everyone says "oh, that's software, what can you do," and moves on. Meanwhile, sometimes obscure people build amazing new software quickly and well most often using unauthorized methods. Their software is widely used and their companies acquired by the big organizations who can't build anything. Nothing changes. Here is an example and details.
  

To take another example, while there are many ways that the science of medical drug development could be improved, there is little doubt that it is a scientific venture. In terms of science, in spite of its problems, limitations and inefficiencies, drug development is probably a hundred years ahead of "computer science" in general and software development in particular. See this for a comparison.

If software were a science

Think of a list of established principles in software -- if software were in fact a science, the things that would be like the basic equations of non-relativistic motion. What's on the list? I suspect it includes things like: Object-oriented programming, comprehensive test automation, architecting for scalability.

Now think of a list of hot new methods or techniques in software, things that are widely accepted but early in widespread adoption. The list may include, depending on the circles in which you travel, things like micro-services, the Clojure language, Agile methodology with SCRUM, test-driven development.

Which of the items on either list -- your version of the lists, not mine -- went through anything like this process:

There was a bad problem that accepted methods weren't solving. People hypothesized an underlying cause and/or a cure, tested it first on a small scale and then on a larger scale. The evidence was overwhelming in A:B comparison that the cure was effective, so it became accepted.

Or,

There were observations that didn't fit existing theories, data that wasn't explained, or discrepancies that couldn't be accounted for. Someone came up with a theory that made sense out of the rogue data. Others formulated the theory exactly and conducted careful experiments; the results were made public, and maybe there was a period of refinement. Finally, the new theory was accepted, because it was experimentally proven to account for all the measured data, something the old theory could not do.

Anyone with reasonable software experience knows the answer to these simple questions: software doesn't work that way! Not even a little bit! Instead, new practices are invented, promoted and sometimes accepted into common practice. In no case is there a scientific vetting process! People just accept the theory because

• it makes sense to them, or
• it's what they've been taught, or
• it's required by the mandated practice of the group in which they work
• it somehow advances their career or enhances their prestige

Sometimes, happily, software fads just fade away for as little reason as they started. A fairly recent example is pair programming, which I describe and examine here.

In the face of this evidence you may swallow hard and admit that software may not be a science, but it is an established discipline with standards and processes that are widely accepted, as for example you can see in the FDA software regulations. Sadly, that makes it even worse, if it's possible to imagine that. The standards and processes that constitute modern software practice are taken from other fields and jammed onto software. They don't fit and they don't work.

Conclusion

No testing. No hypothesis with controlled experiment. No evidence. No process that resembles either medicine (we know there's a problem, we have a possible solution, let's prove it works before using it widely) or physics (we have this data we can't explain, let's propose a theory that accounts for it and run experiments that will prove it right or wrong) or anything else.

You can say that building software is part of "computer science" until you're blue in the face. You can require CS degrees for your new hires. But the evidence is that software is, without question, pre-scientific.

We need to at least start building towards a true Science of Software.

 

Due Diligence: Financial, Legal and Software

When you are going to acquire or make a major investment in a company, it makes sense to perform due diligence. Not only does it make sense, it's a fiduciary responsibility! Due diligence on the company's financials are a no-brainer. Same thing with legal. Wouldn't want to be surprised by a landmine in one of the contracts, would you? If there is software involved, most people perform technical due diligence. In each case, you hire an expert who combs through the relevant portion of the company. The expert produces a report, at minimum highlighting any deficiencies.

They're all pretty similar, right? They're abstruse subjects involving deep expertise where there can be landmines and hidden messes that can have great consequences. That's why you have due diligence.

Yup, that's the standard view, Due Diligence 1.01. It's even all reasonable ... except for Venture Capital investing in tech companies.

Due Diligence for computer and software technology

The vast majority of tech due diligence involves fairly large companies with large staffs. When you're looking to do a deal with a fairly large tech company as the target, it's business as usual. You want to make sure everything is done to industry standards. There are probably security and other regulations. Are they in compliance? Are they following industry-standard development methods, or is there some kind of willy-nilly, out-of-control stuff going on? Do they actually have a QA process that assures that new software releases won't go belly-up and hurt the company and its customers?

This sounds pretty much like legal and financial diligence. Because it is!

Tech Due Diligence for VC's

How can a VC responsibly invest in a young software-based startup without understanding the software? Most investors aren’t programmers, after all, and can’t “see” the software. No big deal! VC’s usually aren’t lawyers or accountants either, after all, and there’s a standard solution: hire an expert to perform due diligence. Hire lawyers for all the legal things, accountants to pour over the books, and technology due diligence people to handle the tech. Now that wasn’t much of a problem, was it?

There's just one little problem: the standard methods for evaluating software lead to terrible results. This shouldn’t be surprising, because the standard methods for building software lead to terrible results! If the industry’s best minds can’t figure out how to build software effectively, why would anyone think that these best minds would be any better at evaluating software and the organizations that create and maintain it?

You won’t find these industry-leading groups and thinkers wringing their hands or wailing lamentations at their on-going failure – that would be bad for business! So instead, awfulness is accepted as just how things are. So when experts evaluate software groups, the typical process is to compare them to how the vast majority of groups “get things done” – kinda, sorta, eventually, with failures and massive security breaches attributed to some combination of life and the existence of evil criminal masterminds. If the group fails to sing the standard songs and say how wonderful the software dances that are currently in fashion make them feel, then questions are raised.

Large corporate and government organizations know and accept all this. They take comfort in striving to attain industry-standard “best practices,” and give presentations about how great their direction is. They may put signs up on walls, or hang banners making vague but strong assertions about “innovation” stuff involving the future.

What if you’re a small organization, a start-up or not much more advanced? For the VC point of view, this makes little difference. They tend to have due diligence providers with solid methods and a track record the partners of the VC firm are comfortable with. This fact of existence leads to a few major behaviors among entrepreneurs:

• Avoid any VC that wastes your time putting you through what you know to be irrelevant evaluations
  
• Hold your nose and go through with it, while trying to gull the evaluating firm into thinking you're more industry-conforming than you are, and to the extent that you're not, you're planning to fix it all up real soon
• Really change things so that you match industry-standard evaluations, often by hiring a tech leader who's "been there done that."

Many VC's pride themselves on their judgment of character, and won't take the tech due diligence really seriously anyway -- experience has taught them that it's mostly a pile of irrelevant gobbledy-gook anyway. They place their bets on the company leaders, whether they have the vision, drive and smarts to create a winning company, or at least to achieve lift-off.

This is a real problem, but it logically reflects a larger problem in the tech industry, one that I've tried to articulate in a variety of books and blog posts: there is a huge gap between how the vast majority of software organizations build software and how the most effective small ones build it, not unlike the difference between army draftees just out of basic training and a team of elite Rangers. They both fight, but if made to fight each other, it wouldn't be a "fair" fight. Same thing with software, only even more so.

The vast armies of software engineers, armed with their impotent and irrelevant Computer Science degrees, resemble draftees out of basic training more than anyone would like to admit. There are so few software Rangers that most of the plodders rarely encounter one. Rangers certainly don't hang out by the water cooler hoping to chat with one of the sloggers -- a real Ranger wouldn't want to enter the building, much less be employed there.

The very best tech due diligence for VC's attempts to differentiate between these cases, which aren't totally distinct:

1. The company develops software following industry standards.  If what they're doing is amazing and pretty much already baked that could be OK.
2. The company isn't following standard norms or any other norms. It's just disorganized. This is a problem. There's a risk in whether it can be fixed. Know what you're dealing with is essential.
3. The group is not following industry norms, but is achieving rapid progress with good quality using non-standard methods -- not anything-goes wild-west, but non-standard techniques that are actually methods. There is a wide range here, which must be judged.

The trouble of course is that standard, off-the-shelf tech due diligence conflates cases 2 and 3 above. Truly effective tech diligence for VC looks for case 3, and judges to what extent the company achieves it.

Conclusion

Performing effective tech diligence is pretty similar to financial and legal diligence -- unless you're investing in a break-the-rules, innovative startup or young company and you're a VC. In that case, you're best off judging them by different standards than you would normally use. In finance and legal, you're looking to find things that are unusual and diverge from norms -- because they could be bad and hurt you. In tech, you're looking for things that strongly differ from standards and norms -- specifically methods that trump those standards and norms. You don't want a group that fails to live up to standard army discipline. You want a group that chooses to do things differently -- because they do them better, WAY better. That's what makes winners, and that's what VC's want to invest in.

An Immutable Distributed Ledger is now in Large-Scale Production!

First there was Bitcoin, friend of criminals, speculators and tech geeks everywhere. It’s grown amazingly. Then there were alternatives to Bitcoin, often sharing much of the same code, but with different and incompatible tokens. One of those Bitcoin alternatives, Ethereum, introduced the concept of Smart Contracts, which I discuss here. Now, increasing attention is being paid to “blockchain,” said to be the foundation on which crypto-currencies like Bitcoin and Ethereum are built. Large corporations are taking up the charge, places like IBM and Microsoft, and leaders in various industries have projects going to prove out the technology. While the terminology isn’t uniform, it’s easy to see that earlier terms with unsavory associations are being abandoned in favor of more generic terminology, names like blockchain,  “Immutable Distributed Ledger” technology, or just “distributed ledger.”

Once you start talking about the technology in generic terms, what are the chances of this actually working? At scale? In practical reality? Lots of people in the community of blockchain enthusiasts have expressed concern about this. Legitimate concern. The question naturally arises, but appears not to have been asked, has something like this been built before? Something that could legitimately be called an immutable distributed ledger?

The answer is a simple “yes.”

This amazing system, which is one of several in production today, has over 2 billion accounts and over 40 million participating agents. It moves over $10 trillion per year, processing over 150 million transactions a day, and can handle over 50,000 transaction messages per second.

Let’s dive into the technology a bit:

• It’s immutable. Once a transaction gets in the system, it can’t be altered or removed. Despite the volumes mentioned above, spread over more than 200 countries, there are no instances of processed transactions being altered.
• It’s distributed. Computers all over the world are involved. It doesn’t matter where you are: this system enables you to send or receive currency. Even better, currency conversions are built into the system! The distribution has been built in part to enable reliability. While any one computer in the system can fail, the system as a whole has never gone down, and transactions have never been lost.
• It’s a ledger. It’s a gigantic ledger of currency going into and out of accounts. The ledger balances to the penny every single day.
• Consumers don’t lose their money! In spite of all the volume, When bad stuff somehow happens, consumers lose nothing. Nada.

OK, OK, I’ll stop the charade. As you’ve probably guessed, the Distributed Ledger technology I’m describing here is in fact VISA. I’m playing this game because reading about how enthusiasts talk about blockchain, I wonder how many of them know about credit card internals? If they did, they would see that all the goals they have for blockchain have already been achieved in credit cards. And more!

The key assumption at the core of the blockchain craze is that blockchain is an amazing new technology, a breakthrough that enables all sorts of long-intractable problems to be solved. These virtues are primarily the fact that it’s immutable, it’s distributed and it’s a ledger. Sorry, guys, you don’t need blockchain to build a system that has those attributes. It’s already been built using plain old database technology and secure networking.

Yes, I played a game when describing the immutable distributed ledger technology that’s already in massive production, knowing everyone would think I was talking about blockchain. But the blockchain groupies are playing a more serious game, convincing themselves and others that blockchain is uniquely able to do things that haven’t been done before, and could never be done without this amazing new invention. Just to be clear, I’m not saying that VISA technology can be applied to the many problems to which blockchain is being applied. I’m simply saying that if you think you have a problem for which an immutable distributed ledger technology is the best solution – that problem can be solved without blockchain, more quickly and with a lot less effort. VISA is just one of many full-scale, in-production examples.

It's sad that the Blockchain mania is so powerful that no self-respecting executive can risk ignoring it. So all the big banks and even, yes, VISA itself have blockchain based projects underway, nearly all of them involving widespread use of the future tense. I am open to the possibility that some of them may even be deployed in some way, once the people implementing them get realistic and relegate the blockchain technology itself to a tiny, marginal aspect of the project's code base, and realize that everything they're doing could be done better and faster without the distraction of largely irrelevant blockchain. But meanwhile, it's great for reputational enhancement and attention-getting!

A version of this post was published at Forbes.

How We Got Chatbots for Mobile Financial Apps

Chatbots for financial and other applications aren’t just a cool new thing. They’re a necessity! They solve the worsening problem of too many options to choose from on shrinking screens, with unhelpful help screens.

Do you access your financial accounts online? If you do, perhaps you’ll remember that the first time you tried to do something, you had the fun of poring over the menu system to find what button to click, sometimes only to reach another screen full of buttons and menus. On the plus side, online financial systems let you get a lot done. On the minus side, jumping through the hoops to actually get them to do what you want can be a long slog. Have you ever gotten frustrated and tried to click on Help? And gotten the long, unhelpful Help stuff? Helped a whole lot, didn’t it?

Today, nice big screens with high resolution, sitting on a desk somewhere, are used less and less. Cute, portable little screens with phone and camera built in are used more and more, partly because it’s in your pocket, right there when you need it. What happens to all those giant screens packed with menu choices? I guess the designers could have reduced the typeface so much that you’d need a magnifying glass to read them, but they bowed to reality and put just a few menu choices on each screen. Nice to read, but the result is that your multi-screen journey to get to the one you want got even longer. Assuming you remember how to get there. And what if you want something more elaborate, like how close am I getting to hitting my budget for restaurants this month? Fuggedabout it.

The designers of mobile apps for financial applications aren’t between a rock and a hard place. It’s worse. They’re stuck way back in a long, narrow cave that floods. What to do?

Echo and Siri have been training us to just ask for things. Who won the game last night? What’s the weather forecast? Even jokes! But banks … now that’s a different matter altogether. Banks are serious things. There’s money involved. Not just any money – MY money!

So what’s a bank app creator supposed to do?

It’s actually pretty simple, because there’s not much choice. If you want the bank app to be, you know, USED by the people who have it, you have to make it USABLE. Period. There’s not enough room for menus, except maybe a couple super-popular buttons. Help files? Waste of time. You’re down to one choice: make it so you can TALK (or chat) to the app, and ask it to do stuff for you.

That’s the logic. Even better, it’s really happening. In real life!

Bank of America is advertising its chatbot Erica heavily in some parts of the country. The reason is simple: they want their customers to be able to USE the BofA app, not be frustrated by it. There’s the chatbot technology provider Kasisto (Disclosure: Oak HC/FT is an investor) used by multiple banks to power human interactions. The bar has now been raised for financial institutions with apps.

Let’s review some history. Years ago, what financial institutions had to have was a website for customers to access their accounts. As smartphones spread, the bar was raised: OK, you’ve got a website, but do you have an app? Not having an app was a reason for customers to move to a place that did. Just as most of the financial institutions were breathing sighs of relief that they’ve caught up, the bar is raised again: you mean we have to make the app USABLE? What’s a chatbot, anyway? The pattern here is clear: technology keeps marching along, some financial institution applies it to their customers’ and their own benefit, and the others scramble to catch up. What’s next?

To find out what’s next, all we have to do is look at the non-financial domains that use chatbots. For example, look at Amazon’s Echo. It started out pretty primitive, but it’s adding capabilities all the time – as Amazon puts it, new “skills.” The bar is already being raised by technology vendors in two specific, technically challenging areas:

It’s one thing to answer a simple question, like what’s my balance? But real chatting as done by people requires that the chatbot take into account complex questions, context and history.

A complex question requires doing some real “thinking.” For example, “How much did I spend on eating out last month?” has loads of complexity. The system has to know that “eating out” means spending money on the merchant category “restaurants.”  It’s got to find all those transactions that took place in the month before today’s month, add them up and give you the answer.

Taking conversation history and context in account is even trickier. Suppose you get the answer to the eating out question. Suppose you now ask “ How about the prior month?” Easy for a human; for a bot, not easy at all! The bot has to figure out that you’re still talking about eating out (restaurants) and that you want the total for a month. It’s also got to figure out that you want not last month, but the one before that. These are the kind of amazing interactions supported by Kasisto.

Chatbots for financial apps are just being rolled out to solve the otherwise unsolvable problem of screen real estate. Even though we’re still in the early roll-out stages, the next battle is clear. Most of today’s chatbots are in elementary school – soon they’ll need to graduate to middle school!

A slightly different version of this post originally appeared in Forbes.

How Can an Immutable Distributed Ledger have Assets Lost or Stolen?

As I’ve described, cryptocurrency losses started long ago and have kept on mounting over the years. Most recently, the largest cryptocurrency exchange in Canada, QuadrigaCX,  has experienced a little problem that has resulted in what appears to be a permanent loss of over $130 million for its many customers.

The losses don’t seem to make any sense. My account is stored in a ledger that’s immutable – it can’t be changed because it’s locked down by unbreakable cryptography. It can’t be lost because the ledger is distributed, so even if a few computers are lost, there are still loads of computers with a copy of my unchangeable account balances in them. So how can it be that I can lose my money? To further increase security, Bitcoin and the other crypto-currencies don’t identify me by my name, but by a really loooong string of letters and numbers. And that’s just the name – to make any changes, someone would need access to my private key. Remember, “private key” isn’t just some password that’s easy to remember and maybe someone could guess – it’s a long number that's part of a proven-unbreakable cryptography scheme, totally guaranteeing that no one but me can access my accounts? It’s impossible for anyone to break through these layers of security, that math guys everywhere calmly assert just can’t be cracked. So what are these losses about???

The only reason this is mysterious is because the vast majority of stuff we read about Bitcoin, blockchain, distributed ledger technology and the rest is produced by adherents to the new cult. They’re promoters, not evaluators. Let’s start with encryption. Loads of things are encrypted – it’s hardly something that started with Bitcoin. Most sessions with websites are encrypted, as you can see by the https in the URL bar in the browser. I hope you only use encrypted WiFi. Email is encrypted. All databases run by moderately competent professionals are encrypted. And so on. Are websites, email systems and corporate servers hacked in spite of it? You know the answer. People can wave their arms and babble with passion about the power of unbreakable encryption all day long, but bad things happen in spite of it. Is it because encryption is in fact breakable? Nope! It’s because even the biggest castle with the most unbreakable walls has to have doors that are easy to open and close, though which properly authorized people and goods go in and out. Bad guys waltz right into the castle with unbreakable walls through the same easy-to-open doors that all the residents and servants of the castle use! How are those doors locked and guarded? Using absolutely proven, tested, tried-and-true methods such as … user names and passwords! And sometimes even PIN codes on top of it!!

Oh, no, everyone but everyone talks about how safe and immutable the distributed ledger is – how is it possible for it to be broken, and why are ancient, obsolete things like usernames and passwords involved? It’s simple, really. Say you’re an experienced break-and-entry person. You walk down a block at night. Most of the houses you see have fences, big locked doors and lights on. One of the houses is dark, and its front door is swinging free. Which house do you pick? Everyone scans for opportunities and goes for the weakest point. So no one even tries to break the cryptography – it really IS secure. But what about the place where the long ID and the no-way-can-I-remember-it private key are held? AHA! We’ve gotten to the weak point of the system: the wallet.

Wallets are just pieces of software that run on your phone, your computer or even online. There are dozens of wallets available. They all claim to store your Bitcoin securely, and maybe they do – except they need to allow a normal human being to actually use them, so they need to be opened by someone a normal human being can handle, like a user name and password! The super-secret but impossible-to-remember private key is stored in the wallet, among other things. Without the private key, you have nothing. If you lose or break the phone that has the wallet, you are screwed. And of course if your phone or computer is hacked, someone else will immediately transfer your Bitcoin out of your account into theirs. There is no recourse. None. Think that never happens? It’s been going on for years, see this. A 20-year-old college kid was just sentenced to 10 years in prison for stealing more than $5 million by phone hacking, see this. That was a first-ever case – usually, no one is caught.

Some people, concerned about the security of their wallets, have turned to professionally-managed exchanges, which both store crypto-currency and enable it to be converted and transferred in and out. An example of such a company is Quadriga, which was the largest such exchange in Canada last year. Now it has made various court filings, and is unable to return its customer’s roughly $200 million in assets, see this. Why? Because those assets were held in a wallet on the now-deceased founder’s laptop computer, and no one can get into the computer – its contents are encrypted and the access information was last seen somewhere in what was the founder’s brain.

Yes, the distributed ledger is secure and immutable. Yes, the cryptography used is in fact unbreakable. Most descriptions stop there, leading you to think you’re protected against loss or theft. As I hope is now clear, Bitcoin and its numerous offshoots do indeed involve some amazing technology, as I’ve described here. But that doesn’t mean that it’s any more secure or protected against loss than any other software. Because of its immaturity, it is proving to be more vulnerable to loss and theft than “normal” money. And the non-monetary blockchain solutions are just as vulnerable.

This post originally appeared on Forbes.

If You Like Private Blockchain, You Should Also Like Living in a Tent Instead of a House

Bitcoin is an amazing technology. I admire it. The central idea of how to implement a virtual currency with no one in charge, but where the “bank vault” is nonetheless pretty safe, is clever, as I explained here. However, the second you take this clever idea and apply it to situations for which it was not designed, it quickly becomes ridiculous – inferior by factors of thousands compared to existing solutions. It’s as though you liked hiking and camping in a tent -- and went back to your home on your suburban block, knocked your house down, disconnected from electricity, municipal water and sewer, stopped garbage collection, sold your car and bicycle, and gloried in your new, improved way of living during cold winter nights. Good idea, but wrong place, and there's probably a reason few people choose to live that way.

As a start for understanding why private blockchain is a ridiculous notion, let’s imagine that we all live in buildings with municipal water supply, and that suddenly someone decided it would be cool to live “off the grid” in a dry area where it does rain, but infrequently. How are you going to get and store the water you need to live? Obviously, you have to somehow make maximum use of the rare rainfall that happens. You construct as wide and varied a system of rain-catchers as you can. If you have a house with a roof, you arrange the gutters to go to downspouts to rain barrels. You carefully construct the rain barrels so they don’t leak, since water is precious, after all – the rain barrels have to be “immutable.” You also stretch out any canvas or anything else you can scrounge up to capture the rain before it hits the ground, and route it to barrels. You construct a set of pipes to connect the barrels, to make sure that all the barrels have water, and none has too much. You end up with a distributed set of barrels, each containing the precious water you need to collect and preserve. The system is even more impressive when it supplies a small encampment of people, with pipes distributing the water among the barrels, assuring that everyone has enough water.

 

Anyone wandering in the wilderness who encountered this maze of connected, distributed, immutable barrels and rain catchers would be impressed at what a good solution it was to the problem of having enough water when there’s no municipal water supply. Someone might come out to the place, take lots of pictures and blog about it. It might catch on, and some homeowners with regular water supply might be attracted to the notion of being ready to survive when civilization collapses and everyone will be forced to live off the grid. Most people, of course, will be happy to continue enjoying normal hot and cold running water, available by turning the faucet.

The Bitcoin solution was specifically designed when you really don’t want a municipal currency authority. Like the distributed water catchers and barrels, it’s a clever solution for exactly that problem. What happens when you decide it’s OK after all to have someone in charge – you’re not in the desert, you’re not a survivalist, and you just want a convenient water supply? What kind of sense does it make to somehow get a private corporation to be completely in charge of the system of water catchers, pipes and barrels in a place where connection to central water is readily available? Do you think the new system would be less expensive, more convenient and less obtrusive? Do you think the privately run immutable distributed water system, with all its barrels, catchers and pipes would be able to handle sudden demands like filling your pool or even a few houses running their lawn sprinkling systems at the same time?

That’s exactly what’s happening with private blockchain implementations. Every single vendor that so enthusiastically promotes its private blockchain tells you quietly what’s wrong, things like the transaction rate is worse by factors of thousands compared to normal DBMS’s (of course they don’t put it that way), and a host of other deficiencies that they’re overcoming … by step-by-step adapting standard database techniques first deployed decades ago and by now standard methods, and making an “improved” private blockchain. Improved, but still dramatically worse by all measures compared to standard technology.

Blockchain is a pile of new software that was designed to solve a very special problem that does not occur in normal life. Private blockchain is an attempt to take that highly unique solution, designed for wilderness living with no central authority in charge, and apply it to normal urban/suburban life with a central authority. The amazing, cool and different things about blockchain were invented specifically to solve the problem of having no central authority. The second you introduce a central authority, i.e., make it a private blockchain, all those special things that make blockchain unique suddenly become huge impediments, obstacles with no redeeming virtues.  It makes as much sense as camping out in your suburban back yard -- OK if you're a kid or want to give your tent a dry run, but nothing any sane person would think is an improved way of living.

This post originally appeared on Forbes.

Are Blockchain Smart Contracts Smart? Are They Contracts?

Smart contracts are all the rage in the blockchain world these days. They are the key feature that has pushed Ethereum to prominence. They’re everywhere!

There are just a few little problems. They’re not smart. They’re not contracts. They’re rife with security issues. And they violate the core principles that are supposed to make blockchain wonderful. Other than that, they’re great!

There is a huge amount of rhetoric and propaganda about what Smart Contracts are supposed to be. Here’s the reality: A smart contract is a software program. It’s written in one of a variety of mostly brand-new languages, chief among them Solidity. A smart contract is the exact equivalent in the blockchain world of a “stored procedure” in the database world; this means that it’s embedded in the blockchain and has access to its internal functions.

At first glance, smart contracts can seem like a clever idea that enables endless extensions to the underlying “immutable distributed ledger” technology in which they’re embedded, greatly extending their flexibility and fields of application. Let’s take a look at how this first glance holds up under scrutiny.

Here’s a typical explanation of smart contract:

“Here’s a very reductive way of establishing a smart contract: let’s say you and I have agreed that if I write you a history of bitcoin, you’ll send me $10 on my birthday this year. We can do that via a legally enforceable contract, which involves lawyers, notaries, and so on — or we can do it via Ethereum. In the latter case, you put $10 worth of smart coins in escrow, and when the terms of the contract are met, those coins are released to me. If I don’t meet the terms of our agreement, the coins are released back to you.”

The key, innocent-sounding phrase in the above description is “when the terms of the contract are met.” When I write you a history of bitcoin, will you accept it if it’s a piece of crap? Probably not. Here’s what has to happen with the deal:

• We have to agree on the terms of the deal
  • This happens verbally, no matter what mechanism is used.
• We have to express the deal terms in a mutually acceptable way.
  • In the land of normalcy, either our verbal agreement would be OK, or we’d have an email or paper exchange.
  • In smart-contract-land, someone would have to write a program in an acceptable language such as Solidity. We would both have to have wallets and accounts in the same crypto-currency among the hundreds that are out there. We would have to agree that the Solidity program expressed our mutual agreement. How good are your Solidity reading and writing skills? You would also have to deposit $10 in your account.
• I would write up my history and send it to you
  • In real life, I’d get it to you using paper or email.
  • In smart-contract-land, there is no good way to send an email – and putting an email address into a smart contract would make it available to the public! There are some clever hacks involving external services that monitor accounts in the blockchain, but there’s no direct solution. So I’d have to get my history to you using plain old real-life methods.
• A decision would be made about whether the history I send satisfied the contract.
  • This would be done in real life by you reading the history and making a judgment.
  • Smart-contracts would remain blissfully unaware of this crucial step, unless and until an amazing advance in AI/NLP is somehow embodied in them.
• If the terms of the contract are met, the money would be sent to me.
  • In real life, you’d hand me the $10, mail it to me, or electronically transfer it to me via one of the widely available methods, for example Venmo, which handled over $12 billion in transactions in 2018.
  • In smart-contract land, you’ve had $10 tied up in your crypto account since the contract was agreed to and the program – oops, I meant the “smart contract” – was created by one or both of us. You would then send a transaction to the contract to transfer the money to my account, after which I could convert it to normal money, with fees taken out along the way.

Someone please explain to me how using a smart contract to embody and execute our agreement is an improvement on normalcy?

In addition to the problems mentioned along the way, we’ve got these:

• If you love my history but want to cheat and not pay me, how does the smart contract help?
• If I’m worried about you paying me and want an enforceable contract, how does the smart contract help?
• If we make a written agreement, even a simple email one, at least we can probably both understand it. What are the chances that we are both fluent in Solidity?
• Assuming we somehow manage to write the code in some language, what if we’re less than perfect and have a bug in the code? The teensy weensy little problem with smart contracts is … they’re immutable (i.e., can’t be changed), along with the blockchain in which they’re stored!
• The key part of any agreement of this kind is the characterization of the history I’m supposed to write and the acceptance criteria. How can that be expressed or evaluated in lines of code?
• In terms of just moving the money quick and easy with minimal overhead, how is anything in crypto-land easier or better in any way than Venmo?

Smart contracts are a really cool idea. The best way to use them is in a sandbox where really smart, unemployed people can play games and make experiments, keeping them out of trouble and far away from real life and real problems that need to be solved.

This post originally appeared in Forbes.

Nobody Cares about Data

Nearly everyone professes to LOVE data. Just think about all the talk about Big Data, Data Lakes and the rest. Lies. Liars lying big LIES. Everyone says they like data ... until they get near it. Suddenly they develop fevers and rashes. They're allergic! Someone else will have to actually handle the data!

Data, the foundation of AI, ML, Analytics

All you have to do is get a job in one of these fancy subjects, and you quickly get hit with reality. When you were in school, you had wonderful exercises where you could develop your skills in deep learning, random forest, or whatever. Now in the real world, some older person assigns you to some juicy-sounding task where you'll get to use your skills. Where's the data? you ask. A wrist-wave of the hand tells you it's over there. You go over there and can't believe what you see. Why, it's nothing like it was in school! You try for a couple hours to clean things up. Then a couple days. It's still bad, but maybe good enough. So you run it through some models. Disaster. The system crashes and/or generates garbage. You complain. "Grow up," you're told. "This is what we've got to work with. Deal with it."

At the end of a year, you realize you've spent half your time in meetings of one kind or another, and 90% of the "working" time has been spent trying to get the data in order. With unsatisfying results. You've got some choices to make. You can lie. You can get into management, marketing or sales. You can roll up your sleeves, forget the fancy stuff you learned in school, and become a data clean-up specialist, which is actually more like a create-decent-data-from-scratch specialist. Which is NOT what you signed up for. Waaaaahhhhh.

What's maybe worse of all is the status. AI and machine learning are clearly the prestigious upper floors of a grand apartment building. Deep learning thinks it's the penthouse, but whatever. The lower floors are occupied by simple analytics. The ground floors are occupied by people managing the databases and Hadoop clusters, and maybe even some ETL tools.

And then there are the basements. The sub-basement where the garbage chutes end. Where the janitors live. Where the crap from the elegant apartments is taken to be discarded. Where the water and oil and natural gas enter the building -- the things the fancy people on the upper floors need to wash up, keep warm and prepare to dress elegantly. That's the floor ... and the status ... of the data specialists. 

You can tell yourself until you're blue in the face that without good data, none of the fancy stuff would work. It's the foundation, dammit! The janitors probably tell themselves the same thing about the heat, cooling, hot and cold water, cleaning and garbage removal. True -- but they're still janitors, wearing a uniform and passed in the halls by the upper-floor people as though they don't exist. 

Bad data equals bad results

There's a simple reason why the incredible potential of the Big Data movement has now morphed into AI/ML and is even incorporating Blockchain. The time passes, tick. Tick. Tick. Tick. No results! Uniform use of the future tense! Claimed successes aren't really, when you dig into them.

Some of the reason is typical organizational incompetence. But much is also due to the fact that we are swimming in a sea of big data and no one wants to clean it up! It's so bad, we mostly don't acknowledge it; much easier just to ignore it.

This problem isn't new. See this:

I've talked about the importance of data as the foundation of AI/ML here. I've illustrated the horrendous problem of bad medical data here. Even basic data, like what providers are where, is wrong too often. By the way, these illustrations should be considered informal tips of massive icebergs. When I talk with true experts who are themselves knee-deep in this stuff, I find the situation is ... even worse.

As today's illustration of the problem, let me show you a piece of mail I got. It's from a major corporation, one of the big regional cable companies and internet service providers. They've got decades of experience working with customers in their geography. They've got to know every address, every household, with complete histories of using their service, dropping it, signing up again. How could they not know the basic demographics and the kind of approaches that work and the ones that don't?

Here's the mail I got (I blocked out the street number):

Looks OK, right? Nice and clear. Specific to the town, so it feels personal. Lots of good things about it. They even designed the envelope so you could see the plastic card on the right, with an eye-catching banner over it.

There's just one little problem. JO Black died in May 2001. More than seventeen years ago.

I don't think there's anything else I need to say except, good job, Optimum! You're doing a great job illustrating the near-universal toxic, rotten ocean of data in which we swim, and doing your part in keeping it that way.

Wait, you might say, this is a trivial little problem. In a way, it is: one piece of mail that shouldn't have been sent. But it's an illustration of a problem that's broad and deep. The notion that a wrongly sent piece of mail "means nothing, is trivial" is an attitude that is EXACTLY why people who care about data metaphorically wear uniforms and work out of the basement. Maybe Optimum is worse than all the rest. Sorry, they're not. JO Black gets a VERY slowly diminishing stream of mail at this address from a wide variety of vendors, large and small. So does Mrs. Grace Black, who died 4 years ago. So does Ms. Jessica Black, who lived here for awhile before moving 20 years ago. So do Mr. Samuel Black and Ms. Elspeth Black, who never lived at this address.

The Problem is Everywhere

To be totally clear: the problem isn't just wasteful mail solicitations. It's everywhere, and every stage of data collection and utilization. The problem with healthcare data is immense, for example, as I've illustrated often. Bad healthcare data, which is ubiquitous, has the direct result that normal, innocent people needlessly suffer and die. It doesn't get better, because all the smart people and the important decision-makers are busy attending conferences about how AI is transforming medicine and how blockchain will solve all the medical data problems -- leaving the ragged crew of people who are supposed to fix the problem ignored in the dank basement, spending their time scheming how they can at least get to the first floor, since it's perfectly obvious that no one is actually interested in ... fixing the data problem!!!

Conclusion

Everybody says they want data. BIG data. But what they really want is a springboard to do something prestigious, which turning a toxic stream of severely polluted data into something textbook-clean is not. While hardly the only factor, this is a major factor in the widespread untalked-about failures of fancy modern techniques to deliver practical results. The plain fact is, nobody cares about data.

The Hierarchy of Software Skills and Status in Data Science

There is a striking hierarchy of skills in software, as I've explained here. When you dive into any particular aspect of software, you usually find that it's got a hierarchy all its own. Data science is a subject of intense interest these days, so in this post I'll explain some of the basics of the data science skills hierarchy.

A skills hierarchy is very much an insider's game. What most people care about is status. I talk about the basics of software status here. Remember, the skills hierarchy is a whole world away from the status hierarchy that most people care about. Don't confuse the two!

Data Science skills

First and foremost, it's important to understand the incredibly broad range of subjects covered  by the term "data science." I attempt to explain the basics of the range in this blog post. You can be just amazing in one of those subjects, while being a neophyte in one that the outside world may consider to be "related," but which in practice is not just down the hall, but in a different building on a different campus.The general understanding of this range is SO pathetic that "data science" is typically managed as a completely independent, free-standing group. Which makes about as much sense as believing that a sous-chef belongs in something that isn't a kitchen. Or that everything that everyone does in a kitchen is basically the same thing.

Here's one cut at the hierarchy in data science, starting from the base:

1. Tool users.These are people who have learned how to use some software tool, or maybe a couple. Most "data scientists" fall into this category.
   • They don’t understand how the tools are built or any of the underlying software
   • They may know their tool, but aren't real clear on what the other tools are about, much less when you might consider using one.
2. Some have broader knowledge of tools
3. Very few have the sophistication to understand real data analysis, per my series of AI/ML; see this and the links in it: http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-3-closed-loop.html
4. Of those, even fewer can understand the underlying algorithms and follow the latest research literature
5. Of those, even fewer can make real algorithmic advances and implement them as tools and deliver practical value.
6. Of anyone who can do all of the above, it is rare to meet anyone who can address and solve deep tool-level problems in other domains needed to make their code practical in the real world.
7. Finally, it is extremely rare to meet people who, in addition to their deep and broad prowess in data science and relevant skills needed to make it real-world practical, have similarly deep skills in an associated domain needed to make the data science fully effective for a business.

The issues of this hierarchy are compounded by the usual over-selling by people who are good promoters but little else, and corporate/government big-wigs who don't want to be bothered with details, but are keen to be seen as "doing something" on such a high-visibility topic. Getting results that make a difference is pretty low on the typical priority list.

And then of course there are the "data scientists" themselves, who most often are sincere people who are trying to do a good job as they've been taught to do it -- mostly by professors and others who have no idea what real-world success looks like, much less how to bring it about.

Finally, there is the usual "manage something that's invisible to you" phenomenon I have often discussed in this blog, which leads to so much dysfunction and so many wonderful Dilbert cartoons.

Conclusion

People talk as though "data science" were a thing, with the usual kind of hierarchy based on level of management and/or "experience." Those typical patterns of hierarchy just don't cut it for understanding what's going on in data science, just like they don't cut it for understanding software development. We will continue to see waste and dead-end efforts until we at least make a start at making our understanding of data science more sophisticated, and aligned with the facts on the ground.

What Software Experts think about Blood-letting

Software experts do NOT think about blood-letting. But ALL medical doctors thought about blood-letting and considered it a standard and necessary part of medical practice until well into the 1800's. They continued to weaken and kill patients with this destructive "therapy," even as the evidence against it piled high.

The vast majority of software experts strongly resemble medical doctors from those earlier times. The evidence is overwhelming that the "cures" they promote make things worse, but since all the software doctors give nearly the same horrible advice, things continue.

Blood-letting

Blood-letting is now a thoroughly discredited practice. But it was standard, universally-accepted practice for thousands of years. Here is blood-letting on a Grecian urn:

Consider, for example, the death of George Washington, a healthy man of 68 when he died.

Washington rode his horse around his estate in freezing rain for 5 hours. He got a sore throat. The next day he rode again through snow to mark trees he wanted cut down. He woke early in the morning the next day, having trouble breathing and a sore throat. Leaving out the details, by the time of his death, after treatment by multiple doctors, about half the blood in his body had been purposely bled in attempt to "cure" him of his sickness!!! If he hadn't been sick before, losing half the blood in his body would have killed him.

If you are at an accident and you or someone else is bleeding badly, what do you do? You stop the bleeding, because if you don't, the person will bleed to death. That's now. Then? You bleed the sick person because it's the universally accepted CURE for a wide variety of sicknesses.

Bloodletting was first disproved by William Harvey in 1628. It had no effect. It remained the primary treatment for over 100 diseases. Leaches were a good way to keep the blood flowing. France imported over 40 million leaches a year for medicinal purposes in the 1830's, and England imported over 6 million leaches from France in the next decade.

While blood-letting faded in the rest of the 1800's, it was still practiced widely, and recommended in some medical textbooks in the early 1900's. We are reminded of it today by the poles on barber shops -- the red was for blood and the white for bandages; barbers were the surgeons who did the cutting prescribed by doctors.

Blood-letting in software

By any reasonable criteria, software is at the state medicine was in 1799, when everyone, all the experts, agreed that removing half the blood from George Washington's body was the best way to cure him.

If you think this is an extreme statement, you either don't have broad exposure to the facts on the ground or you haven't thought about what is taken to be "knowledge" in software compared to other fields.

I hope we all know and accept that the vast majority of what we learn and come to believe is based on authority and general acceptance. This is true in all walks of life. Of course not everyone believes the same thing -- there are different groups to which you may belong that have widely varying belief systems. But if you're somehow a member of a group, chances are very high that you accept most things that most members of that group believes.

This is no less true in science-based fields than others. The difficulty of changing widely-held beliefs in science has been deeply studied, and the resistance to change is strong. See for a start The Structure of Scientific Revolutions. I have described this resistance in medical-related subjects, and in particular showed how the history of scurvy parallels software development methods all too well.

But at least, to its great credit, medicine has gone through the painful transition to demanding facts, trials and real evidence to show that a method does what it's supposed to do, without awful side-effects. That's why we hear about evidence-based medicine, for example, while there is no such thing in software!

I hear from highly-qualified and experienced software CTO's that they are going to lead a transition of their code base so it conforms to some modern cool fashion. One of the strong trends this year has been the drive to convert a "monolithic code base" (presumed to be a bad thing) to a "micro-service-based architecture." When I ask "why" the initial response ranges from surprise to a blank stare -- they never get such a question! It's always smiling and nodding -- my, that CTO is with-it, no question about it.

Eventually I get the typical list of virtues, including things like "we've got a monolithic code base and have to do something about it" and "we've got to be more scalable," none of which solves problems for the company. When I press further, it becomes obvious that the CTO has ZERO evidence in favor of what will be a huge and consequential investment, and has never seriously considered the alternatives.

As is typical in cases like this, when you scan the web, you see all sorts of laudatory paeans to the micro-service thing, very little against it. Most important, you find not a shred of evidence! No double-blind experiments! No evidence of any kind! No science of any kind! What you also don't find is stories of places that have embarked on the micro-service journey and discovered by experience all the problems no one talks about, all the problems it's supposed to solve but doesn't, and the all-too-frequent declarations of success accompanied by a quiet wind-down of the effort and moving on to happier subjects. Because of my position working with many innovative companies, this is exactly the kind of thing I do hear about -- quietly.

Conclusion

We've got a long way to go in software. While software experts don't wear white coats, the way they dress, act and talk exudes the authority of 19th century doctors, dishing out impressive-sounding advice that is meekly accepted by the recipients as best practice. No one dares question the advice, and the few who demand explanations generally just accept the meaningless string of words that usually result -- empty of evidence of any kind. It's just as well; the evidence largely consists of "everyone does it, it's standard practice." And that's true!

Software experts don't think about blood-letting. But they regularly practice the modern equivalent of it in software, and have yet to make the painful but necessary transition to scientific, evidence-based practice.

 

Apple's Facetime Problems Illustrate What's Wrong with Blockchain

Apple’s had a rough time recently, with bugs, security problems and sales issues. The recent Facetime bug is particularly embarrassing. It’s made the news! There are stories about it all over. Apple is scrambling to fix the issue and end the pain and embarrassment, pronto.

Blockchain has also had a rough time, with recent cavernous losses that extend a years-long pattern. Blockchain enthusiasts march on, seemingly oblivious to the intractable problems that cripple their beloved technology. So far as anyone can tell, no one is scrambling to fix the problems.

Comparing the two situations is interesting and educational.

The Apple bug was first discovered by a teenager while he was setting up a Facetime group chat.

• Most blockchain problems are discovered sometime after a substantial loss has taken place, when you go to check your account and are shocked to find it has a whole lot less value than it had last time you checked.

The boy and his mom were frustrated by how hard it was to get through to anyone at Apple to report the bug. They were doing the right thing, while Apple was being a typical lumbering, unresponsive bureaucracy.

 

• You discover the loss in your crypto account. You’re upset. Who do you call? Where’s the 800 number for customer support? If you didn’t know it already, you quickly discover that there’s no number, no one to call. There’s no organization in charge at all! And in those cases where there sort of is, they refuse to fix the problem.

In less than a week, Apple officials woke up to the fact that they had a problem. A big, ugly, embarrassing one. To their credit, they did two things.

• In some of the most important cases, such as Bitcoin, there is literally no one in charge – that’s the whole point of Bitcoin – it’s a system that was designed to have no one in charge. It’s brilliant, as I describe here. But it’s also fatal when the system is hacked.
• In other cases, such as exchanges, there is someone in charge. But their typical response is to claim, with good justification, that “fixing” the problem would destroy the fabric of blockchain. And after all, in many important cases, the funds have long since been converted to cash and are long gone – how can that be “fixed” without catching the bad guys? And as I’ve described, while bad guys in normal banking are often caught, in the crypto-currency world, they almost never are. So your money is gone. Gone!

The first thing Apple did was shut down their servers so that group Facetime was no longer possible. This didn’t happen all at once, but rolled through the system pretty quickly.

• Immediate action to fix the problem in blockchain? Doesn’t happen.

The second thing Apple did was announce that the bug would be fixed and released in about a week.

• Here’s where it gets really bad for blockchain: where’s the bug that can be fixed to solve the underlying problem? No one can say! So far as all the blockchain “experts” are concerned, there is no problem to be fixed. The silence is deafening. When have you ever read about any kind of crypto-currency loss, after which someone “in charge” said something like: “the loss was due to [this bug}, which we’ve found and fixed and will be effective on [this date]”? Anyone?
• When the responsible problem is in a so-called smart contract, it’s even worse. Smart contracts are stored in the immutable blockchain itself, and so, unlike normal programs, can’t be fixed or changed in any way. The geniuses who invented and support smart contracts consider this fatal flaw to be one of their great advantages. Go figure.

The net effect of the Apple bug is that the privacy of certain Facetime users was compromised. Embarrassing. Bad. But your wallet is unaffected. But at least the news got out quickly, people can avoid the feature for a bit and then go back to using it, should they choose.

• The net effect of the myriad of Crypto and blockchain bugs and hacks is loss of the equivalent of real money, sometimes to the tune of tens of millions of dollars. The only way to avoid the problem is to get out of crypto. Totally. And never go back.

The contrast between the Apple Facetime bug and the various crypto/blockchain bugs and hacks couldn’t be more stark. With Apple, there’s someone in charge; the someone wakes up to the problem within days, and moves decisively to first block the problem and then fix it. In blockchain, there’s no one in charge (by design!); the affected people wake up to their problem, whine about their often substantial financial loss, but are largely ignored by the community of experts and operators, who soldier on, promoting the wonders of the amazing immutable distributed ledger technology.

Here’s my prediction: the blockchain mania will continue to spread for a while, but then will slowly fade away, with few of the promoters admitting their lapse in judgment. As with most technology fads, everyone’s attention will simply shift to something shinier and newer as the problems grow too large to be ignored. There's a long shot there will be some shining successes with blockchain -- but I predict that when you look under the covers, it won't really be blockchain doing most of the work.

(Disclosure: While I’ve read API’s and source code, I’ve never owned cryptocurrency of any kind, and don’t plan to any time soon.)

This post originally appeared on Forbes.

Crypto-currency Hacks and Losses Mount While Supporters Remain Silent

Insiders most like to describe Blockchain as Immutable Distributed Ledger technology. They love that it’s distributed, and a “ledger” rather than a database. But most of all, they seem to like that it’s “immutable.” To enthusiasts, this means that the unbreakable cryptography and other techno-nerd elements result in impregnable, hack-proof software. In a world filled with crappy software that’s thoroughly “mutable,” hackable, breakable and a smorgasbord of other criminal, consumer-hurting things, this is a wonderful thing. No wonder so many people and corporations are jumping on the Blockchain Bandwagon. The smell of FOMO (Fear Of Missing Out) fills the air.

The FOMO is really strong on Blockchain. So strong that it appears to prevent enthusiasts from paying attention to the fact that has been established over the last few years: Blockchain may indeed be Distributed and a Ledger (more on those in subsequent posts), but it’s hardly immutable. In fact, it’s just as hackable as any other piece of software – even more so because no one’s in charge of keeping it safe!

The latest loss is small by comparison to some of the earlier ones. The one announced on January 8, 2019 amounts to “just” $200,000 worth of ethereum classic. What’s worse is that the attack was at the core of  the blockchain. Apparently the attack was carried out by miners, the servers that are at the core of blockchain’s operations and security, the ones that perform the magic cryptography that supposedly prevents bad things from happening. The hack itself involved the absolutely worst thing that can happen to a crypto-currency – about 40,000 ETC was double-spent.

If this were the first loss, I would understand the blockchain folks minimizing its importance. It’s hardly the first loss. Who talks about the Mt. Gox hack, in which nearly half a BILLION dollars was lost? That happened about 5 years ago! Mt. Gox has been followed by an un-ending stream of other successful (for the criminals) attacks and losses. One of the more famous was the $50 million lost in the DAO hack. Less famous hacks resulting in losses over $10 million took place in 2018. The pattern of criminal success shows no signs of slowing down.

How can any sane person continue to back blockchain as a transforming technology due (in part) to its immutability and implied greater security in the face of this evidence? Obviously, what’s happening is that people are simply ignoring the evidence. That’s it!

Let’s put this in context. What would the stories be if anything comparable took place with plain old banks, with their supposedly obsolete software and security that’s primitive by comparison? While lots of normal banks are robbed every year, these are small-scale occurrences, and many of the perpetrators are caught. For example, here is the FBI’s latest news about bank robberies. You’ll see that there are loads of convictions and prison sentences. Here is a story from December 2018 about a man who robbed a local bank of $536 in order to pay his rent. He has just been sentenced to 46 months in prison.

Are there bank robberies in which large amounts are stolen? Check out the list of them in Wikipedia. The list goes back more than a century. The largest robberies don’t come close to the criminals of the blockchain world. The most recent one listed was 20 years ago, when the Bank of America was robbed of less than $2 million. Peanuts!

OK, you might say, but what about cyber-crime? There’s actually quite a bit of it. But digging into the exact nature of the crimes and how they’re committed is quite interesting. It may exist, but I couldn’t find ANY cases of the core bank systems being hacked. And NONE of the central database (the equivalent of blockchain)! In every case I’ve seen, it was plain old systems network hacking and/or criminal employees that were the problem. Yes, some large amounts were involved, for example in the Bangladesh robbery. But in that case as in many others, corrupted insiders were involved. It had nothing to do with the security of the banking software itself!

If normal banks had been hacked the way blockchain has been hacked, you’d find that the core banking system itself was breached, or the central database itself. In no case that I can find has this happened. What this means is that blockchain, in its short existence and with its relatively tiny fraction of money, has been hacked more successfully and more deeply than any normal banking software has been. “Immutable,” huh? Explain that again, please. But stick to the facts this time.

The conclusions we can draw from these facts are simple, clear, and hard to dispute:

• Blockchain is highly susceptible to being hacked in a wide variety of ways.
  • This has been demonstrated by events for more than 5 years; the hacks are on-going.
  • Large amounts of money are lost in the hacks.
  • The hackers aren't caught, much less punished.
• While there are lots of physical robberies of old-style banks, the amounts involved are small, the perpetrators are often caught, and consumers are not hurt.
• While there are hacks on old-style banks, they have had little success in US banks, and the same kind of cyber-security breaches that occur everywhere are involved.
  • In no case have the hacks been as deep in the software as many attacks on blockchain have been.

There's lots more that could be said about this, but here's the bottom line: if you want to keep your money safe, put it in a traditional bank, whose software systems are indeed immutable. Any blockchain storage is more susceptible to attack and loss than the software used by traditional banks.

This post originally appeared at Forbes.

The Novel Idea at the Heart of Bitcoin

There aren’t many true and surprising new things in software technology, in spite of all the gushing about new stuff. At the center of Bitcoin is a tech advance. Not a minor step forward. Not an enhancement fueled by faster chips. An amazing idea that is the engine that has fueled its explosive growth. It’s not something people talk much about, sadly. They should. The core of the idea is the miners that are the heart of the Bitcoin engine.

If you’ve heard anything about Bitcoin, you’ve probably heard that it’s a crypto-currency. You’ve heard it’s totally secure because lots of computing locks the data in an air-tight vault secured by the latest cryptology algorithms. You’ve heard that it’s a ledger of transactions, and that the ledger is distributed, which somehow makes it better. All these things that you’ve heard are correct – and it’s the miners at the heart of every one of those true things.

First, let’s step back a minute and understand the problem that Bitcoin solves. Bitcoin isn’t cool in the abstract – it’s cool because it’s a creative solution to a really hard problem. The problem, in a nutshell, is to create a currency, like US dollars or Euros, that isn’t controlled by any central authority. That’s a HARD problem. How can you have a currency that people accept with no one to issue it? If it’s somehow issued, who’s going to do the work of creating and managing it?

Long ago, currency consisted of fairly scarce, valuable objects, like sea shells. Then, precious metals were used. Since it was hard to judge how valuable a piece of metal was, people in authority created standard sizes, shapes and values – today’s coins. Then paper money was issued by early banks, with the precious metals in the banks backing it up. Central government authorities then replaced the banks, and currency became a per-country thing. Finally, it became detached from the coins, a.k.a. “the gold standard.” That’s where we are today, with huge government authorities issuing and controlling abstract and paper currency at will, manipulating it to meet political goals. The problem at the heart of Bitcoin is, how can you create an abstract currency that people can trust without a central authority of any kind, much less a government? I trust you’ll agree, that’s a truly hard problem.

Before diving into the solution to this problem, it’s worth understanding why it’s a problem. The core issue is the money supply, and how central authorities manipulate it to implement monetary policy in various ways, shifting with the political winds. Central bankers can print more money, take money out of circulation, raise and lower interest rates and do other things on a whim. Other branches of the government closely monitor who does what with their money via extensive, ever-growing, onerous regulations that make everything harder, slower and more expensive. How can we get out of this? How can we escape the armies of faceless bureaucrats who control the money and watch what we do with it?

The solution has to somehow make everything work with no one in charge. Get people to do lots of work and spend lots of normal money to create and maintain a robust system of virtual currency, and somehow get those people to be absolutely incorruptible?! They’ll be in charge, but not tempted even a little to use their power to enrich themselves? How is THAT supposed to happen??

That, my friends, is the genius of Bitcoin. The genius is embodied in the design of the miners.

Miners are volunteers. No one selects them – they just step up, get their hardware and software together, and start mining. All on their own – without permission and without even an invitation! Here’s the key part: when you mine, you make money, in the form of newly-issued Bitcoin. The formula and the rules are built into the software that everyone uses. When you mine, you make money. The more you mine, the more you make. If you’re ever tempted to think about fiddling with the software, cheating and just taking a bunch of money (Bitcoin), you immediately think of the huge investment you’ve made in mining equipment, which isn’t good for much of anything except mining. If people started thinking that miners were self-dealing corruptocrats, the value of Bitcoin would immediately plummet, and the miner’s investment would be worthless. Your thought of cheating, just a little, quickly flies out of your head, and you go back to being a straight-up miner – and, by the way, watching the other miners closely to make sure none of THEM cheat; if they did it would hurt you. Badly.

The miners are un-recruited, unmanaged groups who put up their own money and time to make money, and are thoroughly incented to play it straight, without cheating.

What the miners actually do is solve computationally intensive problems – all using standard software on juiced-up hardware – that does two important things.

• First, the computing assures that each new transaction that someone tries to put in the ledger follows the rules. Simple rules that are essential to virtual currency working. Things like you can only spend money you have. You can only spend it once. Stuff like that, things you don’t even think about when your money is physical and sits in a wallet -- but when it’s digital, has to be enforced.
• Second, the computing puts a lock on the new transaction, a special fancy lock that links to all the earlier locks on all the prior transactions. For ease of computing, the transactions are grouped into blocks, and it’s actually the blocks that are locked up tight and chained together with cryptology. Thus the name “blockchain.”

The rules built into the Bitcoin/blockchain software used by all the miners are the key to everything. Since all the miners run the same software, everyone follows the same rules. These rules enforce the fact that, at any given moment, there are a known amount of Bitcoin, with the ledger tracking who owns how much. The number of Bitcoin is fixed – until a miner earns some as a result of the mining work. In that case, brand-new Bitcoin are created – according to an established formula – and deposited in the miner’s own account in the ledger. Once the miner has the earned Bitcoin, he can do anything he likes with it, like any normal owner of Bitcoin.

Finally, it’s true that the Bitcoin miners see each and every transaction. Each transaction is vetted to assure that the rules are followed. But the owner is identified only by a VERY long string of letters, a key, so no one knows who the owner of Bitcoin is in physical life. This is the capstone of Bitcoin’s solution to the problem of government-issued currency. No snooping!

Net-net: There is a publicly known amount of Bitcoin in the world, which slowly grows as it is created to pay the miners who earn it by running the system. There are a large number of volunteer miners keeping transactions flowing, safe and secure, without depending on any of them. Bitcoin buying and selling is easy, inexpensive and private. Because of thousands of volunteer miners crunching away. No one’s in charge. Miners want to work and are incented to be honest. No governments, no bureaucracies, no politics, no one snooping on you. Problem solved!

That’s why Bitcoin/blockchain is new and deserves the attention and credit it’s gotten.

This post originally appeared on Forbes.

Adventures with Health Insurance Software: Customer Feedback

I got an email from my health insurance company, telling me I had an important message I could read if I clicked and got to their website. Here's what happened. While I was on the site, I discovered they were delivering break-through functionality making it easier to pay those annoying doctor bills that appear in the mail long after the visit. Here's the scoop. This post tells what happened next.

One of the best things Anthem did to enhance their customer website experience is to be humble. So many stuck-up website creators are sure they’ve done the best job that can be done, and simply put the site out there. Here it is, visitors, we’re sure you’ll agree that it’s a truly excellent website!

The experienced professionals at Anthem are well past this kind of immature attitude. They worry that the site they’ve build isn’t as good as it could be; how better to find this out than by asking the customers, the actual people who use the site?

Certain modern website designers get this kind of information in a sneaky, underhanded way. They closely monitor each click and keystroke made by visitors, and track the time between each. This method of surreptitious shadowing enables them to discern exactly when and where visitors get stuck, bogged down, get lost or whatever. That way they can enhance the site and run experiments to make everyone’s experience smoother – without telling anyone what they’re doing!

It’s amazing the public puts up with this kind of spy-movie tactics! The folks at Anthem aren’t about to be sneaky or underhanded in any way. They’re committed to open, fair and above-board methods. I experienced this myself. When I was in the middle of experiencing their amazing new billing features, I was presented with this screen:

It’s an offer to give feedback, via a third-party tool. Excellent! I think I’ll say yes, and give them my feedback.

I proceeded to experience the fullness of the new, break-through patient billing feature I’ve described here. Then, when I left the site, sure enough, I had my opportunity to provide feedback. I dove in and first saw this screen with a couple questions to which they “require” the answers.

What’s that about? This is voluntary, remember? I’m helping you guys. You should be glad to get any feedback I care to give. And I’m not starting out in a great mood because the first couple questions are completely generic.

So please forgive me, but I zoomed ahead, getting to here

Question 23! Sorry, I wasn’t able to get it done. I zoom to what looks like the end:

Getting personal with the income, are we? After a whole pile of b.s. questions. I’m outa here. But then this appears:

Yup, 26 questions. ALL OF THEM “REQUIRED.”

I wonder if anyone tracks the completion rate of these surveys. I suspect not.

I bet Anthem gets incredibly useful information from these surveys. And increases customer satisfaction along the way. No wonder their site is clearly top-of-the-line.

 

Adventures with Health Insurance Software: Provider Billing break-through!

I personally experienced the roll-out of Anthem BC-BS’s new patient billing initiative. As is well known, and as I’ve discussed in detail, patient billing, both from providers and payers, is a nightmare for everyone involved. Fixing it appears to be tough – it’s worse than being between a rock and a hard place; billing is between a rock, a hard place, the devil and the deep blue sea.

As I’ve said many times, Anthem is one of the best health insurance companies out there, full of hard-working, well-meaning people who just want to do the right thing. Their many efforts at making things better demonstrate this. But there’s clearly something about being a giant organization with lots of extensive, complex software that seems to make things go wrong. Small, entrepreneurial organizations can sometimes avoid the usual traps and turn out great, high-quality software quickly. But the big, lumbering bureaucracies just can’t seem to follow the clear patterns of success demonstrated by the winning the upstarts. Everything is against winning. They follow the regulations, the best advice of the best experts from industry and academia, and the results are consistent. Consistently bad.

Getting things badly wrong in software is most often a team effort. When it’s wrong, it’s rarely an ooops, how did that happen? I’ll fix it and then we’ll be OK! It’s more often pervasive, multi-dimensional badness. Anthem’s billing break-through is clearly the typical case.

In this post, I’ll describe the break-down, the fact that the new billing effort is broken. Again: this is not about Anthem specifically. It’s an example of a pervasive issue.

What led me to discover the new patient billing feature was an email I got from Anthem. That’s a story in itself, which I tell here.

While I was on the site, I had occasion to carefully examine the welcome page:

I noticed something new since last time I was there – the ability to pay provider bills. Wow! I’d never heard of any insurer offering this capability before. Of course I want to try out this amazing bill-paying break-through.

I’d better step back for a moment and explain the billing situation in healthcare, because when you just glance at what you’re about to see, you might think it’s no big deal.

In US healthcare insurance, there’s something called a “co-pay.” This innocent-sounding term is the cheerful face of a nightmare for all involved – patients, providers and insurers. At some point, I have no idea when, someone decided that insured patients should pay a part of their cost of their care, a bit like the well-known general insurance of a deductible, but fancier. The idea is that, for each service a patient gets, insurance would pay part of it, if the service is covered at all. Then, depending on a whole bunch of other things like whether a provider is “in-network,” the patient should pay something to the provider, an amount that the insurance company calculates using arcane formulas that are secret. Here is a detailed example.

So get this: you have insurance. You go to a covered provider to get a covered service. The insurance company pays some of the provider’s bill. The provider generates a bill for you, sometimes when you check out, but often much later, in the mail. The provider’s bill includes an amount you have to pay directly to the provider. That’s the co-pay.

This is the context that helps you understand Anthem’s billing break-through: instead of getting random bills in the mail from random providers and having to write checks and mail them various places, Anthem is letting you go to one central place and take care of them. Even electronically! A great convenience.

Health insurance companies are ideally positioned to provide this service! They already have the complex systems set up to transfer money to providers (doctors and hospitals) electronically, because they’re already paying them! Setting this kind of system up from scratch is hard, and maintaining it with all the changes to bank accounts, etc. takes a lot of work. Leveraging all that infrastructure to enable patients to pay their bills is a great idea.

Even better, the insurance company knows how much the patient is supposed to pay the provider! Even though the normal procedure is that the providers bills the patient, all the numbers ultimately come from the insurance company. So all the required data and systems are already in place! What a fabulous opportunity!

Naturally, I dove right in to trying it out. Here’s what I saw right away:

Now I’m getting really pumped – it’s exactly what’s needed!

I ask for the list and see a juicy target to pay:

Perfect, a typical co-pay. I click to see the details of the claim. Typical stuff, as expected:

Of course, it's a bill that no one but a clueless health insurance company would have the nerve to create. It doesn't tell me who provided the service, when the service was provided, where it was provided or what the service was. Other than that, it's great! Oh and by the way, they don't seem to know whether I actually owe this money or whether I've already paid it. In fact, I happen to know I have already paid it!

I know, complain, complain. But still, moving ahead...

I’ll spare you the details of how I had to provide payment information. I did it and authorized payment. Then I got this lovely screen:

Yes, that’s the whole screen. The technical term for this kind of screen is “face plant.” A “face slap” is when there’s an error, but the software knows it and can give you a nice error message with something soothing about how things aren’t as bad as they look. When you’re on a site labelled Anthem and then suddenly you’re no longer on their site and some other company’s logo appears, you know that the two bodies of software are in all-out war. You know the war has gone nuclear when the dreaded “500” error appears. That’s not an error message generated by software that is confused and knows it’s in trouble – that’s an internal website server error. This means things have gone so bonkers that the software that’s supposed to handle things has gone AWOL, and all the server can do is say, in technical terms, is OOPS.

So I re-grouped, went back to the start, picked another bill, put in my credit card information (again!), and authorized payment. Here’s what I got for my trouble:

Things are getting better – we’ve advanced from a full-on face plant to a relatively mild face slap error – no one else’s logo is in sight, no servers are complaining (at least visibly). It just plain doesn’t work.

Anthem presents bills that fail nearly all the criteria required of a reasonable bill. Their bill payment mechanism doesn't work. Even if the final step of paying the bill worked instead of face-planting or face-slapping, it would still be near-worthless! What good is a bill that doesn't actually show what you owe? Doesn't show when, where or by whom the service was provided, or even what the service was?

Here is the simple conclusion: Anthem claims to have a revolutionary method to enable patients to pay their providers, a great convenience. If it works. Which it doesn’t, to varying levels of embarrassment.

Adventures with Health Insurance Software: Email and Primary Care

Giant organizations have trouble building effective software that works and gets the job done. I have gone into depth on this subject, giving examples of the problems. But there’s something about being a large organization that seems to prevent even being aware that there’s a problem, much less being able to fix it.

I recently had occasion to dive into my health insurance company’s website, enticed by an email to do so. What I experienced was a travesty. If this company were run like a company should be run, heads should have rolled. It’s as bad as a trucking company having a large fraction of their trucks wander around getting lost, and another fraction driving off the road and crashing.

Unfortunately, this story is not about an unfortunate bug or two that somehow snuck into otherwise fine software, which is what any self-respecting manager would start by trying to claim. This is story is about software that is broken in concept and in execution – even when it “works,” it’s simply awful!

What I’m saying here flies in the face of what nearly everyone says and appears to think – including all the managers at all the places that preside over this nightmare of dysfunction. You also don’t hear any lofty academics decrying the “crisis in software,” as they should. So I’m going to lay out the facts, point by point; this is NOT fake news.

This is the first of three blog posts on this subject. This first one is pretty mild.

I got an email from my insurance company. Here it is:

I have a new message – and it’s not sales or promotional! Nothing about what the message could be about. It must be too secret and confidential to put it in regular email. Maybe it’s something about my health? I’d better check. So I click.

Oh, yuck. I’ve got to log in.

Now I have to decide how badly I want to read this non-sales email. They seem to have decided that giving me an intelligence test combined with an endurance test was the best way to determine whether I was worthy to read this non-promotional, possibly health-related message. I persisted. I dug out my user name and password for this site I rarely use, and logged in.

Or rather, I attempted to login. Here’s what I got after successfully entering my user name and password:

My user name and password weren’t good enough! This is clearly an incredibly confidential message! Even though I was using a computer I use all the time, including when accessing Anthem. I picked email, and then got this screen:

I entered the 6 digit code.

This is classic 2-factor authentication. The security “experts” at Anthem probably felt pretty good about how they increased the security at Anthem, particularly after their past embarrassments. But it’s all GARBAGE! Nothing but security Kabuki Theater! Think about it: I got to the login screen by clicking on an email that Anthem sent me!! It’s trivial to include in the email link’s URL the information about the email. So when the request comes in … Anthem knows it’s coming from the email they sent! A simple check would tell them it also is coming from a computer associated with that email. By going through the send-email-enter-6 digit-code b.s., all they’re doing is wasting my time because they already have proof that it’s my email.

Next, there’s the remarkable screen telling me how hard Anthem is working on my behalf:

All this hard work will surely result in displaying the information that the email I clicked on long ago was enticing me to click for, right? Well, no.

A completely generic welcome page!

This is a problem. A big one. You’re supposed to click to read an important message. In every system I know, a “click-me” email is a “deep link,” i.e., it doesn’t go to the home page of the web page; it goes “deep” into the site, to the place the email wants me to see. You’ve experienced this. When Facebook or LinkedIn sends you an email about something, when you click, it always deep-links you to the place referenced by the email. My blankity-blank BANK does this. Even confidential document stores that need to be highly secure do the same – once you’ve verified yourself, you go right to the document. Makes sense.

Except to Anthem. Anthem’s email link brings me to the generic welcome page of Anthem, exactly the same thing I’d see if I’d gone to the site directly.

I can barely remember how I got here, it was such an annoyingly long time ago. Oh, yeah, the email – I’ve got an important message! Now, where might that be? I look at the screen. Why don’t you check it out too – do you see anything that says “messages?” Me neither. Clearly this page, the front splash page of the Anthem patient website, has received the best vetting that the skilled professionals at Anthem can muster. And the vetting somehow failed to notice that they were going to send me to a page looking for a “message” without those seven wonderful letters appearing anywhere on the page.

Again, a combined test of intelligence and endurance. Let’s see if I can pass. Taking a closer look at that generic landing page, look at where I've put the big red arrow…

Aha!  I wonder if, by any remote chance, that red shape means messages (in the secret Anthem language), and I have 10 messages that have piled up? Let’s try clicking.

Score!

The endurance test continues. Click again. Finally, the important message in question:

At this point, all I can say is OMG.

1. I have a primary care doctor, Anthem. You know it because you pay insurance bills for that doctor covering suspiciously primary-care items like “wellness visit.”
2. The primary care doctor you’ve selected for me is indeed in the same state as me. But “close?” Not even in the same county. Sorry. No chance.

I’m so glad I endured the obstacle course and endurance test, making my way past the elaborate privacy protections to read this important message with spot-on recommendation, so cleverly refined with accurate GPS data. I can’t put into words what this has done for my admiration of the excellent insurance company that orchestrated this software ballet.

Using Advanced Software Techniques in Business: Fashion or Real Value?

Using advanced software techniques can make a dramatic positive impact on business. It’s important for everyone to assure that your software efforts aren’t stuck in out-moded, last-generation tools and techniques.

Nearly everyone, including me, agrees with this simple statement. Nearly everyone also agrees on at least the top members of a list of reasonable candidates of “advanced software techniques” that are not “out-moded” or “last-generation.” That last statement is where the best technical people part ways with the crowd.

No, I’m not talking about hard-to-understand weird-o’s babbling about esoterica in some corner. The best technical people understand and support using the best and most appropriate techniques for solving a given problem, regardless of the recency of the technique or its prevalence. Sadly, it is often the case that the most-talked-about hot trends in software should not be used in any business that actually wants to spend money widely and get stuff done, quickly and well.

This is a BIG subject. It’s important. It’s deep. And it’s extensive. So let’s start with a big, fat, juicy example, one that was hot, hot, HOT but is now fading away, so it’s possible to talk about it somewhat more rationally. Maybe. I hope.

Big Data and Hadoop

There is little doubt that Big Data is a huge trend in software, though at least talking about it with that name appears to be undergoing a typical slow fade. Here is a review I wrote more than 5 years ago of the Big Data fashion trend as it existed at that time. It was everywhere you looked! Magazine covers! Ads! Conferences! Books! If you weren't somehow doing Big Data you were nothing and nobody.

I've been working with data my entire professional life. The data has been small, medium, large, big, huge, totally awesomely huge and even gi-normous. Since I've been faced with space and time constraints, I have long since settled on a fundamental concept of computing, simple but rarely done: Count the data! It sounds ridiculous, but it's almost a secret weapon, and appears to be rarely done. Here is some analysis of data sizes in the context of the big data trend. Here is a more detailed example of a big data set that Harvard bragged about. Hint: the data isn't very big.

I shouldn't have to say this, but here it is: For anything that people say is "big data," the very first step should be to ... count the data. Sounds simple, but apparently it's not. It's also not common to dig into the data a bit. I guess it's uncommon because in most cases of data that starts out looking big after you count -- this by itself is rare -- is that you find that most of the data is just not needed or not relevant. Which makes it not big anymore. Which means you don't need Hadoop!

Hadoop

I know I'm being silly here. Hey, we're talking Big Data! Surely we've got some somewhere. We've got to get in an expert and crunch away so we can get those virtuous, business-enhancing juices flowing through our company.

In this situation, at least until recently, what that meant was that you dove into the next level of detail and found out that the go-to tool was Hadoop. Dig in some more, and it sounds great. It's scalable without limit. You build your Hadoop cluster, script up some calculations, tap into the ocean of data you've got somewhere, and hear about how the Hadoop spins those computers up and down, and crunches all the data, using the computers that are available, and even working around ones that fail without anyone having to respond to some old-style beeper or something. No wonder Hadoop is the go-to tool for Big Data!

In the vast majority of situations, it's "decision made" time at this point. You get in your experts, they build their Hadoop cluster and away you go, climbing the Hadoop stairway to Big Data Heaven, with a glow of virtue surrounding everyone involved.

Very few people seem to dive in and understand what Hadoop and its main programming paradigm MapReduce are all about. The Hadoop "experts" don't seem to know what the reasonable alternatives are, and when they might be applicable.

Here's an example. In 2011, one of our large web companies had a huge problem caused by Google's move to a new search algorithm. The CTO grabbed a massive web log file, wrote some code to boil the terrabytes (Big Data for sure!) of data down to the key data elements, and then loaded them into the 512GB of DRAM of his powerful laptop computer and ran some advanced machine learning against it. You can see the CTO doing the work here. A few days later he had figured out Google's algorithm, reflected it in the company's website family, and traffic increased back to nearly the pre-change norm. If he had taken the Hadoop path, he would have worked for months, spent huge amounts of money, and found that the cluster and Hadoop thing would have basically been irrelevant to the problem.

Here are a couple things to consider:

• Hadoop, by definition, spreads its computing over the many machines available to it in the cluster, using HDFS (the Hadoop file system) for reading and writing data.
  • It is literally thousands of times faster to get data from local memory than it is to get it from a disk-based file system. The fewer file reads and writes needed to perform a computation, the faster it will be. Hadoop doesn't care.
    
  • Using more computers in a cluster means that there will be more I/O than using fewer. Many important calculations can be performed on a single properly-configured machine!
• MapReduce, the key processing engine of Hadoop, is one of those cool-sounding ideas whose job can be done perfectly well with normal code, which can do way more, vastly more efficiently.
• Why would anyone consider such an insanely wasteful approach? Once you know the origins, it makes sense.
  • If you're a big search engine company, you have to have loads of servers, enough to hold all the data and handle all the search queries at peak traffic times.
  • As is typical in situations like this, loads of servers will be under-used a large fraction of the day. Why not write some code that sucks up these "free" cycles and put them to work? Why not build a framework so you can just specify what you want done, without worrying about what resources from what machine where is used? Who cares if it's inefficient? It gets stuff done with the computers I already have. Brilliant!
  • Now it makes sense that Hadoop started and grew at Yahoo, copying some ideas about a narrowly-applicable (MapReduce) system and framework built at Google.
  • Except that at Yahoo, they somehow decided to make the Hadoop machines dedicated! Last I heard, they were up to, get ready ... 40,000 servers. Wow.

• With such an investment in getting value out of Big Data, Yahoo must be booming, just sky-rocketing with all the juice that has come out of the investment. Not. Why would anyone want to use an expensive, strange tool that generated no value for its originator? One word: fashion.
• Yes, there are some narrow situations in which Hadoop might be applicable. But in the vast majority of cases, you'll spend too much time getting way too many computers to do too little processing on not all that much data, and taking way too much time to get it done.

Conclusion

There is no doubt -- none! -- that you should use advanced software techniques in your business, because it will give you a competitive edge over everyone else.

The trouble is telling the difference between (1) value-adding advanced software techniques and (2) hype and software fashion. Even most software people have trouble telling the difference! In fact, software people who insist that there is a difference between value-adding software techniques and the latest thing that everyone is talking about run the serious risk of being marginalized, and categorized as being old farts who are unable or unwilling to do the work to learn the new methods.

In sharp contrast to the general thinking, software is a pre-scientific, fashion-driven field that resists holding new ideas to reasonable standards of proof and evidence. This makes it tough for business executives to know what to do. There is only one approach that works: roll up your sleeves, put ego and pride to the side, and figure it out using evidence and common sense.

Here's what we can learn from the shift to smart credit card terminals

I’ve been involved in computer software for decades. Lots has changed over that time. One thing that hasn’t changed is the question people most like to ask me. It’s this: “What do you see that’s new and interesting?” It’s a perfectly reasonable question, though one for which I rarely have a ready-made answer.

A question I never hear goes something like this: “What do you see that’s touted as the newest new thing, but is mostly old stuff, and was completely predictable?” Now that’s an interesting question. And the un-helpful but honest answer is “Practically everything that’s touted as a new thing is mostly old stuff, with a little bit of ‘obvious next step’ thrown in for variety.”

Still, there are some unpredictable aspects of the fancy new things: it’s really hard to know WHEN the new thing will happen and WHO will make it happen.

A case in point is … [I’m not sorry about the pun] … the new smart card terminal company Poynt. (Disclosure: my VC fund, Oak HC/FT, is an investor.) I can see eager marketing people at Poynt are raising their hands in the back at this … ahem … point, all anxious to point (groan…) out that Poynt is a pioneer in the market, arguably the inventor of the smart terminal, an amazing device that not only takes card payments, but also rings up items just like a POS terminal and hosts endless numbers of third-party apps. True! I happily concede the point. But I hasten to point out that there are robust competitors in the market, notably including Square and Clover.

The smart terminal is a new thing, and the market is glad to have it, but it’s hardly a NEW new thing, or something where you’d knock your head and say “now who’d-a thought-a that!?”

It’s natural for consumers of technology to look at the new devices and appreciate them for what they are. That’s like being a tourist, driving on a road through the country-side, appreciating the nice new views. That’s nice for the tourist, but are there patterns here, patterns that would enable an educated person to expect something like a smart terminal to appear, and womdering when it would happen?

Yes there are. The main pattern at work here is the rate of change of the underlying hardware. Today’s hardware is something like 1,000 times faster than much larger, more expensive hardware at the turn of the last century, less than 20 years ago. That number may not seem like much, but think of this: the average human walking speed is about 3 mph. The speed of a commercial jet while flying is less than 600 mph, about 200 times faster. Now imagine a human evolving so quickly that the human could walk at the speed of a jet -- and increase in computer speed is 5 times greater than that, in less than 20 years!

What’s the point? Or Poynt? Here it is: there are underlying geology-like forces in the world of computing that make it highly likely that something very much like Poynt would be invented – though as I said, predicting who will do it and when they’ll do it is a whole other thing.

The first step in creating a technology solution to a problem is often building problem-specific hardware. Then the technology evolves, getting faster, cheaper and more capable. Then there’s a tipping point, at which the purpose-specific hardware is replaced by general-purpose hardware, and most of the specific features of the device are implemented in sofware that runs on the general-purpose hardware. Then a new era begins. The general pattern is that special-purpose devices are supplanted by general-purpose ones.

In the case of card processing technology, first we see imprints of cards made on paper, with the physical paper being sent to a central place for processing. Then the big jump to computer technology and networking: a series of increasingly-better charge terminals, specifically made for processing card charges. The terminals evolved from dial-up networking to the internet, and from stand-alone to connected to a point-of-sale system. Wonderful devices!

Now think about cell phones. If you’ve been around for a little while, you remember big phones getting better and smaller and finally evolving into flip phones. Great phones, … but they’re phones. Then came the big shift, to a next generation of phones that were really small, portable, general-purpose cmputers that can run a myriad of applications … with cell phone hardware and software built in. Yes, it’s a phone. But it runs Facebook, email, and any of thousands of applications avialable in the app store. It’s a “smart phone!”

You know this. The reason I’m reminding you of that history is that it’s exactly the transition that card-charging “terminals” are going through right now – as they become “smart terminals,” i.e., small, portable, general-purpose computers that can run a myriad of applications … with card charging hardware and software built in. Yes, it’s a terminal, but a smart one.

How often do you see “I’m just a phone” devices? Flip phones? Yup! The old card-charge terminal will become just as rare a sight in the next couple of years.

So are the new “smart terminals” new? Yes! But hardly unexpected, at least to those who see the clearly repeating patterns of the underlying technology.

A less Poynted version of this post was previously published at Forbes.

Computer Security Breach Response Excellence

Here's what the experts do for computer security:

• Hire security experts to implement best-in-class security.
• Follow all the regulations.
• Pass all the audits.
• Spend lots of money.

Then, of course, you get breached, because in spite of doing the above, you have no idea what you're doing...

Here's how you respond:

• Get more experts to find what happened.
• Establish a carefully-thought-out strategy to recover from the breach and minimize damage to your reputation.
• Alert the public and your users about the event and your concerned, respectful response.

Then, of course, you change your website, put lots of money into attractive graphics, while making it hard for users to login or reset their passwords.

The share-your-expertise website Quora is surely in the running for best-in class when it comes to computer security; they have followed the above plan with true excellence.

The Quora Story

I got this email from Quora, of which I'm an occasional user, on December 3, 2018:

Dear David B. Black, We are writing to let you know that we recently discovered that some user data was compromised as a result of unauthorized access to our systems by a malicious third party. We are very sorry for any concern or inconvenience this may cause. We are working rapidly to investigate the situation further and take the appropriate steps to prevent such incidents in the future. What Happened On Friday we discovered that some user data was compromised by a third party who gained unauthorized access to our systems. We're still investigating the precise causes and in addition to the work being conducted by our internal security teams, we have retained a leading digital forensics and security firm to assist us. We have also notified law enforcement officials. While the investigation is still ongoing, we have already taken steps to contain the incident, and our efforts to protect our users and prevent this type of incident from happening in the future are our top priority as a company. What information was involved The following information of yours may have been compromised: Account and user information, e.g. name, email, IP, user ID, encrypted password, user account settings, personalization data Public actions and content including drafts, e.g. questions, answers, comments, blog posts, upvotes Data imported from linked networks when authorized by you, e.g. contacts, demographic information, interests, access tokens (now invalidated) Non-public actions, e.g. answer requests, downvotes, thanks Questions and answers that were written anonymously are not affected by this breach as we do not store the identities of people who post anonymous content. What we are doing While our investigation continues, we're taking additional steps to improve our security: We’re in the process of notifying users whose data has been compromised. Out of an abundance of caution, we are logging out all Quora users who may have been affected, and, if they use a password as their authentication method, we are invalidating their passwords. We believe we’ve identified the root cause and taken steps to address the issue, although our investigation is ongoing and we’ll continue to make security improvements. We will continue to work both internally and with our outside experts to gain a full understanding of what happened and take any further action as needed. What you can do We’ve included more detailed information about more specific questions you may have in our help center, which you can find here. While the passwords were encrypted (hashed with a salt that varies for each user), it is generally a best practice not to reuse the same password across multiple services, and we recommend that people change their passwords if they are doing so. Conclusion It is our responsibility to make sure things like this don’t happen, and we failed to meet that responsibility. We recognize that in order to maintain user trust, we need to work very hard to make sure this does not happen again. There’s little hope of sharing and growing the world’s knowledge if those doing so cannot feel safe and secure, and cannot trust that their information will remain private. We are continuing to work very hard to remedy the situation, and we hope over time to prove that we are worthy of your trust. The Quora Team

What a bunch of careful, responsible people, those folks at Quora are! So appropriate for a share-your-expertise site!

After this notice, I kept getting the occasional teaser email from Quora, tempting me to click and answer a question or see an answer someone else gave. For example I got this one a couple weeks before the breach:

I know, it's not click-bait for the general public, but definitely a good one for me.

Yesterday I got the first teaser I'd gotten since the breach email reproduced above. Here's the lead:

Not a killer issue, but I clicked out of mild curiosity about the answer, and also to see whether Quora was up and running normally. What I got was a lesson in how to respond to a security breach by driving your customers off. It's true, after all, that if there aren't any users, there won't be any meaningful security breaches -- problem solved!!

Here's the landing page -- a new thing in itself, because clicking on an email used to be enough to identify you.

The cute graphics are all new. I put in my password and got the box in red above, telling me I had to reset the password by responding to the email they sent. OK.

I got a typical password reset email:

I clicked on the link. I got to see even more wonderful new graphics! These guys are really trying! Then I put in my old password, because I wanted to; it's my password, I should be able to pick any one I want, unless they tell me there are rules.

Can't use my old password, huh? If you're so sensitive and caring, you could just possibly have warned me about that up front. Oh well. Here's a new one:

I put it in. It's new. They match. I click on the Reset Password button. Nothing. I change the password and click again. Nothing. Again. Nothing again.

They just don't want me, it's clear. If I were a normal user, it would have been game over. But I'm not, so I went back to the password reset email and clicked again. This time I put in a brand-new password. Then, clicking worked -- it got me to the login page, where I had to enter my email and new password yet again.

Quora has a big, fat, ugly, super-obvious, BUG in their "we're taking responsibility for this breach and hoping to win back the trust of our users" new entry door to their site, not bothering to perform super-elementary QA on one of the main pathways of the new code. Not some obscure condition. Software QA 1.01.

So just who are these geniuses at Quora? Are they the super-smart, rich, cool kids that have such a track record of excellence at other tech sites? Like Facebook and Twitter and the rest? It takes a bit of looking, but the simple answer is: yes. Super-smart. Beyond cool. Rich. And still can't get the most elementary details right!

Business as usual in software. Whether it's government, big corporation or cool young hip tech company the story is the same: getting stuff to actually, you know, old-fashioned WORK is beneath, beyond, above or whatever for whoever's involved. Not to mention make software that protects customer data.

 

Software Planning is Impossible

What else is new? Everyone knows that software planning is, well, just impossible. Live with it!

I mean something else entirely. I mean that software planning, in the usual sense of the word, is literally impossible to do. Planning for a new house? OK. Planning a new road or intersection? You may not like the cost, disruption or time, but it can be done. "Planning" in the sense that everyone means it and uses it for everything else, is literally impossible for software.

Here's why, in a nutshell: a "software plan" that is the exact equivalent of an architectural plan for a building, including the materials list is the software itself. Anything less is a vague sketch of what part of the building might look like, something a builder asked to quote on it would say, "I'll be glad to give you a quote when you give me a plan. Which this is not."

Let me explain.

Building plans

Most people understand the basics of planning a building, say a house.

You start by answering some very basic questions, like How Big, and Where. This location and size information determine much of the eventual cost of the building.

You then go with architecture, i.e., the guts of building planning. Starting with size and place, you then go into the general shape and style of the house, things like How Many of What Kind of rooms and finishes. You may interact with the architect, creating several rounds of plans. As you agree to one level of generality, you dig deeper, ending with things like colors, appliances, and materials (wood clapboard vs. Hardie board, etc.). If the architect is modern, he'll show you 3-D renderings and give you 3-D walk-throughs of fully furnished rooms.

Here's an overview that I built using common software, two snapshots from different directions:

Notice the landscape, the terrain, the bushes, the shadows -- these are just snapshots from a complete 360 degree view. You can pick the angle, distance, height and even the position of the sun. Those are images of real clapboards, doors and windows that are commercially available, everything.

Here's part of a floor plan I built with the same software:

 

And here's a variation of that plan with a bunch of furniture added and some other changes:

Here's a sample 3D interior view from the software's website:

The software enables you to not just have pictures, but to do a full 3-D live walk-through, just like in a video game.

If the architect uses modern software, the software will not only assure that the building is structurally sound, the software will fill in all the structural elements, including electrical and mechanical, and produce an item-level materials list.

That's the plan! You can give it to builders for time and money quotes, and to the local building department for permits. Once you make an agreement with the builder, off starts the construction project. Hopefully it finishes about when promised, and you end up paying the agreed-upon amount. Done!

When it's done, unless the builder has screwed up, it will look in physical life just like the drawings and renderings, with all the selected materials.

Software planning

Software planning is basically the same as building planning, right? Except even better! With buildings, you're digging holes, pouring concrete, putting up framing, and all sorts of time-consuming, physical things that are costly to buy and install. By contrast, software is just writing code, a very non-physical activity -- any change you want or mistake you make is easily fixed, without ripping out building materials or breaking up concrete. Moreover, building planning is like old-style "waterfall" project management. Agile, the modern trend in software, isn't possible with buildings.

Why is it, then, that everyone with practical experience knows that software planning is a mess, a total nightmare, pretty much no matter what planning regime you follow? Why do you think the industry keeps coming up with new ways of planning things? For building planning, the methods were always effective, and with the increased use of software, the methods have stayed the same, but the ability to render the results to clients and automate error-prone architecture work has gotten better and better. For software planning, we experience a ever-ending sequence of planning "revolutions," each of which promises to eliminate the perennial problems. Except the benefits are always in the future, and the future somehow never happens. What's going on here?

At root, the cause of the problem is simple: the analogy of building planning and software planning doesn't work! But we keep trying.

Here's why: a plan for a building at the level of detail I described above is truly comprehensive. The ability to render detail in 3D makes that clear, and the ability to generate the structural elements and item-level material list makes it crystal clear. While it's incredibly comprehensive for a building plan, no software plan comes close to that level of detail in terms of building software!

The reason is simple:

A building (or road or bridge) is passive, built by workers and machines from a carefully assembled inventory of passive parts and materials, things like lumber, wiring, pipes, siding, concrete, etc. A program is active, a carefully organized set of instructions and data (mostly instructions, i.e., software) that do things in response to active inputs and stored data, as a result of which stored data is added or altered and new data and/or actions are created in the outside world (e.g., new screen displays, messages sent to distant servers).

This difference is a BIG DEAL. It's EVERYTHING!

The Building metaphor for software is bogus

Think back to the beautiful building plan I showed above, complete with its ability to enable a person to do a virtual walk-through of the building. That's like a software plan, right? It's kind of like what people call wire frames, mock-ups of the UI, right?

Now imagine that the software plan we're building is for a combat-oriented, multi-player video game. How close is that ready-to-build physical building plan to a plan for the video game? Ahhh, maybe 5%? The easiest 5%, for sure.

The detailed building plan we can walk through is like the "world" that video games create. Except the world is always changing. No building plan can deal with a player racing through the building, crashing through a window and firing a blast that blows a hole in a wall -- a hole that is customized to the location and angle of the shooter, the type of wall and the kind of blaster used. The wall explosion could further impact another game player on either side of the wall, in real time. This is the hard part, and where the vast majority of effort of building a video game goes. It's the action part.

Is building commercial transaction software like building a video game? No, of course not. There are some elements of video games that are unique. In other ways, building commercial software is harder than building video games! Video games are completely self-contained "worlds." For better or worse, commercial software is very much part of an existing, amazingly extensive world -- of other software! Existing software that is always changing, has bugs, is extremely elaborate, and whose actions depend not only on what you ask of it, but what's been asked in the past -- context. Commercial software is a complex series of actions, many of which involve interactions with other software that is no less complex.

What's important in software, and takes the VAST majority of the work, is the action part. A building is passive. It just sits there. Even the "passive" parts of software are created, on the fly, by the action part. For example, when you first go to a web page, it might be fairly passive. But the second you click on something, action starts. Once you login, the action gets fierce, and is created, on the fly, just for you: with data that's been stored for you and things on the screen that are customized for you.

This is the difference between walking into one of those Amazon retail book stores, which are what they are, and going to the Amazon website as a logged-in visitor. In the store, personalization is performed by a worker who walks up to you, who of course doesn't know your complete history of interactions with Amazon, which the software does and reflects in how it interacts with you. This goes all the way down to simple things; for example, if I go to the web page of a book I bought years ago but forgot, the software helpfully tells me I already own the book, would I like to get it again?

Conclusion

As usual, metaphors lead us astray -- even the universally-accepted metaphors for software. This is really sad, and the fact that it's gone on for decades is even sadder, with no signs of changing for the better. I've gone into the sources of the bogus building planning metaphor in great detail in my book on Software Project management, along with more detail on why the metaphor doesn't work.

All this history and logic is besides the point, in a way: the important thing is that, even today, the bogus building metaphor for software enjoys near-universal acceptance, in spite of its persisting monumental failures. Instead, the "leading minds" of the industry yammer on about irrelevancies like Agile and Scrum that may make people feel better, but don't change the underlying problem.

You wish for a better world? Be one of the renegades who actually gets stuff done; try Wartime software. Here's the background.

Patient Incentives in Healthcare: Case Study

We all know that incentives work. That's why you always read about the "low prices" and the "sale" about to start, or the "limited-time offer." They're incentives to buy this or that. A server at a restaurant is incented to provide good service to get a good tip, and a salesperson is incented to sell by getting a commission. What a good idea it must be to apply this idea to healthcare, right?

Maybe the idea of incentives applies to healthcare. I take no position on that subject. But I do know that when the sprawling bureaucracy of a health insurance company tries to apply the idea, it turns into yet another costly bit of overhead that yields no benefits beyond allowing top executives who float blissfully above the facts and reality to claim that they're modern and innovative. Right.

Incentives in business

A business has strong incentives -- to get incentives right! A giant commercial or government bureaucracy has NO incentive to get them right. The business is aware that it's spending money to get customers, money that could be spent on an endless number of good things, from advertising to improving the product/service, to improving customer service to get more repeat customers, and on and on.

Here's the key thing: incentives are old news in businesses that have customers and need to make a profit. There is a long history of giving incentives, measuring how effective they are, and adjusting accordingly.

For example, in retail there were specialists who carefully controlled each season's products and set the incentives based on the experience of measuring the results. Already in the 1990's software products began to emerge that would take line-item POS (point-of-sale terminal) data from the last few years, and predict what would be the best time to start a sale, on which products, in which exact stores, and how deep the cuts should be. After side-by-side testing in multiple retail chains, the math-driven sales proved to be more effective than the ones generated by the best, most experienced people. So the industry transitioned to the algorithmic approach -- it's now malpractice if you don't use algorithmic sale incentives in retail.

Retail has gotten amazingly effective with incentives, following the classic path to excellence in AI/ML as I described in this series of posts.

Incentives by health insurers

Given this background, when the health insurance giants finally decided to apply incentives to their insured population, naturally they carefully studied incentives in other fields and adapted state-of-the-art techniques to their situation, right? WRONG!! In every case I've seen, they've done the dumbest things possible, not just starting from incentives 1.01, but screwing up so badly that any internal measurement system (which of course was NOT in place) should have resulted in the prompt cancellation of the project and demoting everyone concerned to a starting position opening mail. None of which happened, of course.

Incentive Case Study: Anthem

Anthem is an excellent health insurance company. It and its managers strive to be industry-best and provide great service to the employers and individual customers it serves. I am using Anthem for the case study here NOT because they're the worst -- far from it! I'm featuring them simply because I'm a customer, and so get a ground-level view of how things work there.

Sadly, as a giant, heavily-regulated bureaucracy, Anthem lumbers along and, along with its peers, gets really important things wrong. I wrote about the mess they made a couple years ago, first by allowing themselves to be hacked, and second by responding to the hack somewhat, ahem, ineptly.

Some time ago Anthem decided that Wellness and Health Incentives were something they should dive into. They now appear to be thoroughly committed to it, as they say loudly and clearly on their website:

Some of these programs may be truly wonderful. I make no comment on them. But I doubt that the one that was pushed onto me was exceptional, so let's dive in.

I decided it had been too long since I'd had a general health check, so I signed up for one and got it. After a while, I got a packet of stuff in the mail. Here's the top page:

Anthem clearly had gotten the claim for my visit and auto-enrolled me in their incentive program to get me to do the stuff they say, including getting such check-ups regularly. Wow, the incentive-program-babies who designed this program thought, we'll pay him some money for getting a check-up and maybe he'll do it again next year, hoping to get another card in the mail -- though of course we won't breathe a word about that.

Let's check out the rest of the package. Next page I get a wonderful, inspiring picture  of how exercise makes me healthy:

Next page, reality starts to hit. No more nice pictures and color. Just the facts, ma'am (the card itself was glued onto this page):

It's a gift card kind of thing! Except it's "pre-paid," and is accepted where Visa debit cards are accepted, but when you use it, you've got to lie and say it's "credit." Hmmm. And NOWHERE does it say how much money is on the card! I have to call or go online to find out. And of course the card doesn't say Anthem, it says SVM. Who are they?

Maybe I'll find out. Let's keep on. Next page:

Great. When I get a regular credit card, I call a number to make it live, and then away I go. For this one I have to read something and then check somehow what's on it. I wonder how much IS on it? I hope eventually they'll tell me. First I better read the rest.

Man, this print is getting mighty small! I wonder if I have to pass a test before I'll be allowed to use the card that may contain some secret amount of money that no one will tell me. Minus whatever fees and other stuff that they jam into it when they feel like.

Reading carefully, I find out that, even though it's a debit card, like for your bank, I can't just get the cash out of the card! What the &*&&$&*()$ is THAT about?? It's an incentive, darn it! A incentive that's a money incentive -- and I can't get the cash and spend it?? What is this, if I go to Burger King and order something they don't approve of the card won't work?? Who knows??!!

Reading more, I can't use it at an automated gas pump. Or at most restaurants. And there's a PIN, which I have to call to set -- in TINY print in the middle of the TINY PRINT page? I thought I was supposed to select CREDIT, and credit cards don't have PIN's. What's going on?

Maybe the next page will help:

Or maybe it won't. Or the next 3 after that, which are more of the same, and which I refuse to read.

Finally, last page, bigger print:

This appears to be the cheat sheet. This I can read. It appears I really do have to go online and enter a bunch of stuff, and remember all the conditions on using it, including the ones they don't repeat on this sheet. Maybe I can even find out how much money they're giving me!

If I were a regular person with some mild interest in whatever this incentive is, I would have dropped out by now. But I'm a fanatic and want to see how this story ends, so I'm going on to the next step. On-line we go! First step:

 

Get my card, copy the numbers in. Next step:

 

 

 

Put some more numbers in. Next step:

 

 

I appear to be in, but not really. I have to go back, enter the security code again and also enter the hard-to-read code they put in to stop robots. Wow -- all I can think is, this incentive must be huge. Why else would they be making it so hard??

Finally, I get logged in:

 

I think I've said "wow" a few too many times, but I should have held off for now. Tucked away in the upper right corner is the size of the golden goose I've spent all this time and effort seeking: $50! Maybe. Sort of. Except not in cash. And minus whatever fees. And not at restaurants or gas stations unless you follow the rules. And maybe there's a PIN, check the amount before each time you use it because you might have been dinged a fee, and...

And before I can touch any of it anywhere, I've got lots more information to enter. I'm outa here!

Conclusion

This incentive card program is one of the more bone-headed, dysfunctional things I've encountered in a while. Lots of lawyers, bureaucrats, managers and even publicity/image people contributed to it, but did anyone with, you know, real knowledge of how things like this work ever get a shot at it? The number of steps to get rewarded and all the uncertainty and conditions are guaranteed to produce maximum drop-out. In the reward card business this is called "breakage," and unethical rewards people try to maximize breakage; Anthem should be up for "rookie of the year" in the breakage stat.

It's one thing to try to create good behavior by giving an incentive. It's another to dangle the promise of an incentive, trick people into going through a horrific maze that most won't make it through, to get the incentive (of unknown value!), with the primary result that your basic impression of the health insurance company as incompetent and wasteful and something you should ignore whenever possible is strengthened.

Is anyone out there listening?

What are Software Fashions?

“Fashion” is a word we associate with clothes. Software is hard, it’s objective, it’s taught in schools as “computer science.” Software can’t have anything to do with “fashion” if it’s a “science,” can it?

Sadly, software is infected by fashion trends and styles at least as much as clothes. Fashion has a huge impact on how software is built. Understanding this, along with other key concepts like those involved in Wartime Software, can contribute greatly to building great software that powers a business to great success.

Fashion

We all know what fashion is, exemplified by fashion shows like this one:

 with impossibly thin female models strutting down cat walks wearing clothes that no one is likely to wear in real life.

Not as often, but guys too:

Far more important than models wearing extreme clothes is everyday fashion. I grew up looking at men dressed like this:

It's what men wore to church and to aptly-named "white-collar" businesses all the time. But there's nothing special about a suit and tie. Here's a look at Dutch fashion in the early 1600's:

Fashion is arbitrary! it's just what people wear. Everyone judges you by what you or don't wear, according to prevailing fashions.

Of course, "fashion" goes way beyond how people dress. It's how you act, how you speak, the accent you use, the interests you express, just about everything. If you don't think it matters, just trying wearing NY Yankee regalia into a South Boston sports bar and start spouting trash about the Red Sox. Lots of people who think they're the kind of people who are "above" fashion are driven by it nonetheless -- just look how they respond to people walking into a room, and if there's any doubt, seconds after the newcomer opens his mouth.

Fashion is about people. It's about belonging, status, fitting in or "making a statement." We live in a fashion-dominated world, like it or not.

Fashion in Software

I was one of those people who was convinced I was "above" fashion. Being above it made me superior, in my mind, to those who were slaves to it. During and beyond college, I bought the few clothes I wore from a local used clothing store and wore hiking boots most of the time. Once when I was in my first post-college programming job, I was called out of the cubical where I spent most of my time, heads-down, programming away, and asked to come into a meeting in the front of the building. I walked in to a meeting populated entirely by men in suits. One of the men I didn't know glanced at me and immediately exclaimed "finally! We get to talk with someone who knows things!"

I had been called into a sales meeting, and one of the visitors had software questions no one knew the answer to, so "the suits" had called in the guy who knew the answers. How I dressed and acted in fact made a statement to the visitors -- I dressed the way a programmer dressed, the kind of programmer who wanted to program, not one who aspired to management. So, like it or not, I was making a "fashion statement," while fooling myself thinking that I was "above" fashion. The hard fact is, no one is "above" fashion. The way we dress and act and talk, the choices we make, says loads about us. Those unavoidable choices clearly establish our place in various groups, social and status hierarchies.

Software Fashions

Given how fashion-driven our lives are, it would be shocking if programmers weren't fashion-driven in their shared activity of software. In fact, they are! Sadly, the vast majority don't think their choices are fashion-driven. They believe they're modern, with-it software professionals who are using the proven, advanced methods for doing software. The trouble is, few of them take the trouble to cast a knowledgeable, cold, hard eye on the arguments, experience and facts concerning their chosen methods and tools. They've made their choices so they can be with whatever software social group they identify with and/or aspire to. It's all about relationships and status. If you're ambitious, you may want to be with the "cool kids," members of an elect social group, yes, in software. And it works! If it didn't "work" (elevate their software group status), they wouldn't do it.

We like to think of people who wear fashion-forward clothes as being empty-headed, shallow people. Surely, programmers aren't that! But to the extent that they adapt fashion-forward software, that's exactly what they're doing -- only worse! They're lying to themselves, deceiving others, and making believe they're pushing some trendy software thing because it's advanced technology, yielding results superior to the obsolete stuff that used to be the standard.

Just like with clothes, software fashions evolve. Fads start and may become hot. The fad may evolve into a fashion as it spreads, with people who haven't adopted it taking notice. The fashion may further evolve into standard practice, with eyebrows being raised for anyone who dares to question it. More often, the fashion simply fades away.

It's rare for any fad or fashion to be explicitly repudiated -- oh, that was a terrible idea, people are turning away from it for good reason and here's why. No one says that! In the "advanced clothing" area, everyone knows that fashion is "just fashion." In software, fashions aren't considered "fashions;" they are considered "advances," emerging modern techniques that are objectively better than what came before, like a new drug or operation that has emerged from clinical trials and now saves lives that used to be lost! Saying that a widely adopted software fashion was never proven, was always a bad idea, but got widely used and promoted anyway would expose the game. So when software fashions die, they fade slowly away and simply stop getting used and talked about.

After a fashion fades away, it's generally forgotten by nearly everyone, usually except for a band of true believers. Some of the more intellectually heavy-weight fashions retreat to academia, where they live on, always with "exciting futures."

Some of these flourished-but-died fashions rise to live renewed lives. In one pattern, the fashion was so broadly accepted but such a failure (though rarely discussed as such) that when it becomes fashionable again, it has a new name. No one ever refers to last time, why the older fashion didn't work out, and why this slightly altered version of the same thing will. In another pattern, the fashion baldly re-emerges with exactly the same name, and nearly the same blazing-bright future as the last time. Sometimes there are even some successes. But it remains a fashion and therefore has disappointing results to anyone who cares to look, which is essentially no one -- such is the social power of fashion!

Not all fashions die. Some fashions have such powerful support that they become locked in as part of modern mainstream practice, sometimes even becoming part of so-called Computer Science, or at least IT Management. I don't fully understand how and why this happens, but I know that in part, the fashions that become standards address some widely felt need in the people involved in software. When this happens, there is often a series of waves of renewal or reform -- while the reformers refuse to acknowledge fundamental problems with the fashion-enshrined-as-best-practice, they latch onto some minor tweak or addition and promote it, usually with a new name, as the best way to get results with standard-practice X.

What are these software fashions exactly?

I have already talked about a few important toxic software fashions. I have gone into huge detail for a couple of them, with multiple blog posts and even books. I'm gradually starting to understand this bizarre phenomenon in terms of powerful social fashions with bad results, masquerading as "advances." I'm seeing the resistance to seeing the Emperor's New Clothes for what they are because of the self-delusional conception of software as a science/math-based STEM field, rather than as the pre-scientific collective group-think that it largely is.

I have already challenged a couple of the modern hot fashions, for example my series of posts on AI/ML -- a classic example of a once-hot fashion that has died away and been re-born multiple times. This particular fashion is distinguished from some of the others because there are some truly excellent algorithms at the heart of it that can be applied to great benefit -- and this has been true for decades! But because of widespread fashion-itus, the money and effort spent on them is mostly wasted.

In future posts and at least one future book (in process), I will continue to dive into and expose specific software fashions for what they are. I do this in part to strengthen the resolve of those special people and groups (some of them ones we've invested in) with the understanding that the "wrong" or "uncool" things they're doing give them a fundamental business/technical advantage, and they should stick to their guns and ride their truly effective methods to success, to the benefit of all concerned.

Further reading:

Resistance to treating scurvy compared to software disease treatments.

The modern AI/ML fashion.

The story of how I discovered the fashion vs. what-really-works issue.

Deconstructing project management.

The story of fashions-becoming-standard-practice.

The Cloud fashion.

Big Data fashion. Big Data bubble.

Evidence-based software methods don't exist.

The recurring fashion of data definition location.

 

Medicine as a Business: Medical Testing 5: The Results

I've gone through quite a bit to get the results of my MRI. See here for the previous installment, and here for the start of the saga. I glanced at the report and it looked good. In this post, I'll describe the unsettling things I found when digging deeper. In sum: the whole baroque nightmare of scheduling, performing and delivering results of medical tests is not only inefficient and riddled with needless high cost and waste, but more important there are serious quality problems leading not just to delay and waste, but bad results.

I fully acknowledge that what has happened to me pales beside the waste, incompetence and fraud that pervades the worst medical systems. My point is that, even in the best healthcare systems, bad things are happening.

The Results

I glanced at my hard-won test results and felt OK, mostly because I had been told that radiation therapy took a looooong time to show results, and I shouldn't expect anything to change at the first MRI. There was no OMG or IT'S GROWN LIKE CRAZY in the notes when I glanced at them, so I let it rest. I was tired out from all the effort of getting the darn thing.

Then I looked more carefully. I'm still OK for my personal case, but on close examination, I realized that the final report of the MRI was consistent with the crazy things that led up to it: scheduling the test, taking it, and getting the results. It's all part of a bizarre system that has GLARING flaws that seem like they should be easily fixed, but nothing much happens.

Here is the key paragraph from the earlier of the scans, the one that led to the radiation treatment:

Here is the corresponding paragraph from the second scan, the one I struggled to get:

The first thing that jumped out at me was the simple observation that there are no standards! Radiologists are incredibly smart, well-educated people. College degrees. Super scores on GMAT tests. Degrees from incredibly-hard-to-get-into medical schools that have TINY numbers of students. Then more years getting further training to become medical imaging specialists -- usually five more years, on top of the four years of college and the four of medical school! 

If I showed you the whole report, you'd immediately see that even the paragraph and subject-matter organization was different. About the most glaring thing to me was that the second report gave actual dimensions of the tumor, while the first did not! Don't you think that when tumors were involved, specifying the actual size would be the standard?

There's lots more that could be said, but I'll leave it with these simple observations:

• There is no system in place to record and assure that the required location is being imaged. The key thing can be missed because it wasn't imaged.
• There is no consistency of exactly what is reported on and how it is reported. It is difficult to compare reports and assure that what you need is there.
• For tumors, there is no consistent positioning and measurement of size. You could miss a tumor altogether, and easily miss size/location changes.

Conclusion

I'm in remarkably good shape, having a scary diagnosis of an extremely rare cancer. I received great treatment from highly skilled professionals at every step of the way. I received chemo that had only 25% chance of working, but it shrank a rapidly growing tumor. Then, when it started growing again, I got right into radiation, which has at least prevented further growth, and should finally stamp it out. I have nothing to complain about, and a great deal to be thankful for, including all the professionals who treated me. 

I've written this series of posts about Medical Testing NOT as an indictment of the individuals who have treated me, but as a serious indictment of the system in which they work. Here is a summary list of the things that "could be improved;" details are in the prior posts of this series:

• Scheduling and getting a pre-auth for a test can be a labyrinth and delay-filled nightmare.
• There are multiple issues with the wasteful, expensive and time-consuming blood test.
• There are multiple issues with specifying and following an exact procedure for the location and mechanism of the scan itself. Instead of being in the system, the nurse has to get information from the patient and guess about other things!
• The equipment and software is built in a regulation-protected bubble, which results in 10X or greater cost and trailing-edge technology.
• Getting results that have already been created by the radiologist can be an obstacle-filled maze, even if you try to use a “patient portal” that is supposed to make things to easy and transparent.
• The patient portal is mostly a sales pitch about how the hospital is wonderful – getting the information is a big problem, and then important information is wrong or just plain not there.
• Finally, the results produced by super-highly-trained doctors based on these expensive and questionable inputs don’t meet any modern standard for content.

Why doesn't anyone in management seem to care? I've often wondered this, and speculated about why several times. What's clear is that there's a hierarchy of prestige in every society, including ours, and that the top of the hierarchy is populated by people who focus on strategy, policy, direction and messaging. They are, for the most part, "above" getting "lost in the weeds." Sorry guys; the action is on the ground, where real things happen to real people. That's where you discover what's wrong, and when you "fix" something, that's where it's got to change.

Medicine as a Business: Medical Testing 4: Getting the Results 2

I've done everything I can to use the Mount Sinai patient access portal to access my test results, without result. (See here for the start of this saga, and here for the previous post.)

Now it's time for desperate measures. I finally take the radical step of picking up the phone and calling for help. Surely the results are there!

Here's what happened.

• I called.
• I was put on hold.
• I explained the situation.
• I was put on hold while the CSR checked something out.
• More questions. More holding. Rinse and repeat several times.
• Hold while I check with my supervisor.
• Rinse and repeat several times.
• Final result: we can't help you, call your doctor and have them help.
• But what can they do that you, the specialist can't??
• They have a number they can call to get help.
  

More than half an hour on the phone, and I get to ask my doctor to call someone who won't be able to help either. And I'm sure my doctor would jump at the chance to fix this problem, since he looooooves the EMR so much!!

Desperate and out of options, I call the doctor's office.

• I got transferred to a 5 minute wait before getting a dial tone.
• I got transferred to voice mail.
• I got transferred to nowhere again.
• Again.

Finally, someone picked up whose voice I recognized -- the office receptionist. I explain the problem, and he tells me that the Mount Sinai Radiation Center uses a different EMR than the rest of Mount Sinai!! Apparently one that doesn't send patient data to MyChart.

He promises to get me into the Radiation Center EMR patient portal AND send me the results. "What's your fax number?" he asks. "Umm, can you send it by email?" Pause... "Sure, I can figure out how to do that. What's your email?" I gave him the information, and five minutes later, I got an email with a PDF document attached. The document had the test results and instructions on how to get into the patient portal.Thank you!

Problem solved! I read the report, and the news was good. The thing that had been growing had stopped growing. But self-sacrificing guy that I am, I didn't stop there. What would have happened had I not persisted in my calling, and connected with a helpful and knowledgeable receptionist? After all, the report was supposed to be in the patient portal.

So I persisted. I decided to get into this special patient portal and finally see that the test results were actually posted there and available to me.

The Radiation-only EMR and patient portal

Leaving out all the details, I followed the procedure and after only a moderately odious amount of work (I had an access code!), I got into the portal:

 

Then I went to the test results, where my report should be:

It's not there, of course. Why am I not surprised?

The test results report should have been in My Mount Sinai Chart. It was not there, as confirmed by multiple levels of customer support people. It should also have been in the Radiation Oncology patient portal. It was not there, as you can see above. Given that an insider was able to access the report quickly and send it to me, the report was certainly in both EMR's. It was in the normal Mt. Sinai EMR, because that's where the doctor who wrote the report put it. It was also in the Radiation Oncology EMR, because that's the EMR of the doctor who requested the test -- and as I learned early in the process, it was easy for people in the radiation center to put orders into the "main" system.

Here's the key point:

Neither of the two EMR's at Mount Sinai that were involved with my test put a copy of it into the relevant patient portal so that I could see it. While I managed to avoid the usual doctor's appointment to find out the results, it's not clear how much time and frustration I saved in the end. Here's what was promised:

What was the reality?

• The test results were not available in MyChart. Is Mt. Sinai management unaware of this? Are they just lying and hoping to avoid embarrassment, as they do with other important "low-level" things? See this for a juicy example, and this for context. Either choice is unacceptable.
• The customer support service, when finally available, was unable to help.
• The original doctor's office was unavailable.
• The SURPRISE! special, different EMR used by my Mount Sinai department also didn't have the report.
• I only got the report because of repeated calling and a chance encounter with a kind receptionist.

Yeah, yeah. I'm computer and math guy, and I know statistics, and I know this is just one example. But can you really imagine that what I went through was a giant, almost-never-happens, tiny blip in a uniform fabric of excellence? Right. Wanna buy a bridge? I've got one real cheap for ya...

The E-mail!

Wait! There's more! After I drafted the saga of getting my greedy hands on the MRI results, something happened.

About a week later I got an email:

WHAT!!?? This test result was supposed to be on the radiation center's portal!

What's more, the only reason I got the email telling me the result was available on the Mount Sinai patient portal was because I was previously a patient and had signed up for it. If I had come into the Radiation Center directly, without having a history at the broader hospital system, I'd still be waiting.

On July 24 I'm told that the result was posted and available to me. A result from a test that was posted to Mount Sinai's system on July 3, 3 weeks earlier. It's a good thing we've got computers -- if it took 3 weeks to make a copy of a short document from one place in the Mount Sinai computer system and store the copy in another place in a related program in the same computer system, imagine how long it would have taken to do it manually! Years, probably!

I'm writing this on July 31, 2018, so by now the result surely will be posted on the patient portal for my doctor, nearly a month after the test was taken, right? Let's check:

Nothing is available. So much for the Radiation Center's patient portal.

Now I'm curious. Is the test really there, even though on the wrong portal? Here's the results list:

Yes, it's there, top of the list.

MyChart also provides a convenient to-do list, things I'm supposed to do, and there's something on the list. Better check it out, even though no one's told me there's something for me to do; this subject is important to me, to put it mildly, and I wouldn't want anything to slip through the cracks. Here's the to-do:

Oops. The MRI that was expected to be taken on June 11, actually taken on July 2 because of my initiative, and posted to the portal on July 24 is listed as a to-do item. The EMR evidently failed to connect the work order with the fact that the work ordered was performed and the results delivered.

This sounds benign, but it's actually scary. Deeply scary. The system doesn't match orders placed with results delivered, which means that orders could hang in space, ignored, with patient-essential work undone, unless a concerned and involved patient tracks it. In my case, there was a concerned and involved, not to mention detailed-oriented patient. What about the normal case? How many important things just hang out on a to-do list, undone, until they are "cleaned up?"

There's more trouble coming. When I glanced at the results, I got the impression things were OK. But when you dive in, ... see the next and final post in this series.