The Black Liszt

What Software Experts think about Blood-letting

Software experts do NOT think about blood-letting. But ALL medical doctors thought about blood-letting and considered it a standard and necessary part of medical practice until well into the 1800's. They continued to weaken and kill patients with this destructive "therapy," even as the evidence against it piled high.

The vast majority of software experts strongly resemble medical doctors from those earlier times. The evidence is overwhelming that the "cures" they promote make things worse, but since all the software doctors give nearly the same horrible advice, things continue.

Blood-letting

Blood-letting is now a thoroughly discredited practice. But it was standard, universally-accepted practice for thousands of years. Here is blood-letting on a Grecian urn:

Consider, for example, the death of George Washington, a healthy man of 68 when he died.

Washington rode his horse around his estate in freezing rain for 5 hours. He got a sore throat. The next day he rode again through snow to mark trees he wanted cut down. He woke early in the morning the next day, having trouble breathing and a sore throat. Leaving out the details, by the time of his death, after treatment by multiple doctors, about half the blood in his body had been purposely bled in attempt to "cure" him of his sickness!!! If he hadn't been sick before, losing half the blood in his body would have killed him.

If you are at an accident and you or someone else is bleeding badly, what do you do? You stop the bleeding, because if you don't, the person will bleed to death. That's now. Then? You bleed the sick person because it's the universally accepted CURE for a wide variety of sicknesses.

Bloodletting was first disproved by William Harvey in 1628. It had no effect. It remained the primary treatment for over 100 diseases. Leaches were a good way to keep the blood flowing. France imported over 40 million leaches a year for medicinal purposes in the 1830's, and England imported over 6 million leaches from France in the next decade.

While blood-letting faded in the rest of the 1800's, it was still practiced widely, and recommended in some medical textbooks in the early 1900's. We are reminded of it today by the poles on barber shops -- the red was for blood and the white for bandages; barbers were the surgeons who did the cutting prescribed by doctors.

Blood-letting in software

By any reasonable criteria, software is at the state medicine was in 1799, when everyone, all the experts, agreed that removing half the blood from George Washington's body was the best way to cure him.

If you think this is an extreme statement, you either don't have broad exposure to the facts on the ground or you haven't thought about what is taken to be "knowledge" in software compared to other fields.

I hope we all know and accept that the vast majority of what we learn and come to believe is based on authority and general acceptance. This is true in all walks of life. Of course not everyone believes the same thing -- there are different groups to which you may belong that have widely varying belief systems. But if you're somehow a member of a group, chances are very high that you accept most things that most members of that group believes.

This is no less true in science-based fields than others. The difficulty of changing widely-held beliefs in science has been deeply studied, and the resistance to change is strong. See for a start The Structure of Scientific Revolutions. I have described this resistance in medical-related subjects, and in particular showed how the history of scurvy parallels software development methods all too well.

But at least, to its great credit, medicine has gone through the painful transition to demanding facts, trials and real evidence to show that a method does what it's supposed to do, without awful side-effects. That's why we hear about evidence-based medicine, for example, while there is no such thing in software!

I hear from highly-qualified and experienced software CTO's that they are going to lead a transition of their code base so it conforms to some modern cool fashion. One of the strong trends this year has been the drive to convert a "monolithic code base" (presumed to be a bad thing) to a "micro-service-based architecture." When I ask "why" the initial response ranges from surprise to a blank stare -- they never get such a question! It's always smiling and nodding -- my, that CTO is with-it, no question about it.

Eventually I get the typical list of virtues, including things like "we've got a monolithic code base and have to do something about it" and "we've got to be more scalable," none of which solves problems for the company. When I press further, it becomes obvious that the CTO has ZERO evidence in favor of what will be a huge and consequential investment, and has never seriously considered the alternatives.

As is typical in cases like this, when you scan the web, you see all sorts of laudatory paeans to the micro-service thing, very little against it. Most important, you find not a shred of evidence! No double-blind experiments! No evidence of any kind! No science of any kind! What you also don't find is stories of places that have embarked on the micro-service journey and discovered by experience all the problems no one talks about, all the problems it's supposed to solve but doesn't, and the all-too-frequent declarations of success accompanied by a quiet wind-down of the effort and moving on to happier subjects. Because of my position working with many innovative companies, this is exactly the kind of thing I do hear about -- quietly.

Conclusion

We've got a long way to go in software. While software experts don't wear white coats, the way they dress, act and talk exudes the authority of 19th century doctors, dishing out impressive-sounding advice that is meekly accepted by the recipients as best practice. No one dares question the advice, and the few who demand explanations generally just accept the meaningless string of words that usually result -- empty of evidence of any kind. It's just as well; the evidence largely consists of "everyone does it, it's standard practice." And that's true!

Software experts don't think about blood-letting. But they regularly practice the modern equivalent of it in software, and have yet to make the painful but necessary transition to scientific, evidence-based practice.

 

Apple's Facetime Problems Illustrate What's Wrong with Blockchain

Apple’s had a rough time recently, with bugs, security problems and sales issues. The recent Facetime bug is particularly embarrassing. It’s made the news! There are stories about it all over. Apple is scrambling to fix the issue and end the pain and embarrassment, pronto.

Blockchain has also had a rough time, with recent cavernous losses that extend a years-long pattern. Blockchain enthusiasts march on, seemingly oblivious to the intractable problems that cripple their beloved technology. So far as anyone can tell, no one is scrambling to fix the problems.

Comparing the two situations is interesting and educational.

The Apple bug was first discovered by a teenager while he was setting up a Facetime group chat.

• Most blockchain problems are discovered sometime after a substantial loss has taken place, when you go to check your account and are shocked to find it has a whole lot less value than it had last time you checked.

The boy and his mom were frustrated by how hard it was to get through to anyone at Apple to report the bug. They were doing the right thing, while Apple was being a typical lumbering, unresponsive bureaucracy.

 

• You discover the loss in your crypto account. You’re upset. Who do you call? Where’s the 800 number for customer support? If you didn’t know it already, you quickly discover that there’s no number, no one to call. There’s no organization in charge at all! And in those cases where there sort of is, they refuse to fix the problem.

In less than a week, Apple officials woke up to the fact that they had a problem. A big, ugly, embarrassing one. To their credit, they did two things.

• In some of the most important cases, such as Bitcoin, there is literally no one in charge – that’s the whole point of Bitcoin – it’s a system that was designed to have no one in charge. It’s brilliant, as I describe here. But it’s also fatal when the system is hacked.
• In other cases, such as exchanges, there is someone in charge. But their typical response is to claim, with good justification, that “fixing” the problem would destroy the fabric of blockchain. And after all, in many important cases, the funds have long since been converted to cash and are long gone – how can that be “fixed” without catching the bad guys? And as I’ve described, while bad guys in normal banking are often caught, in the crypto-currency world, they almost never are. So your money is gone. Gone!

The first thing Apple did was shut down their servers so that group Facetime was no longer possible. This didn’t happen all at once, but rolled through the system pretty quickly.

• Immediate action to fix the problem in blockchain? Doesn’t happen.

The second thing Apple did was announce that the bug would be fixed and released in about a week.

• Here’s where it gets really bad for blockchain: where’s the bug that can be fixed to solve the underlying problem? No one can say! So far as all the blockchain “experts” are concerned, there is no problem to be fixed. The silence is deafening. When have you ever read about any kind of crypto-currency loss, after which someone “in charge” said something like: “the loss was due to [this bug}, which we’ve found and fixed and will be effective on [this date]”? Anyone?
• When the responsible problem is in a so-called smart contract, it’s even worse. Smart contracts are stored in the immutable blockchain itself, and so, unlike normal programs, can’t be fixed or changed in any way. The geniuses who invented and support smart contracts consider this fatal flaw to be one of their great advantages. Go figure.

The net effect of the Apple bug is that the privacy of certain Facetime users was compromised. Embarrassing. Bad. But your wallet is unaffected. But at least the news got out quickly, people can avoid the feature for a bit and then go back to using it, should they choose.

• The net effect of the myriad of Crypto and blockchain bugs and hacks is loss of the equivalent of real money, sometimes to the tune of tens of millions of dollars. The only way to avoid the problem is to get out of crypto. Totally. And never go back.

The contrast between the Apple Facetime bug and the various crypto/blockchain bugs and hacks couldn’t be more stark. With Apple, there’s someone in charge; the someone wakes up to the problem within days, and moves decisively to first block the problem and then fix it. In blockchain, there’s no one in charge (by design!); the affected people wake up to their problem, whine about their often substantial financial loss, but are largely ignored by the community of experts and operators, who soldier on, promoting the wonders of the amazing immutable distributed ledger technology.

Here’s my prediction: the blockchain mania will continue to spread for a while, but then will slowly fade away, with few of the promoters admitting their lapse in judgment. As with most technology fads, everyone’s attention will simply shift to something shinier and newer as the problems grow too large to be ignored. There's a long shot there will be some shining successes with blockchain -- but I predict that when you look under the covers, it won't really be blockchain doing most of the work.

(Disclosure: While I’ve read API’s and source code, I’ve never owned cryptocurrency of any kind, and don’t plan to any time soon.)

This post originally appeared on Forbes.

Crypto-currency Hacks and Losses Mount While Supporters Remain Silent

Insiders most like to describe Blockchain as Immutable Distributed Ledger technology. They love that it’s distributed, and a “ledger” rather than a database. But most of all, they seem to like that it’s “immutable.” To enthusiasts, this means that the unbreakable cryptography and other techno-nerd elements result in impregnable, hack-proof software. In a world filled with crappy software that’s thoroughly “mutable,” hackable, breakable and a smorgasbord of other criminal, consumer-hurting things, this is a wonderful thing. No wonder so many people and corporations are jumping on the Blockchain Bandwagon. The smell of FOMO (Fear Of Missing Out) fills the air.

The FOMO is really strong on Blockchain. So strong that it appears to prevent enthusiasts from paying attention to the fact that has been established over the last few years: Blockchain may indeed be Distributed and a Ledger (more on those in subsequent posts), but it’s hardly immutable. In fact, it’s just as hackable as any other piece of software – even more so because no one’s in charge of keeping it safe!

The latest loss is small by comparison to some of the earlier ones. The one announced on January 8, 2019 amounts to “just” $200,000 worth of ethereum classic. What’s worse is that the attack was at the core of  the blockchain. Apparently the attack was carried out by miners, the servers that are at the core of blockchain’s operations and security, the ones that perform the magic cryptography that supposedly prevents bad things from happening. The hack itself involved the absolutely worst thing that can happen to a crypto-currency – about 40,000 ETC was double-spent.

If this were the first loss, I would understand the blockchain folks minimizing its importance. It’s hardly the first loss. Who talks about the Mt. Gox hack, in which nearly half a BILLION dollars was lost? That happened about 5 years ago! Mt. Gox has been followed by an un-ending stream of other successful (for the criminals) attacks and losses. One of the more famous was the $50 million lost in the DAO hack. Less famous hacks resulting in losses over $10 million took place in 2018. The pattern of criminal success shows no signs of slowing down.

How can any sane person continue to back blockchain as a transforming technology due (in part) to its immutability and implied greater security in the face of this evidence? Obviously, what’s happening is that people are simply ignoring the evidence. That’s it!

Let’s put this in context. What would the stories be if anything comparable took place with plain old banks, with their supposedly obsolete software and security that’s primitive by comparison? While lots of normal banks are robbed every year, these are small-scale occurrences, and many of the perpetrators are caught. For example, here is the FBI’s latest news about bank robberies. You’ll see that there are loads of convictions and prison sentences. Here is a story from December 2018 about a man who robbed a local bank of $536 in order to pay his rent. He has just been sentenced to 46 months in prison.

Are there bank robberies in which large amounts are stolen? Check out the list of them in Wikipedia. The list goes back more than a century. The largest robberies don’t come close to the criminals of the blockchain world. The most recent one listed was 20 years ago, when the Bank of America was robbed of less than $2 million. Peanuts!

OK, you might say, but what about cyber-crime? There’s actually quite a bit of it. But digging into the exact nature of the crimes and how they’re committed is quite interesting. It may exist, but I couldn’t find ANY cases of the core bank systems being hacked. And NONE of the central database (the equivalent of blockchain)! In every case I’ve seen, it was plain old systems network hacking and/or criminal employees that were the problem. Yes, some large amounts were involved, for example in the Bangladesh robbery. But in that case as in many others, corrupted insiders were involved. It had nothing to do with the security of the banking software itself!

If normal banks had been hacked the way blockchain has been hacked, you’d find that the core banking system itself was breached, or the central database itself. In no case that I can find has this happened. What this means is that blockchain, in its short existence and with its relatively tiny fraction of money, has been hacked more successfully and more deeply than any normal banking software has been. “Immutable,” huh? Explain that again, please. But stick to the facts this time.

The conclusions we can draw from these facts are simple, clear, and hard to dispute:

• Blockchain is highly susceptible to being hacked in a wide variety of ways.
  • This has been demonstrated by events for more than 5 years; the hacks are on-going.
  • Large amounts of money are lost in the hacks.
  • The hackers aren't caught, much less punished.
• While there are lots of physical robberies of old-style banks, the amounts involved are small, the perpetrators are often caught, and consumers are not hurt.
• While there are hacks on old-style banks, they have had little success in US banks, and the same kind of cyber-security breaches that occur everywhere are involved.
  • In no case have the hacks been as deep in the software as many attacks on blockchain have been.

There's lots more that could be said about this, but here's the bottom line: if you want to keep your money safe, put it in a traditional bank, whose software systems are indeed immutable. Any blockchain storage is more susceptible to attack and loss than the software used by traditional banks.

This post originally appeared at Forbes.

The Novel Idea at the Heart of Bitcoin

There aren’t many true and surprising new things in software technology, in spite of all the gushing about new stuff. At the center of Bitcoin is a tech advance. Not a minor step forward. Not an enhancement fueled by faster chips. An amazing idea that is the engine that has fueled its explosive growth. It’s not something people talk much about, sadly. They should. The core of the idea is the miners that are the heart of the Bitcoin engine.

If you’ve heard anything about Bitcoin, you’ve probably heard that it’s a crypto-currency. You’ve heard it’s totally secure because lots of computing locks the data in an air-tight vault secured by the latest cryptology algorithms. You’ve heard that it’s a ledger of transactions, and that the ledger is distributed, which somehow makes it better. All these things that you’ve heard are correct – and it’s the miners at the heart of every one of those true things.

First, let’s step back a minute and understand the problem that Bitcoin solves. Bitcoin isn’t cool in the abstract – it’s cool because it’s a creative solution to a really hard problem. The problem, in a nutshell, is to create a currency, like US dollars or Euros, that isn’t controlled by any central authority. That’s a HARD problem. How can you have a currency that people accept with no one to issue it? If it’s somehow issued, who’s going to do the work of creating and managing it?

Long ago, currency consisted of fairly scarce, valuable objects, like sea shells. Then, precious metals were used. Since it was hard to judge how valuable a piece of metal was, people in authority created standard sizes, shapes and values – today’s coins. Then paper money was issued by early banks, with the precious metals in the banks backing it up. Central government authorities then replaced the banks, and currency became a per-country thing. Finally, it became detached from the coins, a.k.a. “the gold standard.” That’s where we are today, with huge government authorities issuing and controlling abstract and paper currency at will, manipulating it to meet political goals. The problem at the heart of Bitcoin is, how can you create an abstract currency that people can trust without a central authority of any kind, much less a government? I trust you’ll agree, that’s a truly hard problem.

Before diving into the solution to this problem, it’s worth understanding why it’s a problem. The core issue is the money supply, and how central authorities manipulate it to implement monetary policy in various ways, shifting with the political winds. Central bankers can print more money, take money out of circulation, raise and lower interest rates and do other things on a whim. Other branches of the government closely monitor who does what with their money via extensive, ever-growing, onerous regulations that make everything harder, slower and more expensive. How can we get out of this? How can we escape the armies of faceless bureaucrats who control the money and watch what we do with it?

The solution has to somehow make everything work with no one in charge. Get people to do lots of work and spend lots of normal money to create and maintain a robust system of virtual currency, and somehow get those people to be absolutely incorruptible?! They’ll be in charge, but not tempted even a little to use their power to enrich themselves? How is THAT supposed to happen??

That, my friends, is the genius of Bitcoin. The genius is embodied in the design of the miners.

Miners are volunteers. No one selects them – they just step up, get their hardware and software together, and start mining. All on their own – without permission and without even an invitation! Here’s the key part: when you mine, you make money, in the form of newly-issued Bitcoin. The formula and the rules are built into the software that everyone uses. When you mine, you make money. The more you mine, the more you make. If you’re ever tempted to think about fiddling with the software, cheating and just taking a bunch of money (Bitcoin), you immediately think of the huge investment you’ve made in mining equipment, which isn’t good for much of anything except mining. If people started thinking that miners were self-dealing corruptocrats, the value of Bitcoin would immediately plummet, and the miner’s investment would be worthless. Your thought of cheating, just a little, quickly flies out of your head, and you go back to being a straight-up miner – and, by the way, watching the other miners closely to make sure none of THEM cheat; if they did it would hurt you. Badly.

The miners are un-recruited, unmanaged groups who put up their own money and time to make money, and are thoroughly incented to play it straight, without cheating.

What the miners actually do is solve computationally intensive problems – all using standard software on juiced-up hardware – that does two important things.

• First, the computing assures that each new transaction that someone tries to put in the ledger follows the rules. Simple rules that are essential to virtual currency working. Things like you can only spend money you have. You can only spend it once. Stuff like that, things you don’t even think about when your money is physical and sits in a wallet -- but when it’s digital, has to be enforced.
• Second, the computing puts a lock on the new transaction, a special fancy lock that links to all the earlier locks on all the prior transactions. For ease of computing, the transactions are grouped into blocks, and it’s actually the blocks that are locked up tight and chained together with cryptology. Thus the name “blockchain.”

The rules built into the Bitcoin/blockchain software used by all the miners are the key to everything. Since all the miners run the same software, everyone follows the same rules. These rules enforce the fact that, at any given moment, there are a known amount of Bitcoin, with the ledger tracking who owns how much. The number of Bitcoin is fixed – until a miner earns some as a result of the mining work. In that case, brand-new Bitcoin are created – according to an established formula – and deposited in the miner’s own account in the ledger. Once the miner has the earned Bitcoin, he can do anything he likes with it, like any normal owner of Bitcoin.

Finally, it’s true that the Bitcoin miners see each and every transaction. Each transaction is vetted to assure that the rules are followed. But the owner is identified only by a VERY long string of letters, a key, so no one knows who the owner of Bitcoin is in physical life. This is the capstone of Bitcoin’s solution to the problem of government-issued currency. No snooping!

Net-net: There is a publicly known amount of Bitcoin in the world, which slowly grows as it is created to pay the miners who earn it by running the system. There are a large number of volunteer miners keeping transactions flowing, safe and secure, without depending on any of them. Bitcoin buying and selling is easy, inexpensive and private. Because of thousands of volunteer miners crunching away. No one’s in charge. Miners want to work and are incented to be honest. No governments, no bureaucracies, no politics, no one snooping on you. Problem solved!

That’s why Bitcoin/blockchain is new and deserves the attention and credit it’s gotten.

This post originally appeared on Forbes.

Adventures with Health Insurance Software: Customer Feedback

I got an email from my health insurance company, telling me I had an important message I could read if I clicked and got to their website. Here's what happened. While I was on the site, I discovered they were delivering break-through functionality making it easier to pay those annoying doctor bills that appear in the mail long after the visit. Here's the scoop. This post tells what happened next.

One of the best things Anthem did to enhance their customer website experience is to be humble. So many stuck-up website creators are sure they’ve done the best job that can be done, and simply put the site out there. Here it is, visitors, we’re sure you’ll agree that it’s a truly excellent website!

The experienced professionals at Anthem are well past this kind of immature attitude. They worry that the site they’ve build isn’t as good as it could be; how better to find this out than by asking the customers, the actual people who use the site?

Certain modern website designers get this kind of information in a sneaky, underhanded way. They closely monitor each click and keystroke made by visitors, and track the time between each. This method of surreptitious shadowing enables them to discern exactly when and where visitors get stuck, bogged down, get lost or whatever. That way they can enhance the site and run experiments to make everyone’s experience smoother – without telling anyone what they’re doing!

It’s amazing the public puts up with this kind of spy-movie tactics! The folks at Anthem aren’t about to be sneaky or underhanded in any way. They’re committed to open, fair and above-board methods. I experienced this myself. When I was in the middle of experiencing their amazing new billing features, I was presented with this screen:

It’s an offer to give feedback, via a third-party tool. Excellent! I think I’ll say yes, and give them my feedback.

I proceeded to experience the fullness of the new, break-through patient billing feature I’ve described here. Then, when I left the site, sure enough, I had my opportunity to provide feedback. I dove in and first saw this screen with a couple questions to which they “require” the answers.

What’s that about? This is voluntary, remember? I’m helping you guys. You should be glad to get any feedback I care to give. And I’m not starting out in a great mood because the first couple questions are completely generic.

So please forgive me, but I zoomed ahead, getting to here

Question 23! Sorry, I wasn’t able to get it done. I zoom to what looks like the end:

Getting personal with the income, are we? After a whole pile of b.s. questions. I’m outa here. But then this appears:

Yup, 26 questions. ALL OF THEM “REQUIRED.”

I wonder if anyone tracks the completion rate of these surveys. I suspect not.

I bet Anthem gets incredibly useful information from these surveys. And increases customer satisfaction along the way. No wonder their site is clearly top-of-the-line.

 

Adventures with Health Insurance Software: Provider Billing break-through!

I personally experienced the roll-out of Anthem BC-BS’s new patient billing initiative. As is well known, and as I’ve discussed in detail, patient billing, both from providers and payers, is a nightmare for everyone involved. Fixing it appears to be tough – it’s worse than being between a rock and a hard place; billing is between a rock, a hard place, the devil and the deep blue sea.

As I’ve said many times, Anthem is one of the best health insurance companies out there, full of hard-working, well-meaning people who just want to do the right thing. Their many efforts at making things better demonstrate this. But there’s clearly something about being a giant organization with lots of extensive, complex software that seems to make things go wrong. Small, entrepreneurial organizations can sometimes avoid the usual traps and turn out great, high-quality software quickly. But the big, lumbering bureaucracies just can’t seem to follow the clear patterns of success demonstrated by the winning the upstarts. Everything is against winning. They follow the regulations, the best advice of the best experts from industry and academia, and the results are consistent. Consistently bad.

Getting things badly wrong in software is most often a team effort. When it’s wrong, it’s rarely an ooops, how did that happen? I’ll fix it and then we’ll be OK! It’s more often pervasive, multi-dimensional badness. Anthem’s billing break-through is clearly the typical case.

In this post, I’ll describe the break-down, the fact that the new billing effort is broken. Again: this is not about Anthem specifically. It’s an example of a pervasive issue.

What led me to discover the new patient billing feature was an email I got from Anthem. That’s a story in itself, which I tell here.

While I was on the site, I had occasion to carefully examine the welcome page:

I noticed something new since last time I was there – the ability to pay provider bills. Wow! I’d never heard of any insurer offering this capability before. Of course I want to try out this amazing bill-paying break-through.

I’d better step back for a moment and explain the billing situation in healthcare, because when you just glance at what you’re about to see, you might think it’s no big deal.

In US healthcare insurance, there’s something called a “co-pay.” This innocent-sounding term is the cheerful face of a nightmare for all involved – patients, providers and insurers. At some point, I have no idea when, someone decided that insured patients should pay a part of their cost of their care, a bit like the well-known general insurance of a deductible, but fancier. The idea is that, for each service a patient gets, insurance would pay part of it, if the service is covered at all. Then, depending on a whole bunch of other things like whether a provider is “in-network,” the patient should pay something to the provider, an amount that the insurance company calculates using arcane formulas that are secret. Here is a detailed example.

So get this: you have insurance. You go to a covered provider to get a covered service. The insurance company pays some of the provider’s bill. The provider generates a bill for you, sometimes when you check out, but often much later, in the mail. The provider’s bill includes an amount you have to pay directly to the provider. That’s the co-pay.

This is the context that helps you understand Anthem’s billing break-through: instead of getting random bills in the mail from random providers and having to write checks and mail them various places, Anthem is letting you go to one central place and take care of them. Even electronically! A great convenience.

Health insurance companies are ideally positioned to provide this service! They already have the complex systems set up to transfer money to providers (doctors and hospitals) electronically, because they’re already paying them! Setting this kind of system up from scratch is hard, and maintaining it with all the changes to bank accounts, etc. takes a lot of work. Leveraging all that infrastructure to enable patients to pay their bills is a great idea.

Even better, the insurance company knows how much the patient is supposed to pay the provider! Even though the normal procedure is that the providers bills the patient, all the numbers ultimately come from the insurance company. So all the required data and systems are already in place! What a fabulous opportunity!

Naturally, I dove right in to trying it out. Here’s what I saw right away:

Now I’m getting really pumped – it’s exactly what’s needed!

I ask for the list and see a juicy target to pay:

Perfect, a typical co-pay. I click to see the details of the claim. Typical stuff, as expected:

Of course, it's a bill that no one but a clueless health insurance company would have the nerve to create. It doesn't tell me who provided the service, when the service was provided, where it was provided or what the service was. Other than that, it's great! Oh and by the way, they don't seem to know whether I actually owe this money or whether I've already paid it. In fact, I happen to know I have already paid it!

I know, complain, complain. But still, moving ahead...

I’ll spare you the details of how I had to provide payment information. I did it and authorized payment. Then I got this lovely screen:

Yes, that’s the whole screen. The technical term for this kind of screen is “face plant.” A “face slap” is when there’s an error, but the software knows it and can give you a nice error message with something soothing about how things aren’t as bad as they look. When you’re on a site labelled Anthem and then suddenly you’re no longer on their site and some other company’s logo appears, you know that the two bodies of software are in all-out war. You know the war has gone nuclear when the dreaded “500” error appears. That’s not an error message generated by software that is confused and knows it’s in trouble – that’s an internal website server error. This means things have gone so bonkers that the software that’s supposed to handle things has gone AWOL, and all the server can do is say, in technical terms, is OOPS.

So I re-grouped, went back to the start, picked another bill, put in my credit card information (again!), and authorized payment. Here’s what I got for my trouble:

Things are getting better – we’ve advanced from a full-on face plant to a relatively mild face slap error – no one else’s logo is in sight, no servers are complaining (at least visibly). It just plain doesn’t work.

Anthem presents bills that fail nearly all the criteria required of a reasonable bill. Their bill payment mechanism doesn't work. Even if the final step of paying the bill worked instead of face-planting or face-slapping, it would still be near-worthless! What good is a bill that doesn't actually show what you owe? Doesn't show when, where or by whom the service was provided, or even what the service was?

Here is the simple conclusion: Anthem claims to have a revolutionary method to enable patients to pay their providers, a great convenience. If it works. Which it doesn’t, to varying levels of embarrassment.

Adventures with Health Insurance Software: Email and Primary Care

Giant organizations have trouble building effective software that works and gets the job done. I have gone into depth on this subject, giving examples of the problems. But there’s something about being a large organization that seems to prevent even being aware that there’s a problem, much less being able to fix it.

I recently had occasion to dive into my health insurance company’s website, enticed by an email to do so. What I experienced was a travesty. If this company were run like a company should be run, heads should have rolled. It’s as bad as a trucking company having a large fraction of their trucks wander around getting lost, and another fraction driving off the road and crashing.

Unfortunately, this story is not about an unfortunate bug or two that somehow snuck into otherwise fine software, which is what any self-respecting manager would start by trying to claim. This is story is about software that is broken in concept and in execution – even when it “works,” it’s simply awful!

What I’m saying here flies in the face of what nearly everyone says and appears to think – including all the managers at all the places that preside over this nightmare of dysfunction. You also don’t hear any lofty academics decrying the “crisis in software,” as they should. So I’m going to lay out the facts, point by point; this is NOT fake news.

This is the first of three blog posts on this subject. This first one is pretty mild.

I got an email from my insurance company. Here it is:

I have a new message – and it’s not sales or promotional! Nothing about what the message could be about. It must be too secret and confidential to put it in regular email. Maybe it’s something about my health? I’d better check. So I click.

Oh, yuck. I’ve got to log in.

Now I have to decide how badly I want to read this non-sales email. They seem to have decided that giving me an intelligence test combined with an endurance test was the best way to determine whether I was worthy to read this non-promotional, possibly health-related message. I persisted. I dug out my user name and password for this site I rarely use, and logged in.

Or rather, I attempted to login. Here’s what I got after successfully entering my user name and password:

My user name and password weren’t good enough! This is clearly an incredibly confidential message! Even though I was using a computer I use all the time, including when accessing Anthem. I picked email, and then got this screen:

I entered the 6 digit code.

This is classic 2-factor authentication. The security “experts” at Anthem probably felt pretty good about how they increased the security at Anthem, particularly after their past embarrassments. But it’s all GARBAGE! Nothing but security Kabuki Theater! Think about it: I got to the login screen by clicking on an email that Anthem sent me!! It’s trivial to include in the email link’s URL the information about the email. So when the request comes in … Anthem knows it’s coming from the email they sent! A simple check would tell them it also is coming from a computer associated with that email. By going through the send-email-enter-6 digit-code b.s., all they’re doing is wasting my time because they already have proof that it’s my email.

Next, there’s the remarkable screen telling me how hard Anthem is working on my behalf:

All this hard work will surely result in displaying the information that the email I clicked on long ago was enticing me to click for, right? Well, no.

A completely generic welcome page!

This is a problem. A big one. You’re supposed to click to read an important message. In every system I know, a “click-me” email is a “deep link,” i.e., it doesn’t go to the home page of the web page; it goes “deep” into the site, to the place the email wants me to see. You’ve experienced this. When Facebook or LinkedIn sends you an email about something, when you click, it always deep-links you to the place referenced by the email. My blankity-blank BANK does this. Even confidential document stores that need to be highly secure do the same – once you’ve verified yourself, you go right to the document. Makes sense.

Except to Anthem. Anthem’s email link brings me to the generic welcome page of Anthem, exactly the same thing I’d see if I’d gone to the site directly.

I can barely remember how I got here, it was such an annoyingly long time ago. Oh, yeah, the email – I’ve got an important message! Now, where might that be? I look at the screen. Why don’t you check it out too – do you see anything that says “messages?” Me neither. Clearly this page, the front splash page of the Anthem patient website, has received the best vetting that the skilled professionals at Anthem can muster. And the vetting somehow failed to notice that they were going to send me to a page looking for a “message” without those seven wonderful letters appearing anywhere on the page.

Again, a combined test of intelligence and endurance. Let’s see if I can pass. Taking a closer look at that generic landing page, look at where I've put the big red arrow…

Aha!  I wonder if, by any remote chance, that red shape means messages (in the secret Anthem language), and I have 10 messages that have piled up? Let’s try clicking.

Score!

The endurance test continues. Click again. Finally, the important message in question:

At this point, all I can say is OMG.

1. I have a primary care doctor, Anthem. You know it because you pay insurance bills for that doctor covering suspiciously primary-care items like “wellness visit.”
2. The primary care doctor you’ve selected for me is indeed in the same state as me. But “close?” Not even in the same county. Sorry. No chance.

I’m so glad I endured the obstacle course and endurance test, making my way past the elaborate privacy protections to read this important message with spot-on recommendation, so cleverly refined with accurate GPS data. I can’t put into words what this has done for my admiration of the excellent insurance company that orchestrated this software ballet.