The Black Liszt

Making Things Worse with AI

AI is on a hot streak. Businesses claim they’re applying the latest AI and/or ML to their operations. There are articles and books about the wonders of AI and how it deeply threatens to replace humans doing various things. Awwwwk! What can we do??

Dilbert to the rescue

When in doubt about any important technology issue, for example about whether a programmer should become a manager, it often helps to turn to the wisdom of Dilbert for help. Here he illustrates how AI will transform HR and management in technology firms.

AI has built a perfect data-driven case on the uselessness of perpetually-lazy employee Wally. How can Wally possibly defend himself?

As you see in the cartoon, Wally pulls off the feat with aplomb. First he points out a typical flaw is logic (how different is Wally from the rest), and second he identifies what in statistics is called a confounding factor to account for his perfect record of failure. The boss’s method of job assignment was not, of course, included in the data processed by the AI algorithm, nor of course the classic error of correlation vs. causation.

The main point of the Dilbert cartoon is that AI made things worse in this case. Without Wally being given the chance to defend himself to a human being, he would have been out the door.

A little history

AI has been on the verge of ousting human beings from important jobs since the 1950’s. First it was checkers and chess. By the mid-1960’s, ELIZA and SHRDLU were having conversations with people in English. ELIZA impressed many with its conversational, interactive abilities, while SHRDLU could not only talk but it could answer questions and perform actions in its world of blocks. By the early 1970’s many experts were talking about how AI would soon rule the world.

I won’t go through all the history, but all the talk about what would happen “soon” faded away and was forgotten. Some event like Big Blue beating human chess masters would capture the news, which would again fill with experts opining about what would happen “soon.”

We’re in a hot cycle again. People in traditional roles are again being cajoled into “thinking outside the box” in order to change what they do to incorporate the wonders of AI. If they do it right, maybe they can dramatically improve things and put boatloads of people out of work! Thinking outside the box is easy enough to say, but making practical results happen with it is another things altogether. It happens less often than you might think.

Funerals for misbegotten AI projects are extremely rare, in spite of the astounding death rate.

There is real value in AI

My undergraduate thesis at Harvard was written about AI. It was an early attempt to design the data structures that would represent real-world knowledge in an intelligent robot. I focused on “simple” things like walking and catching a ball. In contrast, the vast majority of AI efforts then and since focus on the hardest and most uniquely human things people do.

Where AI and ML have made the most headway is performing tasks that most humans think are repetitive. And before you get to that, there is the perpetually challenging job of getting all the required data. Even then it is challenging to get a real-world system working. These posts illustrate the main strategies for getting good results.

Meanwhile, the current furor about the imminent taking over the world by AI is likely to continue making heavy use of the future tense, as it gradually fades as it has in the past, along with the broken promises and failed predictions.

How to Integrate AI and ML with Production Software

Most enterprises that build software are proudly flying the flag of AI/ML. "We're technology leaders!" their leaders crow in annual reports and at conferences. At the same time, any objective observer usually sees a lack of common sense in the operation of the company's systems. It often appears that, far from needing beyond-human artificial intelligence, they could use some insect-level functioning instincts that get things done. What's going on? Can it be fixed?

The Industry-standard way to fix the problem

The usual fix to the problem is to completely ignore the fact that there's a problem in public, while following something like these proven strategies:

• Brag, loudly and often, about you and your organization's commitment to AI/ML. The commitment is serious; it's deep and it's broad!
• Talk about the initiatives you've funded and the top experts you've hired.
• Talk about the promising things you've got in the works.
• Use extra phrases to demonstrate your seriousness, things like "1-to-1 personalization" and "adaptive processes" and "digital-first transformation."
• Put your top executives with fancy titles out there to follow the same strategy, using their own words.

I've given a detailed example of how a top healthcare insurance company follows this strategy while operating at a sophistication level that is best described as "hey, this electronic mail stuff sounds neat, let's give it a try."

Sometimes one of these organizations puts something in practice that works. It typically takes a great deal of time and effort to find and modify the relevant production systems. The efforts that are mostly likely to make it into production are those that can be done with the least amount of modification. For example, minimal-effort success can sometimes be achieved by extracting data from production systems, subjecting it to AI/ML magic and then either feeding a new system or making it effective with just a couple of insertion points.

The Obstacles to AI/ML Success

The obstacles to AI/ML success have two major aspects:

• The typical practice of leap-frogging all the predecessors to AI/ML to maximum sophistication.
• The extensive, incompatible existing production systems into which AI/ML power has to somehow be inserted.

A good way to understand these obstacles is to imagine that you're in a world in which boats are by far the most important means of bulk transportation. In other words, the world in which we all lived at the start of the 1800's. Suppose by some miracle a small group has invented nuclear power and has decided it would be a great way to provide locomotion to large boats instead of the sails and wind power then in use. What prevents the amazing new technology from being used?

Easy: the boats were designed for sails (with masts and all that) and have no good place to put a nuclear engine, and no way to harness its power to make the boat move.  The strong steel and other materials required to make a turbine and propellers doesn't exist. You can demonstrate the potential of your engine in isolation, but making it work in the boats available at the time won't happen. You can spend as much time as you like blaming the boats, but what's the point?

The solution is clear by studying boat locomotion: there were incremental advances in boat materials and design, and the systems used for powering them. Paddle wheelers have been around for over a thousand years. Here's a medieval representation of a Roman ox-powered paddle wheel boat.

For serious ocean travel, the choice became the large sail boat, as in this painting of boats near a Dutch fortified town:

Suppose you had a nuclear engine of some kind and were somehow able to make it with materials that were generally not available in the 1600's. How would you use it to power the sail boat? The very thought is ridiculous. The problem is that the boats have no way to accept or utilize the nuclear engine.

How to overcome the obstacles to AI/ML

What would a sensible person do? Exactly what real-life people did in history: incrementally make boats suitable for more powerful means of locomotion, and make more powerful means of locomotion that would make boats go more quickly. Practically. You know, in real life.

That means, among other things, once steam power was created, gradually make it suitable for powering ships with sails -- using the sails to conserve coal when the wind was strong, and using coal to power paddles when the wind wasn't blowing. Then, after materials advanced, invent the screw propeller -- which didn't happen until the late 1800's -- to make things even better. Eventually, the engine and the ship would converge and be suitable for the introduction of nuclear power.

This is an excellent model for understanding how to overcome the obstacles to powering existing enterprise applications with AI/ML:

• The AI/ML can only be jammed into existing systems with great effort and by making serious compromises.
  • With a few exceptions, simpler methods that can make real-life improvements should be devised and introduced first, with the portion of AI/ML gradually increasing.
• The existing enterprise applications are like wooden sailing ships, into which generation-skipping advanced locomotion simply can't be jammed.
  • Evolve the applications with automated decision-making in mind, first putting in simple methods that will produce quick returns.
  • The key to AI-friendly evolution is to center the application architecture on metadata in general, and in particular with metadata for workflow.

The important thing is this: increase the "intelligence" of your applications step by step, concentrating on simple changes for big returns. Who cares whether and to what extent AI/ML is used to make improvements? All that matters is that you make frequent changes to improve the effectiveness, appropriateness and personalization of your applications. Experience shows that relatively simple changes tend to make the greatest impact. See this series of posts for more detail.

 

Getting Results from ML and AI 6: Fintech Chatbot

While the success patterns laid out in the prior posts in this series may seem clear in the abstract, applying them in practice can be hard, because nearly everyone who thinks or talks about AI (these sets over overlap very little, sadly) takes a different approach.

http://www.blackliszt.com/2018/03/getting-results-from-ml-and-ai-1.html

http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-2.html

http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-3-closed-loop.html

I previously discussed the application of the principles to healthcare, with specific examples:

https://www.blackliszt.com/2018/08/getting-results-from-ml-and-ai-4-healthcare-examples.html

I've discussed the application of the principles to fintech, with a focus on anti-fraud:

https://www.blackliszt.com/2019/12/getting-results-from-ml-and-ai-5-fintech-fraud.html

In this post, I'll show how things can play out with a stellar example in fintech chatbots.

Computers talking with People -- Using People Language

The idea that computers should talk with us in our language rather than people struggling to learn computer talk has been around nearly as long as computers have. The earliest "high level languages" like FORTRAN and COBOL were each attempts to let people use English-like language for telling a computer what to do. They were baby steps, and people quickly decided that computers can and should do better. This thought was one of the earliest practical drivers towards Artificial Intelligence (AI) in general, and Natural Language Processing (NLP) in particular. 

One of the acknowledged milestones towards a computer that can talk like people was the work of Terry Winograd at MIT during 1968-1970. He created a talking robot, SHRDLU, that could talk about and act in a special world of blocks:

SHRDLU could have conversations about the block world that amazed people at the time. Here's an excerpt, see this for more.

I was at college at the other end of Cambridge at the time, heard about SHRDLU, and got even deeper into the AI work I was engaged in at the time as a result, for example this project. SHRDLU was wonderful, but I wondered about and worked on how to represent knowledge and actions internally. After I graduated, "everyone" was convinced that "talking computers" were going to burst on the scene any day now. A couple decades passed, and nothing.

Talking Computers Today

Here we are, 50 years later, and things have definitely advanced. We have Alexa and Siri, and in banking we have Bank of America's much-promoted Erika. But in many ways, these programs remain primitive, and are far from being able to understand context and sequence the way people do.

One of the reasons for this is that human language understanding and generation is really hard. Another is that the vast majority of people who work on the problem are thoroughly immersed in the other-worldly vapors of academia, in which publishing papers and gaining esteem among your fellows are the all-consuming goals -- to the exclusion of building products that do things that normal people value and, you know, work.

The State of the Art in Chatbots

The founders of the Oak HC/FT portfolio company Kasisto come from that world, and also from the industrial labs of Stanford and IBM that try hard to commercialize such fancy stuff, with products like Siri as results. The Kasisto people are obviously exceptional not just in their field, but in their drive to make code that does real things in the real world. If that's your goal, there is exactly one standard for measurement: what people do and say.

That's the background of the first remarkable thing about Kasisto, which I learned when I probed details of their process. Pretty much everyone who deals with AI/ML builds and builds in the lab, until they have achieved amazing things that they're ready to roll out ... which promptly belly-flops in the real world. We'll do better next time, promise! Kasisto works the way everyone should work, but practically no one does -- people first!

This means they get in the middle of a huge number of human-to-human chat sessions, building and tuning their software first by human judgment, but increasingly by machine judgment. They "try" to answer the human's question, and just as important, they rate their ability to answer the question appropriately. After tens of thousands of tests, their ability to respond like a human would improves -- and just as important, their self-rating of how well they're likely to do improves as well.

As their answers and the associated self-ratings get good, they start actively playing a role in the live question-answer flow. For exactly and only the human questions for which Kasisto is "confident" (using an adjustable threshold) it will answer well, its answers go to the human -- but are still routed to a human for double-checking, with a frequency that goes down over time.

There will always be questions that are beyond the capability of the Kasisto bot, but so long as it is "self-aware" of which ones, the customers continue to get great service, with an ever-shrinking fraction of the questions being routed to humans for answers. If Kasisto can handle 95% of the questions, this might means that instead of a staff of 100 to respond to customers, a staff of 5 could do the job.

This way of thinking is far removed from the typical academic head-set -- and I've just given a couple highlights, there's actually much more where that came from!

Beating the Pack

What I've already described enables Kasisto to deliver chat results to customers that are far superior to anyone else in the field. But the overall head-set and people-first technique, combined with some true tech smarts, leads to further things.

The first thing is context. All the other chatbots are lucky if they can give reasonable answers to isolated questions. The combination of human-first, good tech and practical methods enables Kasisto to not only answer isolated questions far better than others, but follow-on questions as well -- questions that make no sense by themselves, but perfect sense when taken in the context of an interactive sequence. As a simple example, consider this:

"How much did I spend last month?"

"How much of it was on restaurants?"

"How about the previous month?"

That's a simple one for humans, but way beyond what other chatbots can do. Why? The computer has to figure out that "how much of it" means "How much did I spend on restaurants last month?" Not to mention, handling multiple languages; performing transactions; making changes and selling products.

While Kasisto started with consumer banking, it's already added high-value functionality for treasury, business banking and more. The key is: Kasisto, unlike everyone else in the field, didn't just start selling into those applications -- they followed the laborious but effective, bottoms-up, people-first method of exhaustive training and seamless integration with humans doing chat.

Conclusion

Kasisto is an excellent example of the incredible results that can come from following the path briefly outlined in these blog posts. You would think with all the talk about AI and ML, and the all efforts and announcements, that highly capable systems would be popping out all over. They're not! That makes the few that follow the success patterns I've discussed here all the more impressive.

Getting Results from ML and AI 5: Fintech Fraud

While the success patterns laid out in the prior posts in this series may seem clear in the abstract, applying them in practice can be hard, because nearly everyone who thinks or talks about AI (these sets over overlap very little, sadly) takes a different approach.

http://www.blackliszt.com/2018/03/getting-results-from-ml-and-ai-1.html

http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-2.html

http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-3-closed-loop.html

I previously discussed the application of the principles to healthcare, with specific examples: https://www.blackliszt.com/2018/08/getting-results-from-ml-and-ai-4-healthcare-examples.html

In this post, I'll show how things can play out with a stellar example in fintech anti-fraud.

The Use of ML in Fraud Detection

Credit card fraud is a difficult, ever-evolving problem, not unlike cyber-security in general. In most cases, real money is involved, billions of dollars of it -- over $20 Billion world-wide as of a few years ago!

Robert Hecht-Nielsen was a pioneer in machine learning in the 1980's. He started HNC software to apply ML to a variety of problems. He was particularly fond of Neural Networks. After a few years of not doing well, he encountered a visionary executive of Household Finance, a large company that provided credit, including cards, to sub-prime customers. The HFC executive convinced Hecht-Nielsen to concentrate on card fraud, which was a large and growing problem for the card industry in general, and HFC in particular.

With HFC's support, HNC built the first effective card fraud detection system. It centered around humans analyzing recent instances of card fraud, building a detector for each particular kind, and training neural nets to combine all the detectors to be applied to each new transaction as it came through.

Since the model's effectiveness depended on having an ever-growing library of fraud detectors, HFC supported HNC creating a network model, in which each participating card issuer would contribute the details of each new case of fraud it encountered. HNC analysts would add detectors, train new models and give updated fraud detecting models to each participating company. It quickly became the industry standard, since all the participants benefited from the experience of all the members. HNC went public in the mid-1990's and later merged with FICO.

Enter Feedzai

How could any new entrant possibly compete against the data monopoly and neural networks of FICO/HNC? Somebody new wouldn't have access to the incredible database of card fraud. Feedzai, backed by Oak HC/FT, had some things that were more important.

First of all, it helped that the Feedzai founders were broadly expert in machine learning and other analytic methods. It took them a while to focus on "just" card fraud. Being a general expert and then deciding to focus on a narrow problem is just like HNC, and is a general success pattern. But HNC got frozen and dabbled in many other domains, failing to continue to innovate in their core product.

Once Feedzai got engaged with a card issuer, they noticed that the FICO solution was so slow that it couldn't detect whether an authorization was fraudulent in real time -- only after it was already approved. Result: every fraudster got at least one free! Beating FICO would require real-time analytics. Next, Feedzai noticed that FICO was trying to see if each transaction was BAD, and that seeing if each transaction was NOT GOOD was just as good -- even better, since you don't care how inventive the bad guys are, just that they're not acting like the real cardholder would act. That meant you could train your models on a single card issuer's customer base, ignoring the rest of the world. Cool! Finally, that meant you could use modern, transparent ML algorithms, and dispense entirely with the time-consuming and error-prone FICO method of modeling each type of fraud. And that meant that your first couple of appropriately skeptical customers could try you out, side-by-side, against the incumbent FICO, and see who won the catch-the-fraud race. Hooray, Feedzai won!

It's important to note that Feedzai also won because they did all the foundational steps described in the earlier posts in this series correctly, including the all-important know-your-data and closed-loop steps.

FICO/HNC moved on from their success in card fraud, thinking they had an absolute lock on the market -- which they did, for many years, but only because someone like Feedzai didn't go after them sooner. Feedzai isn't making the FICO/HNC mistake; they are continuing to build on their lead, both deepening and extending it. It's a big subject, but here are a couple of the highlights:

• Feedzai has built a tool that does much of the work highly skilled engineers used to have to do when supporting a new client. The original tool detects fraud; the new tool automates building a fraud detection tool. The speed and quality that results is unprecedented.
• While rarely discussed, every practical detection system has a set of human-created rules. Feedzai is the first company to automate the evaluation of the rules, and decide which should be changed or dropped. This is huge.
• Feedzai is now, step by step, introducing their technology to adjacent areas such as AML. The incentives created by the heavy regulation in AML are perverse, and has led to bloated costs in banks with no real improvement in AML detection. Feedzai's novel approach is actually making headway in spite of the regulations, and not just doing normal reg-tech automation.

Conclusion

Being an excellent generalist is often a good way to become a superb specialist. But it's rare to find a group of people who are truly up to snuff in the abstruse technologies of AI/ML, but also able to dive into the ugly details of data and real-life cases to make a system that beats the real-world incumbents. As we know from FICO/HNC and many other cases, the pattern is then to declare victory and move on. Feedzai is an object lesson in how to go deeper and deeper, continuing to innovate and automate. They are an excellent example of the principles described in the earlier posts in this series.

Surprising Bugs at Amazon Show the Fragile Foundation of AI

I promise I didn't plan it this way, but when I looked on Amazon for a product to help me deal with an infestation of bugs, I encountered a major ... yes, bug. I described the bug in detail here, and at the time thought it might be isolated -- after all, in my all-too-extensive use of Amazon, I had never before encountered such a bug.

Not long after, as the issue continued to "bug" me, I went onto Amazon again, and found more interesting bugs. This could be just some scamming or corruption. I'm taking the trouble to describe what I discovered because it's exactly the kind of data that modern systems are based on, and by playing with the data, people with bad intentions can cause the systems that depend on that data to come out with the results the bad people want. It's a way of causing destruction or stealing that's indirect, effective, and likely to grow as evil-doers catch onto the technique.

The new bug

I went into Amazon looking for a pest repeller again.

This one looked pretty close to the first product I saw -- same vendor, same product appearance, nearly the same language. But the price was different, and the number of reviews, while huge by pest repeller standards, was about half that of the original listing.

I went on, and the questions & answers looked like they were real, and for the product itself. Maybe Amazon has fixed things!

Let's scroll down more and see.

Whoops!

Still an astounding number of positive reviews, but look at the categories and things mentioned. Children's books again, this time with religion and a magic tree.

But the pictures are more creative. Look at the second one above -- clearly snitched from some product far, far away from children's books! One of the leading reviews makes absolutely clear that some serious messing with Amazon's data has taken place:

The only way this is about pest repellers is if you consider the devil to be a pest. No, I don't think so.

Implications

This is a bug. And some serious messing with data inside Amazon's systems. And a GAPING hole in Amazon's security apparatus, showing us yet again that effective cybersecurity is NOT about getting certifications and passing tests devised by government lawyers and bureaucrats.

The direct implications of this are pretty small to the average person. But if best-of-breed Amazon can be hacked this way, what about the labs where data is collected and stored concerning our health? What if, instead of just stealing the data to make a couple bucks selling it to scammers, someone decides to mess with the data to achieve an outcome, like happened here? In cases like this, the data is manipulated to get buyers to see a product based on the large number of positive reviews. The same thing could be done for a drug, a therapy or a health center! Going deeper, the data could be manipulated to impact "knowledge" of the kind that AI/ML discovers and implements. Even more deviously, it could have a devastating impact on "personalized medicine," the most data-driven of all, with a screwed-with set of data that says that a certain innocent-sounding treatment would be just right for someone like you -- except, for someone like you, it's precisely personalized disaster.

In order for super-charged AI/ML engines to actually do good things, we have to make absolutely sure that their data foundations and underpinning are sound, secure, and un-manipulated. Sadly, this is not a "what-if" issue. For details, see this. The assurances of the usual serious-sounding, credentialed experts with stentorian voices and/or soothing manners are not enough. Not even close.

Nobody Cares about Data

Nearly everyone professes to LOVE data. Just think about all the talk about Big Data, Data Lakes and the rest. Lies. Liars lying big LIES. Everyone says they like data ... until they get near it. Suddenly they develop fevers and rashes. They're allergic! Someone else will have to actually handle the data!

Data, the foundation of AI, ML, Analytics

All you have to do is get a job in one of these fancy subjects, and you quickly get hit with reality. When you were in school, you had wonderful exercises where you could develop your skills in deep learning, random forest, or whatever. Now in the real world, some older person assigns you to some juicy-sounding task where you'll get to use your skills. Where's the data? you ask. A wrist-wave of the hand tells you it's over there. You go over there and can't believe what you see. Why, it's nothing like it was in school! You try for a couple hours to clean things up. Then a couple days. It's still bad, but maybe good enough. So you run it through some models. Disaster. The system crashes and/or generates garbage. You complain. "Grow up," you're told. "This is what we've got to work with. Deal with it."

At the end of a year, you realize you've spent half your time in meetings of one kind or another, and 90% of the "working" time has been spent trying to get the data in order. With unsatisfying results. You've got some choices to make. You can lie. You can get into management, marketing or sales. You can roll up your sleeves, forget the fancy stuff you learned in school, and become a data clean-up specialist, which is actually more like a create-decent-data-from-scratch specialist. Which is NOT what you signed up for. Waaaaahhhhh.

What's maybe worse of all is the status. AI and machine learning are clearly the prestigious upper floors of a grand apartment building. Deep learning thinks it's the penthouse, but whatever. The lower floors are occupied by simple analytics. The ground floors are occupied by people managing the databases and Hadoop clusters, and maybe even some ETL tools.

And then there are the basements. The sub-basement where the garbage chutes end. Where the janitors live. Where the crap from the elegant apartments is taken to be discarded. Where the water and oil and natural gas enter the building -- the things the fancy people on the upper floors need to wash up, keep warm and prepare to dress elegantly. That's the floor ... and the status ... of the data specialists. 

You can tell yourself until you're blue in the face that without good data, none of the fancy stuff would work. It's the foundation, dammit! The janitors probably tell themselves the same thing about the heat, cooling, hot and cold water, cleaning and garbage removal. True -- but they're still janitors, wearing a uniform and passed in the halls by the upper-floor people as though they don't exist. 

Bad data equals bad results

There's a simple reason why the incredible potential of the Big Data movement has now morphed into AI/ML and is even incorporating Blockchain. The time passes, tick. Tick. Tick. Tick. No results! Uniform use of the future tense! Claimed successes aren't really, when you dig into them.

Some of the reason is typical organizational incompetence. But much is also due to the fact that we are swimming in a sea of big data and no one wants to clean it up! It's so bad, we mostly don't acknowledge it; much easier just to ignore it.

This problem isn't new. See this:

I've talked about the importance of data as the foundation of AI/ML here. I've illustrated the horrendous problem of bad medical data here. Even basic data, like what providers are where, is wrong too often. By the way, these illustrations should be considered informal tips of massive icebergs. When I talk with true experts who are themselves knee-deep in this stuff, I find the situation is ... even worse.

As today's illustration of the problem, let me show you a piece of mail I got. It's from a major corporation, one of the big regional cable companies and internet service providers. They've got decades of experience working with customers in their geography. They've got to know every address, every household, with complete histories of using their service, dropping it, signing up again. How could they not know the basic demographics and the kind of approaches that work and the ones that don't?

Here's the mail I got (I blocked out the street number):

Looks OK, right? Nice and clear. Specific to the town, so it feels personal. Lots of good things about it. They even designed the envelope so you could see the plastic card on the right, with an eye-catching banner over it.

There's just one little problem. JO Black died in May 2001. More than seventeen years ago.

I don't think there's anything else I need to say except, good job, Optimum! You're doing a great job illustrating the near-universal toxic, rotten ocean of data in which we swim, and doing your part in keeping it that way.

Wait, you might say, this is a trivial little problem. In a way, it is: one piece of mail that shouldn't have been sent. But it's an illustration of a problem that's broad and deep. The notion that a wrongly sent piece of mail "means nothing, is trivial" is an attitude that is EXACTLY why people who care about data metaphorically wear uniforms and work out of the basement. Maybe Optimum is worse than all the rest. Sorry, they're not. JO Black gets a VERY slowly diminishing stream of mail at this address from a wide variety of vendors, large and small. So does Mrs. Grace Black, who died 4 years ago. So does Ms. Jessica Black, who lived here for awhile before moving 20 years ago. So do Mr. Samuel Black and Ms. Elspeth Black, who never lived at this address.

The Problem is Everywhere

To be totally clear: the problem isn't just wasteful mail solicitations. It's everywhere, and every stage of data collection and utilization. The problem with healthcare data is immense, for example, as I've illustrated often. Bad healthcare data, which is ubiquitous, has the direct result that normal, innocent people needlessly suffer and die. It doesn't get better, because all the smart people and the important decision-makers are busy attending conferences about how AI is transforming medicine and how blockchain will solve all the medical data problems -- leaving the ragged crew of people who are supposed to fix the problem ignored in the dank basement, spending their time scheming how they can at least get to the first floor, since it's perfectly obvious that no one is actually interested in ... fixing the data problem!!!

Conclusion

Everybody says they want data. BIG data. But what they really want is a springboard to do something prestigious, which turning a toxic stream of severely polluted data into something textbook-clean is not. While hardly the only factor, this is a major factor in the widespread untalked-about failures of fancy modern techniques to deliver practical results. The plain fact is, nobody cares about data.

The Hierarchy of Software Skills and Status in Data Science

There is a striking hierarchy of skills in software, as I've explained here. When you dive into any particular aspect of software, you usually find that it's got a hierarchy all its own. Data science is a subject of intense interest these days, so in this post I'll explain some of the basics of the data science skills hierarchy.

A skills hierarchy is very much an insider's game. What most people care about is status. I talk about the basics of software status here. Remember, the skills hierarchy is a whole world away from the status hierarchy that most people care about. Don't confuse the two!

Data Science skills

First and foremost, it's important to understand the incredibly broad range of subjects covered  by the term "data science." I attempt to explain the basics of the range in this blog post. You can be just amazing in one of those subjects, while being a neophyte in one that the outside world may consider to be "related," but which in practice is not just down the hall, but in a different building on a different campus.The general understanding of this range is SO pathetic that "data science" is typically managed as a completely independent, free-standing group. Which makes about as much sense as believing that a sous-chef belongs in something that isn't a kitchen. Or that everything that everyone does in a kitchen is basically the same thing.

Here's one cut at the hierarchy in data science, starting from the base:

1. Tool users.These are people who have learned how to use some software tool, or maybe a couple. Most "data scientists" fall into this category.
   • They don’t understand how the tools are built or any of the underlying software
   • They may know their tool, but aren't real clear on what the other tools are about, much less when you might consider using one.
2. Some have broader knowledge of tools
3. Very few have the sophistication to understand real data analysis, per my series of AI/ML; see this and the links in it: http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-3-closed-loop.html
4. Of those, even fewer can understand the underlying algorithms and follow the latest research literature
5. Of those, even fewer can make real algorithmic advances and implement them as tools and deliver practical value.
6. Of anyone who can do all of the above, it is rare to meet anyone who can address and solve deep tool-level problems in other domains needed to make their code practical in the real world.
7. Finally, it is extremely rare to meet people who, in addition to their deep and broad prowess in data science and relevant skills needed to make it real-world practical, have similarly deep skills in an associated domain needed to make the data science fully effective for a business.

The issues of this hierarchy are compounded by the usual over-selling by people who are good promoters but little else, and corporate/government big-wigs who don't want to be bothered with details, but are keen to be seen as "doing something" on such a high-visibility topic. Getting results that make a difference is pretty low on the typical priority list.

And then of course there are the "data scientists" themselves, who most often are sincere people who are trying to do a good job as they've been taught to do it -- mostly by professors and others who have no idea what real-world success looks like, much less how to bring it about.

Finally, there is the usual "manage something that's invisible to you" phenomenon I have often discussed in this blog, which leads to so much dysfunction and so many wonderful Dilbert cartoons.

Conclusion

People talk as though "data science" were a thing, with the usual kind of hierarchy based on level of management and/or "experience." Those typical patterns of hierarchy just don't cut it for understanding what's going on in data science, just like they don't cut it for understanding software development. We will continue to see waste and dead-end efforts until we at least make a start at making our understanding of data science more sophisticated, and aligned with the facts on the ground.

Getting Results from ML and AI 4: Healthcare Examples

While the success patterns laid out in the prior posts in this series may seem clear in the abstract, applying them in practice can be hard, because nearly everyone who thinks or talks about AI (these sets over overlap very little, sadly) takes a different approach.

http://www.blackliszt.com/2018/03/getting-results-from-ml-and-ai-1.html

http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-2.html

http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-3-closed-loop.html

So here are a couple examples in healthcare to illustrate the principles.

The spectrum of problems

One useful way of understanding the winning patterns in AI is to understand the range of problems to which it may be applied. It's not difficult to arrange the problems as a spectrum. While there are many ways to characterize the spectrum -- here's a prior attempt of mine to characterize the spectrum in healthcare -- perhaps it's easiest to understand it in terms of the typical salary of the person whose work is being replaced or augmented by the AI technology.

At one end of the spectrum are low-paid people performing relatively mundane, repetitive tasks. These people have relatively little education and minimal certifications compared to those higher on the spectrum. Think back-office clerical staff.

At the other end of the spectrum are highly paid, educated and certified people performing what are understood to be highly skilled and consequential tasks. Think doctors.

The very name "artificial intelligence" tells you at which end of the spectrum AI is normally applied. The popular image, supported by the marketing of the relevant vendors, is that AI is amazingly smart, smarter than the smartest person in the room, just like the way that IBM's Deep Blue (predecessor of Watson) beat the human world champion playing "Jeopardy," and beat the then-reigning world champion playing chess.

To put it plainly, while these achievements of Deep Blue were amazing, they were victories playing games. They were not victories "playing" in the real world. Games are 100% artificial. The data is 100% clear and unambiguous. There are no giant seas of uncertainty, ignorance or unknowability -- unlike the real world, which is chock full of them. Nonetheless, IBM and whole piles of people who self-identify as being "smart," and are widely perceived as being smart, jumped on the "AI does what smart people do" bandwagon.

This was and is incredibly stupid and 100% bone-headed wrong. Not only is it bone-headed in terms of intelligent application of AI, it violates simple common sense. If you knew a talented high school kid who played a mean game of chess, would you drop them into a hospital and give them a white coat? Even after the kid claimed to have read and understood all the medical literature?

The smart thing to do is to apply AI to tasks that are relatively simple for humans, at the "low" end of the spectrum, and see if you can get a win. If you can make it work, by all means graduate to the next more complicated thing. It turns out that replacing/augmenting human tasks that are mundane, "simple" and repetitive is amazingly challenging! Yes, even for super-advanced AI!

IBM Watson in healthcare

I know I've made some strong statements here. It's little old me vs. a multi-billion dollar effort by that world-wide leader in AI technology, IBM. Who's going to win that one? Turns out, it's easy. See this, for example.

IBM claims to get many billions of dollars in revenue from Watson. But everything about getting it to do what doctors can do has proven to be vastly more challenging than anyone thought, and its advice rarely makes any difference, even when it's not wrong. And this, after years of work by top doctors at top institutions doing their best to help IBM "train" it!

Here is a summary of the situation:

Let's note: the Watson effort is built on the most famous "smart computer" technology ever, funded to the tune of billions of dollars, with technology acquisitions and expert help from all corners. The "disappointing" outcomes are not the result of having picked the wrong algorithm or something easily fixed. The failures are a direct result of not following the success patterns described in the earlier posts of this series, combined with applying AI to the wrong end of the job-complexity spectrum described earlier in this post.

Olive in healthcare

If IBM can't manage to pull off a win in healthcare, after years of applying the most advanced AI and spending billions of dollars with the best help that money can buy, I guess it's impossible, right?

Wrong. IBM made a fatal strategic mistake. They used AI to attack the hardest problem of all, at the wrong end of the complexity spectrum. Has anyone done this the right way? Applied modern AI and related automation technology to the right end of the complexity spectrum? Yes! Olive has!

Olive is making a positive difference today (please note the use of the present tense here) in many hospital systems by reducing costs, reducing error rates and getting patient information where it needs to go more quickly and efficiently, saving time and aggravation of medical workers along the way. The money and time it saves in the back office may not seem glamorous or "leading edge," but every minute and dollar it saves is time and money that can go to making patients healthier, instead of disappearing down the "overhead" sink-hole.

Getting a pre-auth for a key procedure so it can be performed. Submitting all the right information so a claim can be paid. Getting information to pharmacies so patients can get the life-saving drugs they need. Getting all the information from incompatible, hard-to-navigate EMR's so doctors have all the information they need to give patients the most appropriate care. These absolutely essential tasks are largely performed in windowless rooms far removed from patient settings by people who work hard at largely thankless jobs that aren't well-paid -- but are absolutely essential to providing care to patients. And they're harder than they look! Anyone who's spend any time with a modern EMR can't help but think of the endless meetings attended by skilled professionals at the software vendor trying to find yet new ways to confuse and confound the users. And anyone who has dealt with getting insurance companies the information they demand can't help but think of a cranky three-year-old who lost emotional maturity when he grew up. Bottom line: this stuff is hard!

Olive gets it done, using an exotic collection of works-today technology, silently learning from the people who do the work today. And gets it done without having to upgrade or replace existing computer systems. Amazing.

The founders at Olive are doing AI the right way, attacking the right end of the complexity spectrum. They follow most of the rest of the success patterns laid out in the prior posts of this series, above all attention to data and detail, working from the bottom up in terms of algorithmic complexity, and using closed loop. It's a hard problem, and it was hard work to get it done. But they did it, without the massive armada IBM fruitlessly assembled.

Disclosure: Olive is an investment of Oak HC/FT, the venture firm at which I'm tech partner.

 

Getting Results from ML and AI: 3, Closed Loop

Getting practical, real-world results with ML and AI involves more than getting data, doing calculations, and building models. You can do everything else right, but if you don’t get this last step right, you’ll join the rapidly growing ranks of people who may have tried hard, but ended up accomplishing little in real-world terms.

The first part of this series laid out the issues and concentrated on the indispensable foundation of success, the data. The second part of this series dove into the analytic methods that can be used to generate value, with some advice about how to sequence the methods used.

In this post, we’ll concentrate on the relationship between the real world and the back office analytical work. What we’ll find is that an integrated, collaborative, closed-loop relationship between measuring, calculating and real world application is the path to success.

Loops, open and closed

Whether you run an operation closed loop or open loop is one of those absolutely key concepts, highly correlated with success, that is rarely discussed. Generation after generation discovers it by itself, or not, nearly always without fanfare. Who talks about the key role played by the invention of the governor in 1788 by James Watt – the invention that made his steam engine practical? In that case, the governor was the newly-minted part of a steam engine that kept the pressure of the steam reasonably constant. With a governor, steam engines no longer blew up, as they regularly did before its use.

It’s important to understand that the reason a governor works is that it’s an integral part of the steam engine. Steam goes into the governor, which then controls the throttle valve of the engine, slowing it down when it’s getting dangerously hot. This is closed-loop.

In more modern terms, running open loop means going on and on down a path without real-world feedback and testing of your work as it is being developed. It’s a little like trying to walk to a goal post at the opposite end of a football field with your eyes closed, using a carefully planned sequence of steps and turns. That’s open-loop, which essentially mean no feedback. But shockingly enough, a huge fraction of highly technical efforts in software and analytics operate in just this way!  The people in charge insist they’re experienced, they’ve got a thoroughly vetted plan, and everyone should let them alone to get their work done.

There are many similarities between war-time software and running closed loop for analytics. Driving towards a goal, letting nothing get in the way. Optimizing for speed, not expectations. Leaping to a place that's better than today, and then cycling improvements.

The easiest way to see the difference is thinking about the previous posts in this series. Have you spent lots of time with data, and applied simple calculations to it? If not, you should. Once you have, … you should put your new understanding into practice! It may not be the very best solution that’s possible, but if it’s marginally better than what’s in place today, you should roll it out at least in a limited way and see how the world reacts to it. You’ll learn stuff! You may end up learning there are more variables you need to account for, different ways it needs to be applied, all sorts of things! In other words, don’t sit on the beach by the water for months – wade right in and see what it’s like. That’s when you’ll really start learning.

The World Responds and Changes

The key concept to understanding why running closed loop is so important is that the “world” is an incredibly complex, ever-changing set of actors. When you do something – almost anything – the world changes in response to what you did, if only in a small way. You have to run closed loop to respond appropriately as the world responds to your actions.

Oh, you may say, I’m just the genius in the back room who’s an expert in this or that branch of ML. I’m not acting on the world. I just need the time and support to get my amazing modeling work done.

That may be true. And that’s the problem! The whole point of doing ML/AI/etc. is to change something in the world –  and it’s guaranteed that the world will change in response! Accounting for the responsive changes is just as important as whatever it is you first put out there. Even worse, the world constantly changes independent of anything you may do. So the solution you modeled for may not be valid, given the changes that happened.

Think about the carefully planned walk on the football field to the goal posts I described above, and how hard it would be to accomplish with your eyes closed, i.e., with no feedback. Now think about the same situation, except there's an opposing team on the field! You carefully study everything about the opposing team. You know who they are and where they are. Then the play starts and you start to execute your exquisitely planned march to the goal posts. Here's the trouble: opposing team members see what you're doing, and they change their positions! They move! Even worse, they run towards you and try to tackle you. And you are helpless, because you are carefully executing your wonderful plan with your eyes closed, unable to react to the other team's movements. Is that stupid or what? It's not just stupid, it's inconceivably stupid. That's why I spelled it out, because that's exactly how most ML and other analytics efforts are carried out. Open loop. Assuming that the world does not change in response to what you do.

Of course, the world is unlikely to be quite as single-minded and determined as members of an opposing football team. But you'd be surprised! You're making changes in the real world. Whatever you do, there are probably losers. Losers who won't be happy, and will change their behaviors so they become winners again. Or simply fail to act in the predicted ways.

Conclusion

Running closed-loop is absolutely indispensable to achieving success. Put something simple in the real world and then cycle, making it better and better, using increasingly sophisticated techniques. Whatever your final crowning technique is, whether it's ML, AI or something else, success will be yours, and you'll enjoy it all along, without the risk, anxiety and likely failures of the usual highly planned methods.

Getting Results from ML and AI: 2

Getting practical, real-world results with ML and AI isn’t just a matter of hiring some people with the right credentials and throwing them at the project. Most such efforts start with fanfare but then fade into failure, usually quietly. The first part of this series laid out the issues, described the path to success, and concentrated on the indispensable foundation of success, the data. Data that has to be collected, corrected, enhanced and augmented – a time-consuming process that has no “advanced algorithm” glory, but MUST be done, and done well.

In this post, we’ll concentrate on the analytic methods that a successful project uses to generate value from the data so arduously collected and corrected.

A Little Background

I say I’m focused here on ML and AI. I just said that because it’s what everyone is talking about. What I’m really focused on is algorithms for understanding and getting value out of data. So I lied. Even worse, I’m not sorry – because just thinking that what’s important is to use the latest ML and AI techniques is central to the failure of most such efforts to deliver value.

I guess I can get over my programmer-ish prissiness that things are getting new names. What I refuse to get over is that lots of important, really valuable techniques are usually left out of the grab-bag of “ML and AI.” I won’t be comprehensive, but I think a glance at the landscape might help here.

There are a couple different ways to understand useful algorithms and how they came to be. Roughly, they are:

• Follow the algorithm, taking a fuzzy lens for the naming and details
• Follow the academic departments that “own” the algorithm
• Follow the problems the algorithm has proven to be good for

These ways overlap, but provide useful angles for understanding the algorithms and where to find them.

Let’s illustrate this with an amazing, powerful algorithm that is usually sadly ignored by people who are into ML and AI. It’s most often called linear programming (LP).Those who are into it think of it as being one of a category of algorithms called mathematical programming. More broadly, it’s normally “owned” by academic departments of operations research (OR). OR studies repeating operations like responding to repairs for appliances or controlling the output of oil refineries when prices and costs vary and optimizes the results. It’s been used for decades for this purpose in many industries, and is being rolled out today to schedule infusion centers and operating rooms in hospitals.  

This isn’t the place to spell it all out, but knowledge of amazing algorithms like LP is scattered over departments of Engineering, Computer Science, Math, Operations Research, Statistics, AI and others. The point is simple: the world of useful algorithms and modeling techniques is vastly greater than ML and AI.

The Natural Sequence

There are dozens and dozens of methods that can be used to analyze and extract value from data, which after all is the point of ML and AI (and, by implication, all the other great algorithms). As I described in the prior post, there is a natural progression or sequence of methods, which roughly follows their order of discovery and/or widespread use. Success usually comes from using the methods in the rough order of their invention as you climb the mountain of understanding from simple and obvious (in retrospect) results to increasingly non-obvious and subtle results.

I often see the following reaction to this concept, rarely articulated but often acted upon: “Why would I want to waste everyone’s time playing around with obsolete, outdated methods, when I’m an expert in the use of the most modern ML and/or AI techniques? I’m sure that my favorite ML technique … blah, blather, gobbledygook … will yield great results with this problem. Why should I be forced to use an ancient, rusting steam engine when I’m an expert in the latest rocket-powered techniques, ones that will zoom to great answers quickly?!”

The unspoken assumption behind this modern-sounding plea is that analytical techniques, ranging from simple statistics and extending to the latest ML, are like computers or powered vehicles. With those things, the latest versions are usually WAY better than prior versions. You would indeed be wasting everyone’s time and money if you insisted on using a personal computer from the 1980’s when modern computers are many thousands of times better and faster.

The trouble with this line of thinking is simple: the metaphor is inapplicable. It’s wrong! Analytic techniques are NOT like computers; they are like math, in which algebra does not make simple math obsolete – algebra assumes simple math and is built on it. Calculus does not make algebra obsolete – calculus assumes algebra and is built on it! And so on. Each step in the sequence is a refinement that is built on top of the earlier one. No one says, now that I know calculus, I refuse to do algebra because it’s old and obsolete. See this for more on this subject.

So it does make sense to quickly apply simple methods to the data to get simple answers, and at the same time vet your data. No time is wasted doing this. On the other hand, if you jump straight to someone’s favorite ML technique, not only is it likely that inaccurate and incomplete data will render the results useless … you won’t even know anything is wrong! Because most ML techniques do nothing to reveal problematic data to the researcher, while simpler methods often do!

Fundamental Analytical Concepts: Calculate it methods

The simplest and most useful methods are ones in which you simply calculate the answer. There’s no modeling, no training, no uncertainty. These methods are highly useful for both understanding and correcting the data you’ve got. The basic methods of statistics like regression apply here, and so do the methods of data organization and presentation usually called OLAP, BI and dimensional analysis. The tools associated with a star schema in a DBMS apply here, which are roughly the same as pivot tables in Excel.

Graphing and visualization tools are important companions to these methods; they help you really understand the numbers and see to what extent they make common sense and match reality. For example, you can see to what extent a doctor’s years of experience correlate with ordering tests or issuing prescriptions of a certain kind; or simply identify the doctors whose actions stand out from the rest. There could be a good reason why they stand out; wouldn’t you like to find out why? Maybe the doctor should be emulated by others, or maybe the doctor should be corrected; either way, you should figure it out.

Until you’ve pursuing all lines of thinking based on these simpler methods, it’s premature to move on.

Fundamental Analytical Concepts: Solve/Optimize it methods

These are, IMHO, the gold standard of algorithmic improvement. When applicable, they tell you how to reach a provably optimal result! No training. It takes experience and judgment to apply the generic algorithms to a particular problem set, and sometimes the problem needs to be adjusted. But the results are stellar.

First, you create an equation that measures what you’re trying to optimize. Is it fastest time? Lowest cost? Least waste? Some combination? Whatever it is, that’s what you’ll maximize or minimize as the case may be.

Next, you determine the constraints. You only have so many operating rooms? This kind of machine failure requires a repairman with that kind of skills?  Then you put in the inputs and solve. While I’m leaving out lots of detail, that’s the basic idea.

These methods, usually of the OR kind, have been applied with great success for decades. In certain fields and industries, they are part of the standard operating procedure – it would be unprofessional to fail to apply them. And you would rapidly lose to the competition.

Fundamental Analytical Concepts: Train it methods

The training methods all require sample data sets on which to “train” the model. Selecting and controlling the data set is key, as is avoiding over-training, in which the trained model can’t generalize what it’s been trained on, and thus loses most of its utility.

Fundamental Analytical Concepts: Train it methods: white box

What characterizes these methods is something incredibly important: what the model does can generally be explained in human-understandable terms, i.e., it’s “white box.” This has huge value, if only to gain acceptance for what the model does – but it may also bring up problems with data that can lead to further improvements.

There are lots of ML algorithms that are in this category. All the decision tree methods are here, among them the very important random forest method, along with methods that arose within the field of statistics such as CART.

Fundamental Analytical Concepts: Train it methods: black box

These methods can produce amazing results, and should be used whenever necessary, i.e., whenever earlier methods in the sequence can’t be used. The fact that the model is “black box” means that it’s difficult if not impossible to understand how the model makes its decisions in human terms – even for an expert.

These methods include neural networks in all forms, including all the variations of “deep learning.”

Fundamental Analytical Concepts: Rules

Finally, I add the indispensable attribute of success in many practical systems: human-coded rules. These can be inserted at any point in processing, as early as enhancing the data before any methods work on it, and as late as modifying the results of final processing. While not often explicitly discussed, few practitioners with successful track records avoid the use of rules altogether. They may not be pretty or fancy or elegant – but they work, darn it.

More elaborate than sets of rules is the technique in AI of expert systems. This is a whole big subject of its own. Generally speaking, if you can get useful results from one of the sequence of methods up to and including white box training systems, you should do so. But important categories of problems can only be solved using expert systems, which ideally should be as white box as possible.

Conclusion

There is a broad range of analytic techniques that can be applied to a given problem. There is an optimal sequence for understanding the data and the problem. Going from one step in the sequence to the next, when done correctly, isn’t abandoning a method for something better, but first picking the low-hanging fruit and then moving on to catch tougher stuff. Prejudging the best technique to use before really getting your hands dirty is a mistake. Being a specialist in a particular method, e.g. “deep learning,” and confining your activities to that method alone can get you hired, paid and busy, but may lead to no useful results, or results far less useful than they could be.

Getting Results from ML and AI: 1

We hear quite a bit these days about ML (machine learning) and AI (Artificial Intelligence), as the drumbeats of Big Data and Analytics fade away. You’ve got to be using these things to get great results and transform your business! You’d better round out the staff of Data Scientists you hired last year, and add appropriate numbers of ML and AI experts to the mix. Otherwise, you’re hopelessly behind the times, and you’ll eat the dust generated by the winners!

Most efforts to apply these technologies fail. Not loudly, of course – no one admits failure. But after enough time passes with exciting results being right around the next corner, people stop talking about catching a glimpse of light from the end of the tunnel, and accept the fact that they’re digging a tunnel deeper and deeper, a tunnel to nowhere.

Is there a way to get amazing value out of these exotic technologies? Yes. Decades of solid results show the way.

Typical Failure Patterns

In real-life cases, what happens all too often is that “data scientists” are called in to apply their magic. They take the available data and apply their favorite techniques. They may not produce results that are promising. If the results are promising, there is trouble applying them to the real-life situation. Or if they can somehow be applied, they don’t work or produced the promised results.

The failures are the direct result of taking a naïve approach to applying these kinds of techniques in the real world. There are proven methods for attaining good results, but those methods are rarely discussed, for reasons I don’t understand. If they’re discussed, they’re lightly brushed over – instead of being given center stage as they deserve. Following are the ways to be successful with ML and AI.

Build from the bottom up

No one tries to sell a house based on how solid its foundation is; but a house with a crappy foundation will collapse. No one brags about their arithmetic skills when trying to get a job as an ML expert; but if you can’t add, how can you do ML? There is a clear sequence of learning ML and AI. You start with learning how to count; if you don’t know the numbers, you know nothing. Then you learn basic manipulation of numbers, like addition and multiplication. Then it’s on to algebra and the following stages. You don’t try to learn exotic techniques until you’ve mastered the more basic ones on which they’re based, and on which they depend. These are things that are nearly universally accepted.

What’s not so common is applying the same sequence when analyzing any particular problem. The foundation of the sequence, the first step, is the numbers, the data. All too often, the data is incorrect and incomplete. If the data is bad, the results will be worthless.

I’ve never found anyone who disputes this, once the issue is raised. But I also rarely find people who act on it, and take it seriously. Why? Among other reasons, it's a multi-trillion dollar issue.

I have worked with some of the most advanced people in the field, including someone who’s been the chairman of one of the top academic departments in the field. This person and his methods have been behind a few of the most widely used success stories. Here’s his “secret” for when he dives into a new problem: he loads the data into Excel and looks at it, first line-by-line, and then using functions and visualizations. Yes, I know, Excel is something accountants use and try to avoid. But it’s ideal for fast, visual analysis of data sets, and has some of the most advanced algorithms available as add-in libraries. Why would you start programming in Python when you can move quickly mostly without programming using a tool?

The important thing isn’t the tool. The important thing is the activity – look at the data. Seriously look at it! Don’t just scan and move on. Understand it! What you’ll almost always find is that there are errors. Mistakes. Important stuff that’s missing. Graph it and look for basic correlations, and see if it makes sense. Make sure a true subject-matter expert is by your side.

Then the fun begins. You have to fix the data. It’s the foundation of everything else. Without a solid foundation, nothing of value can be built on it.

It’s also important to understand that this isn’t something mechanical, like spell-checking. What you often find is that really crucial data is missing, or that real important data can be added. This simple-sounding fact can be a project-maker. I have been involved with several projects in which the mundane-sounding effort of adding more relevant data has been the difference between failure and incredible, world-changing success.

OK, your data’s pretty good. Time to dive into ML? No way!! Way too soon! We’re going to go in sequence here, applying the very most basic techniques to the data first.

The point of all this is simple: you squeeze all the value you can out of a given “level” of technique before advancing to the next one. There are lots of reasons why this makes sense. The simpler techniques yield results more quickly than fancier ones. They tend to be larger and more obvious, which means the impact will be big. People will understand them, and so are more likely to buy into the changes needed to apply them in real life.

I’m not going to spell out all the techniques you should apply and the proper sequence, but generally speaking, the order is the same as the order in which the techniques were discovered historically, which is the roughly the same order in which the techniques are taught in school. So you try simple linear regression before multi-variate, for example. And you always look and use visual methods, because a surprising number of the advances are often ones that people in the field know or expect, or that at least “make sense” to them.

Finally, at long last, you get to use the fancy stuff you’ve been itching to use all along. But by then … your data’s in great shape. Your system is already up and running. People are already accustomed to change and the improvements that result from applying math. And interestingly, there may not be that much “juice” to be squeezed out of the system by then. Depending on your scale, that remaining juice may be tasty indeed, but it’s the icing on the cake.

In applying AI, the pattern is the same, except that in addition to applying simpler analytic techniques, you may be writing common-sense-understandable rules by hand. Why not? It gets the job done, it’s simple and direct, and the AI can focus on that yummy icing.

Other Issues

We’re not done! There are a couple other major, overriding issues to be considered in order to get great results from these advanced methods. I’ll cover them in future posts. They are:

• Most people have a favorite method, in which they have expertise and experience. That’s wonderful, except that there is a world of different methods, and many of them are simple inapplicable to certain kinds of problems – but great on others. Picking the right method (or methods) is absolutely key to success.
• Closed loop. All too many projects run open loop. In my descriptions above, I sneakily assumed closed loop – that’s where the best feedback comes from.

Conclusion

Wonderful results can be obtained by applying modern analytic methods to real-world problems. But you have to choose: do you want an academic prize or do you want real-world improvements? Sadly, those goals don’t have loads of overlap. If you want real-world results, you should build your effort on a solid foundation of accurate and complete data, and move from simple to increasingly refined as you apply algorithms to it. If you do, you’ll see positive results fairly quickly, and those results will get better and better as you climb up the mountain of sophistication.

Devil's Dictionary for 21st Century Computing 3

More cynical definitions in the series introduced here, for Deep Learning and Blockchain.

Ambrose Bierce

A couple definitions from his book:

Stan Kelly-Bootle

Mr. Kelly-Bootle sometimes provided extended explanations of the words he defined:

Sometimes he even needed illustrations. See the two definitions below, followed by illustrations:

Definitions for 21st Century Computing

A couple more from the student:

Deep Learning

Deep learning is an evolution of shallow neural networks, in which the neural networks are stacked in many layers, making them “deep.”

Decades after the 1959 biological model introduced by Nobel Prize-winning scientists Hubel and Wiesel inspired artificial intelligence pioneers at MIT and elsewhere to invent neural network technology, someone noticed that biological neurons are connected in many layers, unlike the single-layer neural networks that AI researchers had been touting for years as the basis for recreating human intelligence inside a machine. Since everyone knows that prestigious artificial intelligence researchers don’t commit errors, or at least simple ones, “deep learning” was introduced as a brand-new idea that would finally crack the code of making machines as smart as the average fifth grader. Someday. Maybe.

Blockchain

A hot new technology that is sweeping through the world of finance,  healthcare and elsewhere, whose greatest practical success to date has been the secret transfer of funds between cooperating parties in a criminal enterprise.

A newly discovered database that has recently been freed from the nearly unbreakable bonds of its cryptocurrency prison; however, as a new kind of database, it stubbornly refuses to be classified as a “database,” preferring to be known as a “distributed ledger,” of which it is apparently the only known exemplar. A cynic might point out that that the stubborn refusal to agree to be part of genus database-imus may be due to the wholly inadequate functionality and performance of blockchain on generally accepted measures of database value, but this is almost certainly unfair to such a widely hailed future solution to problems that undoubtedly are pressing, and have resisted solution for many years.

Conclusion

I apologize in advance: there could be more to come.

Devil's Dictionary for 21st Century Computing 2

Another in the series introduced here.

Ambrose Bierce

A couple definitions from his book:

Stan Kelly-Bootle

Mr. Kelly-Bootle sometimes provided extended explanations of the words he defined:

 

Definitions for 21st Century Computing

A couple more from the student:

Cognitive Computing

A totally, absolutely brand-new approach to making computers that are really smart. Cognitive computing is already a success primarily because it has NOTHING whatsoever to do with certain lame technologies that have a decades-long, proven track record of achieving perpetually imminent success. Cognitive computing is primarily backed by a giant company whose roots go back to the technology that popularized the term “hanging chads,” whose TLA name is alphabetically adjacent to HAL, the star of movie set in 2001.

Machine Learning

The term for a growing collection of dozens of techniques that have been developed in the continuing quest to teach machines enough so that they can score better than they do on the college entrance exams. Until the quest for effective machine learning yields better results, machines will continue to be relegated to second-class status among the company of educated things.

The advocates of machine learning are known to be a fiercely contentious lot, each asserting that its own approach is superior to all others, and that any evidence adduced to the contrary is propaganda, fake news of the worst sort, stemming from jealous advocates of inferior approaches. The closest approximation to the internecine warfare of the machine learning field is the human learning field, in which advocates of public, government-run and union-staffed schools exchange harsh words with advocates of charter schools, with a level of invective and passion that indicates that someone is strongly in favor of hopelessly uneducated machines and/or humans.

Conclusion

I apologize in advance: there could be more to come.

Devil’s Dictionary for 21st Century Computing

Ambrose Bierce wrote the Devil’s Dictionary in 1910, delighting and edifying cynics everywhere. Stan Kelly-Bootle wrote a new version for the world of computing called the Devil’s DP Dictionary in 1981, and a later edition in 1995 called the Computer ContraDictionary. These are timeless works, providing valuable insight and inspiration for cynics to this day. But there are modern computing terms that came into use after these geniuses had passed onto their reward. It’s time for at least a first draft of a Computer Cynic’s Dictionary for the 21^st Century.

Ambrose Bierce

Mr. Bierce started publishing definitions many years before the first book appeared. Here is the start of a column from 1881:

You can see that from the very start, Mr. Bierce had the ability to get at the heart of things using few words.

Stan Kelly-Bootle

Ambrose Bierce was clearly a tough act to follow, but the new computer technology was such rich soil that Mr. Kelly-Bootle felt that an attempt had to be made. And a heroic attempt it was, providing insight and edification all these years later. The following couple of simple definitions get right to the point:

In other definitions, he gets a bit more cutting:

Cynicism in the 21st Century

Many new terms have entered the world of computing since Mr. Kelly-Bootle last graced us with his wisdom. Reasonable people may ask, "is cynicism dead?" "Will such juicy targets remain unskewered?"

I have searched high and (especially) low, and found nothing but piles of dry computer-babble, peppered with ignorance and misinformation. I have yet to find a good source of penetrating definitions for any the terms being thrown wildly about in today's discourse. I feel I have no choice but to offer some of my own definitions, sad exemplars of the type though they be, in hope of challenging those with the true, deep knowledge of a Bierce or Bootle to counter with their own superior definitions.

Here is the first installment. Should I somehow avoid assassination, more will follow in future posts.

Big Data

A subject of which no self-respecting executive may claim ignorance; an expensive, ever-growing collection of hardware and software managed by people who spout a dizzying array of acronyms with confidence and certainty, with mounting expenses and benefits that are just about to be realized.

A collection of data, presumed to be large but normally fitting in a backpack with room to spare, which is said to contain untold riches if only they can be found and unlocked with mysterious keys like Hadoop.

An approach to analyzing incredibly huuuuge collections of data that has been recently invented, bearing no resemblance whatsoever to outdated technologies such as data warehousing and business intelligence, and sharing none of their drawbacks.

Artificial Intelligence

A kind of intelligence, sometimes implemented by computers, which would be decisively rejected by all right-thinking people if it were food. It is the opposite of organic, free-range, unprocessed intelligence – it is chock-full of GMO’s, fructose and artificial ingredients of many kinds.

The growing crisis of insufficient intelligence is being addressed by some leading scientists, who are leading the way in the creation of artificial intelligence to fill in the gaps left by inadequate supplies of naturally-occurring intelligence. Like the green revolution in agriculture, many hope that this emerging “grey revolution” will put a stop to the persistent intelligence shortages that make so many miserable. While some elites sneer that artificial, non-organic intelligence is deeply harmful, most of the deprived are glad to be served intelligence of any kind, however artificial it may be, rather than their current meager diets containing precious little intelligence of any kind.

A purposely vague term, referring to an ever-growing set of tools and techniques, that are said to do stuff that people usually do, only better. AI programs have advanced from early victories in playing checkers to wins against chess masters. They have finally achieved the pinnacle of human intelligence, winning the game show Jeopardy. After decades of marching from success to success, today's leaders of Artificial Intelligence anticipate that practical applications of the technology are certain to emerge. If not, they threaten to further inflate the definition of Artificial Intelligence to encompass normal computer programs written by ordinary human beings, at which point success will be theirs -- since a computer program is, without doubt, artificial.

Conclusion

I expect to release more definitions in the course of this year.

The Healthcare Innovation Spectrum: From Washing Hands to AI

There's a spectrum of ways to innovate in healthcare. On one end is simple stuff, like making sure things are clean and germ-free. On the other end is exotic stuff, like using AI: Artificial Intelligence and Cognitive Computing. Obvious questions: (1) where is the money going? (2) where is the value? (3) Is the money going where the value is? Simple answer: the "smart" money is going to exotic gee-gaws, ignoring near-term value and patient health.

Where the Money is going

The money is clearly going to exotica. Ignoring for the moment the billions IBM and others are pouring into what they call Cognitive Computing, VC's are investing heavily in healthcare-directed AI. See this:

We're talking serious money here:

While there are loads of conferences, trials, talks and articles talking about the great future here, there is an obvious conclusion to be drawn: while the money is being spent now, the benefits (if any) are in the future.

That's about all you need to say about it.

The middle of the spectrum

While things like AI are clearly at one far end of the spectrum of healthcare innovation, there are intelligent, educated things in the middle of spectrum. Lots of people are pursuing these innovations with great energy. I've discussed an example of one such approach here.

The Oak HC/FT portfolio company VillageMD is another clear example of data-driven innovation in healthcare. No new math or fancy computers are required. "Just" educated, dedicated people looking at the data and making required behavioral changes based on those facts. The founder of VillageMD, Clive Fields, just won a major award for his work, using all-organic and natural intelligence -- no artificial ingredients! Guess what: it's here and now! The outcomes of real patients are being improved as you read this!

The basic end of the spectrum

On the other end of the spectrum from AI, we've got things that shouldn't need "innovation." They should be standard practice. They have huge impact. They are the shocking, scandalous modern equivalent of antiseptic surgery -- things that no one seriously disagrees with, but which the important experts and leadership type people somehow can't lower themselves to pay serious attention to. Or when they pay attention, it's with actions that do nothing to solve the problems.

A good candidate for the poster child of this end of the spectrum is what the CDC calls healthcare-associated infections, HAI's. In other words, getting sick from going to the hospital. Here is the CDC's summary of the situation:

I don't know about you, but this makes me sick. 75,000 preventable deaths in a year, preventable using non-exotic methods. No Cognitive Computing required! There are cures, demonstrated at multiple hospitals that have put serious effort into it. This article summarizes the efforts and approaches, ranging from simple changes of cleaning practices to fancy new machines.

Conclusion

There's a clear spectrum of innovation in healthcare, ranging from blocking-and-tackling basics at one end, to exotic new things based on various forms of Artificial Intelligence at the other end, with smart, non-exotic, data-driven methods occupying the middle ground. Most of the "smart" money appears to be going to the fancy exotic end, with results sometime in the indefinite future, while the rest of the spectrum trundles along, largely under the radar, delivering results to patients today.

Healthcare Innovation: Can Big Data and Cognitive Computing Deliver It?

Most people seem to agree that healthcare is ripe for innovation, and badly needs it. Lots of people are talking up two potential sources for that innovation: Big Data and Cognitive Computing.

I'm strongly in favor of data, the bigger the better. But is the Big Data movement going to make a difference? I'm strongly in favor of cognition, computing, and computing that is smarter rather than dumber. But is the Cognitive Computing movement likely to make a difference? Here's a summary of some thoughts.

Process Automation and continuous improvement

Here is a description of the core process automation process implemented by a company I've invested in, Candescent Health. It describes the process that can and should be applied to all of health care.

The point isn’t that there’s data and analytics – the point is that there’s a closed-loop process of continuous improvement where actions are based on rules. This is the framework that is required to make anything happen. Without it, you can’t put your proposed new clinical action into practice with double-blind A-B test and see if the results of your analytics actually deliver benefits in the real world! Or even just deploy it!

How about just making the basics work?

Here is the story illustrated by Mt Sinai hospital about how everyone focuses on “innovation” and fancy new things, when just having the computer systems run reliability has a huge impact on patients – and unless those systems run, the results of fancy new analytics can’t be delivered to benefit patients.

If the car won't start or run reliably, who cares how good the fancy sound and navigation systems are?

How about making the computers work?

I love data and analytics. But doesn’t it make sense to focus on getting the operational computer systems to actually run well before moving on to the fancy stuff?

Paying top dollar for computers doesn't make them work

In fact, just about anything you do with healthcare data that is going to be brought to the front line of care requires functioning computer systems to be able to pull off – the big healthcare systems pay Greenwich CT prices and get trailer park results.

Clean data isn't easy to get

Both data warehousing and the fancy new Big Data movement share the under-appreciated problem of getting good quality data in analytics-ready form. Sounds simple, but the difficulties make progress a grinding crawl on many efforts. See this for example.

Big data sets tend to have Big problems

Massive data sets have built-in problems that make it hard to get actionable results.

AI: How about under-promise and over-deliver for a change?

Skepticism about Cognitive Computing in health care is warranted. There is a rich history of over-promise and under-deliver for AI efforts in general.

Real-world solutions waiting to be automated

Meanwhile, there are proven gems in the medical literature just waiting to be disseminated to the front lines of health care via point-of-care computer systems that are languishing in journals.

What can make a difference?

There are lots of practical, tangible ways to make things better, in spite of all the obstacles to change pervading our healthcare system. Here are some examples of people doing the right thing, all them with investments by Oak HC/FT:

• Candescent delivers better imaging results with less expense by applying basic continuous-improvement workflow automation.
• VillageMD delivers better results with lower cost by feeding back results and advice to PCP’s.
• Aspire delivers better results at lower cost for end of life – by having one person be in charge, managing everything from the patient point of view.
• Quartet makes a difference by applying behavior health as needed to help other conditions.

These companies embody some common themes:

• Knock down the silos, have a patient-experience-centric point of view.
• Applying common sense has huge benefits.
• Focus on delivering results to the front line (patient) is hard but necessary.
• A system of continuous learning and delivery is a pre-condition to delivering any results of analytics for patient benefit.

Conclusion

The big hot topics in healthcare of Big Data and Cognitive Computing are little more than fashion statements. Data, of course, is a good thing; so is having computers do smart things. But without doing some basic blocking-and-tackling and applying some practical common sense, a great deal of time, money and energy will be spent accomplishing nothing.

Human-Implemented Cognitive Computing in Healthcare

I'm pretty skeptical about "cognitive computing." It's hard even for people to "cognitively compute." In fields like medicine, only the brightest, most educated and experienced people are capable of producing useful results. But when they do, the results can be powerful, useful and save lives.

Cognitive Computing

I've worked closely with computers since I was in my teens. I've seen their impact on society, and the huge productivity gains when properly applied. But ever since the early days, a subset of the computer industry and the public has insisted on seeing computers as versions of human brains. Periodically, the computer industry gets excited about how the latest computer hardware and software will enable computers to do things that only the smartest and most educated humans can do. When the excitement gets frothy, the movement is called something new, so no one will be "confused," and think it's the same as all the earlier, essentially identical movements that have failed and quietly faded away. The latest such movement is called "cognitive computing."

Medicine

Doctors who specialize in a field of study both contribute to advancing the state of the art and stay on top of advances made by others. Action-oriented review articles are particularly valuable -- they both summarize advances made by many people, and advance clinical practice by making those advances practical and actionable.

I'll take an example from emergency medicine, this journal in particular.

One of the articles that appeared in that journal earlier this year was about Horner's Syndrome in children.

There is lots of fascinating and useful information in the article. It's just amazing the things that go on in the human body. Knowing about all the things that sometimes go wrong makes it all the more impressive that most bodies work so well most of the time!

Here's a chart in the article that boils it all down. You've got a child presenting with Ptosis. What's going on here? There are a couple options, and the chart guides you to figuring it out.

One of the outcomes is Horner's Syndrome. What do you do next? Here's a chart that makes it all clear.

This example of human-implemented "cognitive computing" is similar in principle to many valuable intellectual and scientific results. It's the result of years of effort and study by many people handling many cases, and publishing the results. The advance here is boiling it all down and translating it into simple, unambiguous flowcharts that guide you to do the right thing, without leaving out anything important.

Can and should flowcharts like this be made available to front-line clinicians as they are seeing cases? Yes, of course, if only to enable the clinician to make sure something new hasn't emerged since last time he/she checked. Does it require fancy computing to make this happen? It does not.

Conclusion

Getting simple results like these is amazingly tough work for highly educated and motivated human beings. Meanwhile, computers are not yet capable of tying my shoes. When they are, I will gladly put them in the running for doing something more sophisticated and important, like picking who should have the lead role in the next James Bond movie. "Reading" the medical literature and figuring out how to respond when a child presents with ptosis? The crew that can't even keep a hospital's critical computer systems running is going to one-up humans? Maybe it's something you'll welcome for your child; for mine, I'll pass on the opportunity, thanks very much.

Cognitive Computing and Healthcare

They say that cognitive computing, the term-du-jour for Artificial Intelligence (AI), is in the process of transforming healthcare. Billions of dollars of investment are behind the effort. Sadly, there are good reasons to believe that little good will come of it.

Cognitive Computing

Whatever it is, people are pretty sure it's BIG. Here's what a major investor and the former GM of IBM's Watson unit says about it:

$80 billion dollars! Before long, we'll be talking serious money here!

Where's this money going? Lots of places. But there's one special target for the money. The same expert tells us:

Cognitive Computing in Healthcare

Is Cognitive Computing really happening in healthcare? You betcha. IBM's Watson by itself is making major inroads into healthcare, with terrific-sounding projects at Sloan Kettering, Cleveland Clinic, MD Anderson and others. Good things are coming! For example, C. Martin Harris, MD, chief information officer of Cleveland Clinic, says:

Cleveland Clinic's collaboration with IBM is exciting because it offers us the opportunity to teach Watson to 'think' in ways that have the potential to make it a powerful tool in medicine.

And here is how the CEO of IBM explained it in an interview:

It's so hot that IBM has created a separate division for Watson, investing more than $1 billion just to get it started, and will have a headquarters group employing more than 2,000 people.

So What's the Problem?

Big investments like this should mean that there's a big problem to be solved. What is it? Not enough doctors? The doctors are too expensive, and somehow automating what they do with this mega-expensive effort will help that? The doctors aren't as smart or educated as Watson will (by presumption) be?

Someone involved should let the rest of us know.

Meanwhile, count me a skeptic. The reason is simple: there is a decades-long history of researchers and big companies making claims more modest than the ones being made for "cognitive computing," and they've all failed, technically and/or in business terms. In the end, computers do get used for more and more, as we all know from personal experience. That's a trend that will certainly continue. But "cognitive computing," i.e., AI reincarnated and re-named? Uh-uh.

Human and Inhuman Analytics

While people talk about analytics in general, there are really two distinct varieties: human analytics and inhuman analytics. First, there is analytics for and by humans, i.e., numbers, tables and graphs designed by humans for human consumption and consideration. Second, there is algorithmic analytics, originally designed by humans but then set off to make observations, decisions and perhaps actions on its own. I dub this "inhuman analytics," because that's what it is. It is incredibly important to understand the differences between these two things, related in name but little else.

Human Analytics

When most people think about analytics, they're usually thinking about things like Data Warehouse (DW), Online Analytic Processing (OLAP), Business Intelligence (BI), and related subjects.

This is a subject that is broad and deep, with many products and vendors that have evolved over time. But there is a simple unifying theme: these are tools intended to provide information to people, often in the form of graphics, so that those people can understand what's going on and take any action that may be appropriate.

Oracle, for example, has a wide variety of such tools:

Microsoft also has a variety of such tools.

Note that both companies illustrate their approach using screens and people. That's what this type of analytics is all about.

There are a wide variety of BI tools from many vendors, in addition to open source.

Inhuman Analytics

Inhuman analytics, a terms that no one else uses, so far as I am aware, is a whole different thing. This is also a subject that is broad and deep and undergoing constant innovation. It includes such diverse subjects as machine learning (ML), advanced statistics, operations research (OR) and related subjects.

In general, inhuman analytics are far more specialized than human analytics. They are nearly impossible for anyone but a specialist to understand. There is often lots of math involved. They are not primarily about presenting information so that it makes sense to human beings -- they are about figuring stuff out that most humans wouldn't be able to figure out at all, or figure it out with a precision that exceeds human capability.

Because of this, there aren't great pictures to illustrate inhuman analytics. But here's an illustration of the ML process from one company's ML toolkit:

Inhuman analytics are behind a large number of modern innovations, though they rarely get credit for it, since the way they work is essentially like magic to most people This is a vibrant subject with a rich history. I suspect I will come back to this in some future post.

Conclusion

Human analytics has many uses and is a good thing. The visual tools it emphasizes enables knowledgeable and motivated people to explore and understand a data set, and to track it over time. Sometimes you can even discover new things, particularly in the early stages of understanding and optimization

However, inhuman analytics are the serious, heavy-duty tools to help derive value from data. They can and regularlly do figure things out and solve problems that are beyond human capability, even with the aid of human analytics.

Human analytics has its place. But it's no substitute for inhuman analytics for serious value creation.