The Black Liszt

An Immutable Distributed Ledger is now in Large-Scale Production!

First there was Bitcoin, friend of criminals, speculators and tech geeks everywhere. It’s grown amazingly. Then there were alternatives to Bitcoin, often sharing much of the same code, but with different and incompatible tokens. One of those Bitcoin alternatives, Ethereum, introduced the concept of Smart Contracts, which I discuss here. Now, increasing attention is being paid to “blockchain,” said to be the foundation on which crypto-currencies like Bitcoin and Ethereum are built. Large corporations are taking up the charge, places like IBM and Microsoft, and leaders in various industries have projects going to prove out the technology. While the terminology isn’t uniform, it’s easy to see that earlier terms with unsavory associations are being abandoned in favor of more generic terminology, names like blockchain,  “Immutable Distributed Ledger” technology, or just “distributed ledger.”

Once you start talking about the technology in generic terms, what are the chances of this actually working? At scale? In practical reality? Lots of people in the community of blockchain enthusiasts have expressed concern about this. Legitimate concern. The question naturally arises, but appears not to have been asked, has something like this been built before? Something that could legitimately be called an immutable distributed ledger?

The answer is a simple “yes.”

This amazing system, which is one of several in production today, has over 2 billion accounts and over 40 million participating agents. It moves over $10 trillion per year, processing over 150 million transactions a day, and can handle over 50,000 transaction messages per second.

Let’s dive into the technology a bit:

• It’s immutable. Once a transaction gets in the system, it can’t be altered or removed. Despite the volumes mentioned above, spread over more than 200 countries, there are no instances of processed transactions being altered.
• It’s distributed. Computers all over the world are involved. It doesn’t matter where you are: this system enables you to send or receive currency. Even better, currency conversions are built into the system! The distribution has been built in part to enable reliability. While any one computer in the system can fail, the system as a whole has never gone down, and transactions have never been lost.
• It’s a ledger. It’s a gigantic ledger of currency going into and out of accounts. The ledger balances to the penny every single day.
• Consumers don’t lose their money! In spite of all the volume, When bad stuff somehow happens, consumers lose nothing. Nada.

OK, OK, I’ll stop the charade. As you’ve probably guessed, the Distributed Ledger technology I’m describing here is in fact VISA. I’m playing this game because reading about how enthusiasts talk about blockchain, I wonder how many of them know about credit card internals? If they did, they would see that all the goals they have for blockchain have already been achieved in credit cards. And more!

The key assumption at the core of the blockchain craze is that blockchain is an amazing new technology, a breakthrough that enables all sorts of long-intractable problems to be solved. These virtues are primarily the fact that it’s immutable, it’s distributed and it’s a ledger. Sorry, guys, you don’t need blockchain to build a system that has those attributes. It’s already been built using plain old database technology and secure networking.

Yes, I played a game when describing the immutable distributed ledger technology that’s already in massive production, knowing everyone would think I was talking about blockchain. But the blockchain groupies are playing a more serious game, convincing themselves and others that blockchain is uniquely able to do things that haven’t been done before, and could never be done without this amazing new invention. Just to be clear, I’m not saying that VISA technology can be applied to the many problems to which blockchain is being applied. I’m simply saying that if you think you have a problem for which an immutable distributed ledger technology is the best solution – that problem can be solved without blockchain, more quickly and with a lot less effort. VISA is just one of many full-scale, in-production examples.

It's sad that the Blockchain mania is so powerful that no self-respecting executive can risk ignoring it. So all the big banks and even, yes, VISA itself have blockchain based projects underway, nearly all of them involving widespread use of the future tense. I am open to the possibility that some of them may even be deployed in some way, once the people implementing them get realistic and relegate the blockchain technology itself to a tiny, marginal aspect of the project's code base, and realize that everything they're doing could be done better and faster without the distraction of largely irrelevant blockchain. But meanwhile, it's great for reputational enhancement and attention-getting!

A version of this post was published at Forbes.

How We Got Chatbots for Mobile Financial Apps

Chatbots for financial and other applications aren’t just a cool new thing. They’re a necessity! They solve the worsening problem of too many options to choose from on shrinking screens, with unhelpful help screens.

Do you access your financial accounts online? If you do, perhaps you’ll remember that the first time you tried to do something, you had the fun of poring over the menu system to find what button to click, sometimes only to reach another screen full of buttons and menus. On the plus side, online financial systems let you get a lot done. On the minus side, jumping through the hoops to actually get them to do what you want can be a long slog. Have you ever gotten frustrated and tried to click on Help? And gotten the long, unhelpful Help stuff? Helped a whole lot, didn’t it?

Today, nice big screens with high resolution, sitting on a desk somewhere, are used less and less. Cute, portable little screens with phone and camera built in are used more and more, partly because it’s in your pocket, right there when you need it. What happens to all those giant screens packed with menu choices? I guess the designers could have reduced the typeface so much that you’d need a magnifying glass to read them, but they bowed to reality and put just a few menu choices on each screen. Nice to read, but the result is that your multi-screen journey to get to the one you want got even longer. Assuming you remember how to get there. And what if you want something more elaborate, like how close am I getting to hitting my budget for restaurants this month? Fuggedabout it.

The designers of mobile apps for financial applications aren’t between a rock and a hard place. It’s worse. They’re stuck way back in a long, narrow cave that floods. What to do?

Echo and Siri have been training us to just ask for things. Who won the game last night? What’s the weather forecast? Even jokes! But banks … now that’s a different matter altogether. Banks are serious things. There’s money involved. Not just any money – MY money!

So what’s a bank app creator supposed to do?

It’s actually pretty simple, because there’s not much choice. If you want the bank app to be, you know, USED by the people who have it, you have to make it USABLE. Period. There’s not enough room for menus, except maybe a couple super-popular buttons. Help files? Waste of time. You’re down to one choice: make it so you can TALK (or chat) to the app, and ask it to do stuff for you.

That’s the logic. Even better, it’s really happening. In real life!

Bank of America is advertising its chatbot Erica heavily in some parts of the country. The reason is simple: they want their customers to be able to USE the BofA app, not be frustrated by it. There’s the chatbot technology provider Kasisto (Disclosure: Oak HC/FT is an investor) used by multiple banks to power human interactions. The bar has now been raised for financial institutions with apps.

Let’s review some history. Years ago, what financial institutions had to have was a website for customers to access their accounts. As smartphones spread, the bar was raised: OK, you’ve got a website, but do you have an app? Not having an app was a reason for customers to move to a place that did. Just as most of the financial institutions were breathing sighs of relief that they’ve caught up, the bar is raised again: you mean we have to make the app USABLE? What’s a chatbot, anyway? The pattern here is clear: technology keeps marching along, some financial institution applies it to their customers’ and their own benefit, and the others scramble to catch up. What’s next?

To find out what’s next, all we have to do is look at the non-financial domains that use chatbots. For example, look at Amazon’s Echo. It started out pretty primitive, but it’s adding capabilities all the time – as Amazon puts it, new “skills.” The bar is already being raised by technology vendors in two specific, technically challenging areas:

It’s one thing to answer a simple question, like what’s my balance? But real chatting as done by people requires that the chatbot take into account complex questions, context and history.

A complex question requires doing some real “thinking.” For example, “How much did I spend on eating out last month?” has loads of complexity. The system has to know that “eating out” means spending money on the merchant category “restaurants.”  It’s got to find all those transactions that took place in the month before today’s month, add them up and give you the answer.

Taking conversation history and context in account is even trickier. Suppose you get the answer to the eating out question. Suppose you now ask “ How about the prior month?” Easy for a human; for a bot, not easy at all! The bot has to figure out that you’re still talking about eating out (restaurants) and that you want the total for a month. It’s also got to figure out that you want not last month, but the one before that. These are the kind of amazing interactions supported by Kasisto.

Chatbots for financial apps are just being rolled out to solve the otherwise unsolvable problem of screen real estate. Even though we’re still in the early roll-out stages, the next battle is clear. Most of today’s chatbots are in elementary school – soon they’ll need to graduate to middle school!

A slightly different version of this post originally appeared in Forbes.

How Can an Immutable Distributed Ledger have Assets Lost or Stolen?

As I’ve described, cryptocurrency losses started long ago and have kept on mounting over the years. Most recently, the largest cryptocurrency exchange in Canada, QuadrigaCX,  has experienced a little problem that has resulted in what appears to be a permanent loss of over $130 million for its many customers.

The losses don’t seem to make any sense. My account is stored in a ledger that’s immutable – it can’t be changed because it’s locked down by unbreakable cryptography. It can’t be lost because the ledger is distributed, so even if a few computers are lost, there are still loads of computers with a copy of my unchangeable account balances in them. So how can it be that I can lose my money? To further increase security, Bitcoin and the other crypto-currencies don’t identify me by my name, but by a really loooong string of letters and numbers. And that’s just the name – to make any changes, someone would need access to my private key. Remember, “private key” isn’t just some password that’s easy to remember and maybe someone could guess – it’s a long number that's part of a proven-unbreakable cryptography scheme, totally guaranteeing that no one but me can access my accounts? It’s impossible for anyone to break through these layers of security, that math guys everywhere calmly assert just can’t be cracked. So what are these losses about???

The only reason this is mysterious is because the vast majority of stuff we read about Bitcoin, blockchain, distributed ledger technology and the rest is produced by adherents to the new cult. They’re promoters, not evaluators. Let’s start with encryption. Loads of things are encrypted – it’s hardly something that started with Bitcoin. Most sessions with websites are encrypted, as you can see by the https in the URL bar in the browser. I hope you only use encrypted WiFi. Email is encrypted. All databases run by moderately competent professionals are encrypted. And so on. Are websites, email systems and corporate servers hacked in spite of it? You know the answer. People can wave their arms and babble with passion about the power of unbreakable encryption all day long, but bad things happen in spite of it. Is it because encryption is in fact breakable? Nope! It’s because even the biggest castle with the most unbreakable walls has to have doors that are easy to open and close, though which properly authorized people and goods go in and out. Bad guys waltz right into the castle with unbreakable walls through the same easy-to-open doors that all the residents and servants of the castle use! How are those doors locked and guarded? Using absolutely proven, tested, tried-and-true methods such as … user names and passwords! And sometimes even PIN codes on top of it!!

Oh, no, everyone but everyone talks about how safe and immutable the distributed ledger is – how is it possible for it to be broken, and why are ancient, obsolete things like usernames and passwords involved? It’s simple, really. Say you’re an experienced break-and-entry person. You walk down a block at night. Most of the houses you see have fences, big locked doors and lights on. One of the houses is dark, and its front door is swinging free. Which house do you pick? Everyone scans for opportunities and goes for the weakest point. So no one even tries to break the cryptography – it really IS secure. But what about the place where the long ID and the no-way-can-I-remember-it private key are held? AHA! We’ve gotten to the weak point of the system: the wallet.

Wallets are just pieces of software that run on your phone, your computer or even online. There are dozens of wallets available. They all claim to store your Bitcoin securely, and maybe they do – except they need to allow a normal human being to actually use them, so they need to be opened by someone a normal human being can handle, like a user name and password! The super-secret but impossible-to-remember private key is stored in the wallet, among other things. Without the private key, you have nothing. If you lose or break the phone that has the wallet, you are screwed. And of course if your phone or computer is hacked, someone else will immediately transfer your Bitcoin out of your account into theirs. There is no recourse. None. Think that never happens? It’s been going on for years, see this. A 20-year-old college kid was just sentenced to 10 years in prison for stealing more than $5 million by phone hacking, see this. That was a first-ever case – usually, no one is caught.

Some people, concerned about the security of their wallets, have turned to professionally-managed exchanges, which both store crypto-currency and enable it to be converted and transferred in and out. An example of such a company is Quadriga, which was the largest such exchange in Canada last year. Now it has made various court filings, and is unable to return its customer’s roughly $200 million in assets, see this. Why? Because those assets were held in a wallet on the now-deceased founder’s laptop computer, and no one can get into the computer – its contents are encrypted and the access information was last seen somewhere in what was the founder’s brain.

Yes, the distributed ledger is secure and immutable. Yes, the cryptography used is in fact unbreakable. Most descriptions stop there, leading you to think you’re protected against loss or theft. As I hope is now clear, Bitcoin and its numerous offshoots do indeed involve some amazing technology, as I’ve described here. But that doesn’t mean that it’s any more secure or protected against loss than any other software. Because of its immaturity, it is proving to be more vulnerable to loss and theft than “normal” money. And the non-monetary blockchain solutions are just as vulnerable.

This post originally appeared on Forbes.

If You Like Private Blockchain, You Should Also Like Living in a Tent Instead of a House

Bitcoin is an amazing technology. I admire it. The central idea of how to implement a virtual currency with no one in charge, but where the “bank vault” is nonetheless pretty safe, is clever, as I explained here. However, the second you take this clever idea and apply it to situations for which it was not designed, it quickly becomes ridiculous – inferior by factors of thousands compared to existing solutions. It’s as though you liked hiking and camping in a tent -- and went back to your home on your suburban block, knocked your house down, disconnected from electricity, municipal water and sewer, stopped garbage collection, sold your car and bicycle, and gloried in your new, improved way of living during cold winter nights. Good idea, but wrong place, and there's probably a reason few people choose to live that way.

As a start for understanding why private blockchain is a ridiculous notion, let’s imagine that we all live in buildings with municipal water supply, and that suddenly someone decided it would be cool to live “off the grid” in a dry area where it does rain, but infrequently. How are you going to get and store the water you need to live? Obviously, you have to somehow make maximum use of the rare rainfall that happens. You construct as wide and varied a system of rain-catchers as you can. If you have a house with a roof, you arrange the gutters to go to downspouts to rain barrels. You carefully construct the rain barrels so they don’t leak, since water is precious, after all – the rain barrels have to be “immutable.” You also stretch out any canvas or anything else you can scrounge up to capture the rain before it hits the ground, and route it to barrels. You construct a set of pipes to connect the barrels, to make sure that all the barrels have water, and none has too much. You end up with a distributed set of barrels, each containing the precious water you need to collect and preserve. The system is even more impressive when it supplies a small encampment of people, with pipes distributing the water among the barrels, assuring that everyone has enough water.

 

Anyone wandering in the wilderness who encountered this maze of connected, distributed, immutable barrels and rain catchers would be impressed at what a good solution it was to the problem of having enough water when there’s no municipal water supply. Someone might come out to the place, take lots of pictures and blog about it. It might catch on, and some homeowners with regular water supply might be attracted to the notion of being ready to survive when civilization collapses and everyone will be forced to live off the grid. Most people, of course, will be happy to continue enjoying normal hot and cold running water, available by turning the faucet.

The Bitcoin solution was specifically designed when you really don’t want a municipal currency authority. Like the distributed water catchers and barrels, it’s a clever solution for exactly that problem. What happens when you decide it’s OK after all to have someone in charge – you’re not in the desert, you’re not a survivalist, and you just want a convenient water supply? What kind of sense does it make to somehow get a private corporation to be completely in charge of the system of water catchers, pipes and barrels in a place where connection to central water is readily available? Do you think the new system would be less expensive, more convenient and less obtrusive? Do you think the privately run immutable distributed water system, with all its barrels, catchers and pipes would be able to handle sudden demands like filling your pool or even a few houses running their lawn sprinkling systems at the same time?

That’s exactly what’s happening with private blockchain implementations. Every single vendor that so enthusiastically promotes its private blockchain tells you quietly what’s wrong, things like the transaction rate is worse by factors of thousands compared to normal DBMS’s (of course they don’t put it that way), and a host of other deficiencies that they’re overcoming … by step-by-step adapting standard database techniques first deployed decades ago and by now standard methods, and making an “improved” private blockchain. Improved, but still dramatically worse by all measures compared to standard technology.

Blockchain is a pile of new software that was designed to solve a very special problem that does not occur in normal life. Private blockchain is an attempt to take that highly unique solution, designed for wilderness living with no central authority in charge, and apply it to normal urban/suburban life with a central authority. The amazing, cool and different things about blockchain were invented specifically to solve the problem of having no central authority. The second you introduce a central authority, i.e., make it a private blockchain, all those special things that make blockchain unique suddenly become huge impediments, obstacles with no redeeming virtues.  It makes as much sense as camping out in your suburban back yard -- OK if you're a kid or want to give your tent a dry run, but nothing any sane person would think is an improved way of living.

This post originally appeared on Forbes.

Are Blockchain Smart Contracts Smart? Are They Contracts?

Smart contracts are all the rage in the blockchain world these days. They are the key feature that has pushed Ethereum to prominence. They’re everywhere!

There are just a few little problems. They’re not smart. They’re not contracts. They’re rife with security issues. And they violate the core principles that are supposed to make blockchain wonderful. Other than that, they’re great!

There is a huge amount of rhetoric and propaganda about what Smart Contracts are supposed to be. Here’s the reality: A smart contract is a software program. It’s written in one of a variety of mostly brand-new languages, chief among them Solidity. A smart contract is the exact equivalent in the blockchain world of a “stored procedure” in the database world; this means that it’s embedded in the blockchain and has access to its internal functions.

At first glance, smart contracts can seem like a clever idea that enables endless extensions to the underlying “immutable distributed ledger” technology in which they’re embedded, greatly extending their flexibility and fields of application. Let’s take a look at how this first glance holds up under scrutiny.

Here’s a typical explanation of smart contract:

“Here’s a very reductive way of establishing a smart contract: let’s say you and I have agreed that if I write you a history of bitcoin, you’ll send me $10 on my birthday this year. We can do that via a legally enforceable contract, which involves lawyers, notaries, and so on — or we can do it via Ethereum. In the latter case, you put $10 worth of smart coins in escrow, and when the terms of the contract are met, those coins are released to me. If I don’t meet the terms of our agreement, the coins are released back to you.”

The key, innocent-sounding phrase in the above description is “when the terms of the contract are met.” When I write you a history of bitcoin, will you accept it if it’s a piece of crap? Probably not. Here’s what has to happen with the deal:

• We have to agree on the terms of the deal
  • This happens verbally, no matter what mechanism is used.
• We have to express the deal terms in a mutually acceptable way.
  • In the land of normalcy, either our verbal agreement would be OK, or we’d have an email or paper exchange.
  • In smart-contract-land, someone would have to write a program in an acceptable language such as Solidity. We would both have to have wallets and accounts in the same crypto-currency among the hundreds that are out there. We would have to agree that the Solidity program expressed our mutual agreement. How good are your Solidity reading and writing skills? You would also have to deposit $10 in your account.
• I would write up my history and send it to you
  • In real life, I’d get it to you using paper or email.
  • In smart-contract-land, there is no good way to send an email – and putting an email address into a smart contract would make it available to the public! There are some clever hacks involving external services that monitor accounts in the blockchain, but there’s no direct solution. So I’d have to get my history to you using plain old real-life methods.
• A decision would be made about whether the history I send satisfied the contract.
  • This would be done in real life by you reading the history and making a judgment.
  • Smart-contracts would remain blissfully unaware of this crucial step, unless and until an amazing advance in AI/NLP is somehow embodied in them.
• If the terms of the contract are met, the money would be sent to me.
  • In real life, you’d hand me the $10, mail it to me, or electronically transfer it to me via one of the widely available methods, for example Venmo, which handled over $12 billion in transactions in 2018.
  • In smart-contract land, you’ve had $10 tied up in your crypto account since the contract was agreed to and the program – oops, I meant the “smart contract” – was created by one or both of us. You would then send a transaction to the contract to transfer the money to my account, after which I could convert it to normal money, with fees taken out along the way.

Someone please explain to me how using a smart contract to embody and execute our agreement is an improvement on normalcy?

In addition to the problems mentioned along the way, we’ve got these:

• If you love my history but want to cheat and not pay me, how does the smart contract help?
• If I’m worried about you paying me and want an enforceable contract, how does the smart contract help?
• If we make a written agreement, even a simple email one, at least we can probably both understand it. What are the chances that we are both fluent in Solidity?
• Assuming we somehow manage to write the code in some language, what if we’re less than perfect and have a bug in the code? The teensy weensy little problem with smart contracts is … they’re immutable (i.e., can’t be changed), along with the blockchain in which they’re stored!
• The key part of any agreement of this kind is the characterization of the history I’m supposed to write and the acceptance criteria. How can that be expressed or evaluated in lines of code?
• In terms of just moving the money quick and easy with minimal overhead, how is anything in crypto-land easier or better in any way than Venmo?

Smart contracts are a really cool idea. The best way to use them is in a sandbox where really smart, unemployed people can play games and make experiments, keeping them out of trouble and far away from real life and real problems that need to be solved.

This post originally appeared in Forbes.

Nobody Cares about Data

Nearly everyone professes to LOVE data. Just think about all the talk about Big Data, Data Lakes and the rest. Lies. Liars lying big LIES. Everyone says they like data ... until they get near it. Suddenly they develop fevers and rashes. They're allergic! Someone else will have to actually handle the data!

Data, the foundation of AI, ML, Analytics

All you have to do is get a job in one of these fancy subjects, and you quickly get hit with reality. When you were in school, you had wonderful exercises where you could develop your skills in deep learning, random forest, or whatever. Now in the real world, some older person assigns you to some juicy-sounding task where you'll get to use your skills. Where's the data? you ask. A wrist-wave of the hand tells you it's over there. You go over there and can't believe what you see. Why, it's nothing like it was in school! You try for a couple hours to clean things up. Then a couple days. It's still bad, but maybe good enough. So you run it through some models. Disaster. The system crashes and/or generates garbage. You complain. "Grow up," you're told. "This is what we've got to work with. Deal with it."

At the end of a year, you realize you've spent half your time in meetings of one kind or another, and 90% of the "working" time has been spent trying to get the data in order. With unsatisfying results. You've got some choices to make. You can lie. You can get into management, marketing or sales. You can roll up your sleeves, forget the fancy stuff you learned in school, and become a data clean-up specialist, which is actually more like a create-decent-data-from-scratch specialist. Which is NOT what you signed up for. Waaaaahhhhh.

What's maybe worse of all is the status. AI and machine learning are clearly the prestigious upper floors of a grand apartment building. Deep learning thinks it's the penthouse, but whatever. The lower floors are occupied by simple analytics. The ground floors are occupied by people managing the databases and Hadoop clusters, and maybe even some ETL tools.

And then there are the basements. The sub-basement where the garbage chutes end. Where the janitors live. Where the crap from the elegant apartments is taken to be discarded. Where the water and oil and natural gas enter the building -- the things the fancy people on the upper floors need to wash up, keep warm and prepare to dress elegantly. That's the floor ... and the status ... of the data specialists. 

You can tell yourself until you're blue in the face that without good data, none of the fancy stuff would work. It's the foundation, dammit! The janitors probably tell themselves the same thing about the heat, cooling, hot and cold water, cleaning and garbage removal. True -- but they're still janitors, wearing a uniform and passed in the halls by the upper-floor people as though they don't exist. 

Bad data equals bad results

There's a simple reason why the incredible potential of the Big Data movement has now morphed into AI/ML and is even incorporating Blockchain. The time passes, tick. Tick. Tick. Tick. No results! Uniform use of the future tense! Claimed successes aren't really, when you dig into them.

Some of the reason is typical organizational incompetence. But much is also due to the fact that we are swimming in a sea of big data and no one wants to clean it up! It's so bad, we mostly don't acknowledge it; much easier just to ignore it.

This problem isn't new. See this:

I've talked about the importance of data as the foundation of AI/ML here. I've illustrated the horrendous problem of bad medical data here. Even basic data, like what providers are where, is wrong too often. By the way, these illustrations should be considered informal tips of massive icebergs. When I talk with true experts who are themselves knee-deep in this stuff, I find the situation is ... even worse.

As today's illustration of the problem, let me show you a piece of mail I got. It's from a major corporation, one of the big regional cable companies and internet service providers. They've got decades of experience working with customers in their geography. They've got to know every address, every household, with complete histories of using their service, dropping it, signing up again. How could they not know the basic demographics and the kind of approaches that work and the ones that don't?

Here's the mail I got (I blocked out the street number):

Looks OK, right? Nice and clear. Specific to the town, so it feels personal. Lots of good things about it. They even designed the envelope so you could see the plastic card on the right, with an eye-catching banner over it.

There's just one little problem. JO Black died in May 2001. More than seventeen years ago.

I don't think there's anything else I need to say except, good job, Optimum! You're doing a great job illustrating the near-universal toxic, rotten ocean of data in which we swim, and doing your part in keeping it that way.

Wait, you might say, this is a trivial little problem. In a way, it is: one piece of mail that shouldn't have been sent. But it's an illustration of a problem that's broad and deep. The notion that a wrongly sent piece of mail "means nothing, is trivial" is an attitude that is EXACTLY why people who care about data metaphorically wear uniforms and work out of the basement. Maybe Optimum is worse than all the rest. Sorry, they're not. JO Black gets a VERY slowly diminishing stream of mail at this address from a wide variety of vendors, large and small. So does Mrs. Grace Black, who died 4 years ago. So does Ms. Jessica Black, who lived here for awhile before moving 20 years ago. So do Mr. Samuel Black and Ms. Elspeth Black, who never lived at this address.

The Problem is Everywhere

To be totally clear: the problem isn't just wasteful mail solicitations. It's everywhere, and every stage of data collection and utilization. The problem with healthcare data is immense, for example, as I've illustrated often. Bad healthcare data, which is ubiquitous, has the direct result that normal, innocent people needlessly suffer and die. It doesn't get better, because all the smart people and the important decision-makers are busy attending conferences about how AI is transforming medicine and how blockchain will solve all the medical data problems -- leaving the ragged crew of people who are supposed to fix the problem ignored in the dank basement, spending their time scheming how they can at least get to the first floor, since it's perfectly obvious that no one is actually interested in ... fixing the data problem!!!

Conclusion

Everybody says they want data. BIG data. But what they really want is a springboard to do something prestigious, which turning a toxic stream of severely polluted data into something textbook-clean is not. While hardly the only factor, this is a major factor in the widespread untalked-about failures of fancy modern techniques to deliver practical results. The plain fact is, nobody cares about data.

The Hierarchy of Software Skills and Status in Data Science

There is a striking hierarchy of skills in software, as I've explained here. When you dive into any particular aspect of software, you usually find that it's got a hierarchy all its own. Data science is a subject of intense interest these days, so in this post I'll explain some of the basics of the data science skills hierarchy.

A skills hierarchy is very much an insider's game. What most people care about is status. I talk about the basics of software status here. Remember, the skills hierarchy is a whole world away from the status hierarchy that most people care about. Don't confuse the two!

Data Science skills

First and foremost, it's important to understand the incredibly broad range of subjects covered  by the term "data science." I attempt to explain the basics of the range in this blog post. You can be just amazing in one of those subjects, while being a neophyte in one that the outside world may consider to be "related," but which in practice is not just down the hall, but in a different building on a different campus.The general understanding of this range is SO pathetic that "data science" is typically managed as a completely independent, free-standing group. Which makes about as much sense as believing that a sous-chef belongs in something that isn't a kitchen. Or that everything that everyone does in a kitchen is basically the same thing.

Here's one cut at the hierarchy in data science, starting from the base:

1. Tool users.These are people who have learned how to use some software tool, or maybe a couple. Most "data scientists" fall into this category.
   • They don’t understand how the tools are built or any of the underlying software
   • They may know their tool, but aren't real clear on what the other tools are about, much less when you might consider using one.
2. Some have broader knowledge of tools
3. Very few have the sophistication to understand real data analysis, per my series of AI/ML; see this and the links in it: http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-3-closed-loop.html
4. Of those, even fewer can understand the underlying algorithms and follow the latest research literature
5. Of those, even fewer can make real algorithmic advances and implement them as tools and deliver practical value.
6. Of anyone who can do all of the above, it is rare to meet anyone who can address and solve deep tool-level problems in other domains needed to make their code practical in the real world.
7. Finally, it is extremely rare to meet people who, in addition to their deep and broad prowess in data science and relevant skills needed to make it real-world practical, have similarly deep skills in an associated domain needed to make the data science fully effective for a business.

The issues of this hierarchy are compounded by the usual over-selling by people who are good promoters but little else, and corporate/government big-wigs who don't want to be bothered with details, but are keen to be seen as "doing something" on such a high-visibility topic. Getting results that make a difference is pretty low on the typical priority list.

And then of course there are the "data scientists" themselves, who most often are sincere people who are trying to do a good job as they've been taught to do it -- mostly by professors and others who have no idea what real-world success looks like, much less how to bring it about.

Finally, there is the usual "manage something that's invisible to you" phenomenon I have often discussed in this blog, which leads to so much dysfunction and so many wonderful Dilbert cartoons.

Conclusion

People talk as though "data science" were a thing, with the usual kind of hierarchy based on level of management and/or "experience." Those typical patterns of hierarchy just don't cut it for understanding what's going on in data science, just like they don't cut it for understanding software development. We will continue to see waste and dead-end efforts until we at least make a start at making our understanding of data science more sophisticated, and aligned with the facts on the ground.