The Black Liszt

Surprising Bug at Amazon About a Bug Repeller

I have found ample reason to mock the golden-glowed tech reputations of most of the tech giants, in addition to the supposed tech prowess of organizations such as the NSA. There is good reason to believe that old-style libraries are more secure.

I recently stumbled upon a rather glaring bug or result of hacking at Amazon -- and found that Amazon provides no way I could find to report the problem.

The problem was glaring and amusing -- a whole set of over 3,000 reviews of a book attached to a pest repelling product. As I'll describe in a future post, this isn't just a bug concerning bug repellers -- it's the tip of an iceberg about the foundations of AI/ML.

Here's how I found the bug bug.

Stumbling on the bug at Amazon

I was looking for a product to scare away some pests that have found their way into my house. I gave a close look at the following product:

I immediately noticed and was impressed by the large number of reviews, and how favorable they trended. What a popular product! I've got to look at this one. I scrolled down:

Things are still looking good. More details:

Uh-oh! Right after we see that it has 3,051 reviews with an average rating of 4.8, which is super-good, we see that this pest repeller is #2,695 ... in Books!! Something is seriously wrong. Scrolling down, we see some really weird answers to questions:

A couple of the people answering clearly tried to tell Amazon there was a problem.

This next one blew me away -- over 3,000 reviews for a pest repeller??!!

Now we know for sure something's seriously wrong -- a pest repeller is not Children's Literature!

Here is the nail in the coffin, the pictures of the product and a list of the phrases that frequently appear:

 

And here's one of the glowing reviews. It brought tears to the reviewer's eyes, and not because of the strong odor generated by the pest repeller:

 

What's going on, Amazon? I don't usually get to mock you. It's usually the other tech giants that receive my sarcasm, places like Google, Facebook, Twitter, Apple and Microsoft. Amazon has done a reasonable job maintaining quality and growing into new areas. This is such a peculiar bug -- it's as though some disgruntled employee were messing with things to show his displeasure with the world.

As we'll see in a following post, this is not an isolated bug, and it has implications that go far beyond Amazon -- implications concerning the glorious future promised by all the AI/ML enthusiasts.

Laser Disks and Workflow Illustrate the Insane, Fashion-Driven Nature of Software Evolution

Computers are so exact and numbers-based that we tend to think of the people in charge of them as white-jacketed scientists with esoteric knowledge driving towards calculated optimal results, much the same way we imagine that the scientists and engineers calculated the path of the Apollo capsule to the Moon. If the Apollo program were run the way most computer projects are, it would have been a miracle if the capsule made it off the launch pad, much less traced an incredibly exact path from Cape Canaveral to the chosen landing spot on the Moon, a journey of over a quarter million miles.

The reality is that the application of computers and software to automation is painfully slow, usually failing to achieve optimal results by large margins, and often lagging behind what is achievable by decades. The reasons appear to be a semi-random mixture of commercial interests, technology fashion trends, ignorant and risk-averse managers, and technical specialists rooted in the past and wearing blinders. That’s all!

Here’s the story of a typical example.

Microfilm, document imaging and laser disks

Long before computers, microfilm was the preferred medium of librarians and archivists to preserve paper documents for future use. Microfilm was more compact than paper, and lasted much longer. What happened when computers came along is a curious and educational story, a prime example of how computer use and software evolve.

Disclosure: I lived and worked as a participant in this history during the late 1980’s and early 1990’s. The story I will tell is not systematic or comprehensive, but like a traveler’s tale of what he did and saw.

A side show in the long evolution of computer storage is the laser disk. A laser disk has data recorded onto it either as a whole, during manufacturing, or incrementally, as part of a computer system. The laser disk had quite a run in the consumer market in the form of CD’s, and then DVD’s. They were an improvement on existing methods for distributing digital content of all kinds.

In the computer industry, they took the form of WORM (write-once-read-many) drives, and were used to record archival data that should not be updated, but could be read as often as needed. The technology, as often happens, went searching for problems it could solve. In an effort to reduce human labor more than microfilm could, jukeboxes were invented. Each jukebox had a disk reader and many shelves for disks, with a mechanism to load a selected disk into the reader. FileNet, now part of IBM, built one of the first of these systems in the mid-1980’s.

At around the same time that WORM drives became practical, paper document scanners were evolved from systems to capture images of the paper onto microfilm. In each case, light was shown onto paper and focused onto a target; a scanner replaced the film with an optical sensing device.

How could we get people to buy WORM drives and jukeboxes, the creative marketing people wondered? The departments that put documents onto microfilm weren’t going for it – the new systems were much more expensive, microfilm wasn’t accessed often enough to make it worth connecting it to computers, and the records retention people were understandably skeptical that any computer disk would be readable 100 years from now, as microfilm will be.

Adding workflow to document imaging

I have no idea who made the audacious leap, but some creative person/group came up with the idea of doing computer document scanning at the start of the process, when paper arrived at a location for processing, instead of microfilmed at the end for archival purposes. But WHY??? The creative types scratched their heads until they were losing hair rapidly. Why would anyone make responding to loan requests or whatever even longer than it is today by introducing the step of scanning?

GENIUS TIME: LET’S INVENT “WORKFLOW”!!!

Everyone has an image for what “workflow” is. For documents, it’s the path of processing steps from arrival to filing. In larger organizations, there are often multiple departments involved in processing a document, so there are in-boxes and out-boxes and clerks transporting documents from one to the other.

Rubbing their hands and cackling, the scammers worked out their strategy: We’ll scan the documents and convert them to images so that we can move images from place to place electronically instead of piles of paper! We’ll call it “document imaging workflow.” It will deliver massive benefits to the whole organization, to everyone who touches the paper, not just to the tiny group who files and archives at the end – and storing the scanned documents onto WORM drives will do their job too! Hooray! We will leap-frog the old-fashioned, paper-flooded back office into the modern electronic age. What’s not to like?

It was audacious and brilliant. It was a scam that would wither if confronted with just a little common sense or cost accounting. Which, true to the pattern of such fashion-driven trends, it never had to confront!

Implementing workflow

A typical target environment for implementing workflow was large offices with large numbers of documents, often forms of some kind, arriving every day. The forms would go from the mail room to the relevant people’s desks for processing. The person handling the form would often have an IBM 3270 terminal for interacting with a mainframe computer. When the person was done with the form, they would place it in one or more outboxes for further processing at other desks, or for filing. Sometimes documents would go into file cabinets for a period of time, but all would end up being archived, and sometimes microfilmed for permanent storage.

Implementing workflow first of all required scanning the documents as a first step, and storing them immediately on normal computer disks, and ultimately on WORM drives, thought to be the equivalent of microfilm. Once scanned, the documents would be managed by workflow software, which would route them to the appropriate desk in the appropriate department for processing. Every desk needed to have the old computer terminal replaced or augmented with a large, high-resolution image terminal, capable of handling at least the display of the document, and sometimes also the computer screen. The old, slow connections between terminals and mainframe needed to be replaced with a fast local area network, and there needed to be substantial servers for handling the local image handling and workflow routing.

Of course, lots of analysis and programming was involved in addition to the equipment. Workflows had to be analyzed and programmed, and all the management, reporting and exception handling taken care of.

The benefits of workflow

When a movement like document imaging workflow has a head of steam up, no one seems to ask whether it’s a good idea, and if so, how good of a good idea it is – quantitatively. When I was involved, everyone threw around there would be at least a 30% productivity improvement due to workflow, and that would make all the expense and disruption worth it. I never encountered a single study that measured this.

Think about it for a minute. Would looking at an image of a document make you work faster than you would looking at the original paper? Would picking the next paper from the in-box be slower than getting the system to show you the next image? What about the labor of the clerks moving stacks of paper from one person’s outbox to the next one’s inbox? It would certainly be saved, but chances are a single clerk could handle the paper movement for many dozens of people. And remember, there’s the scanning and indexing that’s been added and the massive computer and software infrastructure that has to be justified.

It’s obvious why no one EVER did a cost-benefit analysis of workflow – the back-of-the-envelope version easily shows that it’s a couple zeros away from making sense.

I remember a couple of out-of-it, tech-fashion-disaster know-nothings mildly thinking out loud about how workflow could possible be worth it financially. The immediate response from workflow fashion leaders was upping the ante – don’t you know, the wonderful workflow tool from (for example) FileNet lets you program all sorts of efficiencies for each stage of work; it’s something we really need to dive into once we get the basic thing installed. End of subject!

So who fell for this stuff? Just a who’s-who list of major organizations. Each one that fell for it increased the pressure for the rest to go for it. None of them did the numbers – they just knew that they couldn’t fall too far behind their peers.

Conclusion

It’s hard to think of a field that’s more precise and numbers-oriented than computers. Who has a clue what actually goes on inside those darn things? And the software? There are millions of lines of code; if a single character in any of those lines is wrong, the whole thing can break The impression nearly everyone has, understandably, is of a field that is impossibly precise and unforgiving of error. When the software experts in an organization say that some new technology should be adopted, sensible people just agree, particularly when there’s lots of noise and other major organizations are doing it. It can be even more gratifying to get known as pioneering, as one of the first organizations to jump on an emerging tech trend!

Somehow, the smoke and noise from the software battlefield is so intense that the reality is rarely seen or understood. The reality, as I’ve illustrated here, should result in everyone involved being sent to the blackboard by a stern teacher, and being made to write, over and over: “I pledge to try harder to rise above the level of rank stupidity next time.”

Simple Data Entry Technology Illustrates How In-Old-Vation in Software Evolution Works

Since when is “data entry” (entering data into a computer) a pivotal, innovative technology? When the difference between doing it the normal way and doing it with advanced technologies is a …ten-to-one productivity difference … that’s when.

I’ve described how the Operations Research algorithm of Linear Programming is fifty years into an agonizingly slow roll-out through different applications, from scheduling oil refineries in the 1960’s to scheduling retail sales in the 1990’s, and now scheduling medical infusion centers and operating rooms in the late 2010’s. In each case, laborious and error-prone human scheduling was replaced by the algorithm, with improvements ranging from no less than 10% to over 50%. This is major! Why did such an innovation wait for decades to be applied, and for many applications, is still waiting!!?? This is the mystery of how what’s called “innovation” works in reality, and why it should be called “in-old-vation” instead.

You may think that part of the cause is that LP is an exotic algorithm – even though it’s a standard part of the Operations Research engineering curriculum, most so-called normal people haven’t heard about it. While it appears that even the hosts of people who wax eloquent about AI and ML are clueless about LP, it’s not exactly a secret. So let’s see if obscurity is the reason why LP remains a “future innovation” in many potential applications, by examining the super-plain, ordinary, completely-understandable-by-normal-people case of data entry.

Data Entry

Just as process optimization is done by hand or stupid methods for decades until some genius comes up with the brilliant idea of applying tried-and-tested LP to the problem and dramatically improves it, so is Data Entry widely performed by primitive methods until some “innovator” comes along and applies “Heads-Down Data Entry” (HDDE) methods to the process – and typically gets improvements of 3 to 10X! The only difference is that while LP is taught in Engineering departments and studied by math nerds, Heads-Down-Data-Entry is “just” a collection of common-sense techniques that require no math and no professors to understand or implement. It’s so “common” that it doesn’t even have a generally accepted name – though it’s been implemented in many places and been thoroughly proven in practice. It’s far too humble to merit an academic department – and yet, when applied, has delivered truly massive gains, far higher proportionately than exotic Linear Programming has!

The methods of HDDE were first implemented in places that had huge volumes of repetitive data to be entered into computers. Banks were early users for check entry, and so were the credit card companies who, at the start, had huge volumes of paper charge slips to process. Simple ideas like minimizing keystrokes and eye movement were implemented, and then taking advantage of the eye-to-fingers pipeline, when people noticed that showing clerks the next item to be entered before the current one was complete led to a big jump in speed. Other methods like double-blind techniques were invented, so that entry clerks just entered – whether their work was original or used to check someone else’s work was entirely handled by the Data Entry system.

As soon as scanning and image display became practical, HDDE adopted them. That led to another jump in productivity, enabling large, complex forms to be broken up into pieces, so that a clerk would see the image on a screen of the same piece of data from a whole set of forms instead of entering a whole form from start to finish. No HDDE shop would even consider having the entry clerk think about anything on the form, stuff like “if this field is missing, do this instead of that,” because it would just slow them down.

Finally, there’s ICR (Image Character Recognition), which is having the computer “read” the image instead of a human. This technology has existed for many decades. Once you’ve got HDDE in place, phasing in ICR is a natural, so that the proportion of entry done by humans gradually decreases as the effectiveness of ICR increases.

Remember, applying LP to scheduling might result in a 30% improvement, which in most cases, is major to the point of being revolutionary. What about HDDE? Entering data from a paper form into a computer using primitive methods might get between 1,000 and 1500 KPH (keystrokes per hour). There are lots of stages of improvement, including things I’ve mentioned like eliminating the thinking and breaking up the form, but levels of 10,000 to 15,000 KPH in a professional environment are widely achieved – with superior quality. That’s a minimum of 5X! Typically much more. Of course, as you incorporate ICR into the process, it gets even better, gradually reducing the human factor, so that most fields are entered with no human involvement. At this point, the technology is probably best called ICR+HDDE, though there is no generally accepted term.

Given all this, it would be insane to handle computer data entry by anything other than HDDE methods, right? Welcome to the software industry, where insanity of this kind is the accepted state of affairs. And where almost no one practices the most simple and basic of computer fundamentals, such as counting.

How HDDE gets implemented

I described how Linear Programming went from problem domain to domain, each time acting like an innovation, as indeed it was in that area of application. Once it gets established, it tends to stay. The case of HDDE is different, I think because it’s not a recognized “thing” in the halls of academia, or among the poo-bahs of big business. It’s the kind of thing that no self-respecting Professor of Computer Science would stoop to consider, assuming he ever encountered such a low-status thing – you know, the kind of thing that “merely” makes common sense and, well, works.

HDDE has appeared in competitive, high-volume service businesses, where it has a major role to play in delivering results for the customers of the business. There have been software products that directly support HDDE, so that all you have to do is buy and implement them. It’s neither obscure nor hidden. But it’s never been talked about at conferences as the “coming thing.”

Case Study: HDDE rejected

In the early 1990’s, when document imaging and workflow technology were hot and something people talked about the way they talk about AI/ML and innovation today, the government-backed student loan organization, Sallie Mae, decided to apply the technology to improve the operations at the handful of processing centers they had at the time, employing many thousands of people and processing millions of documents a year. The popular thing to do at the time was to scan documents on receipt, and then send them to the same places the paper was sent, so that workers could process the images of the documents displayed on new big screens instead of paper. The job was basically to type the data into the right places of the software application they used.

Everyone at the time said that converting the documents was important ONLY because it enabled wonderful workflow, the elimination of inboxes and outboxes for paper. And the bits of other stuff you could do, like having a group of people taking from a common inbox instead of each having their own. The common “wisdom” was that you could gain 30% productivity improvement by implementing this marvelous new technology.

I got involved, since I was a recognized expert on document imaging technology at the time, and had personally coded one of the early workflow systems. I figured out and showed in detail that by canning the workflow and implementing HDDE techniques, they could gain a minimum of 5X productivity improvement. No one disputed my thoughts or detailed plan. They just ignored it, and proceeded to implement the standard stuff. I strongly suspect that after considerable time and expense, there were walk-throughs of the Sallie Mae sites showing visitors the big screens and absence of paper – what a big success the project was!

Case Study: ICR-HDDE applied with success

There are two current cases I know of where ICR-HDDE is being applied and winning. Each are classic, narrow service businesses where converting forms to data is the key value of the business, and where the companies buying the service just want fast, accurate data delivery, they don’t care how it’s done. Disclosure: each of these is an Oak HC/FT investment.

At Groundspeed, insurance forms and reports are captured and the relevant data is extracted from them by the most effective relevant means, often involving forms of ICR-HDDE. There is lots of forms recognition from documents and images that are often computer output, with the relevant data appearing at varying places on a page. Nonetheless, Groundspeed is able to deliver the data stream the customer needs, quickly and accurately, The results are so powerful that new levels of analytics are enabled by the newly available stream of structured data.

At Ocrolus, financial documents of all kinds including bank statements and pay stubs are converted to data in a standard format to enable fast and effective operations like giving loans for business and personal use, along with a growing list of other operations that also need good data. An effective combination of ICR-HDDE techniques are applied to get results for companies that need accurate data to make fast decisions.

Conclusion

HDDE is a collection of methods that have been proven in practice for many decades. The technology that is behind it continues to deepen and reduce human effort even more, with the addition of ICR. But it remains a niche technology, ignored by the numerous places that could benefit from it, even more than LP.

The big difference between LP and HDDE is that LP is a formal piece of magic that’s in academia. HDDE is nowhere. In fact, it’s really just an example of classic industrial engineering applied to computer software and the people who use it. Which makes it all the more mysterious that it's largely ignored.

In-old-vation is real. Most “innovations” are minor variations on things long-since proven and demonstrated in practice, but are unimplemented in the many situations that would benefit from them until some mysterious combination of circumstances arises to let them explode into practical reality.

Facebook's Libra Cryptocurrency and the P2P Apps Venmo and CashApp

Facebook is working hard on building a brand-new cryptocurrency system called Libra, sort of like Bitcoin and Ethereum, except it will be much better, at least according to Facebook.

With all the talk about Libra, cryptocurrency, regulation and the rest, no one seems to wonder about what existing solutions normal people will be using to solve the problems for which Libra is suited. This isn’t strange at all actually – in all you’ve read about Facebook’s Libra, how much have you read about the pressing problems it will solve, the unmet needs it will address – right? Mostly what you read about is how Libra will solve all sorts of problems that today’s crypto-currency systems have, how many partners they have and how wonderful it will be.

Libra will be an infrastructure “out there” somewhere, with lots of important people and organizations making sure it’s wonderful. But in practical terms, most people will use it via an e-wallet, an app that they install on their smart phones. That’s where a name that hasn’t appeared a great deal pops up: Calibra. Calibra is a newly-formed Facebook subsidiary that will be the e-wallet for Libra. It will “integrate with” Facebook’s WhatApp and Messenger, giving it incredible consumer reach.

You can read all about what it will do, but it’s basically an e-wallet for holding e-cash and providing basic functions like sending and receiving e-cash to and from another e-wallet. Except for the little “detail” that instead of sending real money, you’re sending the Libra cryptocurrency, and will have to go to an additional step to move dollars in a bank account to and from your Calibra wallet, converting to or from dollars along the way.

Putting aside the fancy new terms of cryptocurrency and the rest, does using a phone to make person-to-person payments remind you of anything? How about Venmo, the P2P e-wallet used by over 51 million people, which is now part of PayPal? How about Cash App, the rapidly growing P2P e-wallet installed in about 60 million phones?

These are proven consumer applications which have already gone through numerous upgrades and feature additions, used at least weekly by tens of millions of people.

Facebook has incredible reach, and billions of cash in the bank gotten by selling your private information to advertisers. They will certainly make a lot of noise. How does Facebook's proposed system compare to Venmo and Cash App?

• Venmo and Cash App just use dollars. Simple. Facebook will use the newly invented Libra, which needs to “work,” something Facebook isn’t good at doing.
• If you split a bill and need to send $7.30 to your friend, you just do it with Venmo and Cash App. With Facebook, you’ll have to convert it to Libra, send that, and have it be converted back. Hopefully the exchange rate won’t move too much.
• Venmo and Cash App support free P2P payments. The Calibra website claims it will be “low cost,” but they have yet to say what the cost will be; after all, there is a HUGE cryptocurrency infrastructure to support, none of which is needed by the existing cash apps.
• It’s easy to imagine Facebook will find a way to sell the information about your transactions to the highest bidder, or somehow find a way to "monetize" what you do with your money. It’s what they do!
• Libra and Calibra will work for international payments! With exchanges from local to Libra to foreign currency, two exchanges instead of one. That’s certainly something Cash App and Venmo don’t do today, and will appeal to some fraction of a hundredths of a percent of the market. Except for the proven massive cryptocurrency uses for money laundering and international crime, who will have yet another channel to support their illicit activities.

Facebook's Libra is getting all the attention any giant corporation could want, including some attention I suspect they'd rather not have, from regulators. But in the end, will they be able to make this massive software effort work? Will it do anything consumers want better than existing apps like Venmo and Cash App that are in widespread use? There is good reason to be skeptical.

This was originally posted at Forbes.

Understanding Software Evolution Helps you Predict the Future of Software

There are patterns in software evolution. Because almost no one studies software history, and even fewer study software evolution, these patterns are almost never discussed.

The patterns are amazing. In some cases, you can be pretty sure that a trend that “everyone” says is going to be the future will fizzle out. In other cases, you can predict with a high degree of certainly that software of a certain definite kind and description will be built – even though it hasn’t been built yet – it hasn’t been built anywhere by anyone, but nonetheless you know it will be built.

The reason I started to study software evolution is that it’s just plain interesting. The patterns that emerge from it are striking and educational. But most important, knowing software evolution can help you predict the future!!

Knowing software evolution can help you predict the future for the simple reason that software does not evolve the way pretty much everyone thinks it does. It is often the case that, when a new software system appears on the market that has a big impact, people who know the patterns were sitting around waiting for it to happen! They knew it was coming!

I remember living in New Haven CT in the early 1970’s. I was a vegetarian. A new restaurant had just opened, a kind I’d never been to, an Indian restaurant. I went, and it changed my culinary life. Indian food – made by vegetarians for vegetarians, healthy and wonderful flavors!

The new restaurant was definitely new in Connecticut, a true innovation. The owners had to go through quite a bit to open it up, get supplies, etc. But was it otherwise innovative? Did the owners invent a single one of the dishes or recipes? No, of course not. Just like with a piece of software, they had to create it from scratch in a new setting, making appropriate adaptations, and so to the locals, it was new. But in reality, it was a newly adapted instance of a proven winner. Software works the same way – a piece of software that is a big new winner, the first of its kind, has most often been proven elsewhere.

The first Indian restaurant in New Haven could have opened years earlier or later than it did, with different people doing the work. There was no way to predict when the Indian restaurant would have opened, or who would have done it. But isn’t it obvious that, in the context of ethnic restaurants enjoying increasing success and the growing Indian population that someone would have opened an Indian restaurant there? Shockingly to most people, software works exactly the same way! In a surprising number of cases, you can’t predict WHO or WHEN, but you can sure as heck predict WHAT will be built and even WHERE (i.e., in what business domain).

Here is a specific example of software that resembles in important ways the example of Indian restaurants. Instead of Indian food, the star is a specific optimization method within the field of Operations Research (OR) called Linear Programming (LP). While most people know there are loads of Indian people in India and elsewhere and they have a unique cuisine, understanding this is helped by the fact that we can picture India on a map of the globe and we all eat some kind of food. How many people have even heard of OR or LP, or have seen and understood software of any kind? To be clear, what I mean by "having seen software" is seeing the source code, not the results of it on a screen. This LP software sits around, widely used in some domains, and completely unknown in others, just like there were no Indian restaurants in New Haven before a certain date.

Knowing this helps you predict the future of software, because in a shocking number of cases, it doesn't need to be invented, but "merely" imported into an area of use where it is not currently in use. That is a big reason to be interested in understanding software evolution.