The Black Liszt

Job requirements for software engineers should stop requiring CS

I've read thousands of job requirements for computer programmers over the years, and written or edited quite a number. I’ve interacted with hundreds of software groups and seen the results of their work. I’ve spent a couple decades cranking out code myself in multiple domains. There are a couple near-universal problems with job requirements that, if changed, would improve the quality of software groups and their productivity.

Of course, it’s not just the job requirements and what the hiring people do – it’s also the managers, from the CEO down. They also have to not just support but champion the changes I describe. If they do, everyone will enjoy better results from the software team.

In this post, I'm going to concentrate on just one of the issues: academic degrees

A near-universal job requirement is an academic CS degree. When analytics, ML or AI is involved, the requirement is often “upgraded” to a Master’s or PhD.

There are many capable programmers who have degrees of this kind. Often it doesn’t seem to hurt or hold them back much. But all too often it does! The more specialized the training, for example in project management or quality, the more likely the “education” the person has received is an active impediment to getting good work done.

Here’s the simple, raw, brutal fact: Getting a degree in Computer Science does NOT train you to become an effective programmer in the real world. All too often, the degree results in the person performing worse than someone with self-training and apprentice experience.

That is the fact. Surprised?

Let’s do a little compare and contrast with medicine. Yes, I know that an MD is a graduate degree, while CS is often undergrad. Medical training has evolved to what it is now after decades and centuries of finding out what it really takes to help make people healthy. By contrast, CS degrees just started being granted 50 years or so ago, and are far from even trying to figure out what kind of training helps create good programmers.

First let’s look at the training and testing:

• You don’t even get into med school without taking the MCAT, a challenging test that takes over 7 hours that few do well on.
• Once you’re in med school you take a four year course to get your MD.
• The first two years are academic, including hands-on labs. Then you take the USMLE-1. If you don’t pass this challenging test you’re out. End of med school.
• The second two years are clinical! You’re in hospitals, clinics and offices seeing patients under supervision. And you’re graded. And then you take the USMLE-2, which is harder than part 1 and has lots of clinical stuff. If you fail, you’re not an MD.
• To practice, even as a general practitioner, you have to apply and be accepted into a Residency. Depending on specialty, this can be 3-7 years of mostly hands-on practice, under close supervision.
• During your first year, you have to take and pass the USMLE-3. Fail and you’re out.
• During your last year you have to take and pass the test specific to your specialty. Fail and you’re out.

Here’s the equivalent of the training and testing in CS:

• There is NO equivalent in CS. No entry testing. No exit testing. Just grades on courses determined by professors who usually pass everyone.

A little compare and contrast between medicine and CS:

• Medicine is taught by doctors who practice medicine
• CS is taught by professors, most of whom have never practiced programming in the real world.
• A large part of medical training is working with real patients with real problems, under the supervision of practicing doctors.
• CS is primarily classroom teaching with textbooks and homework exercises. You have to write programs as exercises, but it’s completely artificial. There is nothing apprentice-like or truly clinical about it.
• Medical training is led by doctors who are incented to produce great doctors.
• CS training is led by academic PhD’s with no real-world experience who are incented to publish papers read by people like them.
• Medical journals publish essential information for practicing doctors, giving advances and new discoveries.
• CS journals are read by the tiny group of academics who publish in them. Practicing programmers pay no attention for good reason.
• Bad doctors are fired for incompetence and barred from practicing.
• CS graduates are rarely fired for incompetence. If CS graduates can’t program well, they usually shift into using their non-skills in “management.”
• In medicine, best practices are increasingly codified. You rapidly fall under scrutiny for deviating.
• CS grads seek out and follow fashions that are the software equivalent of blood-letting, enthusiastically promoting them and getting them adopted with disastrous results.
• Hospitals are compared with each other in terms of results. It’s not hard to find which are the best hospitals.
• Groups of CS grads make it impossible to make comparisons between groups, with the result that huge groups produce major disasters at great expense, while tiny groups of effective programmers perform 10X or more better.

All this doesn’t make things uniformly wonderful in medicine. But it goes a long way towards explaining why software is so bad. It’s awful. The awfulness is so widespread that it’s rarely talked about!. If bridges fell down at 1/100 the rate that software projects fail, there would be a revolt! Instead, everyone in the industry just sighs and says that’s the way things are.

You think things are great in software? Check out a couple of these:

https://www.blackliszt.com/2015/09/software-quality-at-big-companies-united-hp-and-google.html

https://www.blackliszt.com/2014/12/fb.html

https://www.blackliszt.com/2014/02/lessons-for-software-from-the-history-of-scurvy.html

The fact is, CS training leads to horrible results because Computer “Science” is roughly at the same level as medicine was when bleeding patients was the rule. See this:

https://www.blackliszt.com/2019/11/computer-science-is-propaganda-and-computer-engineering-is-a-distant-goal.html

Conclusion

There are lots of things you can do to improve the results of hiring software programmers and managers. Here's how the usual interview process goes; here is specific advice about interviewing. There is a whole pile of advice in my book on software people. If all you did was drop the CS degree requirement, you would have taken a big step forward in quality improvement.

Why is a Monolithic Software Architecture Evil?

Why is a monolithic software architecture evil? Simple. There is no need to explain “why,” because monolithic is not evil. Or even plain old bad. In fact it’s probably better than all the alternatives in most cases. Here’s the story.

The Cool, Modern Programmers Explain

The new, modern, with-it software people come in and look at your existing code base. While admitting that it works, they declare it DOA. They say DOA, implying “dead on arrival.” But since the software apparently works, it can’t be “dead,” except in the eyes of the cool kids, as in “you’re dead to me.” So it must be “disgusting,” “decrepit,” “disreputable,” or something even worse.

Why is it DOA (whatever that means)? Simple: … get ready … it’s monolithic!! Horrors! Or even better: quelle horreur!!

Suppose you don’t immediately grimace, say the equivalent of OMG, and otherwise express horror at the thought of a code base that’s … monolithic!! … running your business. Suppose instead you maintain your composure and ask in even, measured tones: “Why is that bad?” Depending on the maturity level of the tech team involved, the response could range from “OK, boomer,” to a moderate “are you serious, haven’t you been reading,” all the way up to a big sigh, followed by “OK, let me explain. First of all, if an application is monolithic, it’s so ancient it might as well be written in COBOL or something people who are mostly dead now wrote while they were sort of alive. But whatever the language, monolithic applications don’t scale! You want your business to be able to grow, right? Well that means the application has to be able to scale, and monolithic applications can’t scale. What you need instead is a micro-services architecture, which is the proven model for scalability. With micro-services, you can run as many copies of each service on as many servers as you need, supporting endless scaling. Even better, each micro-service is its own set of code. That means you can have separate teams work on each micro-service. That means each team feels like they own the code, which makes them more productive. They’re not constantly stepping on the other teams’ toes, running into them, making changes that break other teams’ work and having their own code broken by who-knows-who else? With monolithic, nobody owns anything and it’s a big free-for-all, which just gets worse as you add teams. So you see, not only can’t the software scale when it’s a monolith, the team can’t scale either! The more people you add, the worse it gets! That’s why everything has to stop and we have to implement a micro-service architecture. There’s not a moment to lose!”

After that, what can a self-respecting manager do except bow to the wisdom and energy of the new generation of tech experts, and let them have at it? All it means is re-writing all the code, so how bad can it be?

One of the many signs that “computer science” does little to even pretend to be a science is the fact that this kind of twaddle is allowed to continue polluting the software ecosphere. You would think that some exalted professor somewhere would dissect this and reveal it for the errant nonsense it is. But no.

Some Common Sense

In the absence of a complete take-down, here are a few thoughts to help people with common sense resist the crowd of lemmings rushing towards the cliff of micro-services.

For starters, let’s acknowledge that modern processor technology has simply unbelievable power and throughput. Handling millions of events per second is the norm. The only barrier to extreme throughput and transaction handling is almost always the limits of secondary systems such as storage.

Without getting into too many details, modern DBMS technology running on fairly normal storage can easily handle thousands of transactions per second. This isn’t anything special – look up the numbers for RDS on Amazon’s AWS for example. Tens of thousands of transactions per second with dynamic scaling and fault tolerance are easily within the capacity of the AWS Aurora RDBMS;  with the key-value DynamoDB database, well over 100,000 operations per second are supported.

Keeping it simple, suppose you need to handle a very large stream of transactions – say for example 30 million per hour. That’s a lot, right? Simple arithmetic tells you that’s less than ten thousand transactions per second, which itself is well within the capacity of common, non-fancy database technology. What applications come even close to needing that kind of capacity?

OK, what if something extreme happens? What if you need more, and that somehow it’s your code that’s the barrier. Here the micro-services groupies have it right – to expand throughput, the right approach is sometimes to spin up another copy of the code on another machine. And another and another if needed. I talk about how to scale with a shared-nothing architecture here. Why is this only possible if the code has been re-written into tiny little slivers of the whole, micro-services?

The micro-service adherent might puke at the thought of making copies of the whole HUGE body of code. Do the numbers. Do you have a million lines of code? Probably not, but suppose each line of take 100 bytes, which would be a lot. That’s 100 MB of code. I’m writing this on a laptop machine that’s a couple years old. It has 8GB of RAM in it. That’s 80 times as large as the space required for the million lines of code, which is probably WAY more than your system has. Oh you have ten million lines? It’s still 8 times larger. No problem. And best of all, no need to rewrite your code to take advantage of running it on as many processors as you care to allocate to it.

I can see the stubborn micro-services cultist shaking his head and pointing out that micro-services isn’t only about splitting up the code into separate little services, but making each service have its own database. Hah! With each service having its own database, everything is separate, and there are truly no limits to growth!

The cultist is clearly pulling for a “mere” tens of thousands of transactions a second not being nearly enough. Think of examples. One might be supporting the entire voting population of California voting using an online system at nearly the same time. There are fewer than 20 million registered voters in that state. Fewer than 60% vote, usually much less. Suppose for sake of argument that voter turnout was 100% and that they all voted within a single hour, a preposterous assumption. A monolithic voting application running on a single machine with a single database would be able to handle the entire load with capacity to spare. Of course in practice you’d have active-active versions deployed in multiple data centers to assure nothing bad happened if something failed, but you’d have that no matter what.

Suppose somehow you needed even more scaling than that. Do you need micro-services then?

First of all, there are simple, proven solutions to scaling that don’t involve the trauma of re-writing your application to micro-services.

The simplest one is a technique that is applicable in the vast majority of cases called database sharding. This is where you make multiple copies of not just your code but also the database, with each database having a unique subset of the data. The exact way to shard varies depending on the structure of the data, but for example could be by the state of the mailing address of the customer, or by the last digit of the account, or something similarly simple. In addition, most sharding systems also have a central copy of the database for system-wide variables and totals, which usually requires a couple simple code changes.

Sharding is keeping the entire database schema in each copy of the code, but arranging things so that each copy has a subset of all the data. Micro-services, in contrast, usually involve creating a separate database schema for each micro-service, and attempting to arrange things so that the code in the service has ALL the tables it needs in its subset of the overall schema, and ONLY the tables it needs. In practice, this is impossible to achieve. The result is that micro-services end up calling each other to get the fields it doesn’t store locally and to update them as well. This results in a maze of inter-service calling, with the attendant errors and killing of elapsed time. If all the code and the entire schema were in one place, this wouldn’t be needed.

I am far from the only person who has noticed issues like this. There was even a list of problems in Wikipedia last time I looked.

Making sure your application is scalable and then scaling it doesn’t arise often, but when it does you should definitely be ready for it. The answer to the question of how best to architect a software application to be scalable from day one is simple: assure that it’s monolithic! Architect your application so it’s not database centric – this has been a reasonable approach for at least a decade, think it might be worth a look-see? If you do have a RDBMS, design your database schema to enable sharding should it be needed in the future. Make sure each software team “owns” a portion of the code; if you work towards eliminating redundancy and have a meta-data-centric attitude, you’ll have few issues with team conflict and overlap.

Do yourself and your team and your customers and your investors a BIG favor: stubbornly resist to siren call to join the fashion-forward micro-services crowd. Everything will be better. And finally, when you use the term “monolithic,” use it with pride. It is indeed something to guard, preserve and be pleased with.

The Map for Building Optimal Software

So you want to build optimal software, do you? What’s “optimal” in software, anyway? Is it building the software that’s needed quickly and well, with no bugs? Is it building software that’s easy to enhance, adding new features – including ones you never thought of – with minimal fuss and bother? Is it building software that’s easy to scale endlessly with little trouble? How about all of the above, all at once? Yup, that would be optimal, sure enough.

I’ve described in general terms about optimal software. Here’s a map, a specific example, of how to get there.

Going to new places

Let’s think about going places we haven’t been before, or taking routes that are new to us. What are the options for figuring out how to do it?

If you’re talking with someone who is familiar with the place to which you want to go, you might ask them how to get there. If there are more than a couple of turns, you might try to simplify the directions or write them down. If you’re on your own, you might consult a map. If you’re computer-oriented you might use one of the on-line mapping programs. If you’re driving in a car, you might use a navigation system.

No matter what you do, you’ll end up (let’s say) getting in a car and driving. When you drive, one way or another, you’ll be following directions and making choices of turns. Nothing happens unless you drive and to drive you need directions.

It’s obvious that while you need directions, a fixed set of directions by itself is adequate only if nothing goes wrong – there are no accidents, no construction detours, no recent changes to the roads, etc. The minute there is any problem, you need a map or a live, map-driven navigation system.

What you do when you look at a map, or what a navigation system does, is execute a generic direction-creating algorithm. Either your brain or a computer looks at the map, sees where you are and where you want to go, and figures out the sequence of roads and turns to get you there. Is there an obstruction? The nav system doesn't care -- it just computes a fresh set of directions for you, based on where you are now.

Here's an example. It shows two different routes for me to drive from my home to the wonderfully close source of some of the world's best home-made ice cream: Denville Dairy.

A direction-creating algorithm is a relatively small amount of code that is independent of the map that it uses. You may need to enhance it when different kinds of things are introduced into the map (for example, you decide to support avoiding toll roads or taking ferries), but once you’ve added the capability, it should automatically apply to all the maps you have.

So “where” is the application functionality here? The ability to create directions is in a small amount of abstract code. This code accepts a reference to a map, a starting point, an ending point and perhaps some parameters, like whether to minimize distance or time or avoid toll roads. The actual directions themselves are drawn from the substance of the map. In this case,

• what you’ve got is the map;
• what you do to what you’ve got is create directions.

Isn’t it interesting that most of the time and effort is creating the set of maps, without which there are no directions? And isn’t it interesting that the maps are data?

Maps are:

• Extensive
• Subject to frequent updating and change

An error in a map is bad, but should not “crash the program” – it should just lead to bad results, and should be easily fixed, without going into debuggers, single-stepping and other geeky things.

The direction-creating program is:

• Short
• Infrequently updated
• Unaffected by additional maps
• Unaffected by changes to maps

Best of all, the direction-creating program is unaffected by the driver not following the directions – the program just builds new directions to the destination from wherever the car happens to be.

Errors in this program are crippling, but it’s likely that errors will be found early in the process, after which it won’t change much.

Maps and writing optimal code

So what does this have to do with writing code? Lots.

A map (meta-data) is an efficient, compact, re-purpose-able representation of what the user cares about. It is truly Occamal, without redundancy. Each piece of knowledge you have about the world is represented in the map exactly once.

A reasonably well-written direction-generating program is an efficient, compact representation of strategies for generating routes through the entire universe of possible maps. Each algorithm or strategy for generating directions should be represented in the program exactly once.

Combined, the map and the direction-generating program are a pretty good template/metaphor for an Occamal program. Here’s the pattern:

• You should make everything you possibly can into a “map” (metadata).
• Abstract actions that cannot be reduced to metadata should be coded once, and should be “driven” to the largest extent possible by the metadata.

The map to building optimal software is a simple idea: build the smallest amount of code you possibly can, and put as much information as possible into passive, declarative meta-data. That will get you to your destination safely and in the shortest amount of time without breaking laws.

Lessons for Better Software from Washing Machine Design

The power of Occam-optimality, a.k.a. occamality, can be difficult to appreciate if your head is awash in myriad software buzzwords, and explaining it in software terms can make understanding it challenging to anyone unfamiliar with those terms. So it makes sense at this point to explain the core concept in a different context.

Let’s think about the design of physical things, the kind of things that require discrete manufacturing, have bills of material, and so on, for example a washing machine.

If every section of the washing machine were designed by someone different, each designer may call for a particular screw here, or a particular composition of aluminum there, making his best judgment on what is the best to use. When you broke down the bill of material into a consolidated parts list, you may see two of this kind of screw, one of that kind, and a couple of yet another kind. It may be that the various screws need to be sourced from multiple suppliers. As far as the designer is concerned, the various screws are completely legitimate requirements. Each was chosen to have everything required to get the job done, but only to get the job done. The designer is probably proud, with good justification, of the care and effort he expended in specifying exactly the right screw for the job. When the design is complete, the washing machine will have a list of unique parts and designs and/or instructions for how to construct and/or assemble them.

While the designers may be proud, everyone else’s jobs have been made more difficult and the overall expense of building washing machines has been increased. If instead the designers had been motivated to get together and find a way to use a single screw type that could be sourced from a single supplier, more screws would be bought from a single supplier in greater quantities, reducing both unit and shipping costs, and enabling all assembly points to be supplied from a single pool of screws. Only one type of screw driver would be needed, which would also help keep all the screw drivers in working order during manufacturing. Even the repair manual would be slightly easier to write and would be slightly shorter, and repair people would need only a single type of screw and a single type of driver, which would increase the odds that they would have what they needed to do a repair on a single visit. If the design were done in this way, the overall design would be virtually unchanged, but the parts list would be shorter – there would be a smaller number of unique parts, with each used more frequently.

How about the designers? Instead of starting from scratch when they needed a screw, picking the “right” screw from the universe of available screws, which is large, they would agree up front on a screw that would meet all their anticipated needs closely enough, and then, every time they needed a screw, they would just check to make sure the pre-chosen screw was adequate to the job. This is a simpler job than selecting the “best” screw from a large selection, because you’re just applying a short list of disqualifying characteristics to the pre-selected screw. Moreover, you’re arranging your design as you go along (for example, the thickness of the plates) to match up with the qualities of the screw, making the selected screw very likely to be OK in each case. So the “shortest” (without going too crazy) parts list, the one with the smallest number of unique parts, clearly is the best design.

Notice that the original design, the one with lots of types of screw types, would be defended by its designers as the “best” design, because they understood that the important thing for them to do was to create the “best” solution for each design problem they faced, considered in isolation. Let’s assume they did. But once the designers look at the whole problem, and try to pick the best design for the entire lifecycle of the washing machine, including sourcing, manufacturing and repair, it’s obvious that the different screw selections were incidental aspects of the washing machine design. The design could just as well have been embodied in one that had fewer screw types, and as we now understand, having fewer parts is better.

Suppose there are a couple of established washing machine manufacturers. Suppose you wanted to enter the market with a superior product. If things worked the way they normally do, the earlier products are probably far from occamal. That means it's possible to design a washing machine with just as high quality as the ones on the market, but less expensive to build and easier to service. Everyone would like your product better than the existing ones. Cheaper to build, cheaper and easier to fix in a single visit; what's not to like? This means that, in addition to being a core design principle, occamality helps us understand and predict the evolution of products in a market, all other things being equal.

In a virtually identical way, a program that has the smallest (within reason) number of unique “parts” is the best, for reasons that are very similar to the way that the best physical design has the shortest list of parts. It definitely helps the designers, and it helps everyone else even more. Just like with a washing machine, a program can be built, from the requirements all the way through, to have lots of unique “parts” and custom things, each of which may make sense when considered in isolation. But when you look at the whole software lifecycle, taking a view over the entire program and everything that will eventually be done to it, including later maintenance, change and enhancement, it becomes clear that designing it to contain the smallest number of “unique parts,” so that every “program concept” is defined in exactly one place, yields the best design.

The parable of the screw can help you, your organization and your users avoid being screwed during software design. You’ll get there if you keep this principle in mind: always use the smallest number of unique “parts” that you can, and be willing to adjust your design to make the number even smaller. This may sound easy to you, or it may sound like an essentially trivial exercise in neatness. It is neither. It can be challenging in practice, and the implications of it are actually profound and far-reaching.

Experts vs Innovation New Book

Experts and anointed authorities of various kinds, both academic and commercial, have been the front lines of resistance to innovation for centuries, up to the present. They are the firewall keeping what they consider to be rogue ideas outside the protected environments they oversee, protecting them from bad influences that their naïve but innocent charges might inadvertently adopt. It’s a good thing they’re on the job – otherwise things would be chaos and nothing would get done!

This pattern is raised to a new level when the subject isn’t some specific business domain like healthcare, but the process of innovation itself. As you may have noticed, many organizations now have the expensive modern equivalent of “suggestion boxes,” departments devoted to fostering innovation in the organization, led by a Chief Innovation Officer. Government officials have gotten into the game, establishing centers for innovation, and “incubators” for startups. Eager not to be left behind, academia has jumped into the game, with august professors doing what they do best: pronouncing truths and activities designed to promulgate them.

There was a time in my relatively innocent past when I was willing to give the experts a pass. Hey, how can you know everything? I know I don’t!  They’re probably just trying to keep their organizations from being taken down by harebrained ideas, and sometimes they fail to recognize a true innovation when it appears! I no longer believe that pleasant, forgiving fiction. They’re also not evil geniuses immediately recognizing a juicy innovation when it sniffs around the door, and stamping it out before it can start making changes. The truth is far worse: the vast, vast majority of them wouldn’t know a practical, effective innovation if it came up to them and slapped them in the face! See this and this for more.

The bulk of front-line experts act this way to “protect” their organizations against scary change. They go to great lengths on multiple dimensions to assure that nothing upsets the status quo. Here is detail about the measures they take to prevent innovation, and simple methods to overcome the measures – which the experts are universally ignoring.

But the elite of the experts are experts in entrepreneurship – people who are “expert” in enabling people who want to create and lead innovation to make it happen. These experts on innovation are like experts on being expert. We are definitely still in the middle of an “innovation” bubble, with everyone acting like it’s a new thing. In fact, it’s looking like less of a bubble these days, and looking more like something that’s going to stick around. Here’s some information about the bubble and how we got here.

I’ve seen so much of this over so many years, and been so disgusted with the useless wisdom of experts, that I’ve put together some preliminary thoughts, drawn from experience and observation, about how innovation happens. See this.

Imagine my surprise when I read an article about entrepreneurs that actually made sense! I'd never heard of the guy, Carl Schramm.  It appears from the article that he knows a bunch of stuff about entrepreneurs and innovation that matches up pretty well with my observations, but is actually backed by … real data! OMG! One of my initial shocks was learning that he was a real professor at a real university, even with a PhD, but that … he had worked as an entrepreneur! How is that possible? How did he manage to slip by the super-strict requirements that prevent anyone who actually knows something from experience becoming a Professor?? Maybe it helped that he had been the head of the Kaufman Foundation, the largest foundation devoted to the study of entrepreneurs and innovation, and that instead of just doling out grants, he did real studies to gather real data. What an idea. You think maybe the people who run Departments of Computer Science could get inspired? Sorry, forgive me, you caught me dreaming the impossible dream there...

I’m not finished reading the book yet, but here it is:

As you might guess from the title, he talks a lot in the beginning about that near-universal requirement for getting innovation funded, the dread business plan. He trashes it. Vigorously and effectively. No, he doesn’t trash this or that business plan, he trashes the very idea that business plans are both essential and good. He’s right! For exactly the same reason (he doesn’t say this, I’m saying it) that project management in software development is not just brain-dead, but a positive impediment to getting good software done!

If you are ready to learn about a different and better way to be an entrepreneur, check out this book.

Here's How the FDA Can Reduce Medical Device Costs While Improving Healthcare

The FDA wants to keep us safe. They want the drugs we take to be what they’re supposed to be, and they want the medical equipment used on us to be safe and without fault or error. We all want that!

However, the way they choose to achieve the goal for the software that is an essential part of most medical devices is deeply flawed, and leads to huge expense with only a small number of companies willing and able to follow the FDA’s regulatory regimen for software. The net result is medical equipment and software (which is increasingly a key component of medical equipment – think MRI machines) that is wildly expensive and uses seriously outdated technology.

There is a simple, easily understandable reason for this horrible state of affairs, which the grandees of the FDA refuse to acknowledge or even understand. The root of the problem is that they don’t understand software. Which doesn’t stop any of them from being certain they can regulate it. Because of this inexcusable ignorance, they take the regulatory approach developed over many years for drugs and manufacturing and apply it with only cosmetic changes to software. Safety is safety, they probably say to themselves. We’ve proven our approach for drugs; why start from scratch for software?

The mistake made by the FDA, along with nearly all the hard-charging graduates of MBA programs, law schools and bureaucrats everywhere is in thinking that the process of manufacturing is like the process of building software, except not as visible or physical.

In manufacturing, you have a factory with raw material arriving, being processed in a series of steps, with quality checks along the way, and emerging as finished goods at the end. The important thing is to check the quality of the raw materials as they enter and the results of each step of processing to make sure it’s up to snuff. At the end all you need is a cursory quality check, because so long as everything is done right along the way, the result is probably good.

Similarly, they think, in software you have a set of requirements, with lines of code and software components being produced along the way, with careful unit testing being performed at each stage, and more tests as the components are woven together. The end product is subject to more testing, but the important testing has already been done. The idea is that quality is designed in.

This method for building software is exactly what, in gruesome detail, the FDA requires. It’s spelled out in highly detailed regulations. Sounds good, right? Why would anyone want crappy software, particularly when it comes to our health?

The trouble is that this whole way is thinking is based on a blatantly false analogy.

What they think is that the manufacturing process of converting raw materials to finished goods is just like the process of creating lines of code and combining them into a finished software product. People even talk about “software factories,” and how important it is to churn out quality code, on time and on budget. Still sounds good, right?

Here’s the problem: a factory that produces finished goods, whether they’re drugs or cars, is making copies of a design that’s already been created and tested. In the drug development process a drug is created and validated through testing. All that’s done in the drug factory is assure that the copies that are made of the already-designed-and-proven drug are exactly and only what the drug creators intended.

Designing and building a new piece of software is like the drug development process, not the drug manufacturing process. The software is created for the very first time, with changes made along the way. The software equivalent of a drug factory is trivial: it’s taking a piece of executable software and making a copy of it. There is a universal software “factory” that works on all software: the copy utility. It’s what happens when you go to the Apple or Google software store and download the software you want. The download makes a bit-for-bit-100%-accurate copy of the original software for your use. That is software “manufacturing!” There’s even a universal quality check – a checksum is always incorporated in the original prior to copying that the receiver can check. The checksum tells you whether the copy is perfect, just like all the drug manufacturing quality checks do, only with software, it’s easy. Yes, because software is different. Here is more about software factories.

What the FDA regulations do is specify and control in gruesome, expensive and time-wasting detail the process of building the very first, original copy of the software – like creating the drug in the first place. This is a complete and total waste. The methods and processes of building software are constantly evolving, with the most innovative companies, the ones that actually create new software, at the forefront. These companies have small, focused teams who crank out great software, and do it quickly. They use what can be called “wartime” methods of building software.

The FDA should scrap its mountain of software regulations and replace them with a simple set of regulations that achieve the same goal, more effectively. I describe this in detail here. The new regulations amount to something like “We don’t care how you build your software, but its your responsibility to assure that the software performs its stated job each and every time, without fail. If the software has errors that cause medical harm, you are responsible for the damage it causes, and you may be barred from supplying software to the medical market in the future.”

This of course, shifts the entire burden onto the software creators – as it should. Inspections are no longer required. Employment at the FDA should go down, but of course, since it’s the government, it probably won’t.

Changing the medical software regulations in this way will unleash a wave of innovative, low-cost medical software. It will be as though runners were required to carry 100 pound backpacks and walk on stilts; as soon as they can dump the pack and use real running shoes, just watch them set records! They will be a race with each other to see who can cross the finish line in style the fastest, with no stumbling along the way.

William of Occam is the Inventor of the Method for Building Optimal Software

Lots of people would like the credit for establishing, once and for all, the core, immutable principle of optimal software design/architecture – the method for measuring which, among an infinite number of embodiments of software requirements, is provably the best among them. I am one of the crowd that would love to take credit! Alas, I and my competitors have been beaten to the punch, by a mere 700 years or so, by a guy named William of Occam.

As it turns out, the software crowd is, as usual, late to the party: loads of luminaries in other fields have recognized William’s achievement, viz., Occam’s Razor, for the indispensable concept that it is. Various software folks have been nipping around the edges of it, for example with the DRY (don’t repeat yourself) principle that’s gotten some attention. But they haven’t grasped the fundamental, touches-nearly-everything, deep nature of the Razor. Too bad! Crappy software continues to be churned out by the ton by people who are enthralled by false idols, convinced they’re on the righteous path toward software excellence, when in fact they’re doing something else, words for which should not be used in polite company.

William of Occam (ca. 1285 to 1349) was an English logician and Franciscan friar.

He is credited with formulating a principle that is already widely applied to various aspects of computer systems. In those areas of computing to which it has been applied, it reigns supreme – it unquestionably supplies the most fundamental, relevant and optimal understanding and solutions to the relevant problems, so much so that the people involved don’t think about it or question it.

However, there are large areas of computing to which Occam’s razor has not been applied. Worse, it is not even one of the candidates under consideration. As a result, those aspects of computing are fractured, inefficient, unpredictable, and driven by fashion and politics. Even while the practitioners call themselves "computer scientists," which is something that none of them are.

Everyone involved knows that the whole process of specifying, designing, building, testing and supporting software is hopelessly inefficient, unpredictable and error-prone. The leading methodology that concentrates on the process of building software, “project management,” is theoretically bankrupt and in any case has an empirical track record of failure. In terms of the content of good software, there are fierce battles among competing approaches, none of which is anything but a collection of unsupported and unfounded assertions, and which in practice don’t contribute to building good software.

Occam’s razor leads to the principles on which good software may be built, and supplies a single simple, widely applicable theme that, when applied, cuts away (as a razor should) all the inefficiency, inapplicability and generally what we don’t like about software. Once the principle is understood and widely applied, I expect it will become the un-discussed and undisputed standard for how software is built, just as it has in the other areas of computing to which it has been applied.

What is this amazing standard? It’s simple. I mean literally simple. That’s what Occam’s Razor is all about.

Occam’s razor is:

    Entia non sunt multiplicanda praeter necessitatem.

    No more things should be presumed to exist than are absolutely necessary.

Occam’s razor applied to software would be:

    No more software entities should be created than are absolutely necessary.

Another way of expressing this is that all redundancy, of any kind and in any way, should be eliminated from a piece of software. This often requires writing imperative code to implement any and all abstract actions, and to create largely declarative meta-data to specify everything that is specific to an application. The meta-data, of course, should also have no redundancy; this is normally achieved by use of multiple inheritance with override. See this for more.

To be extremely brief, the reason why eliminating redundancy is good is that practically everything that happens to a piece of software is that it is enhanced, altered, bug-fixed or otherwise modified. Creation happens just once; everything that follows is growth and change. Implementing change of any kind to a program takes the least effort with the least risk if everything is defined in exactly one place – once you’ve found the place, you make the change and you’re done. Among the infinite universe of programs solving the same problem that work, the speed, cost and risk of change is the overriding virtue, both in technical and business terms.

Thanks to William of Occam, I can’t claim to have invented the core principle of software, and the way to express the measure of goodness for software. William has me beat by over 700 years. Thanks a lot, Bill! Lots of people have applied Occam’s Razor to lots of tough things with optimal success, even in things closely related to software, such as information theory. Way too late again. About all I can do is mess with his name and make up a term that expresses goodness in software. Here it is: I propose that a piece of software can be measured by its “Occamality.” The more “Occamal” it is, the better it is. And I propose that Occamality is strictly correlated with the extent to which there is no redundancy of any kind in a program, which is almost always strictly correlated with the program being at the high end of the hierarchy of abstraction.

How to Pay Down Technical Debt

When we look at a body of software, how do we judge it? Clearly, there are strong opinions on this subject – but there is no general consensus on the standards by which a body of code should be judged! This is shocking and unacceptable!

For most normal people, i.e., people who aren’t software developers, software is “good” if is doesn’t crash and pretty much does what you want most of the time. What more could anyone want? Well, how about changing the software. This is one of the BIG things that makes software different from pretty much anything else in our experience. How often do you try to make changes to your car in the same sense that you change software? The answer is never.

There are standards for nearly everything in life, certainly for things that are built by engineers and other technical people. How is it that, not only are there NO standards for what constitutes “good” software, but the fact that there are no such standards is not decried, and no one appears to be sheepish or unapologetic about admitting it – because they’re never asked to admit it. The subject never comes up – not only in “polite company” but even in “impolite company,” i.e., among programmers talking quietly among themselves.

“Ya know,” someone might say, “we tell them there’s technical debt. Sounds fancy, right? Anyone ever been asked what it means? They seem to think that WE know what it means, and that’s good enough for them. I wonder how much longer we’re going to be able to get away with it?” Someone else says, “Forever, man. This has been going on since before we were BORN, ya know? Unless word somehow gets out, and how could it, we’ll die without anyone being the wiser. Good thing.”

I wish programmers did have quiet dialog among themselves like the one I made up above. But they don’t, at least that I’ve ever heard. There’s nearly always someone who has strong opinions about what software should look like internally, and of course the current body of software doesn’t measure up. So there is agitation and there are threats about the awful future consequences of failing to rein in the spreading disease – soon! RIGHT AWAY!! – unless the transformation is authorized.

Then the re-writing death march starts off, with various sections of the code slated for repair, renewal or even flat-out re-write. What is the target? It varies. Favorite criticisms include that the code is spaghetti, it’s a monolith, it’s not structured according to some set of object-oriented principles, it’s written in some obviously inferior language, there aren’t components, there are no services, micro or otherwise, and on and on. The solution is obvious to the converted, and value to be achieved by the solution is so obvious that it’s hardly worth stating.

Is there any value to be gained by the clean-up of “technical debt?” Maybe. But often, nothing really changes except a lot of time has been spent with no improvement to the business. No one can point to a proof or a study or a clear historic pattern demonstrating the effort is worth making.

This is a shame, because there is a clear pattern of success for a method of organizing and/or re-organizing code for business benefit.

Let’s start from the business benefit. Of all the ways a piece of software could be written that performs the same function, which is the best in business terms? I’m assuming here roughly the same level of performance, etc.

Tick, tick, tick… Think about it!

When you come right down to it, no one cares about today’s software doing today’s things for today’s customers – so long as it works and performs reasonably well. What matters is simple, and everyone knows it: it’s the next request by an important existing customer, and more important, the next potential customer who is balking about things they need that aren’t there, the time, cost and risk to implement the software, etc. In other words, what matters about today’s software is TOMORROW’s changes, whatever they may be – we can suspect, but we won’t know until we know them!

Software people have talked forever about things like “architecting for change” and various fantasies that mostly go poof when the harsh wind of reality rushes at them.

There are exactly two rarely-discussed aspects of code that position it optimally for the widest possible range of unanticipated changes and the needs of installation:

• A minimum of redundancy in the code – anything you want to change can be changed by going to one place
• To the largest extent possible, particular things the application does are defined in meta-data instead of code, with the result that the code is smaller and more abstract with minimal redundancy -- and that any change can more likely be made by changing meta-data, which is quicker and safer than changing code.
  

That’s it! See this for more.

Why the Equifax Hacking Disaster Wouldn't Happen at a Car Dealership

The 2017 Equifax data breach is in the news again because of the recent indictment of four Chinese government hackers for committing the break-in. How did they pull off such a feat? From China? As it turns out, Equifax’s defenses were so pathetic that a couple of bright nerd wise guys anywhere could have done it.

By contrast, the Equifax hacking could not possibly happen at a car dealership – unless the dealership were run by government and corporate cybersecurity experts. Understanding why that’s so tells you everything you need to know about the expertise of the experts.

The 2017 Equifax hack

First, a quick – VERY quick – review of the Equifax hack. Equifax has websites. One of them is just for people who want to dispute their credit rating. This particular site was run on a computer with software that had a serious flaw, allowing a skilled person to get past the normal consumer web pages and run other programs on the computer. The flaw had a fix that could easily have been installed, but because of a series of bungling and delays, the fix wasn’t installed for months.

The flawed software on the Equifax site was widely used. The fact of the flaw and the fix for it were publicly available. Anyone could have read about it and fired off a search for websites on which the patch correcting the flaw had not been installed. Someone found the Equifax site had the issue, exploited the flaw, and ran a couple programs without doing much else.

More than a month later, the flaw still unpatched, someone got into the Equifax server again and started “looking around.” They found an unencrypted file with names and passwords. They used the information to log in and got access to a series of databases that had Equifax customer information. For 76 days, they used the databases like any authorized user would, and issued queries that returned a great deal of customer information, which they stored in files. They encrypted the files and used standard programs to ship the files out of Equifax, presumably to themselves.

Hacking the Equifax-run car dealership

Suppose someone had tried the equivalent of the Equifax hack at a car dealership. The hacker would have physically walked in the service entrance, along with the other existing customers. The hacker knew that many doors had a security flaw in their lock that had not been fixed. The exact nature of the flaw had been publicized, and any reasonably skilled person who knew about doors and locks could exploit it. Once inside the customer service area of the dealership, the hacker looked for the “employees only” door, and quickly saw that it had not been fixed. The hacker walked up to the door, fiddled with it for a couple minutes, opened it and walked in. In a real car dealership, the employees would have immediately noticed a strange person and challenged him, politely showing him out. In the Equifax-run car dealership, the intruder is ignored.

Once in the employee section of the dealership, the hacker wanders around, poking into lots of things. Finally he walks into the finance department and wanders some more, again unchallenged. He notices a row of file cabinets against the wall, and figures there must be valuable information in there about customers and cars, with all sorts of details like names, addresses, driver’s licenses and who knows what else. But he sees the cabinets are locked. Darn! So he looks around some more and spots some keys sitting on someone’s desk. Even though it’s daytime and people are working at the desks, on the phone, etc., no one says a word when the intruder picks up the keys, walks over to the file cabinets, and tries one key after another until finding the one that works. He opens the cabinet, takes out a handful of folders, walks over to the copy machine, and makes a copy of every document in the handful of files. He then goes back to the file cabinet, returns the originals, puts the keys back on the desk, and walks out of the building the way he came, holding a big pile of copied pages in his arms. Unchallenged.

The next day, the hacker returns to the Equifax car dealership and goes through the same drill – goes into the employee-only section, then the finance department, uses the keys to open another drawer, makes copies and walks out, all without a single one of the many employees working there saying a thing. He does this day after day for 76 days.

Finally, someone notices that there’s supposed to be a guard by the employee entrance checking everyone – there always used to be one! So a guard is put there again. The guard notices that the daily visitor carrying lots of paper doesn’t look like everyone else – he’s carrying big piles of paper! The guard doesn’t stop the visitor, but reports to his boss; meanwhile, the visitor notices the guard, figures the jig is up, and stops coming.

More than a month later, the Equifax-run car dealership’s bosses finally let the word out that they had been hacked.

The Equifax breach vs. the car dealership

You already know that nothing even vaguely like the Equifax breach could have taken place in a car dealership. In the dealership, people have common sense and are dealing with physical things, while at Equifax, everything important happens on computers that are in locked rooms, with software that is invisible to nearly everyone, doing things that most experts barely understand, managed by people who have no real knowledge of software using management methods taught in business schools by professors who are ignorant of software, and following rules and regulations written by lawyers and bureaucrats. What can you expect but madness and chaos??

What went wrong at Equifax

Equifax’s cybersecurity methods followed regulations and industry-standard practices, with all required certifications. The result of these methods, when executed perfectly, is security that is worse than that of retail stores or libraries. In the end, Equifax made exactly two mistakes – but each mistake mattered because of further bungling and the failure of ordinary follow-up checks.

• The web server patch, the start of the trouble. The patch was announced on March 7, 2017.
• What they did: On March 9, administrators were told to apply the patch. They didn’t. On March 15 a scan was run that was supposed to detect unpatched systems. It didn’t work. It wasn’t run again or fixed. Result: the patch wasn’t applied until August or later.
• The bad traffic detection box. A great deal of web traffic is encrypted. Equifax had installed a box at the “edge” of their system to stop all incoming encrypted traffic, unencrypt it, make sure nothing “funny” is going on, report suspicious activity, and re-encrypt and send along each message. It’s like a traffic stop for incoming traffic.
• What they did: encryption works with keys, and keys expire periodically. The key used by the box expired 10 months before the hack started. Equifax failed to renew the key, and the traffic-stop box was set up to let all traffic through without checking unless it had a valid key. No one noticed until July 29, 2017, when the certificate was finally renewed. When the traffic stops started again, an administrator noticed suspicious activity, and sounded alarms bells.

I’m going to add this fact, just because I find it amusing: the Chief Security Officer was a music major, Susan Mauldin. She had degrees from U Georgia in music composition. Her background was scrubbed from the internet as soon as the scandal broke. Of course, you don’t need a college diploma to be excellent at software, IMHO. But in this case…

What car dealership management would do at Equifax

People who run car dealerships aren’t usually thought of as geniuses, but part of succeeding in business is protecting your customer records, and everyone involved takes care to do that and do it well. All that was required for Equifax to have avoided being hacked was to do the software equivalent of what every sensible car dealer employee would do. Obviously, such common sense is beyond the ken of the high-paid professionals and certified experts at large places like Equifax. Here are the main things that would be done if car dealer people replaced Equifax management.

• Recall/repair notice.
• The second a car dealer hears about a recall/repair issue for cars, they jump on it. Similarly, just exercising common sense, if they learn that the restricted entry system to the employees-only area has a fixable flaw, they would get it fixed immediately. No excuses. Car dealer management would have seen to it that the flaw was patched, and checked.
• Guard duty.
• At the car dealership, there is extra security around the finance department, which holds all customer data. During working hours, there is always someone alert to the door, challenging anyone entering who shouldn’t be there. Off-hours, there are deadbolts on the metal doors, along with an alarm system that has motion detectors and cameras, which gives an immediate alert on any sign of trouble. If the alarm system drops its constant, real-time communication with the monitored center, electronic alerts are sent, so the problem can be immediately fixed. The room and its data are NEVER left unprotected.
• With car dealership management, the encryption key on the traffic checker wouldn’t have been allowed to expire. Instead of setting up the traffic checker to stop checking unless it had a valid key, car management would have made it set off alarms so it could be fixed right away.
• When car dealership management learned that only incoming traffic was being checked, they would have pitched a fit. What? You’re just letting anyone waltz out with anything?? They would have installed a system to stop and check outbound data traffic before letting it out.
• Keys
• Car dealership management would never allow the file cabinet keys (user names and passwords) to be lying around for anyone to pick up.
• Employee behavior monitoring.
• Car dealership management would make sure anyone opening a file cabinet was a person authorized to do so, and that their actions were reasonable. When they found out that anyone could open a drawer and pull out and copy ALL the files in that drawer, they would have been enraged. They would have immediately put a software system in place to ring a bell and prevent anyone from taking more than a single file.
• While at Equifax the only real checks were with people as they were coming in, after which they could do anything without being checked, car dealership management and employees know that everyone has roles and acts in certain ways – and that everyone is responsible for noticing unusual behavior and questioning it. With car dealership people in charge, the software equivalent of such monitoring for “normal” behavior would have been implemented and strictly enforced, with immediate shut-down of a user if they stepped outside the bounds of their normal actions.
• Car dealership management understands that audits need to be done, and that auditors need broad access to customer files. This is the ONLY time mass access to customer files would be permitted, and ONLY under the watchful, suspicious eyes of multiple dealership managers, who would ASSURE that all files would be replaced WITHOUT BEING COPIED.

Expert recommendations for Equifax

The “experts” have had a wide-ranging set of advice for Equifax. Equifax has spent over $1.4 billion dollars making largely useless “improvements” to its security. I haven’t read ANYWHERE recommendations of the kind of changes any sensible car dealership would make, as described above.

Here are some of the leading recommendations of what Equifax should do to improve their cybersecurity:

• Change management reporting, processes and procedures.
• I love this one. It’s a commonly-recommended “cure” for cybersecurity ills.
• Encrypt all customer data
• This is a favorite, and widely recommended. It is USELESS. It would NOT have prevented the Equifax hack!! Why? Easy: once the hackers were in and were using employee user names, they just issued SQL queries against the database. If the data on the disk, the database un-encrypts the disk blocks, processes the SQL query, and returns UN-encrypted data. Otherwise, the data can’t be used!
• Encrypting data on disk is like having locked, strong metal file cabinets. But authorized people still need access. Therefore, file cabinets have drawers and keys. When you open the drawer, the data is easily accessed. Encrypting data “at rest,” as they say, protects only against the hackers who somehow get close to the cabinet and drill into it from the side or bottom. Who would do that? It’s easier to steal the keys or break the lock!
• Create more silos
• The nice-sounding theory is that breaking everything into silos would limit a break into just one silo. But far from being a solution, silos were actually part of the problem at Equifax – applying the delayed software patch required writing memos and asking multiple people to do things, when in a uniform environment, a single script could have updated everything.
• Change the reporting structure
• Because when you change who reports to whom, everything changes, right? In Equifax’s case, the Chief Security Officer reported to the Chief Legal Officer, while the Chief Information Officer reported to the CEO. “That’s the cause of the breach!” shrieked an amazing number of pundits!
• What there should probably be is two Security Officers:
• A CSRO, Chief Security Regulation Officer, who reports to the Chief Legal Officer. This person is in charge of the massive, ever-changing, lawyer-created body of regulations that are supposed to assure Security. There are severe penalties if you fail to conform to the regulations, which require loads of reports, processes, documents, etc. But they have little to do with real security.
  
• A CRSO, Chief Real Security Officer, who reports to the Chief Technology Officer. This person is in charge of making sure that real security is performed, in spite of the regulations.

Conclusion

Computer software and systems are hard to understand, a problem made worse by the fact that they’re literally invisible. You'd think that would be OK, since for example even fewer people understand quantum physics and we’re OK in physics. The trouble is, the people doing physics really do understand it, while the people doing software in general, and cybersecurity in particular, are faking it – without even suspecting, in spite of the mountain of evidence to the contrary, that they’re faking it.

The result is that government agencies, powerful consultants and weighty experts recommend more of the same medicine that created the problem, without a shred of recognition that it was their own rank stupidity that caused the problems to begin with.

For more perspective on this subject, see these posts.

Note: this post first appeared at Forbes.

How to Build Applications that can be Changed Quickly

Nearly everyone who talks about fast and reliable application building seems to talk about the process (Agile etc.) and/or a couple architectural things (object-oriented, micro-services, etc.). From what I can tell, these haven’t led to any break-throughs in application velocity or predictability. However, it’s clear that there are methods for high-velocity application growth. A wide variety of small, motivated groups have re-invented them over the years. I have had the privilege of observing these groups and their methods, and seen the impact when they add or drop velocity-aiding methods.

Here is an overview of some of the methods I’ve seen used. The improvements can range from cutting time in half to whole number factors, things like turning weeks into days or in some cases hours.

Forget requirements, focus on change

The normal requirements process starts with the often non-technical people who have a need. The need can be expressed directly, or through refusal to buy/use a product, which gets the attention of sales and marketing people. There then ensues a process of requirements gathering and refinement. Wire-frame UI’s may be involved. Finally, the requirements are ready for engineering, and away we go on hopefully not too many “sprints.” This is an “outside-in” approach.

Speed is achieved by taking an “inside-out” approach – instead of focusing almost exclusively on requirements, center your efforts on the code and what it can do or easily be changed to do today to go in the direction of the requirements. Take steps to change and add to the existing code to meet the unmet needs. Do this in small steps, getting feedback at each step. When you’re done, go back and “clean up” the code to eliminate redundancies as much as possible, to make the next change as easy as possible.

What this does is ground the changes in the existing code, resulting in the changes and additions being smaller than they otherwise would be. As a additional benefit, it usually results in a greater uniformity of both the code and the interfaces (UI and API) to the code.

Minimize the customization and optimization of the code and UI

When product people think about a new process that has UI, they usually try to do a good job, understanding all the details and nuances involved. The result of this sophistication is often requirements that are finely adapted to the particular function being performed. What’s wrong with this, you may wonder? One problem is that this approach ends up taking longer to built and results in more code. More important is the users – the less someone has to learn to use a new feature, the more quickly and easily they’ll be able to use it. Familiarity is incredibly important. Which means, among other things, that literally using rarely changing templates to generate interfaces will both make any new interface element recognizable and easy to learn, but also lead to the least amount of code changes to get the job done.

This means that the fineness and sophistication of product people can be a problem. The more time they spend on a subject and the more they think and reflect on it, the worse the temptation is likely to get to do something other than just crank out another instance of a familiar pattern.  Resist!

Avoid planning and designing for change and future requirements

When starting a project, accomplished software professionals often insist on a period devoted to design or architecture, in which the overall technical approach to the project is determined. When things go wrong later, inadequate or insufficient up-front design is often cited as the reason. “We’re gong to insist on being able to do it right next time, so this won’t happen again.” But all too often, projects slip and slide regardless of any up-front design work.

Generally, up-front design time is actually counter-productive. The reasons are simple: you spend time “designing” when you could have been building; the result of the design is often building for anticipated change, which takes time; when the anticipated change doesn’t happen and unanticipated change arrives, the code added because of the design almost always makes things harder.

None of this happens when you build incrementally and strive toward uniformity, eliminating code redundancies and moving functionality from code into meta-data.

Wartime software

The methods I have described as wartime software maximize speed and efficiency of building new software that meets business needs. See here for details. Generally, the methods include:

• Eliminate overhead
• Avoid things resembling “project management”
• Optimize for speed instead of expectations
• Use production environments for testing instead of sand boxes
• Use comparison QA; avoid test-driven development, unit testing and test scripts
• Minimize the use of documents, meetings and other internal overhead
• The main architectural value should be speed and scale
• Avoid technical fashions

Move up the mountain of abstraction

When you think about code and methods, the main values should be the elimination of redundancy in both data definitions and code, and migrating functionality from code into meta-data. See this and this.

The other methods described here are important for moving quickly, and are complementary to this one. But moving up Abstraction Mountain is the most powerful of the methods, often yielding qualitative improvements.

Conclusion

This is a short post, and the methods are described briefly. In a different world, each topic here would be at least a course in the Computer Science curriculum, and in a couple cases multiple books. I don't claim anything I've said here is fully described. But every single item has been proven in practice by many groups over many years, in different ways, always resulting in better quality software that meets business needs being built dramatically more quickly than other methods could possibly have achieved.

My goal here is simply to provide a short list of the main methods for fast development in a single place.

The Iowa Caucus Software Problem Exemplifies Widespread Software Management Problems

Everyone who’s even vaguely in touch with the headlines knows there was a problem getting results from the Iowa caucuses – a problem blamed on a software app built to automate reporting voting results. No one, including the maker of the app, disputes there was a serious software problem.

Most of the commentary has focused on steps that should have been taken to assure that the app worked better than it did. The Wall Street Journal quoted various “experts” who stated – authoritatively, no doubt – that “testing could have prevented” the problems.

What’s not being said is the emperor-has-no-clothes “secret” – our society is irrationally obsessed with software, and acts convinced that getting new software to perform some function that is now getting done with older software or without software will make things better, be worth the trouble and have no downside. Otherwise sensible and experienced people continue to ignore the ongoing, rolling disaster of software building and deployment, and blissfully welcome new software nightmares, as though there’s no chance that anything could go wrong. The Iowa Democrat caucus face-plant is just the latest of a decades-long parade of software disasters. To make it explicit: I'm using this highly visible event as an occasion to illustrate a general, widespread problem; it has nothing to do with Democrats or politics.

The 2016 Caucus Software

Both Democrats and Republicans hold Caucuses in Iowa. Each party sets its own rules and changes those rules as it sees fit. They even do the voting a bit differently. Nonetheless, the parties cooperated to have a single piece of software that would service all their needs. They jointly announced this in the Jan 31, 2016 Des Moines Register:

They also had phone backup in case of software problems. The net result is that the Democrat Party chairman announced final results:

The software wasn’t perfect, of course – software never is. But it got the job done:

The 2020 Caucus Software

In the summer of 2019, the Iowa Democrats decided that, instead of upgrading the software that worked for them last time, they would pay for brand-new software to be built by a tiny, brand-new firm run by a non-programming History major who’d had a role in the tech of the 2016 campaign for Hillary Clinton. New firm. New software. Has to work out of the box, with over 1,700 people using it for the first time. No problem, right? That’s what the chair of the DNC proclaimed just hours before the caucuses opened.

Many people at the caucuses had trouble using the software, and in the end, it just didn’t work – it wasn’t able to load the results it captured from the people who were finally able to use it into the DNC reporting database. An analysis shows the kind of amateur-hour flaws you'd expect from the kind of group that built it.

Of course they forgot or ignored, like most people do, the train wreck of so many high-visibility new software unveilings, illustrated well by the ACA software releases of 2013 such as the one in Oregon I discuss here.

The Curse of Technology

Technology can be a wonderful thing. I’m grateful for the good things it delivers. But even relatively simple, physical technology can have massive problems. Did you know that 3,613,732 people have died in motor vehicle accidents in the US alone between 1899 and 2013? In 2016, 37,461 people died, as the carnage continues unabated. Just to put that number in context, the total number of US military deaths in all wars starting with the Revolution is well under half that number.

Does this surprise you? Here’s an example: during the entire period of the Vietnam war, the US had a total of 47,424 military deaths. During EACH year in the period 1966 to 1973, there were more US traffic deaths than the total for the war, with a high of 53,543 traffic deaths in 1969. You know which category of numbers people paid attention to.

The problems with computer and software technology massively dwarf those of physical technology, by huge factors.

Physical bank and stage coach robbers make for good visuals and press. One of the most famous was Willie Sutton, who spent decades in jail for having gotten about $2 million from all his hold-ups over many years. Let’s introduce computers and software to make things better! Sure. Banks lose over $10 Billion a year due to computer-driven credit card fraud alone! Banks vs. banking software? No contest.

Loads of people in the industry are promoting the latest software fad, for crypto-currency, like Bitcoin. It’s supposed to be super-secure. Sure. It turns out to be even worse, as the mounting software failures and losses demonstrate.

What all of this means is that once a piece of software does what it’s supposed to do, in production and at scale, no sensible person even thinks about replacing it with a whole new application. The reality is that most of our bedrock software systems took years to build, are often decades-old – and continue to work solidly and reliably.

An excellent example of this is the body of COBOL code produced by the small software company Paysys. When I was CTO of the company, in the late 1990’s, it ran over 150 million credit cards, including for Citibank, GE Capital and other name brands. The company was bought by the world’s largest credit card processor, First Data, and an incrementally evolved version of the same code now runs over 600 million cards, more than any other piece of software. The code is nearly 30 years old! It’s written in “obsolete” COBOL! There have multiple major attempts to re-create it using “modern” technology, sometimes with efforts taking years and costing tens of millions of dollars. Failures, every one.

The reality is that software is hard to build and get right, even today, after decades of time to figure it out. If we were as good at building bridges as we are at software, no one would dare drive over one. The big tech companies, for all their glittering reputations, are lousy software builders. When they try to build something new, they usually take years and end up failing – which is one of the reasons they acquire so many software companies, companies that in many cases have somehow managed to build software that they couldn’t. Here are details about Facebook’s ineptitude. If it's so hard for tech giants, chock full of super-bright nerds, why should anyone expect brand-new software written by a tiny group of amateurs to perform well the first time, with no testing or training, and in a national spotlight?

Building Software is different than building other technologies

People make decisions about software that, if it were any other technology, they would never make. In fact, they would consider those decisions to be beyond stupid.

Suppose you needed to get 1,700 people to an important public event from scattered places. While they didn’t all have to arrive at exactly the same, they had to be there within a couple hour window. Without fail. What would you do? You might consider asking a couple of established limo companies with fleets of reliable cars and experienced drivers to get it done. If a couple people insisted on driving themselves, you’d make sure their cars were OK and they had a backup plan. You might arrange for some car-pooling, or bring some people to a central spot and arrange for an experienced bus service with professional drivers. You might do any number of things.

Let’s further suppose you’d done something like this once, but when you did it again, you wanted to do some important things differently. Suppose you needed to make two trips on the same day. You would probably make some changes to what you did last time, while adjusting for whatever didn’t work perfectly before, right?

There are lots of things you wouldn’t even think of doing. Would you do any of the following?  Design, build and distribute brand-new vehicles for doing the transporting? Contract with a tiny, new company with no real track record to design the vehicles from scratch, not even modifying an existing model that works? Distribute the vehicles with no real test-rides? Give them to each of the 1,700 people to drive themselves? Given that the vehicles are new, not built according to standards, expect the new drivers to drive themselves with no training and no help? When the people have trouble driving, have no help available? When the vehicles break down, have no back-up plan to get the job done? When the disaster drags on for days, still be unable to fix it?

No, I don’t think any person would consider making these kinds of decisions with physical-world technology. The very thought would be ludicrous. But this is exactly what happened with the 2020 Iowa Caucus software! Endorsed and supported from the top of the DNC, and not questioned by any of the Iowa leaders!

Conclusion

There is a problem with how we think about and build software. A big problem. It’s nearly universal, and shared by normal people and supposed software experts. What happened at the 2020 Iowa Caucus technology is exceptional only in that it was so highly visible, which is why I use it as an example. The insane decision-making process that took place there takes place every day, from the way normal people and organizations work with software to the way tech giants and software so-called experts work.

The small groups of people who build software effectively and well are largely ignored by experienced managers and organizations. But these are the ones who are most likely to create amazing new bodies of software that change industries, before their software is absorbed and subsumed into the larger organizations who try to avoid breaking it too badly.

The mainstream thinking about software, both by managers and by software experts, is badly flawed. Things won't get better until fundamental things are changed.

The Three Dimensions of Software Architecture Goodness

In my work on Wartime Software, I describe the methods used by small groups of smart, motivated programmers to compete with large, established groups using standard software techniques – and win. I haven’t invented those methods; I’m simply collecting, organizing and describing what such groups of software ninjas actually do.

Similarly, after observing the internal structure of many bodies of software over a long time, patterns emerge about the internal structures that yield competitive advantage, and tend to take market share. These internal structures are a kind of continuum rather than an either/or: a group that is farther along the continuum of goodness I’ve observed has a substantial competitive advantage over any group whose software is more primitive in the continuum. This pattern is so strong that you can see it play out with many companies competing over a long period of time.

The patterns are fundamentally simple, but rarely discussed among programmers or in the literature – and not taught (to my knowledge) in Computer Science departments. Using the patterns, it’s possible to rank any piece of software along three independent but conceptually related dimensions.

The dimensions are:

• The code. The not-good end of this axis is code that has a great deal of redundancy, and the good end is code that has no redundancy.
  
• The data. The not-good end of the axis is data that is defined and/or stored in multiple places, and the good end is data that is uniquely defined and stored.
• The meta-data. The not-good end of the axis is a software system with no meta-data, and the good end is one in which most application functionality is defined and controlled by meta-data, and can be changed by editing it with no code changes. I often think of this as the vertical dimension.
  

These dimensions are closely related conceptually, but in practice aren’t necessarily linked. Nonetheless, in practice, movement towards the good end of any dimension encourages and helps movement towards the good end of the others.

Why should anyone care about dimensions of software architecture goodness?

It’s an abstract idea. It’s new to many people. It’s a tough addition for managers whose heads already are chock full of barely understandable buzzwords and acronyms. Why pile on more?

Simple: investments in moving software towards the goodness end of the dimensions is HIGHLY correlated with shorter times, lower costs and lower risks of meeting the evolving needs of existing and new customers in a highly competitive market. The correlation is clearly demonstrated historically, and incremental progress yields immediate benefits in sales and business satisfaction. There is no single thing a software group can do that has greater impact on fast, low-cost and trouble-free delivery of existing and new product features to customers.

I know these are bold claims, particularly in an environment that swirls with buzzwords for software. Everything will be better once we start programming in Golang, like Google! Our teams will work better and our software will be scalable once we re-organize using micro-services! We’ve got to automate testing and move to test-driven development! We have to add Kanban to our Agile environment, and hire a certified SCRUM master! These subjects and more have passionate adherents and are widely implemented. But none of them yield the promised results, which is part of why there’s such a merry-go-round of methods becoming fashionable, only to fade quietly into obscurity or ho-hum, nothing’s changed but it’s just what we do.

Details of the Dimensions

Data

The data dimension is the most widely understood and practiced of the three. I hope everyone who can spell “relational database” is familiar with the concept of a normalized schema – the whole point of which is to assure that each piece of data is defined and stored in exactly one place. But the concept of a normalized schema applies strictly within the bounds of the DBMS itself – there is no widely used term or concept for a truly central definition of data, including inside the code.

Is having fewer, less redundant, more centralized definitions a good idea? You bet. This is largely why the RAILS framework for Ruby rapidly grew to widespread use. It had nothing to do with the design of Ruby or its object-orientation – it was all about the RAILS framework and it’s central DRY principle; DRY = “don’t repeat yourself.” The idea was/is simple: data definitions in RAILS are created in a central place – and then applied by the framework both in the DBMS schema and in the data definitions in the code to access the data. To change a data definition, you change it in the RAILS definition, and it changes it in both the database and the Ruby code.

It’s important to note that the data dimension in my definition encompasses data wherever it appears – program files, schema files, anywhere.

Code

The code redundancy dimension is much less understood and recognized for its importance. But the value of reducing code redundancy is easily understood: when you need to make a change to a program, how many places do you have to go to make the change? The answer any appropriately lazy programmer would like is “exactly one place.” The programmer finds the place, makes the change, and all is good. As you add code to a system to make it do new things, often you end up repeating or creating, perhaps inadvertently, variations on existing themes in the code. When you’re done, what’s the best thing you can do to make the next change as easy as possible? Eliminate the redundancy. That’s it! Objects, layers, components, services and the rest don’t matter. Redundancy matters.

Meta-data

The third dimension is meta-data. Having no redundancy in the meta-data dimension is important, and is often achieved by measures such multiple inheritance hierarchies and links. But the most important thing is this: to the maximum extent possible, application-specific behaviors of all kinds should be defined in meta-data rather than lines of code. Goodness in this dimension is measured by growing meta-data. Over time, decreasing code redundancy is achieved by increasing the size, extent and power of the meta-data.

This evolution directly powers business benefits for the simple reason that meta-data can be changed and augmented quickly without creating a new version of the code and without fear of introducing bugs into the code. In the end, you end up with something like Excel: the source code never needs to be touched, while supporting endless variations of spreadsheets, with no programming skills required. When you make a mistake in a spreadsheet, you may not get the result you want, but Excel doesn’t crash.

Conclusion

Wartime Software is a collection of techniques that have evolved to build software quickly, and to win in a software-fueled business.  People who practice some subset of this collection of techniques often find themselves moving towards the goodness end of the software architecture dimensions as they rapidly cycle their code, since every small step along any of the three dimensions helps them move more quickly and satisfy emerging customer needs more rapidly than the competition. It’s one of the most powerful tools in the Wartime Software arsenal.

How to Design User Interfaces for Heavily-Used Software

There is lots of knowledge about software user interfaces -- standards, models, experts and the all rest. But there's a problem: there is no difference, the way things are now, between designing a UI for someone using a piece of software for the first time, and someone who uses it over and over. This results in astounding waste of time for the heavy users of software. It's long since time to fix this glaring hole in UI theory!

Most UI’s are built to optimize the initial experience of the person using it. The assumption is made that the person is using the software for the first time and needs to have a good experience. Otherwise the user will reject the software, not use it again, and it might feel bad! So the software needs to be “user-friendly.”

But what if the people using the software use it for a good part of the day, every day? If we can make them just 10% more efficient, then that’s worth something. So what if they need some training at the beginning? And the fact is, in many cases we’re looking at way more than 10% improvement. In large categories improvements of 50% are available, and in some large-scale cases, substantial whole-number factors of gain.

Since everyone thinks they know how to build user-friendly software (if only more of it were!) and since most programmers don’t even think about high-use, high-productivity software, here are the main principles of building a user interface whose users will spend a great deal of time using, and who want to get more done in less time:

• Arrange the work to minimize movement
  • The first and most important step is to eliminate is anything that takes the human’s eyes or hands away from the computer. Paper is a prime example – having an image of paper on the screen is vastly more productive than having a physical paper.
  • Given that eyes and hands are on the computer, the next step is to eliminate the use of the slow input device – the mouse or touch pad. Everyone loves the mouse. They think requiring its use is the most user-friendly thing you can do. But we’re talking about productivity here, and in any productivity race between mouse and keyboard, the mouse is a sad, distant last. So with the possible exception of logging in and out, just lose the mouse. Don’t compromise or be “nice” about it.
  • Now that all inputs are keyboard inputs (and they are, aren’t they???), reduce the number of keystrokes to the bare minimum. You would be amazed, when you count keystrokes (yes, you should actually count keystrokes; yes, you), how many can be eliminated in the average application.
  • Finally – don’t laugh – minimize eye movement. It’s not the time, it’s the attention.
• Arrange the work to minimize thought.
  • Sometimes, like when people are writing something, they just need to think. That’s OK.
  • But any other kind of thinking is just a waste of time. Think about your own thought-free actions; I hope typing is a good example. When you have to look at the keyboard or think about it at all, instead of just typing, what happens to your rate of typing? Does it improve as a result of the thought? I didn’t think so.
• If some training is needed at the beginning – OK; if training is needed on an on-going basis, you are probably making the user do things the computer should do – instead of training, automate.
• Embodying domain knowledge is really tempting. You should make your system so that the users don’t need domain knowledge. It’s made particularly hard because you typically need domain experts at the beginning to make sure your system is sensible. They want to see the way they think about the problem embodied in the system.
  • A good example is working with health care forms. There is a huge amount of domain knowledge involved in doing this right. But the vast majority of this knowledge can be put into the system, so that the people end up just doing things that only people can do.
• Arranging the UI and the people so they know what they’re doing and why is really tempting – resist that temptation! The more your users just do their work, focusing just on productivity and accuracy, leaving the rest to the “system,” the better off everyone will be.
  • A prime example of this is QA. In most systems, the users know whether they are doing the work for the first time or checking someone else’s work. This knowledge is built into the user roles and the queuing system. It’s true that in some cases this can’t be avoided, because of the nature of the work. But you’re well advised to avoid it, if you possibly can.
  • What does role hiding look like? An example is in heads-down data entry. In the early days, experienced people would look at what was supposed to be typed, look at what was actually typed, and check for errors. It turns out that it was faster and more accurate to simply have the “checker” enter the data as though for the first time – and it always was the first time they were entering it. Then the system would compare the results, and flag an inspection or a third keying to resolve the conflict. This came to be called “blind double-key entry,” and remains the gold standard for productivity and quality in the industry.
  • Why does role hiding work? When someone knows what someone else thought the outcome or result should have been, it influences them, one way or another, and it takes them time. It’s like when you suspect a teacher of giving biased grades or you want to know how good a food is. You get the most objective results by giving the tests to second teacher for grading because the (anonymous) other teacher got sick (or something), or you give the food for testing without labels of any kind. Remember the famous tests of Pepsi vs. Coke? Coke drinkers would always prefer Coke when it was labeled as Coke – but when two colas were unlabeled, most would prefer Pepsi, even the Coke loyalists.

Building a UI that optimizes the productivity of the people who use it is a new and different way thinking for many software folks. But it's well worth pursuing -- the results speak for themselves, and the professionals who use the UI will appreciate being able to get their work done in less time.

The Progression of Abstraction in Software Applications

This post describes a little-known concept for understanding and creating software architecture that small groups use to defeat large, powerful incumbents and nimble competitors. It is one of a small number of powerful, repeating patterns that help us understand and predict the evolution of software. Understanding these patterns can help entrepreneurs direct their efforts; if they do it well, they greatly enhance their chances of success. Understanding the patterns can also help investors choose to invest in groups that are walking a path to success.

Evolution of Applications Towards Abstraction on a Platform

One of these patterns is the stages that applications naturally evolve through on a technology platform. Each step or stage brings a big increase in the power of the software, decreasing the effort and increasing the speed and effectiveness of being deployed to meet customer needs.

A category of software applications may well get “stuck” at a particular stage for a long time, sometimes even decades. During this time, the software may appear to move forward, and of course the marketing people and managers will always put things in the best possible light. But it’s always vulnerable to being supplanted by a next-stage version of the functionality.

While there aren’t clear lines of delineation between the stages, it’s nonetheless useful to understand them roughly as:

• Prototype. A hard-coded body of code.
• Custom Application. Does a job reliably, but most changes require changing source code.
• Basic Product. The code now has parameters, maybe user exits and API’s. Real-life implementations tend to require extensive professional services, and the cost of upgrading to new versions tends to be high.
• Parameterized Product. The level of parameterization is high, with interface layers to many things outside the core code, so that many implementations can be done without changing source code. There may be some meta-data or editable rules.
• Workbench Product. A large portion of the product’s functionality has migrated from code to editable meta-data, so that extensive UI, workflow, interface and functionality changes can be accomplished via some form of workbench, which could be just a text editor. The key is that the details of application functionality are expressed as editable data, meta-data, instead of code. Nonetheless, all fundamental capabilities are expressed in highly abstract code.

As a body of code goes through this sequence of abstraction, it is increasingly able to meet the needs of new customers and changing needs of existing customers, with decreasing amounts of effort, risk and changes to source code. At the same time, the more abstract a program, the more functionality is expressed as data that is not part of the software itself, a.k.a. meta-data, and the more the software implements generic capabilities, as directed by the meta-data.

The pattern applies both to individual bodies of code and to collections of them. It applies to code built internally for an organization and to code that is sold as a product in any way.

I defined the stages above as a convenience; in reality, the categories are rarely hard-and-fast. A body of code could be given a big transformation and leap along the spectrum, or it could take a long series of small steps. One body of code could remain stuck with little change in abstraction, while other bodies of code doing similar things could be ahead, or progress rapidly towards abstraction.

The Driver of Abstraction Evolution

In biological nature, competitive pressures and external change appear to drive evolutionary changes. Similarly, when we look at categories of software, if little changes to make the software change, it doesn’t change – why take the trouble and expense to change software that meets your needs or the needs of your customers?

In reality, someone always seems to want changes to an application. A prospective user would gladly use the software if it did this, that or the other thing. A current user complains about something – it’s too slow, too hard to use, too complicated, or it just doesn’t work for X, Y or Z. How often does a piece of software not have a “roadmap?” If it doesn’t, it’s probably slated for retirement before long. Brand-new software is rare. The vast majority of software effort goes into making changes to an existing piece of software.

How much time and effort is needed to change a particular body of software? That is the key touch-point between techies and business people. This is the point at which the level of abstraction of the application comes into play. Regardless of the level of abstraction of the application, the “change” required either has been anticipated and provided for or it has not.

• If the change has been anticipated, the code can already do the kind of thing the user wants – but not the particular thing. Doing the particular thing requires that a parameter be defined, a configuration file changed, a template created or altered, workflow or rule changes made, or something similar that is NOT part of the program’s source code. This means that the requirement can be met quickly, with little chance of error.
• If the change has not been anticipated, then source code has to be changed in some way to make the change. The changes required may be simple and localized, or complex and extensive – but the source code is changed and a new version of the program is created.

This is what the level of abstraction of a software application is all about: the more abstracted the application, the more ways it can be changed without altering the source code and making a new version of the program.

This is the fundamental driver of applications towards increasing abstraction: as changes have to be made to the application, at some point the technical people may decide to make the change easier to make in the future, and create the appropriate abstraction for the kind of change. This may happen repeatedly. As more complex changes are required, the program may just get more complicated and more expensive to make further changes, or more sophisticated abstractions may be introduced.

While historically it appears that outside forces drive applications towards increasing abstraction, smart programmers can understand the techniques of abstraction and build an application that is appropriately abstract from the beginning. Similarly, the abstracting methods can be applied by smart programmers to existing bodies of code to transform them, just because it's a way to build better code and meet ever-evolving business requirements.

Conclusion

The hierarchy of abstraction in software is one of the most important concepts for understanding a specific piece of software, or a group of related software. Over time, software tends to become more abstract because of competitive and business pressures, or because of smart programmers working to make things better. The more abstract a piece of software is, the more likely that it can respond to business and user demands without modifications to the source code itself, i..e., quickly and with low risk.

The hierarchy of abstraction is certainly a valuable way of understanding the history of software. But it is more valuable as a framework for understanding a given piece of software, and the way to evolve that software to becoming increasingly valuable. It is most valuable to software developers as a framework for understanding software, and helping them to direct their efforts to get the greatest possible business impact with the smallest amount of time and effort.

The Fundamentals of Computer Automation

The principles underlying computer automation are clear and strong. They account for most of the automation we see. They tell us clearly what will happen, why it will happen, and what the benefits will be. What the principles do NOT tell us is who will first apply automation to what process in what sector.

Understanding the principles lets you predict the rough order of the sequence of automation.

The principles are extremely simple. Perhaps that's why they're rarely stated and appear not to be taught in schools or understood by practitioners. So much the better for people who want to innovate and win by automating!

The Principles

You can probably guess the principles of automation from the definition of the word: "automate" means to do by machine or computer what a human would otherwise do.

At core, there's really a single simple principle:

• Replace time that a person would have spent with work done by a computer.

The core principle is demonstrated by a before-and-after comparison: the principle requires that human time/value doing whatever activity is being automated, net of the investment in the computer hardware and software, is reduced after the addition of the computer and software. Lots of words, same meaning: apply the computer stuff, and there's less total human time/effort.

Expanding on the replace theme, "replace" means anything that reduces human time to do something:

• Replace time that a human would have spent with work done by computer; REPLACE!
  
• Reduce the amount of time spent by a human; REDUCE!
  
• Arrange the human's time more efficiently, eliminating waste; RE-ARRANGE!
• Decide Who does What work, How and Where they do it; RE-ORGANIZE!

That's it! 

This notion of what software was REALLY about struck me quite early in my career. The thought was simple: the ultimate purpose of most of the software I wrote is to replace humans. The thought made me uncomfortable. But after a while, I connected the thought with all the rest of the mechanization and industrialization in society for hundreds of years. A new machine is valuable only to the extent that it somehow reduces the total amount of human labor to reach a given result, everything taken into account! This was true for the Jacquard Looms to which the luddites violently objected in the early 1800's, and the same principles are at work today with computers, resisted by modern-day luddites.

In other words, computers are the next step in a long sequence of people figuring out how to get stuff done with less effort by themselves and/or other people.

Here are some simple examples:

REPLACE

Replace what humans do with computers doing it is the core principle of automation. E-mail is a super-simple example, since computers automate the entire process of delivering the result of typing to another human being, eliminating the many steps involved in doing it physically. The replacement can be low-level and mechanical, like delivering email, or it can be high-level, like the work of deciding who should do what work in what order.

REDUCE

Electric light is a pre-computer example, since it reduces all the time required to, for example, deal with the oil lamp, and replaces it with flicking a switch. A modern example is spreadsheets, which automate the time people used to spend making calculations with adding machines.

RE-ARRANGE

Introducing chat into customer call centers eliminated huge amounts of wasted waiting time, so that phone operators could handle chat when they weren't on the phone.

RE_ORGANIZE

Routing certain kinds of calls to the optimal customer service person (Who does What work), giving them scripts and hints.

The Principles apply to Manual and Mental work

At the start of automation, the computer and software automate work that it's pretty plain that people are doing. As it gets more advanced, a couple interesting things happen.

1. Once the work is done by computer, it sometimes happens that better ways of getting the work done are possible. This is the same principle we see in mechanical flight: even though all birds have wings that flap, airplane wings don't flap. Instead of flying the ways birds do, engineers have figured out of the core principles of flying and reflected those in airplane design. Similarly, human work is often first translated to computer automation simplistically, and later rebuilt without reference to the human origins.
   
2. Once computers are a significant part of the landscape, computer people figure out valuable things to do that have little to do with what humans have done in the past, though this is uncommon.

The Principles are recursive

At the beginning, getting computers to do things is incredibly labor-intensive. As time goes on, some people look at the work humans do to get computers to do things -- and automate the automation process! Yes, the very programmers whose job is to put people out of work can be put out of work by computers. And on and on.

The Principles applied to human interaction

We can see the principles in action in the sequence of changes that have taken place in normal interactions between human beings.

Prior to automation, the only way humans could communicate was by voice, and by being in each other's physical presence. Here are the major automation steps that have taken place:

• Writing a letter.
  
  • Automation is making and acquiring paper, pen and ink, and transporting letter
    
  • Eliminates the travel time of the sender and/or receiver
• Publishing books, newspapers.
  • Automation is printing and distributing the books or papers
  • Eliminates the travel time of all the receivers
• Telephone
  • Eliminates the travel time for the parties to get together
  • Eliminates the time spent writing and reading, and the delay in transmission of prior methods
    
• Mobile phone
  • Eliminates the need to travel to a phone to make and/or receive calls
• Voice mail
  
  • Eliminates the requirement that the receiver be available when the caller calls, for those cases when it's applicable
    
• E-Mail
  
  • Writing a letter with near-real-time delivery and the ability to read when ready, like letter reading and voice mail.
• Chat
  • Like email, only with a different interface, making it better for real-time, written exchanges, with minor delays acceptable.
• Scripting
  • In customer service centers, reduces training, reduces human error, and reduces the time to produce an optimal answer to a question.
    
• VRU, chatbot
  • A recorded and/or machine-generated party in a conversation.
  • Applicable to all of the above methods.
  • Eliminates a human giving repetitive responses or messages.

This all seems obvious, right? Like we know all this stuff, what's new?

For the steps above that are computer-based, what's interesting is that the technical capabilities were often in place considerably before they were brought into production. The gap was usually not decades-long, like it has been with more esoteric technologies. And in each case, knowing the technology and this fundamental principle could enable you to predict that the automation would be built and deployed.

Conclusion

If you want to predict the future of technology and evolution, study the evolution of software and the fundamental principles that drive its application. The principles are obvious! But they are nonetheless rarely discussed or applied.

 

Luddites Smashed Looms then and Resist Automation today

The few people who are familiar with the term "luddite" think that a luddite is an unfortunate but stupid person who fights against advancements that make things better, while clinging bitterly to their crappy, low-end jobs. "Luddites" in this common view, are uneducated, progress-preventing people who need to be moved to the side so that society can be improved.

The reality is that, in most cases, luddites were highly skilled craftsmen performing difficult and challenging jobs. It's not that much different today. Luddites are often highly educated professionals and managers who are convinced they bring value to their complex jobs every day The reality is that there are luddites everywhere. Who wants to have their job disrupted? Who wants to be told that their expertise is no longer needed, and that a machine or software system can do a better job?

Stepping back

This notion of what software was REALLY about struck me quite early in my career. The thought was simple: the ultimate purpose of most of the software I wrote was to replace humans. The thought made me uncomfortable. But after a while, I connected the thought with all the rest of the mechanization and industrialization in society for hundreds of years. A new machine (or program) is valuable only to the extent that it somehow reduces the total amount of human labor to reach a given result, everything taken into account!

The replacement of humans by machines was extremely clear to the luddites, the secret society of workers whose jobs were being eliminated or reduced to unskilled labor by the increased use of Jacquard Looms. The movement was named after one of the first people to smash a loom, Ned Ludd:

The core of the movement was actions to destroy the job-killing looms that automated and de-skilled the worker's jobs:

This is particularly interesting and relevant to computers eliminating human labor. No one thinks of what could be the modern equivalent of workers smashing the looms that threaten their jobs. An ironic twist is that the revolutionary Jacquard looms were mechanical computers -- they executed a "program" that was encoded physically, enabling them to execute flawlessly elaborate patterns in the woven cloth.

Luddites today

Yes, there are luddites today. Lots of them. But they're not so crass as to pick up big sledge hammers and smash the looms that threaten their livelihoods. They're more subtle, and far more supported by elite society than the original luddites ever were. They're not even seen as resisting change -- they're seen as highly trained professionals that we're so grateful to have. Very much the way that the skilled craftsmen displaced by machines were seen by themselves and most of their contemporaries!

So who exactly are these modern luddites? They include many of the elite, highly-paid professions that ambitious young people strive for.

• Lawyers. A growing fraction of lawyers are under attack by automation. The pain is shown by declines in law school graduate hiring, the growth of firms such as Legalzoom, and growing outsourcing to low-wage locations. The trend has been growing for at least 20 years. I have personally seen the resistance of lawyers to automation efforts.
• Doctors. The ready-to-be-luddite feelings are strong here, though automation is still in its early stages. Doctors increasingly feel like they work in a non-stop assembly line, with patients questioning their statements because of Doctor Google. With insurance companies increasingly questioning and challenging their actions and clinical decision-tree software chomping at the bit, doctors can feel like targets. While hiring remains strong, automation efforts continue to gain strength.
• Software engineers. It seems only fair that the people who drive automation should have their jobs automated. This isn't in the news, but in fact there have been many waves of software job elimination -- masked by strong growth in newly emerging technologies. I have run into many programmers over the years who jobs were eliminated; some of them have left the field, and others slide into management positions, managing people doing jobs they barely understand. For example, during the 1970's there was an explosion of jobs building operating systems for minicomputers. Minicomputers are long gone, along with the diverse operating systems needing to be built. A long trend is skills degrading -- today you can build a functioning web site without programming, while years ago building software with a sophisticated user interface required serious software effort.
• Actors and musicians. We have more visual and musical entertainment available than at any time in human history. But the vast, vast majority of it is recorded and replayed! The vast majority of musicians, for example, were employed making adequate, best-available performances at local places. The number of such jobs is a tiny fraction of what it once was because of recordings. All the supporting jobs are greatly reduced as well. Even jobs in movie theaters have cratered, since people watch on personal and home devices.

What will become of all those people and jobs? The answer has been the same for hundreds of years: there will be great pain for the individuals involved, but overall, the wealth of the population will grow. A clear example is agriculture. At the time of the American revolution, over 90% of the population was involved in agriculture. One step at a time, those jobs were automated, so that in 2019, only about 1% of the US population was employed in agriculture. Yet there's food enough for everyone, and jobs that didn't exist back then.

Conclusion

As a person who has been directly involved in automation for over 50 years, I have seen the jobs I've had and the skills I've acquired become obsolete over and over again. I've also seen or been part of eliminating through automation many jobs, a surprising number of them highly skilled jobs. An early one was the capable engineers who ran the oil refinery in Venezuela in the late 1960's whose jobs were eliminated by the software I worked on at the time -- the software just did a better job than the engineers could possible do! I describe this here.

Luddites can and often do postpone their replacement by automation. But they lose in the end. I don't see an end to this process any time soon.

Getting Results from ML and AI 6: Fintech Chatbot

While the success patterns laid out in the prior posts in this series may seem clear in the abstract, applying them in practice can be hard, because nearly everyone who thinks or talks about AI (these sets over overlap very little, sadly) takes a different approach.

http://www.blackliszt.com/2018/03/getting-results-from-ml-and-ai-1.html

http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-2.html

http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-3-closed-loop.html

I previously discussed the application of the principles to healthcare, with specific examples:

https://www.blackliszt.com/2018/08/getting-results-from-ml-and-ai-4-healthcare-examples.html

I've discussed the application of the principles to fintech, with a focus on anti-fraud:

https://www.blackliszt.com/2019/12/getting-results-from-ml-and-ai-5-fintech-fraud.html

In this post, I'll show how things can play out with a stellar example in fintech chatbots.

Computers talking with People -- Using People Language

The idea that computers should talk with us in our language rather than people struggling to learn computer talk has been around nearly as long as computers have. The earliest "high level languages" like FORTRAN and COBOL were each attempts to let people use English-like language for telling a computer what to do. They were baby steps, and people quickly decided that computers can and should do better. This thought was one of the earliest practical drivers towards Artificial Intelligence (AI) in general, and Natural Language Processing (NLP) in particular. 

One of the acknowledged milestones towards a computer that can talk like people was the work of Terry Winograd at MIT during 1968-1970. He created a talking robot, SHRDLU, that could talk about and act in a special world of blocks:

SHRDLU could have conversations about the block world that amazed people at the time. Here's an excerpt, see this for more.

I was at college at the other end of Cambridge at the time, heard about SHRDLU, and got even deeper into the AI work I was engaged in at the time as a result, for example this project. SHRDLU was wonderful, but I wondered about and worked on how to represent knowledge and actions internally. After I graduated, "everyone" was convinced that "talking computers" were going to burst on the scene any day now. A couple decades passed, and nothing.

Talking Computers Today

Here we are, 50 years later, and things have definitely advanced. We have Alexa and Siri, and in banking we have Bank of America's much-promoted Erika. But in many ways, these programs remain primitive, and are far from being able to understand context and sequence the way people do.

One of the reasons for this is that human language understanding and generation is really hard. Another is that the vast majority of people who work on the problem are thoroughly immersed in the other-worldly vapors of academia, in which publishing papers and gaining esteem among your fellows are the all-consuming goals -- to the exclusion of building products that do things that normal people value and, you know, work.

The State of the Art in Chatbots

The founders of the Oak HC/FT portfolio company Kasisto come from that world, and also from the industrial labs of Stanford and IBM that try hard to commercialize such fancy stuff, with products like Siri as results. The Kasisto people are obviously exceptional not just in their field, but in their drive to make code that does real things in the real world. If that's your goal, there is exactly one standard for measurement: what people do and say.

That's the background of the first remarkable thing about Kasisto, which I learned when I probed details of their process. Pretty much everyone who deals with AI/ML builds and builds in the lab, until they have achieved amazing things that they're ready to roll out ... which promptly belly-flops in the real world. We'll do better next time, promise! Kasisto works the way everyone should work, but practically no one does -- people first!

This means they get in the middle of a huge number of human-to-human chat sessions, building and tuning their software first by human judgment, but increasingly by machine judgment. They "try" to answer the human's question, and just as important, they rate their ability to answer the question appropriately. After tens of thousands of tests, their ability to respond like a human would improves -- and just as important, their self-rating of how well they're likely to do improves as well.

As their answers and the associated self-ratings get good, they start actively playing a role in the live question-answer flow. For exactly and only the human questions for which Kasisto is "confident" (using an adjustable threshold) it will answer well, its answers go to the human -- but are still routed to a human for double-checking, with a frequency that goes down over time.

There will always be questions that are beyond the capability of the Kasisto bot, but so long as it is "self-aware" of which ones, the customers continue to get great service, with an ever-shrinking fraction of the questions being routed to humans for answers. If Kasisto can handle 95% of the questions, this might means that instead of a staff of 100 to respond to customers, a staff of 5 could do the job.

This way of thinking is far removed from the typical academic head-set -- and I've just given a couple highlights, there's actually much more where that came from!

Beating the Pack

What I've already described enables Kasisto to deliver chat results to customers that are far superior to anyone else in the field. But the overall head-set and people-first technique, combined with some true tech smarts, leads to further things.

The first thing is context. All the other chatbots are lucky if they can give reasonable answers to isolated questions. The combination of human-first, good tech and practical methods enables Kasisto to not only answer isolated questions far better than others, but follow-on questions as well -- questions that make no sense by themselves, but perfect sense when taken in the context of an interactive sequence. As a simple example, consider this:

"How much did I spend last month?"

"How much of it was on restaurants?"

"How about the previous month?"

That's a simple one for humans, but way beyond what other chatbots can do. Why? The computer has to figure out that "how much of it" means "How much did I spend on restaurants last month?" Not to mention, handling multiple languages; performing transactions; making changes and selling products.

While Kasisto started with consumer banking, it's already added high-value functionality for treasury, business banking and more. The key is: Kasisto, unlike everyone else in the field, didn't just start selling into those applications -- they followed the laborious but effective, bottoms-up, people-first method of exhaustive training and seamless integration with humans doing chat.

Conclusion

Kasisto is an excellent example of the incredible results that can come from following the path briefly outlined in these blog posts. You would think with all the talk about AI and ML, and the all efforts and announcements, that highly capable systems would be popping out all over. They're not! That makes the few that follow the success patterns I've discussed here all the more impressive.

Getting Results from ML and AI 5: Fintech Fraud

While the success patterns laid out in the prior posts in this series may seem clear in the abstract, applying them in practice can be hard, because nearly everyone who thinks or talks about AI (these sets over overlap very little, sadly) takes a different approach.

http://www.blackliszt.com/2018/03/getting-results-from-ml-and-ai-1.html

http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-2.html

http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-3-closed-loop.html

I previously discussed the application of the principles to healthcare, with specific examples: https://www.blackliszt.com/2018/08/getting-results-from-ml-and-ai-4-healthcare-examples.html

In this post, I'll show how things can play out with a stellar example in fintech anti-fraud.

The Use of ML in Fraud Detection

Credit card fraud is a difficult, ever-evolving problem, not unlike cyber-security in general. In most cases, real money is involved, billions of dollars of it -- over $20 Billion world-wide as of a few years ago!

Robert Hecht-Nielsen was a pioneer in machine learning in the 1980's. He started HNC software to apply ML to a variety of problems. He was particularly fond of Neural Networks. After a few years of not doing well, he encountered a visionary executive of Household Finance, a large company that provided credit, including cards, to sub-prime customers. The HFC executive convinced Hecht-Nielsen to concentrate on card fraud, which was a large and growing problem for the card industry in general, and HFC in particular.

With HFC's support, HNC built the first effective card fraud detection system. It centered around humans analyzing recent instances of card fraud, building a detector for each particular kind, and training neural nets to combine all the detectors to be applied to each new transaction as it came through.

Since the model's effectiveness depended on having an ever-growing library of fraud detectors, HFC supported HNC creating a network model, in which each participating card issuer would contribute the details of each new case of fraud it encountered. HNC analysts would add detectors, train new models and give updated fraud detecting models to each participating company. It quickly became the industry standard, since all the participants benefited from the experience of all the members. HNC went public in the mid-1990's and later merged with FICO.

Enter Feedzai

How could any new entrant possibly compete against the data monopoly and neural networks of FICO/HNC? Somebody new wouldn't have access to the incredible database of card fraud. Feedzai, backed by Oak HC/FT, had some things that were more important.

First of all, it helped that the Feedzai founders were broadly expert in machine learning and other analytic methods. It took them a while to focus on "just" card fraud. Being a general expert and then deciding to focus on a narrow problem is just like HNC, and is a general success pattern. But HNC got frozen and dabbled in many other domains, failing to continue to innovate in their core product.

Once Feedzai got engaged with a card issuer, they noticed that the FICO solution was so slow that it couldn't detect whether an authorization was fraudulent in real time -- only after it was already approved. Result: every fraudster got at least one free! Beating FICO would require real-time analytics. Next, Feedzai noticed that FICO was trying to see if each transaction was BAD, and that seeing if each transaction was NOT GOOD was just as good -- even better, since you don't care how inventive the bad guys are, just that they're not acting like the real cardholder would act. That meant you could train your models on a single card issuer's customer base, ignoring the rest of the world. Cool! Finally, that meant you could use modern, transparent ML algorithms, and dispense entirely with the time-consuming and error-prone FICO method of modeling each type of fraud. And that meant that your first couple of appropriately skeptical customers could try you out, side-by-side, against the incumbent FICO, and see who won the catch-the-fraud race. Hooray, Feedzai won!

It's important to note that Feedzai also won because they did all the foundational steps described in the earlier posts in this series correctly, including the all-important know-your-data and closed-loop steps.

FICO/HNC moved on from their success in card fraud, thinking they had an absolute lock on the market -- which they did, for many years, but only because someone like Feedzai didn't go after them sooner. Feedzai isn't making the FICO/HNC mistake; they are continuing to build on their lead, both deepening and extending it. It's a big subject, but here are a couple of the highlights:

• Feedzai has built a tool that does much of the work highly skilled engineers used to have to do when supporting a new client. The original tool detects fraud; the new tool automates building a fraud detection tool. The speed and quality that results is unprecedented.
• While rarely discussed, every practical detection system has a set of human-created rules. Feedzai is the first company to automate the evaluation of the rules, and decide which should be changed or dropped. This is huge.
• Feedzai is now, step by step, introducing their technology to adjacent areas such as AML. The incentives created by the heavy regulation in AML are perverse, and has led to bloated costs in banks with no real improvement in AML detection. Feedzai's novel approach is actually making headway in spite of the regulations, and not just doing normal reg-tech automation.

Conclusion

Being an excellent generalist is often a good way to become a superb specialist. But it's rare to find a group of people who are truly up to snuff in the abstruse technologies of AI/ML, but also able to dive into the ugly details of data and real-life cases to make a system that beats the real-world incumbents. As we know from FICO/HNC and many other cases, the pattern is then to declare victory and move on. Feedzai is an object lesson in how to go deeper and deeper, continuing to innovate and automate. They are an excellent example of the principles described in the earlier posts in this series.

Cybersecurity is Almost Impossible to Achieve -- Here's Why

As with most computer software issues, cybersecurity is badly misunderstood by the vast majority of people, including, sadly but as usual, most computer professionals. The result of this is that the vast majority of people have wrong ideas about the source and methods of security breaches and how they can be prevented. Unfortunately, sometimes these wrong ideas have major consequences.

A Typical Phishing Attack

Phishing is a kind of attack on a user by a bad guy. The bad guy sends the target an email that contains something to tempt the target to click on it. Clicking on the hyperlink is like a fish biting the bait on the hook -- you're caught!

I've been the target of a slew of phishing attacks in the last couple of months. Here is a typical email I've received:

 

What a scary email! It's to me, from my company's email administrator, and I'd better do something about it!

An amazing number of recipients of such an email would have clicked on the red "Resolve Errors Now" box. I wouldn't because:

• I know that my email admin person's domain is not "team-admin.net"
• There are multiple suspicious minor grammatical errors.
• Why would the final line be a copyright?

And many other reasons.

I went to the trouble of digging a bit. I exposed the URL at the hyperlink I was supposed to click and copied it. Here is the place I would have gone had I done what the phisher intended me to do:

Yup, it's a one-time domain set up by the phisher with the information about who I was so that the right next hacking steps could be executed -- that's the string after the question mark. This is not an amateur phisher!

There are more where that came from.

The email above was hardly an isolated example. Just yesterday I received an email from someone I knew at another VC firm, asking me to click to download an important document. Half an hour later, I got a legit email from the firm's admin saying sorry, the sender's account was hacked, please don't click on the email, and if you did, you may want to change all your passwords.

Here is another I recently received, apparently from DHL instead of apparently from my email administrator:

Yes, the place they want you to click is totally bogus -- and bonus points for doing an exceptional job mimicking DHL!

Cybersecurity Experts Know all about Phishing Attacks!

Sure they do. It's on everybody's list, and they all put "solutions" in place. Solutions that don't work.

A colleague of mine has been a top tech executive in major financial institutions. He told me how none of the anti-phishing solutions work because too many valid emails end up in the "spam" folder, and/or too many phishing attacks are passed through as OK. If users know there's a phishing filter in place, they tend to assume that every email in their inbox is OK, and click away, making the problem worse.

The executive tried phishing education for users. He ran several different kinds of education, and sent test phishing emails to people as tests of the education. Net result: no form of education improved the practical willingness/ability of users to notice and avoid clicking on phish attacks. Conclusion: highly paid office workers are too stupid to be educated on this subject.

The Place of Phishing Attacks in Cybersecurity

Regardless of your familiarity with computer security, I suspect you've got the image nearly everyone has about it -- an image that is like a medieval castle with high walls and ponderous, thick gates that are heavily guarded. It's all about those evil bad guys roaming around out there, always probing for ways to get around the gate or over the wall to get inside and wreak havoc.

This is a fun image, and accurately reflects something which is part of the truth. Phishing attacks, along with insiders who have become bad guys, are an important part of the rest of the truth. While there are "solutions" to phishing, they're mostly badly-performing filters that dump loads of emails into a "spam" folder, many of which are perfectly legit. Making you manually go through it anyway. What's the point?

Phishing and bent insiders are the poor step-children of cybersecurity, always acknowledged as being part of the family, but surviving on hand-me-downs and left-overs as best they can. This in spite of the fact that they continue to be the source of the most flagrant "breaches" of security we know about! High-end retail stores and even libraries have had working solutions to this kind of problem in place and working since before computers were invented, but there's something about being a fancy-shmancy computer science cybersecurity expert that prevents such solutions from being considered.  Among other things, check out:

https://www.blackliszt.com/2017/05/computer-security-problems-solutions.html

Conclusion

The image of bad guys cackling in distant, dark rooms lit with screens, as they manage increasingly devious attacks against the walls and defenses of the innocent computer systems of the world continues to cause regulators, cybersecurity experts and the other grandees of computer administration to ignore the real threats -- the ones that work, the ones that actually lead to failures of computer security. It's hard to grasp the full extent of combined stupidity, self-satisfaction and blindness that leads the ongoing non-response to bad actors recruiting insiders to do their work for them.

 

Take-aways from the 2019 Edition of the Money 2020 Conference

The 2019 US edition of Money 20/20 is a wrap. All of us who attended are recovering – and digesting.

Money 20/20 is now an established event – even though the first one was just seven years ago, in 2012! You might think it would be getting stodgy and repetitive by now, with bosses sending their underlings. Not so! Among the over 7,000 people attending were an amazing number of top, big-company executives – along with hundreds of startups, investors, and companies ranging from emerging to established. All the tech firms you’d expect also attended.

So what happened? Anything new? Big themes? Here are a couple:

Entrepreneurs

The startup energy was amazing, partly because there were literally hundreds of startups in attendance, many in little booths, many participating in the events and meetups that were organized for them and the larger groups – investors, partners and buyers – who want to know what’s going on. There’s no way I’m going to pick a couple as examples of the hundreds, but as a test, I’d suggest you come up with a couple ideas you think are novel, walk next year’s show, and see how novel your ideas really are. Unless you’re incredibly better than most, you’ll find a startup with your idea – and if not, maybe it’s your time to start a company!

The Feedzai Financial Crime Summit

We’re glad to be investors in Feedzai, and amazed at the sub-conference they’ve run the last couple of years. It’s too bad Agatha Christie couldn’t attend this year to help solve financial crimes, though the stellar crowd recruited by Feedzai did a fine job of it. The top person from Microsoft described how he spends nearly $1 Billion a year on cyber-security, with competing teams of hackers and defenders, and much else.

With so much of money being electronic and transactions being conducted on phones and computers, the silent, remote hacker is indeed the modern equivalent of Willie Sutton – whose answer to why he robbed banks was simple – because that’s where the money is. Now that the money is in computers and transmitted by electronic networks, that’s where the money is, and there was a great deal to be said about this ongoing battle between good guys and bad guys – and the rapidly escalating arms race that makes the outcome of any one battle so uncertain.

It’s clear that this topic will continue to grow in importance in coming years.

Cryptocurrency and blockchain

There was a whole track devoted to this hot topic, and some time on the big stage. In particular, David Marcus, the leader of Facebook’s Libra cryptocurrency effort, had the center seat of the giant center stage to himself, answering softball questions from a moderator. His smooth explanations acknowledged the widespread attention and questioning the Libra effort has engendered, while giving comforting answers and putting a calm, confident face on the effort.

There was quite a contrast between all the sessions devoted to this subject and what you saw on the show floor, which wasn’t much. Given that Money 20/20 is a show about payments, and given that cryptocurrency is, after all, a kind of currency, it’s natural there should be widespread interest in the topic, fueled by a generous portion of FOMO. There are lots of efforts and noise on the subject, but still not much evidence of real action. Everyone follows and will continue to follow the high-profile Facebook Libra effort with interest.

New Technologies: AI and ML

AI and ML continue to be the buzzwords nearly everyone wants to be associated with, in one way or another. This was particularly remarkable when you walked the show floor, with a high fraction of the booths claiming leadership in these subjects. Given the arcane nature of the underlying technology and the widely diverse technologies that are reasonably described as AI or ML – combined with the fact that most peoples’ eyes quickly glaze over when a nerdish person tries to explain what makes one of the dozens of machine learning algorithms different from another – it’s clear that the marketing person has an impossible task, and most of the show attendees have a challenge figuring out which approach is better than the others.

Nonetheless, this is one of those cases where the hype is clearly justified. For example, Feedzai is catching more of the bad guys than its direct competitors because its technology is just plain better, even though the others arguably also use AI/ML.

Eventually, people will figure out that this has happened before. It’s like decades ago, when a few leading companies claimed to be better than the competition because … they used computers! Yes, those incredibly advanced things that no one really understands. It wasn’t long before everyone first claimed to be using them, and then, eventually, everyone still in business was actually using computers! This year’s Money 20/20 clearly demonstrated that we’re still in the intermediate stage, in which the claims to be using AI/ML are widespread, and the reality is still in catch-up mode. We’ll know we’re really there when it’s just assumed – if you’re still in business, you must be using AI/ML; the only question is, who does it better. We’re getting there!

Lending, payments, banking

The show contained an amazing demonstration of continuing innovation in making it easier for people and businesses to send money, get money and borrow money. The days of checking that you’re properly dressed and groomed for your important meeting with the bank lending officer are long gone. Even the days of filling out forms, not just on paper, but also on the computer. As ever-growing portions of your financial history are stored and available in various online repositories, the job is mostly making sure you really are who you say you are. Once that’s settled, a couple simple actions on your phone are increasingly sufficient to get, send and borrow money, cheaply and quickly.

We saw an array of companies from point products to comprehensive, one-stop services, coming ever-closer to making every aspect of banking as quick and painless as messaging on your phone. The competition is fierce. It’s enough to make you wonder why the big banks survive. But all the big-name players are in the game and were at the show as well, with their own approach to making transactions as easy and efficient as possible for consumers and merchants. While everyone likes convenience, change isn’t particularly convenient; the big institutions have most of the customers and aren’t going to give them up without a fight. So they’re offering their own brands of ease and convenience to their customers. It was all on display at the show, and it’s clear that the war for the customer is far from over.

Regulation

As one of the most regulated industries, payments is doing an amazing job of innovating while avoiding the long arm of the bureaucratic enforcers. This is partly because regulation technology is advancing, as we saw at the show. But it’s got a ways to go. Automating regulation is a bit like teenagers eating vegetables – they respond to parental demands, but it’s not the first choice. As regulators continue to ratchet up the stakes and catch up to modern technologies, this is a field that continues to grow in importance at Money 20/20, both in the speakers and the exhibits.

Money 20/20

Next year will be 2020. Will this show be re-named Money 20/30? Or will moving towards 20/20 vision be the metaphor that justifies sticking with the name? Regardless of the name, the show will go on. It’s the one place you can go and see everyone who is anyone, and everyone who wants to be someone, all in one place in a short period of time. We’re already looking forward to next year.

Facebook’s Libra is not a Cryptocurrency

“Everyone” says that Facebook’s Libra is a cryptocurrency. Long before Libra had been imagined, Bitcoin pioneered and established the brand new world of cryptocurrency. Bitcoin created the category, and has always been its leading exemplar. The white paper by the still-unknown Bitcoin creator and inventor spelled out his design goals and the main aspects of Bitcoin that supported those goals. Once you read and understand what cryptocurrency is, it becomes very clear that, whatever Libra may be, it is NOT a cryptocurrency. To claim that it’s a cryptocurrency is like claiming that a locked desk drawer is a bank vault – yes they both have keys and are supposed to keep things safe, but other than that…

Satoshi, the brilliant creator of Bitcoin, designed a currency that involves cryptography. If you want to be extremely loose, you could say that Libra is the same thing, because it’s also a currency that somehow involves cryptography. But that’s like saying that the thing you use to “buy” properties and hotels in the board game Monopoly is “money.” Try depositing some of it at an ATM and see how far you get.  Let’s explore the basics of what makes a cryptocurrency the way Bitcoin is a cryptocurrency.

First and foremost, there’s the concept that in Bitcoin, no one is in charge. How can you possibly make a computer system that works, does lots of computing, keeping lots of financial transactions and makes sure everyone’s account balance is correct … without anyone being in charge?? These things are hard to do when someone IS in charge! There’s quite a bit involved in making this happen, as I illustrate here, but here are some of the key points:

• Anyone who wants to can sign up to be a “miner,” who are the folks that make Bitcoin work.
• A miner has to put money into buying fast computers, running the mining software, and connecting with all the other miners to share work.
• Miners get new transactions that Bitcoin users want to perform and “make them happen.”
• This means that miners race each other to solve complex problems involving cryptography, the net result of which is a new page (block) of transactions that have been vetted, and “locked” by crypto-key.
• Every piece of work a miner does is paid for by newly-minted Bitcoin – the miners are paid with Bitcoin!
• Miners are highly incented to do the work and do it right, because they want to get lots of Bitcoin, and they want Bitcoin to continue to be viable.
• Miners come and go as they see fit – no one “approves” them, literally no one’s in charge.
• Miners can be anywhere, in any country.

Big corporations and regulators don’t like the unsupervised free-for-all of Bitcoin. They like to control things. And that’s exactly why Bitcoin was invented – to escape the control of a central authority but still have a system that works. It’s a brilliant concept, and Bitcoin’s success shows that it works.

Along comes Facebook and Libra. Facebook is ambitious. They keep trying to invent new things. They mostly fail when they build things themselves, so they buy companies instead. Facebook would LOVE to buy Bitcoin – but it’s not for sale, because no one owns it – darn! They’re forced to try to build it. But being a big corporation, they just can’t stop themselves from building their version of Bitcoin in a style that makes them comfortable – violating every single core principle of Bitcoin – the original cryptocurrency – along the way!

Here’s what Facebook is doing with Libra:

• In Bitcoin, literally no one is in charge. With Libra, Facebook is designing and building it. Facebook is in charge and owns it.
• Facebook has gone to considerable lengths to create the illusion that it’s not in charge with this fake Swiss-based consortium of prestigious companies that supposedly control things. Either way, some combination of big name-brand companies are in charge, which is pretty far from Bitcoin’s really-truly NO ONE is in charge.
• Just like Facebook owns and controls all the computers that run Facebook, Libra will own and control all the computers that run Libra in a private data center. To all the corporate computer types, this is a good thing, but it totally and completely violates a core principle of Bitcoin, leaving it open to the same kind of insider corruption that all such places are rife with. It’s also a silly idea, as explained here. Microsoft and Intel explain the issues here.
• One of the less pleasant side effects of Bitcoin’s miners and what they do with cryptography is the fact that “proof of work” takes time. It’s a cornerstone of getting all these strangers to play nice and do good things, but it takes a number of minutes to complete a transaction. To Facebook, this is unacceptable. So they’ve blithely discarded the key cryptographic cornerstone of Bitcoin, and replaced it with some light-weight encryption, so they can still say they’re a “cryptocurrency,” even though they’re not.

There’s more to be said, but that should be sufficient to make the basic point that Libra is a cryptocurrency the same way my cousin, who is sometimes allowed to sing in bars, is an opera singer. My cousin likes to think she is, and I’m nice to her. But she’s never so much as attended a performance at the Metropolitan Opera in New York, much less appeared on stage in front of an audience. Similarly, Facebook’s Libra likes to think it’s a cryptocurrency even better than the original, Bitcoin, but it swore off the core principles of Bitcoin from the start, and doesn’t deserve to be called by the same terminology.

Note: This was originally posted at Forbes.

Computer Science is Propaganda and Computer Engineering is a Distant Goal

To call what is taught in the “computer science” departments of universities a “science” is a mind-game to get everyone involved to believe that what is taught meets the normal criteria for being a “science.” It doesn’t come close. Well, you might say, some of those departments are more humbly and accurately called “computer engineering.” True. At some point in the distant future, what is taught in computer engineering might rise to the level of what is taught in, say mechanical or electrical engineering. Until that goal is in sight, it would be more accurate to call the classes something like “Fads, fashions and sects in computer software practice.”

Physics, Chemistry and Biology

I hope we can all agree that physics, chemistry and biology are sciences. It wasn’t always that way! They have only gotten to be sciences after long struggles. Physics started emerging about 400 years ago, chemistry a couple hundred years ago, and biology just in the last 150 years or so. In each case, they are studies of reality, with generally accepted statements of how that reality works, as shown by many experiments. In each case, they advance by someone making a hypothesis that can be dis-proven by experiment. If the hypothesis is supported by experiment, more tests are done by many people to refine it, and then it becomes part of the accepted science. Sometimes a new hypothesis contradicts something that’s accepted, but more often refines it – for example, the best way to understand most of Einstein’s work is that it refined Newton’s for special, highly unusual cases. In the vast majority of cases, you can safely use Newton’s laws of motion without being concerned about relativity theory.

How does Computer Science stand up to these paragons of science?

In physics, you learn the rules of matter, energy and motion. In chemistry, you learn first of all, the periodic table of elements, and then all the molecules into which they assemble themselves and how they interact. In biology, you learn about all the living things, from viruses, through plants and animals. What’s the equivalent in computer science? You can say, “oh, it’s the science of computers;” except that physics, chemistry and biology are things that exist in the world. Humans didn’t create them. Computers are 100% human creations, no less than spoons and baseballs. There is no science of spoons – there is a bit of history and a range of modern methods and styles of making them, just like computers. Until we decide that it’s OK to create an academic department of Tablecloth Science, we should be able to agree that there is no such thing as Computer “Science.”

How did it happen that loads of departments with courses about obviously bogus Computer “Science” come to be?

The innocent explanation is that, in the early days, computers were closely related to math, and were seen basically as giant programmable calculators. In fact, the word “computer” was originally the name of a person, almost always female, who “computed” the answers to math problems. While math isn’t a “science” in the normal sense of the word, it is a kind of pre-existing non-physical reality that, in ways and for reasons that have never been explained, pervades and underlies everything about us and the world we live in. It’s the grounding of all of science. Computers were often studied by the math people in academia, and the precision naturally associated with computers seems to justify the association with science.

But in the end, computers are just fancy machines that people design and build to do stuff. How would you feel about “Refrigerator Science?” Refrigerators are great, and I like what they do. The advances in Refrigerator Science since we used to call them “ice boxes” is amazing. Uhhh, maybe not.

The less innocent explanation is that everyone involved realized that no way is there such a thing as computer science, if we’re at all serious about the term “science.” But there’s no doubt that it makes everyone involved feel better about themselves, so as propaganda, “computer science” gets an A+.

OK, OK, We’ll call it Computer Engineering

Many places do call it Computer Engineering. Is that any better? Think about mechanical engineering, for example. Let’s look at my favorite example of building bridges. I’ve gone into huge detail about the differences between bridge-building in peace and war and how it applies to software.

Let’s step back and compare the normal peace-time bridge engineering project with the normal computer software one. At the outset, they seem pretty similar. They start with requirements, then an overall design, then build, testing and inspection and finally production. In fact, many engineering-type methods are used in software, including project management.

For bridges, it works out pretty well. Bridges get built and work, 24 by 7, year after year, with routine maintenance. There are exceptions, but they're rare. Software? Not so much. Just to hit some highlights, the problems include:

• Bridge-building project management methods are sound and generally produce reasonable results. Software project management methods don’t work
• Bridges are generally built in-place, so installation is an integral part of the build process. Installing standard software is a nightmare
• Once built, bridges work. Software QA is broken
• Bridges just keep working. Software is full of horrible bugs
• No one stealthily sneaks onto bridges and steals the cars that are on it. Software is full of horrible security holes
• Bridge-building is driven by solid, proven engineering practice, backed by scientific principles. Software is driven by fashion instead of sound practice

Given all this, I guess we can still call it Computer Engineering. But to be fair, we should have it on probation, and call it “Computer Sadly-Deficient Engineering” until it rises to the level of mediocrity -- which it may, with some luck, someday achieve.

Conclusion

Computer Science is not a "science." Not close. To call it science is propaganda, fraud, ignorance, whatever. Computer Engineering is another matter. Computer Engineering, both as taught and as practiced, is horrible. The methods, largely lifted from other disciplines, just don't produce good results most of the time. There isn't even a movement that recognizes this and agitates to fix it!

Happily, the solutions -- good computer engineering practices -- exist and have been proven in practice, many times over by different groups in different times and places. I have discussed what I know of these methods extensively in this blog and in books. Top programmers discover large parts of them on their own, and use them to achieve stellar results -- outside the purview of corporate types, regulators and bureaucrats. For entrepreneurs versed in these methods, it's a serious competitive advantage, while the crippled, lumbering giants of software shuffle along.

Microsoft And Intel Detail The Deep-Seated Problems With Blockchain

Both Microsoft and Intel are big supporters of blockchain. They think it's going to be "bigger than the internet," contributing trillions of dollars to the economy before long. At the same time, they spell out the overwhelming obstacles blockchain must overcome to reach this pinnacle of achievement. Guess what, surprise surprise, the special version of blockchain created by Intel and Microsoft is indispensable to solving the problems and achieving success!

You can see their deep thinking here and here. Before diving in, I'd like to point out that the custom, private blockchain they advocate is a contradiction in terms, as I illustrate here -- even if they implement what they claim perfectly, it will still be a joke.

Here are a few of the little obstacles that blockchain has to overcome before becoming acceptable for enterprise use, according to Microsoft and/or Intel:

• Performance: Normal blockchain performance is a few transactions per second. "Reed said the trusted execution environment of Intel SGX enables Coco to deliver a novel consensus mechanism that can deliver up to 1600 transactions per second..."
• Confidentiality:Normally, everything on a blockchain is public information.  "Microsoft uses Intel Software Guard Extensions (Intel SGX) to protect the Coco Framework. Reed said the trusted execution environment of Intel SGX ...  helps Coco transactions remain confidential among blockchain participants."
• Governance: With a normal public blockchain, no one is in charge. This doesn't come close to meeting enterprise requirements. Microsoft's private blockchain enables classic management, access controls and all the rest.
• Processing power:Intel says "Public cryptocurrency blockchains require huge amounts of energy to verify transactions through node consensus. Analysts have estimated a single bitcoin transaction can require as much energy as the average American home uses in a week."

The other big vendors, like IBM with its team of 1,500 people working on its Blockchain effort, have similar stories about what's wrong with Blockchain and why you should use theirs. When you add it all up, it does make you wonder about this revolutionary new technology, and exactly why important new initiatives should depend on this brand-new, largely untested code that obviously was not built with practical, enterprise use in mind.

This was originally posted at Forbes.

Software Professionals Would Rather Be Fashionable Than Achieve 10X Productivity Gains

I’ve been involved in a large number of pioneering software efforts, and I’ve lived through many tech fashion seasons. It’s very clear that the vast majority of experienced software professionals and managers would rather follow current technology fashion than actually making things better. Because of the pathetic state of software knowledge and analytics, not only do they get away with throwing away truly massive gains for their organization, no one in power has a clue what they’ve done! This happens over and over and over and over again. A few people may know there’s a vastly better approach available – one that’s proven and risk-free – but no one hears them. The siren song of the hot new thing wins most every time.

I have been involved in many such farces in one way or another. Here’s the story of one of them from many years ago. See this and this for general background of the technologies and patterns. It's important to remember: I'm not saying that Sallie Mae was a disaster when everyone else was fine. Sallie Mae was a normal, mainstream project at the time, and therefore an excellent example of the overall insane pattern.

The Workflow project at Sallie Mae

In the early 1990’s, document imaging and workflow technology were hot; they were something people talked about the way they talk about AI/ML and innovation today.

Sallie Mae, then as now, was the government-backed student loan organization. In the early 1990’s they decided to upgrade their computer-based processes to something that was modern in order to make things more efficient. At the time they had over 10 million paper documents per year arriving at their processing centers. They had six of them at the time, each employing thousands of people. Even a 10% productivity improvement would have been big at that scale! Sallie Mae managers had read about the new document imaging and workflow technology that was rapidly growing at the time, and set up a task force of professionals to study it, create an RFP and get it implemented.

The task force knew they weren’t experts at this technology, so they sought out people to help them. Among other things, they attended the AIIM industry association’s annual convention, at which I happened to be a featured speaker due, among things, to my book they had just published. Here is the forward to the book, written by AIIM:

I had also led the technology at one of the industry’s tech vendors, had personally written their workflow software, been involved with major buyers including Amex, etc. So they approached me and asked me to help them out. I agreed. They also contracted with a small consulting firm to join in.

I dove in to analyze the situation. Sallie Mae had loads of IT people who were concerned about the new technology. The existing technology was basically IBM mainframes with many thousands of 3270 terminals on people’s desks. Implementing the new technology would involve buying expensive workstations with large image displays for each worker. The IT people worried about the LAN that would be needed to connect the workers to the image storage system, so I created a spreadsheet to enable them to calculate the transmission speeds that would be needed. While the other consultants were busily drawing up diagrams of paper movement so that workflow could be implemented, I dove into the details of what actually happened at a couple representative workstations.

I observed some people at one of the processing centers doing the work of processing the form. I timed the work and carefully observed the screens. I read the manual that was used to train the workers for the job. I arrived at a pretty radical (for the time) hypothesis: the vast majority of the work was logic-enhanced data entry. If the humans did no thinking but just entered the data, they could work 10X faster. The logic spelled out in the training manual could all be programmed on the data as entered, with a tiny number of exceptions kicked out for human handling.

If this could really be achieved, it had consequences for Sallie Mae. Instead of a massive per-employee technology investment with the work remaining roughly unchanged, a small fraction of the employees could do the same work in a completely new way. I figured I better make sure I was right before I started rattling cages and maybe embarrassing myself.

Logic-enhanced heads-down data entry at Sallie Mae

I knew that everyone assumed that implementing workflow was the goal, with productivity improvements assumed to be 30% after massive investment and disruption. This was broadly accepted, and no one involved was interested in challenging it or testing it. If there was any chance of them taking a new approach, I knew I better have the numbers.

The main things I discovered were:

• Most people entered 1,000 to 1,500 KPH (keystrokes per hour).
• Roughly 20% of those keystrokes were overhead, i.e., navigating to the next field on the mainframe screen, often navigating to a whole new screen. The mainframe screens followed how the data was organized inside the computer, not at all how it appeared on the paper forms.
• The workers weren't bad typists -- they were obviously doing some thinking between fields, essentially applying the special rules they were taught in training to guide them what to enter where.

I refreshed my knowledge of practical image-enabled heads-down data entry (HDDE), See this for an explanation and details. I laid out some plans.

• Image-enabled HDDE could reasonably achieve 12,000 KPH, as it was doing at multiple high-volume operations. Moreover, the navigation overhead could be brought down to a couple percent.
• The training manual was amazingly close to a natural language description of a program. If this field on the form has that, then enter X in Y mainframe field, and so on.
• The current method interwove the entry of the form with the rules. In the new system, all the data would be entered exactly as it was on the paper form, and then a new set of software would apply the rules and enter the data into the mainframe system. When the rules changed, instead of everyone taking a training class, the software would just need to be changed.
• Some forms in the existing method were sent to a customer service person for exception handling. Much of that would still happen, but the numbers were low.
• I knew that Image character recognition could handle some fraction of the data entry work. But it was a new technology, and I figured a 10X productivity improvement and avoiding a massive technology investment for workflow was more than enough to make the case.

I wrote up my results. Since “re-engineering” was a popular concept at the time, and since “zero-based budgeting” was talked about a fair amount, I wrote a paper and called my approach “zero-based re-engineering” to explain the approach and link it to fashionable things.

My involvement with Sallie Mae ended, quickly and quietly. No one objected to my proposal. No one questioned my analysis or numbers. There were some vague comments thanking me for my creative, detailed work. But I was invited to no more meetings, got no more assignments, and that was that.

What happened was that the workflow-at-Sallie-Mae train had long since left the station. My work was nothing but a distraction, with the potential of becoming an embarrassment if things went badly. I had demonstrated that I wasn’t a “team player,” and in most organizations, there’s little worse than that.

Conclusion

Like most history, the workflow project at Sallie Mae in the 1990’s doesn’t matter at all. It’s history! Its value is that it’s a typical example of a recurring pattern in software implementation and evolution. Here is a more recent example. Sallie Mae was not unusual at the time -- they were absolutely typical! Yes, they had an "insider" recommend a simple, radically better way of doing things and rejected it -- but that was the overwhelming mainstream pattern. By recognizing the pattern for what it is, the few who care can learn from it, recognize the pattern when it’s taking place, and maybe even do things better.

Most organizations will almost always follow the tech fashion rather than achieve dramatic gains using proven but unfashionable technologies. Many entrepreneurs are part of the problem, since they need to sell into buyers who value fashion over real, objective, provable value. The best entrepreneurs find a way to fit in with buyers’ hunger for fashion, while also delivering real value. Sometimes they cloak the value in super-attractive, flimsy outerwear. That's OK. They’re entrepreneurs – they find a way!

Surprising Bugs at Amazon Show the Fragile Foundation of AI

I promise I didn't plan it this way, but when I looked on Amazon for a product to help me deal with an infestation of bugs, I encountered a major ... yes, bug. I described the bug in detail here, and at the time thought it might be isolated -- after all, in my all-too-extensive use of Amazon, I had never before encountered such a bug.

Not long after, as the issue continued to "bug" me, I went onto Amazon again, and found more interesting bugs. This could be just some scamming or corruption. I'm taking the trouble to describe what I discovered because it's exactly the kind of data that modern systems are based on, and by playing with the data, people with bad intentions can cause the systems that depend on that data to come out with the results the bad people want. It's a way of causing destruction or stealing that's indirect, effective, and likely to grow as evil-doers catch onto the technique.

The new bug

I went into Amazon looking for a pest repeller again.

This one looked pretty close to the first product I saw -- same vendor, same product appearance, nearly the same language. But the price was different, and the number of reviews, while huge by pest repeller standards, was about half that of the original listing.

I went on, and the questions & answers looked like they were real, and for the product itself. Maybe Amazon has fixed things!

Let's scroll down more and see.

Whoops!

Still an astounding number of positive reviews, but look at the categories and things mentioned. Children's books again, this time with religion and a magic tree.

But the pictures are more creative. Look at the second one above -- clearly snitched from some product far, far away from children's books! One of the leading reviews makes absolutely clear that some serious messing with Amazon's data has taken place:

The only way this is about pest repellers is if you consider the devil to be a pest. No, I don't think so.

Implications

This is a bug. And some serious messing with data inside Amazon's systems. And a GAPING hole in Amazon's security apparatus, showing us yet again that effective cybersecurity is NOT about getting certifications and passing tests devised by government lawyers and bureaucrats.

The direct implications of this are pretty small to the average person. But if best-of-breed Amazon can be hacked this way, what about the labs where data is collected and stored concerning our health? What if, instead of just stealing the data to make a couple bucks selling it to scammers, someone decides to mess with the data to achieve an outcome, like happened here? In cases like this, the data is manipulated to get buyers to see a product based on the large number of positive reviews. The same thing could be done for a drug, a therapy or a health center! Going deeper, the data could be manipulated to impact "knowledge" of the kind that AI/ML discovers and implements. Even more deviously, it could have a devastating impact on "personalized medicine," the most data-driven of all, with a screwed-with set of data that says that a certain innocent-sounding treatment would be just right for someone like you -- except, for someone like you, it's precisely personalized disaster.

In order for super-charged AI/ML engines to actually do good things, we have to make absolutely sure that their data foundations and underpinning are sound, secure, and un-manipulated. Sadly, this is not a "what-if" issue. For details, see this. The assurances of the usual serious-sounding, credentialed experts with stentorian voices and/or soothing manners are not enough. Not even close.

Surprising Bug at Amazon About a Bug Repeller

I have found ample reason to mock the golden-glowed tech reputations of most of the tech giants, in addition to the supposed tech prowess of organizations such as the NSA. There is good reason to believe that old-style libraries are more secure.

I recently stumbled upon a rather glaring bug or result of hacking at Amazon -- and found that Amazon provides no way I could find to report the problem.

The problem was glaring and amusing -- a whole set of over 3,000 reviews of a book attached to a pest repelling product. As I'll describe in a future post, this isn't just a bug concerning bug repellers -- it's the tip of an iceberg about the foundations of AI/ML.

Here's how I found the bug bug.

Stumbling on the bug at Amazon

I was looking for a product to scare away some pests that have found their way into my house. I gave a close look at the following product:

I immediately noticed and was impressed by the large number of reviews, and how favorable they trended. What a popular product! I've got to look at this one. I scrolled down:

Things are still looking good. More details:

Uh-oh! Right after we see that it has 3,051 reviews with an average rating of 4.8, which is super-good, we see that this pest repeller is #2,695 ... in Books!! Something is seriously wrong. Scrolling down, we see some really weird answers to questions:

A couple of the people answering clearly tried to tell Amazon there was a problem.

This next one blew me away -- over 3,000 reviews for a pest repeller??!!

Now we know for sure something's seriously wrong -- a pest repeller is not Children's Literature!

Here is the nail in the coffin, the pictures of the product and a list of the phrases that frequently appear:

 

And here's one of the glowing reviews. It brought tears to the reviewer's eyes, and not because of the strong odor generated by the pest repeller:

 

What's going on, Amazon? I don't usually get to mock you. It's usually the other tech giants that receive my sarcasm, places like Google, Facebook, Twitter, Apple and Microsoft. Amazon has done a reasonable job maintaining quality and growing into new areas. This is such a peculiar bug -- it's as though some disgruntled employee were messing with things to show his displeasure with the world.

As we'll see in a following post, this is not an isolated bug, and it has implications that go far beyond Amazon -- implications concerning the glorious future promised by all the AI/ML enthusiasts.

Laser Disks and Workflow Illustrate the Insane, Fashion-Driven Nature of Software Evolution

Computers are so exact and numbers-based that we tend to think of the people in charge of them as white-jacketed scientists with esoteric knowledge driving towards calculated optimal results, much the same way we imagine that the scientists and engineers calculated the path of the Apollo capsule to the Moon. If the Apollo program were run the way most computer projects are, it would have been a miracle if the capsule made it off the launch pad, much less traced an incredibly exact path from Cape Canaveral to the chosen landing spot on the Moon, a journey of over a quarter million miles.

The reality is that the application of computers and software to automation is painfully slow, usually failing to achieve optimal results by large margins, and often lagging behind what is achievable by decades. The reasons appear to be a semi-random mixture of commercial interests, technology fashion trends, ignorant and risk-averse managers, and technical specialists rooted in the past and wearing blinders. That’s all!

Here’s the story of a typical example.

Microfilm, document imaging and laser disks

Long before computers, microfilm was the preferred medium of librarians and archivists to preserve paper documents for future use. Microfilm was more compact than paper, and lasted much longer. What happened when computers came along is a curious and educational story, a prime example of how computer use and software evolve.

Disclosure: I lived and worked as a participant in this history during the late 1980’s and early 1990’s. The story I will tell is not systematic or comprehensive, but like a traveler’s tale of what he did and saw.

A side show in the long evolution of computer storage is the laser disk. A laser disk has data recorded onto it either as a whole, during manufacturing, or incrementally, as part of a computer system. The laser disk had quite a run in the consumer market in the form of CD’s, and then DVD’s. They were an improvement on existing methods for distributing digital content of all kinds.

In the computer industry, they took the form of WORM (write-once-read-many) drives, and were used to record archival data that should not be updated, but could be read as often as needed. The technology, as often happens, went searching for problems it could solve. In an effort to reduce human labor more than microfilm could, jukeboxes were invented. Each jukebox had a disk reader and many shelves for disks, with a mechanism to load a selected disk into the reader. FileNet, now part of IBM, built one of the first of these systems in the mid-1980’s.

At around the same time that WORM drives became practical, paper document scanners were evolved from systems to capture images of the paper onto microfilm. In each case, light was shown onto paper and focused onto a target; a scanner replaced the film with an optical sensing device.

How could we get people to buy WORM drives and jukeboxes, the creative marketing people wondered? The departments that put documents onto microfilm weren’t going for it – the new systems were much more expensive, microfilm wasn’t accessed often enough to make it worth connecting it to computers, and the records retention people were understandably skeptical that any computer disk would be readable 100 years from now, as microfilm will be.

Adding workflow to document imaging

I have no idea who made the audacious leap, but some creative person/group came up with the idea of doing computer document scanning at the start of the process, when paper arrived at a location for processing, instead of microfilmed at the end for archival purposes. But WHY??? The creative types scratched their heads until they were losing hair rapidly. Why would anyone make responding to loan requests or whatever even longer than it is today by introducing the step of scanning?

GENIUS TIME: LET’S INVENT “WORKFLOW”!!!

Everyone has an image for what “workflow” is. For documents, it’s the path of processing steps from arrival to filing. In larger organizations, there are often multiple departments involved in processing a document, so there are in-boxes and out-boxes and clerks transporting documents from one to the other.

Rubbing their hands and cackling, the scammers worked out their strategy: We’ll scan the documents and convert them to images so that we can move images from place to place electronically instead of piles of paper! We’ll call it “document imaging workflow.” It will deliver massive benefits to the whole organization, to everyone who touches the paper, not just to the tiny group who files and archives at the end – and storing the scanned documents onto WORM drives will do their job too! Hooray! We will leap-frog the old-fashioned, paper-flooded back office into the modern electronic age. What’s not to like?

It was audacious and brilliant. It was a scam that would wither if confronted with just a little common sense or cost accounting. Which, true to the pattern of such fashion-driven trends, it never had to confront!

Implementing workflow

A typical target environment for implementing workflow was large offices with large numbers of documents, often forms of some kind, arriving every day. The forms would go from the mail room to the relevant people’s desks for processing. The person handling the form would often have an IBM 3270 terminal for interacting with a mainframe computer. When the person was done with the form, they would place it in one or more outboxes for further processing at other desks, or for filing. Sometimes documents would go into file cabinets for a period of time, but all would end up being archived, and sometimes microfilmed for permanent storage.

Implementing workflow first of all required scanning the documents as a first step, and storing them immediately on normal computer disks, and ultimately on WORM drives, thought to be the equivalent of microfilm. Once scanned, the documents would be managed by workflow software, which would route them to the appropriate desk in the appropriate department for processing. Every desk needed to have the old computer terminal replaced or augmented with a large, high-resolution image terminal, capable of handling at least the display of the document, and sometimes also the computer screen. The old, slow connections between terminals and mainframe needed to be replaced with a fast local area network, and there needed to be substantial servers for handling the local image handling and workflow routing.

Of course, lots of analysis and programming was involved in addition to the equipment. Workflows had to be analyzed and programmed, and all the management, reporting and exception handling taken care of.

The benefits of workflow

When a movement like document imaging workflow has a head of steam up, no one seems to ask whether it’s a good idea, and if so, how good of a good idea it is – quantitatively. When I was involved, everyone threw around there would be at least a 30% productivity improvement due to workflow, and that would make all the expense and disruption worth it. I never encountered a single study that measured this.

Think about it for a minute. Would looking at an image of a document make you work faster than you would looking at the original paper? Would picking the next paper from the in-box be slower than getting the system to show you the next image? What about the labor of the clerks moving stacks of paper from one person’s outbox to the next one’s inbox? It would certainly be saved, but chances are a single clerk could handle the paper movement for many dozens of people. And remember, there’s the scanning and indexing that’s been added and the massive computer and software infrastructure that has to be justified.

It’s obvious why no one EVER did a cost-benefit analysis of workflow – the back-of-the-envelope version easily shows that it’s a couple zeros away from making sense.

I remember a couple of out-of-it, tech-fashion-disaster know-nothings mildly thinking out loud about how workflow could possible be worth it financially. The immediate response from workflow fashion leaders was upping the ante – don’t you know, the wonderful workflow tool from (for example) FileNet lets you program all sorts of efficiencies for each stage of work; it’s something we really need to dive into once we get the basic thing installed. End of subject!

So who fell for this stuff? Just a who’s-who list of major organizations. Each one that fell for it increased the pressure for the rest to go for it. None of them did the numbers – they just knew that they couldn’t fall too far behind their peers.

Conclusion

It’s hard to think of a field that’s more precise and numbers-oriented than computers. Who has a clue what actually goes on inside those darn things? And the software? There are millions of lines of code; if a single character in any of those lines is wrong, the whole thing can break The impression nearly everyone has, understandably, is of a field that is impossibly precise and unforgiving of error. When the software experts in an organization say that some new technology should be adopted, sensible people just agree, particularly when there’s lots of noise and other major organizations are doing it. It can be even more gratifying to get known as pioneering, as one of the first organizations to jump on an emerging tech trend!

Somehow, the smoke and noise from the software battlefield is so intense that the reality is rarely seen or understood. The reality, as I’ve illustrated here, should result in everyone involved being sent to the blackboard by a stern teacher, and being made to write, over and over: “I pledge to try harder to rise above the level of rank stupidity next time.”

Simple Data Entry Technology Illustrates How In-Old-Vation in Software Evolution Works

Since when is “data entry” (entering data into a computer) a pivotal, innovative technology? When the difference between doing it the normal way and doing it with advanced technologies is a …ten-to-one productivity difference … that’s when.

I’ve described how the Operations Research algorithm of Linear Programming is fifty years into an agonizingly slow roll-out through different applications, from scheduling oil refineries in the 1960’s to scheduling retail sales in the 1990’s, and now scheduling medical infusion centers and operating rooms in the late 2010’s. In each case, laborious and error-prone human scheduling was replaced by the algorithm, with improvements ranging from no less than 10% to over 50%. This is major! Why did such an innovation wait for decades to be applied, and for many applications, is still waiting!!?? This is the mystery of how what’s called “innovation” works in reality, and why it should be called “in-old-vation” instead.

You may think that part of the cause is that LP is an exotic algorithm – even though it’s a standard part of the Operations Research engineering curriculum, most so-called normal people haven’t heard about it. While it appears that even the hosts of people who wax eloquent about AI and ML are clueless about LP, it’s not exactly a secret. So let’s see if obscurity is the reason why LP remains a “future innovation” in many potential applications, by examining the super-plain, ordinary, completely-understandable-by-normal-people case of data entry.

Data Entry

Just as process optimization is done by hand or stupid methods for decades until some genius comes up with the brilliant idea of applying tried-and-tested LP to the problem and dramatically improves it, so is Data Entry widely performed by primitive methods until some “innovator” comes along and applies “Heads-Down Data Entry” (HDDE) methods to the process – and typically gets improvements of 3 to 10X! The only difference is that while LP is taught in Engineering departments and studied by math nerds, Heads-Down-Data-Entry is “just” a collection of common-sense techniques that require no math and no professors to understand or implement. It’s so “common” that it doesn’t even have a generally accepted name – though it’s been implemented in many places and been thoroughly proven in practice. It’s far too humble to merit an academic department – and yet, when applied, has delivered truly massive gains, far higher proportionately than exotic Linear Programming has!

The methods of HDDE were first implemented in places that had huge volumes of repetitive data to be entered into computers. Banks were early users for check entry, and so were the credit card companies who, at the start, had huge volumes of paper charge slips to process. Simple ideas like minimizing keystrokes and eye movement were implemented, and then taking advantage of the eye-to-fingers pipeline, when people noticed that showing clerks the next item to be entered before the current one was complete led to a big jump in speed. Other methods like double-blind techniques were invented, so that entry clerks just entered – whether their work was original or used to check someone else’s work was entirely handled by the Data Entry system.

As soon as scanning and image display became practical, HDDE adopted them. That led to another jump in productivity, enabling large, complex forms to be broken up into pieces, so that a clerk would see the image on a screen of the same piece of data from a whole set of forms instead of entering a whole form from start to finish. No HDDE shop would even consider having the entry clerk think about anything on the form, stuff like “if this field is missing, do this instead of that,” because it would just slow them down.

Finally, there’s ICR (Image Character Recognition), which is having the computer “read” the image instead of a human. This technology has existed for many decades. Once you’ve got HDDE in place, phasing in ICR is a natural, so that the proportion of entry done by humans gradually decreases as the effectiveness of ICR increases.

Remember, applying LP to scheduling might result in a 30% improvement, which in most cases, is major to the point of being revolutionary. What about HDDE? Entering data from a paper form into a computer using primitive methods might get between 1,000 and 1500 KPH (keystrokes per hour). There are lots of stages of improvement, including things I’ve mentioned like eliminating the thinking and breaking up the form, but levels of 10,000 to 15,000 KPH in a professional environment are widely achieved – with superior quality. That’s a minimum of 5X! Typically much more. Of course, as you incorporate ICR into the process, it gets even better, gradually reducing the human factor, so that most fields are entered with no human involvement. At this point, the technology is probably best called ICR+HDDE, though there is no generally accepted term.

Given all this, it would be insane to handle computer data entry by anything other than HDDE methods, right? Welcome to the software industry, where insanity of this kind is the accepted state of affairs. And where almost no one practices the most simple and basic of computer fundamentals, such as counting.

How HDDE gets implemented

I described how Linear Programming went from problem domain to domain, each time acting like an innovation, as indeed it was in that area of application. Once it gets established, it tends to stay. The case of HDDE is different, I think because it’s not a recognized “thing” in the halls of academia, or among the poo-bahs of big business. It’s the kind of thing that no self-respecting Professor of Computer Science would stoop to consider, assuming he ever encountered such a low-status thing – you know, the kind of thing that “merely” makes common sense and, well, works.

HDDE has appeared in competitive, high-volume service businesses, where it has a major role to play in delivering results for the customers of the business. There have been software products that directly support HDDE, so that all you have to do is buy and implement them. It’s neither obscure nor hidden. But it’s never been talked about at conferences as the “coming thing.”

Case Study: HDDE rejected

In the early 1990’s, when document imaging and workflow technology were hot and something people talked about the way they talk about AI/ML and innovation today, the government-backed student loan organization, Sallie Mae, decided to apply the technology to improve the operations at the handful of processing centers they had at the time, employing many thousands of people and processing millions of documents a year. The popular thing to do at the time was to scan documents on receipt, and then send them to the same places the paper was sent, so that workers could process the images of the documents displayed on new big screens instead of paper. The job was basically to type the data into the right places of the software application they used.

Everyone at the time said that converting the documents was important ONLY because it enabled wonderful workflow, the elimination of inboxes and outboxes for paper. And the bits of other stuff you could do, like having a group of people taking from a common inbox instead of each having their own. The common “wisdom” was that you could gain 30% productivity improvement by implementing this marvelous new technology.

I got involved, since I was a recognized expert on document imaging technology at the time, and had personally coded one of the early workflow systems. I figured out and showed in detail that by canning the workflow and implementing HDDE techniques, they could gain a minimum of 5X productivity improvement. No one disputed my thoughts or detailed plan. They just ignored it, and proceeded to implement the standard stuff. I strongly suspect that after considerable time and expense, there were walk-throughs of the Sallie Mae sites showing visitors the big screens and absence of paper – what a big success the project was!

Case Study: ICR-HDDE applied with success

There are two current cases I know of where ICR-HDDE is being applied and winning. Each are classic, narrow service businesses where converting forms to data is the key value of the business, and where the companies buying the service just want fast, accurate data delivery, they don’t care how it’s done. Disclosure: each of these is an Oak HC/FT investment.

At Groundspeed, insurance forms and reports are captured and the relevant data is extracted from them by the most effective relevant means, often involving forms of ICR-HDDE. There is lots of forms recognition from documents and images that are often computer output, with the relevant data appearing at varying places on a page. Nonetheless, Groundspeed is able to deliver the data stream the customer needs, quickly and accurately, The results are so powerful that new levels of analytics are enabled by the newly available stream of structured data.

At Ocrolus, financial documents of all kinds including bank statements and pay stubs are converted to data in a standard format to enable fast and effective operations like giving loans for business and personal use, along with a growing list of other operations that also need good data. An effective combination of ICR-HDDE techniques are applied to get results for companies that need accurate data to make fast decisions.

Conclusion

HDDE is a collection of methods that have been proven in practice for many decades. The technology that is behind it continues to deepen and reduce human effort even more, with the addition of ICR. But it remains a niche technology, ignored by the numerous places that could benefit from it, even more than LP.

The big difference between LP and HDDE is that LP is a formal piece of magic that’s in academia. HDDE is nowhere. In fact, it’s really just an example of classic industrial engineering applied to computer software and the people who use it. Which makes it all the more mysterious that it's largely ignored.

In-old-vation is real. Most “innovations” are minor variations on things long-since proven and demonstrated in practice, but are unimplemented in the many situations that would benefit from them until some mysterious combination of circumstances arises to let them explode into practical reality.

Facebook's Libra Cryptocurrency and the P2P Apps Venmo and CashApp

Facebook is working hard on building a brand-new cryptocurrency system called Libra, sort of like Bitcoin and Ethereum, except it will be much better, at least according to Facebook.

With all the talk about Libra, cryptocurrency, regulation and the rest, no one seems to wonder about what existing solutions normal people will be using to solve the problems for which Libra is suited. This isn’t strange at all actually – in all you’ve read about Facebook’s Libra, how much have you read about the pressing problems it will solve, the unmet needs it will address – right? Mostly what you read about is how Libra will solve all sorts of problems that today’s crypto-currency systems have, how many partners they have and how wonderful it will be.

Libra will be an infrastructure “out there” somewhere, with lots of important people and organizations making sure it’s wonderful. But in practical terms, most people will use it via an e-wallet, an app that they install on their smart phones. That’s where a name that hasn’t appeared a great deal pops up: Calibra. Calibra is a newly-formed Facebook subsidiary that will be the e-wallet for Libra. It will “integrate with” Facebook’s WhatApp and Messenger, giving it incredible consumer reach.

You can read all about what it will do, but it’s basically an e-wallet for holding e-cash and providing basic functions like sending and receiving e-cash to and from another e-wallet. Except for the little “detail” that instead of sending real money, you’re sending the Libra cryptocurrency, and will have to go to an additional step to move dollars in a bank account to and from your Calibra wallet, converting to or from dollars along the way.

Putting aside the fancy new terms of cryptocurrency and the rest, does using a phone to make person-to-person payments remind you of anything? How about Venmo, the P2P e-wallet used by over 51 million people, which is now part of PayPal? How about Cash App, the rapidly growing P2P e-wallet installed in about 60 million phones?

These are proven consumer applications which have already gone through numerous upgrades and feature additions, used at least weekly by tens of millions of people.

Facebook has incredible reach, and billions of cash in the bank gotten by selling your private information to advertisers. They will certainly make a lot of noise. How does Facebook's proposed system compare to Venmo and Cash App?

• Venmo and Cash App just use dollars. Simple. Facebook will use the newly invented Libra, which needs to “work,” something Facebook isn’t good at doing.
• If you split a bill and need to send $7.30 to your friend, you just do it with Venmo and Cash App. With Facebook, you’ll have to convert it to Libra, send that, and have it be converted back. Hopefully the exchange rate won’t move too much.
• Venmo and Cash App support free P2P payments. The Calibra website claims it will be “low cost,” but they have yet to say what the cost will be; after all, there is a HUGE cryptocurrency infrastructure to support, none of which is needed by the existing cash apps.
• It’s easy to imagine Facebook will find a way to sell the information about your transactions to the highest bidder, or somehow find a way to "monetize" what you do with your money. It’s what they do!
• Libra and Calibra will work for international payments! With exchanges from local to Libra to foreign currency, two exchanges instead of one. That’s certainly something Cash App and Venmo don’t do today, and will appeal to some fraction of a hundredths of a percent of the market. Except for the proven massive cryptocurrency uses for money laundering and international crime, who will have yet another channel to support their illicit activities.

Facebook's Libra is getting all the attention any giant corporation could want, including some attention I suspect they'd rather not have, from regulators. But in the end, will they be able to make this massive software effort work? Will it do anything consumers want better than existing apps like Venmo and Cash App that are in widespread use? There is good reason to be skeptical.

This was originally posted at Forbes.

Understanding Software Evolution Helps you Predict the Future of Software

There are patterns in software evolution. Because almost no one studies software history, and even fewer study software evolution, these patterns are almost never discussed.

The patterns are amazing. In some cases, you can be pretty sure that a trend that “everyone” says is going to be the future will fizzle out. In other cases, you can predict with a high degree of certainly that software of a certain definite kind and description will be built – even though it hasn’t been built yet – it hasn’t been built anywhere by anyone, but nonetheless you know it will be built.

The reason I started to study software evolution is that it’s just plain interesting. The patterns that emerge from it are striking and educational. But most important, knowing software evolution can help you predict the future!!

Knowing software evolution can help you predict the future for the simple reason that software does not evolve the way pretty much everyone thinks it does. It is often the case that, when a new software system appears on the market that has a big impact, people who know the patterns were sitting around waiting for it to happen! They knew it was coming!

I remember living in New Haven CT in the early 1970’s. I was a vegetarian. A new restaurant had just opened, a kind I’d never been to, an Indian restaurant. I went, and it changed my culinary life. Indian food – made by vegetarians for vegetarians, healthy and wonderful flavors!

The new restaurant was definitely new in Connecticut, a true innovation. The owners had to go through quite a bit to open it up, get supplies, etc. But was it otherwise innovative? Did the owners invent a single one of the dishes or recipes? No, of course not. Just like with a piece of software, they had to create it from scratch in a new setting, making appropriate adaptations, and so to the locals, it was new. But in reality, it was a newly adapted instance of a proven winner. Software works the same way – a piece of software that is a big new winner, the first of its kind, has most often been proven elsewhere.

The first Indian restaurant in New Haven could have opened years earlier or later than it did, with different people doing the work. There was no way to predict when the Indian restaurant would have opened, or who would have done it. But isn’t it obvious that, in the context of ethnic restaurants enjoying increasing success and the growing Indian population that someone would have opened an Indian restaurant there? Shockingly to most people, software works exactly the same way! In a surprising number of cases, you can’t predict WHO or WHEN, but you can sure as heck predict WHAT will be built and even WHERE (i.e., in what business domain).

Here is a specific example of software that resembles in important ways the example of Indian restaurants. Instead of Indian food, the star is a specific optimization method within the field of Operations Research (OR) called Linear Programming (LP). While most people know there are loads of Indian people in India and elsewhere and they have a unique cuisine, understanding this is helped by the fact that we can picture India on a map of the globe and we all eat some kind of food. How many people have even heard of OR or LP, or have seen and understood software of any kind? To be clear, what I mean by "having seen software" is seeing the source code, not the results of it on a screen. This LP software sits around, widely used in some domains, and completely unknown in others, just like there were no Indian restaurants in New Haven before a certain date.

Knowing this helps you predict the future of software, because in a shocking number of cases, it doesn't need to be invented, but "merely" imported into an area of use where it is not currently in use. That is a big reason to be interested in understanding software evolution.