The Black Liszt

Trusting Science: the Whole Milk Disaster

I trust science. The gradual emergence of science has led to a revolution in human existence that has happened so quickly and with such impact that it is hard to gain perspective on it.

Trusting science is not the same as trusting the pronouncements of people who are designated scientific experts. Establishing the truth of a scientific theory is an entirely different process than the social dynamics of rising to a position of leadership in a group of any kind. Official experts, whether government, corporate or academic, nearly always defend the current version of received truth against challenge of all kinds; most of those challenges are stupidity and ignorance. My go-to expert on this subject, as so many others, is Dilbert:

Sadly, those same establishment experts tend to be the strongest opponents of genuine innovation and scientific advances of all kinds. As I explain here, with examples from Feynman and the history of flight, one of the core elements of successful innovation is ignoring the official experts.

My skepticism is well proven in the case of so-called Computer Science, which doesn't even rise to the level of "useful computer practices" much less science. As I have shown extensively, Computer Science and Engineering is largely a collection of elaborate faith-based assertions without empirical foundation. And computers are all numbers and math! If it's so pathetic in such an objective field, imagine how bad it can get when complex biological systems are involved.

This brings us to the subject of saturated fat (solid fat of the kind that's in meat), whole milk and human nutrition. This ongoing scandal -- for which no one has been imprisoned, sued or even demoted -- in spite of its leading to widespread obesity and other health-damaging conditions -- is still rolling along. The hard science concerning the supposed connection between saturated fat, cholesterol and heart disease is in. The results are clear. It is positively healthy for people to eat saturated fat. Period. The scandal is that the "expert" people and organizations that have declared saturated fat and cholesterol to be dangerously unhealthy for many decades refuse to admit their errors and continue to waffle on the subject.

This is relevant to computer science because of the stark differences between the two fields. Software is esoteric and invisible to nearly everyone, while the results of eating are tangible to everyone, and the statistics about the effects are visible and measurable. The common factor is ... people. In both cases there is a wide consensus of expert opinion about the right way to build and main software, and the right way to eat and live in order to be healthy. Experts! From blood-letting to flying machines, they lead the way!

Usually the Science-challengers are wrong

It has taken me a great deal of time to dig in to this scandal, in part because there are so many cases of "the experts are all wrong -- me and my fringe group have the truth." I wanted to make absolutely sure "it's good to eat saturated fat" wasn't another of these. After all, the simple notion that eating fat makes you fat makes common sense!

An example of a harmfully bogus claim is the anti-vax movement, which has been supported by a number of famous people. The idea is that vaccinations in general and childhood vaccinations in particular have horrible consequences -- for example, causing autism in children. A study led by Dr. Andrew Wakefield was published in the British journal Lancet that claimed to prove the association. After years of growing fear and resistance to childhood MMR vaccines, the study was shown to by fatally flawed and corrupt, funded by trial attorneys who wanted to sue drug makers. Later claims that mercury-containing thimerosal in some vaccines continued to fuel the anti-vax cause. Also wrong. Here's a brief history.

The Scandal

Just as vaccinations are provably good things, surely the diet recommendations of the major medical and government institutions in favor of limiting fat must also be! Sadly, this is not the case. Rather, it's a wonderful example of how hard paradigm shifts are to accomplish, particularly when the prestige of major institutions are involved. And, sadly, how prestige and baseless assertions have substituted for science, shockingly similar to bloodletting and other universally-accepted-on-no-objective-basis practices.

A basic, understandable summary of the subject may be found in The Big Fat Surprise, which is loaded with appropriate detail. Here is a summary:

"the past sixty years of low-fat nutrition advice has amounted to a vast uncontrolled experiment on the entire population, with disastrous consequences for our health.

For decades, we have been told that the best possible diet involves cutting back on fat, especially saturated fat, and that if we are not getting healthier or thinner it must be because we are not trying hard enough. But what if the low-fat diet is itself the problem? What if those exact foods we’ve been denying ourselves — the creamy cheeses, the sizzling steaks — are themselves the key to reversing the epidemics of obesity, diabetes, and heart disease?"

Yes, this sounds like what an anti-science crank would say. All I can say is, dig in. You'll find the shoddy beginnings of the fat-cholesterol-heart hypothesis; the biased studies that seemed to support it; the massive, multi-decade Framingham study which was trumpeted as supporting the anti-fat theory, but whose thoroughly confirmed and vetted results were actively suppressed for many years; the uncontested studies that disprove the anti-fat recommendations; and the improved understanding of the biological systems that thoroughly debunks the widely promoted campaign against saturated fat and LDL, the "bad" cholesterol.

More detail

If you want a start on more detail, I recommend Dr. Sebastian Rushworth at a high level and the recent book by long-term cardiac doctor Malcolm Kendrick that gives the details of the studies and biology that explain what really happens.

Here are a couple explanations from Dr. Rushworth:

"the LDL hypothesis basically says that heart disease happens because LDL somewhow ends up in the arterial wall, after which it is oxidized, which starts an inflammatory reaction that gradually leads to the hardening of arteries and eventually to bad things like heart attacks and strokes."

"... the LDL hypothesis is bunk. There is by now a wealth of evidence showing that LDL has little to do with heart disease, such as this systematic review from BMJ Evidence Based Medicine, which showed that there is no correlation whatsoever between the amount of LDL lowering induced by statins and other LDL lowering drugs, and the benefit seen on cardiovascular disease risk (if indeed any benefit is seen – it often isn’t)."

Rushmore's summary of the Kendrick book is:

"The ultra-short elevator pitch version of what he argues in the book is that heart disease is what happens when damage to the arterial wall occurs at a faster rate than repair can happen. That’s why everything from sickle cell disease to diabetes to high blood pressure to smoking to rheumatoid arthritis to cortisone treatment to the cancer drug Avastin increases the risk of cardiovascular disease – they all either increase the speed at which the arterial wall gets damaged or slow down its repair. It’s why heart disease (more correctly called “cardiovascular disease”) only affects arteries (which are high pressure systems) and not veins (which are low pressure systems), and why atherosclerosis (the hardening of the arteries that characterizes heart disease) primarily happens at locations where blood flow is extra turbulent, such as at bifurcations.

This alternative to the LDL hypothesis is known as the “thrombogenic hypothesis” of heart disease. It’s actually been around for a long time, first having been proposed by German pathologist Carl von Rokitansky in the 19th century. Von Rokitansky noted that atherosclerotic plaques bear a remarkable similarity to blood clots when analyzed in a microscope, and proposed that they were in fact blood clots in various stages of repair.

Unfortunately, at the time, von Rokitansky wasn’t able to explain how blood clots ended up inside the artery wall, and so the hypothesis floundered for a century and a half (which is a little bit ironic when you consider that no-one knows how LDL ends up inside the artery wall either, yet that hasn’t hindered the LDL hypothesis from becoming the dominant explanation for how heart disease happens). We now know the mechanism by which this happens: cells formed in the bone marrow, known as “endothelial progenitor cells”, circulate in the blood stream and form a new layer of endothelium on top of any clots that form on the artery wall after damage – thus the clot is incorporated in to the arterial wall.

In spite of the fact that probably at least 99% of cardiologists still believe in the LDL hypothesis, the thrombogenic hypothesis is actually supported far better by all the available evidence. While the LDL hypothesis cannot explain why any of the risk factors listed above increases the risk of heart disease, the thrombogenic hypothesis easily explains all of them.

One of the things Malcolm Kendrick mentions in his new book is the molecule Lipoprotein(a), a.k.a. Lp(a). It has been found to correlate far better with risk of heart disease than LDL does. Interestingly, it is in fact identical to LDL, with one small difference – it has an extra protein, called Apo(a), bound to its surface. This gives it a function that is quite different from LDL. As mentioned, LDL transports cholesterol in the blood stream. What Lp(a) does instead is to stabilize blood clots. Why does this matter?

Because whenever the arterial wall is damaged, a blood clot forms to prevent you bleeding out. The arterial wall then, as mentioned, rebuilds itself on top of the blood clot, and the clot is gradually degraded and absorbed by the body. People with high levels of Lp(a) have more stable blood clots, which means that their blood clots are absorbed more slowly. In other words, arterial wall repair happens more slowly, so people with high Lp(a) have more atherosclerosis and therefore more heart disease."

What does all of this have to do with PCSK9 inhibitors?

PCSK9 inhibitors work by increasing the expression of the LDL receptor on the surface of liver cells. The traditional LDL hypothesis explanation for how PCSK9 inhibitors reduce heart disease is that the increased expression of the LDL receptor results in increased uptake of LDL from the blood stream and therefore lower levels of LDL in the body. There is, however, a tantalizing alternate explanation for how PCSK9 inhibitors reduce heart disease, that fits perfectly with the thrombogenic hypothesis.

As mentioned, LDL and Lp(a) are virtually identical. And the LDL receptor is not able to differentiate between them, so an increased expression of LDL receptors will result not just in a decrease in LDL in the bloodstream, but also in a decrease in Lp(a). That is why the PCSK9 inhibitor evolocumab has been found to decrease Lp(a) by around 30%. It’s likely also why there is any correlation at all between LDL and heart disease – the LDL receptor is responsible for removing both LDL and Lp(a). But it’s the Lp(a) that’s contributing to heart disease, not the LDL. It’s a classic case of correlation being mistaken for causation."

Conclusion

Many major institutions have dialed down their fervent condemnation of the low-fat and LDL-is-bad myths, but haven't done what they should do, which reverse their positions and mea culpa. They should at minimum take at least part responsibility for the explosion of obesity, useless pharma mega-dollars wasted, and the attendant health disasters for countless humans. The fact that they're not helps us understand the resistance to correction of the similarly powerful mainstream myths about software. It's not about the LDL or the software; it's about people, pride, institutions, bureaucracy and entrenched practices and beliefs that fight change.

 

Computer Science and Kuhn's Structure of Scientific Revolutions

If bridges fell down at anywhere close to the rate that software systems break and become unavailable, there would be mass revolt. Drivers would demand that bridge engineers make radical changes and improvements in bridge design and building. If criminals took over bridges and held the vehicles until they paid a ransom anywhere close to the number of times criminals rob organizations or their data or lock their systems until a ransom is paid, there would be mass revolt. In the world of software, this indefensible state of affairs is what passes for normal! Isn't it time for change? Has something like this ever happened in other fields that we can learn from?

Yes. It's happened enough that it's been studied, and the process of resistance to change until the overwhelming force of a new paradigm breaks through.

Thomas Kuhn was the author of a highly influential book published in 1962 called The Structure of Scientific Revolutions. He introduced the term “paradigm shift,” which is now a general idiom. Examining the history of science, he found that there were abrupt breaks. There would be a universally accepted approach to a scientific field that was challenged and then replaced with a revolutionary new approach. He made it clear that a paradigm shift wasn’t an important new discovery or addition – it was a whole conceptual framework that first challenged and then replaced the incumbent. An example is Ptolemaic astronomy in which the planets and stars revolved around the earth, replaced after long resistance by the Copernican revolution.

Computer Science is an established framework that reigns supreme in academia, government and corporations, including Big Tech. There are clear signs that it is as ready for a revolution as the Ptolemaic earth-centric paradigm was. Many aspects of the new paradigm have been established and proven in practice. Following the pattern of all scientific revolutions, there is massive establishment resistance, led by a combination of ignoring the issues and denying the problems.

The Structure of Scientific Revolutions

Thomas Kuhn received degrees in physics, up to a PhD from Harvard in 1949. He was into serious stuff, with a thesis called “The Cohesive Energy of Monovalent Metals as a Function of Their Atomic Quantum Defects.” Then he began exploring. As Wiki summarizes:

As he states in the first few pages of the preface to the second edition of The Structure of Scientific Revolutions, his three years of total academic freedom as a Harvard Junior Fellow were crucial in allowing him to switch from physics to the history and philosophy of science. He later taught a course in the history of science at Harvard from 1948 until 1956, at the suggestion of university president James Conant.

His path for coming to his realization is fascinating. I recommend reading the book to anyone interested in how science works and the history of science.

After studying the history of science, he realized that it isn't just incremental progress.

Kuhn challenged the then prevailing view of progress in science in which scientific progress was viewed as "development-by-accumulation" of accepted facts and theories. Kuhn argued for an episodic model in which periods of conceptual continuity where there is cumulative progress, which Kuhn referred to as periods of "normal science", were interrupted by periods of revolutionary science. The discovery of "anomalies" during revolutions in science leads to new paradigms. New paradigms then ask new questions of old data, move beyond the mere "puzzle-solving" of the previous paradigm, change the rules of the game and the "map" directing new research.^[1]

Real-life examples of this are fascinating. The example often given is the shift from "everything revolves around the earth" to "planets revolve around the sun." What's interesting here is the planetary predictions of the Ptolemaic method were quite accurate. The shift to Copernicus (Sun-centric) didn't increase accuracy, and the calculations grew even more complicated. The world was not convinced! Kepler made a huge step forward with elliptical orbits instead of circles with epicycles and got better results that made more sense. The scientific community was coming around. Then when Newton showed that Kepler's laws of motion could be derived from his core laws of motion and gravity the revolution won.

While the book doesn't emphasize this, it's worth pointing out that the Newtonian scientific paradigm "won" among a select group of numbers-oriented people. The public at large? No change.

Anomalies that drive change

One of the interesting things Kuhn describes are the factors that drive a paradigm shift in science -- anomalies, results that don't fit the existing theory. In most cases, anomalies are resolved within the paradigm and drive incremental change. When anomalies resist resolution, something else happens.

During the period of normal science, the failure of a result to conform to the paradigm is seen not as refuting the paradigm, but as the mistake of the researcher, contra Popper's falsifiability criterion. As anomalous results build up, science reaches a crisis, at which point a new paradigm, which subsumes the old results along with the anomalous results into one framework, is accepted. This is termed revolutionary science.

The strength of the existing paradigm is shown by the strong tendency to blame things on mistakes of the researcher -- or in the case of software, on failure to follow the proper procedures or to write the code well.

The Ruling Paradigm of Software and Computer Science

There is a reigning paradigm in software and Computer Science. As you would expect, the paradigm is almost never explicitly discussed. It has undergone some evolution over the last 50 years or so, but not as radical as some would have it.

At the beginning, computers were amazing new devices and people programmed them as best they could. Starting over 50 years ago, people began to notice that software took a long time and lots of effort to build and was frequently riddled with bugs. That's when the foundational aspects of the current paradigm were born and started to grow, continuing to this day:

1. Languages should be designed and used to help programmers avoid making mistakes. Programs should be written in small pieces (objects, components, services, layers) that can be individually made bug-free.
2. Best-in-class detailed procedures should be adapted from other fields to assure that the process from requirements through design, programming, quality assurance and release is standardized and delivers predictable results.

The ruling paradigm of software and computer science is embodied in textbooks, extensive highly detailed regulations, courses, certifications and an ever-evolving collection of organizational structures. Nearly everyone in the field unconsciously accepts it as reality.

Are There Anomalies that Threaten the Reigning Paradigm?

Yes. There are two kinds.

The first kind are the failures of delivery and quality that continue to plague by-the-book software development, in spite of decades of piling up the rules, regulations, methods and languages that are supposed to make software development reliable and predictable. The failures are mostly attributed to errors and omissions by the people doing the work -- if they had truly done things the right way, the problems would not have happened. At the same time, there is a regular flow of incremental "advances" in procedure and technology designed to prevent such problems. This is textbook Kuhn -- the defenders of the status quo attributing issues to human error.

The second kind of anomalies are bodies of new software that are created by small teams of people who ignore the universally taught and proscribed methods and get things done that teams 100's of times larger couldn't do. Things like this shouldn't be possible. Teams that ignore the rules should fail -- but instead most of the winning teams are ones that did things the "wrong" way. This is shown by the frequency of new software products being created by such rule-ignoring small groups, rocketing to success and then being bought by the rule-following organizations, including Big Tech, who can't do it -- in spite of their giant budgets and paradigm-conforming methods. See this and this.

When will this never-ending stream of paradigm-breaking anomalies make a paradigm-shifting revolution take place in Computer Science? There is no way of knowing. I don't see it taking place any time soon.

Conclusion

The good news about the resistance of the current consensus in Computer Science and software practice to a paradigm shift is that it provides the room for creative entrepreneurs to build new things that meet the unmet needs of the market  The entrepreneurs don't even have to go all-in on the new software paradigm! They just need to ignore enough of the bad old stuff and use enough of the good new stuff to get things done that the rule-followers are incapable of. Sadly, the good news doesn't apply to fields that are so outrageously highly regulated that the buyer insists on being able to audit compliance during the build process. Nonetheless, there is lots of open space for creative people to build and grow.

The Dimension of Software Automation Breadth

Computers and software are all about automation. See this for the general principles of automation. When you dive into details and look at lots of examples, patterns emerge. The patterns amount to sequences or stages of automation that emerge over time, with remarkable consistency. When you apply your knowledge of the pattern to the software used in an industry at any given time, you can identify where the software is in the sequence. You can confirm the earlier stage in the sequence and predict with accuracy the stage that will follow. This gives you ability to say what will happen, though not when or by whom.

The patterns of automation play out in multiple dimensions. I have described what may be called the depth of automation, in which software evolves from recording what people do through helping them and eventually to replacing them.

In this post I will describe another dimension, which may be thought of as the breadth of automation. The greater the automation breadth the more functions are incorporated into the automation software and the more highly integrated the functions are with each other.

The Dimension of automation breadth

Automation breadth has these basic levels, independent of the automation depth of the products involved:

Component	Not a stand-alone product, but a component that could be incorporated into many custom applications and/or products
Point product	Implements a single function
Product collection	A group of point products from a single vendor
Product suite	An integrated set of separate products, with meaningful benefits from the integration
Integrated product	A single body of source code that performs a variety of related functions that would otherwise require separate products, with meaningful benefits from the unification
Integrated product with selective outsourcing	A product that is written and delivered in such a way that the using organization can choose to have the vendor staff a number of functions off-site.

Components rarely appear first in historical terms, but they are the beginning of this sequence. Typically, functionality that is very difficult to write or whose requirements change rapidly is separated out and delivered in the form of a component. The quality of the component may become so important that it becomes an industry standard.

Example: The Ocrolus service (disclosure: Oak HC/FT is an investor) is a classic example of a valuable, narrow component. It takes images of documents of nearly any kind, recognizes them, extracts their data and returns the data to the component user. This functionality is so challenging to write, and the consequence of errors so great, that most applications that need the functionality are likely to use the component, which is delivered as a service, rather than write their own.

In a time of rapidly emerging functionality, point products are usually first, because they can be gotten to market quickly. Buyers typically talk in terms of “best of breed,” but have the problem of negotiating and maintaining relationships with multiple vendors, who frequently have conflicting interests. The buyer also has to take responsibility for integrating the various products he has bought. Buyers reasonably worry about whether their typically small vendor will stay in business and continue to invest in their product.

Example: Captura (now part of Concur) provided a point product to enable a company to automate the process of entering, approving and paying expense reports.

Product collections solve some of these problems. There is now a single vendor; the vendor is probably much larger and more likely to stay in business; the products will be maintained and there is no conflict of interest. Once a product collection becomes available in a category, buyers will typically prefer an adequate product from a collection to a superior point product. Product collections can be formed by acquisition.

Example: CA, Computer Associates, is a typical vendor that acquires products in a category and puts them together into a product collection.

It is often desirable to share data among the products in a collection. Frequently, they maintain copies of essentially the same information, have essentially the same security roles defined, etc. Users have to go through considerable time and effort to accomplish this on their own, and then again when there is a new release, and desirable integrations are sometimes not even possible. Product suites solve these problems to a large extent, since a single vendor performs the integration and sells the integration as part of the product collection. Once a product suite becomes available in a category, buyers will typically prefer an adequate product suite to a product collection whose products are superior, because the cost of installation and maintenance is lower and the benefits of integration often outweigh the benefits of individual product features. Building a product suite typically requires source-code level coding and functionality design changes, but the code bases of the products can be separate.

Classic example: Microsoft Office was one of the first product suites for office products. While the benefits of integration are not overwhelming in this functional area, users clearly benefit from having a suite rather than individual products. Once the benefits of a suite became accepted, buyers no longer wanted individual office products.

Recent example: The market for business formation has long been dominated by the point product Legalzoom. A new, rapidly growing product suite called Zenbusiness (disclosure: Oak HC/FT is an investor) is taking classic advantage of moving to the next stage of automation breadth, by offering small business customers not only business formation services, but also services for websites, banking and accounting.

Not all functionality areas benefit from having an integrated product, but for those that do, integrated products are better for vendors (reduced costs due to elimination of redundancy) and for buyers (a simpler, more unified product that is easier to learn and operate, and has deep, fully-automated function integration), and typically win in the marketplace.

Example: Everyone thinks of SAP’s R3 as being the first client/server ERP product, but its real break-through was being the first truly integrated product on which a large enterprise could run its business. Using a single body of code and a common shared DBMS, its many modules each ran different parts of the enterprise business. In spite of the high cost of implementation and operation, the advantages of running your business on a single integrated product were overwhelming. Like most projects to build a unified product, the vendor had deep domain knowledge, it took a long time to get it right, and the early installations were painful.

Once functionality markets reach a level of maturity, the competition becomes intense, and organizations often have to choose areas of distinction, outsourcing the functions they choose not to compete in to a low-cost provider. Products that enable organizations to selectively outsource in this way typically take business from products that must be fully staffed by the buyer.

Example: FDC is the leading processor for credit cards in the US. With a billion cards outstanding, the market is huge and highly competitive. While the technology base of the FDC product is decades old, the functionality it delivers is highly sophisticated. In addition to delivering a completely integrated, multi-department product, FDC offers off-site staffing for the various departmental functions, where the staff sounds on the telephone as though they worked for you.

Conclusion

The dimension of automation breadth helps us understand the evolution of software and the progression that naturally takes place in a given market space. The companies that win are most often the ones that dominate a narrow market segment with a particular product and then broaden the range of software they can sell to a given organization. They typically move step by step according to the progression I have described here.

The Destructive Impact of Software Standards and Regulations

In many fields of life, standards and regulations are a good thing. Standards are why you can get into a new car and be comfortable driving it; without standard steering wheels and brake pedals, no one would be able to drive a rental car. Software is different. The misguided effort to impose standards and regulations on software development has played a key role in the nonstop cybersecurity disasters and software failures that most organizations try to minimize and ignore.

Do software standards and regulations literally cause software and cybersecurity disasters? Yes, in much the same way as looking at your cell phone while driving causes auto accidents. Everyone agrees that distracted driving is bad, but somehow distracted programming is ignored. It’s a classic case of ignorance and good intentions that have horrible unintended consequences. Seeing the bad consequences, the community of standards writers believes the problem is that their standards aren’t deep, broad and detailed enough -- let's distract the programmers even more!

Software and Bicycles

Suppose that writing a software program were like riding a bicycle, and that finishing writing the program was like reaching your destination on the bicycle.

Even if you're not a bike rider, you’ve likely seen lots of bikes being ridden. You've probably seen kids learning on training wheels:

(Image)

Kids eventually learn to balance and learn how to handle curves, hills and the rest. Then there are serious riders whose bodies are tuned to the task. They ride with focus and concentration, assuring that they handle every detail of the road they're on to maximum advantage.

(Image)

Because people who create standards and regulations on software are appallingly ignorant of creating code with maximum speed and quality, they impose all sorts of strange requirements on programmers. It's as though the bicycles were invisible to everyone but the programmers, but the leaders have it on best authority that the bicycles they demand their programmers use are up to the most modern standards.

(Image)

They create increasingly elaborate processes and methods that are supposed to assure that bicycles reach their destinations quickly and safely, but in fact assure the opposite. The programmers are required to juggle a myriad of meetings, planning discussions, reviews and other activities while still making great progress.

(Image)

The oh-so-careful regulators go to great lengths to assure that at each stage of the bicycle’s journey the rider does his riding flawlessly and without so much as a swerve from the proscribed path. When developers are forced to follow regulations and standards, they don’t just pedal, quickly and smoothly, to the finish line. They constantly stop and engage in myriad non-programming activities. No sooner do they start to pick up speed after being allowed back on the bike than ring! ring! It's time for the security review meeting!

Riding a bicycle competitively is not the most intellectual of activities. Neither is being a batter in baseball. But both require exquisitely deep focus and concentration. The batter's head can't be swimming with advice about what to do when the pitch is a sinker; the batter has to be present in the moment and respond to the pitch in real time with the evolving information he gets as the pitch approaches the plate.

In the same way and arguably even more so, the programmer has to be immersed in the invisible evolving "world" of the software around him, seeing what should to added or changed to what, where in that world. The total immersion in that world isn't something that can be flicked on with a switch -- though it can be brought crashing down in an instant by an interruption. In the same way, a biker doesn't get to maximum speed and focus in seconds. It takes time to sink in to the flow.

Meanwhile, all the managers judge programmers primarily by how well they juggle, like the skilled fellow in the picture above, blissfully unaware and seemingly uncaring that the unicycle is better for stopping, starting and going in circles than it is for making forward progress.

Conclusion

This is the ABP (Anything But Programming) factor in software -- make sure the programmers spend their time and energy on things other than driving the program to its working, useful, high-quality goal. The managers feel great about themselves. They are following the latest standards, complying with all the regulations, and assuring that the programmers under their charge are doing exactly and only the right thing -- doing it right the first time. When such managers get together with their peers, they exchange notes about which aspects of the thousands and thousands of pages of standards they have forced their programmers to distract themselves with recently. Because that's what modern software managers do.

The promising origins of object-oriented programming

The creation of the original system that evolved into the object-oriented programming (OOP) paradigm in design and languages was smart. It was a creative, effective way to think about what was at the time a hard problem, simulating systems. It’s important to appreciate good thinking, even if the later evolution of that good thinking and the huge increase in the scope of application of OOP has been problematic.

The evolution of good software ideas

Lots of good ideas pop up in software, though most of them amount to little but variations on a small number of themes.

One amazingly brilliant idea was Bitcoin. It solved a really hard problem in creative ways, shown in part by its incredible growth and widespread acceptance. See this for my appreciation of the virtues of Bitcoin, which remains high in spite of the various things that have come after it.

The early development of software languages was also smart, though not as creative IMHO as Bitcoin. The creation of assembler language made programming practical, and the creation of the early 3-GL’s led to a huge productivity boost. A couple of later language developments led to further boosts in productivity, though the use of those systems has gradually faded away for various reasons.

The origin of Object-Oriented Programming

Wikipedia has a reasonable description of the origins of OOP:

In 1962, Kristen Nygaard initiated a project for a simulation language at the Norwegian Computing Center, based on his previous use of the Monte Carlo simulation and his work to conceptualise real-world systems. Ole-Johan Dahl formally joined the project and the Simula programming language was designed to run on the Universal Automatic Computer (UNIVAC) 1107. Simula introduced important concepts that are today an essential part of object-oriented programming, such as class and object, inheritance, and dynamic binding.

Originally it was built as a pre-processor for Algol, but they built a compiler for it in 1966. It kept undergoing change.

They became preoccupied with putting into practice Tony Hoare's record class concept, which had been implemented in the free-form, English-like general-purpose simulation language SIMSCRIPT. They settled for a generalised process concept with record class properties, and a second layer of prefixes. Through prefixing a process could reference its predecessor and have additional properties. Simula thus introduced the class and subclass hierarchy, and the possibility of generating objects from these classes.  

Nygaard was well-rewarded for the invention of OOP, for which he is given most of the credit.

What was new about Nygaard’s OOP? Mostly, like Simscript, it provided a natural way to think about simulating real-world events and translating the simulation into software. As Wikipedia says:

The object-oriented Simula programming language was used mainly by researchers involved with physical modelling, such as models to study and improve the movement of ships and their content through cargo ports

In some (not all) simulation environments, thinking in terms of a set of separate actors that interact with each other is natural, and an object system fits it well. Making software that eases the path to expressing the solution to a problem is a clear winner. Simula was an advance in software for that class of applications, and deserves the credit it gets.

What should have happened next

Simula was an effective way to hard-code a computer simulation of a physical system. The natural next step an experienced programmer would take would be to extract out the description of the system being modeled and express it in easily editable metadata. This makes creating the simulation and making changes to it as easy as making a change to an Excel spreadsheet and clicking recalc. Here's the general idea of moving up the tree of abstraction. Here's an explanation and illustration of the power of moving concepts out of procedural code and into declarative metadata.

This is what good programmers do. They see all the hard-coded variations of general concepts in the early hard-coded simulation programs. They might start out by creating subroutines or classes to express the common things, but you still are hard-coding the simulation. So smart programmers take the parameters out of the code, put them into editable files and assure that they're declarative. Metadata! There are always exceptions where you need a calculation or an if-then-else -- no problem, you just enhance the metadata so it can include "rules" and you're set.

The later evolution of OOP

In spite of the power of moving from hard-coding to metadata, the language-obsessed elite of the computing world focused on the language itself and decided that its object orientation could be taken further. Simula’s object orientation ended up inspiring other languages and being one of the thought patterns that dominate software thinking. It led directly to Smalltalk, which was highly influential:

Smalltalk is also one of the most influential programming languages. Virtually all of the object-oriented languages that came after—Flavors, CLOS, Objective-C, Java, Python, Ruby, and many others—were influenced by Smalltalk. Smalltalk was also one of the most popular languages for agile software development methods, rapid application development (RAD) or prototyping, and software design patterns.The highly productive environment provided by Smalltalk platforms made them ideal for rapid, iterative development.

Lots of the promotional ideas associated with OOP grew along with Smalltalk, including the idea that objects were like Lego blocks, easily assembled into new programs with little work. It didn’t work out for Smalltalk, in spite of all the promotion and backing from influential players. The companies and the language itself dwindled and died.

It was another story altogether for the next generation of O-O languages, as Java became the language most associated with the internet as it grew in the 1990’s. I will tell more of the history later. For now I’ll just say that OOP is the paradigm most often taught in academia and generally in the industry.

Conclusion

OOP was invented by smart people who had a new problem to solve. Simula made things easier for modeling physical systems and was widely used for that purpose. As it was expanded and applied beyond the small category of problems for which it was invented, serious problems began to emerge that have never been solved. The problems are inherent in the very idea of OOP. The problems are deep and broad; the Pyramid of Doom is one of seemingly endless examples.

When OOP principles and languages were applied to databases and GUI’s, they failed utterly, to the extent that even the weight of elite experts couldn’t force their use instead of simple, effective approaches like RDBMS for database and javascript for GUI’s. OOP has evolved into a fascinating case study of the issues with Computer Science and the practice of software development, with complete acceptance in the mainstream along with widespread quiet dissent resulting from fraudulent claims of virtue.

Deep-Seated Resistance to Software Innovation

Everyone says they're in favor of innovation. Some organizations promote innovation with lots of publicity. Many organizations even have CIO's (Chief Innovation Officers) to make sure it gets done. But the reality is that resistance to innovation runs strong and deep in organizations; the larger the organization, the greater the resistance usually is. The reason is simple: innovation threatens the power and position of people who have it. They feel they have nothing to gain and much to lose.

It's not just psychology. Innovation resistance throws up barriers that are thick and high. See this for examples.

A good way to understand the resistance is look at sound technologies that have been proven in practice that could be more widely applied, but are ignored and/or actively resisted by the organizations that could benefit from them. I have called these In-Old-vations. Here is an innovation that is still waiting for its time in the sun, and here's one that's over 50 years old that is still being rolled out VERY slowly.

In this post I will illustrate the resistance to technology innovation in a little-known extreme example: the people in charge of Britain's war effort resisted innovations that could help them win the war. They were literally in war and losing and decided, in effect, that they'd rather lose. Sounds ridiculous, I know, but this is normal behavior of people in organizations of all kinds.

The Battle of Britain

Britain was at war, facing Hitler’s much larger, better-prepared military, who had already rolled over its adversaries. Life and death. Literally. The established departments did all they could do defend from attacks. The so-called Battle of Britain is well-known. What is not as widely known is the battle in the seas. German submarines were sinking British ships at an alarming rate. The Navy had no answers other than to do what they were already doing harder.

The situation was desperate. If there was ever a time to "think outside the box" it would seem this was it. The response of the Navy to new things? NO WAY. Amazing new weapons developed by uncertified people outside the normal departmental structures? NO WAY. Once those weapons are built and proven, use them to stop the submarines that were destroying boats and killing men by the thousands? NO WAY!!

Of course, you might think that someone would have known that the fairly recent great innovation in flying machines was achieved by "amateurs" flying in the face of the establishment and the acknowledged expert in flying as I describe here. You might think that Navy men would remember that perhaps the greatest innovation in naval history was invented by Navy-related people. But no. Protecting our power and the authority of our experts is FAR more important than a little thing like losing a war!

The story of the new way to fight submarines is told in this book:

Someone who was not part of the Navy establishment invented a whole new approach to fighting submarines. The person wasn't a certified, official expert. He was rejected by all relevant authorities and experts. Fortunately for the survival of England, Churchill made sure the concept was implemented and tested. The new devices were delivered to a ship.

This all took time and it was not until the spring of 1943 that the first Hedgehogs were being installed on Royal Navy vessels. When Commander Reginald Whinney took command of the HMS Wanderer, he was told to expect the arrival of a highly secret piece of equipment. ‘At more or less the last minute, the bits and pieces for an ahead-throwing anti-submarine mortar codenamed “hedgehog” arrived.’ As Whinney watched it being unpacked on the Devonport quayside, he was struck by its bizarre shape. ‘How does this thing work, sir?’ he asked, ‘and when are we supposed to use it?’ He was met with a shrug. ‘You’ll get full instructions.' Whinney glanced over the Hedgehog’s twenty-four mortars and was ‘mildly suspicious’ of this contraption that had been delivered in an unmarked van coming from an anonymous country house in Buckinghamshire. He was not alone in his scepticism. Many Royal Navy captains were ‘used to weapons which fired with a resounding bang’, as one put it, and were ‘not readily impressed with the performance of a contact bomb which exploded only on striking an unseen target’. They preferred to stick with the tried and tested depth charge when attacking U-boats, even though it had a hit rate of less than one in ten. Jefferis’s technology was too smart to be believed.

Here's what the new mortars looked like:

What happened? It was transformative:

Over the course of the next twelve days, Williamson achieved a record unbeaten in the history of naval warfare. He and his men sank a further five submarines, all destroyed by Hedgehogs. Each time.

If resistance to change to true technological innovation is so strong when you’re desperate, literally at death’s door, how do you think it’s going to be in everyday life? The rhetoric is that we all we love innovation! The reality is that anything that threatens anybody or anything about the status quo is to be ignored, shoved to the side and left to die. Anyone who makes noise about it obviously isn’t a team player and should find someplace to work where they’ll be happier. And so on.

Conclusion

Innovation happens. Often nothing "new" needs to be invented -- "just" a pathway through the resistance to make it happen. Here is a description of the main patterns followed by successful innovations. If you have an innovation or want to innovate, you should be aware of the deep-seated resistance to innovation and rather than meeting it head-on, craft a way to make it happen without head-on war. Go for it!

Franz Liszt: 210

This week is the 210th anniversary of the birth of Franz Liszt on Oct 22, 1811. I wrote a couple highlights about Liszt for his 200th anniversary. Aside from loving his achievements in life, he was the inspiration for this blog.

Liszt was one of those rare people who was world-class brilliant and also appreciated the valuable work of others, supporting that work in word and deed.

One way he demonstrated this was by championing the work of other composers by transcribing their work and performing it for audiences who might otherwise not have a chance to hear it. He made amazing transcriptions of operas and many other kinds of works. Probably the most impressive transcriptions he made were making piano versions of all nine Beethoven symphonies!

Oddly enough, the symphony transcriptions were largely ignored in the world of music, even by Liszt's pupils. It wasn't until Glenn Gould recorded a couple of them in 1967 that they began to appear on the scene. But they remain obscure, even among classical music lovers.

I thought I knew all the Beethoven symphonies, but hearing a Liszt piano transcription performed was like hearing it for the first time. After a great deal of struggle he finally managed to transcribe the fourth (choral) movement of the ninth symphony. And did it again for duo pianos.

Frederic Chiu has recorded the 5th and 7th Symphonies for Centaur Records. In his program notes he suggests that "Liszt's piano scores must therefore be taken as a sort of gospel in regards to Beethoven's intentions with the Symphonies" because of Liszt's unique perspective, having met Beethoven in person, having heard collaborators and contemporaries of Beethoven perform the Symphonies, having studied and performed the works both as a pianist/transcriber and as a conductor in Weimar. No one in history could claim to have as much exposure, insight and journalistic integrity as regards Beethoven's intentions around the Symphonies.

I personally recommend the recordings by Konstantin Scherbakov. You will be listening to the unique inspiration of Beethoven, further enhanced by the incredible genius of Liszt.

Listening to these transcriptions can pull you out of normal physical space into a dimension filled with drama, beauty and inspiration, one that, unlike the austere, timeless beauty of math, drives through time from a start to a conclusion.

Software Programming Language Evolution: Libraries and Frameworks

We've talked about the major advances in programming languages and the enhancements that brought those languages to a peak of productivity -- a peak which has not been improved on since. Nonetheless there has been an ongoing stream of new programming languages invented, each of which is claimed to be "better" -- with no discussion about what constitutes "goodness" in a language! This is high on the list of causes of the never-ending chaos and confusion that pervades the world of software. While loads of people focus on language, the most important programming tools that make HUGE contributions to the productivity of programmers using them goes largely unremarked, in spite of the fact that they are part of every programmer's day-to-day programming experience. What are these essential, always-used but largely in the background thing? Libraries and frameworks.

Libraries

A library is a collection of subroutines, often grouped by subject area, which perform commonly used functions for programs. Every widely used language has a library that is key to its practical use.

For example, the C language has libraries that contain hundreds of functions for

• input and output, including formatting
• string manipulation and character testing
• math functions
• standard functions, dozens of them
• date and time

There are libraries of functions for controlling displays and input devices and many other things. After all these years and the "fatal" flaw of not being object-oriented (gasp!!), it is currently the #2 language in popularity. While the libraries don't play as important part of its popularity as certain other languages, they are still important.

Python became widely used among people doing analytics not particularly because of the virtues of the language itself, but because it grew one of the richest modern libraries of routines for calculations available. It ended up with great support for statistics and the things you need to do with large number sets including array and matrix manipulation. When doing analytics, the library functions do nearly all the heavy lifting. All that the program has to do is deploy the rich set of functions against the problem in the right way!

A 2021 survey of language popularity confirms the popularity of Python, and the importance of its library in giving it its popularity. Here's a summary:

Have you ever heard of the language R? It's an open-source language which became usable in the year 2000. Chances are if you work in statistics, data analytics, operations research or other areas involving serious data manipulation you know all about it and probably use it. The language itself has valuable features for math programming that most languages don't have, for things like vectors and matrices. More importantly it has an amazing rich library of the R equivalent of libraries, called packages. R packages contain reusable collections of functions and data definitions to perform valuable calculations. When you're trying to do something you're pretty sure someone else has tried to tackle in some way, the first thing you do is look for an appropriate package. Packages are available for most common data science tasks; they do much of the work and nearly all the "heavy lifting" of performing any job for which the package applies. R is a prime example of a language having virtues, but the bulk of the value and productivity coming from the available libraries.

One of the sad turns taken in programming language evolution was driven by the ascendancy of the modern RDBMS. Ironically, the DBMS emerged at about the same time as do-everything language environments dominated the new minicomputer landscape. With a single environment like MUMPS or PICK you could get a whole job done -- user interface, core program, data storage and access, everything! These environments enabled massive productivity gains. See this for more. But the DBMS blasted in, took over, and HAD to be used. So 3-GL's that were less productive than the new environments became even less so by finding cumbersome ways to use DBMS's. The way they did it I explain here. This was a first major step down the degenerate, productivity-killing road of layers. How did 3-GL's pull off the integration? With libraries, of course! For example, Java's JDBC library became a requirement for burdensome, error-prone access to standard databases from within Java, while those who didn't mind the overhead and "impedance mismatch" could use one of the ORM's (Object-Relational Mapping systems) that emerged.

Not all libraries are tightly associated with a language! One such library is redis.io, an open source project started by an Italian developer to build a powerful in-memory key/value datastore and cache. It has taken off and has many more powerful features including queuing. While unknown among non-programmers, it is incredibly popular, widely used and available on the major cloud providers.

Other libraries are directed towards a particular aspect of programming. With the emergence of the web UI, a succession of libraries to ease their creation have emerged.

A UI library that has become dominant is React, created and then open-sourced by Facebook. It is specifically for the Javascript language and greatly enhances the productivity of building and the resulting performance of web UI's. The benefits of javascript or any other language are trivial compared to the productivity gain produced by using React. Unlike most libraries, React comes close to being a framework; it resembles an R package, in that it's a comprehensive solution to the problem of building web UI's. Interestingly, part of the programmer productivity gain is due to the fact that it has a major declarative aspect to its design, like AngularJS (see below).

Bottom line: while the details of software language features have some impact on productivity, the availability and richness of libraries enhances productivity many times over. It's no contest.

Frameworks

Frameworks take the idea of libraries but with an important reversal. Libraries are sets of routines, any of which may be called by a program written in a supported language. The program written in the language is completely in charge. Frameworks, by contrast, provide an environment in which a language can operate. You select exactly one framework to work in. When you play by its rules, you normally enjoy large productivity benefits.

A library is a rich set of resources covering many issues, like a book library. A framework is a selected set of resources enabling rapid work on a given category of issues, like a kitchen for cooking.

One impressive framework is the RAILS framework for the language Ruby. While the Ruby language had become fairly successful, Ruby on Rails (as it was called) established it as a player because many groups discovered that it was many times faster than other tools at building a web application that has a UI and a database. The reason is simple: normally to build such an application you have one expert creating the UI probably using javascript and some library, another one building the server-side business logic, and another one creating the DBMS -- with a great deal of work and trouble relating the database to the server program; see for example JDBC and ORM's above. With RAILS, you defined your database, which automatically defined the names you use inside Ruby to access the data. Similar story with the UI.

The philosophy of RAILS centered on the DRY principle -- Don't Repeat Yourself.

Unlike the endless repetition of the usual layered environment, RAILS took a major step towards the principle of Occamality. Was RAILS original, a real break-through? In its object-oriented environment, yes. In programming in general, no. RAILS is a typical example of the appalling lack of knowledge of history in software. Most of the benefits of RAILS were delivered in a more integrated way many years before by the Powerbuilder environment. I discuss the context here.

What comprehensive frameworks like this REALLY do is attempt to re-create the comprehensive, everything-in-one-place development environments that emerged to enhance programmer productivity beyond what was achievable in 3-GL's, mostly by incorporating data storage and user interface functions, as I explain here. Their emergence and widespread adoption is clear evidence of the power of eliminating the insanity of layers in software.

There are also frameworks that are much narrower in scope, addressing only part of the programming puzzle. While they leave much of an overall programming job alone, they can give a major boost to the narrow area on which they focus. One that is focused exclusively on building web UI's  that caught on and became widely used is AngularJS, a new version of which is simply Angular. This framework is highly declarative, focused more on describing the elements of the UI than the actions required to implement it.

This led to HUGE programmer productivity gains. Why would anyone build a web UI from scratch? Nonetheless, the similarities between the ReactJS library and the AngularJS framework are strong, and both of them powered huge programmer productivity gains that had very little to do with the virtues (or lack thereof) of the associated language, javascript.

Languages vs Libraries and Frameworks

 New languages continue to flow out of the creative minds of groups of programmers who obviously don't have enough to do to keep themselves fully occupied. Each new language tends to be lauded, if only by its creators, with extreme claims of virtue on many dimensions. None of these claims are ever subjected to verification and testing, much less of the rigorous kind. In any case, there is simply no contest between the gains delivered by a rich library or framework and a new programming language.

I'll just give glaring example. One of the main virtues that Object-oriented languages are supposed to have is code reuse. The concept that is bandied about is that good class system is like a set of Lego blocks, enabling new programs to be easily assembled. Mostly it doesn't happen. For re-use, libraries and frameworks are the gold standard. Think of a normal, old-fashioned book library. The whole reason they exist is that people reuse books! It's the same thing for software libraries -- code gets into software libraries because it's re-used often! That's how they got to be called "libraries." Duh.

 

What is technical due diligence for Venture Capital?

I’m Technology Partner in a VC firm, Oak HC/FT. I help look at companies we’re considering investing in, and I help out as needed after investment. Recently I’ve been thinking about what I do, mostly because I’ve encountered multiple cases of technology due diligence performed on companies I know well. To put it plainly, I don’t do it the same way.

The firms whose work I’ve seen are professional, well-regarded groups. Their work fits into a pattern of similar work I’ve seen over many years. Their work can have value.

My Evolution in Tech Due Diligence

When I started performing tech due diligence for the VC firm Oak Investment Partners in 1992, I had no specific idea of how to do it. My background had been almost exclusively on the “other side,” i.e., working and building software, mostly for small, entrepreneurial companies. But I’d had occasion to dive into bodies of software over the years and figure them out, so I guessed I’d be able to figure it out better than any non-programmer could. All I can say is, I went through quite an evolution from there.

The starting point was clear and simple: evaluate the software. There’s a lot of work involved and not many people have the skills to do it well, but there’s no magic. I quickly developed an outline for what an evaluation consists of. It was a few pages long. Here’s the last part:

The earlier parts … well, let’s just say it was comprehensive.

I marched forward doing the work and interacting with the people in the company being evaluated and with the partners in the VC firm. I evolved my views of tech due diligence.

In the late 1990’s, I did an on-site tech diligence of a little company called Inktomi. It was started by a couple guys from U Cal Berkeley, one a young professor and the other a grad student. They were hard at work on a project called NOW – the network of workstations, which was an early attempt to build software that would enable a bunch of workstations connected by fast networking to solve really big computing problems, of the kind normally only so-called super-computers could handle. They built the software, and needed a big, big problem to solve to demonstrate that it worked. There were already full-text databases around that ran on standard computers. Also at the time, the internet was exploding, with no end in sight. There was an emerging need to help people find things on the internet. At the time, the best available solutions were “portals,” of which Yahoo was a leading example. A portal was a densely populated page managed by people who would put up links to the main things people were interested in. But what if you wanted more? Wouldn’t it be great to have a full-text search engine that could index and search the entire internet, even as it grew endlessly?

This is the problem Inktomi took on. No existing search engine could come within a factor of 100 of indexing and searching the whole internet at the time. The boys at Inktomi used their NOW infrastructure to attack the problem, first building a crawling and index-building system, then building the search. It wasn’t easy, but it turns out that it fit right into the NOW approach. If you understood it, you could see that by adding workstations to the NOW, you could scale the index endlessly, no matter how large the internet grew.

I walked into the second-floor warren of offices above stores on Shattuck Ave. in Berkeley, with machines and people crammed in anywhere they would fit. I already had heard the basic idea. I looked over people’s shoulders at screens, and noticed some things about the C code there. I asked questions and discovered they had implemented parallelism not by using standard UNIX multi-tasking, but by a super-low-overhead coding method. I saw the cables and boards, and found out the clever things they had done to minimize inter-workstation latency. And some more stuff.

Before long, and without doing anything like the exhaustive inventory of my original approach to due diligence, I had discovered the originality of their approach and their impressive implementation of it, which would give them a lead over any competitors that would emerge. I became a strong advocate of the deal.They took our money, the largest investment we had made in a company to that date, and ended up making over 100X return.

Of course today, we don’t “inktomi” for things on the internet, we “google.” That’s a story for another time, and happened after Inktomi had attained a public market valuation in excess of $10 Billion.

I tell the story to illustrate one of the fulcrum points of my evolution in performing tech due diligence. The way I did it and the resulting win helped spur me on to doing what the VC really needs, which is to leverage my experience and skills into getting insights into a potential deal that are invisible to the other members of the firm because of their lack of detailed knowledge of software and systems. The insights can range from “makes no difference here” to “OMG negative” to “how could I have known that super-positive.”

What I finally evolved to, and continue to work at, revolves around this simple fact: the VC firm wants to know if this technology, the way it is today and is likely to grow into the future, will make the company into a success and end up making money for us and our investors. That’s it! Everything else is a footnote or appendix.

The vast majority of tech due diligence I see performed by firms who specialize in performing that work, and who are regularly retained by groups doing investments and/or acquisitions, is not about this. What I almost universally see is something like what I started with long ago, with this very important added element: how do the tech, the people, the organization, the process, the deployment and the architectural choices made by the firm compare to widely held industry standards, best practices and regulations? When I look back at my old outline, I see this element entirely missing! Thinking back, I realize that the reason is that I had already decided that those widely held standards, unlike the ones in law and accounting, were a pile of crap. I had already noticed what I later began to systematically study and write about, the difference between the "war time" software practices of winning startups, compared to the widely accepted "peace time" methods.

Conclusion

If you want to acquire a tech-oriented company, you naturally want to perform due diligence. If you’re a big company, you may want to acquire it for its momentum and market edge; Facebook and Google do lots of acquiring for this reason, among others. You want to know how well the target company conforms to the kinds of standards, practices and procedures that your internal organization is held to. But if you’re a smart VC, you want different things from tech diligence. Does the tech give the company an “unfair advantage?” Do they approach tech with a speed-oriented, get-stuff-done attitude? The tech may be 2X better than the incumbents, but could a newcomer pretty easily get 5X? Sounds improbable, but this kind of thing happens all the time. If you’re a VC about to place a bet on an up-and-coming company, that’s the kind of tech input you should want.

Software Programming Language Evolution: the Structured Programming GOTO Witch Hunt

In prior posts I’ve given an overview of the advances in programming languages, described in detail the major advances and defined just what is meant by “high” in the phrase high-level language. I've described the advances in structuring and conditional branching that brought 3-GL’s to a peak of productivity.

The structuring and branching caught the attention of academics. Watch out! What happened next was that a theorem was proved, a movement was declared and named, and a certain indispensable part of any programming language, the GO TO statement, was declared to be a thing only used by bad programmers and should be banned. Here's the story of the nefarious GOTO.

Structures in Programming Languages

I've described how structures were part of the first 3-GL's and how they were soon elaborated to more clearly express the intention of programmers, making code even more productive to write. The very first FORTRAN compiler, delivered in 1957, included primitive versions of conditional branching and loops, two of the foundations of programming structure. It was so powerful that the early users figured it decreased the number of statements needed to achieve a result more than 10 times.

These are the people who actually WRITE PROGRAMS! They want to make it easier and jumped on anything that gave a dramatic improvement.

“Significantly, the increasing popularity of FORTRAN spurred competing computer manufacturers to provide FORTRAN compilers for their machines, so that by 1963 over 40 FORTRAN compilers existed. For these reasons, FORTRAN is considered to be the first widely used cross-platform programming language.”

Before long, the structuring capabilities of the original IF (conditional branching) and DO (controlled looping) statements were enhanced and augmented to something close to their current form. I describe this here. The result was a peak of programmer productivity that has not substantially been increased since, and often been degraded.

The Bohm-Jacopini Theorem

Completely independent of the amazing advances in languages and programming productivity that were taking place, math-oriented non-programmers were hard at work deciding how software should be written. Here is the story in brief:

The structured program theorem, also called the Böhm–Jacopini theorem,^[1][2] is a result in programming language theory. It states that a class of control-flow graphs (historically called flowcharts in this context) can compute any computable function if it combines subprograms in only three specific ways (control structures). These are

1. Executing one subprogram, and then another subprogram (sequence)
2. Executing one of two subprograms according to the value of a boolean expression (selection)
3. Repeatedly executing a subprogram as long as a boolean expression is true (iteration)

The structured chart subject to these constraints may however use additional variables in the form of bits (stored in an extra integer variable in the original proof) in order to keep track of information that the original program represents by the program location. The construction was based on Böhm's programming language P′′.

The theorem forms the basis of structured programming, a programming paradigm which eschews goto commands and exclusively uses subroutines, sequences, selection and iteration.

This theorem got all the academic types involved with computers riled up. The key to good software has been discovered! The fact that math theorems are incomprehensible to the vast majority of people, and the fact that perfectly good computer programs can be written by people who aren't math types didn't concern any of these self-anointed geniuses.

The important thing to note about the theorem is that it was NOT created in order to make programming easier or more productive. It just "proved" that it was "possible" to write a program under the absurd and perverse constraints of the theorem to compute any computable function. Assuming you were willing to use a weird set of bits to store location information in ways that would make any such program unreadable by any normal person. Way to go, guys -- let's go back to the days of writing in all-binary machine language!

The Crisis in Software and its solution

Not long after this, the academic group of Computer Science “experts” formed. They had a conference. They looked at the state of software and declared it to be abysmal. The whole conference was about the "crisis" in software. See this for details.

One of the most prominent of those Computer Scientists was Edsger W. Dijkstra. He looked at the powerful constructs for conditional branching, loops and blocks that had been added to 3-GL's and invented the term "structured programming" to describe them. He related those statements to the wonderful but useless math proof about the minimal requirements for programming a solution to any "computable function." The proof "proved" that such programs could be written without the equivalent of a GOTO statement. BTW, I do not dispute this. He wrote "the influential "Go To Statement Considered Harmful" open letter in 1968."

Among the solutions to the software crisis they proclaimed was strict adherence to the dogma of what Dijkstra called “structured programming,” which prominently declared that the GOTO statement had no place in good programming and should be eliminated.

Does the fact that's it is POSSIBLE to program a solution to any computable function without using GOTO mean that you SHOULD write without using GOTO's? When children go to school, it's POSSIBLE for them to crawl the whole way, without using "walking" at all. Everyone accepts that this is possible. When you're on your feet all sorts of bad things can happen -- you can trip and fall! Most important, you can get the job done without walking ... and therefore you SHOULD eliminate walking for kids getting to school. QED.

This is academia for you – a prime example of how Computer Science works hard to make sure that programs are hard to write, understand and deliver, all in the name of achieving the opposite.

The debate about structured programming

There was no debate about the utility of the conditional branching, controlled looping and block structures that rapidly became part of any productive software language. They were there and programmers used them, then and now. The debate was about "structured programming," which by its academic definition outlawed the use of the GOTO statement. That wasn't all. It also outlawed having more than one exit to a routine, breaks from loops and other productive, transparent and generally useful constructs.

I remember clearly as a programmer in the 1980's having a non-technical manager type coming to me and quizzing me about whether I was following the rigors of structured programming, which was then talked about as the only way to write good code. I don't remember my answer, but since I knew the manager would never go to the trouble of actually -- gasp! -- reading code, my answer probably didn't matter.

The most important thing to know about the leader of the wonderful movement to purify programming is his lack of interest in actually writing code:

Fortunately, there are sane people in the world, including the incomparable Donald Knuth (an academic Computer Scientist who's actually great!) and a number of others.

An alternative viewpoint is presented in Donald Knuth's Structured Programming with go to Statements, which analyzes many common programming tasks and finds that in some of them GOTO is the optimal language construct to use.^[9] In The C Programming Language, Brian Kernighan and Dennis Ritchie warn that goto is "infinitely abusable", but also suggest that it could be used for end-of-function error handlers and for multi-level breaks from loops.^[10] These two patterns can be found in numerous subsequent books on C by other authors;^{[11][12][13][14]} a 2007 introductory textbook notes that the error handling pattern is a way to work around the "lack of built-in exception handling within the C language".^[11] Other programmers, including Linux Kernel designer and coder Linus Torvalds or software engineer and book author Steve McConnell, also object to Dijkstra's point of view, stating that GOTOs can be a useful language feature, improving program speed, size and code clarity, but only when used in a sensible way by a comparably sensible programmer.^[15][16] According to computer science professor John Regehr, in 2013, there were about 100,000 instances of goto in the Linux kernel code.^[17]

Any programmer can make mistakes. Any statement type can be involved in that mistake. For example, I think nearly everyone accepts that cars are a good thing. But over 30,000 people a year DIE in car accidents! So where's the movement to eliminate cars because of this awful outcome! It makes as much sense as outlawing the GOTO because sometimes it's used improperly. Like every other statement type.

Software Programming Language Evolution: Structures, Blocks and Macros

In prior posts I’ve given an overview of the advances in programming languages, described in detail the major advances and defined just what is meant by “high” in the phrase high-level language. In this post I’ll dive into the additional capabilities added to 3-GL’s that brought them to a peak of productivity.

History

Let’s remember what high-level languages are all about: productivity! They are about the amount of work it takes to write the code and how easy code is read.

The first major advance, from machine language to assembler, was largely about eliminating the grim scut-work of taking the statements you wanted to write and making the statements “understandable” to the machine by expressing them in binary. Ugh.

The second major advance, to 3-GL’s like FORTRAN and COBOL, was about eliminating the work of translating from your intention to the assembler statements required to express that intention. A single line of 3-GL code can easily translate into 10 or 20 lines of assembler code. And the 3-GL line of code often comes remarkably close to what you actually want to “say” to the computer, both writing it and reading it.

FORTRAN achieved this goal to an amazing extent.

“with the first FORTRAN compiler delivered in April 1957.^[9]:75 This was the first optimizing compiler, because customers were reluctant to use a high-level programming language unless its compiler could generate code with performance approaching that of hand-coded assembly language.^[16]”

“While the community was skeptical that this new method could possibly outperform hand-coding, it reduced the number of programming statements necessary to operate a machine by a factor of 20, and quickly gained acceptance.”

The reduction in the amount of work was the crucial achievement, but just as important was the fact that each set of FORTRAN statements were understandable, in that they came remarkably close to expressing the programmer’s intent, what the programmer wanted to achieve. No scratching your head when you read the code thinking to yourself “I wonder what he’s trying to say here??” This meant easier to write, fewer errors and easier to read.

Enhancing conditional branching and loops

The very first iteration of FORTRAN was an amazing achievement, but it’s no surprise that it wasn’t perfect. At an individual statement level it was nearly perfect. When reading long groups of statements there were situations where the code was clear, but the intention of the programmer not clearly expressed in the code itself – it had to be inferred from the code.

The first FORTRAN had a couple of the intention-expressing statements: a primitive IF statement and DO loop. Programmers soon realized that more could be done. The next major version was FORTRAN 66, which cleaned up and refined the early attempts at structuring. Along with the appropriate use of in-line comments, it was nearly as clear and intention-expressing as any practical programmer could want.

The final milestone in the march to intention-expressing languages was C.

It’s an amazing language. While FORTRAN was devised by people who wanted to do math/science calculations and COBOL by people who wanted to do business data processing, C was devised to enable computer people to write anything – above all “systems” software, like operating systems, compilers and other tools. In fact, C was used to re-write the first Unix operating system so that it could run on any machine without re-writing. C remains the language that is used to implement the vast majority of systems software to this day.

I bring it up in this context because C added important intention-expressing elements to its language that have remained foundational to this day. It enhanced conditional statements, creating the full IF-THEN-ELSE-ENDIF that has been common ever since. This meant you could say IF <condition is true> THEN <a statement>. The statement would only be executed if the condition was true. You could tack on ELSE <another statement> ENDIF. This isn’t dramatic, but the only other way to express this common thought is with GOTO statement – which certainly can be understood, but takes some figuring. In addition, C added the ability to use a delimited block of statements wherever a single statement could be used. When there are a moderate number of statements in a block, the code is easy to read. When there would be a large number, a good programmer creates a subroutine instead.

C added a couple more handy intention-expressing statements. A prime example is the SWITCH CASE BREAK statement. This is used when you have a number of conditions and something specific to do for each. The SWITCH defines the condition, and CASE <value> BREAK pairs define what to do for each possible <value> of the SWITCH.

Lots of following languages have added more and more statements to handle special cases, but the cost of a more complex language is rarely balanced with the benefit in ease and readability.

The great advance that's been ignored: Macros

C did something about all those special cases that goes far beyond the power of adding new statements to a language, and vastly increases not just the power and readability of the language but more important the speed and accuracy of making changes. This is the macro pre-processor.

I was already very familiar with macros when I first encountered the C language, because they form a key part of a good assembler language – to the extent that an assembler that has such a facility is usually called a macro-assembler. Macros enable you to define blocks of text including argument substitution. They resemble subroutine calls, but they are translated at compile time to code which is then compiled along with the hand-written code. A macro can do something simple like create a symbol for a constant that is widely used in a program, but which may have to be changed. When a change is needed, you just change the macro definition and – poof! – everywhere it’s used it has the new value. It can also do complex things that result in multiple lines of action statements and/or data definitions. It is the most powerful and extensible tool for expressing intention and enabling rapid, low-risk change in the programmer’s toolbox. While the C macro facility isn’t quite as powerful as the best macro-assemblers, it’s a zillion times better than not having one at all, like all the proud but pathetic modern languages that wouldn’t know a macro if it bit them in the shin.

The next 50 years of software language advances

The refrain of people who want to stay up to date with computers is, of course, "What's new?" Everyone knows that computers evolve more quickly than anything else in human existence, by a long shot. The first cell phones appeared not so long ago, and they were "just" cell phones. We all know that now they're astounding miniature computers complete with screens, finger and voice input, cameras and incredible storage and just about any app you can think of. So course we want to know what's new.

The trouble is that all that blindingly-fast progress is in the hardware. Software is just along for the ride! The software languages and statements that you write are 99% the same as in the long-ago times before smart phones. Of course there are different drivers and things you have to call, just as in any hardware environment. But the substance of the software and the lines of code you use to write it are nearly the same. Here is a review of the last 50 years of "advances" in software. Bottom line: it hasn't advanced!

Oh, an insider might argue: what about the huge advances of the object-oriented revolution? What about Google's new powerhouse, Go? Insiders may ooh and ahh, just like people at a fashion catwalk. The come-back is simple: what about programmer productivity? Claims to this effect are sometimes made, but more often its that the wonderful new language protects against stupid programmers making errors or something. There has not been so much as a single effort to measure increase of programmer productivity or quality. There is NO experimental data!! See this for more. Calling what programmers do "Computer Science" is a bad joke. It's anything but a science.

What this means is simple: everyone knows the answer -- claims about improvement would not withstand objective experiments -- and therefore the whole subject is shut down. If you're looking for a decades-old example of "cancel culture," this is it. Don't ask, don't tell.

Conclusion

3-GL's brought software programming to an astounding level of productivity. Using them you could write code quickly. The code you wrote came pretty close to expressing what you wanted the computer to do with minimal wasted effort. Given suitable compilers, the code could run on any machine. Using a 3-GL was at least 10X more productive than what came before for many applications.

A couple language features were incorporated into the very first languages, like conditional branching and controlled looping, that were good first steps. The next few years led early programmers to realize that a few more elaborations of conditional branching and controlled looping would handle the vast majority of practical cases. With those extra language features, code became highly expressive. All subsequent languages have incorporated these advances in some form. Sadly, the even more productive feature of macros has been abandoned, but as we'll see in future posts, their power can be harnessed to an even greater extent in the world of declarative metadata.

Software Components and Layers: Problems with Data

Components and layers are supposed to make software programs easier to create and maintain. They are supposed to make programmers more productive and reduce errors. It’s all lies and propaganda! In fact, the vast majority of software architectures based on components (ranging from tiny objects to large components) and/or layers (UI, server, storage and more) lead to massive duplication and added value-destroying work. The fact that these insane methods continue to be taught in nearly all academic environments and required by nearly all mainstream software development groups is cornerstone evidence that Computer Science not only isn’t a science, it resembles a deranged religious cult.

Components and Layers

It has been natural for a long time to describe the different “chunks” of software that work together to deliver results for users in dimensional terms.

First there’s the vertical dimension, the layers. Software that is “closer” to end users (real people) is thought of as being on “top,” while the closer software is to long-term data storage and farther from end users is thought of as being on the “bottom” of a software system. In software that has a user interface, the different depths are thought of as “layers,” with the user interface the top layer, the server code the middle layer and the storage code the bottom layer. Sometimes a system like this is called a “three tier” system, evolved from a two tier “client/server” system.

Second there’s the horizontal dimension, the components. These are bodies of code that are organized by some principle to break the code up into pieces for various reasons. I've given a detailed description of components. A component may have multiple layers in it or it may operate as a single layer.

Layers are often written in different languages and using different tools. Javascript is prominent in user interface layers, while some stored procedure language is often used in the lowest data access layer. The server-side layer may be written in any of dozens of languages, including java and python. Sometimes there are multiple layers in server-resident software, with a scripting language like PHP or javascript used for server code on the “top,” communicating with the client software.

Components are self-contained bodies of software that communicate with other components by some kind of formal means. Formal components always have data that can only be accessed by the code in the component. The smallest kind of component is an object, as in object-oriented languages. The data members of the object can only be accessed by routines attached to the object, known as methods.  Methods can normally be called by methods of other objects. Larger components are often called microservices or services. These are usually designed so that they could run on separate machines, with calling methods that can span machines, like old RPC’s or more modern RESTful API’s. Sometimes components are designed so that instead of being directly called, they take units of work from a queue or service bus, and send results out in the same way. When a component makes a call, it calls a specific component. When a component puts work onto a queue or bus, it has no knowledge or control over which component takes the work from the queue or bus.

Layer and components are often combined. For example, microservice components often have multiple layers, frequently including a storage layer. With a thorough object-oriented approach, there could be multiple objects inside each layer that makes up a service component.

Why are there layers and components? Software certainly doesn’t have to use these mechanisms to be effective. In fact, major systems and tools have been built and widely deployed that mostly ignore the concepts of layers and components! See this for historic examples.

The core arguments for layers and components typically revolve around limiting the amount of code (components) and the variety of technologies (layers) a programmer has to know about to make the job easier, along with minimizing errors and maximizing teamwork and productivity. I have analyzed these claims here and here.

Data Definitions in Components and Layers

All software consists of both procedures (instructions, actions) and data. Each of those needs to be defined in a program. When data is defined for a procedure to act upon, the data definitions are nearly always specific to and contained within the component and/or layer they’re part of. For the general concept and variations on how instructions and data relate to each other, see this.

Thinking about data that’s used in a component or layer, some of the data will be truly only for use by that component or layer. But nearly any application you can imagine has data that is used by many components and layers. This necessity is painful to object/component purists who go to great lengths to avoid it. But when a piece of data like “application date” is needed, it will nearly always have to be used in multiple layers: user interface, server and database. To be used it must be defined. So it will be defined in each layer, typically a minimum of three times!

When data is defined in software, it always has a name used by procedures to read or change it. It nearly always also has a data type, like character or integer. The way most languages define data that’s it! But there’s more.

• When the “application date” data is shown to the user in the UI layer, it typically also needs a label, some formatting information (month, day, year), error checking (was 32 entered into the day part?) and an error message to be displayed.
• When application date is used in the server layer by some component, some kind of error checking is often needed to protect against disaster if a mistaken value is sent by another component.
• When a field like social security number is used in the storage layer, sanity checks are often applied to make sure, for example, that the SSN entered matches the one previously stored for the particular customer already stored in the database.
• There need to be error codes produced if data is presented that is wrong. When the user makes an error, you can’t use a code, you have to use a readable message, which the user layer might need to look up based on the error code it gets from another component or layer.
• Each language has its own standards for defining data and the attributes associated with it. Someone has to make sure that all the definitions in varying languages match up, and that when changes are made they are made correctly everywhere.
• If the data is sent from one component to another, more definitions have to be made: the procedure that gets the data, puts it into a message and sends it, possibly on a queue; the procedure that gets the message, maybe from a queue and sends the data to the routine that will actually do the processing.
• When data is sent between components, various forms of error checking and error return processing must also be implemented for the data to protect against the problems caused by bad data being passed between components that, for example, might have been implemented and maintained by separate groups. Sometimes this is formalized into "contracts" between data-interchanging components/layers.

So what did we gain by breaking everything up into components and layers? A multiplication of redundant data definitions containing different information, expressed in different ways! What we “gained” by all those components and layers was a profusion of data definitions! The profusion is multiplied by the need for components and layers to pass data among themselves for processing.The profusion can’t be avoided and can only be reduced by introducing further complexity and overhead into the component and layer definitions.

See this for another take on this subject.

I’ve heard the argument that unifying data definitions makes things harder for the specialists that often dominate software organizations. The database specialists are guardians of the database, and make sure everything about it is handled in the right way. The user interface specialists keep the database specialists away from their protected domain, because if they meddled the users wouldn’t enjoy the high quality interfaces they’ve come to expect. There is no doubt you want people to know their stuff. But none of this is really that hard – channeling programmers into narrow specialties is one of the many things that leads to dysfunction. Programmers produce the best results by thoroughly understanding their partners and consumers, which can only be done by spending time working in different roles – for example spending time in sales, customer service and different sub-departments of the programming group.

Data Access in Components and Layers

Now we’ve got data defined in many components and layers. In a truly simple system, data would be defined exactly once, be read into memory and be accessed in the single location in which it resides by whatever code needs it for any reason. If the code needs to interact with the user, perform calculations or store it, the code would simply reference the piece of data in its globally accessible named memory location and have at it. Fast and simple.

If this concept sounds familiar, you may have heard of it in the world of relational DBMS’s. It’s the bedrock concept of having a “normalized” schema definition, in which each unique piece of data is stored in exactly one place. A database that isn’t normalized is asking for trouble, just like the way that customer name and address are frequently stored in different places in various pieces of enterprise software that evolved over time or were jammed together by corporate mergers.

Components and layers performing operations on global data is neither fast nor simple. Suppose for example that you’ve got a new financial transaction that you want to process against a customer’s account. In an object system, the customer account and financial transaction would be different objects. That’s OK, except that the customer account master probably has a field that is the sum of all the transactions that have taken place in the recent time period.

In a sensible system, adding the transaction amount to a customer transaction total in the customer record would probably be a single statement that referenced each piece of data (the transaction amount and the transactions total) directly by name. Simple, fast and understandable.

In a component/object system that’s properly separated, transaction processing might be handled in one component and account master maintenance in another. In that case, highly expensive and non-obvious remote component references would have to be made, or a copy of the transaction placed on an external queue. In an object system, a method of the transaction object would have to be called and then a method of the account master object.

It’s a good thing we’ve got smart, educated Computer Scientists arranging things so that programmers do things the right way and don’t make mistakes, isn’t it?

Conclusion

With the minor exception of temporary local variables, nearly every layer or component you break a body of software into leads to a multiplication of redundant, partly overlapping data definitions expressed in different languages -- each of which has to be 100% in agreement to avoid error. The communication by call or message between layers and components to send and receive data increases the multiplication of data definitions and references. Adding the discipline of error checking is a further multiplication.

Not only does each layer and component multiply the work and the chances of error, each simple-seeming change to a data definition results in a nightmare of redundancy. You have to find each and every place the data is defined, used or error-checked and make the right change in the right language. Components and layers are the enemy of Occamality. Software spends the VAST majority of its life being changed! Increasingly scattered data definitions make the thing that is done 99% of the time to software vastly harder and riskier to do.

The catastrophic effect of bad metaphors in software

Software is invisible. The vast majority of people who talk about it have little or no practical experience writing it. But software is incredibly important and people need to be able to talk about it: creating it, modifying it, installing and using it and making it work as intended. How do you talk about something that's invisible? Answer: you use real-world metaphors that everyone knows and talk in those terms. Given the importance of the process of creating and modifying software, you also use real-world metaphors for the processes, methods and techniques that people use for those things on software.

Given the high cost, long time periods and risky outcomes of software development, real-world metaphors are used as models for how to create what seem like the equivalent things in the invisible world. Software professionals use the same metaphors and think in the same terms as their non-programming counterparts, the more so as advancement in software normally correlates with doing less hands-on programming.

Before long, the metaphors turn into highly detailed, structured things embodied in software tools for software management, and regulations for how software must be built in order to be acceptable for acquisition by governments and major enterprises. The software in medical devices, for example, is highly regulated in this way, everything from requirements to testing and modifications.

These metaphors are great for communication among people involved in software in some way, but disastrous for building fast, efficient software, quickly and effectively. The metaphors are largely to blame for the rolling crisis/disaster of software that was first recognized over 50 years ago.

Small groups of smart, hard-working software nerds sometimes break out of metaphor prison and use techniques that naturally grow out of the experience of programmers who live in the land of software, which, though invisible to the vast majority of people, is vividly visible to them. They are usually simple ideas that come out of the writing real-world software that meets needs and evolves with experience. Top programmers often think of themselves as "lazy," meaning that they are aware of how they spend their time and find ways to avoid doing things that don't advance the cause. They visualize the world of software they're working on, always looking for ways to spend less time writing less code to get more stuff done without errors.

The methods that they use rarely conform to any of the established software methodologies and typically make non-software people uncomfortable. I've observed and described many of those techniques, which as a whole I term "wartime software."

In this post I will identify and summarize some of the widely accepted metaphors that continue to wreak havoc on software development

The factory metaphor

The metaphor of the software factory tends to float around. Wouldn’t it be nice if we could churn out software as reliably and predictably as a factory? We’d put the requirements and other goals and constraints into the factory, and after a series of careful, orderly processing steps, out would come a wonderful piece of software! Unfortunately, factories make copies of things that are already designed and tested. All the effort goes in to making the design of a new car or drug; sometimes the design is no good and has to be discarded. Once you’ve made test versions and made sure they operate the way you planned, you turn the design over to the factory for making lots of identical copies of it. See this for more.

The building metaphor

Buildings and roads are widely used as metaphors for software. Lots of the words are the same: we have requirements, architects, standards that must be met, common design patterns. We have project plans with resource requirements and target dates. We might put the project out to bid. We use specialists for each aspect of the work. We build in phases and have regular reviews by the owners and quality inspectors. At the end we have walk-throughs, which may result in punch lists of defects to be corrected. Then we have the closing and we’re off and running!

It’s a powerful metaphor. It’s better than the factory metaphor because each house is uniquely built by a team, just like each piece of software is uniquely built by a team. The trouble is that a house just sits there. It’s a passive physical object. Software is NOT a physical object; it’s mostly is a set of actions to be taken by a computer, along with some built-in data. See this for more detail.

There's another issue with the metaphor. Buildings are built and then mostly used. Modifications are sometimes made, but they are normally small compared to the effort put into the original building. Software, on the other hand, is mostly changed. Vastly more effort is typically put into changing a body of software over time than was spent with the original building. Most of what most employees of software departments do is make changes. See this for more detail.

The Architect metaphor

The architect metaphor is closely related to the building metaphor. It's just as bad for the same reasons. Here is a detailed description that uses architecture software to illustrate the difference.

Waterfall and Agile

The waterfall method is the universally derided classic method of applying project management to building software. After years of failure with increasingly rigorous and detailed project planning to improve software development outcomes, it was supposed that waterfall just wasn't flexible enough. It was too rigid. More agility to respond to emerging knowledge and conditions would solve the problems. Thus, Agile software development method was invented, purposely built on the metaphor of a nimble-footed engineering team going through the process with lots of re-calibration and obstacle avoidance.

Hah. Nothing has changed except the rhetoric. Waterfall is a method that fixes the requirements and figures out the time to build them. Agile is a method that fixes the time periods and figures out what can be built in each. The difference in practice is that Agile has more overhead, and no better results. See this for more.

Objects in software

Could there be a more inappropriate metaphor? A bundle of data definitions and code is an "object?" A thing? Sorry, programs are NOT things. They're not passive; programs are actions!

Object-orientation started as an obscure design pattern for creating simulation programs, grew into a fad and has evolved into a standard part of software orthodoxy. I remember long ago hearing the sales pitch for the early O-O languages like Smalltalk. The business sales pitch was that it would greatly enhance the speed and quality of software development, while enabling changes to be more easily accomplished. The tech pitch was that by creating "objects" consisting of a definition of a related set of data and by including in that definition all the code that would read and write the data, concerns would be isolated and mistakes avoided by having data manipulated only by "its own" code.

The common metaphor was that objects were like Lego blocks, lending themselves to easily build elaborate structures and making changes to them without collateral damage. By throwing in things like inheritance -- which was always illustrated by physical things like the hierarchy of species, families, etc. -- the Lego effect was even stronger.

As it turned out, O-O has been a giant step backwards for software. While appropriate in some situations, as a rigid structure with no exceptions it imposes difficult and unnatural constraints on how code and data can be related and arranged, making everything more complicated than it needs to be. Lego blocks are great as physical objects, but their software version is a failure.

The walls and defenses metaphor

This one is huge in cybersecurity. Everyone talks about putting up walls to protect our computers, establishing and maintaining an effective periphery. We talk about attackers who want to "penetrate" our security defenses.

A great deal of software security is built according to this metaphor. The fact that software is invisible to everyone who makes important decisions about computer security leads directly to the string of monstrous security disasters that continue to this day. Here is a good way using baseball to understand the impact of the invisibility. Here is a detailed description of the impact of invisibility.

Conclusion

We all think using metaphors. It's not inherently a bad thing. But when the metaphor has a strong real-world grounding that everyone understands, it can lead to disaster when applied to software in ways that don't match the correct understanding of the software itself. This is unfortunately the case for a great deal of software methods and processes, and helps explain the ongoing software disaster that was first recognized over 50 years ago.

 

The Dangerous Drive Towards the Goal of Software Components

Starting fairly early in the years of building software programs, some programs grew large. People found the size to be unwieldy. They looked for ways to organize the software and break it up into pieces to make it more manageable. This effort applied to the lines of code in the program and to the definitions of the data referred to by the code, and to how they were related. How do you handle blocks of data that many routines work with? How about ones whose use is very limited? The background of this issue is explained here.

These questions led to a variety of efforts that continue to this day to break a big body of software code and data definitions into pieces. The obvious approach, which continues to be used today, was simply to use files in directories. But for many people, this simple, practical approach wasn't enough. They wanted to make the walls between subsets of the code and data higher, broader and more rigid, creating what were generally called components. The various efforts to create components vary greatly. They usually create a terminology of their own, terms like “services” and “objects.” Such specialized, formulaic approaches to components are claimed to make software easier to organize, write, modify and debug. In this post I’ll take a stab at explaining the general concept of software components.

Components and kitchens

When you’re young and in your first apartment, you’ve got a kitchen that starts out empty. You’re not flush with cash, so you buy the minimum amount of cooking tools (pots, knives, etc.) and food that you need to get by. You may learn to make dishes that need additional tools, and you may move into a house with a larger kitchen that starts getting filled with the tools and ingredients you need to make your growing repertoire of dishes. You may have been haphazard about your things at the start, but as your collection grows you probably start to organize things. You may put the flour, sugar and rice on the same shelf, and probably put all the pots together. You may put ingredients that are frequently used together in the same place. You probably store all the spices and herbs together. It makes sense – if everything were scattered, you’d have a tough time remembering where each item was. The same logic applies to a home work bench and to clothes.

The same logic applies for the same reason to software! It’s a natural tendency to want to organize things in a way that makes sense – for remembering where they are, and for convenience of access. This natural urge was recognized in the early days of software programming. Given that there aren’t shelves or drawers in that invisible world, most people settled on the term “component” as the word for a related body of software. The idea was always that instead of one giant disorganized block of code, the thing would be divided into a set of components, each of which had software and data that was related.

A software program consists of a set of procedures (the actions) and a set of data definitions (what the procedures act on). Breaking up a large amount of code into related blocks was helped by the early emergence of the subroutine – a block of code that is called on to do something and then returns with a result; kind of like a mixer in a kitchen. The problems that emerged were how to break a large number of subroutines into components (each of which had multiple subroutines), and how to relate the various data definitions to the routine components. This problem resembles the one in the kitchen of organizing tools and ingredients. Do you keep all the tools separate from the ingredients, or do you store ingredients with the tools that are used to process them?

In the world of cooking, this has a simple answer. If all you do is make pancakes, you might store the flour and milk together with the mixing bowls and frying pans you use with them. But no one ever just makes pancakes – duh! And even if you did, you’d better put the milk and butter in the fridge! Ditto with spices. Nearly everyone has a spice drawer or shelf, and spices used for everything from baking to making curries are stored there.  Similarly, you store all the pans together, all the knives, etc.

In software it’s a little tougher, but not a lot. One tough subject is always do you group routines (and data) together by subject/business area or by technology area? The same choice applies to organizing programmers in a department. It makes sense for everyone who deals primarily with user interfaces to work together so they can create uniform results with minimal code. Same thing with back-end or database processing. But over time, the programmers become less responsive to the business and end users; the problem is often solved by having everyone who contributes to a business or kind of user work as a team. Responsiveness skyrockets! Before long, things begin to deteriorate on the tech side, with redundant, inconsistent data put into the database, UI elements that varying between subject areas, confusing users, etc. There’s pressure to go back to tech-centric organization. And so it goes, round and round. Answer: there is no perfect way to organize a group of programmers!. Just as painfully, there is no perfect way to organize a group of procedures and their relationship to the data they work on!!

The drive towards components

There are two major dimensions of making software into components. One dimension is putting routines into groups that are separate from each other. The second dimension is controlling which routines can access which data definitions.

Separating routines into groups can be done in a light-weight way based on convenience, similar to having different workspaces in a kitchen. In most applications there are routines that are pretty isolated from the rest and others that are more widely accessed. Enabling programmers to access any routine at any time and having access to all the source code makes things efficient.

Component-makers decided that programmers couldn't be trusted with such broad powers. They invented restrictions to keep routines strictly separate. While there were earlier versions of this idea, a couple decades ago "services" were created to hold strictly separate groups of routines. An evolved version of that concept is "micro-services." See this for an analysis.

The second major dimension of components is controlling and limiting the relationship between routines and data. The first step was taken very early. It was sensible and remains in near-universal use today: local variables. These are data definitions declared inside a routine for the private use of that routine during its operation. They are like items on a temporary worksheet, discarded when the results are created.

Later steps concerning "global" variables were less innocent. The idea was to strictly separate which routines can access which data definitions. Early implementations of this were light-weight and easily changed. Later versions built the separation into the architecture and code details. For example, each micro-service is ideally supposed to have its own private database and schema, inaccessible to the other services. This impacts how the code is designed and written, increasing the total amount of code, overhead and elapsed time.

Languages that are "object oriented" are an extreme version of the component idea. In O-O languages, each data definition ("class") can be accessed only by an associated set of routines ("methods"). This bizarre reversal of the natural relationship between code and data results in a wide variety of problems.

These ideas of components can be combined, making the overhead and restrictions even worse. People who build micro-services, for example, are likely to use O-O languages and methods to do the work. Obstacles on top of problems.

Components in the kitchen

All this software terminology can sound abstract, but the meaning and consequences can be understood using the kitchen comparison.

What components amount to is having the kitchen be broken up into separate little kitchenettes, each surrounded by windowless walls. There are two openings in each kitchenette's walls, one for inputs and one for outputs. No chef can see or hear any other chef. The only interaction permitted is by sending and receiving packages. Each package has an address of some kind, along with things the sending chef wants the receiving chef to work on. For example, the sending chef may mix together some dry ingredients and send them to another chef who adds liquid and then mixes or kneads the dough as requested. The mixing chef might then put the dough into another package and return it to the sending chef, who might put together another package and send it to the baking chef who controls the oven.

If the components are built as services, the little kitchenettes are built as free-standing buildings. In one version of components (direct RESTful calls), there is a messenger who stands waiting at each chef's output window. When the messenger receives a package, the messenger reads the address, jumps into his delivery van with the package and drives off to the local depot and drops off the package; another driver grabs the package, loads it into his van and delivers it to the receiver's input window. If the kitchenettes are all in the same location the vans and depot are still used -- the idea is that a genius administrator can change the location of the kitchenettes at any time and things will remain the same.

Another version of components is based around an Enterprise Service Bus (ESB). This is similar to the vans and the central office depot except that the packages are all queued up at the central location, which has lots of little storage areas. Instead of a package going right to a recipient, it's sent to one of these little central storage areas. Then, when the chef in a kitchenette is ready for more work he sends a requests to the central office, asking for the next package from a given little storage area. Then a worker grabs the oldest package, gives it to a driver who puts it in his van and delivers the package to the input window of the requesting chef.

If this sounds bizarre and lots of extra work, it's because ... it's bizarre and requires lots of extra work.

The ideal way to organize software

The ideal way to break software into components is actually pretty similar to the way good kitchens are organized. Generally speaking, you start by having the same kinds of things together. You probably store all the pots and pans together, sorted in a way that makes sense depending on how you use them. You probably pick a place that’s near the stove where they’ll probably be used – maybe even hanging on hooks from the ceiling. You probably have a main store of widely used ingredients like salt, but you may periodically put containers of it near where it is most often used. An important principle is that most chefs work at or near their stations – but (it goes without saying) can move anywhere to get anything they need. There aren't walls stopping you, and you don’t bother someone else to get something for you when you can more easily do it yourself.

Exactly the same principle applies whether you are creating an appetizer, an entree, a side dish or a dessert -- you do the same gathering and assembly from the ingredients in the kitchen, but deliver the results on different plates at different times.

In software this means that while routines may be stored in files in different directories for convenience, and that usually your work is confined to a single directory, you go wherever you need to in order to do your job. Same thing with data definitions; you can break them up if it seems to make things more organized, but any routine can access any data definition it needs to. When you’re done, you make a build of the system and try it out. That's what champion/challenger QA is for.

Are you deploying on a system that has separate UI, server and storage? No problem! That's what build scripts (which you would have anyway) are for! This approach makes it easier to migrate from the decades-old DBMS-centric systems design and move towards a document-centered database with auto-replication to a DBMS for reporting, which typically improves both performance and simplicity by many whole-number factors.

Are the chefs in your kitchen having trouble handling all the work in a timely way? In the software world you might think of making a "scalable architecture" with separate little kitchenettes, delivery vans, etc. -- which any sensible person knows adds trouble and work and makes every request take longer from receipt to ultimate delivery. In the world of kitchens (and sensible software people) you might add another mixer or two, install a couple more ovens and hire another chef or two, everyone communicating and working in parallel, and churning out more excellent food quickly, with low overhead.

If you think this sounds simple, you’re right. If you think it sounds simplistic, perhaps you should think about this: any of the artificial, elaborate, rigid prescriptions for organizing software necessarily involves lots of overhead for people and computers in designing, writing, deploying, testing and running software. Each restriction is like telling a chef that under no circumstances is he allowed to access this shelf of ingredients or use that tool when the need arises.

Arguments to the contrary are never backed with facts or real-world experiments – just theory and religious fervor. Instead, you should consider the effective methods of optimizing a body of software, which are centered on Occamality (elimination of redundancy) and increasing abstraction (increasingly using metadata instead of imperative code). Both of these things will directly address the evils that elaborate component methods are supposed to cure but don't.

Fundamental Concept: Software is not a passive thing, it is actions

Even lay people have heard that there are computer languages. They've heard that the languages process data -- they take data in and put it out. They gather there are commands in the language to do stuff to data. So of course software is actions.

In spite of this knowledge, something very different is in the forefront of peoples' thinking about software, both in the industry and not. This different way of thinking results from the fact that software is invisible to nearly everyone, and that in order to think and communicate about it, language is used that assumes software is a thing.

We talk about a body of software, the 2021 version of a piece of software, as though it were a car. Speaking of cars, we talk about how software crashes. We talk about how software has bugs, a phrase that originated from a bug found in a long-ago computer that had broken.

We talk about creating requirements for a new piece of software that someone will architect and the building may be put out to bid. The rest of the software building process sounds remarkably like building a house. We think of a time when the building project is done and we can start using the house.

The trouble is that a house just sits there. It’s a passive physical object. Software is NOT a physical object; it’s mostly is a set of actions to be taken by a computer, along with some built-in data. A piece of software is like a data factory – it takes data in, makes all sorts of changes, sometimes puts some of it in a storage room, and spits out data in all sorts of volumes and formats. A software program responds to each piece of data it gets, e.g. mouse click or data file; software is an actor, while a house is a thing that does nothing. Designing a piece of software is more like writing a play for an improv theater – instead of having a fixed script, the play can go in all sorts of directions depending on the input it gets from the audience. The improv theater script even has to tell the actor that when an audience member uses a swear word, the actor has to respond saying that those words aren’t permitted, would he like to try again. This is NOTHING like designing a house! Here is more detail.

Suppose you use an advanced architecture program and design a house in great detail. Modern programs go down to the level of the framing, the details of the steps, doors and windows, literally every object that will be part of the eventual house. When you're done with such a design, you can give it to the building department for approval and to a construction company to build. If they follow the instructions correctly, you'll get exactly what you saw in the 3-D renderings and the live-action walk-throughs. Here's the killer: the equivalent of such a detailed design for computer software is ... a computer program! Or at least enough detail in pseudo-code, data definitions and screen designs so that coding is fairly mechanical. A detailed design for a computer program is like the results of extensive interactions by the script-writing team for a new episode of a series TV show, with recordings and lots of notes. Chances are a junior script-writer who has been intensely following everything with some participation is assigned the job of producing a clean draft of the script, ready for refinement and editing. That is the program, ready for actors to give it a studio reading, like you would run a test version of the software for the developers.

Software isn't a thing, regardless of how often we talk about it as though it were. It isn't a thing, even though we nearly always use methods like project management developed for building things for organizing its creation. Software is a collection of actions, like an incredibly detailed script for improvisational theater with loads of conditional branching and the ability to bring things up that were talked about before in creative and useful ways.

This matters because how you think influences what you do. If you're thinking about running a marathon while you're climbing a mountain, chances are you'll slip and fall. If you're thinking about baseball plays while you're trying to choreograph a ballet, I suspect you won't win any awards. But at least those are thinking about actions. If you're thinking about the design of a book while you're writing a comedy, I doubt you'll get many laughs.

This fundamental concept is a core aspect of all of programming. The fact that people so often have the wrong things in their heads while they are dealing with software has profound effects.

And ... just as bad, we mostly don't "build" software -- mostly we change it!

The Relationship between Data and Instructions in Software

The relationship between data and instructions is one of those bedrock concepts in software that is somehow never explicitly stated or discussed. While every computer program has instructions (organized as routines or subroutines) and data, the details of how the data is identified, named and accessed vary among programming languages and software architectures. Those differences of detail have major consequences. That’s why understanding the underlying principles is so important.

Programmers argue passionately about the supposed virtues or defects of various  software architectural approaches and languages, but do so largely without reference to the underlying concepts. Only by understanding the basic concepts of data/instruction relationships can you understand the consequences of the differences.

The professional kitchen

One way to understand the relation between instructions (actions) and data is to compare it to something we can all visualize. An appropriate comparison with a program is a professional chef’s kitchen with working cooks. But in this kitchen, the cooks are a bit odd -- only one of them is active at a time. When someone asks them to do something they get active, each performing its own specialty. A cook may ask another cook for help, giving the cook things or directing them to places, and getting the results back. The cooks are like subroutines in that they get called on to do things, often with specific instructions like “medium rare.” The cook processes the “call” by moving around the kitchen to fetch ingredients and tools, brings them to a work area to process the ingredients (data) with the tools, and then delivers the results for further work or to the server who put in the order. The ingredients (data) can be in long-term storage, working storage available to multiple chefs or in a workspace undergoing prep or cooking. In addition, food (data) is passed to chefs and returned by them.

The action starts when a server gives an order to the kitchen for processing.

• This is like calling a program or a subroutine. Subroutines take data as calling parameters, which are like the items from the menu written on the order to the kitchen.

The person who receives the order breaks the work into pieces, giving the pieces to different specialists, each of whom does his work and returns the results. Unlike in a kitchen, only one cook is active at a time. One order might go to the meat chef and another to whoever handles vegetables.

• This is like the first subroutine calling other subroutines, giving each one the specifics of the data it is supposed to process. The meat subroutine would be told the kind and cut of meat, the finish, etc.

In a professional kitchen there is lots of pre-processing done before any order is taken. Chefs go to storage areas and bring back ingredients to their work areas. They may prepare sauces or dough so that everything is mixed in and prepped so that it can be finished in a short amount of time. They put the results of their work in nearby shelves or buckets for easy access later in the shift.

• This is like getting data from storage, processing it and putting the results in what is called static or working storage, which is accessible by many different subroutines.

There is a storage area and refrigerator that stores meat and another that stores vegetables. The vegetable area might have shelves and bins. The cook goes to the storage area and brings the required ingredients back to the cook’s work space. Depending on the recipe, the cook may also fetch some of the partly prepared things like sauces, often prepared by others, to include.

• This is like getting data from long-term storage and from working storage and bringing it to automatic or local variables just for this piece of work.

The storage area could be nearby. It could be a closet with shelves containing big boxes that have jars and containers in it. A cook is in charge of keeping the pantry full. They go off and get needed ingredients and put them in the appropriate storage area as needed. They could also deliver them as requested right to a chef.

• This is like having long-term storage and access to it completely integrated with the language, or having it be a separate service that needs to be called in a special way.

The chef does the work on the ingredients to prepare the result.

• This is like performing manipulations on the data that is in local variables until the desired result has been produced. In the course of this, a chef may need to reach out and grab some ingredient from a nearby shelf.

The chef may need extra space for a large project. He grabs some empty shelves from the storage area and uses them to store things that are in progress, like dough that needs time to rise. later a chef might call out “grab me the next piece of dough” or “I need the dough on the right end of the third shelf.

• This is like taking empty space and using it. Pointers are sometimes used to reference the data, or object ID’s in O-O systems.

The cook delivers the result for plating and delivery.

• This is like producing a return variable. It may also involve writing data to long-term storage or working storage.

I’m not a cooking professional, but I gather that the work in professional kitchens and how they’re organized has evolved towards producing the best results in the least amount of elapsed time and total effort. As much prep work as done before orders are received to minimize the time and work to deliver orders quickly and well. The chefs have organized work and storage spaces to handle original ingredients (meat, spices, flour, etc.) and partly done results (for example, a restaurant can’t wait the 45 minutes it might take to cook brown rice from scratch).

In the next section, this is all described again somewhat more technically. If you’re interested in technology or have a programming background, by all means read it. The main points of this post and the ones that follow can be understood without it.

Instructions and data in a computer program

The essence of a computer program is instructions that the computer executes. Most of the instructions reference data in some way – getting data, manipulating it, storing results. See this for more.

In math, from algebra on up, variables simply appear in equations. In computer software, every variable that appears in a statement must be defined as part of the program. For example, a simple statement like

          X = Y+1

Means “read the value stored in the location whose name is Y, add the number 1 to it, and store the result in the location whose name is X.” Given this meaning, X and Y need to be defined. How and where does this happen? There are several main options:

• Parameters. These form part of the definition of a subroutine. When calling a subroutine, you include the variables you want the subroutine to process. These are each named.
• Return value. In many languages, a called routine can return a value, which is defined as part of the subroutine.
• Automatic or local variables. These are normally defined at the start of a subroutine definition. They are created when the subroutine starts, used by statements of the subroutine and discarded when the subroutine exits.
• Static or working storage variables. These are normally defined separately (outside of) subroutines. They are assigned storage at the start of the whole program (which may have many subroutines), and discarded at the end.
• Allocated variables. Memory for these is allocated by a subroutine call in the course of executing a program. Many instances of such allocated variables may be created, each distinguished by an ID or memory pointer.
• File, database or persisting variables. These are variables that exist independent of any program. They are typically stored in a file system or DBMS. Some software languages support these definitions being included as part of a program, while others do not. See this for more.

There are a couple concepts that apply to many of the places and ways variables can be defined.

• Grouping. Groups of variables can be in an ordered list, sometimes with a nesting hierarchy. This is like the classic Hollerith card: you would have a high-level definition for the whole card and then a list of the variables that would appear on the card.
• There might be subgroups; for example, start-date could be the name of a group consisting of the variables day, month, year.
• Referring to such a variable might look like “year IN start-date IN cust-record” in COBOL, while in other languages it might be year.start-date.cust-record.
• Multiples. Any variable or group can be declared to be an array, for example the variable DAY could be made an array of 365, so there’s one value per day of a year.
• Types or templates. Many languages let you define a template or type for an individual variable or group. When you define a new variable with a new name like Y, you could say it’s a variable of type X, which then uses the attributes of X to define Y.
• Definition scope. Parameters, return values and local variables are always tied to the subroutine of which they are a part. They are “invisible” outside the subroutine. The other variables, depending on the language, may be made “visible” to some or all of a program’s subroutines. Exactly how widely visible data definitions are is the subject of huge dispute, and is at the core of things like components, services and layers.
  

When you look at a statement like X = Y+1, exactly how and where X and Y are defined isn’t mentioned. X could be a parameter, a local variable or defined outside of the subroutine in which the statement appears. Part of the job of the programmer is to name and organize the data definitions in a clean and sensible way.

The variety of data-instruction organization and relationships

Most of the possibilities for defining variables listed above were provided for by the early languages FORTRAN, COBOL and C, each of which remains in widespread use. Not long after these languages were established, variations were introduced. Software languages and architectures were created that selected and arranged the way instructions related to data definition. Programmers and academics decided that some ways of referencing and organizing data were error-prone and introduced restrictions that were intended to reduce the number of errors that programmers made when creating programs. In software architecture, the idea arose that all of a program's subroutines should be organized into separate groups, usually called "components" or "services," each with its own collection of data definitions. The different components call on each other or send messages to ask for help and get results, but can only directly operate on data that is defined as part of the component.

The most extreme variation of instruction/data relationship is a complete reversal of point of view. The view I've described here is "procedural," which means everything is centered around the actor, the chef who does things. The reversal of that point of view is "object-oriented," called O-O, which organizes everything around the data, the acted upon, the ingredients and workspaces in a kitchen. Instead of following the chef around as he gets and operates on ingredients (data), we look at the data, called objects, each of which has little actors assigned to it, mini-chefs, that can send messages for help, but can only work on their own little part of the world. It's hard to imagine!

The basic idea is simple: instead of having a master chef or ones with broad specialties like desserts, there are a host of mini-chefs called "methods," each of  which can only work on a specific small group of ingredients. A master chef has to know so much -- he might make a mistake! By having a mini-chef who is 100% dedicated to dough, and never letting anyone else create the dough, we can protect against bad chefs (programmers) and make sure the dough is always perfect! Hooray! Or at least that's the theory...

Conclusion

Computers take data in, process it and write data out. Inside the computer there are instructions and data. Software languages have evolved to make it easier for programmers to define the data that is read and created and to make it easier to write the lines of code that refer to the data and manipulate it. As bodies of software have grown, people have created ways to organize the data that a computer works on, for example putting definitions for the in-process data of a subroutine inside the subroutine itself or collecting a group of related subroutines into a self-contained group or component with data that only it can work on.

Understanding the basic concepts of instruction/data relationships and how those relationships can be organized and controlled is the key to understanding the plethora of approaches to language and architecture that have been created, and making informed decisions about which language and architecture is best for a given problem. The overall trend is clear: Programming self-declared elites decide that this or that restriction should be placed on which variables can be accessed by which instructions in which way, with the goal of reducing the errors made by normal programming riff-raff. Nearly all such restrictions make things worse!

Fundamental Concept: Software is rarely Built, mostly it is Changed

Nearly everything we hear about the construction of software is that it is built. This is a fundamental misconception that pervades academia and practice. This misconception is a major underlying cause of why software has so many problems getting created in a timely way and functioning with acceptable speed and quality.

The software-is-built misconception is woven into every aspect of software education and practice. You are first taught how to write software programs; all the exercises start with a blank screen or sheet of paper. You’re taught how to test and run them. In software engineering you’re taught how to perform project management, from requirements to deployment. You may learn how to automate software quality testing. Everything about these processes assumes that software is built, like a house or a car.

The idea that software is built isn’t wrong – it just applies to a tiny fraction of the total effort that goes in to a body of software from birth to its final resting place in the great bit-bucket in the sky. The vast majority of the effort goes into changing software – making it do something different than the version you’re working on does. Maybe you’re fixing a bug, maybe you’re adding a feature or adapting to a new environment. Sometimes the changes are extensive, like when you’re changing to a new UI framework. For a typical  body of software, however much effort is put into the first production version, a minimum of 10X and often over 100X of the work goes into changing it.

We don't think of software as ever-changing, ever-growing partly because we think in terms of physical metaphors. We use the same project management techniques that are used for houses that we use for software. While sometimes houses are changed, mostly they're just built and then used for years. It's an extremely rare house that is endlessly extended the way software is. See this for more.

When it comes to enterprise software, the fraction of effort that goes into change sky rockets even more. A good example is the kind of software that runs hospitals and health systems. The software already exists, and it’s already had 100X the effort put into change than the original building. For example, standard software from the leading industry supplier was installed in the NYC hospital system. The original contract -- just to buy and install existing production software -- was for over $300M and 4 years! A couple years later the carefully planned project cost more than doubled. All because of changes. Here are details.

Why does thinking software is built rather than changed matter?

The idea that software is about building and not changing is responsible for most of the processes, procedures and methods that are overwhelmingly common practice either by custom or by regulation. The mismatch between the practices and the fact that software is nearly always changed instead of being created in the first place results in the never-ending nightmare of software disasters that plague us. The never-ending flow of new tools, languages and methods are futile attempts to solve these problems. No one would find the new things attractive if there weren’t a problem to fixed – a problem everyone knows and that no one wants to explicitly discuss!

Example: Project Management, Architecture, Design and Coding

Mostly what happens when you get a job in software is that you join a group that works with a substantial existing body of code. You may be a project manager, a designer, a customizer, any number of things, all of which revolve around the body of code.

Yes, you have to start any project from an understanding of what you want to accomplish. Here's the thing: what you want to get done is in the context of the existing code. What this means is that you have to understand as much of the code as possible to do things. If what you want to do is fairly circumscribed in the existing code, you may get away with ignoring the rest -- which is where comprehensive, nothing-left-out regression testing comes in, which means better make darn sure you didn't break anything.

In this context, lots of skills are relevant. But the overwhelming number one thing is ... knowledge of the ins and outs of the existing code base!

I've worked in a couple places where there were large code bases of this kind. Forget about departmental hierarchy -- the go-to person in every case of doing anything important was the person who best knew that particular aspect or section of the code.

In this context, what's the point of all the fancy project management stuff people think is the cornerstone of effective, predictable work? When you're traveling through an unknown land with lots of mountains, jungles, rivers and swamps, what do you want above all else (besides nice paved roads that don't exist)? You want an experienced GUIDE, someone who's been all over the place and knows the best way to get there from here.

Kind of turns all the project management stuff on its head, doesn't it?

Example: Software QA

Most software QA revolves around writing test scripts of one kind or another that mirror the new code you write: if the code does X, the test should assure that the code does X correctly. This is true whether you're doing test-driven development or whether QA people write scripts. As you build the code, you're supposed to build the test code. When you make a new release, you run integration tests that include running all the individual test scripts.

Everyone with experience knows the problems: there are never enough tests, they don't cover all the conditions, there are omissions and errors in the tests, with the result that new releases put into production often have problems.

The root cause of this is the never-spoken assumption that software is about building, when the reality is that it's mostly changed. What this means is that the focus should be on the existing code and what it does -- do the changes that you made break anything that used to work? When you make changes, you should be thinking champion/challenger like in data science: the new code is the challenger: did it make things better, and did it (probably unintentionally) make anything worse? The best way to do this is to run the same data through the production program and the challenger program in parallel, and have a tool that compares the output. Looking at the output will tell you what you want to know. Even better, running the new code in live-parallel-production, with lots of real customer data going through both versions but only the current champion output going to the customer provides a safe way to see if the customers are OK, while testing out the new functionality on the champion code only to make sure it does what you wanted.

Who needs test scripts, TDD and the rest? They're backwards! See this for more.

Conclusion

Software is literally invisible to the vast majority of people. Therefore, we all talk and think about software in terms of metaphors. That would be OK if the metaphors matched reality; unfortunately, the metaphors don’t match reality, and play a major role in software dysfunction. In real life, we build houses, bridges, and other structures -- we mostly don't change them.

So what are the methods that are appropriate when you recognize that the main thing you do with software is change it instead of building it? I've done my best to abstract the methods I've seen the best people use, calling them (for want of a better term) war-time software. Instead of creating a whole adult human at once, like Frankenstein, it's like starting with a baby that can't do much yet, but is alive. The software effort is one of growing the baby instead of constructing a Frankenstein.

 

The Crisis in Software is over 50 years old

Everyone knows there’s a problem in software. No one likes to talk about it. Waves of new tools and techniques are introduced, hyped and fade away. Are there waves of new tools and techniques in accounting? During interviews, are accountants asked if they practice the currently “hot” methods? Of course not! What about cars? Hah, you might say – look at the wave of new electric cars! Yes, let’s look at them – do electric cars crash more than normal ones? If cars crashed or mis-performed at the same rate as software, we would all be afraid to ride in them!

The existence of a software crisis was first prominently identified in 1968. The results of the crisis were clear at that time, and haven’t changed a bit since then. Nothing – no wonderful new language, paradigm, project management or quality method, architectural technique or anything else has made a dent in the crisis.

The solutions to the crisis have largely been identified and repeatedly proven in practice by small groups of programmers who desperately need to get things done. Their methods are ignored or suppressed by the ruling elites in academia, government and corporations – including Big Tech. Sadly, this is unlikely to change any time soon. Happily, effective methods for building software that are ignored and suppressed provide a way for small, ambitious groups to do new stuff and win!

What is the software crisis?

The term “software crisis” was coined during the 1968 NATO Software Engineering Conference. Many computer science experts attended. There was general agreement that there was a serious problem. Part of the solution was identified by creating a discipline of “computer engineering,” which would structure and formalize the techniques for building software. Papers identifying the problems and solutions were generated as a result of the conference and a similar one held the following year.

The problems are summarized in the Wiki article:

Look familiar?

How do we know there’s a crisis?

When things are really bad, it’s typical for people to avoid talking about it. When there’s a whiff of a solution, though, it’s likely to receive attention. When people engage in something that’s likely to go wrong, most of them will get the best advice and be sure to follow widely accepted practices in order to avoid trouble. If something goes wrong anyway, having followed the standard accepted techniques provides a good defense – I did what people in my position are supposed to do! When hiring people, the hiring organization will often trumpet their use of these authorized and accepted methods to assure avoidance of blame. And finally, when things go wrong, great care is taken to assure that as few people know about the problem as possible.

When you hire someone in physics, do you advertise that the position requires knowledge and acceptance of relativity theory? When you hire a doctor, do you need to make sure that the doctor adheres to the infectious theory of disease?

We know there’s a crisis because, in spite of all the effort to keep it quiet, disasters happen that are so public, widespread and annoying that they get talked about. The trouble is that the highly publicized disasters are just the visible tip of a truly gigantic iceberg that takes up most of the space in the ocean of software.

We know there's a crisis because the academic field devoted to the subject, Computer Science, isn't a "science." Not even close. Check out these.

We know there's a crisis because the famous big tech companies have amazing reputations, like they're all geniuses -- while the facts show that they're pretentious bumblers with astounding salaries.

We know there's a problem because of the fashion-driven nature of the field. Here is a discussion of specific fashions, and see these for examples.

Conclusion

Look again at the list of problem identified over 50 years ago: software that's late, over budget, not what you wanted, poor quality, and sometimes has to be thrown out. This is one of the reasons managers take estimates from programmers and multiply the time by 3X before passing them on up the chain. Higher managers pad even more. People try to avoid publicizing the wonderful thing that's coming real soon because all too often, days before completion, it blows up.

There have been decades of widely-touted solutions to the problem that never solve the problem. Here is a review of 50 years of "progress" in programming languages, for example.

There are solutions to this 50 year problem. Here is one of those solutions. Small groups of programmers, pressured to produce great results in short period of time, re-invent the solutions. There are underlying errors of thought that are the primary factors behind the problem and its incredible persistence. How else can you understand the near-universal resistance to winning methods-- proven in practice! -- to be ignored?

 

Software programming languages: the Declarative Core of Functional Languages

In a prior post I described the long-standing impulse towards creating functional languages in software. Functional languages have been near the leading edge of software fashion since the beginning, while perpetually failing to enter the mainstream. There is nonetheless an insight at the core of functional languages which is highly valuable and probably has played a role in their continuing attractiveness to leading thinkers. When that insight is applied in the right situations in a good way it leads to tremendous practical and business value, and in fact defines a path of advancement for bodies of software written in traditional ways.

This is one of those highly abstract, abstruse issues that seems far removed from practical values. While the subject is indeed abstract and abstruse, the practical implications are far-reaching and result in huge software and business value when intelligently applied.

Declarative and Imperative

A computer program is imperative. It consists of a series of machine language instructions that are executed, one after the other, by the computer's CPU (central processing unit). Each Instruction performs some little action. It may move a piece of data, perform a calculation, compare two pieces of data, jump to another instruction, etc. You can easily imagine yourself doing it. Pick up what's in front of you, take a step forward, if the number over there is greater than zero, jump to this other location, etc. It's tedious! But every program you write in any language ends up being implemented by imperative instructions of this kind. It's how computers work. Period.

Computers operate on data.  Data is passive. Data can be created by a program, after which it's put someplace. The locations where data is held/stored are themselves passive; those locations are declared (named and described) as part of creating the imperative part of a computer program.

Data is WHAT you've got; instructions are HOW you act. WHAT is declarative; HOW is imperative. WHAT is like a map; HOW is like an ordered set of directions for getting from point A to point B on a map.

The push towards declarative

From early in the use of computers, some people saw the incredible detail involved in spelling out the set of exacting instructions required to get the computer to do something. A single bit in the wrong position can cause a computer program to fail, or worse, arrive at the wrong results. This detailed directions approach to programming is wired into how computers work. Is there a better way?

There is in fact no avoiding the imperative nature of the computer's CPU. As high level languages began to be invented that freed programmers from the tedious, error-prone  detail of programming in machine language, some people began to wonder if there were a way to write a low-level program, a program that of necessity would be imperative, that somehow enabled programs to be created in some higher level language that were declarative in nature.

Some people who were involved with early computers, nearly all with a strong background in math, proceeded to create the declarative class of programming languages, the most distinctive members of which are functional programming languages as I described in an earlier post.

The ongoing attempt to create functional languages and use them to solve the same problems for which imperative languages are used has proven to be a fruitless effort. But there are specific problem areas for which a declarative approach is well-suited and yields terrific, practical results -- so long as the declarative approach is implemented by a program written in an imperative language, creating a workbench style of system for the declarations. The current fashion of "low code" and "no code" environments is an attempt to move in that direction. But I'd like to note that there's nothing new in those movements; they're just new names for things that have been done for decades.

The Declarative approach wins: SQL

DBMS's are ubiquitous. By far the dominant language for DBMS is SQL. Data is defined in a relational DBMS by a declarative schema, defined in DDL (data definition language). Data is operated on by a few different kinds of statements such as SELECT and INSERT.

SELECT certainly sounds like a normal imperative keyword in any language, like COBOL's COMPUTE statement. But it's not. It's declarative. A SELECT statement defines WHAT data you want to select from a particular database, but says nothing about HOW to get it.

This is one of the cornerstones of value in a relational DBMS system. A SELECT statement can be complicated, referencing columns in multiple rows of various tables joined in various ways. The process of getting the selected data from the database can be tricky. Without query optimization, a key aspect of a DBMS, a query could take thousands of times longer than a modern DBMS that implements query optimization will take. Furthermore, table definitions can be altered and augmented, and so long as the data referenced in the query still exists, the SELECT statement will continue to do its job.

If all you're doing is grabbing a row from a table (like a record from an indexed file), SQL is nothing but a bunch of overhead, and you'd be better off with simple 3-GL Read statements. But the second things get complicated, your program will require many fewer lines of complex code while being easier to write and maintain if you have SQL at your disposal. A win for the declarative approach to data access, which is why, decades after it was created, SQL is in the mainstream.

The Declarative approach wins: Excel

I don't know many programmers who use Excel. Too bad for them; it's a really useful tool for many purposes, as its widespread continuing use makes clear.

Excel is a normal executable program written in an imperative language, but it implements a declarative approach to working with data. Studying Excel is a good way to understand and appreciate the paradigm.

An Excel worksheet is two dimensional matrix of values (cells). A cell can be empty or have a value of any kind (text, number, currency, etc.) entered. What's important in this context is that you can put a formula into a cell that defines its value. The formula can reference other cells, individually or by range, absolutely or relatively. A simple formula could be the sum of the values in the cells above the cell with the formula. It could be arbitrarily complex. It can have conditionals (if-then-else). Going beyond formulas, you can turn ranges of cells into a wide variety of graphs.

If you're not familiar with them, you should look at Pivot tables, and when you've absorbed them, move on to the optimization libraries that are built in, with even better ones available from third parties. Pivot tables enable you define complex summaries and extractions of ranges of cells. For example, I have an Excel worksheet in which I list each day of the year and the place where I am that day. A simple Pivot table gives me the total of days spent in each location for the year, something that simple formulas could not compute.

The key thing here is that even though there are computations, tests and complex operations taking place, it's all done declaratively. There is no ordering or flow of control. If your spreadsheet is simple, Excel updates sums (for example) when you enter or change a value in a cell. For more complex ones, you just click re-calc, and Excel figures out all the formula dependencies and evaluates them in the right order. This makes Excel a quicker way to get results than programming in any imperative language, assuming the problem you have fits the Excel paradigm.

The Declarative approach wins: React.js

One of the most widely-used frameworks for building UI's is React.js. Last time I looked, the header page of react.js included this:

It says right out that it's declarative! And that makes it easy! I have found a number of places (example, example) that have nice explanations of how it works and why it's good.

The Declarative approach wins: Compilers

The best computer-related course I took in college by far was a graduate course on the theory and construction of compilers. The approach I was taught all those decades ago remains the main method for compiler construction. First you have a lexical parser to turn the text of the program into a string of tokens. Then you have a grammar parser to turn the tokens into a structured graph of semantic objects. Then you have a code generator to turn the objects into an executable program.

The key insight is that each stage in this process is driven by a rich collection of meta-data. The meta-data contains all the details of the input language and its grammar and the output target. A good compiler is really a compiler-compiler, i.e., the imperative code that reads and acts on the lexical definitions, the grammar and the code generation tables. The beauty is that you write the compiler-compiler just once. If you want it to work on a new language, you give it the grammar of the language without changing any of the imperative code in the compiler-compiler! If you want to generate code for a new computer for a language you've already got, all you do is update the code generation tables! Once you've got such a compiler, you can write the compiler in its own language, at the start "boot-strapping" it.

While they didn't exist at the time I took the course, tools to perform these functions, YACC (Yet Another Compiler Compiler) and LEX, were built at Bell Labs by one of the groups of pioneers who created Unix and the C language. I didn't have those tools but used the concepts to build the FORTRAN compiler I wrote in 1973 while at my first post-college job. Since the only tool I had to use was assembler language, taking the meta-data, compiler-compiler approach saved me huge amounts and time and effort compared to hard-coding the compiler without meta-data.

This is meta-data at its best.

The Declarative spectrum

Excel and SQL are 100% all-in on the declarative approach. But it turns out that it doesn't have to be all one way or the other. If you're attacking an appropriate problem domain, you can start with just programming it imperatively, and then as you understand the domain, introduce an increasing amount of declaration into your program, by defining and using increasing amounts of meta-data. This is exactly what I have described as climbing up the tree of abstraction. Each piece of declarative meta-data you introduce reduces the size of the imperative program, and moves information that is likely to be changed or customized into bug-free, easily-changed meta-data.

Conclusion

Functional languages as a total alternative to imperative languages will perpetually be attractive to some programmers, particularly those of a math theory bent. Except in highly limited situations, it won't happen. No one is going to write an operating system in a functional language.

Nonetheless, the declarative approach with its emphasis on declaring facts, attributes and relationships is the powerful core of the functional approach, and can bring simplicity and power to otherwise impossibly complicated imperative programs.

The Revolutionary Champion/Challenger Method of Software QA

Having issues with software quality? Have you tried test-driven development? Do you have test scripts based on test requirements? Do you have a rich sandbox with good test data? What’s your code coverage? How often is a fully tested version released into production with problems that were missed? How often does the new feature work adequately in production but with destructive side effects that integration testing missed? How long and expensive is the pipeline from programmer’s release to working production? Are you proud of being on the forefront of development methods with Agile, Scrum and the rest but still avoid releasing what comes out of each Sprint to production because you can’t risk another disaster?

If any of these apply to you, you may want to consider a decades-old, widely proven in production method of software QA that enables frequent, error-free releases to production with almost no overhead or traditional QA work. It’s not taught, there are no certifications and It’s ignored by mainstream software experts. But it works.

I used to call it “comparison-based QA.” The term is appropriate because the core of the method is comparing results of the production version of the code with a test version. A CTO with a data science background suggested a better term, which I will use henceforth: champion/challenger software QA. It’s like when you’ve got a good model, the champion, and you want to see if a new model, the challenger, yields better results. Or it’s like A::B testing of a consumer UI, when you want to see if a proposed variation works better. In both cases, you’ve done something new and you want to do two things: (1) see if the new thing works like it should, and (2) make sure nothing that used to work is broken. Sounds like feature testing and regression testing, doesn’t it? Yup!

Nobody writes test requirements or feature tests for champion/challenger. You just feed the new thing the same data you fed the old thing and compare the results. You expect differences. If something is better and nothing is worse, you could have a winner – you test with a wider range of data, and if the results hold up, the challenger becomes the new champion and you move on.

Here are the main beauties of champion/challenger:

• The tests are nothing but inputs and saved outputs of real-world processing. Nothing artificial, no carefully crafted "test data."
• You create the code for comparison of output data once. The code/model could get endlessly complex, but the output comparison identifies differences whether it has 10KB or 10GB to compare. No test scripts!
  
  • You can do the comparison with a UI with some extra work, but again one-and-done.
• The comparison pulls out all the differences. You look to see if each difference is something you expected: if everything new you expected is there, and crucially if there are any unexpected differences. If there are, you’ve got a regression failure to fix.
• You can do the comparison in batch with large samples of old data, giving you great coverage.
  
• You can do the comparison live, in a production environment, using the challenger code/model output for comparison and sending only the champion’s output to the user.
  • You can test the changes you've made on the challenger code alone, looking at the challenger code's output, to see if you're happy with the change you made. While you're doing that, the challenger code will simultaneously be running production data, which the comparison with the champion will make sure is still OK.
    
  • After you’ve done enough live parallel production testing, you can be confident that the challenger will do everything the champion did – with real-world data in the production environment -- so you merely turn the switch so that challenger output becomes live and the old champion is retired.
  • You can do this in stages to crank the risk way down, so that the challenger becomes champion for just 10% of the inputs and then gradually turn up the fraction. This guarantees problem-free production releases.

I've written about about the details of how to get this done in a book.

I've written a post explaining the core concepts of the two main aspects of QA and how they're different and best served by different methods.

I've written about the path that led me to writing this and related books and summaries of the books.

I've written about how this method provides crucial fuel enabling startups to surpass tech groups hundreds of times their size along with specific strategies those groups use here and here.

The method is a "secret" -- because the vast majority of industry elites choose to ignore it. Want to succeed against industry giants? Use secret methods like champion/challenger to enable rapid release of quickly evolving code with zero errors in production and minimal QA effort. You'll run circles around the lumbering, stumbling giants and then leave them in your dust.

The IRS Could Have Prevented the Tax Data Leak

If the IRS had wanted to prevent the leak of tax returns recently reported by Propublica, they could have done it. The methods are simple, effective and in use. They just didn’t implement leak prevention methods. Why? The problem isn’t money; the IRS spends billions of dollars a year on computer systems. Will this embarrassment get them to fix things? I’ve read through the “IRS Integrated Modernization Business Plan,” the April 2019 document that describes how the IRS will spend many billions over the next 5 years to “modernize” their computer systems, and nowhere in the document is there a hint that they’ll do anything but spend more money to implement more of the ineffective security systems they already have.

The IRS doesn’t create or invent cybersecurity methods; they try to adhere to all the security regulations, follow the standards and take the advice of agencies that specialize in cybersecurity. These other agencies employ top experts who set the standards that institutions follow to protect their computer systems and confidential data. So what’s going on here? Did the IRS suffer the tax data leak because they failed to implement one of these clear standards? Or is there something missing or wrong with the standards that affects the IRS and all the other organizations that are guided by them? Let’s see.

Cybersecurity is a complex issue. I’ve used the metaphor of a gated community to explain general computer security; while the walls and gates of a gated community tend to be secure and well-maintained, the equivalent in the computer world is a patch-work of incompatible wall sections from different manufacturers which are never built properly and often need fixes to be applied, which the computer managers too often take months to apply if they do the work at all.

It’s possible that a hacker broke into the IRS. But what probably happened is that an IRS employee or contractor with legitimate access to IRS data decided to make a political statement by grabbing the files of ultra-wealthy Americans, smuggling them out of the agency and giving them to Propublica. This is known as an “insider” threat. Here’s the shocker: modern corporate and government cybersecurity standards and regulations fail to prevent or even detect insider threats!

Insiders stealing the data of the company or agency they work for has happened many times. The famous Edward Snowden case is a classic example of an insider stealing secret information and leaking it for publication. Snowden was a contractor who worked at the super-secret NSA (National Security Agency). He saw the surveillance of citizens that was being performed by the agency and didn’t think it was right, so he gathered lots computer files documenting the behavior and sent the files outside the agency for publication.

Snowden did electronically what Daniel Ellsberg did decades ago physically. Ellsberg was a military officer who had helped create reports describing in detail secret operations the US conducted during the Vietnam war. While working at the Top Secret RAND Corporation he gained access to a copy of the reports and walked out the door with them in his briefcase. He gave them to the press, where they were headlined as the Pentagon Papers.

The NSA has a positive reputation for cybersecurity. The cover story in Wired Magazine in June 2013 featured a description of a visit to NSA HQ in Fort Meade with its elaborate security measures. The strong impression given is that an organization that has so many strong walls, locks and cameras must be able to do the equivalent in the invisible world of computers. The timing of the cover story was perfect. Edward Snowden started leaking secret NSA documents in December 2012; the leaked documents were published shortly after the publication of the Wired Magazine issue praising the ultra-security of the NSA.

There are systemic issues that result in most of the successful hacks of governments and large companies which I describe here. What it comes down to is two main factors: the people in charge don’t understand the world of computers; the people in charge take a slow, regulatory approach to security, while the opposition is fast and creative.

For the IRS, the data loss is similar to books being taken from a library without being checked out, and can be fixed using electronic versions of methods that librarians use: check the books anyone walks out with!

Personal tax information is valuable, like the goods sold by high-end retailers. Think about jewelry stores; nearly anyone can go in the store, but all the valuable jewels are closely watched as they are taken out of display cases, tried on and put down. You don’t get away with slipping a diamond into your pocket and walking out of the store. Systems like this can be and have been implemented in the world of computers. I go into more detail here.

Going beyond basic monitoring of the behavior of computer users, it’s possible to translate methods that are in production today for catching credit card fraud to the problem of data leaks. Basically what you do is use machine learning to model everyone’s normal behavior concerning data access. When someone does something that is not normal for them, the model immediately notices and calls software to stop them and raise an alert.

In the case of the IRS the general behavior monitoring behavior could be refined, since IRS employees work on cases that have been assigned to them. The software would look at each file a user accesses and make sure that file is relevant to a case they’re working on; if not, the software would prevent access and raise an alarm. That way an errant employee who tried to pull Warren Buffet’s tax data who wasn’t specifically assigned to the case wouldn’t be allowed to do so. And the person working on Warren Buffet’s case wouldn’t be able to access Elon Musk’s case.

It’s less likely but possible that instead of the bad guy being an employee, it was a hacker who gained access to internal systems using methods similar to the ones that resulted in financial records of 147 million Americans being stolen from Equifax in 2017. I describe that hack here.

If the internal monitoring systems I have described were in place, it would also catch a person who had gotten into the IRS by hacking – the beauty of the method is that you don’t worry about who the actor is – you just worry about what they do, just like in a library or jewelry store.

The cybersecurity problem isn’t limited to giant government bureaucracies with outdated computer systems. It’s widespread, in part because they all follow experts, standards and regulations that ignore the insider threat. I analyzed in detail the various experts who were quoted in articles published by the New York Times about the Wannacry ransomware attacks based on software that had been leaked from the NSA. I found that the experts were simply wrong about the reasons, methods and responses to the attack.

It is ironic that the same government authorities who force everyone to follow ineffective regulations they craft by the ton are spending even more money training young people in their methods. My local community college was conducting training sponsored jointly by the NSA and DHS (the Department of Homeland Security); when I looked into it I found that the experts couldn’t even build functioning, secure websites with accurate information.

I sincerely hope that the ongoing flood of illegal leaks and ransomware attacks will end soon. But so long as the current batch of bureaucrats, regulators and experts are in charge of things, we’re likely to spend ever-increasing amounts of money on cybersecurity with ever-worsening results.

Note: this was originally published at Forbes.

Bitcoin Enables Criminals to Thrive While Promoters Deny the Facts

Many people believe, for good reason, that the cryptocurrency Bitcoin is widely used by criminals. The growing number of firms looking to profit from the use of Bitcoin as a legal investment don’t like being associated with crime. So they decided to form an organization and pay an ex-CIA director to lend his prestige and credibility to a report that distorts the truth and whitewashes the huge and ongoing use of Bitcoin and other cryptocurrencies as key parts of criminal enterprises.

The Crypto Council for Innovation

The IPO of crypto firm Coinbase at a valuation of about $100 billion shines a bright light on the main asset it manages, Bitcoin. Coinbase, along with other crypto firms and major financial firms such as Fidelity, have formed a trade group called Crypto Council for Innovation whose purpose is to promote the benefits and general wonders of crypto and to “encourage the responsible regulation of crypto in a way that unlocks potential and improves lives.” They also will be “…addressing misperceptions and misinformation…”

To that end, the group has sponsored a paper. Here’s their description:

In An Analysis of Bitcoin's Use in Illicit Finance, a study authored by Michael Morell, former Acting Director, Deputy Director and Director of Intelligence at the Central Intelligence Agency (CIA) examines the general assertion that the Bitcoin market is rife with illicit activity. Morell concludes that Bitcoin's use in illicit finance activity is limited and orders of magnitude lower than what has been cited by government officials. Morell's analysis also reveals that the blockchain ledger is a highly effective crime-fighting and intelligence-gathering tool.

The group is already meeting its goal of influencing public opinion. The headlines of articles include:

Embracing Bitcoin is now a matter of national security says former CIA director

Bitcoin is a ‘Boon for Surveillance’ says former CIA Director

 

How an Ex-CIA Director Proved Bitcoin Use in Crime is Declining

Former CIA Director Comes Out in Favor of Bitcoin

The pro-Bitcoin propaganda is working!

There’s the Report and then there are the Facts

Like all propaganda of its kind, the Morell report is all about starting with the conclusion you want – Bitcoin is great, criminals are fleeing from it! – and marshaling an impressive-sounding array of name-brand institutions and experts to say what you want. In this case, since the facts diverge from the desired conclusion so drastically, a good deal of work is required to reach that goal. Here are some of the major things the report did to whitewash the truth.

The report is self-serving

First and foremost, the people who wrote the report were bought and paid to do the job they did. Of course the report included words about how it was “objective.”

Think about it this way: suppose a governor of a state were accused of sexual harassment by multiple women and further accused of hiding actions leading to the deaths of thousands of seniors during the pandemic; how credible would a report be that was commissioned and paid for by that same governor? That’s what we have here.

The “experts” are nearly all anonymous

The report references experts from a wide variety of name-brand institutions who are said to support the report’s conclusions. This is an important subject. Don’t you think the report’s authors could get more than one such expert to go on the record? The one expert they got on the record has been retired for years.

Bitcoin technology is difficult for most people to understand

None of the report authors have any technical expertise. The ex-CIA man spent his early years on energy and East Asia, and then became a manager and communicator – which is what you would expect of someone who now spends lots of time working for media outlets like CBS.

The report shows little understanding of Bitcoin technology

The words used in the second conclusion are one example among many: “The blockchain ledger on which Bitcoin transactions are recorded…” The way this is worded shows a lack of the most basic knowledge of how Bitcoin works, implying that Bitcoin is somehow not part and parcel of what some call the “blockchain ledger,” a phrase made up to describe one of the inextricable parts of the Bitcoin code base. In the normal world, transactions like buying a hose at a hardware store take place; some of those real-world transactions may later be recorded in a ledger; in Bitcoin, there is no difference between the transaction, the Bitcoin and the ledger.

Why do criminals like Bitcoin?

The report claims that criminals are fleeing from Bitcoin. Let’s step back and see what it is about Bitcoin that criminals like. The reason why criminals like Bitcoin and other cryptocurrencies is simple: it’s easy for them to avoid getting caught! It’s even better than wearing a mask when robbing a bank! It’s as anonymous as cash except that it enables the transacting parties to be an ocean away from each other.

How does Bitcoin enable criminals to exchange money secretly?

It is true, as the whitewashing report claims, that the Bitcoin ledger contains a complete record of Bitcoin transactions and is open for viewing to anyone. A knowledgeable person can look at any current Bitcoin holding and trace it back to the prior owner, the owner before, and so on to the transaction that created the Bitcoin. It’s totally transparent!

There’s a little wrinkle, though, that’s the key to everything: the buyer and seller of each Bitcoin transaction are identified solely by the public side of the encryption keys controlled by the transactors. The physical-world identity of the sender and receiver of Bitcoin is not recorded in any way, shape or form! In terms of the Bitcoin ledger itself, everything is 100% anonymous.

The report mostly disputes claims few people make

In the introduction they talk about “…public statements from officials on both sides of the Atlantic who have suggested that Bitcoin is used primarily for illicit activities.” In fact the argument most often made is NOT about the fraction of Bitcoin used for criminal purposes, but about the fact that the nature of Bitcoin makes it DESIRABLE for criminals to use, and that criminals in fact make use of it. At no point do the authors argue against the fact of criminals’ preference for Bitcoin. The authors’ first major conclusion is “The broad generalizations about the use of Bitcoin in illicit finance are significantly overstated.”

The first major section of the report is “Bitcoin’s Use in Illicit Activity is Relatively Limited” The authors don’t deny that criminals like to use Bitcoin. They even admit that it is the currency most often found in Dark Net Markets, i.e., places where illegal substances and objects are bought and sold.

They argue that the fraction of Bitcoin activity performed by criminals is decreasing

As the speculative frenzy for Bitcoin buying and selling continues to grow, this is plausible, but irrelevant to the core observation that criminals like using Bitcoin and that it enables their activity.

Suppose a city suffers 100 murders per year. Then the population of the city greatly increases, but 100 people a year are still murdered. Is the fact that a decreasing fraction of the population is murdered every year something to be celebrated? Would you want to proclaim that your city’s murder rate is going down?

The report they rely on does not support their conclusion

The Chainalysis report they quote does not claim that criminal use is decreasing, but that overall use is growing: “One reason the percentage of criminal activity fell is because overall economic activity nearly tripled between 2019 and 2020”

The also report states: “However, as always, cryptocurrency remains appealing for criminals as well due primarily to its pseudonymous nature and the ease with which it allows users to send funds anywhere in the world instantly, despite its transparent and traceable design.”

The uncertainty of the Chainalysis data about criminal use is ignored

Because of the difficulty of identifying the participants in Bitcoin transactions outside of highly regulated exchanges that enforce standard bank KYC provisions, identifying which are criminal is mostly guesswork.

For example, the 2020 report identified 1.1% of 2019’s transactions to be criminal activity. In the latest 2021 report that number was revised to 2.1%, nearly double! They admit the same uncertainty about their numbers for 2020 saying “we should expect 2020’s reported criminal activity numbers to rise over time as well.”

The authors minimize the extent of the use of Bitcoin by criminals

The revised Chainalysis 2019 number shows “$21.4 billion worth of transfers” by criminals. This is no small amount! It is about the same as the worldwide total amount of money lost to credit card fraud by banks! Vastly more people use credit cards than hold Bitcoin, and the yet the total amount of crime is about the same!

The supposed use of blockchain analysis for fighting crime

The report’s second conclusion finishes with: “Put simply, blockchain analysis is a highly effective crime fighting and intelligence gathering tool.” If this is true, don’t you think the report would have included some juicy examples of crimes that had been foiled or intelligence that was gathered from Bitcoin? The authors fail to give a single example – not one of a crime that has been solved by using “blockchain analysis.” And not one example of intelligence that has been gathered. Do you think this might have something to do with the fact that exactly zero personal identity information is contained in the blockchain? You know, the reason criminals like it?

On the other hand...

Criminals are everywhere. Criminals like money. Criminals have been finding ways to exchange and launder money as long as there has been money. Fully regulated banks that enforce KYC (Know Your Customer) identification standards are still used for criminal purposes. The most recent report (from 2015) from the US Treasury estimates that about $300 billion is laundered every year in the United States. This is in spite of the massive AML (anti-money-laundering) regulations imposed on banks that cause them to produce a flood of required AML reports to the feds, who proceed to catch and stop only a small fraction of the crime. Only this year, after years of increasing regulations and costs with ongoing ineffectiveness, has the relevant agency started to take steps towards measuring effectiveness instead of just requiring “churning out more data that proves to be less than helpful” in actually catching the bad guys.

Conclusion

Bitcoin promoters are anxious to upgrade the reputation of their currency. The report they sponsored for that purpose marshals an impressive array of classic propaganda techniques to convey its misinformation.  Why not just state the facts?

The facts in this case are simple. Criminals make extensive use of our existing financial institutions. They manage to do so in spite of huge, costly efforts of banks, regulators and enforcement agencies, who end up catching only a small fraction of the crime. Criminals were early to jump on the Bitcoin bandwagon because of the anonymous, instant transactions it enabled. Criminals use Bitcoin today and are highly likely to continuing doing so, just as they continue to use existing banking mechanisms and largely escape capture. There is nothing about the “transparency” of Bitcoin that makes it easier to catch bad guys than existing systems.

I hope that the relevant organizations abandon the time-wasting report generation approach they’ve taken to finding financial crime in most areas other than credit card fraud and shift to a more entrepreneurial, results-oriented model with proper incentives to the participants. Here is the idea of the approach in general, and here’s an example of how it’s worked out in credit card fraud.

This was originally published in Forbes.

Gated Communities Help Us Understand the Ongoing Cyber-Security Disaster

There’s a war going on in our computers and networks. It’s a silent, invisible war. It’s fierce and continues to escalate.

The bad guys are winning. They are aggressive, hard-working, learning, inventing and focused on the goal of making money. The large army of the good guys is led by hapless, incompetent, unmotivated bureaucrats with meaningless certifications in this or that, consumed by building an audit trail showing that they’ve followed the ever-growing body of useless regulations so that when the nearly-inevitable security disaster happens, they can prove it wasn’t their fault. It’s clearly not a fair fight.

The security war isn’t like a war between nations. It’s more like a sprawling collection of gated communities infiltrated and attacked by myriad bands of criminal groups who break in, rob valuables and sometimes take hostages for ransom. The communities spend more money every year building walls that are higher and stronger and hiring ever more highly trained security people. Governments have multiple departments whose purpose is to stop the criminals directly and to help the communities better defend themselves.

Every year the money spent to prevent cyber-crime goes up, and every year the amount of illicit goods the criminals make off with goes up. The criminals are almost never caught. The problem is clearly not that the communities aren’t spending enough money. The problem is that the defenders don’t know much about computers and are going through the motions while the attackers are going for the gold, i.e. bitcoin.

Hardly anyone, including certified computer professionals with academic degrees, understands what goes on inside computers. They attempt to secure computers using ineffective methods that sound impressive but which they themselves don’t understand. They take great pains to do fancy-sounding things that sound impressive but make no difference.

I have explained the details of the massive hack of Equifax hacking by comparing Equifax security in the computer world to a car dealership’s security in the physical world. Translating invisible computer events to common sense physical things can help anyone understand what this cybersecurity war is all about. In this article I’ll attempt to explain what computer-style security would look like if it were applied to a gated community.

 

The Gated Community: Defenders and Attackers

The people who build the walls that protect the outer perimeter of the community are proud of their work. In some places they even have walls inside walls!

• If the walls were built like computer “walls” are built, you’d see that the walls are a patchwork of wall segments designed at different times by different vendors using very different materials and designs and are shipped with so many flaws that they need frequent upgrades. The people in charge of wall installation and maintenance rarely stay on top of the never-ending flow of walls patches and corrections and apply them haphazardly, if at all. The result is that all the savvy criminal has to do is jump on a vulnerability the second the manufacturer announces it and probe all the walls. It’s not hard – a large fraction of wall maintainers leave gaping holes unpatched for months or even years. Shazam! The criminal is inside the community.

The entrance to the gated community is a gate with a 24x7 guard checking the ID of everyone who enters against the list of people who are permitted to enter. The guard permits no exceptions.

• The people who live in gated communities want lots of people to come to their homes and do things for them. They call in and have the person added to the list. The service person comes to the gate, shows ID and is allowed to enter. Maybe they go to the house of the person who wanted them, but there’s no one to stop them going to other houses and doing whatever they feel like, just like a criminal who snuck through the flawed outer wall. Computer programs who “knock on the doors” of heavily guarded computers do the same thing, often with stolen ID’s.

The houses in the community are built securely, with locks on their doors and windows, so that even if a criminal manages to get inside the community, they can’t rob the house.

• In the computer world the houses and their exterior walls (servers, operating systems and applications) are built by the same kind of hodge-podge of vendors that build the exterior fences that protect the community. Houses are supplied by a variety of huge vendors using complicated methods and materials. It’s extremely rare that a house is installed without flaws – the builder will usually claim that it’s flawless, but then will come a stream of patches that need to be applied to the house with varying levels of urgency. A diligent clever criminal can go around probing houses for flaws that have yet to be discovered or repaired by the original manufacturer; a lazy criminal can just wait until the flaws are announced and probe specifically for the known flaws, confident that most homeowners won’t bother to apply the corrections.

People in the community want service workers to come to their homes when they’re out to do jobs like cleaning. It’s a huge convenience to have that guard at the gate checking ID to make sure only authorized people are let in. The guard can also loan the authorized person a spare key so they can enter the house and do the work they’ve been asked to do.

• In the computer world the criminal enters with a fake ID and gets a “key” which usually gives you permission to enter many “houses,” where you can do whatever you want – mess things up, steal things, etc. You can even change the “locks” and scramble things up (encrypt them) so badly that the house can’t be used. Imagine that the frying pan is stored under the hats in the coat closet and all the cooking knives scattered inside pieces of clothing in every room. How would you feel about cooking?

As in many high-class gated communities, mail service to the house is provided. Of course, only an authorized mail person is let in the gate in his truck that contains all the mail and packages to be delivered.

• In the computer world, homeowners aren’t careful about opening their mail, and sometimes packages they open contain invisible little criminal robots that immediately scurry out of the homeowner’s sight, pull out their cell phones and start communicating with criminal HQ. They run around the house and send out all the private information about the people who live there, including ID’s. They’re RAT’s (remote-access trojans) and result from what are called phishing attacks in email. Most homeowners aren’t able to resist opening infected emails of this kind. Sometimes the RAT’s make copies of themselves and send them to neighbors’ homes, spreading the problem.

Some security-minded people in the gated community protect themselves by installing cameras and other security devices so they’re alerted any time a person enters their house when they’re not there.

• In the computer world, tracking activities inside the house is extremely rare; detecting unusual activity and sending alerts when it’s detected is almost unheard-of. The criminal computer invader is free to take his time while sending copies of all the valuable information in the house of the gated community to criminal HQ. If the criminal feels like it, he can massively scramble the contents of the house to make it practically unusable, maybe even putting things in locked cabinets. Then he leaves a sign on the entrance door explaining to the homeowner what he’s done and demanding a ransom to return things to normal. When the ransom is paid, nearly always in untraceable bitcoin, the house often isn’t returned to normal after all. Why is anyone surprised?

The guard at the entrance gate makes sure that moving trucks that enter have been authorized by a homeowner, and that when they leave, the homeowner has approved the exit.

• In the computer world, watching what goes on inside the walls is rare. The equivalent of moving trucks can be created and endless streams of them can go through the exit of the guarded gate and no one checks.

Some homeowners are particularly concerned about the valuables inside their houses, and so they keep those valuables in an expensive, thick safe, secure from thieves.

• In the computer world, data you want to protect from being stolen is encrypted “at rest.” But just like you have to take jewelry out of the safe to wear it, you have to unencrypt data to use it. Therefore the programs in the house that make use of valuable data use a method to access the data that unencrypts it automatically after taking it from the “safe” and before giving it to the program that needs it. When the criminal software is in the house, it simply uses those same programs to access any and all data that it wants, loads massive valuable data into moving trucks and sends it out the unguarded, unwatched exit gates of the gated community. The owners frequently don’t find out about the theft until months later.

Conclusion

I’m sorry to say, things in the computer world are just as bad as I have described. Cyber-security experts and regulators do little but build up the mountains of pointless, ineffective procedures and regulations to ever-growing heights, seemingly without questioning the value of their efforts. Why should they? No one else does – their highly paid jobs are secure!

This was originally posted at Forbes.

T

The Colonial Pipeline Cyber-security Disaster in Context

There’s little new about the Colonial pipeline security disaster; nearly everything was business as usual: expensive but ineffective cyber-security systems and people; penetration and massive data stolen secretly; learning about the breach when ransomware popped up and said “pay me;” shutting everything down; taking days to recover; and the government, which is incapable of protecting itself, making solemn statements about “helping” more.

Colonial did manage to stand out from its fellow victims in a couple of ways: unlike many victims, they paid the big ransom; and their shutdown hurt millions of normal people in significant ways.

The other way Colonial stood out is remarkable: it was big news in the media for days, while the many “successful” ransomware attacks that take place each and every day on businesses, governments, schools and hospitals are rarely made public, much less make the news. Cars being unable to get gas in multiple states, even at inflated prices, may have had something to do with it...

Colonial Pipeline in Context

As usual in disasters of this kind, many important details of the attack and the response to it are closely-guarded secrets.

Let's step back and put what we do know about the disaster in context.

Ransomware has been around for a couple decades. It was usually sent to consumer emails with threats of various kinds if a ransom wasn’t paid. The ransom amount was usually under $1,000 and was paid to prepaid cards and other places. Because of the ability to trace the payment, the scam became less profitable and more dangerous for the criminals involved.

Then bitcoin came along. By 2013 exchanges appeared that enabled criminals to receive instant payments while remaining anonymous and untraceable. Why attack consumers for small amounts when you can get into a large institution and demand large amounts that you can receive instantly and anonymously anywhere on the globe? The threat evolved as well. The attacking software, after gaining entry into the target’s internal computer network, would encrypt all the data, making the organization’s computers nonfunctional until the data was unencrypted using a key known only to the criminals. The victim would be stalled in place, unable to function until the ransom was paid or the computers were otherwise restored. The use of ransomware exploded.

While the well-paid but ineffective defenders sloppily applied what they were taught in school and followed the thousands of pages of security-related regulations, the criminals evolved on multiple fronts. Ransomware evolved rapidly during 2020. The criminal attackers started to take a copy of the target’s data (exfiltrate it) before locking it up. The threat evolved to: if you don’t pay us we’ll make all your data public and you’ll be locked up.

Industry expert EMSISoft tells us: “As the year progressed, more and more groups started to exfiltrate data, using the threat of releasing the stolen information as additional leverage to extort payment. At the beginning of 2020, only the Maze group used this tactic. By the end of the year, at least 17 others had adopted it and were publishing stolen data on so-called leak sites.”

Oh, expose the data — how bad can that be? Pretty bad. “The data that was published included Protected Health Information (PHI), sensitive information related to school children, and police records related to ongoing investigations.”

What? These sound like highly regulated hospitals and government organizations, even law enforcement. Isn't the government, which creates and sometimes enforces all these cyber-security regulations able to protect itself? I guess not: “Unfortunately the barrage continued into 2020 with at least 2,254 US governments, healthcare facilities and schools being impacted. The impacted organizations included 113 federal, state and municipal governments and agencies, 560 healthcare facilities, 1,681 schools, colleges and universities.”

Colleges and universities? Aren't these the places that train all these cyber-security people and create the theory and practice they all learn and put into practice, with their fancy degrees? How is it possible that the security experts can't keep themselves secure? And just look at those numbers: more than four per day were hit and hurt!

On the other hand, it's just people's data getting exposed and ransoms being paid, right? Sadly it’s more than embarrassment: “The attacks caused significant, and sometimes life-threatening, disruption: ambulances carrying emergency patients had to be redirected, cancer treatments were delayed, lab test results were inaccessible, hospital employees were furloughed and 911 services were interrupted.”

More than an attack a day took place at healthcare facilities! Have you seen any headlines about that?

Alright, alright, this is all about governments and similar institutions. Commercial companies want to protect their profits and their reputations. They probably handle things much better — they must, because you almost never hear about them. Ummmm, maybe not; again from EMSISoft: “The private sector was hit hard too. Globally, more than 1,300 companies, many US-based, lost data including intellectual property and other sensitive information. Note, this is simply the number of companies which had data published on leak sites and takes no account of the companies which paid to prevent publication. Multiple companies in the US Defense Industrial Base sector also had data stolen, including a contractor which supports the Minuteman III nuclear missile program.”

Read that last sentence again, please. The bad guys are successful in stealing even from defense contractors. And the number above is the tip of the iceberg, because it's just the ones where someone was able to find their data for sale — it doesn't count all the ones who paid up, hushed it up, etc. Somebody did a survey to find out just how widespread the problem was in commercial businesses. Here's the bad news: “according to a study by security firm Sophos, 51 percent of all surveyed businesses were hit by ransomware in 2020.”

The iceberg is indeed huge. We're talking serious money given to criminals. From Pentest Magazine: “By the end of 2019, cybercriminals using ransomware had made off with a reported $11.5 billion in ransom payments. By the end of 2020, that number is projected to reach $20 billion.”

That's "just" the ransom money — much more money is spent recovering from the attack, even if the ransom is paid.

With all that bad stuff going on and the FBI and other agencies devoting huge resources to it, at least some of the bad guys are being caught and punished, right? No. According to EMSISoft “the effective enforcement rate for cybercrime in the US is estimated at only about 0.05%.”

In case you're not feeling math-y, let me help. This means that out of each 2,000 cybercrimes, only one is prosecuted.

Conclusion

The Colonial Pipeline event was extremely rare — not that it happened, since about half of all businesses get hit with ransomware every year — but because it made the news and was widely covered.

The reality is that, largely invisible to the public, there are gangs of criminals roving secretly and largely unchecked through our computer systems and networks stealing valuables and extorting money in huge volumes. Business and government spend increasing amounts of money with ever-growing staffs of highly educated, certified professionals to prevent the on-going pillaging. They are failing. Horribly. The vast majority of the "cures" that are batted about will definitely cause everyone involved to spend more money, and will equally certainly make little difference.

I have discussed the issues and illustrated the problems and solutions but it won't make a difference — all the power and prestige go, as usual, to people who are proven ponderous pontificators to whom the entire realm of software is invisible.

Note: This was originally published in Forbes.

 

Ingredients: Whole Foods Sneaks in Sugar

You really do have to read and understand the ingredients list of any food you buy. Even places like Whole Foods need to be carefully checked. Yes, the same Whole Foods that charges "whole paycheck" prices to supposedly only sell food that's good for you.

The Whole Foods brand Olives

Whole Foods is a grocery store with a huge selection of products produced by others. Whole Foods also has its own brand of products, some with standard packaging and some that are custom-packed. Someone I know went shopping there the other day and bought some Whole Foods olives, intrigued by what olives with "tangerine and chilis" would taste like.

After getting them home, my friend opened up the package and tried a couple of the olives. Yuck, what's this? These taste outrageously sweet! These are supposed to be flavored hot and with citrus! What's going on? At that point she took a closer look at the label:

Surprise! The third ingredient after olives and water is "cane sugar." More sugar than oil, more than vinegar, more than tangerines, more than red pepper and more than salt.

This being Whole Foods, the ingredient of salt isn't just any old salt, it's supposedly "sea salt," which sounds like it's better for you somehow. And the sugar isn't any old sugar, it's "cane sugar" -- oh, right from sugar cane, it's "natural," right?

Sugar is a Big Problem

Most people like sugar, which is part of why deserts can taste so good. The trouble is that people who make food only care that you like it and want to buy more of it; they mostly don't care whether the food is healthy for you to eat. Regardless of what they say. Therefore, sugar of one kind or another is added to an astounding range and variety of foods, including ones that are "supposed" to be sweet.

Even the CDC, the well-funded government organization whose whole purpose is to keep us healthy, agrees that sugar is a problem.

Weight gain and obesity, type 2 diabetes and heart disease. Is that all? In the section on obesity, the CDC admits that obesity itself is a serious problem:

This being the CDC, all they do is admit that obesity is "associated with" those fatal things and nowhere do they spell out the curse of sugar added to foods and the need to read the ingredients. The picture above shows obviously sweet cupcakes, while much of the problem is with foods that aren't obviously sweet.

Getting information about sugar

You may have noticed my use of phrases expressing cynicism about the CDC. Although the statements I've excerpted above are true, they have appeared on the CDC after decades of being studiously ignored by this corrupt agency. This is because the CDC has long practice in putting its authority and the patina of "science" behind what various powerful pressure groups want to advance their own interests.

Sadly, like with so many other things, it's up to each person to look out for him/her self. In this case, it's pretty simple: moderate your ingestion of sugar in any form; avoid buying and eating packaged food that has any form of sugar added to it; for this purpose, trust only the small-print list of ingredients.

Conclusion

You can't depend on the idea that a "health oriented" store will only sell you healthy food. You can't depend on how a food product is labelled. You have to take control of your own health, which means taking control of what you eat, which means taking care about the ingredients of the food you buy. It's not too hard, it doesn't take much extra effort, and the payback in terms of length and quality of life are more than worth it.

Software programming language evolution: Functional Languages

Once computers were invented and started being used, people discovered that writing programs for them was brutally hard. It didn’t take long before some smart folks figured out ways to make it easier, taking two giant steps forward in ease and productivity in quick succession.

After that wonderful start, people kept on inventing new languages, but somehow never made things better – in fact the new languages often made programming harder and less productive, a situation that the experts and exalted Professors studiously ignored. They never even bothered to spell out what makes one language better than another, which you'd think would be at the heart of promoting a new language.

I this post I’ll discuss a major category of new languages that keep getting created, receive passionate kudos, are rarely used but refuse to die: functional languages. In a subsequent post I'll describe the truly good idea that lurks inside the madness of the functional language world.

The core idea: time and timeless, music and math

I learned math as a kid. In high school I took the most accelerated and advanced courses available, finishing with a course in calculus, which I aced while mostly sitting in the back of the classroom teaching myself differential equations from my father’s grad school text book. I liked it. At the same time I got into music of multiple kinds but with a strong preference for classical. The strong connection between music and math has often been noted.

During my junior year of high school I took a course in computer programming, with FORTRAN as the language. Our “teacher,” scrambling to keep a chapter ahead of us in the text book, had arranged machine time for the class on Saturdays at a computer at a nearby rocket firm, Reaction Motors. After lots more programming, by my first year of college, it was clear that math took a distinct second place to both music and software in my heart.

In this post I explain where software comes from and its relation to music. The key concept is that, while you can read a page of music or software or math, math usually exists in a world “outside” of time – at most, time is a dimension like space; think for example of a proof in geometry. Music and software, by contrast, can only be understood as flowing through time. When you look at a page of music or software you naturally start at the beginning and hear/see it flow through time in your mind, just like reading a page of text.

What this means is that there is a fundamental dis-connect between math and computing. I explore this disconnect here. The math types are always trying to turn software into math, for example plodding for decades on a fruitless pursuit for ways to “prove” the correctness of programs.

This is the best way to understand both the failure and the refusal to admit defeat of the math types to develop a practical math-like way to write software, an effort and category of languages that is usually called “functional.”

The history of functional languages

Wikipedia's introduction to functional languages is accurate:

Turing had nothing to do with making software programming practical. That amazing job fell to others, who created the two giant advances in software language development. But since the math types were hanging around computers in the early days, particularly because doing math calculation was one of the earliest tasks to which computers were applied, finding a way to make programming more math-like, essentially eliminating the factor of time from software was a top priority.

I had already done serious programming, including working on a giant FORTRAN project to optimize oil refinery operation, when a friend introduced me to functional programming for the first time. The language was APL. It came a bit closer to being able to handle practical math than many of the other non-functional "functional languages" because of its focus on the matrix, but it was as easy to read and understand as advanced math. I could and can imagine that it would be appropriate for representing certain math transformations in a compact way, but why anyone who wasn't being tortured would agree to use it for general programming remains beyond me -- assuming of course that what you want is to ... radical idea here, be warned! ... get stuff done. As opposed to demonstrate and revel in your ideological and mathematical purity and exalted status in the pantheon of Computer Science greatness.

None of this is any concern to the cultists, who march on inventing an endless stream of "new" functional languages. Among the names you might see if you look into this are  OCaml, Scheme, Haskell, Clojure, Scala. There are even a few that are object-oriented on top of being functional, like Erlang. Articles continue to pop their heads up out of the underground insisting that functional languages really are making inroads in commercial applications, better for getting a job, more effective to use to create your ground-breaking start-up, etc. I predict no end to the flow.

The fact that functional languages are mostly of interest to a cult of fanatical core believers minimizes their widespread, on-going pernicious impact on software development, fueled by the near-universal math orientation of academic and corporate Computer Science. Functional concepts keep getting introduced by language fanatics into otherwise-sane normal languages.

Conclusion

Computer software is rarely driven by purely practical considerations much less objective knowledge and definition of what constitutes "good" in a language. But when an approach like functional programming is pushed that leads to results that are often 10X worse on multiple dimensions than "normal" programming, the problem is so large that people who aren't already committed members of the cult notice the difference and refuse to put up with the nonsense.

In spite of all this, there is an amazing valuable insight at the core of the functional language movement that leads to highly productive, real-world-valuable results when embodied in the right way. I will spell this insight out and illustrate its use in a subsequent post. The heart of the insight is taking a declarative approach instead of an imperative one when and to the extent that is appropriate.

 

Yelp is Big Tech Incompetent and Corrupt

Yelp is a Big Tech company that isn't listed when people talk about "big tech," but it's nonetheless all about tech and it's unarguably big, with well over 100 million unique visitors per month to their site and a valuation in the $ billions.

How good is their software? Are they a company from which non-tech corporate America should poach talent to upgrade their in-house tech? I recently had occasion to spend a little time with Yelp, and found that they are indeed an exemplar of the Big Tech tradition of tech bumbling as I have documented.

Harvard Business School famously teaches according to the case study method. Students are presented the case and challenged with figuring out what to do. Yelp would make an excellent case study. Any of the bright, aggressive b-school students who declared that the "solution" to the case was to fire all the executives and replace the entire product and tech team with talented high school grads, would get at least a B. Anyone who further suggested avoiding hiring anyone with a Computer Science degree would get an A.

Yelp and Local Business

Yelp is an advertising company that features listings of local businesses and user-generated reviews of those businesses.I recently visited my local dentist and received excellent care on the way to getting a crown put on one of my teeth. She knows I'm a computer guy and I've helped her out in the past, so she brought up her concern about a couple of malicious false reviews that were posted to her Yelp page. I'd said I'd check it out. I did. The results were illuminating. I sometimes complain about bumbling big corporate companies and their painful but fixable software problems. Yelp provides a great example of how second-tier Big Tech companies are no better at building software than the super-famous first tier ones are.

Yelp and a Local Dentist Business

I checked out Yelp's page for my dentist, Doctor Vu. It had a couple long negative reviews. So I posted a review of my experience, along with a selfie of me at her office.

I went back after a couple days to check out the page in more detail.

At the top of the page:

I see that she has 5 reviews with an average score of what looks like 3.5.

I scrolled down and found the photo section with the photo I had posted:

I took the selfie in her office holding the coaster I had given her last year that I couldn't resist buying at my favorite store in Cape May. She's had it in her waiting room ever since; I hear it often elicits amusement. It just seemed right to take the picture at the end of a visit to get a crown. :-)

After scrolling through ads and other information I got to the reviews. Mine was first:

I scrolled through the rest, including the two 1-star slams. Something didn't seem right. I went back and carefully counted the reviews. I double-checked. There were 6 reviews! The header at the top said 5! Then I made note of the star ratings. The two super-baddies were 1 star and the other 4 were 5 stars each. That's 22 points divided by 6 reviews = 3.67. Looking carefully at the points at the top, the dividing line is clearly at the top of the star, so Yelp thinks it's 3.5.

But then Yelp said there were only 5 reviews in spite of displaying 6.  Is it 22/5? Nope, that's 4.4. A complete mystery. Maybe I'll figure it out later.

At the end I found this:

What's this about? In faint type 16 other reviews? I check it out. At the top of the page that shows up there's a video I can watch and this long, serious-sounding thing about how Yelp cares and they're smart and they're doing the best for everyone:

Impressive! Finally I scroll down to check out these malicious reviews and here's the first thing I see:

Wow. Billions of data points! That software must be good, right? So I scroll down to see these scurrilous reviews. Imagine my surprise when I see this:

Wow. My review. Again!

At least the mystery of the rating average is solved. Even though my review was shown among the legit reviews -- first, no less -- it appears that their ludicrously bad software had marked it as not recommended and so some piece of software decided there were just 5 reviews. So 3 5-star plus 2 1-star is 17 stars, divided by 5 is 3.4. Close enough to the 3.5 average rating they displayed.

Even better -- I received a congratulatory email from Yelp after posting the review with the picture, saying how great it was and I should post more pictures along with my excellent reviews. My picture is indeed featured. And my review is both first on the list of legit reviews AND first on the list of those not recommended. Score! This is what you get by employing thousands of Silicon Valley's best super-programmers...

Those billions of data points must have somehow confused the Yelpers. And their software.

Finally I read through the rest of the reviews that weren't recommended. There was nothing fake about them.

I visited Dr. Vu again to get actually crowned -- the first visit was just preparation. She told me she had contacted Yelp about the situation. First she learned that the Yelp algorithms were state-of-the-art great, and in any case there was nothing a Yelp employee could do to alter their results. Second she learned that she really should pay Yelp lots of money so she could fix up her page, post pictures, etc. Maybe if she paid something could be done about those bad reviews...

My conclusion from this wasn't hard to figure. Yelp is corrupt (pay to play!). Yelp is incompetent (bad algorithms, double-posting, bad arithmetic). Sensible people should ignore Yelp.

Conclusion

I'm tempted to say that Yelp fits right in with its first tier Big Tech buddies, and of course it does. If you can stand it, check out the summary of their issues on Wikipedia. I did and also went to some sources to make sure it wasn't a Wiki hit-job. I also checked out the disastrous business and personal judgment and behavior of the leaders. Pathetic. Yelp is now on my blocked list of websites.

The Dimension of Software Automation Depth Examples

I have described the concept of automation depth, which goes through natural stages starting with the computer playing a completely supportive role to the person (the recorder stage) and ending with the robot stage in which the person plays a secondary role. I will illustrate these stages with a couple examples that illustrate the surprising pain and trouble of going from one stage to the next. Unlike the progression of software applications from custom through parameterized to workbench, customers tend to resist moving to the next stage of automation for various reasons including the fear of loss of control and power.

I will use companies to illustrate this progression that are known to me personally but not widely known. Software history generally is the history of the winners as written by the winners, like most history. However if you want to guide your own software strategy by taking advantage of the clear patterns of history, it is essential to include examples of companies that are rarely discussed.

Nextpage

NextPage was active from 1999 to around 2010. It had a distributed document management product that was prominent in service industries, for example lawyers and accountants. The product operated at a “recorder” level in terms of automation depth. It wasn’t directive, it just sat there waiting for a document request, and when it got one, responded with a list of applicable documents and lets you view the documents.

They and some of their larger customers saw that, in fact, many of their engagements weren’t so very different. As a high-powered, highly paid lawyer, you may think that every job is unique and your ability to handle the differences very important, but it was obvious to people involved that certain transactions involved very similar documents and workflows. It made sense to attempt to capture and automate the similarities.

With the cooperation of one of the world’s largest law firms, NextPage built a new product on top of the existing distributed document platform; this new product was automation at the “power tool” level. While recognizing that each transaction of a particular type would end up producing a unique set of documents, for example, it operated like a series of factory distribution belts, assuring that all lawyers working on an M&A transaction, for example, worked on the right things in the right order with the right base documents, reviewed in the right order by the right people, etc. etc. Lawyers continued to craft their documents. But instead of treating each (for example) M&A transaction as though it were the first one the firm had ever handled, the new system treated them like a repeatable process, with variations.

But the lawyers didn’t think of it that way at all! They were used to thinking of themselves as powerful, autonomous, self-directing agents. Now they were to be subservient to an assembly line in a factory, having their work and output measured as though they were hourly workers? No way – I’m a professional – there’s no way I’m going to put my head in that yoke!

NextPage had a good position in the market, existing customers, a prestigious lead customer for the product, a strong business case, and tangible results. But the resistance to moving to a “power tool” level of application was stronger than the force NextPage was able to bring to bear at that time, and the effort failed to bear fruit. Anyone who looks at the application knows it’s going to happen, the benefits are definitely there. But anyone with sense would be reluctant to predict even the decade when it would be likely to actually happen.

The pattern of resistance encountered by Nextpage was strongly correlated with the prestige and power of the people whose work would be affected. The same resistance is seen elsewhere; for example, how eager are the highly paid and prestigious category of medical doctors to have their work automated?

Captura/Concur

My VC firm invested in Captura, which is now part of SAP Concur, the leader in on-line expense management systems. These applications basically automate the process of filling out expense reports, getting them checked and approved, and finally paid.

If you think about expense reports, you imagine a form that you fill in with your expenses during a trip. You then attach your receipts, drop it off, and eventually get reimbursed for out-of-pocket expenses. So it seems like it would be a “recorder” type application. In fact several companies were started and eventually failed with products that weren’t much more than recorder type applications, with some automatic routing added on. This kind of application didn’t provide enough value to offset the expense of installing and running it.

Captura went much farther. While parts of this application operate at the “recorder” level, particularly the entry of un-automated expense information, much of it actually operates at the “robot” level, which is why the payback for its use is so high. It has built-in rules for approval levels and documentation requirements and many other things. It knows whether and when and to whom expense reports should be routed for approval from the HR system. It gets feeds from the corporate card system. It kicks out exceptions. It allocates expenses to the right accounts. It feeds directly into the accounts payable system to cut checks. When it needs something it can’t get from a computer, it asks a human for it, and otherwise does its job without help.

Unlike other “power tool” type applications, this application tended not to threaten people who were powerful in the adopting organizations. The most powerful people who used it tended to regard it as saving them time, and everyone got their reimbursements more quickly and accurately. The administration group saw it as saving thankless clerical work, and the accounting executives saw automatic enforcement of all their standards. Still, this application category didn’t really take off until the cost of implementation was reduced to a painless level; in other words, it had to wait for nearly universal access to the web to be practical and affordable. And it worked much better as a secure service than as an enterprise application, because the burden of installation and maintenance on IT added so much friction that only the most motivated potential customers would go for it.

ClickSoftware

ClickSoftware provides a classic example of a “robot” level application. It takes the problem of a field service operation, in which there are calls for service from companies in different locations, with different equipment to be repaired, with varying levels of urgency and service level agreements, with different hours of operation and times to travel to them, and finally with different service technicians who have varying skills and qualifications and constraints about hours of work. Who gets sent to which location to fix which problem in which order?

Without something like Click Software, this problem is solved in a rudimentary way by the service manager, who uses some combination of experience and common sense to work out the best solution. However, for a large service organization, the difference between a good solution and the optimal solution can be worth a great deal of money. Click Software takes all those inputs and produces the optimal solution, and asks the human manager for help when the problem simply can’t be solved; in response, for example, the human may ask a critical resource to work overtime, or might call a customer and see if their request can be deferred.

Click illustrates another characteristic common to robot-type applications. It is typical that “power tool” type applications do not become robot type ones by incremental addition. A robot application requires a whole new approach to the problem, and a level of math programming that is likely to be entirely foreign to the software group that wrote the power tool application.

Tape backup systems

In tape backup, whenever a new platform has come out, tape backup software products tend to emerge in the same order, starting with products that just record what the backup operator does, moving to power products that give the operator hints and suggestions and automate the repetitive operations, and ending with true robot products, which are driven by a set of rules and which request operator intervention at only and exactly those times when human intervention is required. For example, Palindrome and Cheyenne (now part of CA) introduced robot-level backup products in the early 1990’s for the PC server platform, at a time when similar products had been in general use on earlier platforms (e.g. IBM mainframe) for decades.

Having cycled from primitive to robot level automation several times as new platforms came out that had tape backup, the whole category is dying, frozen in time as the latest computer systems use the kind of disks that have become incredible cheap and tape has become obsolete.

Conclusion

It seems inevitable that an industry would move, step by step, towards robot-level automation. The historical fact is that while the incentives for advancement are always there, advancement happens only when an industry is "ready" to move, which can be decades after it is technically possible and economically practical.

Anthem Needs my Feedback and Reveals Deep Problems

I’m glad I have health insurance; I’ve had some health problems, now under control, that were expensive to fix. My insurance company, Anthem, has paid the claims.

Still, I can’t help being impressed by how many actions of this health insurance giant are stupid, wasteful, incompetent and worse. To be clear: I don't claim they're worse or better than the others; they just happen to be the one that I have.

I’ve just had a new stupidity inflicted on me by them that is low on the “it matters” scale, and high on the “a smart high school student could have done better” scale. What's interesting is that while the stupidity itself is minor in the overall scheme of things, it's the result of a serious dysfunction causing widespread trouble and hassle for patients and providers, while adding expense to Anthem. The extra painful thing is that, for anyone with a moderate amount of software knowledge, it could easily be fixed!

When I think about all the big, important, highly paid executives involved in and in charge of this, it does make me wonder whether we’d all be better off if they refused to hire any more people with college degrees for any job, and in particular, management.

The only reason this story is worth telling is that it's not about Anthem; it illustrates how  the techniques taught in business and computer science schools and embodied in endless standards and regulations keep large organizations locked into acting in ways that are both expensive and ineffective.

Summary

The root of the issue is my company negotiating a new plan for its employees. Anthem then sent an email (how modern!) welcoming me and inviting me to go digital with card images and an app. I suspect there were self-congratulatory executive meetings about the wonderful modernity of all this. See this for the inexcusably bumbling reality.

A few weeks ago I went to the dentist, got an exam and cleaning and then scheduled for a crown. My dentist submitted claims as usual and got denied, and I received denial EOB's in the paper mail. I sent my new digital card to the office, and that led to another denial and another paper EOB. Finally the office person was able to get through to a human being at Anthem and somehow resolved the issue.

Then I received a survey from one of Anthem's contracted firms to see how impressed I was by Anthem's new plan signup experience. The survey design and implementation resembled a 1995 paper-based process cluelessly "updated" to use, uhhhh, computers.

The whole mess could have been avoided by having a simple system that took incoming claims and checked them against a database of patients and plans, including plans that had been updated and/or replaced. When a claim arrived for a person with a no-longer-in-effect plan, a simple lookup would enable translating it to the new plan information and everything would take place seamlessly, with ZERO action, trouble or inconvenience to patient or provider. You wouldn't even need to send out a survey to ask how the transition went, because it would have been seamless.

An Email asking for Feedback

Some Anthem exec wanted to get some customer feedback about how their new plan roll-out was going. I got an email. Here’s the lead paragraph:

As you’ll guess from my past blog posts regarding Anthem (a summary with links is below), I clicked, hopeful that they’ve gotten moderately competent, but suspecting that some new sophomoric amateur-hour performance will ensue. (BTW, I’ve always been curious about the use of “sophomoric,” since it is the second year of a four year education program. Shouldn’t a performance that’s worse than “sophomoric” be called “freshmanic” or something?)

Sure enough, I clicked, my browser came up and showed me … a blank screen. Yup, nothing. Nada. I refreshed, tried again, same results. Then I thought, “remember this is Anthem we’re dealing with here; what would someone who dropped out of Computer Engineering 1.01 do? Sure, they’d test their stuff on their favorite browser and declare it working!” I strongly suspected that the kid’s (or highly paid seasoned professional with the skill of a kid who dropped out) browser was today’s most widely used one, Chrome. The next one is Safari, Apple’s browser on the Mac. Behind Safari’s share but still with substantial use are Firefox and Microsoft’s Edge. I copied the URL, brought up Edge and plugged it in. It worked! But it failed totally with my Firefox browser. I visit a large number of websites, and this is the first time in years that I’ve gone to a site that managed a complete face plant on Firefox. It takes a certain kind of perverse skill to pull off a feat like that!

With such a great start, who knows what joys would follow? Could this be a candidate for “freshmanic” status??

The Survey

The very first question puts this survey in the running for un-great-ness. Would you take a look at the never-before-seen way for formatting answers to a multiple choice question? With the major choices ending with “that”?

The second question continues the fun.

After asking me whether I remembered receiving this or that communication and answering “no,” the next question was always an huge graphic reproducing what they sent with the question (in effect) now do you remember? Not once -- several times.

Finally came a request to allow Anthem to follow up with me to help improve their experience. To see what would happen, I said yes.

I then got a form in which I was supposed to enter my name, telephone number, email, and best time to reach me. Right. You already know all this. Anthem gave you my email with an ID. All you need to do is give them my answers with the associated ID. That would make it easy for me. Instead, in typical brain-dead, big-corporate manner, you make me enter all the information you already know AGAIN. So NO, I’m not going to fill it in. So I hit Next. What do I get? You can guess, I bet. Yup!

Lots of red with the same questions in red backgrounds. Of course if this stupidity was supposedly all about protecting my privacy, they could have said something about that, apologizing for the inconvenience of entering information they already have. But no. Just ERROR. And I've already agreed to have my answers and identity shared with Anthem!

I could have just closed the browser, but they provided a button labelled “log off.” Which leads to a screen titled “Logged Off,” informing me of my “success” in logging off. But I haven’t “exited” the survey! They close with “Please close your browser to exit the survey.” OMG! If I don’t close the browser I haven’t “exited” the survey even though I’ve “logged off of this survey.” All these years in the business and I’m only just now hearing a new level of sophistication, about how “exiting” and “logging off” are different. Could I possibly screw things up here?

The Anthem Market Strategy and Insights team

I went back to the original email, and clicked on the link “To find out more about Anthem’s research surveys.”

I found out lots of interesting things like how “we’ll never ask for any personal data in our surveys, like social security or ID numbers.” I guess my name, phone, email and (on another question) age aren’t personal. Who knew?

I found out that the geniuses who put together this survey are just one of a crowd:

Always friendly, always available to answer your questions:

I could go on, but what’s the point. Anthem is incapable of doing the simplest kind of market research on their own, so they turned to a bevy of outsiders who can’t do it either, but Anthem can’t tell the difference between good and bad, so who cares? Except it takes a certain kind of genius to consistently pick the worst on multiple dimensions. I guess that’s why they have a “team” working on it – no mere “department” could do it on their own.

How Anthem could leap decades ahead and get to 2010

Anthem doesn't need to invent a thing. All they have to do is copy methods that are decades ahead of the ones they use -- and are quicker, cheaper and more effective to boot. Here are things they could do:

• Make the transition of employees to a new plan seamless.
  
  • The best thing would be to keep the plan number the same and put in a switch to adjudicate and give plan information according to the date of the switch-over. No new cards or numbers, digital or otherwise!
  • The next best thing would be to issue new numbers but keep a database of employees/patients enrolled in the prior plan, and automatically update things like newly arriving claims with the old numbers to the new ones as needed.
  • If you do one of these you won't need a survey!
• Dump the whole survey thing.
  • Track customer actions in detail like modern web companies do and detect when things don't go as they should. If someone isn't getting what they want, detect it and fix it, and give them an opportunity to tell you what's wrong at the time they experience it.
    
• Dump the whole paper thing.
  • I've opted out of paper every way I can at Anthem. I still get loads of paper mail. Why is it so hard?
• Fire your design department and start over.
  • I get lots of to-the-point communications from Amazon. Copy them!
  • Keep it simple. Test everything in small scale with real people before inflicting it on your customers.

Those are just highlights. There's lots more Anthem could do if their august team of highly experienced professionals would stoop to it. See the next section for some samples.

Past Anthem issues

Just to make it easy for anyone researching giant health insurance company outstanding achievements, here is the list of Anthem issues that I’ve looked into.

Again, I make no claim that Anthem is better or worse than any other insurance company. It’s just the one I experience, so theirs are the stories I tell.

Sometime in 2014 Anthem “lost” tens of millions of patient records. And completely botched telling their customers about it.

I discovered on my own that my data was in the stolen data. Anthem then offered a worse-than-useless plan to “help” me.

Somebody at Anthem decided that I would be cheaper to insure if I acted better and made an expensive botch job of offering a pre-paid card as an incentive.

At one point they decided to send me an email to get me to use a primary care physician they had selected for me. Everything about the experience was a nightmare. A case study in stupidity.

Then I discovered on an offer on Anthem’s website to enable me to pay my patient co-pays. Wonderful idea! Except it doesn’t work. Not just a big screw-up, a huge face-plant.

While I was discovering the joys of Anthem’s inability to help with co-pays, I got and responded to a customer survey. It was a model of how to p*ss customers off, not to mention being decades behind standard practice.

Most recently I discovered that my insurance ID changed and that they botched the change and the wonderful new app that was supposed to replace my card.

Conclusion

Big companies can't build software that works. They can't even do surveys. Hiring people from Big Tech companies doesn't help, because they can't do it either. The good news is, that gives LOTS of room for small, innovative groups that get stuff done that people need, building effective software that works quickly along the way. Hooray!

Experts are Super Smart

It's something I always knew, but I just couldn't make the nagging doubts that would pop up from time to time go away. Finally, at long last, the evidence has arrived: experts really are super smart and expert at what they say and do!

The evidence can be found in this article.

This is an extra relief to me personally, since I've expressed such strong support for the authority of experts in the past, giving examples of their prescient ability to analyze and predict with super-human accuracy.

I need worry no more. The evidence has arrived. Experts truly are everything I had hoped they were!