The Black Liszt

Summary: Blackliszt and David B Black Writing

I have written about 500 blog posts on Blackliszt.com and published five books. This is an introduction with links to summary posts on the topics I've written about.

Throughout my career I followed the low-level strict rules, i.e., my programs compiled, ran and did what they were supposed to do. As the years passed, I learned that most of the widely accepted high-level rules lead to huge time and effort and produce mostly bad software. I first learned this as a programmer and CTO and then as a VC investor, seeing that the winners often follow a different set of rules. I've tried to explain the status quo and the better alternatives in my blog and books. For details of my experience see this. https://www.linkedin.com/in/david-black-49b2b4a2

The Summaries

Computer Science doesn't come close to being a science.

https://www.blackliszt.com/2023/04/summary-computer-science.html

For something that claims to be a science, software is amazingly fashion-driven.

https://www.blackliszt.com/2023/07/summary-software-fashions.html

Because software is invisible to most people, it's described in metaphors, which are mostly inappropriate and lead to trouble.

https://www.blackliszt.com/2023/06/summary-bad-software-metaphors-lead-to-bad-results.html

Everyone wants to know what's "new" in software; the new thing is most often just a fashion. Innovation? It's supposed to be what fuels software and changes the world. In reality, there is almost no innovation in software itself, and software-driven innovation is most often what I call in-old-vation, i.e., an advance that's proven in one domain that often takes decades to crawl into new domains. This summary also covers business strategies for taking software innovations to success.

https://www.blackliszt.com/2023/07/summary-software-innovation.html

How do you win with software? Nearly everyone starts with requirements, makes estimates and is judged whether they deliver on time and on budget. This optimizes for expectations and is a proven path to failure. The winning path optimizes for speed, customer satisfaction and continuous quality. It's what the people who need to win do.

https://www.blackliszt.com/2023/07/summary-wartime-software-to-win-the-war.html

Software people don't learn the fundamentals of computing; they may be aware of any one of them if asked, but don't apply them. In any other field, knowing and applying the fundamentals -- like blocking and tackling -- is demanded. Not in computing.

https://www.blackliszt.com/2023/06/summary-software-fundamentals.html

A key aspect of software fundamentals is software architecture. A surprising amount of the received wisdom about architecture is counter-productive, the goals are rarely discussed and the key to effective software architecture, Occamality, is largely ignored.

https://www.blackliszt.com/2023/08/summary-software-architecture.html

If you want to architect and build good software, a major helpful factor that is usually ignored is to learn the history and evolution of software. Patterns repeat and knowing where you are in an evolution can help you predict what is likely to happen. Not to mention avoiding going down paths that usually fail.

https://www.blackliszt.com/2023/08/summary-computer-software-history-and-evolution.html

What do software people do? They write programs. What language do they use? What are programming languages, anyway? Are there good ones and bad ones? How have things changed?

https://www.blackliszt.com/2023/06/summary-software-programming-languages.html

Software people are indeed people, but a good fraction of them are different from non-programming people, with differences that are important to understand. Similarly, there are important differences in how to manage them most effectively.

https://www.blackliszt.com/2023/05/summary-software-people.html

Computer security is assured by mountains of regulations, tools, standard practices and audits. But computer security continues to be breached without pause. There are major improvements possible that are largely ignored.

https://www.blackliszt.com/2023/05/summary-the-ongoing-problem-of-computer-security.html

Software quality assurance is an important specialty within the non-science of computing, but in spite of all the time and money spent, quality continues to be a major issue. There are solutions that have been proven in practice that are ignored by the experts and authorities.

https://www.blackliszt.com/2023/04/summary-software-quality-assurance.html

Everyone knows that software project management is essential to producing software that works, on time and on budget. In spite of decades of "innovation," it doesn't get better. The winners follow a different set of rules.

https://www.blackliszt.com/2023/04/summary-software-project-management.html

Huge amounts of attention are paid to AI, ML, Big Data, and the rest. It's a fashion that too often results in vast amounts of money being spent with little to show for it. Results are amazing when achieved, but it takes a combination of deep knowledge and common sense to do it.

https://www.blackliszt.com/2023/08/summary-ai-machine-learning-big-data-math-optimization.html

Data centers and networks are the underpinning of computing. The transition to the Cloud has been a big factor, along with the rapid evolution of hardware and the glacial evolution of software. Net Neutrality has been an annoying furor, utterly divorced from reality.

https://www.blackliszt.com/2023/08/summary-computer-data-centers-and-networking.html

Cryptocurrency and blockchain are highly visible for many reasons, including the fact that crypto fuels the explosion of ransomware. Leading thinkers are convinced that blockchain is a technology that will revolutionize computing.

https://www.blackliszt.com/2023/01/cryptocurrency-and-blockchain-summary.html

Learning about the mechanics and technology of the healthcare system provides an education on the huge opportunities to reduce costs, increase efficiency and make things better for patients and providers.

https://www.blackliszt.com/2023/08/summary-the-business-of-healthcare-medicine-and-insurance.html

I've had some personal experiences with the healthcare system that have resulted in a sobering education about what really goes on. Healthcare is incredibly valuable with many dedicated professionals involved, but there are some serious flaws.

https://www.blackliszt.com/2023/08/summary-the-medical-industrial-complex.html

 

I have published five books on these topics. Here are introductory blog posts to them.

https://www.blackliszt.com/2023/05/summary-david-b-black-books.html

 

Occamality: the problem with layers, components and objects

Modern software orthodoxy endorses the notion of collections of code that are separated by high walls. Object-oriented thinking codifies the supposed virtue of data hiding, and keeping all the code that works on a particular block of data (a class) behind a wall. Everyone seems to like the idea of components; people will talk in terms of assembling applications out of component building blocks (when has this ever happened except in seminar rooms?). Finally layers or tiers are supposed to introduce discipline to an application. The idea is you have the user interface (top) tier; then you have the application or business logic layer; and finally you have the database layer. Frequently, these are designed and built by separate groups, each a specialist in its domain – do you really want those superficial, flashy, image-obsessed UI people messing around with your transaction data – do you??? Of course not. But you also don’t want the groups to “hold each other up,” so the idea of having the layers be strictly separated so the UI people can do their thing without being held up by the others.

This is the theory. Lots of people practice it, or at least say they do. If you’ve never experienced anything else, it makes sense. Just the other day I met a software company chief scientist, a PhD in something or other, who explained to me that they were converting some scripts from PERL to java because java was a “real” language, and the scripting languages – all of them! – were not. I didn’t get a real explanation of why this was so; he took it as self-evident to all educated and reasonable people that it was, and by questioning it I really tried his patience, since I was apparently uneducated and unreasonable…

Components, layers, microservices and objects can be good things in limited circumstances, but it’s really, really important to see that if you accept that the prime measure of goodness of a piece of software is occamality for the reasons already discussed, in general, components, objects and layers reduce occamality to the extent they are applied.

The core problem with these things is that they result in multiple definitions of what amounts to the same data. The theory actually encourages this. Take a simple data element such as “account number.” In any program involving accounts, this piece of data will be all over the place. If you’ve got layers, it will be in all of them, because surely it will appear on screens, business logic and the database. If you’ve got objects, it will be in the interfaces and internal implementation of many of them – think, for example, of “account master” and “transaction.” If you think you’re being clever and have built yourself a message passing architecture, it will be in the message definitions – lots of them.

Naturally, you assume and hope that the definition of account number doesn’t change, and in most cases it won’t, in which case the wild redundancy won’t hurt you much. But problems come most often from things you didn’t think about – what happens if there’s a merger, and suddenly there are a whole set of accounts you’d like to bring into your software, and the new account number definitions are totally incompatible. What if they’re bigger and there’s no way to cram them into the existing scheme? You desperately cast around for translation schemes, but in the end you accept the inevitable: you’re hosed.

Now think about this: what if there was a central place where everything you know about account number was defined? Suppose the central place had everything that all the layers needed, for example label for the UI and foreign key relationships for the database. You could go to this single place and take care of most of the account number issues. I assume the program isn’t perfectly Occamal, and there are a few other places you have to go for everything to work. So what? You’re way better off than you otherwise would be.

The painful, anti-orthodox but blindly obvious conclusion is this:

• Each “layer” you put into your software increases its complexity and time to construct, and increases the time and risk to make changes to it.
• The more you use classes and other forms of object-orientation in your code, the more redundancy you introduce, and thus the harder you make your code to construct and change.
  • You may think – before you’ve had lots of experience – that the data hiding of classes increases the “componentization” of your code, and thus makes it easier to change without trouble. Sadly, your thinking on this subject needs revision. Parameters and calls weave your objects together and make them do useful things, and the greater their interaction, the greater the redundancy.

There will always be people who love components, objects and the rest, just like there will always be people who insist on having half a dozen gin-on-the-rocks before supper to whet their appetites. Words don't work. Logic is irrelevant. They should stay in university or wallow around in some giant software bureaucracy where they'll have plenty of company.

Achieving Occamality through definitions: case study

Quite a few years ago I had the problem of creating a product that would help printers create estimates for potential printing jobs. We had one of the early micro-computers at our disposal, and the only programming tool that was available for it was a macro assembler. We had to get the product out in a ridiculously short period of time, and were very limited in the amount of memory we could use.

We got together and realized that we had a fairly simple problem. The ultimate goal was to create printed estimates. Each estimate was calculated based on a combination of data that was unique to the job (the size of the paper, the number of colors, the number of copies, the type of paper, etc.) and data that was common to most jobs, but could be changed at will (the amount to charge for different kinds of paper, press setup and per-copy charges, etc.).We also had to save estimates and create new versions with changed parameters.

Here’s what we did. We broke the whole problem down into seven overlapping problem domains. We created a set of macros for each domain. The parameters of each macro contained attributes relevant to the domain. When assembled, each macro would deposit coded data in memory – no instructions. For example, we had a set of macros for input, another for printed estimates, and another for the core variables relevant to estimating. Macros could refer to other macros, so we could eliminate redundancy and keep the memory requirements as small as possible.

Each parameter in a macro had to correspond to or do something, so we definitely wrote code, but the code we wrote was highly abstract and was roughly proportional to the extent of the macro parameter definitions. When we added a new macro or a parameter to an existing macro, we would add or modify a couple line of assembler code.

The actual functionality of the program was created by a small amount of code that rarely needed to be touched and was pretty easy to write and debug, and a relatively larger amount of macro calls that deposited meta-data in memory. The instructions walked through the meta-data and created the behavior the user saw. We spent time at the beginning understanding the nature of the problem, defining macros and writing code. As the project went on, we spent less time with code and definitions and more time with writing and modifying macro calls. As time went on, the users got more and more stuff, and we were able to change what they didn’t like quickly and without introducing further errors or side effects.

We lost a good deal of time because we picked a computer that was too early in its lifecycle. It had hardware errors, and the macro assembler that was so important to us was buggy. The operating system was primitive, and we even had to write our own file system. In assembler. With a flakey machine and a buggy macro pre-processor.

The main competitor at the time had a system they sold for roughly $25,000. Ours was faster, easier to use, had far more functions, and could be sold at great margins for $10,000. The project was taken on by me and three totally awesome geeks. We delivered the product on the date that was fixed at the start of the project, a date that based on nothing except when it would be nice to have the product. It proved to be nearly bug-free when delivered to customers; one customer found one bug after a couple of months, and I was able to fix it in an hour. Time from start to finish: ten weeks.

Achieving Occamality: What not How

I don’t think I can improve on Chris Date’s formulation of the issue. His basic point is that most computer programs solve problems by taking an imperative approach, telling the computer how to accomplish a given task. He argues strongly in favor of a declarative approach, telling the computer what needs to be accomplished, and having a core set of application-independent functions that accomplish the goal in an optimal way.

Date’s favorite domain is databases. His approach works for databases by writing a program that knows about schemas (tables and columns), data (rows) and SQL statements (Insert, Select, etc.). If you can fit your problem of representing (the schema) and manipulating (the SQL) your data in the way databases expect, and an amazingly wide variety of problems can be represented in this way, then you get to define, load manipulate and access your data without writing any code at all. Since the one body of DBMS code ends up solving a huge number of problems, this approach is highly Occamal.

There are other well-know examples. Spreadsheets are a good one. If you’ve got a spreadsheet-type problem, you can get your needs met quickly and well. You can even write formulas and small program to perform custom calculations.

While databases and spreadsheets are widely understood “horizontal” applications, the approach is applicable to nearly any domain. In fact we often take an approach of this kind when writing applications anyway! If you look at a body of code, there will probably be some code that actually performs an application-specific function that users, for example, would think is meaningful to them. Then there’s all the “other” code that you had to write to enable you to deliver the application value. What’s unusual is for programmers to go “all the way,” and make a clean separation between domain-related abstract functions and application-specific declarations.

The notion of "what" not "how" is a far-reaching one. For example, most of criminal law is "what" you're not supposed to do -- don't kill anyone! The law doesn't spell out "how" you're supposed to avoid killing -- just don't do it! By contrast, most regulations are written as "how" type rules. For example, instead of just saying "the medical device as to work correctly," typical "how" type regulations spell out in gruesome detail how you must accomplish this; and better ways are not allowed!. See this for more detail.

Achieving Occamality through definitions

This is a deep subject, and challenges the way many programmers think. It is rooted in the most fundamental assumptions about the way computers work and the way we program them. In my experience, the ideas first strike people as being simple, obvious, and uninteresting. They seem irrelevant to anything important, and even seem like far-out crank talk.

One way to approach this is to imagine that you stop a computer and examine its memory, byte by byte. At one level, it’s all data. By definition, what’s stored in a computer’s memory is data. But in practical terms, every single byte falls into one of two categories: (1) “plain” data, and (2) instructions. Instructions are placed or loaded into a computer’s memory like any other data, but unlike plain data, the computer can be “pointed” to a starting address and told to start executing the data that is there, and good things will result if the data conform to the rules for instructions.

Achieving Occamality through definitions essentially amounts to dividing the computer memory into three categories: in addition to instructions and “plain” data, you add “meta-data.” If you wrote a program with just data and instructions, you would have a certain amount of space devoted to each. If you write a program using meta-data, the data stays pretty much the same, but the space devoted to instructions typically shrinks by a great deal, and you have a good deal of space devoted to meta-data.

It is important to note that the meta-data should not be in the form of tokens, and while the instructions in some sense “interpret” the meta-data, the meta-data should not primarily be a directive, imperative program.

In practical reality, what you do is create a descriptive, declarative language that is as close as possible to the problem domain as possible. There should be a minimum of translation between the “natural” way of thinking about the problem and the way the problem is expressed in the language. A good example is a navigation system as described here – the meta-data describes the map in natural terms. If you understand language concepts at all, there is probably a one-to-one relation between elements on a visual map and elements in the map description language.

Of course, even a highly declarative approach has a directive aspect to it. My intention here is to emphasize what is usually ignored, not to present an either/or. For example, while a program that gives directions mostly consists of the map meta-data, the direction-generator itself can usually be built in a partly imperative and partly declarative fashion, and needs some plain old parameters, things like whether to avoid toll roads or interstate highways.

This is not a left-field approach to programming. In fact, every method of programming computers, with the arguable exception of “raw” assembler language, starts with a “model” of the kind of program you are going to write, and a choice of how to think about that program.

One of the earliest approaches to rising above raw assembler was the language FORTRAN, short for “formula translator,” derived from the fact that its creators were scientists and engineers who wanted to put their mathematical formulas into the machine for solution.

The business programmers who wanted to use the machine for common business record-keeping functions didn’t find FORTRAN particularly helpful. So COBOL, an acronym for Common Business-Oriented Language got invented.

While it’s reasonable to think about FORTRAN, COBOL and the various languages that succeeded them in terms of language, and this has often been done, it is also reasonable to think: When I sit down with language X, what kind of problem does it help (or hurt) me to think about? How much “translation” is required from the natural terms of thinking about the problem and the language? Do the very terms of the language talk about things I think about? If not, you probably have an opportunity to create a definition domain in which to express your problem, and write a much shorter program than you would otherwise need to write to get the job done.

I’m happy to say that the approach I advocate here is very close to what many people call “model-based” programming. It is ironic that this is so hard to describe, instead of just being mainstream common sense. When you think about a body of code that solves a problem, it is normally possible to code each abstract function that the code performs many times exactly once, and then reduce things it does uniquely to a set of meta-data. Most programmers are pretty comfortable taking this approach when defining the schema for a database-style problem. The model-based approach is really little more than taking the concept of a database schema and greatly generalizing it.

Case Study: Replacing metadata with fashionable software

A VC firm I worked for made a (sadly, in retrospect) passive, non-controlling investment in an emerging PLM company that was all over model-based, declarative  development. They didn’t use silly, made-up names like “Occamal” to describe what they did, but they were fully self-conscious of the power of meta-data, and used it fully. The CTO chose Microsoft as the target technology stack, and ran into the fact that most programmers who specialize in this stack found the meta-data approach to be a foreign one. He overcame the problem by getting his programming done by a group of mathematically-oriented programmers in Kiev, encountering a phenomenon I have encountered many times, namely, that people who think mathematically find it natural to think in the abstractions demanded by Occamality.

This company, AA, had both the problems and the achievements many companies have at their early stage. It had accomplished amazing things in a remarkably short period of time with a small team. It had created a distinctive product that could be customized with an order of magnitude less effort than competing products. It had some really happy customers.

AA also had some unhappy customers. It had made some promises it could not keep, and its code had some really embarrassing failures in the field. It had gotten speed and innovation pretty well down, but quality and reliability remained works-in-progress.

AA accepted a major investment from a couple of brand-name venture capital firms. Each of the firms had name-brand partners on the investment. They surveyed the customers, and discovered the quality problems. So, naturally, you would think they focused on fixing the quality problems, making sure to keep the overall approach that was the key to the company’s value. Makes sense, yes? What else would experienced software-industry investors do? Well, they could do what they actually did – which was hire a big-name CEO, who brought in his own VP Engineering, who glanced at the product, decided it was no good because it wasn’t written in a proper language, viz. java, and put the existing product “on ice” while his huge new team re-wrote everything in industrial-strength java, which of course, as everyone knows, is a language so magical that all you need to do is write in it and what results is fast, bug-free, infinitely scalable programs.

I came in at this point and raised a strong protest. I defended the model-driven vision and most of its execution. I pleaded for spending the new money to fix what was broken – the quality and release process – and for continuing to nurture the goose that was already laying gold nuggets and clearly gearing up to laying golden eggs. It was like I was from Mars, and not the advanced-civilization Mars of science fiction, but a hick Mars in which the most visionary thinkers could hope there was a Stone Age in their future. They tolerated my presence with curled lip and up-turned nose, and went back to whatever they thought they were doing before I intruded on them.

A year and a half later, all the money was spent, the re-write of the application was “nearly” ready to ship (a status it maintained for an amazingly long period of time), some of the programmers were found to have played with visual basic at home, which accounted for the fact that the magic that normally makes java bug-free had been compromised by exposure to impurities. Worst of all, customers were telling the CEO that they wouldn’t upgrade to the new version even if it were perfect, because it lacked the easy adaptability of the original, model-based application. The CEO did what any sensible, experienced executive would have done under the circumstances – blamed the founder for his flawed product and market vision, flaws so profound that even a total re-write was unable to cure them, and sold himself into another job, where they were glad to land such a seasoned executive who had such practical, down-to-earth judgment.

The investors, left to pick up the pieces, had no choice but to return control to the founder/CTO, who, amazingly, had stuck around while the new team was spewing all the money into the garbage dump while dumping on his vision and execution. He hired a VP Engineering, a Russian with many years of hands-on US-based software experience, who fully appreciated the vision, and was able to get the newly re-hired, largely Slavic software team on board with making quality an equal partner with speed and innovation. The quality problems got fixed, customers were re-enchanted with the product, and AA enjoyed a second chance at success. (A few inessential facts have been changed to protect the guilty.)

Occamality is the best possible software architecture, but it ranges from unknown to miles beyond left field. Successful, experienced software professionals are highly likely to dismiss it out of hand in favor of whatever the cool thing is at the moment. To be successful with Occamality, not only do the engineers need to understand it but the executive management team has to understand that the approach is the keystone of the company's technical advantage over the competition.

Understanding Occam-optimality practically

Here are some basic thoughts to help understand Occam’s razor for software as it is applied in practice:

• Build only the software you need to build in order to be successful.
• With rare exceptions, shorter programs are better than longer programs. The goal of programming should be to produce the minimum lines of code in order to accomplish the job.
• Always keep the “parts count” of your program in mind, and make it your goal to keep the number of “unique parts” to a minimum, with each “thought” expressed exactly once, with everything that “uses” that thought generated from the primary expression of the thought. See this for a practical example.
  
• The best days are ones that reduce the total number of lines of code while increasing the functionality of the program. A day spent doing nothing but writing new code may very well be a day that that reduces the Occamality of your program, so that every line of code adds to the later burden of testing, documenting, learning, using, maintaining and ultimately changing the program, thus making it worse than it needed to be in nearly every respect.
• In situations where there are multiple paths to a software goal, pick the shortest one that involves the least amount of work. Don’t plan for future change; if you write the shortest possible program, future change will be optimally prepared.
• Ignore common practices if they produce redundancy, and thus violate the razor. For example, multi-tier design frequently results in huge redundancy of data definitions. A multi-tier execution environment is fine, so long as the tiers are generated from and share common definitions, so that the program at the source code level is Occam-optimal.
• Frequently, software is complex because the requirements are needlessly extensive and complex. Reducing and/or standardizing the requirements is a key part of Occamal software.
• Apply Occamal principles to the whole process involving the software, from requirements through to training and support. Cases that seem marginal when looking at the software in isolation often become clear when taken in full context.
• Copy, use, ignore or emulate everything that is not uniquely required to be successful
• Avoid over-determination, i.e., putting in things that sound good or say more than you mean to say. For example, there is frequently no mechanism to deliver the results of error checks in a database to the people or programs that need them, and they are therefore useless. Again, some people use screen design programs that specify design elements in far more detail than they really mean, therefore saying too much.

A great deal could be written about this. For an explanation of the "post-hoc design" approach to Occamality, see this. It's a way of speeding the way to a goal.

Understanding Occam-optimality technically

I am not aware of a set of terms I can use to make a technically clear definition of “Occam optimality” for software, so I’ll just re-use some existing ones. I suspect there’s an exact, more mathematical way to do this. I’m hoping someone will pull it off. My goal here is to express the most basic relevant concepts in rough terms.

By “program” I mean the entire source text that is required to build the executable program. If the program uses a DBMS, the schema definition is part of the program. If there are “include” files or resource files, they are part of the program. If the program is a multi-tier one with an ASP GUI, a component-based application and a database layer with stored procedures, it’s all part of the program. If there is information that is part of the program that is not immediately available as text, then for these purposes we convert it to text. While we normally think of these things as being in totally separate categories, for these purposes, we will construct a single source text that has everything required to build the program.

Here’s a tricky, vague but essential definition. Without this definition, an occamal program would be just a version of the original compressed by mechanical means, kind of like a compiler optimizer. The “intention” of a program is its true goals, without technical specification or constraint. The “expression” of a program is a “program” as defined above, i.e., completely determined by its text. A single program intention can have a very large number of possible expressions. The purpose of the normal program design process is, at the very early stages, to clarify the intention, but most of the process is designed to narrow down among all possible expressions to get close to a narrow range of possible expressions. The final determination of program expression is, of course, made by the coders.

The effort of creating Occamality comes during the process of turning “intention” into “expression.” Of all the evaluation criteria that have been used to drive and evaluate this process, and there have been many, the occamal criterion says: express only what has been intended (don’t over-determine), and express what has been intended in the least possible number of “semantic units” possible, with no redundancy among them.

I use the awkward phrase “semantic unit” to indicate a programmer’s thought, and to avoid getting caught up in one language vs. another, the number of characters involved in the keywords and syntax, comments, the length of labels and variable names, etc. A semantic unit is any combination of code, data and meta-data that is required to express a thought. Semantic units can be primary, composite, and have any number of connections and relationships among them. Good definition and use of connections and relationships are how redundancy is eliminated, along with careful attention to avoid “distinctions without a difference” of substance.

Semantic units can have “primary” information in them and/or “reference” information. “Primary” information in a semantic unit is an actual definition or program statement, for example a data type or a conditional expression. “Reference” information is a reference to one or more other semantic units. For example, you would define that a date is a month, day, and year, with these data types, lengths, labels, compute rules, etc. This would all be “primary” information about date. When you defined “birth date,” its most important characteristic would be the “reference” information that it is a type of date. A semantic unit could easily contain a mixture of primary and reference information. For example, “birth date” would have as primary information its unique label and the fact that it is a separate piece of information; everything else about “birth date” would be a reference to “date.” The reference should be a single reference to the “date” semantic unit.

References, as defined in this broad sense, are the key to creating occamal programs. Every reference is something that could have been given a separate expression, which might have been literally identical, but more likely would have had the same intention but differ in trivial detail as expressed. In the example of the Y2K problem, each reference to a central date definition replaces a separate definition of date. Without references, all the separate instances of date would have to be found and changed; with references to a single central definition, all that needs to change is the central definition, and the references cause the change to be rippled to all uses of date.

A “program” is Occam-optimal when it has the minimum possible number of “semantic units” in its expression, and when there is no redundancy among the semantic units, while still fairly expressing the original intention of the program.

Obviously, this only scratches the surface of the technical side of Occam-optimality. I hope it’s enough to convey the general idea.

Classifications of Software Programs

Anyone who has programmed for a while knows that all code is not alike. There are important broad categories of code. I wish there were a generally agreed to categorization, but there doesn’t seem to be. Many years ago, there were these categories (from lesser to greater in terms of perceived expertise and prestige):

• Operator (your hands actually touched a physical computer!) – low status!
• Applications programmer (many variations here, from business analyst to debugging and maintenance). And even they are not all the same! While I was working in a COBOL shop, I discovered that there is an intense, sharp difference among COBOL programmers:
  • Batch COBOL. There is no direct human interaction here.
  • Interactive COBOL, the kind that paints the infamous “green screens.”
  • Guess which is more prestigious?
• Systems programmer (these got the problems when the application programmers were stumped)
• Computer vendor employee (these people wrote the operating systems, compilers, and other tools that systems programmers “maintained = got blamed for” at user sites)

When I worked at a computer vendor, there were clear distinctions among those who were concerned with:

• The operating system kernel (and then there are the drivers, the people who deal with them have to suffer from hardware dependencies)
• The operating system utilities
• Compilers
• Database systems (“simple” file systems fell under quasi-core OS; add a serious index or mention “b-tree” and you moved here)
• Networks

These distinctions were necessary and real. The coding disciplines, received knowledge, experience, and so on were different enough for these things that people could really be good at one and not at another. Unlike in the dreaded and despised world of end users, there was no clear hierarchy. While the kernel guys felt themselves at the center of the world, as in some sense they were, there was no denying the fact that compilers involved all sorts of special knowledge, and this got really annoying once OS’s started getting written in languages requiring compilers.

I suspect there are other reality-based classifications of code. But for these purposes, I would like to break all programming into some categories that cut across these and others I have heard of:

• True algorithms
• Abstract realizations of generic functions
• Application and device specific definitions and logic

Algorithms

Algorithms are the blocks of code studied so well in Knuth’s multi-volume series, The Art of Computer Programming. They are things like sorting, searching, managing lists, etc. There are lots of other sources, but he focuses on the key factors for code that implements algorithms. In spite of the impression that may be given by his many volumes, there are a relatively small number of algorithms, and it is important that a relatively small number of people like Knuth study them intensely, understand everything about them, and make them available (I’ll gladly pay!) in standard, canonic, guaranteed implementations. Most of the metrics in this paper are not relevant to algorithms.

Why is this? Because, for most algorithms, we continue to value the amount of space they use and the amount of time they take to execute. In increasing numbers of business applications, sub-optimal algorithms could be used without ill effect, at least in some places. But if an optimal algorithm is available, why not use it? Why make life complicated (not to mention error-prone) by introducing two classes of algorithms, the optimal ones and the good-enough ones – what’s to gain?

The life and analysis of algorithms in practice is by no means complete. There are important advances to be made. But most algorithms are advanced by someone solving a previously unsolved problem (like Dijkstra and the shortest path) or by taking advantage of a newly altered constraint (disk memory is now much cheaper than it used to be, therefore we can reasonably do X, which was a terrible idea in the days of expensive disk). Because of their different goals (minimization of space and time within constraints), algorithms play by different rules than other programs.

Abstract generic functions

Abstract generic functions implement things useful to many applications, but not specific to any one of them. Application “frameworks” are a mother lode of such functions. The Rails framework in Ruby is a reasonable modern example.

Any programmer who has written a variety of programs knows what I’m talking about here, and has his own candidates for inclusion. This has its origins in the FORTRAN FORMAT statement from Paleolithic times. Major categories of software products have emerged that started as application-specific code, moved on to be abstract generic functions, and then finally hardened into products. CRM and Content Management systems are recent examples. However, software products as currently typically designed and delivered aren’t nearly as useful or flexible as unified collections of abstract generic functions. The closest thing we have is application frameworks.

Application-specific definitions and logic

This is where everything else goes. This is the code and meta-data that makes your application your application, different from others.

Even this can and should be broken into layers of generality! For example, there should be one place for the common aspects of your UI design, and one place that defines what is unique to each particular interface.

The important thing is that, to the largest extent possible, you are mostly selecting and marshaling the general mechanisms available, and making them particular to the problem at hand.

Of course, your problem may have some highly unique elements. These elements may involve writing some serious code. The code may resemble in depth and complexity that of abstract generic functions. But you’d better just write it to solve the problem at hand, and leave generalization to another day.

Occam-optimality applies to all stages of the software life cycle

What the software is in itself is, of course, the core issue with software. It’s natural that Occam-optimality is mostly focused on software. But since software is usually written from needs and requirements of various degrees of formality, if those requirements aren’t Occam-optimal, the software may be occamal in some abstract way, given the requirements, but the requirements themselves may demand a solution that feels like there are all sorts of special widgets and do-hickey’s that could just as well be dispensed with. In other words, Occamality is best achieved if it is a conscious goal from the very start of the requirements process to the end.

The people building the requirements have got to buy into occamality, and I can tell you from personal experience that this is far from a given. Software requirements people frequently resemble the washing machine designers I have described who optimize what they do over a very narrow domain. The result is screens and functions that are highly customized to the job for which they’re intended. Every bit of customization adds that much more to what has to be programmed, tested, documented, taught, learned, used and maintained. This is typically a very large cost for an assumed benefit that often isn’t even there, because people find uniform programs easier to learn and use.

Downstream from the build phase, you mostly enjoy benefits from having occamal software. However, wherever you are building or creating something, there is a danger of non-occamality creeping in, and there is a further danger that it will be infectious and spread! Obviously when the program is changed to fix a bug or add a feature, there are benefits to be enjoyed if the program you’re working on is occamal, but there is also reason to be diligent that your change does not decrease the occamality of the program.

The usual approach to QA is highly redundant and un-optimal. When QA automation is normally performed, it is highly redundant, and leads to a system that doesn’t have better quality but is even harder to change. Typical script-based automation ends up having the effect of copying variables already present in the program in a different environment (the QA one), and creating logic that reflects and is derived from the base program. This approach to quality automation clearly reduces the quality of the overall program, and certainly reduces its occamality of the overall software effort The ideal approach is highly automated, fully supports occamal software, and indeed is occamal itself. See this for more details.

Occamality and other design principles

People use and defend the use of all sorts of methods and tools for building software. The number and range of methods and tools is overwhelming. Is Occam optimality yet another entrant to be added to the already-too-long list? I hope not. In fact, I hope that the notion of Occam optimality will be understood to be independent of all those methods and principles, and will provide a way to both improve each of them and, in some cases, select among them.

If the notion of Occam-optimality is correct, it stands above and cuts across all methods of building software. It is a principle that enables you to take two efforts to build a program in the same environment and tools, and say objectively which is better.

It is certainly true that some tools and methods make it easier to achieve Occam-optimality than others. But it is important to understand that Occam-optimality is not primarily about judging tools and methods – it is a notion of optimality that can be applied given any selection of tools and methods. Programs can be written in assembler language (assuming there’s a good macro pre-processor) that are Occam-optimal, and in fact it is easy to understand the concept in that environment, since everything is laid bare.

Focusing on the occamality of programs is a way of focusing on what we most care about in software. It doesn’t eliminate our other concerns – we still care that programs perform reasonably, that databases avoid corrupting data, etc. But an amazing number of those other concerns will be taken care of by-the-way if our programs are occamal, and if they’re not, we know that occamal programs are almost by definition the easiest to change to address any need or fill any gap.

I worked with a company having horrible problems with the administrative workstation of their storage product. It took a long time to load and many seconds to respond to reasonable-seeming requests. Its user interface was inconsistent and it took a very long time to update to reflect changes and new features in the underlying product of which it was a part. Looking under the covers, it turns out there were more than 200,000 lines of what turned out to be java code, and it was really hard to say what useful function large chunks of it were performing. Java was selected because the people thought it was modern, object-oriented, and would let them build re-useable components and other good things. Then they just started writing code, and here we were.

There were lots of arguments, pro and con, about whether java was any good, all of which were beside the point – wrong discussion. After a great deal of struggle, the whole thing was gutted and replaced with less than 20,000 lines of code and definitions. The people doing the work did not concentrate on performance, changeability, or other architectural concerns – their main goal was to make it small. When they were done, and they did it quickly, they had a complete replacement, a consistent user interface, and it magically became really fast and easy to change!

Occamality in Software History

Outside of databases, Occamal concepts have naturally emerged in software efforts from nearly the beginning of software, and limited formulations of Occamal concepts have been promoted and valued in software. The purpose of this section is to identify those early expressions for what they are; they show the widespread, ever-springing nature of the desire to come to some principle that will yield optimal programs.

One of the earliest pure-software efforts, the work to build the early language FORTRAN, was clearly a step in an Occamal direction. By creating a machine-independent language in the first place, programmers could concentrate on expressing their programs in a single language, FORTRAN, instead of the multiple machine languages that came to exist. Since FORTRAN is closer to the problem domain of most scientific programming, it usually takes fewer statements to express a program in FORTRAN than in assembly language; therefore, writing in FORTRAN eliminates redundancy and is thus better in Occamal terms. The relationship between the statements of the language and the machine language are all centralized in one place, the compiler. The compiler’s run-time library was just as important in this respect – it provided a set of commonly used functions that only needed to be written once, and then used whenever needed. In the early days of FORTRAN, for example, machines typically did not provide native support for floating point operations. The programmer could write his floating-point formulas and calculations without thinking about the machine, knowing that if the machine his program eventually ran on had no support for floating point, the compiler would generate the right calls and the run-time library would do the work.

All computer languages take programs in an Occamal direction. Just as FORTRAN abstracted and centralized the floating point operations commonly needed in scientific computing, so did COBOL abstract and centralize processing records of data and the BCD arithmetic needed for financial calculations.

Similarly, the concept of subroutine and shared subroutine libraries arose very early. One of the important functions performed by early software societies (IBM’s SHARE, the ACM and IEEE) was to collect, refine and standardize libraries of functions widely used by groups of people. Early software companies owed their existence to the time and money it took to build major functions, and built a business on selling many copies of software that was expensive to build, and less expensive for the user to buy than build for himself.

The use of macros is clearly driven by Occamal goals. The idea and motivation of a macro is simple: when there are repeating examples of a string in the text of your program that you suspect you may want to change in the future, you define the text in a macro and use the macro instead of the text itself. When the need for a change arises, you go to the single macro definition, change it, and you’re good to go, regardless of how extensively the macro is used.

The concepts of inheritance, templates and component re-use are clearly Occamal in nature. The idea is you write the common parts of a function in a master class or template, and then sub-class or apply the template to handle special cases or variations. You have the master class or template which gives you the single place to go to make changes, while benefiting from only needing to spell out the variation in each particular application. Using these things tends to reduce the number of lines of code and reduce the redundancy, and is therefore Occamal.

These examples make it clear that Occamality is a thread that weaves through the software industry and underlies many of its developments, although the people who created and promoted the developments generally did not think of them in these terms.

Occamality in Databases

The accepted practice of database schema design is a good application of Occamal principles. In fact, in a broad sense, Occam optimality takes the concepts accepted in schema design and applies them to programs and the software lifecycle as a whole.

Someone at a bank would write a system for automating checking accounts. Naturally, the account information would include the name and address of the person who owns the account. Some else would write a system for automating savings accounts. Naturally, the account information would be essentially the same; it may be the same because someone knew all about the checking system and copied it, or it might be the same because of “convergent evolution.” The account holder moves, and somehow tells the bank. The person at the bank has to look everywhere the person’s address might be stored and change it there. They might miss a place, or they may enter something incorrectly. This places a burden on the bank, makes extra work, is a source of errors, and may lead to customer dissatisfaction. It’s far better to have a single place where a person is identified at the bank, and where the things we know about that person (name, address, phone number, etc.) are stored. In the database world, this is the concept of “normalization,” which essentially means store each piece of unique data exactly once.

The database world also has the concept of “reference” in a couple of ways. If the person actually sets up a checking account, the checking account system needs the account information for the customer. This is done by identifying each account by a unique identifier, known as a “primary key.” The primary key is associated with the master copy of the account-holder’s information. Then, in each place where the customer uses a bank service, for example a checking account, a “foreign key” to the account information is placed. This is actually a copy of the primary key. So if you are customer 123, your primary key is 123, and the number 123 is also made the foreign key for your savings account, your checking account, etc. Calling it a “foreign key” says that it’s a reference to the one place where the information is stored, and not the master copy itself.

The database also uses references to eliminate redundancy in data types. When you need multiple fields that have to be distinct but actually represent the same kind of data, you define the type information once in something called a “domain,” and then each use of the domain is actually a reference to the master definition. If you change the domain, all the uses of the domain automatically change.

Databases represent Occamal principles in another important respect: statements in the data manipulation language (today, that means statements in SQL, for example SELECT data FROM tables WHERE conditions) are limited to what needs to be done, not how it is to be done. For example, if there is a JOIN between two tables, which should be used first? There is an excellent answer to that question, for example if one table is much smaller than the other, use the smaller table, etc. By abstracting how the join is implemented from the fact that you want a join, you get to define the how of joins in exactly one place, while requesting joins in many places. In earlier data manipulation languages, you could get the job done, but you would explicitly go first to the table you thought best to start from, and then go to the next. The trouble is, this puts the knowledge of how to navigate tables to get information in many places! This is bad because everyone has to learn it and apply it correctly, and if you want to change the method because you’ve figured out a better way to do it, you have to go through all the DML in the program and make individual decisions about how and what to change. Separating what we want from how to get it was definitely an advance, because now SQL could benefit from ever-improving execution routines without having to be changed! The point here is that exactly those benefits are the ones we would expect, because the change also made all programs that use SQL closer to being Occam-optimal!

It is true that many database implementations are Occam-suboptimal when considered in isolation. Because DBMS schemas are typically completely isolated from the programs that use them, programs that use databases (taking the database schema and stored procedures to be part of the program) are typically far from Occam-optimal. This is recognized in the RAILS framework of the Ruby language, which unifies the data definitions of the database and the program; this is the reason why programming is so much more efficient when using RAILS. Apart from RAILS, it is wonderful to have modern databases as an example of an island of software in which Occamal principles are accepted and valued, and the benefits widely enjoyed. By understanding the examples of good database practice in terms of Occam optimality, we can get an idea of how the principle can be extended to the rest of software.

 

Why should you build Occamal Programs?

There is a good deal of background and analysis to understand just how the concept of Occamality applies to the practical details of building programs. But before we get to that, perhaps it would be good to review the benefits we can expect. The core benefit of paring down a program to its bare minimum information content is pretty simple:

To the extent that any semantic concept is repeated anywhere, in any form, in the specification of the program, it is a redundancy that, if discarded, would reduce the cost of implementing the concept, the cost of one or more of the downstream components of the program, and the cost of subsequent modifications.

In other words, an Occamal program is one that is quicker than any other program that does the same thing to build, and is overwhelming the easiest to change, even though you did not think you were building “flexibility” into the program!

Let’s review briefly the lifecycle of software. At the beginning of the process for a whole new program or a change to an existing one, there is the need to create or change embodied in a set of requirements. From there, with variations depending on the programming shops’ methodology, we have roughly specifications (business, functional, high level, low level, etc.), design (various levels of focus and detail), code (perhaps with prototypes, code walk-throughs, etc.), test (various levels, sometimes performed by different groups of people using different tools), document (internal and external), train (the people who operate, administer, install, support and use), learn and use the software. This sequence may be linear or it may have iterative cycles. Once the program is in operation, there is an extended support cycle, with support, maintenance and bug fixing. There is typically a demand for new or altered features; these go through a version of a similar chain from requirements through building and use. It is generally recognized that (1) most efforts to build new programs crash and burn prior to roll-out, and (2) most of the money that is spent on a program is spend during its extended “tail,” namely the support and modify cycle. The difficulty of building brand-new programs increases the extent to which existing programs are modified, sometimes through multiple technology cycles.

Today, because occamality is not a familiar concept, it may take longer in practice to build programs that are occamal. Even when it is understood, redundancy frequently does not cost much to build into a program. In fact, it may be quicker to build a program with extensive redundancy (think copy and modify) than with less. But every superficial gain from redundancy is paid for, over and over again. Once we’re past the learning curve, building occamal programs should cost about the same as non-occamal ones (ideally, they would cost less to build), and the majority of the cost of software ownership should experience dramatic benefits. The benefits would mostly be due to the lower cost and lower risk of testing, documenting, learning, using, maintaining, and changing programs.

What exactly is it that makes programs hard to change? Once you understand basically what you have to do, it comes down to finding all the places in the program that will be affected by your change. The side-effects are always the toughest.

What if the change you wanted to make could be accomplished by changing exactly one place in the program? What if, to take a trivial example, you wanted to change the classic

Print (“Hello, world”);

By printing “bubba” instead of “world?” This would be easy, because you could just go to the single program line and make the change.

Now let’s take a well-known horrible example – the Y2K problem. This was the problem of modifying programs so they would continue to work when the year changed from 1999 to 2000. This was a problem because many programs did not use 4 digits to represent the year. For various reasons, usually involving an attempt to save space or time, year was represented as a two digit number in many programs, for example 99, with the leading two digits, 19, understood. If those programs had a single place where the number of digits in a year was represented, solving the Y2K problem would have been trivial – you would just go to each program, find the place where year was defined, change it, re-compile, check for problems, and you’re done. A quick change, low risk, no big deal. So Y2K was a problem not because of anything difficult or mysterious about dates – in fact, it could have been about any similar program change. Y2K was a problem simply and solely because the knowledge of the number of digits in the year of dates was expressed in many ways in many places, and it took a long time to find and fix them all with a substantial risk that some places were missed.

I completely admit, and I remember because I participated in the madness, that there were bizarre variations that made Y2K particularly gruesome. One example is “overloading,” which means that sometimes a programmer would use the value of “99” in the year field to mean “no date was given.” Y2K was a problem before the year 2000 because some programmers used “9999” (which could mean September 9, 1999) to mean “no more records in this sequence.” But all such cases just multiplied the basic fact: the number of digits in a year was expressed in many ways in many places, and in some of those places, other values or indicators were also stored.

So we want to have exactly one place in which the number of digits in date is defined. Notice that doesn’t mean we are limited to a single date. The key concept here is distinguishing between definition and use. Let’s add another level to make it clear. We want to define date in one place. Then we define, say, a couple of dates: birth date, marriage date, date of loan start, date of loan end. Each of these would be defined as types of dates. “Date” would have how many digits are in a year, and “birth date” would be a kind of date. We would define with “date” those things that are true of all dates, like number of digits in the year. We would define in “birth date” only those things that were true of “birth date” in particular, for example, the label to use when displaying it.

This is an elementary concept as I have illustrated it so far. What is different is the extent of the application of the concept that I propose. While it is (I hope) common practice for dates to be defined in this way within a program, it is certainly not common practice for all dates used anywhere in a program for any purpose to be defined in this way, as we saw big time in Y2K. For example, we have one way of defining dates in the database, another one for local storage in application programs, and yet another one for display and reports. While it is reasonable for there to be aspects of dates that some parts of programs care about that others don’t, for example the display label, it is not reasonable that all aspects that are common be defined redundantly – they should be defined exactly once.

More importantly, this concept applies to all aspects of the software lifecycle, not just the building phase. It even applies to documentation, training and using programs, because it means that there is less to document, train and learn, and when changes are made, the user’s common-sense notion that “something changed” actually applies – for example, if zip code is changed from 5 numbers to 5 numbers optionally followed by 4 numbers, that change is universal. The change, made in a single place, ripples though screens, applications, temporary variable definitions and databases.

Occamality directly addresses a lament frequently heard from programmers, about not being given enough time to do a job “right,” to design it for future flexibility. One of the many reasons good managers refuse to support a “good, flexible design” is that you always pay the price of the additional effort, and you rarely actually experience the benefit. That is simply because the designer tries to anticipate the nature of the potential future change, and make provisions for it. But of course, what usually happens is the anticipated change doesn’t happen and something else does, and the fact that the program is longer than it could have been because of the additional code that provides for un-needed flexibility makes the unanticipated change harder to make than it had to be.

Now that we understand what makes change hard – the fact that what we want to change is expressed in many ways in many places, all of which have to be found, understood, and correctly changed – we can design programs that are both as quick as possible to build and as quick to change in the future, regardless of the new requirement; we know that if we need to change something, there’s always one place to go to make the change.

The answer to why you should build Occamal programs is simple:

Occamal programs cost less to specify, design and build (once you are past the learning and acceptance curves), cost much less in the later stages of the software lifecycle where most of the costs are incurred, and enable changes and other forms of support and maintenance to be made with the greatest speed and the lowest cost and risk. And this is true regardless of language, operating system, application environment, and anything else about your software environment and tools.

 

How do you know if a given piece of software is good?

Software has a real problem. Let me explain.

While a huge gulf separates the novices from the experts in every field, I like to think that the widespread simple knowledge in most fields is like writing as taught to elementary school students. Fifth graders use the same alphabet that I use; while my vocabulary is more extensive and my use of grammar more elaborate, in both cases what I do extends and builds upon what the kids do. Fifth graders don’t need to unlearn the “bad” letters when they get to high school. Books for children can be well-written, and understood and enjoyed by both children and adults. Books for adults are supersets of children’s books in terms of vocabulary, sentence structure and experience. And while professional writers may not make extensive use of notes written on 3 by 5 cards the way English teachers used to make kids do in high school, the principles of organization are the same.

In software, however, the doctrines and received methods taught and practiced in the grades and, worse, among mainstream professionals, are simply inadequate to support doing a really good job; the best people don’t extend them, they ignore them and in various ways violate their principles. The typical methods for performing the quality assurance and testing functions, for example, are counter-productive. They don’t need correction – they need complete replacement. They focus on the wrong issues, they have the wrong concepts, and the most widely used methods and tools simply don’t get the job done!! The relationship between programs and data. Same. By reference and by value, implicit and explicit. Other books in this series go into detail for a couple of these subjects.

Things are so bad in software that the most experienced and accomplished people don’t even agree on what makes a program “good.” Given a set of programs that don’t crash and meet a set of requirements, how do you rank them in order of goodness? Is goodness proportional to how deep the inheritance trees are? How well documented they are? To what extent the evil “go to” mechanism is used? What is the test coverage, or the number of unit tests? How few machine resources they consume? Whether they’ve been written in this language or that?

I propose that there is, with a few common-sense qualifications, a measure of software goodness we can all agree to. I suggest this isn’t a new idea, but one that many people have sensed and acted on. The principle even makes common sense! The only thing it lacks is articulation and discussion. Here is a non-technical explanation of it. Here is one of my attempts to state the principle in a historic and more technical context. Here is a deeper explanation of the context and background.

This is not an abstract, nice-to-have concern. Unless you know what “goodness” is in a program, how can you measure whether you’re going in the right direction? Think about target shooting: without a target, how can you decide how close you are, and therefore how accurate a shooter you are? How can you compare two different shots?

Having an explicit agreement of what makes a program “good” is indispensable to making good software, and making it effectively and efficiently.

Occam’s Razor: the key to optimal software development

William of Occam (ca. 1285 to 1349) was an English logician and Franciscan friar. He is credited with formulating a principle that has been applied to various aspects of computer systems. In those areas of computing to which it has been applied, it reigns supreme – it supplies optimal solutions to the relevant problems.

There are large areas of computing to which Occam’s razor has not been applied. Worse, it is not even one of the candidates under consideration. As a result, those aspects of computing are fractured, inefficient, unpredictable, and driven by fashion and politics.

Everyone involved knows that the whole process of specifying, designing, building, testing and supporting software is hopelessly inefficient, unpredictable and error-prone. The leading methodology that concentrates on the process of building software, “project management,” is theoretically bankrupt and in any case has an empirical track record of failure. In terms of the content of good software, there are fierce battles among competing approaches, none of which is anything but a collection of unsupported and unfounded assertions, and which in practice don’t contribute to building good software.

Occam’s razor leads to the principles on which good software may be built, and supplies a single simple, widely applicable theme that, when applied, cuts away (as a razor should) all the inefficiency and generally what we don’t like about software. Once the principle is understood and widely applied, It should be the undisputed standard for how software is built, just as it has in the other areas of computing to which it has been applied.

Here is a shorter attempt to explain the Razor and its application to software. Here is an approach to the same problem from a common-sense layman's point of view.

The razor itself

“Occam’s razor” is a famous principle of thinking from the European Middle Ages. Occam’s razor is applied in situations where there is more than one reasonable explanation for a phenomenon, and basically says that you should pick the simplest explanation consistent with the phenomena you observe. From Wikipedia:

Leonardo da Vinci (1452–1519) lived after Occam's time and has a variant of Occam's razor. His variant short-circuits the need for sophistication by equating it to simplicity.

Simplicity is the ultimate sophistication.

Occam's Razor is now usually stated as follows:

Of two equivalent theories or explanations, all other things being equal, the simpler one is to be preferred.

As this is ambiguous, Isaac Newton's version may be better:

We are to admit no more causes of natural things than such as are both true and sufficient to explain their appearances.

In the spirit of Occam's Razor itself, the rule is sometimes stated as:

The simplest explanation is usually the best.

Other applications of Occam’s razor

While it plays little overt role in intellectual debates, in fact Occam’s razor and concepts derived from it are central to scientific thinking. So while you may not have ever heard of it, or may only vaguely remember having heard it in the past, you shouldn’t think that it’s an obscure little Medieval tidbit that has no relevance to the present that this guy is just pulling out for some weird reason. If you go through the Wikipedia reference, you find that:

Occam's Razor has become a basic perspective for those who follow the scientific method. … without the principle of Occam's Razor science does not exist. The primary activity of science, formulating theories and selecting the most promising theory based on analysis of collected evidence, is not possible without some method of selecting between theories which do fit the evidence. This is because, for every set of data, there are an infinite number of theories which are consistent with those data (this is known as the Underdetermination Problem).

You can find Occam’s razor in detail in astronomy, physics, biology, and medicine. It even appears explicitly in statistics.

There are various papers in scholarly journals deriving versions of Occam's Razor from probability theory and applying it in statistical inference, and also of various criteria for penalizing complexity in statistical inference. Recent papers have suggested a connection between Occam's Razor and Kolmogorov complexity.

It should be evident that Occam’s razor is an important underpinning of the whole scientific enterprise.

Applications of Occam’s razor in computing

We already understand Occam’s razor clearly in information theory, in which we measure the information content of a transmission. Information theory has provided the theoretical foundation of all communications and data transmission since its invention by Claude Shannon in 1948.

Occam’s razor is particularly applied in minimum message length (MML) theory, which focuses on the least number of bits required to encode a given amount of information. 

MML has been in use since 1968. MML coding schemes have been developed for several distributions, and many kinds of machine learners including: unsupervised classification, decision trees and graphs, DNA sequences, Bayesian networks, Neural networks (one-layer only so far), image compression, image and function segmentation, etc.

While the terminology is not normally used, and I am not aware that Occam’s razor was explicitly used to formulate it, it is clear that the principles are clearly expressed in modern relational database theory, and in the practice of schema design. I suggest that is one of the reasons for the success of the DBMS approach to data storage and access.

Shannon’s information theory

Shannon’s application of the concept to information theory provides a good springboard to seeing how it applies to software design. So let’s understand information theory a little better. I don’t think I can do better than Wikipedia. Here is a snapshot of the article on the history of information theory:

Claude E. Shannon (1916–2001) founded information theory with his classic paper "A Mathematical Theory of Communication," published in the Bell System Technical Journal in July and October of 1948. At the beginning of his paper, Shannon asserted that "The fundamental problem of communication is that of reproducing at one point, either exactly or approximately, a message selected at another point." His theory for the first time considered communication as a rigorously stated mathematical problem in statistics and gave communications engineers a way to determine the capacity of a communication channel in terms of the common currency of bits. This problem is called the channel coding problem. The transmission part of the theory is not concerned with the meaning (semantics) of the message conveyed.

A second set of ideas in information theory relates to data compression. Using a statistical description for data, information theory quantifies the number of bits needed to describe the data. There are two formulations for the compression problem -- in lossless data compression the data must be reconstructed exactly, whereas lossy data compression examines how many bits are needed to reconstruct the data to within a specified fidelity level. This fidelity level is measured by a function called a distortion function. In information theory this is called rate distortion theory. Both lossless and lossy source codes produce bits at the output which can be used as the inputs to the channel codes mentioned above.

This division of information theory into compression and transmission is justified by the information transmission theorems, or source-channel separation theorems that justify the use of bits as the universal currency for information in many contexts. …

Communications channels have a fixed capacity for sending information per unit of time. To use that capacity as efficiently as possible, you distinguish between the message that is presented for sending and the actual information content of that message. What this amounts to is saying that the “information content” of a message is the smallest number of bits required to exactly reproduce the message after transmission. In plain language, the “information content” of a message is the message compressed as much as possible, with all repeats and redundancy removed, except for redundancy purposely introduced for error identification and correction.

Like any application of Occam’s razor, the focus on information content comes out of knowing what you care about; you include everything required to get what you care about and throw out everything else. What I care about in communications is accuracy and efficiency. Efficiency means that I should transmit the most information through a communications channel of given capacity as possible. This can be achieved by using the smallest possible number of bits to encode the message. 

The application of Information theory to software

How is all this relevant to building software? Aren’t we supposed to use structured design, use cases, components, or whatever to design software? Most of those techniques focus on the process of a good design, or a supposed ideal structure for a program. If we apply the same thinking that we see in information theory to software design, we will focus instead on the results of the process, and once we are satisfied we know how to judge whether a computer program is optimal or not, we will be able to find ways to build it.

Software specification and design has clearly been an island of technical art, separated from science and technology as a whole, except for those portions that have already come under the sway of “Occamal” thinking such as information and MML theory. Like any relatively isolated island of theory and practice, a wide variety of techniques and practices arise to fill the gap left by a completely baseless, ad-hoc approach. There are innumerable approaches to software design and building and no clear way to decide among them.

When designing software, we care most about things that are a very close analogy to what we care about in communications. In communications, we have a channel of given capacity, and want to squeeze the most information through it we can; therefore, we eliminate all redundancy from the original message, and transform it to contain only its pure information content. Anything extra would take transmission capacity and add nothing. In software design, we normally have capacity to build a representation of the program (source code) that is as long as we would like in terms of the capacity of the computer to execute the code. However, every line of code adds a burden to one or more stages of the whole chain of specify, design, code, test, document, train, learn, use, maintain, modify, and repeat. This chain of effort, the entire lifecycle of the program and everyone who touches it, is like the communications channel in information theory. The idea is that most program specifications are highly redundant, like messages in their original form. We want to define the “information content” of programs just like we define the information content of messages, so that every redundant or repeating group is eliminated, and what is left is everything required to make the program operate and absolutely nothing else. We focus on information content in communications because we want to make the best use of fixed communications capacity; we focus on information content in software because we want to make the best use of all the resources that are involved with the software in any way.

We don't just want to make programs short for general reasons. We also want to make them easy to change, with as little effort and error as possible. By eliminating all redundancy from a program, we make it so that there's exactly one place to go to make a change.

Getting back to Occam (though I admit Shannon and fancy formulas are more impressive and intimidating), if Occam’s razor is:

Entia non sunt multiplicanda praeter necessitatem.

No more things should be presumed to exist than are absolutely necessary.

Occam’s razor applied to software would be:

No more software entities should be created than are absolutely necessary.

Why? Just as sending bits beyond the information content of the message has a cost but doesn’t improve the message that is received, so do additional entities in the specification or expression of the program add to the cost to build and the cost to change without increasing the value of the program.

This may not sound dramatic or exciting. But as someone who has lived with the process of building software for a long time, and has struggled extensively with the question of what makes software “good,” it’s very satisfying to have an objective criterion that enables you to judge the “goodness” of a program, and to say when a program is “optimally” good. It gets exciting when you realize that this abstract notion of optimality has consequences that are extremely practical and down-to-earth. In particular, it shows you a path to building programs more quickly than using any other method; doing less to build the programs than you thought you had to do; and having the resulting programs be as easy and safe to change as it is possible for programs to be.

Software Evolution: Build, Buy then Build

When a kind of software isn’t available, it makes sense to build it. If it is available to buy, everyone says you should buy it. But in a surprising number of cases, building makes sense even when you can buy software.

Here's an example of a company that wasn't in the software development business decided to build what they needed, and grew an amazing business from there.

There was a little bank in Georgia that somehow got the ambition to process credit cards for their customers when that was a new thing. There was no software available to buy, so they built some primitive software. It worked well enough. Then they improved the software and sold some local banks on processing credit cards for them (not necessarily in that order). One thing led to another, and the software evolved into the second-largest credit card processing company in the US, TSYS. Particularly when software categories are new, that’s what people do – they build software.

As the software becomes available, it starts to make sense to buy it. There are always issues at the beginning (see this about developing functionality on a new platform) about the time and effort to make the software work for you, and whether it has all the features you need. Sooner or later, the surviving vendors reduce the pain of customization and implementation, and develop a rich enough set of functionality so that the needs of nearly everyone in a business segment are met. At some point on that spectrum, it becomes kind of nuts to build it when you could buy it. It might take you years to build what you need, and then you’ll have to support it, and you’ll always be behind your competitors who are using off-the-shelf software, not to mention the risk that the software you try to build doesn’t end up working or really meeting your needs. So the mantra in the corporate offices becomes “buy, buy!” And only if you really can’t buy it should you build it; and even then, you should look real hard for excuses to put it off. If nothing else, there is less risk associated with buying software, and corporate managers nearly always vote for the least-risky of alternatives.

However, as these products have been getting built, tools and components for building software have also been advancing, and the ability to make your software systems adapt to changing business conditions has becoming increasingly important as a business advantage. Meanwhile, the major products in any category tend to get bloated, as pressures from various customer groups result in feature after feature being added to the product. In the early days of a product, you are most concerned that the features you need are in the product; as the product ages, the concern shifts to the burden on the software and users from the vast majority of the features in the product that someone may need, but not you. It may be that you only need a tiny fraction of the available features. An example of this that is close to home for many people is the Microsoft Office products – who even knows what most of the menu items in Word mean, much less uses them?

The case against purchased products gets stronger when you need several of them, and they all have to work together. Then you have the problem of application integration, which is typically compounded by different release and upgrade schedules from the vendors, who, try as they might, don’t seem to be able to release a new version without screwing something up that worked in the older version. If, on top of it all, you have an ambitious and low-cost programming group at your disposal, then you’re a good candidate for the final stage in this evolution, which is, “OK, I bought it and I’m using it, now when do I get to turn it off?” “Programming” shops whose main purpose in life is baby-sitting purchased products can’t even imagine doing something like this, which is probably just as well, because they are highly unlikely to pull it off. But groups who are itching to write code, and do stealth projects to prove what they can do, are another matter.

So the evolution returns to the starting point, only typically at a much more advanced level, with more advanced tools and a knowledgeable view of the business and the required functionality. “Why pay the vendors their ransom and put up with the crap they dish out, when I can write it, own it, change it any time I need to, and control my own destiny?” If the numbers work out, why not, indeed?

Software Evolution Spreadsheet Example

The story of the evolution of spreadsheets is well-known, though rarely studied. Spreadsheet evolution illustrates the patterns of software evolution. It shows the power of those patterns; if the leaders of Apple, Lotus and Visicalc had known and acted on the predictive power of those patterns, Office would not have been a Microsoft product, but one of theirs.

As a reminder, here are the basic patterns of software evolution:

Software is created for a new platform.

Software grows more capable on the platform with increasing abstraction.

The automation provided grows deeper.

The breadth of automation grows wider, doing more things.

The Spreadsheet products in evolutionary terms

The first popular spreadsheet program appeared on the then-new Apple computer. It was Visicalc. In terms of the stages of evolution, it was this:

• Emergence on a new platform: Conceptual Breakthrough
• Development on a platform: Basic product
• Automation depth: Recorder
• Automation breadth: Point product

If the platform became dominant and remained stable, we would expect competition by increasing the depth of automation on the Apple, increasing the breadth of automation e.g. by creating a collection of related products, or by moving along the same-platform development dimension. Instead, a powerful and popular new platform emerged, the PC. Visicalc themselves should have become the leaders in spreadsheets for the new platform, but as so often happens, the emergence of the new platform is an opportunity for a new, focused company to do it.

The company that won was Lotus, with their 1-2-3 product.

• Old platform: Apple I/II
• Old functionality: Spreadsheet, like Visicalc
• New platform: IBM PC/DOS and clones
• New functionality: Spreadsheet, much like Visicalc, along with some hype of other applications
• Outcome: They won big time

All the other categories (Basic product, etc.) remained the same.

Meanwhile, back in the land of Apple, the Mac came out, the first practical GUI on a personal computer platform. You would think this would be the big chance for Apple themselves to jump into the business, or for Lotus to convert their category-killing program. And of course, it was their big chance, but they blew it. Instead, Microsoft of all unlikely companies, jumped in.

• Old platform: Apple I/II, PC/DOS
• Old functionality: Character mode spreadsheet, like Visicalc and Lotus 1-2-3
• New platform: Apple Mac GUI
• New functionality: Excel Spreadsheet, but with attractive fonts and graphics
• Outcome: They won on the Mac

It wasn’t easy to get a spreadsheet right for GUI, but Microsoft kept plugging away and really took advantage of the graphics and the mouse.

At the same time, Microsoft was plugging away at their own clone of the Mac operating system for the PC, Windows, which they announced in 1975 and finally came out with a usable release, 3.1, in 1982 (seven loo-oo-oong years later – but that’s another story). Here we have a GUI platform for the PC, Lotus’ home base. The GUI had shown its attractiveness elsewhere. There was a GUI backer who was clearly going to see it through. You would think that the savvy business executive who was in charge of Lotus at the time, surrounded by a heavy-weight group of Cambridge thinkers, would have been all over this. Didn’t happen. While Apple built the GUI but failed to build the spreadsheet for it, and while Lotus built the spreadsheet but failed to convert it to a GUI on their own dominant platform, Microsoft built both the GUI and the spreadsheet.

• Old platform: Apple Mac, PC/DOS
• Old functionality: Mac GUI spreadsheet; PC/DOS character spreadsheet
• New platform: PC Windows
• New functionality: Same as on Mac
• Outcome: Huge win/win: PC/Windows helped Excel, Excel helped Windows

At the time, Lotus had much more revenue than Microsoft, and seemed reluctant to rewrite its industry-leading spreadsheet for Microsoft’s Windows, which was seen as a move that would help the success of that platform, and thus help Lotus’ competitor. But once it was clear that Windows was here to stay, Lotus finally did the re-write, and got 1-2-3 onto Windows. So now 1-2-3 and Excel were competing as point products on the same technology platform.

But now Lotus had another problem: 1-2-3 was a point product (I know, I know, they said they had three applications, thus the “1-2-3,” but who used the “2-3”?), and by this time, Microsoft had an actual product suite: Microsoft Office, combining spreadsheet with word processing and presentations via OLE and drag-and-drop. On the movement from point product to product collection to product suite, Microsoft was consistently a step ahead, and thus had more market pull than Lotus. While Lotus was plugging the virtues of 1-2-3 vs. Excel, Microsoft was promoting the value of its product collection. Then, while Lotus was promoting the value of its acquired collection of office products, Microsoft had a real product suite. The end result, we know today, is the complete dominance of the Microsoft Office product suite.

Conclusion

The evolution of the spreadsheet fits nicely into the general patterns of software evolution I have described. If strategists and company leaders had been aware of these patterns and acted on their proven predictive power, the outcome of the spreadsheet wars would have turned out differently. Leaders today would be well-advised to understand and apply the patterns of software evolution.

Summary: Computer Data Centers and Networking

This is a summary with links to my posts on computer data centers and networks. The two subjects are intimately related because the whole point of networks is to connect computers with each other.

Like everything else in computing, fashions have a strong impact. A bit over a decade ago, the world of computing started yammering about “the cloud” and “virtualization.” These things were the hot subjects all the cool kids talked about. If you weren’t driving towards moving to the cloud, you were obsolete. The reality of course was much different.

https://www.blackliszt.com/2011/12/the-name-game-of-moving-to-the-cloud.html

The Cloud is just another virtue-signaling fashion word like Big Data.

https://www.blackliszt.com/2012/04/im-tired-of-hearing-about-big-data.html

The fact that the underlying reality hardly changes at all is way beyond the technical knowledge of the vast majority of the people who talk about it.

https://www.blackliszt.com/2012/05/the-cloud-and-virtualization.html

When something like “the cloud” heats up, all the vendors related to it rush to promote their products as ideally suited to the new thing.

https://www.blackliszt.com/2013/04/storage-vendors-in-the-cloud.html

The managers of large in-house data centers rarely know much about what they’re buying and what the alternatives could be. As a result, they often spend way more than they need to on equipment, something which competing cloud vendors are less likely to do.

https://www.blackliszt.com/2014/08/data-center-managers-spend-too-much-on-equipment.html

Like anything where people are involved, there are politics and fights over who should perform a given function. One of the classics is making replications of the data in DBMS’s for safety purposes. The data center  people want to use their mechanism and the DBMS people want to use theirs. There is a clear right answer, but it only wins sometimes.

https://www.blackliszt.com/2014/03/replication-good-idea-storage-replication-nah.html

One of the key things people who write applications want is for them to “scale,” i.e., be able to handle any load without slowing down. The way some systems applications are built, based on decades-old designs, can get in the way. But there are solutions.

https://www.blackliszt.com/2014/02/obstacles-to-scaling-centralization.html

People also want their applications to continue to be available in the event of data center failure. For people working in data centers, there are perverse incentives at work.

https://www.blackliszt.com/2010/04/is-your-site-working-do-you-really-care.html

To be fair to data center managers, applications originally written decades ago expect to be able to run in modern data centers. It’s tough!

https://www.blackliszt.com/2010/01/paleolithic-mainframes-discovered-alive-in-data-center.html

Way back in 2015, it was clear that hardware had evolved to make dramatic improvements possible in software.

https://www.blackliszt.com/2015/08/the-data-center-of-the-future.html

With software people ignorantly focused on network-connected services as the way to build scalable applications, they assure that the incredible hardware power will be minimally utilized.

https://www.blackliszt.com/microservices/

Network neutrality was a hot topic and still comes up. The idea is that everyone should be charged the same for internet access and services. When you dig into the subject with actual knowledge of the technology, you see that the whole furor by virtue-signaling ignorance.

https://www.blackliszt.com/2014/11/net-neutrality-it-aint-broke-dont-fix-it.html

One of the arguments made about net neutrality and legal privacy provisions was unusually divorced from reality.

https://www.blackliszt.com/2017/04/on-the-internet-youre-naked-and-for-sale.html

Years later, a weak form of net neutrality was repealed. There were demonstrations with US Senators making predictions about the disaster that would ensure. Here’s an analysis of the non-disaster a year later.

https://www.blackliszt.com/2019/06/the-aftermath-of-the-net-neutrality-disaster.html

Summary: The Medical-Industrial Complex

Modern medicine can do wonderful things. I benefited from remission of an extremely rare form of cancer that was made possible by advances in the last couple of decades. At the same time, a great deal of what is done in medicine is controlled by the Medical-Industrial Complex, which causes untold waste and harm. These posts document the tip of that iceberg as my health journey has led me.

I've long had a concern that what doctors do often doesn't follow the clear evidence. For example, here's a case of blatantly ignoring standard practice with something simple.

https://www.blackliszt.com/2016/12/what-can-cats-teach-us-about-healthcare.html

No big deal. But then I encountered something far more serious. I tell the start of my story here, about heart symptoms caused by blood pressure pills.

https://www.blackliszt.com/2022/07/the-destructive-treatment-of-hypertension.html

The next part of my story is when I discovered there’s a large study demonstrating that taking blood pressure pills more than doubles your chances of getting AMD, which makes you blind.

https://www.blackliszt.com/2022/07/blood-pressure-pills-can-make-you-blind.html

It’s a study that none of the relevant doctors ever tells you about.

I backed up and studied hypertension. Expert opinion on the subject is united.

https://www.blackliszt.com/2022/05/the-experts-are-clear-control-your-blood-pressure.html

When you dig past the pronouncements of authorities, you discover that hypertension isn’t even a disease.

https://www.blackliszt.com/2022/06/the-facts-are-clear-hypertension-is-not-a-disease.html

So what happens when you stop taking the pills? At least in one anecdotal case, things change in good ways, and in any case, no harm.

https://www.blackliszt.com/2022/11/how-to-cure-amd-macular-degeneration.html

More digging led me to the bogus, incredibly destructive diet-heart hypothesis, proven false by the evidence but promoted on food packages and everywhere else.

https://www.blackliszt.com/2021/12/trusting-science-the-whole-milk-disaster.html

The idea is that eating red meat and full-fat dairy leads to heart disease.

https://www.blackliszt.com/2022/02/the-experts-are-clear-dont-eat-much-saturated-fat.html

When you dig past the propaganda, you learn that saturated fat is an essential and healthy part of your diet.

https://www.blackliszt.com/2022/03/the-facts-are-clear-eat-lots-of-saturated-fat.html

Eating saturated fat is supposed to increase your blood cholesterol, which leads to heart trouble, they say.

https://www.blackliszt.com/2022/02/the-experts-are-clear-keep-your-cholesterol-low.html

The widely-prescribed drugs that lower cholesterol don’t help with heart disease and cause problems of their own.

https://www.blackliszt.com/2022/04/the-facts-are-clear-dont-take-cholesterol-drugs.html

There is strong evidence that bad diet recommendations based on the bogus diet-heart hypothesis is a leading cause of the obesity epidemic that continues to worsen.

https://www.blackliszt.com/2022/04/the-forbidden-question-what-caused-the-obesity-epidemic.html

The current recommendations for diet and medical treatment of obesity continue the madness.

https://www.blackliszt.com/2022/09/the-medical-treatment-of-obesity.html

Many people receive reminders to get things like blood pressure and cholesterol checked so that drugs can be prescribed “if necessary.”

https://www.blackliszt.com/2023/02/be-healthy-and-dont-schedule-your-heart-health-visit.html

Disease Prevention and Testing

Disease prevention sounds like a great idea. So does early detection of bad things. When you dig into the evidence and the numbers, a different picture emerges.

Screening for colon cancer is a multi-billion dollar industry. The only large-scale study that’s ever been done shows that it doesn’t result in longer life.

https://www.blackliszt.com/2023/01/value-of-colon-cancer-screening.html

Everyone is supposed to get an annual flu shot. The CDC’s own numbers and massive studies show that you’re better off without it.

https://www.blackliszt.com/2022/12/flu-shots-propaganda-reality.html

Vaccine efficacy is often mentioned. Its technical meaning can be found, but the authorities rarely mention it. A large efficacy can still mean that you only have 1 chance in a hundred of being helped by the drug.

https://www.blackliszt.com/2022/09/does-vaccine-efficacy-of-95-mean-i-wont-get-sick.html

Your chances of being helped (NNT, Number Needed to Treat) must be considered along with the chances of being harmed, a thing that is too-often ignored.

https://www.blackliszt.com/2022/09/nnt-for-benefits-and-for-harms.html

The studies that are supposed to show treatment effectiveness are too often biased, and (shockingly) backed by data that is kept secret by law. If the treatment is wonderful, why keep the data secret?

https://www.blackliszt.com/2022/11/revolutionize-health-by-making-medical-data-and-studies-open-source.html

There is a proven path to make drug discovery and testing a quantum leap better. All the authorities and experts ignore it.

https://www.blackliszt.com/2017/01/using-software-methods-to-speed-drug-discovery.html

Given the results, the common-sense idea of wellness visits stops making sense in most cases.

https://www.blackliszt.com/2023/02/be-healthy-and-schedule-your-annual-wellness-visit.html

Of course there is a great deal more to be said on this subject. There are true experts, far more qualified than I am, some of whom are referenced in the above posts. My intention is these posts was to detail the journey that a normal patient took from trusting and naive to the opposite. Again, there are wonderful benefits for patients from doctors and hospitals; but not everything that is recommended is wise to take/do.

Summary: The Business of Healthcare, Medicine and Insurance

After spending years learning about the software used in medical offices, hospitals and insurers, I came to the conclusion that healthcare technology is the trailer park of the computer world.

I have since realized that I was wrong in the comparison; trailer parks actually work to house their inhabitants, effectively and inexpensively, unlike most medically-related computer systems. Trailer parks of the world, I humbly apologize.

https://www.blackliszt.com/2015/12/healthcare-problems-and-opportunities.html

 

Healthcare Computers and Software

Hospitals have trouble keeping their computers working. When they fail, they often try to hide the failure, which puts patients at risk.

https://www.blackliszt.com/2015/05/healthcare-it-disfunction-the-secret-computer-outage-at-mount-sinai-hospital.html

In contrast, other institutions let their customers know about computers failures and even apologize for the inconvenience.

https://www.blackliszt.com/2015/05/computer-troubles-at-the-hospital-and-at-the-symphony.html

The contrast between putting meals on the table and assuring that data is available for use on providers’ screens couldn’t be more stark.

https://www.blackliszt.com/2015/05/meals-at-downton-abbey-and-it-in-healthcare.html

Even when their computers are working, the people in charge brag about the fancy new things they’re doing instead of assuring that basic processes like patient scheduling work as intended.

https://www.blackliszt.com/2014/07/innovation-made-simple.html

There’s no doubt that hospital computer systems are a horror show that hurt patients and that managers try to hide. But there’s a far worse largely hidden problem leading to patient deaths.

https://www.blackliszt.com/2017/05/hospital-computer-disasters-and-iatrogenic-disease.html

Is making systems work reliably and well a wide-spread problem? Yes. Are there proven solutions? Yes. Hospitals do an outstanding job of ignoring them.

https://www.blackliszt.com/2023/04/summary-software-quality-assurance.html

 

Electronic Medical Records (EMR)

The data about your health and health history are essential for guiding medical interactions of any kind.  The quality of the data is essential. All too often, the data is wrong or missing.

https://www.blackliszt.com/2016/06/healthcare-innovation-emrs-and-data-quality.html

Most medical organizations have EMR’s and also loads of paper records, which both patient and provider fill out.

https://www.blackliszt.com/2016/06/healthcare-innovation-emrs-and-paper.html

Have you ever tried to get your own medical records from the institution that has them – your records?? Nightmare.

https://www.blackliszt.com/2016/06/healthcare-innovation-getting-our-data-out-of-emr-prison.html

They all say they have easy-to-use systems to enable patients to get their own records. Hah. Just try it.

https://www.blackliszt.com/2018/01/who-owns-your-health-data.html

Here are details of the horrible obstacles a major systems places between the patient and his “easy-to-access” data.

https://www.blackliszt.com/2018/10/medical-testing-getting-the-results.html

https://www.blackliszt.com/2018/10/medical-testing-getting-the-results-2.html

Most people have been to more than one institution, each with its own records about you. EMR interchange is supposed to make it easy for the place you’re visiting to get your records from other places.

https://www.blackliszt.com/2016/10/healthcare-emr-why-portability-matters.html

https://www.blackliszt.com/2016/07/healthcare-innovation-how-to-achieve-emr-interchange.html

EMR interchange works? Patients sometimes have to go to ridiculous lengths to get essential data from where it is to where it’s needed.

https://www.blackliszt.com/2016/09/healthcare-emr-interchange-breakthrough.html

The bottom line is that medical institutions should prioritize giving patients their data. There should be an EMR app for your data just like there are money apps like Venmo and Zelle for your money.

https://www.blackliszt.com/2016/01/i-want-an-emr-app.html

One of the many reasons for EMR nightmares is that the software is a huge problem and “just” buying and installing it is a job of “epic” proportions.

https://www.blackliszt.com/2016/05/healthcare-innovation-emr-procurement-is-broken.html

 

Healthcare Business: Scheduling, Testing, Billing and Payment

Medicine is all about helping people get and stay healthy, but it’s also a business.

https://www.blackliszt.com/2018/07/medicine-as-a-business-overview.html

Scheduling, billing and payment are different for medical care than any other service we require. Here are highlights of the differences.

http://www.blackliszt.com/2018/07/medicine-as-a-business-billing-overview.html

You need to get a simple test like an X-Ray? No problem. Bad data at the provider, the insurer and the EMR will ensure that lots of time is wasted.

https://www.blackliszt.com/2017/01/my-cat-taught-me-about-the-state-of-healthcare-provider-data.html

Scheduling sounds simple, right? Maybe for a cat at a vet, but not for a person at a hospital.

https://www.blackliszt.com/2019/04/whats-wrong-with-medical-scheduling-and-why-it-matters.html

The amount of time and effort on everyone’s part to schedule a medically-required MRI is amazing. Without patient persistence, it won’t happen.

https://www.blackliszt.com/2018/09/medical-testing-scheduling-the-test.html

Covid testing illustrated the widespread problems with scheduling – it was about the systems and software, not Covid.

https://www.blackliszt.com/2021/12/the-nightmare-of-covid-test-scheduling.html

Once you’re scheduled, you have to go to the right place, which can be its own adventure. Fortunately, you can usually rest afterwards by filling out reams of paperwork.

https://www.blackliszt.com/2016/10/hospital-wellness-innovation-breakthrough.html

For costlier tests like MRI’s, the extent of useless, costly processes is mind-boggling.

https://www.blackliszt.com/2018/09/medical-testing-doing-the-test.html

MRI’s are an amazing technology. The reports from the specialists are something else again. It turns out that in spite of endless years of training, there are no standards for reading and reporting – even “simple” things like the size of a tumor.

https://www.blackliszt.com/2018/11/medicine-as-a-business-medical-testing-5-the-results.html

What I’ve described aren’t one-off’s. The problems from scheduling to getting the results are business-as-usual in the world of medicine.

https://www.blackliszt.com/2022/02/medicine-as-a-business-medical-testing-6-another-test.html

And then there’s billing. I went for an MRI, and got bills from different places with bad information and a broken online payment system.

http://www.blackliszt.com/2018/07/medicine-as-a-business-billing-1.html

The second bill was also a mess of bad data, with an offer to pay online but the bad data prevented me from paying. So I wrote a check.

http://www.blackliszt.com/2018/07/medicine-as-a-business-billing-2.html

When you dive into the insurance payment part, the nightmare morass of bad data and bad software is overwhelming. Business as usual.

http://www.blackliszt.com/2018/07/medicine-as-a-business-billing-3-insurance.html

My health insurer announced a major new service to help patients pay provider bills. Strike out.

https://www.blackliszt.com/2019/02/giant-health-insurer-reveals-patient-billing-break-through-or-is-it-a-break-down.html

Here is a summary of what’s wrong with medical billing and insurance payments.

https://www.blackliszt.com/2018/08/medicine-as-a-business-billing-4-whats-wrong.html

 

Medical Costs

Medical costs are outrageous. They don’t have to be.

Hospitals could stop spending money on useless fancy innovations and concentrate on simplifying and automating the everyday nuts and bolts of the business.

https://www.blackliszt.com/2014/07/innovation-made-simple.html

Removing the regulatory straight-jacket on medical equipment would be huge.

https://www.blackliszt.com/2016/12/regulations-that-enable-innovation.html

https://www.blackliszt.com/2020/05/heres-how-the-fda-can-reduce-medical-device-costs-while-improving-healthcare.html

https://www.blackliszt.com/2023/01/how-to-reduce-the-cost-of-medical-imaging-and-pacs.html

Some of the same principles apply to drug discovery and costs.

https://www.blackliszt.com/2017/01/using-software-methods-to-speed-drug-discovery.html

https://www.blackliszt.com/2022/11/revolutionize-health-by-making-medical-data-and-studies-open-source.html

 

Health insurance

Health insurance companies maintain a huge trove of your personal data. They are incapable of keeping it secure, and bungle the response when there’s a major breach they can’t cover up.

https://www.blackliszt.com/2015/02/my-anthem-account-was-hacked.html

https://www.blackliszt.com/2015/02/the-anthem-of-cyber-insecurity.html

Health insurers think providing patients incentives to keep healthy is a win all around. Sadly, their incentives tend to be embarrassing failures. Here’s an example with giving a Prepaid debit card.

https://www.blackliszt.com/2018/11/patient-incentives-in-healthcare.html

The marketing departments of health insurers want to communicate with patients to make good things happen. Most of them appear never to have heard of the concept of personalization.

https://www.blackliszt.com/2019/02/patient-outreach-by-health-insurers-case-study.html

https://www.blackliszt.com/2022/04/health-insurance-company-tries-to-keep-me-healthy.html

The same marketing gurus reach out to patients for feedback and reveal that they flunked Marketing 1.01

https://www.blackliszt.com/2021/05/anthem-needs-my-feedback-reveals-deep-problems.html

Here’s an example of why they get terrible response rates when asking for customer feedback.

https://www.blackliszt.com/2019/02/adventures-with-health-insurance-software-customer-feedback.html

Every serious company has to have an app, right? It makes sense that the insurer would spend piles of money to build one that customers hate.

https://www.blackliszt.com/2021/02/why-cant-big-companies-build-or-even-buy-sofware-that-works.html

 

AI and fashionable algorithms in Medicine

There are many examples of AI in healthcare in this summary of AI in general.

https://www.blackliszt.com/2023/07/summary-ai-machine-learning-big-data-math-optimization.html

Here are some of the AI posts relevant to healthcare. A major theme of AI in healthcare is that well-paid managers are desperate to keep up with the AI fashion, and spend lots of money and generate lots of publicity about how wonderful they are. The trouble is that there are simpler, common-sense things that are fully understood that could be done right away that would yield major improvements in patient health; these things are too often ignored or neglected.

https://www.blackliszt.com/2015/07/cognitive-computing-and-healthcare.html

https://www.blackliszt.com/2015/08/human-implemented-cognitive-computing-healthcare.html

https://www.blackliszt.com/2016/05/healthcare-innovation-can-big-data-and-cognitive-computing-deliver-it.html

https://www.blackliszt.com/2016/09/healthcare-innovation-from-washing-hands-to-ai.html

https://www.blackliszt.com/2017/06/how-to-avoid-cutting-off-breasts-by-mistake.html

https://www.blackliszt.com/2018/08/getting-results-from-ml-and-ai-4-healthcare-examples.html

Summary: Computer Software History and Evolution

This is a summary with links to my posts of the history and evolution of computer hardware and software.

In computers and software, history is ignored and its lessons spurned. What little software history we are taught is often wrong. Everyone who writes or uses software pays for this, and pays big. This is true at a large scale for computers.

https://www.blackliszt.com/2012/03/computer-history.html

Software history can be spectacularly bad about important things.

https://www.blackliszt.com/2019/08/the-comes-before-is-not-causation-fallacy-in-software-evolution.html

Bad software history is also rife on a smaller scale, for example cryptography.

https://www.blackliszt.com/2020/10/elizebeth-smith-friedman-the-cancelled-heroine-of-cryptography.html

Software history is largely ignored.

https://www.blackliszt.com/2015/06/dont-know-much-about-history.html

When you learn math, you are learning its history. When you get to calculus, you’re essentially up to the 1600’s. You build from the ground up. Not in computing, to its detriment.

https://www.blackliszt.com/2015/05/math-and-computer-science-vs-software-development.html

Studying history would show modern software people who think they’re solving big problems that the problem is decades old and that the box of failed solutions is overflowing.

https://www.blackliszt.com/2021/08/the-crisis-in-software-is-over-50-years-old.html

Software development is a giant mess. Studying its history and the attempts to clean up the mess is an essential part of constructive change.

https://www.blackliszt.com/2022/04/how-to-fix-software-development-and-security-a-brief-history.html

The history of other fields can have lessons that are highly relevant for software.

https://www.blackliszt.com/2012/07/what-can-software-learn-from-steamboats-and-antiseptic-surgery.html

https://www.blackliszt.com/2014/02/lessons-for-software-from-the-history-of-scurvy.html

https://www.blackliszt.com/2019/02/what-software-experts-think-about-blood-letting.html

Some aspects of computer history are so important and relevant that, when understood and applied, they make dramatic improvements to software development.

https://www.blackliszt.com/2013/12/fundamental-concepts-of-computing-speed-of-evolution.html

Software Evolution

Software evolves. But it evolves differently than other things studied by science. Virtually no one studies software history in any way, much less the patterns of evolution that become apparent when you study that history.

https://www.blackliszt.com/2019/06/the-evolution-of-software.html

Why should anybody bother to study software evolution? Knowing software evolution can help you predict the future! How? Software does not evolve the way most people think it does

https://www.blackliszt.com/2019/09/understanding-software-evolution-helps-you-predict-the-future-of-software.html

When studying software evolution it’s important to understand the underlying principles of automation – the things that computers and software are capable of doing.  The fundamentals will tell you WHAT will happen, but not WHO or WHEN.

https://www.blackliszt.com/2020/01/the-fundamentals-of-computer-automation.html

The way you learn software evolution is to study both its history in general and how it has been used in different industries and domains.

https://www.blackliszt.com/2014/04/continents-and-islands-in-the-world-of-computers.html

One clear pattern of evolution is that a kind of software will be established in one domain and then slowly, sometimes over decades, appear in other domains, one by one.

https://www.blackliszt.com/2019/08/the-slow-spread-of-linear-programming-illustrates-how-in-old-vation-in-software-evolution-works.html

A similar pattern often occurs with simple kinds of software.

https://www.blackliszt.com/2019/09/simple-data-entry-technology-illustrates-how-in-old-vation-in-software-evolution-works.html

A peculiar aspect of software evolution is software fashions. They appear, grow strongly and often fade away as their awfulness becomes hard to avoid. Then they come back, usually re-named.

https://www.blackliszt.com/2019/05/recurring-software-fashion-nightmares.html

A typical result of a fashion-driven software trend is to increase costs without delivering benefits, which can happen even when it involves an intrinsically good thing like workflow automation.

https://www.blackliszt.com/2019/09/laser-disks-and-workflow-illustrate-the-insane-fashion-driven-nature-of-software-evolution.html

One of the forces of evolution is resistance to automation.

https://www.blackliszt.com/2020/01/luddites.html

Software Programming Language Evolution

Software people love to talk about progress and innovation. Most of what they claim as progress is little but milling around in a small box.

https://www.blackliszt.com/2020/09/software-programming-languages-50-years-of-progress.html

There is an odd evolution in programming languages, in which the relationship between data definitions and programs cycles from inside to outside the program and back.

https://www.blackliszt.com/2015/06/innovations-that-arent-data-definitions-inside-or-outside-the-program.html

However, there have been a couple major advances in the evolution of programming languages.

https://www.blackliszt.com/2020/09/the-giant-advances-in-software-programming-languages.html

While some truly powerful advances have become standard, others have been thoughtlessly discarded.

https://www.blackliszt.com/2021/09/software-programming-language-evolution-structures-blocks-and-macros.html

Other powerful advances, widely used for a time in the market, ended up being ignored or abandoned by academia and industry in favor of a productivity-killing combination of tools and technologies.

https://www.blackliszt.com/2020/11/software-programming-language-evolution-beyond-3gls.html

The emergence from academia of “structured programming” and the associated effort to find all GOTO statements and burn them at the stake was a particularly shameful instance of the evolution of programming languages.

https://www.blackliszt.com/2021/09/software-programming-language-evolution-the-structured-programming-goto-witch-hunt.html

While loads of people focus on language, programming tools that make huge contributions to the productivity of programmers have evolved: libraries and frameworks.

https://www.blackliszt.com/2021/10/software-programming-language-evolution-libraries-and-frameworks.html

Some programming languages that were supposed to be so much better than older ones led to major failures. But failures are expected in the normal world of software development, so the bad new languages kept marching along.

https://www.blackliszt.com/2021/01/software-programming-language-evolution-credit-card-software-examples-1.html

The small world of functional software languages has an interesting history.

https://www.blackliszt.com/2021/05/software-programming-language-evolution-functional-languages.html

Software Applications and Systems Evolution

One of the recurring evolution patterns is that functionality emerges on a new platform in roughly the same order as it emerged on earlier platforms. The timescale of the emergence may be compressed; the important aspect of the pattern isn’t the timing but the order. Here is an explanation of the general concept and how it worked for operating systems.

https://www.blackliszt.com/2020/10/software-evolution-functionality-on-a-new-platform.html

Here is how the same pattern of functionality on a new platform worked out in security services.

https://www.blackliszt.com/2020/11/software-evolution-functionality-on-a-new-platform-security-services.html

Here’s the pattern of existing functionality on a new platform as seen for remote access software.

https://www.blackliszt.com/2020/11/software-evolution-functionality-on-a-new-platform-remote-access.html

Just because you’re building a version of existing functionality for a new platform doesn’t mean you’ll succeed. You can screw it up in various ways, including being too early – if the market doesn’t think it has a problem, it won’t buy a solution.

https://www.blackliszt.com/2020/11/software-evolution-functionality-on-a-new-platform-market-research.html

Transaction monitors are a classic example of the pattern of functionality emerging on a technology platform, and then emerging in pretty much the same way in the same order on other platforms.

https://www.blackliszt.com/2021/02/software-evolution-functionality-on-a-new-platform-transaction-monitors.html

Once an application appears on a software platform, there is a consistent way the category of applications evolves on that platform. It goes from a custom application to a basic product, then parameters are added and finally a workbench.

https://www.blackliszt.com/2020/01/the-progression-of-abstraction-in-software-applications.html

Here are examples of the progression from prototype through increasing levels of abstraction.

https://www.blackliszt.com/2021/03/the-progression-towards-abstraction-on-a-software-platform-examples.html

Software automates human effort to varying degrees. One dimension of automation is depth, in which software evolves from recording what people do through helping them and eventually to replacing them.

https://www.blackliszt.com/2021/04/the-dimension-of-software-automation-depth.html

There is surprising pain and trouble of going from one stage of automation depth to the next. Unlike the progression to increasing levels of abstraction, customers tend to resist moving to the next stage of automation for various reasons including the fear of loss of control and power.

https://www.blackliszt.com/2021/05/the-dimension-of-software-automation-depth-examples.html

Here’s how the automation depth pattern plays out in “information access,” the set of facilities that enable people to find and use computer-based information for decision making.

https://www.blackliszt.com/2021/12/the-dimension-of-automation-depth-in-information-access.html

The patterns of automation play out in multiple dimensions. In addition to automation depth, there is automation breadth.

https://www.blackliszt.com/2021/11/the-dimension-of-software-automation-breadth.html

Here are a couple of clear examples of the evolution of software applications to increasing automation breadth.

https://www.blackliszt.com/2022/07/the-dimension-of-software-automation-breadth-examples.html

The evolution of spreadsheets is a good example of how these patterns work out in history, and clearly demonstrate the predictive power they have for anyone who cares to look.

https://www.blackliszt.com/2023/09/software-evolution-spreadsheet-example.html

As software evolves in these ways, it becomes increasingly nuts for a potential customer to build it for themselves. However, there are a surprising number of cases where a smart user can build a private version of commercially available software and win with it.

https://www.blackliszt.com/2023/09/software-evolution-build-buy-then-build.html

You would think that user interfaces would be thoroughly understood. When you look at a variety of UI’s you see that the underlying principles are clear but usually ignored.

https://www.blackliszt.com/2020/12/software-evolution-user-interface.html

https://www.blackliszt.com/2021/01/software-evolution-user-interface-concepts-whose-perspective.html

https://www.blackliszt.com/2020/01/how-to-design-software-user-interfaces-that-take-less-time.html

Does Software Always Evolve?

There is a large body of core software that evolves extremely slowly.

https://www.blackliszt.com/2023/07/does-software-evolve-rapidly.html

Once a kind of software gets built, it tends to live long past the problem for which it was the solution. Data was so big you needed storage. Special SW invented to handle it. Now it will all fit in memory (for most applications). But the SW and code practices live on!

https://www.blackliszt.com/2010/09/databases-and-applications.html

Yes, they’ve been obsoleted by modern hardware – but needed by software.

https://www.blackliszt.com/2010/01/paleolithic-mainframes-discovered-alive-in-data-center.html

While there are things that change in software and management, a surprising number of common things just get new names. The “cloud” is a typical example.

https://www.blackliszt.com/2011/12/the-name-game-of-moving-to-the-cloud.html

Another example is pure software, shown with SOA and micro-services.

https://www.blackliszt.com/2021/03/micro-services-the-forgotten-history-of-failures.html

One of the explosive growth areas of the internet boom was the invention and spread of social media. Most of the people who look at it ignore its deep roots in non-digital media.

https://www.blackliszt.com/2018/10/social-media-has-a-long-history.html

If and when serious study of computer software history and evolution finally starts to take place, perhaps Computer Science will start on the path to being, you know, scientific. And normal software development will stop being dominated by fashions.

https://www.blackliszt.com/2023/04/summary-computer-science.html

https://www.blackliszt.com/2023/07/summary-software-fashions.html

 

Summary: Software Architecture

This is a summary of my posts on software architecture and the key aspect of good architecture: Occamality, i.e., minimizing redundancy.

To start, it's worth pointing out that software people don't know what makes a piece of software "good."

https://www.blackliszt.com/2023/09/how-do-you-know-if-a-given-piece-of-software-is-good.html

Software people often have strong thoughts about software languages and architecture. However, it is extremely rare for those opinions to be grounded in or related to the goals of software architecture. What are the goals? Here’s my proposal.

https://www.blackliszt.com/2022/05/the-goals-of-software-architecture.html

Here are the key specific things you do to accomplish the goals.

https://www.blackliszt.com/2020/02/how-to-build-applications-that-can-be-changed-quickly.html

Here is a layman's, common-sense explanation of the same idea:

https://www.blackliszt.com/2022/10/how-to-improve-software-productivity-and-quality-common-sense-approach.html

Here is a specific explanation explaining why Occamal programs are better than non-Occamal ones and why you should care.

https://www.blackliszt.com/2023/09/why-should-you-build-occamal-programs.html

Occamality isn't another entrant in the myriad of design principles competing for attention in the chaotic world of software -- it's an overriding principle, one that stands above and ranks all the contenders.

https://www.blackliszt.com/2023/09/occamality-and-other-design-principles.html

Occamality isn't confined to writing software. It applies to all stages of the development lifecycle, from requirements through QA and support.

https://www.blackliszt.com/2023/09/occam-optimality-applies-to-all-stages-of-the-software-life-cycle.html

The value of reducing redundancy isn't confined to software; it's a general principle.

https://www.blackliszt.com/2020/05/lessons-for-better-software-from-washing-machine-design.html

Reducing redundancy is accomplished by taking a declarative approach to programming instead of a purely imperative one. There are many examples of this.

https://www.blackliszt.com/2021/07/software-programming-languages-the-declarative-core-of-functional-languages.html

Databases are an excellent, proven example of applying the principle of Occamality.

https://www.blackliszt.com/2023/09/occamality-in-databases.html

It's not just databases -- Occamality is a thread that weaves through much of the history of software.

https://www.blackliszt.com/2023/09/occamality-in-software-history.html

The optimal way to accomplish those goals includes recognizing that in addition to instructions and data, programs include varying amounts of metadata, which is the third dimension of software architecture.

https://www.blackliszt.com/2020/02/the-three-dimensions-of-software-architecture-goodness.html

For all dimensions, lack of redundancy is the main virtue. As a group, the more functionality is expressed in metadata and the less in code, the better.

A focus on metadata is similar to having a generic direction-generating program that refers to an easily-changed map.

https://www.blackliszt.com/2020/06/the-map-for-building-optimal-software.html

Here’s more theoretical depth on the role of metadata in a software system, with a comparison to theories of the solar system.

https://www.blackliszt.com/2022/10/how-to-improve-software-productivity-and-quality-code-and-metadata.html

Why put as much application knowledge into metadata as possible? It's the easiest thing to change, and above all, it's the best place to eliminate redundancy, which is the enemy of fast, error-free change.

https://www.blackliszt.com/2020/03/william-occam-inventor-method-for-building-optimal-software.html

Here is a more extensive explanation of the history and context of Occam's Razor and its relevance to software.

https://www.blackliszt.com/2023/09/occams-razor-the-key-to-optimal-software-development.html

How do you achieve this ideal architecture for a body of code? Not all at once! You avoid the usual nightmare of useless, ever-changing requirements and do something that makes a customer happier than they were. Then fix it. Here’s the process, to which I’ve given a fancy name.

https://www.blackliszt.com/2022/09/better-software-and-happier-customers-with-post-hoc-design.html

Here’s another statement of the basic idea:

https://www.blackliszt.com/2011/06/software-how-to-move-quickly-while-not-breaking-anything.html

https://www.blackliszt.com/2020/03/how-to-pay-down-technical-debt.html

Here is more detail and explanation of how to use increasing amounts of metadata to help build applications quickly, which of course should be a major goal of software architecture.

https://www.blackliszt.com/2022/05/how-to-improve-software-productivity-and-quality-schema-enhancements.html

One of the most basic aspects of software architecture is the data and where it is stored. The default choice for most architecture is to use a standard DBMS. Given the steady advance of Moore's Law, this is often no longer the best choice.

https://www.blackliszt.com/2010/09/databases-and-applications.html

Here are further explanations of Occamality.

https://www.blackliszt.com/2023/09/understanding-occam-optimality-practically.html

https://www.blackliszt.com/2023/09/case-study-replacing-metadata-with-fashionable-software.html

https://www.blackliszt.com/2023/09/achieving-occamality-through-definitions.html

https://www.blackliszt.com/2023/09/achieving-occamality-what-not-how.html

https://www.blackliszt.com/2023/09/achieving-occamality-through-definitions-case-study.html

https://www.blackliszt.com/2023/09/occamality-the-problem-with-layers-components-and-objects.html

 

Bad Software Architectures

Software is infected with architectural religions, none of them with a sound basis in logic or real-world experience. It’s not that you can’t build software that sort of eventually kinda works with them – but it’s like building a car with a steam engine.

Sadly, some programming languages and programming concepts encourage redundancy.

https://www.blackliszt.com/2014/03/how-to-evaluate-programming-languages.html

Starting a couple decades ago the idea of “distributed computing” as an architecture become the thing all the cool kids gravitated to.

https://www.blackliszt.com/2015/04/the-distributed-computing-zombie-bubble.html

A modern incarnation (with a new name and rhetoric of course) is micro-services, which is supposed to boost programmer productivity.

https://www.blackliszt.com/2021/03/how-micro-services-boost-programmer-productivity.html

Not only does micro-services boost programmer productivity, it supposedly is a “scalable” architecture – in sharp contrast to the evil “monolithic” architecture … a word which is usually pronounced with a sneer.

https://www.blackliszt.com/2020/06/why-is-a-monolithic-software-architecture-evil.html

The trouble is, microservices make about as much sense as blood-letting did in medicine. It's widely accepted as useful, but entirely without evidence.

https://www.blackliszt.com/2019/02/what-software-experts-think-about-blood-letting.html

Programmers seem to like to layer their software, often without thinking about it.

https://www.blackliszt.com/2012/06/layers-in-software-fuss-and-trouble-without-benefit.html

Similarly when they link together pieces, a key decision is whether the coupling is loose or tight.

https://www.blackliszt.com/2012/08/coupling-in-software-loose-or-tight.html

Components and layers have been promoted for a long time.

https://www.blackliszt.com/2021/03/micro-services-the-forgotten-history-of-failures.html

https://www.blackliszt.com/2021/09/software-components-and-layers-problems-with-data.html

https://www.blackliszt.com/2021/08/the-dangerous-drive-towards-the-goal-of-software-components.html

For the best results, it’s good to focus on the goals of software architecture described above, and assure that everything that you do contributes to those goals. Part of how you do this is to avoid the always-present temptation of following software fashions.

https://www.blackliszt.com/2023/07/summary-software-fashions.html

 

Summary: AI Machine Learning Big Data Math Optimization

This is a summary with links to my posts about Big Data and the fancy algorithms and math that are variously called Artificial Intelligence, Cognitive Computing, Machine Learning, Big Data Analytics, math optimization, etc. I have worked as an insider in these fields for over fifty years.

Huge amounts of time and money are spent on these things, but getting practical, real-world results is not so easy. You can make things worse.

https://www.blackliszt.com/2023/03/making-things-worse-with-ai.html

How do you make things better? It starts with data.

http://www.blackliszt.com/2018/03/getting-results-from-ml-and-ai-1.html

Everyone knows that data is important. There’s even a field called Big Data.

What people call Big Data often isn’t nearly as big as people think.

https://www.blackliszt.com/2013/02/big-data-some-little-observations.html

Do the arithmetic!

https://www.blackliszt.com/2012/04/im-tired-of-hearing-about-big-data.html

For all the talk about Big Data, it’s clear that making sure that the data is accurate and complete is too much to ask.

https://www.blackliszt.com/2019/03/nobody-cares-about-data.html

Among other things, the status hierarchy in Data Science makes it clear that anyone with ambition needs to get as far away from the actual data as possible.

https://www.blackliszt.com/2019/03/the-hierarchy-of-software-skills-data-science.html

Without data integrity, your analytics is screwed. It’s worse when the data is about your health. The integrity of data in healthcare electronic medical records is a major issue.

https://www.blackliszt.com/2016/06/healthcare-innovation-emrs-and-data-quality.html

Even simple insurance provider network data is too often wrong, putting yet more obstacles between patients and the services they need.

https://www.blackliszt.com/2017/01/my-cat-taught-me-about-the-state-of-healthcare-provider-data.html

Even if all the data were perfect, no one seems to ask why exactly is big data better?

https://www.blackliszt.com/2015/07/fatal-flaws-of-big-data.html

Is Big Data delivering results? Here’s what happened with Big data and Hadoop at Yahoo.

https://www.blackliszt.com/2019/01/using-advanced-software-techniques-in-business.html

With all the money going to Big Data, people are ignoring the massive benefits of leveraging little data.

https://www.blackliszt.com/2016/08/little-data-vs-big-data.html

The few people who study computer history have noticed that Big Data is remarkably similar to what used to be called EDW, Enterprise Data Warehouse.

https://www.blackliszt.com/2015/10/big-data-and-data-warehouses.html

Same thing with a new name. Hmm. Could it possibly be that Big Data is just a fashion trend?

https://www.blackliszt.com/2013/02/the-big-data-technology-fashion.html

https://www.blackliszt.com/2015/03/big-data-the-driving-force-in-computing.html

At least for insiders, there's lots of humor to be found in data. There's even a book of the fun highlights.

https://www.blackliszt.com/2022/01/data-humor-book-by-rupa-mahanti.html

After data, you move to the amazing variety of techniques available, each of which has specific areas of applicability.

http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-2.html

Even a narrow-sounding field like Machine Learning has incredible variety and lots of things you need to do until the algorithm can be effective.

https://www.blackliszt.com/2017/02/learning-machine-and-human.html

The simple concept of closed loop is essential to getting and keeping good results, but isn’t applied as often as it should be.

http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-3-closed-loop.html

The application of the principles to healthcare has proven to be challenging, with high-profile failures. But there are successes.

https://www.blackliszt.com/2018/08/getting-results-from-ml-and-ai-4-healthcare-examples.html

I've discussed the application of the principles to fintech, with a focus on anti-fraud. The current leader in the field displaced the incumbent who was using neural network technology, showing that the algorithm chosen and how it’s applied makes all the difference.

https://www.blackliszt.com/2019/2/getting-results-from-ml-and-ai-5-fintech-fraud.html

Natural language AI has proved notoriously hard to make practical. Here’s an example of how it’s been done in fintech customer support.

https://www.blackliszt.com/2019/12/getting-results-from-ml-and-ai-6-fintech-chatbot.html

All too often, AI research is conducted in isolation. There is a great deal to do to assure that any results that are achieved can be integrated into production systems.

https://www.blackliszt.com/2022/06/how-to-integrate-ai-and-ml-with-production-software.html

As is true in most fields, in healthcare the money and attention tends to go to expensive, fancy methods requiring PhD’s instead of simple things that actually work.

https://www.blackliszt.com/2015/07/cognitive-computing-and-healthcare.html

https://www.blackliszt.com/2016/09/healthcare-innovation-from-washing-hands-to-ai.html

https://www.blackliszt.com/2016/05/healthcare-innovation-can-big-data-and-cognitive-computing-deliver-it.html

You don’t need AI or cognitive computing to discover or promulgate the new discoveries that humans make.

https://www.blackliszt.com/2015/08/human-implemented-cognitive-computing-healthcare.html

There are many ways to spend lots of time and not get practical results.

https://www.blackliszt.com/2019/03/the-hierarchy-of-software-skills-data-science.html

Impressive analytics performed by recognized experts are subject to getting the results the Expert wants.

https://www.blackliszt.com/2017/04/big-datas-big-face-plant.html

Corruption of data or process can lead to bad results anywhere.

https://www.blackliszt.com/2019/10/surprising-bugs-at-amazon-shows-how-ai-can-lead-to-disaster.html

For cynical definitions in the tradition of Ambrose Bierce of Big Data, Machine Learning, Deep Learning, Cognitive Computing and AI, see these.

https://www.blackliszt.com/2017/01/devils-dictionary-for-21st-century-computing.html

https://www.blackliszt.com/2017/01/devils-dictionary-for-21st-century-computing-2.html

https://www.blackliszt.com/2017/03/devils-dictionary-for-21st-century-computing-3.html

 

Does Software Evolve Rapidly?

Some software is changed over time. But there is a great deal of software that is simply part of the landscape or part of the underpinning of the software we see. This largely ignored category of software is important  to understand.

Keeping up with Software

I had a nice conversation with the non-technical CEO of one of our software-based companies. He expressed the widely-held view that the process of developing software is undergoing change at a dizzying pace. Just in the last couple of years, new languages and tools have emerged and are being used and talked about. How can anyone possibly keep up, he wondered?

That certainly is how things seem to most people, both inside and outside of the computer industry. We had desktop machines, personal computers, 25 years ago, and we have them today; of course, they’re less expensive and have way more power and capacity than they did back then, but they’re still personal computers, we put them on or beside our desks, and use them. Software, on the other hand, undergoes constant change. For programmers, we’ve gone through an incredible changing landscape of programming languages and tools during that time. I’ll just give a short list: assembler, BASIC, turbo Pascal, Smalltalk, C, Objective-C, 4-GL’s too numerous to list, C++, Java, C#, PERL, javascript, php, python, ruby and on and on. It’s no wonder that it seems like it’s hard to keep up! And of course, in a way it is – it does take work. But most of that language change is little but noise. The claims of advances don't hold up when examined.

That’s how it seems. Hardware is hardware. Yes, it gets better and cheaper, but software, wow – there’s always something new!

The reality is different. True software change, whether in methods or bodies of code, is slower than glacial.

The Non-Evolution of Software Categories

It’s hardware that undergoes dramatic change, not software. Think Moore’s Law. Think about mobile phones just ten years ago. Here are details. By contrast, change in software is actually glacial. Yes, the surface weather of software changes rapidly and unpredictably; but the underlying principles and structure of software don’t even move as fast as glaciers – it’s more like tectonic plates.

Things are changing all the time.  Software is certainly “soft” compared to hardware. But programmers spend most of their time in their minds, not wandering around touching up the hardware. What do they spend time thinking about? It’s not the hardware – it’s the software environment in which they work, and in which the software they create will live.

Think about kids in a sand box. The sand is something they can push around, pile up, makes holes in, and generally have fun with. How about the box the sand is in? It’s hard. It keeps the sand in one area. No kid thinks about it much – it’s just something that’s there when you’re playing.

It works the same in software. The “part” of the software you are playing with (a.k.a. “programming”) seems “soft” and easily changeable, like sand. The “part” of the software you’re not playing with seems “hard” to you. It’s just part of the landscape. This effect is strongest when the programmer has no access to the source code of the landscape software. An example is a DBMS or an operating system. These things feel like immutable reality. They are what they are, and you have to deal with them. Hardware speed and storage capacity has grown so much, for example, that DBMS's are a data storage solution to a problem that no longer exists. But most software design continues to be built on them.

Just as other pieces of software feel like given realities if you’re a programmer, so does the programming language in which you’re working. The verbs and the syntax are as real as the grains of sand. Just as when you play with sand, you have no sense that each grain is actually a very large aggregation of molecules, so when you use a verb in the programming language, you have no sense that a compiler or interpreter will actually use many machine language instructions to execute each verb.

The “hardness” of software to which you have no source code access (or otherwise no ability to change) is a strong psychological effect.

The Glacial Evolution of Software Applications

Programmers have a special view of software. Regular people also use software personally and/or professionally. While society gives us the impression that the software people use changes like crazy, the reality is that people learn new things slowly, and successful software evolves very slowly. The changes we see are largely incidental things that are enabled by the astounding advance of hardware.

The Bedrock of Production Systems

Large bodies of working, production code are the bedrock on which fancy new layers may be built, but change very slowly. Here's an example of the impact of that hard-to-change code.

It isn’t as though people don’t want to change these systems. Often they are desperate to change them! Sometimes managers will put huge investments into changing or replacing them, only to quietly admit failure and stop trying. Until the next bunch of ignorant-of-history managers come along.

These bedrock systems aren't just systems software like DBMS's. They are also plain old applications. For example, the software that handles the largest number of credit card accounts in the world is written in the prehistoric language COBOL. Several attempts were made to re-write card processing software using fancy new languages. Failed. See also this and this.

Conclusion

Hardware changes with amazing speed, giving no signs of slowing down. While there's always something new to talk about in software, it changes much more slowly than hardware, and is remarkably hard to change.

 

Summary: Wartime Software to Win the War

The vast majority of software development is organized to deliver software that meets a set of requirements on time and on budget. In other words, it’s organized to meet expectations.

What if you don’t have the time or money to do this? What if you’re focused on a need unmet by existing software and you’re driven to create software to meet that need? This is wartime software – get it done or lose the war. It’s optimized for speed, with customer satisfaction and unfailing quality as a given.

In this summary, we’ll cover relevant blog posts and books.

No matter how you do it, there are things about the software development process that are common.

https://www.blackliszt.com/2013/03/software-development-process-simple-terms.html

How do you measure success in software development? Agreeing on the measure of success is crucial to achieving it.

https://www.blackliszt.com/2013/03/software-postulate-the-measure-of-success.html

Why is so much software so bad? there are multiple reasons.

https://www.blackliszt.com/2011/06/why-computer-software-is-so-bad.html

https://www.blackliszt.com/2022/04/how-to-fix-software-development-and-security-a-brief-history.html

Very high on the list of causes of bad software is what is optimized for in the development process. Peacetime software optimizes for predictability, i.e., delivering the requirements on time and budget. Wartime software optimizes for speed.

https://www.blackliszt.com/2010/10/software-project-management-dates-are-evil.html

The best way to accomplish this kind of development goes against the way most software engineers are trained. Here is the key difference.

https://www.blackliszt.com/2022/09/better-software-and-happier-customers-with-post-hoc-design.html

Peacetime Software Development

Peacetime software optimizes for predictability, i.e., delivering the requirements on time and budget. Wartime software optimizes for speed.

https://www.blackliszt.com/2010/10/software-project-management-dates-are-evil.html

Being on time and budget means making an estimate, which experienced people know is nearly impossible.

https://www.blackliszt.com/2012/06/the-nightmare-of-software-estimation.html

Estimates are also hard because software productivity is hard to measure.

https://www.blackliszt.com/2012/09/what-is-software-productivity.html

Whatever the productivity is, doubling the amount of work to be done using popular peace-time development techniques is sure to make it worse.

https://www.blackliszt.com/2012/09/software-productivity-divisors.html

Peacetime development involves loads of activities that aren’t programming.

https://www.blackliszt.com/2012/09/software-productivity-the-abp-factor.html

Project Management is supposed to assure that software development is “under control.” It’s actually a method to assure that programmers spend as little time as possible programming.

https://www.blackliszt.com/2012/10/software-project-management-book.html

https://www.blackliszt.com/2012/10/the-disease-of-software-project-management.html

The vast majority of software development bears a striking resemblance to the meticulous planning and building of Frankenstein’s monster.

https://www.blackliszt.com/2012/04/the-dirty-secret-of-peace-time-software-development.html

You get to near the end of a software project that has been “going well” and you suddenly discover the disaster.

https://www.blackliszt.com/2014/06/building-software-the-bad-old-way-and-the-good-new-way.html

With the usual measures, standard methods deliver terrible results.

https://www.blackliszt.com/2011/07/software-quality-theory-and-reality.html

The normal methods make common sense, like planning and making a Thanksgiving meal – and usually yield the classic mess you have after the meal.

https://www.blackliszt.com/2011/11/thanksgiving-meals-and-software-turkeys.html

Most peacetime software managers know they should act like they’re modern, criticizing old-fashioned waterfall and embracing Agile. Little change in results.

https://www.blackliszt.com/2013/03/software-comparing-waterfall-and-agile.html

Classic software development efforts focus on process instead of the substance of the software itself.

https://www.blackliszt.com/2013/03/process-and-substance-in-software-development.html

Most important decisions about software aren’t made by programmers, which is a big factor in things going wrong.

https://www.blackliszt.com/2014/01/who-makes-the-software-decisions.html

https://www.blackliszt.com/2014/10/joe-torre-and-software-development.html

Standard development embraces the metaphor of “building” software like a house, imagining that there’s a time when it’s done.

https://www.blackliszt.com/2013/10/when-is-software-development-done.html

Another factor in building software is “just” having competent people who do their jobs.

https://www.blackliszt.com/2014/04/delivering-software-is-a-nightmare.html

Why do all the experts cling to the expectation-based approach to software? The history of scurvy is a close parallel to software development diseases.

https://www.blackliszt.com/2014/02/lessons-for-software-from-the-history-of-scurvy.html

A major issue with expectation-based software is incentives. Among other things, when you’re judged on expectations, you’re incented to make huge estimates of the amount of work.

https://www.blackliszt.com/2015/04/software-problems-the-role-of-incentives.html

You might think that because software is related to math that it’s precise. There are huge factors like the exponential growth of computer power that are ignored in Computer Science and peacetime development.

https://www.blackliszt.com/2015/05/math-and-computer-science-vs-software-development.html

There are many paths people take to feel better about software development; they may sell well at the beginning but make things worse.

https://www.blackliszt.com/2015/06/software-problems.html

Wartime Software Development

The essence of wartime software development is that everything is optimized for speed of development.

https://www.blackliszt.com/2013/05/wartime-software-optimizing-for-speed.html

Great software development resembles growing a baby.

https://www.blackliszt.com/2011/06/software-how-to-move-quickly-while-not-breaking-anything.html

Just like with growing a baby, wartime software can’t be allowed to die.

https://www.blackliszt.com/2012/04/field-tested-software.html

A good way to see the contrast between the mainstream way of software development and the method used by software ninjas is to compare building bridges in peacetime and war time.

https://www.blackliszt.com/2012/03/bridges-and-software-in-peace-and-war.html

When you focus on speed, there is a simple concept that explains how it’s better.

https://www.blackliszt.com/2015/12/the-fundamentals-of-speed-optimized-software-development.html

https://www.blackliszt.com/2015/10/building-better-software-more-quickly.html

Is wartime software development like regular development only faster with fewer distractions? No. There are also essential differences in method and code standards.

https://www.blackliszt.com/2020/02/how-to-build-applications-that-can-be-changed-quickly.html

If software is always being changed, you want to make change easy. It’s what the average lay person wants from software.

https://www.blackliszt.com/2022/10/how-to-improve-software-productivity-and-quality-common-sense-approach.html

Here’s the idea of code without redundancies.

https://www.blackliszt.com/2020/06/the-map-for-building-optimal-software.html

While the software world may not have caught on to the value of no redundancy, the ordinary world of machine design is all over it.

https://www.blackliszt.com/2020/05/lessons-for-better-software-from-washing-machine-design.html

Here’s a broader exploration of focusing on redundancy.

https://www.blackliszt.com/2014/03/how-to-evaluate-programming-languages.html

Technical debt should mostly mean too much redundancy.

https://www.blackliszt.com/2020/03/how-to-pay-down-technical-debt.html

Technically, the way to eliminate redundancy is to move concepts from code to metadata.

https://www.blackliszt.com/2022/10/how-to-improve-software-productivity-and-quality-code-and-metadata.html

More about how to make quickly-changeable code.

https://www.blackliszt.com/2022/05/how-to-improve-software-productivity-and-quality-schema-enhancements.html

Code without redundancy is simple. It’s an ancient idea.

https://www.blackliszt.com/2020/03/william-occam-inventor-method-for-building-optimal-software.html

Here’s a summary of the method and techniques.

https://www.blackliszt.com/2022/09/better-software-and-happier-customers-with-post-hoc-design.html

Of course, following good business and product strategies is an essential aspect of wartime software -- among other things, carefully selecting customer focus is key, Like conquering Europe in World War 2 by landing on a small number of beaches in Normandy instead of invading the whole coastline.

https://www.blackliszt.com/2023/07/summary-software-innovation.html

Wartime Software Book

Here is a brief introduction with links to my book on Wartime Software.

https://www.blackliszt.com/2013/05/wartime-software-book-available.html

 

Summary: Software Innovation

This is a summary with links to my posts on software innovation. It includes posts on barriers to innovation and how to grow a winner.

Software is taking over the world. The pace and scope of the transformation of human activity has no precedent. People often assume that this is the result of fierce innovation in software. While brand-new software is built every day, the actual innovation is the result of computer innovation – which does indeed proceed at an unprecedented pace.

https://www.blackliszt.com/2013/12/fundamental-concepts-of-computing-speed-of-evolution.html

https://www.blackliszt.com/2014/03/innovation-with-computers-and-slow-things-1.html

The fact is that innovation in software is incredibly simple. It rarely involves fancy stuff like AI, but mostly figuring out the best way to accomplish things and getting the computer to do as much of the work as possible.

https://www.blackliszt.com/2014/07/innovation-made-simple.html

Here are some of the most important patterns of the small groups that innovate their way to success in software.

https://www.blackliszt.com/2017/01/the-science-of-innovation-success.html

The general impression is that software is all about innovation and rapid evolution. Programmers are on the front lines, constantly making new things. Sadly, this general impression, which is shared by most programmers, doesn't hold up under examination.

https://www.blackliszt.com/2023/07/does-software-evolve-rapidly.html

People love to brag about the software innovations they’ve invented. The fact is, most of the fundamental innovations in software are proven and in place; they’re ignored by practically everyone, including the experts.

https://www.blackliszt.com/2015/06/fundamental-innovations-in-software.html

This holds true even when you look at a narrow field of application such as financial technology.

https://www.blackliszt.com/2016/03/fintech-innovation-the-drivers.html

An amazing fraction of what appears to be innovations are little but taking advances that are proven in a narrow domain and applying them to a new one. Here is the story of an algorithm that was standard practice in oil refinery operation over 50 years ago that, decade by decade, is still crawling its way into new domains.

https://www.blackliszt.com/2019/08/the-slow-spread-of-linear-programming-illustrates-how-in-old-vation-in-software-evolution-works.html

Here is an example of a truly beneficial innovation proven over 50 years ago that is still not used in medical imaging.

https://www.blackliszt.com/2019/07/barriers-to-software-innovation-radiology-1.html

https://www.blackliszt.com/2019/08/barriers-to-software-innovation-radiology-2.html

It’s not just fancy algorithms that are proven and waiting for application in new domains. It’s simple things like production human data entry, where widely proven “heads down” methods are at least 5 times more efficient than what is normally done.

https://www.blackliszt.com/2019/09/simple-data-entry-technology-illustrates-how-in-old-vation-in-software-evolution-works.html

Here are details of the advantages of “heads down” data entry and how it was ignored at a huge project at Sallie Mae.

https://www.blackliszt.com/2019/10/software-professionals-would-rather-be-fashionable-than-achieve-10x-productivity-gains.html

One of the common ways to ignore a major innovation such as “heads down” data entry is to concentrate on a method that is highly fashionable, even though it doesn’t do much good.

https://www.blackliszt.com/2019/09/laser-disks-and-workflow-illustrate-the-insane-fashion-driven-nature-of-software-evolution.html

Given this, it’s all the more amazing that companies have Chief Innovation Officers whose job is usually to “foster innovation.” Heh.

https://www.blackliszt.com/2016/04/the-innovation-bubble.html

Barriers and Resistance to Innovation

Software innovation faces huge barriers from people and established practice, like innovation in medicine, where a cure for scurvy was proven in practice for decades before the authorities grudgingly accepted it.

https://www.blackliszt.com/2014/02/lessons-for-software-from-the-history-of-scurvy.html

Innovation has been strongly resisted for a long time.

https://www.blackliszt.com/2016/05/innovation-some-history.html

Have you heard the story of Samuel Pierpont Langley, the renowned expert on manned flight? He’s a case study in how experts prevent innovation.

https://www.blackliszt.com/2016/07/innovation-and-experts.html

Major advisory institutions reflect common thinking and prevent their customers from "making mistakes" with innovative companies.

https://www.blackliszt.com/2017/05/the-value-of-computer-industry-advisory-groups.html

Here is the story of how the British military resisted a huge innovation for combating submarines in World War 2.

https://www.blackliszt.com/2021/10/deep-seated-resistance-to-software-innovation.html

Even the experts in entrepreneurship resist innovation. Amazing. Experts!

https://www.blackliszt.com/2020/05/experts-vs-innovation-new-book.html

Perhaps you think that the big tech companies are great at innovation? When you look at how they actually innovate, they don’t look so great.

https://www.blackliszt.com/2016/05/organizing-for-successful-innovation-recent-history.html

Often important innovations don’t require software at all, which doesn’t seem to stop people spending loads on money on exotic software.

https://www.blackliszt.com/2016/09/healthcare-innovation-from-washing-hands-to-ai.html

https://www.blackliszt.com/2016/05/healthcare-innovation-can-big-data-and-cognitive-computing-deliver-it.html

Healthcare is a rich source of examples of how to screw up and fail to take advantage of obvious, long-overdue “innovations” like electronic medical records.

https://www.blackliszt.com/2016/06/healthcare-innovation-emrs-and-paper.html

https://www.blackliszt.com/2016/05/healthcare-innovation-emr-procurement-is-broken.html

https://www.blackliszt.com/2016/06/healthcare-innovation-emrs-and-data-quality.html

While experts and big companies put up active resistance to innovation, regulations are an important source of passive resistance.

https://www.blackliszt.com/2016/09/innovation-the-barriers.html

The huge cost of medical imaging systems is a clear example of how regulations prevent innovation and keep costs high.

https://www.blackliszt.com/2023/01/how-to-reduce-the-cost-of-medical-imaging-and-pacs.html

When you look into sample sets of regulations, you see how onerous they are and how they hamstring innovation.

https://www.blackliszt.com/2016/12/regulations-that-enable-innovation.html

There’s a clear path to eliminating regulatory resistance while making the enforcement power of the regulations even stronger, by shifting from massive how-type regulations to simple but effective what-type ones.

https://www.blackliszt.com/2011/12/regulations-goals-or-directions.html

https://www.blackliszt.com/2012/03/lets-criminalize-our-regulations.html

Growing the Innovation to Success

Once you’ve got your software company up and running, there are strategic moves that will keep you on the track to success.

https://www.blackliszt.com/2010/05/from-start-up-to-real-success.html

https://www.blackliszt.com/2010/12/from-startup-to-success-costs.html

Would you like to follow Facebook’s growth path? Their success wasn’t about great software development. It was due to a classic product/company growth strategy.

https://www.blackliszt.com/2014/12/fb.html

Your strategy may be good, but unless you build applications that can be changed quickly, you’ll lose the race.

https://www.blackliszt.com/2020/02/how-to-build-applications-that-can-be-changed-quickly.html

When you’re moving quickly, classically-trained programmers may start whining about the growing amount of “technical debt” and how important it is to pay it down. Here’s how to do it.

https://www.blackliszt.com/2020/03/how-to-pay-down-technical-debt.html

If you follow classic software development methods instead of fast ones, the chances are you could be hit with a big surprise at the end of the project.

https://www.blackliszt.com/2014/06/building-software-the-bad-old-way-and-the-good-new-way.html

There is a way to avoid disasters of this kind. It's practicing wartime software development, optimizing your methods for speed instead of meeting expectations.

https://www.blackliszt.com/2023/07/summary-wartime-software-to-win-the-war.html

When you’ve built a successful narrow-focus software company, how can you grow it further?

https://www.blackliszt.com/2022/07/the-dimension-of-software-automation-breadth-examples.html

The Book

Here are highlights of winning growth strategies and the book with more stories and details.

https://www.blackliszt.com/2016/04/software-business-and-product-strategy-book.html

https://www.blackliszt.com/2016/05/innovation-from-startup-to-success.html

Here are highlights of the companies used as examples in the book.

https://www.blackliszt.com/2016/07/innovation-stories.html

 

Summary: Software Fashions

Software is a precise discipline. A single bit out of place can crash a program. It’s like math. Fashion is the exact opposite – skirts are short or long, you wear a tie or you don’t, whatever. It’s all opinions and taste, nothing objective about it. How could there be fashions in bit-driven, nerdy software? This is a summary of my writing on software fashions.

https://www.blackliszt.com/2018/11/what-are-software-fashions.html

Here’s a survey of some current software fashions and a couple that have become standard practice.

https://www.blackliszt.com/2019/05/recurring-software-fashion-nightmares.html

Software people are, after all, people. People want to have status. One of the main ways to get status is to get identified with a fast-rising fashion trend and ride it.

https://www.blackliszt.com/2018/10/the-hierarchy-of-software-status.html

The status-driven attraction of fashion is so powerful that software people will often go with the fashionable approach and ignore software methods that would achieve dramatically better results.

https://www.blackliszt.com/2019/10/software-professionals-would-rather-be-fashionable-than-achieve-10x-productivity-gains.html

https://www.blackliszt.com/2019/09/laser-disks-and-workflow-illustrate-the-insane-fashion-driven-nature-of-software-evolution.html

Other things that become fashions are OK ideas taken way beyond where they make sense. These fashions often appear repeatedly, usually with new names.

https://www.blackliszt.com/2021/03/micro-services-the-forgotten-history-of-failures.html

Another pattern is wild swings from one end of a spectrum to the other -- and back again. You'd better be on the right end of the spectrum or risk being "out of it!" In computer programs whether data is defined inside the program logic or outside of it is a decades-long example.

https://www.blackliszt.com/2015/06/innovations-that-arent-data-definitions-inside-or-outside-the-program.html

Computers and software are all about innovation -- so how could innovation itself be a fad? Well, for a few years, innovation was a fashion-forward as you could get. Until it faded back to its usual place.

https://www.blackliszt.com/2016/04/the-innovation-bubble.html

Sometimes technical fashions are picked up by nontechnical people, who wildly promote the fashion without having clue about what the underlying technology is about, and therefore get it wrong. DLT (blockchain, the technology underlying cryptocurrencies) is an example of something that is promoted as a solution for problems for which it's not relevant.

https://www.blackliszt.com/2015/11/the-magic-of-block-chain.html

And for problems which have existing solutions that are often thousands of times better.

https://www.blackliszt.com/2017/11/blockchain-a-sailboat-without-sails.html

It wasn’t so long ago that everyone was talking about Big Data:

https://www.blackliszt.com/2015/03/big-data-the-driving-force-in-computing.html

https://www.blackliszt.com/2013/02/the-big-data-technology-fashion.html

https://www.blackliszt.com/2013/02/big-data-some-little-observations.html

https://www.blackliszt.com/2012/04/im-tired-of-hearing-about-big-data.html

Here’s an example of how the focus on fashion with Hadoop (a fashion-driven approach to Big Data) led to bad results.

https://www.blackliszt.com/2019/01/using-advanced-software-techniques-in-business.html

Large bunches of data, shared computers (the cloud) and virtualization are things that have been around since the early days of computing, are still around and always will be. But for a period of time they were what fashion-forward people talked about.

https://www.blackliszt.com/2012/05/the-cloud-and-virtualization.html

https://www.blackliszt.com/2011/12/the-name-game-of-moving-to-the-cloud.html

How is it possible that fashion can play such a large role in software decision-making? It’s because evidence is rarely part of software decision-making.

https://www.blackliszt.com/2017/02/evidence-based-software.html

https://www.blackliszt.com/2014/02/lessons-for-software-from-the-history-of-scurvy.html

In a broader sense, fashion rules in software because, in spite of the name, Computer Science is not a science.

https://www.blackliszt.com/2023/04/summary-computer-science.html

 

Summary: Software Fundamentals

You might think that computer fundamentals would be the first things taught and often returned to in Computer Science. To the contrary, some of the most important things about software may be known to programmers if you ask them, but understood and practiced? Not so much.

https://www.blackliszt.com/2012/02/fundamental-concepts-of-computing.html

https://www.blackliszt.com/2012/04/where-does-software-come-from-what-is-software-.html

What do you do with software -- aside from wasting huge amount of time? You automate!

https://www.blackliszt.com/2020/01/the-fundamentals-of-computer-automation.html

In any field you can think of, there are clear, agreed-upon measures of success. In software? Not so much.

https://www.blackliszt.com/2013/03/software-postulate-the-measure-of-success.html

If you have trouble measuring success, how can you measure productivity?

https://www.blackliszt.com/2012/09/what-is-software-productivity.html

If you have trouble measuring productivity, practices can become widely used that, if you step back from them, you can see obviously reduce productivity.

https://www.blackliszt.com/2012/09/software-productivity-divisors.html

Programming is taught as though all programming is similar. Nothing could be further from the truth. Is arguing a criminal case in a courtroom the same as acting in a Shakespeare play? They both use English, right? Programming has distinctions that are just as sharp.

https://www.blackliszt.com/2023/09/classifications-of-software-programs.html

One of the most obvious but least examined fundamental is that software is data. The fact that there is a memory that contains a mix of instructions and data … and that the instructions are data … is fundamental, with many implications.

https://www.blackliszt.com/2014/04/fundamental-concepts-of-computing-software-is-data.html

Diving into and understanding the possible relationships between data and instructions is bedrock core to being a real programmer.

https://www.blackliszt.com/2021/08/the-relationship-between-data-and-instructions-in-software.html

Everyone knows about numbers and counting. But how many times do people stand back from their work and apply some appropriate numbers to them? Not often enough.

https://www.blackliszt.com/2012/10/fundamental-concepts-computing-counting.html

Closely related to counting is speed. The speed of computing has increased unlike anything else in our existence. If cars got 10X faster, we would change lots of things, including whether to drive or fly. But as computing has gotten 100X or more faster, we too often fail to adapt the way we program.

https://www.blackliszt.com/2013/12/fundamental-concepts-of-computing-speed-of-evolution.html0

When we walk or drive, we always look ahead and change details of our walking or driving to adapt to ever-changing conditions. Adapting what you do to what you see and learn is fundamental common sense. All too often, it is not part of programming.

https://www.blackliszt.com/2013/02/fundamental-concepts-of-computing-closed-loop.html

Some fundamental concepts of software aren’t taught and most people, including programmers, have the wrong ideas in their head about them. One of those concepts is that a program is a passive thing that needs to be built, instead of a set of actions that needs to be choreographed.

https://www.blackliszt.com/2021/08/fundamental-concept-software-is-not-a-passive-thing-it-is-actions.html

Flowcharts and workflow express the actions of a program. They have been around since the start of programming. They're still around, but largely ignored as a fundamental.

https://www.blackliszt.com/2022/06/flowcharts-and-workflow-in-software.html

You think of building a house, after which it’s mostly unchanged. Software is different. Being built is like being born – software spends most of its “life” being changed.

https://www.blackliszt.com/2013/10/when-is-software-development-done.html

https://www.blackliszt.com/2021/08/fundamental-concept-software-is-rarely-built-mostly-it-is-changed.html

What we call “building” software works much better if we treat as “growing a baby.”

https://www.blackliszt.com/2011/06/software-how-to-move-quickly-while-not-breaking-anything.html

It’s easy to think that most of software has been figured out, and that innovation is possible only at the nether reaches of the field. Kind of like how innovation in math isn’t needed or performed from arithmetic through algebra and calculus and way beyond. Not true in software! Some of the most fundamental things, like quality, are ripe for fundamental innovation.

https://www.blackliszt.com/2015/06/fundamental-innovations-in-software.html

Speaking of math, computing started as a way to do math and grew as part of math departments. The misconception that math and Computer Science are closely related is widespread. This is deeply harmful.

https://www.blackliszt.com/2015/05/math-and-computer-science-vs-software-development.html

https://www.blackliszt.com/2015/04/math-and-computer-science-in-academia.html

The usual practice in computing is to dive right into the details. Perspective and putting it in a broader context is tremendously helpful if you aspire to be truly excellent in computer software. Here are some sample books that provide that context.

https://www.blackliszt.com/2023/05/inspiring-books-for-writing-software.html

Computing and software would greatly benefit by establishing the fundamental concepts of the field. Individual programmers can become vastly more productive by understanding the fundamentals and practicing them.

Summary: Software Programming Languages

This is a summary with links to all my posts on software programming languages.

Computers need to be “told” what to do. Each computer has a native language it understands, a “machine language.” Each machine language has a human-readable version called “assembler language.” Starting in the 1950’s “high level” languages for telling the computer what to do (programming it) have been developed.

Here are the basics of that amazing progression.

https://www.blackliszt.com/2020/09/the-giant-advances-in-software-programming-languages.html

Software programming language core concepts

One of the core features of all programming languages is the subroutine. Everyone who programs uses subroutines, but their power can be underappreciated.

https://www.blackliszt.com/2021/03/fundamental-concepts-of-computing-subroutine-calls.html

Among the earliest features added to languages were structures, blocks and macros. Each of these contributed to more concise and understandable programs.

https://www.blackliszt.com/2021/09/software-programming-language-evolution-structures-blocks-and-macros.html

Once you get past the basics of language, you find that widely used languages have rich collections of subroutine libraries and sometimes what are called frameworks, which contribute a large fraction of any program’s solution.

https://www.blackliszt.com/2021/10/software-programming-language-evolution-libraries-and-frameworks.html

The way programs flow is a key aspect.

https://www.blackliszt.com/2022/06/flowcharts-and-workflow-in-software.html

Programs consist of a combination of instructions and definitions of the data on which the instructions operate.  Exactly how those relate to each other has been the subject of major disputes – which continue to this day.

https://www.blackliszt.com/2015/06/innovations-that-arent-data-definitions-inside-or-outside-the-program.html

Some programming "advances" basically try to restrict the possible relationships betwen code and data. Bad idea.

https://www.blackliszt.com/2021/08/the-relationship-between-data-and-instructions-in-software.html

Software programming language evolution

When talking about software languages, the term high level language was introduced early. Here’s what it means.

https://www.blackliszt.com/2020/10/what-is-high-about-a-high-level-language-in-software.html

Decades ago a couple of languages were invented and enjoyed an explosion of use that were major steps ahead of the standard languages of the time in terms of productivity.

https://www.blackliszt.com/2020/11/software-programming-language-evolution-beyond-3gls.html

Not long after these amazing, productivity-enhancing languages had their rapid growth, the modern DBMS came onto the scene, with its own language for data definition, query and update. The impact of the DBMS revolution was huge.

https://www.blackliszt.com/2021/01/software-programming-language-evolution-impact-of-the-dbms-revolution.html

One of the major reactions to the DBMS revolution was the introduction and rise to prominence of what were called 4-GL’s, all which of incorporated the DBMS as an integral part.

https://www.blackliszt.com/2021/02/software-programming-language-evolution-4gls-and-more.html

There has been an explosion of programming language invention.

https://www.blackliszt.com/2014/04/continents-and-islands-in-the-world-of-computers.html

It just goes on and on and on and on, with no end in sight.

https://www.blackliszt.com/2022/04/software-programming-language-cancer-must-be-stopped.html

Software programming language innovations

Experts tout the major advances in programming languages made since the early days, but they don’t amount to much.

https://www.blackliszt.com/2020/09/software-programming-languages-50-years-of-progress.html

A comparison to the virtues of various human languages makes this clear.

https://www.blackliszt.com/2020/12/the-bronte-sisters-and-software.html

Attempts have been made by experts to fix problems that have been widely noted for many decades.

https://www.blackliszt.com/2021/09/software-programming-language-evolution-the-structured-programming-goto-witch-hunt.html

One of the serious detours into unproductive swamps taken by programming languages has been what’s called object-orientation. It had a promising start and would have been fine had it stopped there.

https://www.blackliszt.com/2021/11/the-promising-origins-of-object-oriented-programming.html

Lots of people have things to say about O-O languages. There are the academics and other authorities and then there are the people with real experience who aren’t groupies.

https://www.blackliszt.com/2022/01/object-oriented-software-languages-the-experts-speak.html

Here are some facts and logic to explain why O-O should be referred to exclusively in the past tense.

https://www.blackliszt.com/2022/03/why-object-orientation-in-software-is-bad.html

Another category of hyped but harmful programming languages have been “functional” ones. These arose early in programming evolution and keep popping up.

https://www.blackliszt.com/2021/05/software-programming-language-evolution-functional-languages.html

The attraction of functional language has a strong basis. It’s true that increasing the declarative aspect of a total solution brings strong benefits. But embedding it in the language neuters it.

https://www.blackliszt.com/2021/07/software-programming-languages-the-declarative-core-of-functional-languages.html

Here are some detailed examples of how supposed advances in programming languages have worked out in practice, including failures of Java and 4-GL’s and a huge success with a brand-new system written in that ancient bit of history called COBOL.

https://www.blackliszt.com/2021/01/software-programming-language-evolution-credit-card-software-examples-1.html

https://www.blackliszt.com/2021/02/software-programming-language-evolution-credit-card-software-examples-2.html

https://www.blackliszt.com/2022/08/software-programming-language-evolution-credit-card-software-examples-3.html

After all those failed advances, there's the amazing case of HTML, a wildly successful language without all the bells and whistles that takes the winning principle of "just get the job done."

https://www.blackliszt.com/2010/05/html-assembler-language-easy-to-use-tools.html

Software programming language evaluation

Promoters of new languages tout how they prevent programmers from making errors (ha!) and result in cleaner programs that somehow promote productivity – which is never measured or tested.

The fact is, there is a standard by which programming languages should be judged – along with the support systems and metadata that are an integral part of the solution. It’s simple: how many places do you have to go to make a change? The right answer is one. Here’s the idea.

https://www.blackliszt.com/2014/03/how-to-evaluate-programming-languages.html

 

Summary: Bad Software Metaphors lead to bad results

Most of the people who talk about software can’t create it. They can’t even read it with understanding. So how is anyone who not only can’t write software but can’t even see it supposed to understand it, much less manage it?

https://www.blackliszt.com/2016/12/managing-what-you-cant-see.html

https://www.blackliszt.com/2017/04/managing-software-thats-invisible-to-you.html

A standard way about talking about software has emerged among the people to whom it’s invisible: physical-world metaphors. The metaphors are so ubiquitous that they are also used by programmers.

Because the metaphors are nearly always seriously inappropriate and people act as though they are great, horrible things result.

https://www.blackliszt.com/2016/11/software-myths-and-myth-conceptions.html

https://www.blackliszt.com/2021/09/the-catastrophic-effect-of-bad-metaphors-in-software.html

Wouldn’t it be nice if we could churn out software as reliably and predictably as a factory?

https://www.blackliszt.com/2012/04/how-effective-are-software-factories.html

The building metaphor is widely used. Buildings have architects, and so does software.

https://www.blackliszt.com/2018/12/software-planning-is-impossible.html

https://www.blackliszt.com/2017/06/software-myth-conceptions-building-materials.html

You architect and build and house. It’s a passive physical object. Software is NOT a physical object; it’s mostly is a set of actions to be taken by a computer.

https://www.blackliszt.com/2021/07/fundamental-concept-software-is-not-a-passive-thing-it-is-actions.html

The notion that the core organizing principle of software is that it's a collection of “objects” has had catastrophic effects.

https://www.blackliszt.com/2021/11/the-promising-origins-of-object-oriented-programming.html

Once you’ve built a house, you hope that modifying it is rare. Software, by contrast, undergoes nearly continuous modification.

https://www.blackliszt.com/2013/10/when-is-software-development-done.html

https://www.blackliszt.com/2021/07/fundamental-concept-software-is-rarely-built-mostly-it-is-changed.html

We maintain our cars, homes and devices. Heating and cooling systems need regular maintenance. So do our bodies! If we don’t care for our bodies properly, they break down! Software, by sharp contrast, never needs to be maintained. NEVER!

https://www.blackliszt.com/2022/01/software-never-needs-to-be-maintained.html

Waterfall and Agile are metaphors widely used in software project management. They hurt without helping.

https://www.blackliszt.com/2013/03/software-comparing-waterfall-and-agile.html

Software is amazing. We interact with what it does when using our computers and phones and when playing video games. Understanding what goes on behind the scenes is a whole different thing. It's understandable that metaphors are how people think about it. Sadly, the inappropriate metaphors result in bad software management.