The Black Liszt

Summary: Blackliszt and David B Black Writing

I have written about 500 blog posts on Blackliszt.com and published five books on computer software. This is an introduction with links to summary posts on the topics I've written about.

Throughout my career I followed the low-level strict rules, i.e., my programs compiled, ran and did what they were supposed to do. As the years passed, I learned that most of the widely accepted high-level rules lead to huge time and effort and produce mostly bad software. I first learned this as a programmer and CTO and then as a VC investor, seeing that the winners often follow a different set of rules. I've tried to explain the status quo and the better alternatives in my blog and books.The contrarian ideas and methods I describe are ignored and/or forbidden in software teaching and management. They represent a revolution in quality, efficiency and effectiveness, leading to superior results in a fraction of the elapsed time and often less than a tenth of the labor.

For details of my experience see this. https://www.linkedin.com/in/david-black-49b2b4a2

The Summaries

Computer Science doesn't come close to being a science. It takes baseless beliefs and promotes them as proven truths.

https://www.blackliszt.com/2023/04/summary-computer-science.html

For something that claims to be a science, software is amazingly fashion-driven.

https://www.blackliszt.com/2023/07/summary-software-fashions.html

Because software is invisible to most people, it's described in metaphors, which are mostly inappropriate and lead to trouble. Even the programmers use the metaphors, which often support methods that are consistently bad.

https://www.blackliszt.com/2023/06/summary-bad-software-metaphors-lead-to-bad-results.html

If you want to understand what's going on in software, it's natural to turn to the experts, insiders who share their knowledge so that most of us can understand the mysteries of software. There's just one little problem: software experts are as valuable as doctors were when blood-letting was standard practice.

https://www.blackliszt.com/2023/11/summary-software-experts.html

Everyone wants to know what's "new" in software; the new thing is most often just a fashion. Innovation? It's supposed to be what fuels software and changes the world. In reality, there is almost no innovation in software itself, and software-driven innovation is most often what I call in-old-vation, i.e., an advance that's proven in one domain that often takes decades to crawl into new domains. This summary also covers business strategies for taking software innovations to success.

https://www.blackliszt.com/2023/07/summary-software-innovation.html

How do you win with software? Nearly everyone starts with requirements, makes estimates and is judged whether they deliver on time and on budget. This optimizes for expectations and is a proven path to failure. The winning path optimizes for speed, customer satisfaction and continuous quality. It's what the people who need to win do.

https://www.blackliszt.com/2023/07/summary-wartime-software-to-win-the-war.html

Software people don't learn the fundamentals of computing; they may be aware of any one of them if asked, but don't apply them. In any other field, knowing and applying the fundamentals -- like blocking and tackling -- is demanded. Not in computing.

https://www.blackliszt.com/2023/06/summary-software-fundamentals.html

A key aspect of software fundamentals is software architecture. A surprising amount of the received wisdom about architecture is counter-productive, the goals are rarely discussed and the key to effective software architecture, Occamality, is largely ignored. Occamality is a foundational tool of wartime software: get the job done quickly and flawlessly, meet the real need and beat the competition.

https://www.blackliszt.com/2023/08/summary-occamality-and-software-architecture.html

If you want to architect and build good software, a major helpful factor that is usually ignored is to learn the history and evolution of software. Patterns repeat and knowing where you are in an evolution can help you predict what is likely to happen. Not to mention avoiding going down paths that usually fail.

https://www.blackliszt.com/2023/08/summary-computer-software-history-and-evolution.html

What do software people do? They write programs. What language do they use? What are programming languages, anyway? Are there good ones and bad ones? How have things changed?

https://www.blackliszt.com/2023/06/summary-software-programming-languages.html

Software people are indeed people, but a good fraction of them are different from non-programming people, with differences that are important to understand. Similarly, there are important differences in how to manage them most effectively.

https://www.blackliszt.com/2023/05/summary-software-people.html

Regulations are a problem in many areas, but are particularly pernicious in software and technology. By commanding that developers use largely obsolete methods for building technology, a huge increase in cost and time results, with no improvement in quality or outcome.This is a direct result of ignorant bureaucrats telling technical people in great detail HOW to get things instead of simply defining WHAT should be accomplished, like the criminal law does when it states simply that murder should not be committed.

https://www.blackliszt.com/regulations/

Computer security is assured by mountains of regulations, tools, standard practices and audits. But computer security continues to be breached without pause. There are major improvements possible that are largely ignored.

https://www.blackliszt.com/2023/05/summary-the-ongoing-problem-of-computer-security.html

Software quality assurance is an important specialty within the non-science of computing, but in spite of all the time and money spent, quality continues to be a major issue. There are solutions that have been proven in practice that are ignored by the experts and authorities.

https://www.blackliszt.com/2023/04/summary-software-quality-assurance.html

Everyone knows that software project management is essential to producing software that works, on time and on budget. In spite of decades of "innovation," it doesn't get better. The winners follow a different set of rules.

https://www.blackliszt.com/2023/04/summary-software-project-management.html

Huge amounts of attention are paid to AI, ML, Big Data, and the rest. It's a fashion that too often results in vast amounts of money being spent with little to show for it. Results can be amazing when achieved, but it takes a combination of deep knowledge and common sense to do it.

https://www.blackliszt.com/2023/08/summary-ai-machine-learning-big-data-math-optimization.html

Data centers and networks are the underpinning of computing. The transition to the Cloud has been a big factor, along with the rapid evolution of hardware and the glacial evolution of software. Net Neutrality has been an annoying furor, utterly divorced from reality.

https://www.blackliszt.com/2023/08/summary-computer-data-centers-and-networking.html

The vast majority of major organizations studiously follow the established methods of software design, construction, operation and security. They learn from academia, industry advisory groups and leading experts. The results are consistent. Disastrously consistent. This is true not only of governments but cool big tech companies including Google, Facebook, Twitter and the rest.

https://www.blackliszt.com/2023/11/summary-software-in-government-big-business-and-big-tech.html

Cryptocurrency and blockchain are highly visible for many reasons, including the fact that crypto fuels the explosion of ransomware. Leading thinkers are convinced that blockchain is a technology that will revolutionize computing.

https://www.blackliszt.com/2023/01/cryptocurrency-and-blockchain-summary.html

Learning about the mechanics and technology of the healthcare system provides an education on the huge opportunities to reduce costs, increase efficiency and make things better for patients and providers.

https://www.blackliszt.com/2023/08/summary-the-business-of-healthcare-medicine-and-insurance.html

I've had some personal experiences with the healthcare system that have resulted in a sobering education about what really goes on. Healthcare is incredibly valuable with many dedicated professionals involved, but there are some serious flaws.

https://www.blackliszt.com/2023/08/summary-the-medical-industrial-complex.html

I have published five books on these topics. The books are Software Business and Product Strategy, Software Quality, The Disease of Software Project Management, Wartime Software, and Software People. Here are introductory blog posts to them with links to the books.

https://www.blackliszt.com/2023/05/summary-david-b-black-books.html

 

What is your Software Project Manager's Batting Average?

How often do software projects fail? While it’s never talked about and rarely makes the news, the fact is that software projects fail with alarming frequency. Exactly how often do they fail? The shocking answer is, despite there being august professors of Computer Science and Computer Engineering falling out of the woodwork, doing research, pronouncing their deep knowledge in papers and teaching the next generation of students, the truly shocking answer is that no one knows! Or cares! Oh, they say they care about teaching people how to create great software, all you have to do is learn the intricacies of Object-Oriented Programming (or whatever). But none of them studies the results. There is no science in what is called “computer Science.”  They all declare they know how to create good software – so long as no one takes the trouble to measure and count, judge and evaluate, to see how often it happens.

In the past, I’ve compared bridges falling with software failures. We can understand this from another angle by looking at … baseball.

Baseball Science

Aaron Judge is an amazing professional baseball player, part of the New York Yankees. He’s an excellent fielder, but he’s best known as a hitter. In his nine seasons so far, he’s already hit over 300 home runs, with 50 so far as of this writing. His batting average this season is an awesome .331 so far, with a .288 average for his career.

Let’s think about that enviable .331 batting average he’s got so far this year. Any player would love to have such an excellent average. But that’s in the context of professional baseball. Simple arithmetic tells you that this “excellent” average means that he only gets a hit in one third of his at-bats! More than two thirds of the time he strikes out or otherwise fails to get a hit! What would you think of someone who managed to drive safely to work just a third of the time, getting into accidents or otherwise screwing up two thirds of the time? What would you think of a car that only worked a third of the time you tried to drive it? And so on … you get the idea.

Why is this? Are all these highly paid pro ball players really losers? Of course not. You can see what’s going on if you watch the home run derby, when top hitters go the plate and get soft, easy pitches from a pitcher standing behind a screen so they don’t get hit. Those guys nearly always hit the ball, and lots of them are home runs! But we all know that’s a once-a-year event.

Most of the time, there’s a highly skilled pitcher on the mound whose goal in life is to strike out the batter or sucker him into hitting an out. Pitchers like Gerrit Cole, who is so good that he’s being paid $36 million dollars this year.

When you’re a batter and walk up to the plate facing Gerrit Cole, you know you’ve got someone who’s in an excellent position to lower your batting average.

So is creating software like trying to get a hit with a fearsome skilled pitcher on the mound? Or is it more like trying to drive your car to work or designing and building a reliable car that just works, with remarkably few exceptions?

The sad fact is that building software resembles the home run derby, except that instead of trying to hit the ball out of the park, all the batter has to do is … not miss. With a rule like this, you’d expect something close to a 1.000 batting average. They try to make it even easier in the software world by messing with requirements, tripling estimates and doing everything they can to make the project a “success.”

Is software really that bad? Yup. Just for fun, I’m going to share a couple of the rarely publicized stories of software failures I made note of a dozen years ago. With things like ransomware exposing the ugly underside of most software operations -- not to mention the awfulness of computer security in general -- you can be sure things haven’t gotten better.

Sample Failures

In 2014, the VA admitted that they have over 57,000 patients waiting for their fist visit. What's their excuse?

"The official also said the VA needs to update its scheduling software package, which the department has been using since 1985. “It predates the internet and Blockbuster’s rise and fall,” he said."

Does that count as a software failure? It's more like the software department died years ago and no one noticed.

Here's a good one from 2012:

The U.S. Air Force has decided to scrap a major ERP (enterprise resource planning) software project after spending US$1 billion, concluding that finishing it would cost far too much more money for too little gain.

Dubbed the Expeditionary Combat Support System (ECSS), the project has racked up $1.03 billion in costs since 2005, “and has not yielded any significant military capability,” an Air Force spokesman said in an emailed statement Wednesday. “We estimate it would require an additional $1.1B for about a quarter of the original scope to continue and fielding would not be until 2020. The Air Force has concluded the ECSS program is no longer a viable option for meeting the FY17 Financial Improvement and Audit Readiness (FIAR) statutory requirement. Therefore, we are cancelling the program and moving forward with other options in order to meet both requirements.”

The Air Force will instead need to use its “existing and modified logistics systems for 2017 audit compliance,” the statement adds.

They started spending money in 2005, spent over a billion dollars by 2012, got nothing of value, estimated they'd need to spend the same again for eight more years to get about a quarter of the original plan done. If there were anyone paying attention to batting averages in software, that would likely be a winner.

Here is rare-to-find information on the frequency of failures from a book.

The odds of a large project finishing on time are close to zero. The odds of a large project being canceled are an even-money bet (Jones 1991).

In 1998, Peat Marwick found that about 35 percent of 600 firms surveyed had at least one runaway software project (Rothfeder 1988). The damage done by runaway software projects makes the Las Vegas prize fights look as tame as having high tea with the queen. Allstate set out in 1982 to automate all of its office operations. They set a 5-year timetable and an $8 million budget. Six years and $15 million later, Allstate set a new deadline and readjusted its sights on a new budget of $100 million. In 1988, Westpac Banking Corporation decided to redefine its information systems. It set out on a 5-year, $85 million project. Three years later, after spending $150 million with little to show for it, Westpac cut its losses, canceled the project, and eliminated 500 development jobs (Glass 1992). Even Vegas prize fights don't get this bloody.

If you care to look, you will find loads more examples of failures the group has been unable to keep secret. The failures keep rolling in spite of the huge efforts to reduce requirements, inflate estimates, extend time lines, increase staff and everything else. You have to ask the question::how many software successes are really failures in disguise? If anyone were serious about calculating software batting averages, this would be a key factor.

This pattern has resulted in some fairly widespread humor that you can be sure isn't mentioned in project management meetings. For example, here are the stages of a software development project:

1. Enthusiasm
2. Disillusionment
3. Panic and Hysteria
4. Search for the Guilty
5. Punishment of the Innocent
6. Praise and Honor for the nonparticipants

Why do Software projects fail and how do you win?

When lots of human beings work at something for a long time, they tend to figure out how to do it. Building software appears to be a huge exception to that rule. With decades of experience under our belt, why is software the exception?

This is a long subject. I have gone into great detail spelling out the causes ... and the cures!

Start with history and evolution:

https://www.blackliszt.com/2023/08/summary-computer-software-history-and-evolution.html

Everyone knows that software project management is essential to producing software that works, on time and on budget. In spite of decades of "innovation," it doesn't get better. The winners follow a different set of rules.

https://www.blackliszt.com/2023/04/summary-software-project-management.html

Software quality assurance is an important specialty within the non-science of computing, but in spite of all the time and money spent, quality continues to be a major issue. There are solutions that have been proven in practice that are ignored by the experts and authorities.

https://www.blackliszt.com/2023/04/summary-software-quality-assurance.html

How do you win with software? Nearly everyone starts with requirements, makes estimates and is judged whether they deliver on time and on budget. This optimizes for expectations and is a proven path to failure. The winning path optimizes for speed, customer satisfaction and continuous quality. It's what the people who need to win do.

https://www.blackliszt.com/2023/07/summary-wartime-software-to-win-the-war.html

 

Long-Standing Software Techniques Can Help Election Integrity

There is a massive secret process in election vote counting. It’s invisible, so observers won’t help. It’s subject to error and fraud. Existing standards, even Virginia Governor Youngkin’s Executive Order 35 ignores it. It’s not a hard problem to solve. Standard practices in database and data warehousing software that have been proven and refined in decades of use can be applied. The fragmented group of semi-custom voting equipment, largely driven by software-ignorant bureaucrats and regulators ignores this technology, if they’re even aware of it – no one is asking for it, so why bother? The process is simple in principle; it’s just arithmetic. It’s the process of registering each vote as it’s made, accumulating them with strict transaction logging, and gathering the votes recorded by multiple machines in a location, summing and displaying them in real time, and further gathering the votes from multiple locations and doing the same – by County and then by State.

A version of this process takes place today. But no one will tell you exactly how it’s done. What software exactly is used? How do the votes get from a machine to totals for a location? How is the flow done up the line to larger groups: towns, counties and states? Does someone read a display on one machine and type it into another? What’s the machine that does the summing, using what software? What are the formats used for transmission? No one wants to spell it out, keeping the whole thing invisible.

You think this is trivial? Think about the fact that ballots are different in each town to handle local elections and issues. The ballots are designed and entered into the various machines from different makers that are used. When the votes are reported, exactly how is each vote identified – by the name of the person being voted for? Is the name exactly what appears on the ballot, which is always entered locally? When voting for president, is the president’s full first name used? Middle name or initial? Is the vice president named, and if so exactly how? If there is so much as a single character mis-match in the naming, the totals may not be correct, depending on how they’re done, by whom and/or by what software. The problem gets worse when the votes (and names) are passed up the line to a county, which could have scores of separately programmed machines. Apart from naming issues, the right amounts have to be added to the right places – how is this arranged exactly? Not only is this an opportunity for error, it’s a massive opportunity for corruption and fraud.

The real solution to this problem is to eliminate local machines and controls altogether. This may sound impossible, but it’s not. See this for a description of the approach, with links for more detail. But so long as paper ballot counting machines are used, this problem must be solved to assure election integrity. Fortunately, the problem is one that has been addressed and solved in the commercial computer industry. It’s been used for dozens of years in various forms. The fundamental technology is database (DBMS) technology, of the kind that all the major vendors provide and in open-source. It is available in every major cloud platform. A form of DBMS technology called data warehousing has long been used for accumulating the results of transactions in databases for reporting, display and analysis. Moreover, there are long-established technologies for performing the essential operation of ETL (Extract, Transform and Load). ETL first enables a programmer to view the schema (data definitions) of the source and target DBMS and easily define how the data from the source is transferred and transformed as needed to the destination.

The process starts with the information that goes onto each ballot in the state, each county and each jurisdiction. Each one has a list of candidates and questions that need to go on the ballots. The higher entities would define all the entries that are common to each small entity within it; for example, the state would define state-wide ballot entries, counties would add ones for the county, and so forth. While the ballots are designed, the schemas for the DBMS and Data Warehouse would also be defined. The exact names and descriptions that go onto each line of each ballot would be matched with corresponding data element definitions in the databases (for original recording) and data warehouses (for reporting, accumulating, and rolling up), along with ETL. This could all be tested prior to mail-in ballots and early voting. Moreover, everything about it could be made fully public – the ballot definitions, the schema definitions and the ETL. Multiple processes in multiple (for safety) clouds that implement it could be made visible.

There is no reason why the whole thing couldn’t be made public. The data warehouses could update and display publicly their totals with updates as frequently as desired: each second, every five seconds, whatever. The totals could be displayed simultaneously for each voting locations, precinct., town, county and state. Along with display in real time of the voters who arrive to vote and ballots that arrive in the mail for processing, you have a fully transparent system.

Can databases handle this? Amazon’s cloud database handles over 10,000 transactions per second. Multiple copies could be used, with full redundancy. Capacity is not an issue, nor is reliability or security.

This does not fully solve the problem of intense feelings in local officials and the desire to adjust results to suit their preferences. But it goes a long way to getting those people out of the loop and making the essential back-office operations of accumulating the votes counted by the ballot counting machines transparent, until the proprietary machines can finally be eliminated altogether.

A Key Improvement to Youngkin’s Virginia Voting Integrity Plan

Governor Glenn Youngkin issued Executive Order 35 to assure voting security in the state of Virginia. The summary and the Order itself are worth a read. While some of it talks about existing procedures, together it’s an excellent foundation for assuring election integrity. The procedures include effective voting list maintenance, ballot security and machine testing, using only paper ballots. If all states adhered to this standard, our elections would be more secure than they are.

The biggest vulnerability remains the numerous local voting machines (ballot counters), supporting computers and officials. In this post, I describe the intense feelings some local officials have and their motivation to alter the election results. I propose a solution that eliminates all the propriety hardware with its attendant vulnerability to administrative corruption. Such a system cannot be built in time for this year’s elections. However, there are measures that can be taken this year that will address the vulnerabilities for corruption by local officials that are unaddressed by Executive Order 35.

The issue is the ballot counting machines are proprietary computer-based devices produced in small quantities by specialized companies to meet widely varied state-by-state requirements. In addition, they need to be coordinated with the design of the ballots, which cover not only national and state-wide voting, but also county-wide and often local elections on a single ballot. Each design needs to be coordinated with the ballot machines that will be fed the filled-out ballots. A ballot machine’s administrator sets it up for reading a specific ballot design, with a specific list of candidates and questions. Having the machine certified in general by bureaucrats is useless. What matters is whether the machine has been set up to read the specific ballot types it will be fed, and its ability to do so accurately. Then it matters if the machine is able to recognize that it has been fed the wrong type of ballot and refuse to process it. How are the results stored and given out for totalling? Is it on a display that a human enters into a computer? If so, using what program? How are the numerous ballot types and thus vote totals handled. And how is this able to be displayed along with the voter registrations, as called for in the EO? What computer using what software coordinates with all the local voting place computers to track totals?

I hope that the total transformation of local voting I have proposed happens in time for the next election cycle. Gov Youngkin’s EO handles most of the key issues of voter list maintenance and paper ballot handling and tracking. Paying some attention to the remaining vulnerabilities in the administration of the ballot counting machines and associated data handling would cure the largest remaining vulnerability short of eliminating the proprietary machines altogether.

Voting Integrity: The Problem of Local Control

Suppose some adult who had never played baseball, even for fun as a kid, was quickly taught the rules and played for a day. How well do you think they would play? Suppose they tried again two years later and then again a few times, each with a two year gap between tries. Do you think they would know what the infield fly rule was, much less be able to hit a pitched ball?

That’s what it’s like for the people who administer the voting machines at the more than 170,000 voting precincts in the US on voting day.

You might say it doesn’t matter, because they’re all skilled computer administrators. Sure. That’s like saying they’re good golfers, so when they walk onto the baseball diamond and stand on the pitcher’s mound, it’s no problem.

The kids who practice hard in Little League play the same game of baseball as the New York Yankees, just at a different level. What if the rules varied dramatically from place to place? What if some places had balls, but the bats were flat and there were wickets, like in Cricket, because each league makes up its own rules?

Even worse, suppose that many leagues decided that physical bats and balls were too old-fashioned, and that they should play video games instead? But each county and state would decide the rules for the video game its administrators would control and make different decisions? This is just like the mess with paper ballots and varying levels of automated counting and voting.

What about the makers of the equipment that was supplied for the game that was played every two years? They would have to carefully understand the updated rules each jurisdiction (county, state) would require and update their sporting equipment and video games to follow the requirements – and get it right the first time so that on game day, it would actually work and the inexperienced administrators wouldn’t screw it up too badly. What small company could manage to address such a semi-custom, always-changing market like this and do it error-free?

You’ve got people without experience playing a game once every two years with a wide variety of radically different, changing equipment made by tiny suppliers. That’s not all!

Suppose the point of this voting equipment is to enable local people all over the country to vote on their favorite baseball players. Suppose that officials in each state, county and town (there are over 3,000 counties and over 89,000 local governments) decided what and who should be on each ballot, and exactly how they should be named. There are local Little League players, players on local adult leagues and several levels of professional teams. Officials at each level control what the names are. Suppose one official decides that the player Mr. Berra should be called “Lawrence Berra.” Another decides he should use a more recognizable name, “Larry Berra.” Another decides that a widely used nickname is best, “Yogi Berra.” When it comes time to add up the votes, there’s a reasonable chance that all three Berras could have vote totals. This may sound like making fun, but it happens often in real life!

People feel strongly about baseball. They root for their home team, and can throw vicious insults at players for rival teams. Suppose it’s voting time for the best players. Fans come into the jurisdictions and vote in whatever way it’s supported. The inexperienced administrator controlling the local version of widely varied, changing equipment is in charge of counting the votes correctly, and other officials at the thousands of higher levels are in charge of summing up the votes they’ve been given for people who aren’t named consistently. This is baseball – the stakes are huge.

To give an idea of the intensity of the feelings, there is a widely read book called “Red Sox Fans Are from Mars, Yankees Fans Are from Uranus: Why Red Sox Fans Are Smarter, Funnier, and Better Looking (In Language Even Yankee Fans Can Understand)” How careful and strictly rule-following do you think the voting administrators in the Bronx and in Boston’s South End are going to be working with their peculiar, rarely-used semi-custom machines to do the counting? Not to mention the county and state-level administrators?

Suppose Yogi Berra isn’t elected a favorite baseball player and fraud is suspected as the reason. What can be done? The natural response is to conduct an audit of the voting that is considered suspicious. The trouble is, none of the systems (or people) involved have complete, secure, unalterable audit trails! Someone could have changed the settings on a paper ballot scanning machine and no one would know. If you tried to audit for the whole state, you’d have to go to the many places scanned paper ballots were kept, set up machines for the different rules in thousands of different towns (who’s on the ballot, where they are and how they’re named). Given the problems with inexperienced people and varied equipment, there will be problems during the audit just as there were during the original tally. Auditing electronic votes is even harder, if it can be done at all. And the audit will certainly take a long time, when everyone reasonably wants accurate results soon after voting closes.

Today’s voting and auditing is a highly suspicious mess for many reasons. Decades of incremental changes haven’t made it better. Can anything be done to achieve true voting integrity?

Yes. High integrity voting with near-real-time results can be achieved, but it requires a revolution in the voting process – not much for voters, but a complete re-start of the voting systems and counting process.

The main points of the new system are:

• Eliminate electronic voting machines. These are costly, widely varying devices that require skilled, accurate, bias-free administration.

• Eliminate paper ballot scanning machines that also count the votes; they are also computer systems requiring setup and administration!
  

• If people vote at a voting location, they fill out a paper ballot in private. They take the ballot to a screened-off desk and feed it to a scanning machine connected to a simple off-the-shelf computer with screen. After scanning, they see the image on the screen and can verify it’s accurate, or re-scan. The image is then converted to the intended votes which are displayed for confirmation. Upon confirmation, the voter may optionally be given a print-out record of their votes, like a receipt. The voting totals are instantly updated in a multiple secure Clouds.

• If ballots are received in the mail, the process is much the same, except a poll worker does the work at a voting station that isn’t screened off.

• The voting location is continually monitored with security cameras, including all handling of paper ballots, before and after scanning, and when boxed in a storage room. The rule is that ballots can go into the room but they can't be removed.
  

This revolutionary process eliminates all the local custom equipment and human administrators, with the attendant risk of error and corruption. The new centralized system needs to be designed and administrated carefully to assure effective and secure results.

If the typical government or corporate bureaucracy were in charge of building such a system, we could expect the usual results. But for a small, entrepreneurial group using COTS equipment and software taking an open-source approach with full transparency of results, for example publicly showing vote totals as they were made within seconds, it could be done quickly and rolled out incrementally.

Of course government politicians and bureaucrats would have to support the approach and give up their iron-clad control of the process. But once it was proven in small-scale practice, the only reason to resist would be their support of the existing expensive and fraud-enabling process.

There is a great deal more to be said about how such a system could be implemented, some of which I have described here.

https://www.blackliszt.com/2020/12/how-to-build-a-secure-auditable-voting-system.html

This isn’t the whole story of how to achieve voting integrity. There are the important issues of accuracy of the voting rolls, voter ID and preventing duplicate votes. But a system of this kind would be a big step ahead.

Why is Writing Computer Software Mostly Dysfunctional?

While not much discussed or broadly recognized, the vast majority of efforts to build computer software are disasters. They take too long, cost too much, and result in varying degrees of crap. Lots of solutions have been been promoted and tried. None of them have worked.

There are exceptions, of course. The exceptions prove that it really is possible to create good software quickly and efficiently. It is highly unlikely that the cause is that the methods taught in academia are good, but screwed up when applied; instead, it’s likely that nearly everyone has it wrong, just like doctors did when they bled patients to cure them and refused to sanitize when operating on patients.

A major cause of the dysfunction can be found in the approach to building software that was taken for good reason in the early days of computing. This approach was necessary for the first decades of computing. As computers grew more powerful, the necessity of doing things the same way began to fade away and finally disappear, Many of the early cumbersome practices were discarded, but the key focus has remained the core of software development to this day.

What is this universally accepted, unquestioned aspect of building software that does so much harm? Simple: it’s obsessing on software imperative language, relegating data definitions and attributes to a necessary annoyance, confined to a tiny island in an obscure corner of the vast sea of procedural language.

Does this sound awful? No. Everyone who does it doesn't think they're obsessing -- they're just working, writing their code! Similarly, getting water from the local community well didn’t sound awful in the 1800’s – until people finally found out about water contaminated with diseases like cholera. It took decades for the need for sanitation to be taken seriously, when the result of doing it poorly was death! We can hope that procedural language obsession will in the future be recognized as the main source of disease in software.

Early Computing

The roots of the language obsession are in the earliest days of computing. It was one thing to build a machine, and quite another to get it to do what you wanted it to do. The start was plugs and switches. Then the stored program computer was given step-by-step instructions in binary machine language. In the 1950’s first FORTRAN then COBOL were invented to make the process of creating the precise instructions needed easier, while still enabling the computer to operate at maximum speed.Those were indeed big advances.

In the 1960’s it still took a great deal of careful work to get results in a timely manner from computers. While languages like FORTRAN made writing easier, the fact that a compiler translated them to maximum speed machine language made their use acceptable.

The Apollo space capsule had a custom-built guidance system that was essential to its operation. Here is Margaret Hamilton next to a stack of code she and her team wrote for the Apollo Mission computers.

The Apollo guidance computer was a fast machine for its day, but the programmers had to get all the power out of it that they could to guide the capsule in real time. This is an extreme example, but 20 years into the computer revolution, everyone focused on using compiled procedural languages to get performance, and assembler language when necessary.

It was already evident that getting programs written quickly and well was incredibly hard. In fact, a big conference was held in 1968 to address what was called the "crisis" in software. Nothing got fixed. Meanwhile, efforts continued then and to this day to invent new programming languages that would miraculously make the problem go away. Nothing has changed for the better.

Partial steps towards declarative

From the early days to the present, there have been isolated efforts to go beyond simple definitions of data for procedural commands to operate on. Generally, the idea is that procedural commands spell out HOW to accomplish a task, while data definitions and attributes define WHAT the task is. It's like having a map (what is there) and directions (how to get from place to place on the map). See this for an explanation.

The invention of the SQL database was a small but important early step in this direction. SQL is all declarative. It is centered around a schema (a set of data definitions organized in rows and columns). The SELECT statement states what you want from the database, but not how to get it. WHAT not HOW!

You would think this would have led to a revolution in language (HOW) obsession. It didn't. In fact, because the language obsession stayed in charge, in some ways things got worse.

A few years after the DBMS revolution, people started putting big collections of historic data into what were called data warehouses. The idea was to make reporting easier without impacting production databases. Before long, OLAP (OnLine Analytical Processing) was invented to complement existing OLTP (OnLine Transaction Processing). While there were many differences, the core of OLAP was having a schema definition in the form of a star (star schema), with a central table of transactions and tables related to it containing the attributes (dimensions) of the transactions, typically organized in hierarchies. So there would be a time dimension (days, weeks, months), a location dimension (office, region, state) and others as relevant (sales, profits, department, etc.). After constructing such a thing, it was easy to get things like the change in sales from month to month in the hardware department without writing code.

OLAP was and is powerful. It assigned attributes to data in hierarchies, with unchanging programs that made it easy to navigate around. You could add attributes, dimensions, etc. without changing code! What an idea! But the idea was strictly confined to the isolated, distant island of OLAP and had no impact on software as a whole. The procedural language obsession continued without pause.

Declarative front and center

Procedural code is necessary. Code is what makes a machine run. However, the time for near-exclusive obsession about procedural code has long since passed. Limitations of computer speed and storage space were a legitimate reason to obsess about using the speed you had optimally. Think about cars. Engineers worked for decades to get the maximum speed of cars from about 10MPH to finally breaking 100. Now it's in the hundreds. In a much shorter period of time, computers have increased in speed by millions of times. Computer speed is rarely an issue.

How can we spend a little of the mountains of excess, unused computer speed to help make creating computer software less dysfunctional? Maybe instead of concentrating on procedural languages, there's a another way to get computer software to work quickly and well?

There is a proven path. It's obsessing about the WHAT is to be done. Obsessing about the data and everything we know about the data -- its attributes. This means applying the fruitful but limited approach we took with OLAP, and extending it as far as possible. In other words, instead of creating a set of directions from every possible starting point to every possible destination, we create a map in the form of metadata and a tiny, rarely-changing direction-generating program that takes as input starting and ending points and generates directions. You know, like those direction-generating program that are so handy in cars? That's how they work! See: https://www.blackliszt.com/2020/06/the-map-for-building-optimal-software.html

When we do this, we'll advance from the endless complexities of the solar system as described by Ptolemy to the simple, clear and accurate one described by Newton. What Newton did for understanding the movements of the planets, Metadata obsession will do for software. See this for more: https://www.blackliszt.com/2022/10/how-to-improve-software-productivity-and-quality-code-and-metadata.html

Software development is stuck in the endlessly complex epicycles of Ptolemy; we need to get to Newton.

 

How Many Jobs Will AI Eliminate?

The new ChatGPT technology answers questions in English in a way that can be hard to distinguish from what a human would have written. This is scary! What will happen to all those highly paid knowledge-worker jobs?

Notice how none of the stories talk about the history of AI. AI has been on the verge of ousting human beings from important jobs since the 1950’s. First it was checkers. By the mid-1960’s, ELIZA and SHRDLU were having conversations with people in English. ELIZA impressed many with its conversational, interactive abilities, while SHRDLU could not only talk but it could answer questions and perform actions in its world of blocks. By the early 1970’s many experts were talking about how AI would soon rule the world. See this: https://www.blackliszt.com/2019/12/getting-results-from-ml-and-ai-6-fintech-chatbot.html

All the talk about what would happen “soon” faded away and was forgotten. Then so-called “expert systems” were all over the news, with lots of investment and talk about how this time it would really happen. Before long IBM’s Big Blue beating a human chess master captured the news. Lots of talk, little action.

We’re in a hot cycle again. It’s often hard to tell whether a given paragraph was written by ChatGPT or a human being! Loads of people who write for a living will be put out of work, and before long the robots will take over!

After over 60 years of AI being super-hot and then fading away with little change, what’s different this time? Not much.

So how many jobs will be lost? A couple of centuries of experience with automation gives us the answer: lots of jobs will be eliminated through automation – but slowly, step by step and with lots of resistance along the way.

At the time of the American revolution, over 90% of the population was involved in agriculture. One step at a time, those jobs were automated, so that today, under 1% of the US population is employed in agriculture. Yet there's food enough for everyone. While the distribution of jobs is vastly different today, the population as a whole is more productive, better off and largely doing jobs that did not exist back then.

The driving force of automation is reducing or eliminating human labor. The most repetitive, easily taught jobs that have the lowest status are nearly always the ones eliminated. The people whose jobs are eliminated usually don’t like it, and sometimes fight to keep their jobs. For example, in the 1810’s in England, automated Jacquard looms created textiles faster and better than human-operated looms. The workers didn’t like it; a movement was started by Ned Ludd, the first person to smash a loom in protest. A secret society called Luddites did their best to resist the automation but lost in the end. See this: https://www.blackliszt.com/2020/01/luddites.html

This may not seem relevant to computers eliminating human labor. Looms, after all, aren’t electronic computers. However, the revolutionary Jacquard looms were mechanical computers -- they executed a "program" that was encoded physically in punched cards, enabling them to flawlessly execute elaborate patterns in the woven cloth.

Bottom line: automation continues to replace people in jobs on a regular and ongoing basis, just as it has for the last couple of hundred years. The people who are affected resist it if they can, like the Luddites two hundred years ago who smashed the Jacquard looms that were putting them out of work. But in the end, the automation happens. It's tough for the workers whose jobs are eliminated, but overall, people are better off.

But isn't the new AI, ChatGPT, a total game-changer?

How about lawyers? They write a lot – surely ChatGPT will produce what they do more quickly and inexpensively. Nope. Lawyers who write repetitive documents like wills are already being displaced by technology much simpler than ChatGPT – technology that produces reliable, best-in-class results each of every time. LegalZoom, for example, guides you through questions and choices and assembles a will for you from boilerplate chunks and fill-in’s that meets and the requirements of your state. A human lawyer couldn’t do it better – and in fact could do it worse, since every decision, choice and boilerplate chunk in LegalZoom (and its competitors) is approved by legal experts.

The automation of writing legal documents like wills and LLC’s illustrates the key reasons why amazing AI achievements like world-class chess playing and ChatGPT don’t end up empowering widespread change: narrow, subject-specific algorithms that produce consistent, transparent and auditable results are always better.

How about in the medical world? A battle to apply AI in medicine has been going on for years. IBM’s Watson AI system has made repeated deals with prestigious hospital systems – deals that have mostly fallen apart. This is partly because the doctors resist being replaced, and partly because Watson simply can’t handle the complexity and nuance of being a good doctor. Nonetheless, automation is taking place in medical systems, mostly in the support staff, the people who help schedule and the ones you never see in the back office moving all the information among different systems that keep the system going. Much of this technology is sophisticated RPA (Robotic Process Automation), which is like a robot operating different pieces of software on different screens. See this: https://www.blackliszt.com/2018/08/getting-results-from-ml-and-ai-4-healthcare-examples.html

ChatGPT is indeed amazing technology. Some version of it is likely to play some kind of role in ever-evolving automation. But, like earlier AI "revolutions," it won't have nearly as much impact as people think, and the path to practicality will remain narrow, focused applications.

The Amazing path to High Quality Fast-to-change Software

Everyone wants software that does what it’s supposed to do, runs fast without down time and can be changed quickly without causing problems. Who doesn’t want this? For more details on software goals, see this.

Everyone claims that their methods are great at achieving those goals. Sadly, such assertions are mostly baseless and in fact the touted methods do a terrible job. But they’re standard practice!

I’ve written extensively about they way to meet the goals in dozens of blog posts and a couple of books. Here is a specific step-by step way to move from standard architecture to one that meets the goals in simple terms. It’s a modified version of a path taken by a small, rapidly growing software company that I worked with. Incremental steps of this kind, with value at each step, are usually superior to the massive re-write approach that some software people are tempted to take.

When the new tech leader took over, there was no QA. So he first put in simple, UI, click-script based QA, and a full build process. Having done that, he wanted to move from his existing code organization to components to make change cleaner. But he had a reliability problem – it was good, but because of the nature of this customer base, it needed to be near-perfect. He started by thinking about adding a second data center, putting in database replication and then somehow doing a switch-over when the primary database went down.

Replication

He first focused on replication. All the transactions would arrive at a primary data center, with results stored on disk. The disk contents would be replicated to a backup data center, so that when it needed to take over, everything would be in place. It’s easy to set up a storage system to do this.

The trouble is that replication & availability functions have been moving up the stack, for good reason. While it may be trickier to set up database replication, the results are usually much better.

DBMS replication

Even when handling replication at the DBMS level, the DBMS transaction is guaranteed, but the user’s task may be just half done. The database will think everything is cool, but unless the application is 100% lined up, the results could be bogus.Yuck.

Application-level replication

Serious applications tend to have application-level replication. This basically means recording the field-level changes a user makes during an application session, from start through confirmation. This can and should be done in a generic way, so that when the application adds or changes a field, the recording, transfer and replay at the other site doesn’t need to change. This has the same net effect as database replication and avoids the issues.

Application log

Building an app-level log for replication is VERY close to building a … user transaction log! Which you’d like to have anyway! So make sure the extra information is there. This has many uses, including being able to show the user what they did and when they did it.

Instead of needing a big, intense, fast connection between the sites like you need for storage or database replication, you just need to ship and apply the session transaction log, which you can do at the end.

Replaying application transaction logs

Application transaction logs solve the reliability problem and give the user full access to their history with the application. They are also the crucial foundation of an incredibly important architectural advance: champion/challenger QA. This is how you can dump those QA scripts and enable rapid testing of new features for rapid, problem-free deployment.

In addition to having two identical copies of the application running in the two data centers, you bring up the proposed new version, Once you’ve got that, you can replay logs against a new copy of the application, and make sure you get the same results as last time. Guess what – that’s real-life QA! The only extra thing you need is the comparison, which again can be built once, regardless of whether there are 10 functions or 10,000. See this for much more.

Live parallel test

Once you’ve got that, you can implement live parallel test, which takes all the strain and risk out of releasing. It means doing the replication to another copy of the stack, just like you do for DR, except it’s the new code. If everything works well for however long you feel is necessary, all you do is switch which copy of the application sends results back to the user.

What’s gone when you do this thing that a small number of successful organizations do to their great benefit? Among other things: QA scripting. Double coding. Fixing QA bugs. Changing the script when you change the app. Unit testing. Test-driven development. Etc.

Moving to components

Yes, components and layers and objects and microservices are what experts are in favor of. Beware of experts! And above all, even though you may not give a talk at a conference about it, migrate your code to ... yes, a monolithic architecture.

Can’t you use queuing when it seems to make sense? Of course you can! But it should be something in-memory and simple like redis queues.

Active data migration

It’s standard practice to store all user and transaction data in a DBMS, and to call on the data from the DBMS when the user wants to do something. Sadly, this standard approach has long-since become obsolete as I explain in this post from more than a decade ago. It’s is a killer of application speed, flexibility and everything else. And there are proven in-memory databases like redis that can do the vast majority of jobs. If you’re dying to use a DBMS, they continue to be useful for archive and reporting!

Doing this will also tremendously simplify your champion/challenger testing.

The final push to speed of application change

Doing the foregoing things to your application will help on multiple dimensions. The final push, which can be step by step, is the one that will do the most to enable unanticipated changes to be made to your application. Conceptually, it’s pretty simple. What makes change hard? Finding all the places that need to be changed. What if the number of places you need to go to make a change shrinks, approaching over time a single place needing to be changed? You’re in the winner’s circle! Bonus: as you do this, the number of lines of code in your application will actually decrease! There’s lots to be said about this, but the core concept and value really are as simple as this.

What are you waiting for? Get started! High quality, fast-to-change software is yours for the taking ... err ... fast for the building.

the Conspiracy to Prevent Innovation

There is a conspiracy to prevent innovation. It’s widespread, strong and effective, more in some domains than others. Like all  conspiracies, it goes to great lengths to conceal its actions and goals. The members of the conspiracy are desperate not to be revealed, because they would certainly be subjected at least to ridicule, if not total ostracizing. More important to the active members of the conspiracy, they would lose power, if not their jobs.

The anti-innovation conspiracy is at war with the transformative power of radical improvements of computer and networking technology. Computer evolution, which continues to follow Moore’s Law of exponential growth, without precedent in human experience, largely powers innovation, directly or indirectly. The exploding growth of computer power and network speeds with similarly amazing reductions in size and cost are an overwhelming force, practically begging people and organizations to tap its power and make things better for everyone.  See this:

https://www.blackliszt.com/2013/12/fundamental-concepts-of-computing-speed-of-evolution.html

As anyone can see with the transformation of rotary, wired phones into wireless, mobile computer phones, the forces of innovation are winning.

Victories of the Conspiracy

The entrenched anti-innovation forces refuse to concede defeat and give up. Decades into the computer revolution, the conspiracy lives on, showing few signs of weakening. It has established entrenched positions in a surprising array of places. Even when it can’t prevent innovation, the conspiracy manages to slow the pace of innovation to a crawl in places where conspirators maintain positions of power. They force innovators to slog through sodden, muddy paths, when in other industries, innovators are whizzing along on mag-lev trains.  There are also wide-spread pockets of resistance, in which the conspirators manage to hold off the application of proven innovations to their industry, often by decades.

Skeptical? An amazing fraction of what appears to be innovations are little but taking advances that are proven in a narrow domain and applying them to a new one. Here is the story of an algorithm that was standard practice in oil refinery operation over 50 years ago that, decade by decade, is still crawling its way into new domains.

https://www.blackliszt.com/2019/08/the-slow-spread-of-linear-programming-illustrates-how-in-old-vation-in-software-evolution-works.html

Here is an example of a truly beneficial innovation proven over 50 years ago that is still not used in medical imaging.

https://www.blackliszt.com/2019/07/barriers-to-software-innovation-radiology-1.html

https://www.blackliszt.com/2019/08/barriers-to-software-innovation-radiology-2.html

There are many more examples of the effectiveness of the anti-innovation conspiracy.

It's Secret!

Among the greatest strengths of the anti-innovation conspiracy is that it has not been “outed.” No one talks about an “anti-innovation conspiracy,” or even about “the forces that slow or prevent innovation.” The fact that innovation takes place widely, touching all of our personal lives, leaves everyone in awe of what innovation has wrought. And rightly so! What is not discussed are the deep forces that prevent and/or slow innovation. Without the resistance, the innovation that would be unleashed would make today’s by-itself-amazing innovation look sluggish by comparison.

The anti-innovation conspiracy has two major wings.

The first wing is particularly pernicious, since it establishes positions inside the people and groups who innovate and implement innovation. In this wing, the conspiracy concedes that innovation will take place, but assures that it will be as expensive, slow and ineffective as possible. The first wing is essentially an effort to cripple and co-opt the “offense,” the efforts of the innovators.

The second wing operates silently and secretly, often waving flags supporting innovation and supporting innovation efforts. It operates inside the institutions that will be affected by innovation, and inside the organizations such as regulatory bodies that control the institutions that are targeted by innovation. The second wing is essentially an effort to bolster the “defense,” the ability of institutions that could be impacted by innovation to resist the efforts of innovators.

One of the great strengths of the anti-innovation conspiracy is that the vast majority of people benefit to some extent by the innovation that does take place, and naturally compare  what is available to them today compared to ten or twenty years ago. Progress has happened, and it’s a good thing. Of course I agree with this.

But meanwhile, the conspirators are rubbing their hands together and mumbling something like “heh, heh … still got ‘em.” The reason for the acclaim for the results of innovation and the near-complete lack of awareness of the conspiracy against it is simple: very few people are in a position to see the amazing innovations, true advances that would make everyone’s lives better, that have been and are being today prevented and/or drastically watered down as a direct result of the efforts of the anti-innovation conspiracy. If there were widespread knowledge of “what could have been,” people would see the results of innovation in a completely different light. It would be like a starving person in a room being given a bit of bread and water; they eagerly consume it and are glad to have it. But how would their attitude be different if they knew that, just through a door in the room, there was a sumptuous feast all laid out, ready to eat, and the person giving them the bread and water was preventing them not just from going through that door, but even from letting them know it was there? Gratitude for the bread and water would suddenly be transformed into fury at the person keeping them from the food they so desperately need, even hiding the fact that it’s there! The vast majority of people are like the starving person in the room, grateful for the food they now have, completely unaware of the feast they can’t eat because the conspiracy successfully hides it from them.

I know these are strong claims. But after decades of writing innovative software and more decades of looking deeply at innovative companies and their spread of innovation in various companies and industries, this describes the patterns I have observed.

Finally, let me make something clear: There is no cabal of anti-innovation conspirators who explicitly communicate among themselves that their goal is to prevent or slow innovation. If there were such a cabal, it would have been exposed and shamed long ago. The long-lasting conspiracy is fueled by a wide variety of goals that appear to have nothing to do with preventing innovation; the conspirators, as they deserve to be called, don’t say or even (for the most part) tell themselves that they are trying to prevent innovation. It’s always something else: they are trying to protect the public, prevent people from being harmed, assuring that things are built to proper standards, trying to maintain stability in a well-oiled operation, avoiding wasteful distractions, and on and on. Most of the people involved might sincerely claim they are promoting innovation everywhere they can. The trouble isn’t their thoughts or words; the trouble is their actions, which have the net effect of preventing or slowing innovation.

I have already written about some of the issues involved in the anti-innovation conspiracy.

https://www.blackliszt.com/2023/07/summary-software-innovation.html

There is more to be written, for example to shame those who promote themselves and their institutions as being leaders in innovation. I hope others will take up this long-neglected issue.

Well-trained Managers Create Software Failure

Many software projects fail, or in various ways don’t achieve the expected results. This is widely known. Loads of people offer cures, for example the fact that the critic’s favorite software development method wasn’t used, or it if was used, wasn’t used correctly. Training! Consultants! More SCRUM masters! Not just stand-up meetings, but stand-on-your-tippy-toes meetings!

There is a fatal flaw that often causes software projects to turn out badly. It’s rarely discussed. It’s sometimes called motivation, but it comes down to whether the people doing the actual work – you know, writing the lines of code – feel ownership in what they’re doing. If they don’t own the work, their eyes will be glazed over, and the most productive corners of their minds will be engaged elsewhere.

If they’re engaged, excited and feel they own the project, however, they are likely to be chastised, boxed, referred to HR, etc. Someone will sit down with them and utter phrases containing words like “team,” “supportive,” “respect other people,” and so on, all of which are generically good, I guess, but amount to something like this: “Sit down, shut down, keep your nasty opinions to yourself, stop putting other people down, and just do your work.” In other words, go through the motions.

The people who say things like this are mostly not programmers – they wouldn’t know a line of code if they tripped on it – or are sometimes go-along type programmers who see that corralling wild programmers is the path to higher pay and increasing management responsibilities – in other words, more pay for less work.

Dilbert has an on-point description of software management.

Yes, some good programmers are mean, sometimes. But mostly they just sound mean when what’s really going on is that they’re reacting strongly to stupid ideas or proposals. Because they care. The targets of their wrath should grow up and consider the possibility that they’re doing the equivalent of proposing that the race should be run by people holding their hands behind their back, or hopping on one foot. Is it nice to mock such a brain-dead idea? No. Should the person proposing the brain-dead idea be made to understand that they’d better do a brain transplant and get a brain that’s at least moderately alive, or switch jobs to serving the coffee? Yes. The sooner the better. Perhaps the idiot could even try listening to the “mean programmer” and understand the substance of what they’re saying, instead of just the emotional baggage.

That’s a nice fantasy, but very rarely happens. Most often, it’s the unrestrained programmer who is put down. Top programmers are given a clear choice. Choice 1: leave. Choice 2: do what you’re told and stop caring.

Dilbert boils it down: learn to lie.

Dedicated MBA/HR types will counter with stuff about the importance of teamwork, how the team succeeds together or fails together, the programs they run to inspire motivation, belonging and dedication to quality, and all sorts of good-sounding things. Good intentions! Sorry, but in the end, the only thing that matters is results. If your “team” has carefully included everyone and reached a consensus about the best path to get to the top of the mountain, and gets stuck part way up with nowhere to go except jumping a cliff or retreating to the base of the mountain, well they’ve failed together. I hope they’re happy. Meanwhile, some collection of misfits could be charging up the mountain in leaps and bounds, anticipating obstacles that your wonderful crowd of conformists and liars were blind to, and reacting creatively to obstacles they had failed to foresee, overcoming them quickly

This is one of the big reasons why large organizations can’t get software done: they are infected with large numbers of trained, credentialed managers and HR people who know nothing about the substance of software and are convinced that their generic management methods should reign supreme.

 

Excellence in Government IT: The Social Security Administration

I got an email update from the SSA It said:

Your Social Security Statement is streamlined and easier to read than ever before. That is because we have redesigned the Statement to provide you the most useful information up front and at a glance.

...

You can access your new Statement by signing into your account at socialsecurity.gov/reviewyourstatement.

Now that you can access your Statement instantly and anytime online, we will not automatically send one by mail.

We hope you find your new Statement useful and informative.

i decided to check out this wonderful new statement that's available "instantly and anytime.". Here's what happened when I clicked the link:

Is there any better evidence that government agencies are incapable of achieving the minimal level of software competence that the vast majority of software groups are able to achieve? Is it possible to be more arrogant than to claim that statements are available "instantly and anytime" and then not do it?

I can only guess that the government buys special computers that need a minimum of 4 hours of rest a day, more on weekends. Because 24 by 7 availability of simple things like this have been easily achievable for at least 25 years.

I particularly love the statement in the error message that says "If you need immediate assistance you may call us Monday through Friday 8:00am - 7:00pm." Is it possible that the government has a meaning for "immediate" that isn't what normal people think it means?

Cat litter is more scientific than Computer Science

Which is more scientific: Computer Science, which is all about numbers and data and exact algorithms, or cat litter, which is all about giving cats a way to poop and pee while reducing the annoyance of their human servants? Seems obvious, right?

It’s obvious until you understand the practical disaster of creating software as detailed in these posts,

https://www.blackliszt.com/2023/04/summary-software-quality-assurance.html

https://www.blackliszt.com/2023/04/summary-computer-science.html

and until you pay careful attention to the information typically found on a box of cat litter and observe how closely the litter does what it’s supposed to do.

Result: the litter wins by a mile.

Check out the information on this box of cat litter, a type I often use.

 

Let's see in detail how software stands up against cat litter.

• Problem clearly defined

          The problem solved by a given piece of software can be stated in documents that are huge. And too often vague.

          By contrast, the problem solved by cat litter is clearly defined and already known to everyone who knows how mammals work. There are lots of variations of the details of how cats do what they do; cat litter handles them all.

• Verifiable promises

          When you buy software or request it to be built, we all experience how hard it is to tell whether we actually got what we were promised. Departments tend to go through extensive “acceptance testing” with new software, or even a new release of existing software, because it’s too often not right.

          With cat litter, you can tell pretty quickly whether the promises are kept. You can see and smell the results.

• See the results

          Yes you can see the results of software. But there are a remarkable number of invisible results, like databases and files being created and updated, messages being sent and received, other programs being invoked.

• Consistent results

          Even when software works, it doesn’t work all the time for everyone. Sometimes it goes down totally. Sometimes it just produces bad results – or none at all – for some people under some circumstances. Sometimes you can tell the result is bad but sometimes it seems right but is wrong. Or it crashes.

          People buy new boxes of litter when the old one is used up. They buy it again and again because … it works! All the time, regardless of the variations of the cat’s output.

Conclusion

I could go on, but it should be clear by now: cat litter is an exemplary model of science and the associated engineering and production. What should be jokingly called “Computer Science” may be a lot things, but “Science” is not one of them.

Summary: Software in Government, Big Business and Big Tech

This is a summary with links to my posts on the many ways that large organizations including government, big business, big tech and the rest diligently apply modern software procedures as taught in academia and required by professional management; they consistently produce disastrous results in software quality, cost, security and everything else that matters.

There are of course issues that are common to all these large organizations, for example in cybersecurity.

https://www.blackliszt.com/2015/06/systemic-issues-behind-the-cyber-security-disasters-at-opm-citi-anthem-etc.html

Government

Government software disasters are government-as-usual, so much so that disasters that wreck lives barely make the news. For example, over 10 million people world-wide enter a government-run lottery for immigration slots that can lead to US citizenship. How hard can picking a bunch of random numbers be? Apparently too hard for the government software people, with the result of horrible consequences for the declared lottery winners whose immigration slots were invalidated.

https://www.blackliszt.com/2011/07/software-quality-horror-tales-electronic-diversity-visas.html

Consider the sets "Excellence" and "Government IT." There is a great deal of evidence that these are non-overlapping sets. I learned there are organizations promoting and celebrating digital government. They hold awards ceremonies. I tried to find out what the winner had done to deserve winning. Surprise, surprise, the link at the organization’s website explaining it all was broken. Pathetic.

https://www.blackliszt.com/2015/05/excellence-in-government-it.html

Even simple things like making Social Security statements available on-line appears to be beyond them -- including of course lying about it.

https://www.blackliszt.com/2024/03/excellenece-in-government-it-the-social-security-administration-.html

The NSA (National  Security Agency) has a budget of over $50 Billion and is touted as being the world’s best at cybersecurity. It turns out the only reason we know their super-top-secret budget is because their security was blatantly breached with massive internal data made public.

https://www.blackliszt.com/2014/05/bureaucracy-regulation-and-computer-security.html

Given that this army of highly-paid cyber geniuses can’t protect itself, it’s not surprising that its analysis of a high-visibility security breach may have sounded good to the public, but was in fact entirely fraudulent.

https://www.blackliszt.com/2017/01/russia-hacks-dnc-podesta-email-fake-news.html

What do you do with such a huge budget when you’re unable to do what you’re supposed to do even with your own secrets? You set up a massive program to teach students your excellent methods and hope to train over a million certified experts. I tracked the program from a local community college to the NSA’s own description of its program – which was both broken and insecure!

https://www.blackliszt.com/2017/06/government-cyber-security-tops-the-oxymoron-list.html

Unfortunately, this isn’t just about keeping information safe. Government ineptitude kills people. Instead of taking a quick, simple approach to preventing train crashes:

https://www.blackliszt.com/2015/05/an-app-to-prevent-train-crashes-like-amtrak-philadelphia.html

The government presses on with its super-expensive solution using obsolete technology, which leads to yet more preventable crashes and deaths.

https://www.blackliszt.com/2016/10/scandal-hoboken-train-crash.html

It’s not just big governments. The little government of several islands in the Caribbean managed to create a multi-front disaster using best practices to foist a digital currency system on its innocent citizens.

https://www.blackliszt.com/2022/03/dcash-government-cryptocurrency-shows-why-fedcoin-would-be-a-disaster.html

https://www.blackliszt.com/2022/03/what-is-behind-the-dcash-central-bank-digital-currency-disaster.html

The US government continues to pursue a national digital currency of the kind that has already proved to be a disaster in the Caribbean. They do so ignoring the fact that the US Dollar is already largely digital, with extensive software support structures that are in place and working well..

https://www.blackliszt.com/2020/12/we-dont-need-fedcoin-we-already-have-a-national-digital-currency.html

Important things like voting systems are some combination of broken and insecure. I took the trouble to define a simple combination of tech and non-tech to build a modern, secure voting system that was auditable, with operations visible to every voter while keeping what they voted for secret. Will any government institution pay attention, much less implement it? We all know the answer.

https://www.blackliszt.com/2020/12/how-to-build-a-secure-auditable-voting-system.html

 

Big Business

Executives in big business want to succeed and advance, but this can only happen by avoiding risk. The best way to avoid risk is to do what “everyone else” is doing, what the experts say is best. That’s where industry advisory groups come in.

https://www.blackliszt.com/2017/05/the-value-of-computer-industry-advisory-groups.html

Giant advisory firms counsel their customers on how to make the best decisions. Getting your customers to like you is high on the list. Carefully crafted words are of supreme importance to such large organizations. Actions that match? Not so much.

https://www.blackliszt.com/2016/07/gartner-group-big-company-customer-service.html

A giant health insurance company “lost” the personal information of "tens of millions" of its members sometime in 2014; they're not sure how many, whose records were "lost," or when it happened. The details are an astounding illustration of big-corporate IT incompetence.

https://www.blackliszt.com/2015/02/the-anthem-of-cyber-insecurity.html

I soon found out that my information had indeed been stolen. The company’s response to the theft was right in line with their letting it happen.

https://www.blackliszt.com/2015/02/my-anthem-account-was-hacked.html

What company doesn't want to be part of the digital revolution and have an app? If you're a major health insurance company, why wouldn't you replace old-fashioned insurance cards with something always up-to-date that comes on an app? Here’s what ensued when one of the industry giants tried.

https://www.blackliszt.com/2021/02/why-cant-big-companies-build-or-even-buy-sofware-that-works.html

I've covered many big organization face-plants. The awfulness encompasses a broad range of consumer-dissing inconvenience, Here’s a case of some software that "works" but puts customer inconvenience front and center.

https://www.blackliszt.com/2021/03/why-cant-big-companies-build-software-that-works.html

Here’s a case of a giant company software issue that is low on the “it matters” scale, and high on the “a smart high school student could have done it better” scale. It’s the kind of issue that leads one to wonder whether we’d all be better off if they refused to hire any more people with college degrees for any job, and in particular, management.

https://www.blackliszt.com/2021/05/anthem-needs-my-feedback-reveals-deep-problems.html

Big Tech

Whether the software is a cool social app, an academic website or a real business, there is a common theme: the software is poorly designed and, even worse, it just breaks. You might think the cool internet apps like Facebook and Twitter are an exception, but they’re not.

https://www.blackliszt.com/2012/01/internet-software-quality-horror-shows.html

How can you innovate? Did the leaders of the current big tech companies benefit from training in innovation? Once they became large, have the big guys like Google demonstrated excellence in innovation? Uhh, sorry, the facts indicate otherwise.

https://www.blackliszt.com/2016/05/organizing-for-successful-innovation-recent-history.html

The widely-accepted logic is: Facebook is wildly successful; FB is built on software; therefore, FB software must be excellent. I should hire people from FB to help me build excellent software! The history and facts support neither the logic nor the conclusion.

https://www.blackliszt.com/2014/12/fb.html

I looked at FB’s mobile app when it had over 700 million people using it. Over 20 million people had written reviews, more than 6 million of which were 3 stars or less. A random sample of those reviews yielded juicy results.

https://www.blackliszt.com/2014/11/facebooks-software-quality.html

The difference between image and reality at FB is astounding. Here is an interview and a recent book that should lead any ambitious young company to avoid hiring people from there.

https://www.blackliszt.com/2017/03/software-giants-image-and-reality-facebook.html

Large organizations have trouble building software. This has been true since the dawn of software history, and shows no signs of changing. The decades-long, rolling disaster of Microsoft Windows is a great example of this.

https://www.blackliszt.com/2015/08/large-organization-software-fails-the-case-of-microsoft-windows.html

Microsoft illustrated multiple issues relating to digital ownership in a case I dug into. Among other things they attempted to require use of their own pathetic browser.

https://www.blackliszt.com/2014/05/giant-software-bureaucracies.html

There are big problems with software quality. The social apps in particular have decided it's embarrassing. But instead of actually, you know, fixing the problems, they seem to have decided to mask the problems! Twitter is a great example of this disease.

https://www.blackliszt.com/2013/05/twitter-software-quality-stinks.html

I did detailed studies on Twitter and found that they do indeed produce provably bad search results.

https://www.blackliszt.com/2013/05/twitter-software-quality-an-oxymoron.html

People write and talk about what's "trending on Twitter" as though the trend meant something. It doesn't. It's based on deeply flawed Twitter search software that gives random, widely varying results.

https://www.blackliszt.com/2013/05/the-bogus-basis-of-trending-on-twitter.html

Twitter fired boatloads of software engineers in 2022 leading some to predict that software disaster will ensue. But then, most people don’t know much about software and don’t realize what a disaster Twitter software has been for years.

https://www.blackliszt.com/2022/11/twitter-can-improve-software-quality-by-losing-most-of-its-engineers.html

Then there is Apple, the high-prestige computer company making expensive devices. In 2016, terrorists killed a bunch of people in California. Law enforcement and the FBI worked hard to find out what happened and who else might have been involved. This required looking in the government-issued iPhones used by the killers. What happened? Apple did its best to protect the criminals. Here are the highlights.

https://www.blackliszt.com/2016/03/the-apple-fbi-fiasco.html

And here are the details:

https://www.blackliszt.com/2016/03/apple-can-help-fight-crime-while-maintaining-privacy.html

https://www.blackliszt.com/2016/02/apples-cancer-prevention-strategy.html

https://www.blackliszt.com/2016/02/apples-approach-to-privacy-terrorists-and-criminals.html

I reviewed a book about government security on Amazon. The author was impressive and had loads of experience. Many of the reviews were positive, with a few pointing to obvious bias. I wrote a review that pointed to the positive aspects, but also mentioned some of the bias. The review disappeared. I interacted with Amazon, and was told that suppressing the review was a mistake. It appeared again. Then it disappeared. I tried to write a review and was told I've been banned!

https://www.blackliszt.com/2023/03/early-evidence-of-criticism-suppression-by-intelligence-agencies-.html

Yelp isn’t as big as the industry giants, but it’s pretty big. A random plunge into their system demonstrates the same kind of slick surface with rotten underpinnings as their larger brethren.

https://www.blackliszt.com/2021/05/yelp-big-tech-incompetent-corrupt.html

Conclusion

There is a better way! The winning methods aren’t even new – they’re proven in practice by small groups that need to win. See:

https://www.blackliszt.com/2023/07/summary-software-innovation.html

https://www.blackliszt.com/2023/07/summary-wartime-software-to-win-the-war.html

 

Summary: Software Experts

This is a summary with links to my posts on software experts. For medical experts see this:

https://www.blackliszt.com/2023/08/summary-the-medical-industrial-complex.html

The hype about experts is so extreme, it’s important to take a bit of time mocking it. After all, experts confirm that experts are super-smart and never make mistakes!

https://www.blackliszt.com/2021/04/experts-are-super-smart.html

Medical doctors considered blood-letting to be a standard part of medical practice until well into the 1800's. They continued to weaken and kill patients with this destructive "therapy," even as the evidence against it piled high.

The vast majority of software experts strongly resemble medical doctors from those earlier times. The evidence is overwhelming that the "cures" they promote make things worse, but since all the software doctors give nearly the same horrible advice, things continue.

https://www.blackliszt.com/2019/02/what-software-experts-think-about-blood-letting.html

The Wannacrypt ransomware attack caused havoc world-wide in major corporations and government institutions. It's a textbook lesson in a number of subjects including the worthlessness of most Experts and the rank illiteracy of otherwise highly educated journalists about computing.

https://www.blackliszt.com/2017/05/the-ransomware-hack-attack-lessons-from-the-experts.html

Governments are filled with Experts, particularly on important subjects such as security. In 2017 the US government declared that the Russian government hacked important US entities. The official report was filled with impressive-sounding evidence that demonstrated the incompetence and/or duplicity of the agencies that issued it. The majority of the US press simply echoed the nonsense.

https://www.blackliszt.com/2017/01/russia-hacks-dnc-podesta-email-fake-news.html

Big Data is one of those subjects that is widely touted but opaque to most people. It’s a classic forum for Experts to have their statements taken as infallible. A famous expert in Big Data and elections made an error in a recent election that clearly illustrates typical Expert behavior of seeking an outcome instead of the truth.

https://www.blackliszt.com/2017/04/big-datas-big-face-plant.html

Experts and anointed authorities of various kinds, both academic and commercial, have been the front lines of resistance to innovation for centuries. They are the firewall keeping rogue ideas outside the environments they oversee, protecting them from bad influences that their naïve but innocent charges might inadvertently adopt.

https://www.blackliszt.com/2020/05/experts-vs-innovation-new-book.html

Groups like Gartner sell expertise as a service. Go to them if you want to know the best thing to do! The real value of such groups is to help executives avoid risk and avoid blame when things go wrong. Innovation? Industry-acknowledged experts are not the way to go.

https://www.blackliszt.com/2017/05/the-value-of-computer-industry-advisory-groups.html

If you're doing something new and want to do it right, it's natural to seek the help of someone who's been there and done that. If you want to do the thing in an innovative way, that's all the more reason to seek expert help. The trouble is, however useless experts are in general, they’re worse when it comes to innovation. The history of manned flight is a good example.

https://www.blackliszt.com/2016/07/innovation-and-experts.html

 

Summary: Regulations

This is a summary with links to my posts on regulations.

Regulations are supposed to make things better. Most of the time regulations make things worse by preventing innovation, increasing costs and failing to achieve the goals for which they were created. The negative impact of regulations, whether government or corporate, is greatly magnified in software and technology.

The reason why our ever-growing number of regulations fail to protect us is simple. In the vast majority of cases, they spell out, often in great detail, how to accomplish the goal, instead of plainly and simply defining the goal and leaving it up to the person or company regulated to figure out how to get it done.

https://www.blackliszt.com/2011/12/regulations-goals-or-directions.html

Regulations that define a goal enable innovators to find new ways to accomplish the goal, whether it’s a better medical device, computer security, whatever.

Another way to think about effective regulations is the criminal law, which regulates behavior. There aren’t ever-growing mountains of regulations telling you how to avoid murdering someone, just a simple statement that murder is something you must not do.

https://www.blackliszt.com/2012/03/lets-criminalize-our-regulations.html

Regulations are front-and-center in the bureaucracy-driven battle to prevent innovation. Regulations are ever-growing mountains of words written by lawyers and bureaucrats. The current federal regulations have more than 100 times the number of words than the collected works of Shakespeare.

https://www.blackliszt.com/2016/09/innovation-the-barriers.html

Regulations and standards can be good; without standard steering wheels and brake pedals, no one would be able to drive a rental car. Software is different. The misguided effort to impose standards and regulations on software development has played a key role in the nonstop cybersecurity disasters and software failures that most organizations try to minimize and ignore.

https://www.blackliszt.com/2021/11/the-destructive-impact-of-software-standards-and-regulations.html

Medical device regulations increase costs and prevent innovation. The FDA device regulations provide an excellent example, declaring in massive detail how exactly to achieve quality in classic how-type method. The what-type (criminal version) would simply declare that the device must perform its declared function, accurately and well.

https://www.blackliszt.com/2016/12/regulations-that-enable-innovation.html

A similar story plays out in the field of medical imaging. The essential devices could be vastly improved if the regulators got out of the way.

https://www.blackliszt.com/2023/01/how-to-reduce-the-cost-of-medical-imaging-and-pacs.html

The bureaucrats who write regulations for software are ignorant of software. It’s literally invisible to them! Their understanding, such as it is, tends to be based on false metaphors and is wildly inappropriate. They end up requiring expensive, obsolete methods for building software that no sensible company would use.

The current mountains of regulations should be replaced by something like “We don’t care how you build your software, but it’s your responsibility to assure that the software performs its stated job without fail. If the software has errors that cause medical harm, you are responsible for the damage it causes, and you may be barred from supplying software to the medical market in the future.”

https://www.blackliszt.com/2020/05/heres-how-the-fda-can-reduce-medical-device-costs-while-improving-healthcare.html

Regulation is also about important things like making trains safe so that, even when something goes wrong, you don’t have a crash that kills people. That’s one the government has been all over. They devised a system called PTC (positive train control) to prevent crashes. The cost to implement it was tens of billions of dollars and took many years. Years after it was mandated there was a crash that killed people and injured hundreds.

https://www.blackliszt.com/2015/05/an-app-to-prevent-train-crashes-like-amtrak-philadelphia.html

The crash in Philadelphia wasn't a one-off. The problem is that PTC is built on computing technology that belongs in a museum, not supposedly protecting our lives.

https://www.blackliszt.com/2016/10/scandal-hoboken-train-crash.html

The technology exists to enable a more effective, inexpensive system to be built using modern technology. But of course the regulators ignore it.

Whenever the government wants to step in to improve a company’s software, beware. When has such a move ever had a positive impact on anything, much less the stated goal of the regulation?

https://www.blackliszt.com/2015/04/the-government-wants-to-help-ubers-software-quality.html

There has been a recurring furor about the unfairness of the internet. According to the critics, regulators at the FCC should step in and impose “net neutrality” to make service fairer for everyone. In fact, what few problems there are have been caused by regulation. And where regulation has been imposed elsewhere, government-mandated severe censorship has quickly followed.

https://www.blackliszt.com/2014/11/net-neutrality-it-aint-broke-dont-fix-it.html

It’s not just the government. Big bureaucracies to “control and improve” software emerge in giant companies of all kinds. Even in a software company, the in-house regulators can impose insanity. Here’s a case involving Microsoft and digital goods.

https://www.blackliszt.com/2014/05/giant-software-bureaucracies.html

Regulations have an outsized role in causing the on-going disaster of computer security. See these for examples and explanations:

https://www.blackliszt.com/2017/05/security-regulations-vs-security.html

https://www.blackliszt.com/2014/05/bureaucracy-regulation-and-computer-security.html

https://www.blackliszt.com/2013/07/cyber-security-standards-are-ineffective-against-insiders-like-edward-snowden.html

For more about security and how regulations make things worse, here is the summary post.

https://www.blackliszt.com/2023/05/summary-the-ongoing-problem-of-computer-security.html

Moving towards Occamal software

Someday, there will be tools that actively help you build occamal software. I imagine that the tools will resemble a modern IDE, but will have assists, wizards, visual representations and other methods of helping you see actual and potential commonalities. In addition, there will be common components, both part of the development environment and part of the execution environment, which will make building occamal software easy and natural. Until such tools and components exist, the work and imagination of the developer will have to fill the gap. Remembering that we live in the real world, our goal is not to build software that is perfectly occamal; software that is nearly occamal would be wonderful, and software that is much more occamal than today’s software would be a big improvement.

A good way to think about occamality is the elimination of redundancy. It may be useful to put redundancy into categories that vary by blatancy. So we can identify:

• Simple redundancy

Simple redundancy is really, really blatant redundancy. There’s not much excuse for it, although some programming environments are shockingly encouraging of it. A great deal of the Y2K problem was due, sadly enough, to simple redundancy. Most of the relevant programs were written before databases were in common use, and there was in any case no native support for the “date” data type in the programming environments used; in the vast majority of cases, no one bothered to take the simple steps required to create the then-equivalent of a date data type. During the seventies, instead of eliminating simple redundancy, people spent their time engaged in fierce ideological arguments about whether “structured” programming represented an advance in program quality and programmer productivity or whether it was just a bunch of hoo-hah. The extent and depth of the Y2K problem told us the answer.

Simple redundancy is normally cured by replacing the redundant instances with references to a definition of whatever it is that they have in common.

• Complex redundancy

Complex redundancy takes some real effort and energy to eliminate, though often not a great deal. Complex redundancy involves redundancy over multiple programming environments or some other issue that raises it above the simple. For example, you may display dates on screens and store dates in databases. Is there a truly single place where you define what you know about “date” in general and each date in particular, for both environments, including temporary variables and parameters? If not, you have a case of complex redundancy. Dates are relatively easy these days, because most systems provide some sort of native support for them. So take something more application-specific, which is unlikely to be predefined, like account number, part number, order number or something like that and ask the same questions.

Complex redundancy is normally cured by shared definitions, but those definitions need to be created and maintained outside of the individual programming environments, and generated into them as needed.

• Redundancy due to incidental differences

There are many cases where there are true differences between potentially redundant items, but they are distinctions that don’t really “make a difference.” This kind of incidental difference frequently comes out of “over-determination,” for example, specifying that a certain field should be at this exact X and Y position, when what you really mean is “under” some other field.

• Redundancy due to over-refinement

Sometimes differences that are potentially “incidental” spring from definite user requirements. Typical sources of such requirements are highly experienced and sophisticated users, experts in a field, who know how it’s best to do things. They know, for example, that at step 4 of a certain process, they have always wanted the screen to be different in this or that way, to remove or add these buttons or fields, or do something else that makes that screen different from the others. It’s something they feel strongly about. It expresses their knowledge, experience and judgment. Rejecting the requirement can be taken by such people as ignoring their experience, denying their knowledge and deprecating their judgment. In other words, it’s bad. It may be a great idea to incorporate the suggestion – or it may be yet another distinction that definitely has a cost from the beginning to the end of the software project, and in the end makes no real difference. From the point of view of Occamality, the bar has to be set very high for such things.

• Redundancy due to external software

Another prime place for complex redundancy to show its ugly face is in interfaces. Many interfaces require you to do the same thing over and over and over again; they give you no choice. If there is no way to centralize this and eliminate the repetition (see next section), you’re stuck in a highly non-Occamal situation, and there may be little you can about it.

• Redundancy imposed by the programming environment

Some programming methods, tools, languages and environments naturally lend themselves to redundancy more than others. A good, simple example is the “polish” convention of naming variables, in which the variable name includes its type, for example instead of just naming a variable IDENTIFIER, you would name it IDENTIFIER-INT to indicate that it’s an integer. The convention came out of a real problem – someone would apply an operation to a variable that was inappropriate for its type; wouldn’t it be a nice idea if the name of the variable itself reminded you of the type so you wouldn’t make such mistakes? Yes, of course, but then if you need to change the variable’s type, either its name misleads you or you have to find every instance of it and change it. While well-intended, it’s non-Occamal.

Once you begin to think about simple things like variable naming, you begin to wonder why in most programming environments, the name of a variable has any significance at all? Why can’t you go to the single place where the variable is defined, and change anything at all about it – including its name?

The exercise is pretty simple. Anyplace you see repetition of any kind, you have to ask yourself, when I make a change, do I have to find all the places and make the change? The answer is probably yes, and it’s not good.

• Redundancy to which abstraction can be applied

If you’re just looking at code, redundancies may not spring out at you – in fact, at the level of the code, there may not be redundancies. But that doesn’t mean your code is Occamal! Sometimes there are common ideas that are expressed with great diversity in the code – or so you realize once you grasp the relevant abstraction. In such cases, it is typical that the people who understood the original problem and the people who wrote the code thought they were dealing with many independent things, and the abstraction that unified them simply never occurred to them.

A good example of this is a collections system I worked on. This was a huge body of code that was used to automate the work of hundreds or thousands of people in call centers whose job was to call people who owed money to an institution, for example a credit card company, and get them to pay. The code had a wide variety of concepts implemented in that were specific to the collections industry. I came into the situation with a broad background in multiple workflow-type applications, and quickly recognized that collections was 95% call-center-oriented workflow, with a little customization and a few specific features for collections.

It wasn’t obvious when looking at the code, but if you started with the basic workflow constructs (workstep, queue, conditional routing, etc.) and took it from there, you ended up with a much more compact and easily extensible application. The original application was simply a “collections” application with some parameters; changing anything required finding the relevant places in the code and making the changes. The new application implemented a core set of workflow abstractions, and hardly ever needed to change. It also had a set of easily editable tables that expressed the current state of the workflow, which enabled almost anything to be changed. Finally, it had a small collection of application fragments that were truly specific to collections, things like the “promise to pay” function.

Occamality: the problem with layers, components and objects

Modern software orthodoxy endorses the notion of collections of code that are separated by high walls. Object-oriented thinking codifies the supposed virtue of data hiding, and keeping all the code that works on a particular block of data (a class) behind a wall. Everyone seems to like the idea of components; people will talk in terms of assembling applications out of component building blocks (when has this ever happened except in seminar rooms?). Finally layers or tiers are supposed to introduce discipline to an application. The idea is you have the user interface (top) tier; then you have the application or business logic layer; and finally you have the database layer. Frequently, these are designed and built by separate groups, each a specialist in its domain – do you really want those superficial, flashy, image-obsessed UI people messing around with your transaction data – do you??? Of course not. But you also don’t want the groups to “hold each other up,” so the idea of having the layers be strictly separated so the UI people can do their thing without being held up by the others.

This is the theory. Lots of people practice it, or at least say they do. If you’ve never experienced anything else, it makes sense. Just the other day I met a software company chief scientist, a PhD in something or other, who explained to me that they were converting some scripts from PERL to java because java was a “real” language, and the scripting languages – all of them! – were not. I didn’t get a real explanation of why this was so; he took it as self-evident to all educated and reasonable people that it was, and by questioning it I really tried his patience, since I was apparently uneducated and unreasonable…

Components, layers, microservices and objects can be good things in limited circumstances, but it’s really, really important to see that if you accept that the prime measure of goodness of a piece of software is occamality for the reasons already discussed, in general, components, objects and layers reduce occamality to the extent they are applied.

The core problem with these things is that they result in multiple definitions of what amounts to the same data. The theory actually encourages this. Take a simple data element such as “account number.” In any program involving accounts, this piece of data will be all over the place. If you’ve got layers, it will be in all of them, because surely it will appear on screens, business logic and the database. If you’ve got objects, it will be in the interfaces and internal implementation of many of them – think, for example, of “account master” and “transaction.” If you think you’re being clever and have built yourself a message passing architecture, it will be in the message definitions – lots of them.

Naturally, you assume and hope that the definition of account number doesn’t change, and in most cases it won’t, in which case the wild redundancy won’t hurt you much. But problems come most often from things you didn’t think about – what happens if there’s a merger, and suddenly there are a whole set of accounts you’d like to bring into your software, and the new account number definitions are totally incompatible. What if they’re bigger and there’s no way to cram them into the existing scheme? You desperately cast around for translation schemes, but in the end you accept the inevitable: you’re hosed.

Now think about this: what if there was a central place where everything you know about account number was defined? Suppose the central place had everything that all the layers needed, for example label for the UI and foreign key relationships for the database. You could go to this single place and take care of most of the account number issues. I assume the program isn’t perfectly Occamal, and there are a few other places you have to go for everything to work. So what? You’re way better off than you otherwise would be.

The painful, anti-orthodox but blindly obvious conclusion is this:

• Each “layer” you put into your software increases its complexity and time to construct, and increases the time and risk to make changes to it.
• The more you use classes and other forms of object-orientation in your code, the more redundancy you introduce, and thus the harder you make your code to construct and change.
  • You may think – before you’ve had lots of experience – that the data hiding of classes increases the “componentization” of your code, and thus makes it easier to change without trouble. Sadly, your thinking on this subject needs revision. Parameters and calls weave your objects together and make them do useful things, and the greater their interaction, the greater the redundancy.

There will always be people who love components, objects and the rest, just like there will always be people who insist on having half a dozen gin-on-the-rocks before supper to whet their appetites. Words don't work. Logic is irrelevant. They should stay in university or wallow around in some giant software bureaucracy where they'll have plenty of company.

Achieving Occamality through definitions: case study

Quite a few years ago I had the problem of creating a product that would help printers create estimates for potential printing jobs. We had one of the early micro-computers at our disposal, and the only programming tool that was available for it was a macro assembler. We had to get the product out in a ridiculously short period of time, and were very limited in the amount of memory we could use.

We got together and realized that we had a fairly simple problem. The ultimate goal was to create printed estimates. Each estimate was calculated based on a combination of data that was unique to the job (the size of the paper, the number of colors, the number of copies, the type of paper, etc.) and data that was common to most jobs, but could be changed at will (the amount to charge for different kinds of paper, press setup and per-copy charges, etc.).We also had to save estimates and create new versions with changed parameters.

Here’s what we did. We broke the whole problem down into seven overlapping problem domains. We created a set of macros for each domain. The parameters of each macro contained attributes relevant to the domain. When assembled, each macro would deposit coded data in memory – no instructions. For example, we had a set of macros for input, another for printed estimates, and another for the core variables relevant to estimating. Macros could refer to other macros, so we could eliminate redundancy and keep the memory requirements as small as possible.

Each parameter in a macro had to correspond to or do something, so we definitely wrote code, but the code we wrote was highly abstract and was roughly proportional to the extent of the macro parameter definitions. When we added a new macro or a parameter to an existing macro, we would add or modify a couple line of assembler code.

The actual functionality of the program was created by a small amount of code that rarely needed to be touched and was pretty easy to write and debug, and a relatively larger amount of macro calls that deposited meta-data in memory. The instructions walked through the meta-data and created the behavior the user saw. We spent time at the beginning understanding the nature of the problem, defining macros and writing code. As the project went on, we spent less time with code and definitions and more time with writing and modifying macro calls. As time went on, the users got more and more stuff, and we were able to change what they didn’t like quickly and without introducing further errors or side effects.

We lost a good deal of time because we picked a computer that was too early in its lifecycle. It had hardware errors, and the macro assembler that was so important to us was buggy. The operating system was primitive, and we even had to write our own file system. In assembler. With a flakey machine and a buggy macro pre-processor.

The main competitor at the time had a system they sold for roughly $25,000. Ours was faster, easier to use, had far more functions, and could be sold at great margins for $10,000. The project was taken on by me and three totally awesome geeks. We delivered the product on the date that was fixed at the start of the project, a date that based on nothing except when it would be nice to have the product. It proved to be nearly bug-free when delivered to customers; one customer found one bug after a couple of months, and I was able to fix it in an hour. Time from start to finish: ten weeks.

Achieving Occamality: What not How

I don’t think I can improve on Chris Date’s formulation of the issue. His basic point is that most computer programs solve problems by taking an imperative approach, telling the computer how to accomplish a given task. He argues strongly in favor of a declarative approach, telling the computer what needs to be accomplished, and having a core set of application-independent functions that accomplish the goal in an optimal way.

Date’s favorite domain is databases. His approach works for databases by writing a program that knows about schemas (tables and columns), data (rows) and SQL statements (Insert, Select, etc.). If you can fit your problem of representing (the schema) and manipulating (the SQL) your data in the way databases expect, and an amazingly wide variety of problems can be represented in this way, then you get to define, load manipulate and access your data without writing any code at all. Since the one body of DBMS code ends up solving a huge number of problems, this approach is highly Occamal.

There are other well-know examples. Spreadsheets are a good one. If you’ve got a spreadsheet-type problem, you can get your needs met quickly and well. You can even write formulas and small program to perform custom calculations.

While databases and spreadsheets are widely understood “horizontal” applications, the approach is applicable to nearly any domain. In fact we often take an approach of this kind when writing applications anyway! If you look at a body of code, there will probably be some code that actually performs an application-specific function that users, for example, would think is meaningful to them. Then there’s all the “other” code that you had to write to enable you to deliver the application value. What’s unusual is for programmers to go “all the way,” and make a clean separation between domain-related abstract functions and application-specific declarations.

The notion of "what" not "how" is a far-reaching one. For example, most of criminal law is "what" you're not supposed to do -- don't kill anyone! The law doesn't spell out "how" you're supposed to avoid killing -- just don't do it! By contrast, most regulations are written as "how" type rules. For example, instead of just saying "the medical device as to work correctly," typical "how" type regulations spell out in gruesome detail how you must accomplish this; and better ways are not allowed!. See this for more detail.

Achieving Occamality through definitions

This is a deep subject, and challenges the way many programmers think. It is rooted in the most fundamental assumptions about the way computers work and the way we program them. In my experience, the ideas first strike people as being simple, obvious, and uninteresting. They seem irrelevant to anything important, and even seem like far-out crank talk.

One way to approach this is to imagine that you stop a computer and examine its memory, byte by byte. At one level, it’s all data. By definition, what’s stored in a computer’s memory is data. But in practical terms, every single byte falls into one of two categories: (1) “plain” data, and (2) instructions. Instructions are placed or loaded into a computer’s memory like any other data, but unlike plain data, the computer can be “pointed” to a starting address and told to start executing the data that is there, and good things will result if the data conform to the rules for instructions.

Achieving Occamality through definitions essentially amounts to dividing the computer memory into three categories: in addition to instructions and “plain” data, you add “meta-data.” If you wrote a program with just data and instructions, you would have a certain amount of space devoted to each. If you write a program using meta-data, the data stays pretty much the same, but the space devoted to instructions typically shrinks by a great deal, and you have a good deal of space devoted to meta-data.

It is important to note that the meta-data should not be in the form of tokens, and while the instructions in some sense “interpret” the meta-data, the meta-data should not primarily be a directive, imperative program.

In practical reality, what you do is create a descriptive, declarative language that is as close as possible to the problem domain as possible. There should be a minimum of translation between the “natural” way of thinking about the problem and the way the problem is expressed in the language. A good example is a navigation system as described here – the meta-data describes the map in natural terms. If you understand language concepts at all, there is probably a one-to-one relation between elements on a visual map and elements in the map description language.

Of course, even a highly declarative approach has a directive aspect to it. My intention here is to emphasize what is usually ignored, not to present an either/or. For example, while a program that gives directions mostly consists of the map meta-data, the direction-generator itself can usually be built in a partly imperative and partly declarative fashion, and needs some plain old parameters, things like whether to avoid toll roads or interstate highways.

This is not a left-field approach to programming. In fact, every method of programming computers, with the arguable exception of “raw” assembler language, starts with a “model” of the kind of program you are going to write, and a choice of how to think about that program.

One of the earliest approaches to rising above raw assembler was the language FORTRAN, short for “formula translator,” derived from the fact that its creators were scientists and engineers who wanted to put their mathematical formulas into the machine for solution.

The business programmers who wanted to use the machine for common business record-keeping functions didn’t find FORTRAN particularly helpful. So COBOL, an acronym for Common Business-Oriented Language got invented.

While it’s reasonable to think about FORTRAN, COBOL and the various languages that succeeded them in terms of language, and this has often been done, it is also reasonable to think: When I sit down with language X, what kind of problem does it help (or hurt) me to think about? How much “translation” is required from the natural terms of thinking about the problem and the language? Do the very terms of the language talk about things I think about? If not, you probably have an opportunity to create a definition domain in which to express your problem, and write a much shorter program than you would otherwise need to write to get the job done.

I’m happy to say that the approach I advocate here is very close to what many people call “model-based” programming. It is ironic that this is so hard to describe, instead of just being mainstream common sense. When you think about a body of code that solves a problem, it is normally possible to code each abstract function that the code performs many times exactly once, and then reduce things it does uniquely to a set of meta-data. Most programmers are pretty comfortable taking this approach when defining the schema for a database-style problem. The model-based approach is really little more than taking the concept of a database schema and greatly generalizing it.

Case Study: Replacing metadata with fashionable software

A VC firm I worked for made a (sadly, in retrospect) passive, non-controlling investment in an emerging PLM company that was all over model-based, declarative  development. They didn’t use silly, made-up names like “Occamal” to describe what they did, but they were fully self-conscious of the power of meta-data, and used it fully. The CTO chose Microsoft as the target technology stack, and ran into the fact that most programmers who specialize in this stack found the meta-data approach to be a foreign one. He overcame the problem by getting his programming done by a group of mathematically-oriented programmers in Kiev, encountering a phenomenon I have encountered many times, namely, that people who think mathematically find it natural to think in the abstractions demanded by Occamality.

This company, AA, had both the problems and the achievements many companies have at their early stage. It had accomplished amazing things in a remarkably short period of time with a small team. It had created a distinctive product that could be customized with an order of magnitude less effort than competing products. It had some really happy customers.

AA also had some unhappy customers. It had made some promises it could not keep, and its code had some really embarrassing failures in the field. It had gotten speed and innovation pretty well down, but quality and reliability remained works-in-progress.

AA accepted a major investment from a couple of brand-name venture capital firms. Each of the firms had name-brand partners on the investment. They surveyed the customers, and discovered the quality problems. So, naturally, you would think they focused on fixing the quality problems, making sure to keep the overall approach that was the key to the company’s value. Makes sense, yes? What else would experienced software-industry investors do? Well, they could do what they actually did – which was hire a big-name CEO, who brought in his own VP Engineering, who glanced at the product, decided it was no good because it wasn’t written in a proper language, viz. java, and put the existing product “on ice” while his huge new team re-wrote everything in industrial-strength java, which of course, as everyone knows, is a language so magical that all you need to do is write in it and what results is fast, bug-free, infinitely scalable programs.

I came in at this point and raised a strong protest. I defended the model-driven vision and most of its execution. I pleaded for spending the new money to fix what was broken – the quality and release process – and for continuing to nurture the goose that was already laying gold nuggets and clearly gearing up to laying golden eggs. It was like I was from Mars, and not the advanced-civilization Mars of science fiction, but a hick Mars in which the most visionary thinkers could hope there was a Stone Age in their future. They tolerated my presence with curled lip and up-turned nose, and went back to whatever they thought they were doing before I intruded on them.

A year and a half later, all the money was spent, the re-write of the application was “nearly” ready to ship (a status it maintained for an amazingly long period of time), some of the programmers were found to have played with visual basic at home, which accounted for the fact that the magic that normally makes java bug-free had been compromised by exposure to impurities. Worst of all, customers were telling the CEO that they wouldn’t upgrade to the new version even if it were perfect, because it lacked the easy adaptability of the original, model-based application. The CEO did what any sensible, experienced executive would have done under the circumstances – blamed the founder for his flawed product and market vision, flaws so profound that even a total re-write was unable to cure them, and sold himself into another job, where they were glad to land such a seasoned executive who had such practical, down-to-earth judgment.

The investors, left to pick up the pieces, had no choice but to return control to the founder/CTO, who, amazingly, had stuck around while the new team was spewing all the money into the garbage dump while dumping on his vision and execution. He hired a VP Engineering, a Russian with many years of hands-on US-based software experience, who fully appreciated the vision, and was able to get the newly re-hired, largely Slavic software team on board with making quality an equal partner with speed and innovation. The quality problems got fixed, customers were re-enchanted with the product, and AA enjoyed a second chance at success. (A few inessential facts have been changed to protect the guilty.)

Occamality is the best possible software architecture, but it ranges from unknown to miles beyond left field. Successful, experienced software professionals are highly likely to dismiss it out of hand in favor of whatever the cool thing is at the moment. To be successful with Occamality, not only do the engineers need to understand it but the executive management team has to understand that the approach is the keystone of the company's technical advantage over the competition.

Understanding Occam-optimality practically

Here are some basic thoughts to help understand Occam’s razor for software as it is applied in practice:

• Build only the software you need to build in order to be successful.
• With rare exceptions, shorter programs are better than longer programs. The goal of programming should be to produce the minimum lines of code in order to accomplish the job.
• Always keep the “parts count” of your program in mind, and make it your goal to keep the number of “unique parts” to a minimum, with each “thought” expressed exactly once, with everything that “uses” that thought generated from the primary expression of the thought. See this for a practical example.
  
• The best days are ones that reduce the total number of lines of code while increasing the functionality of the program. A day spent doing nothing but writing new code may very well be a day that that reduces the Occamality of your program, so that every line of code adds to the later burden of testing, documenting, learning, using, maintaining and ultimately changing the program, thus making it worse than it needed to be in nearly every respect.
• In situations where there are multiple paths to a software goal, pick the shortest one that involves the least amount of work. Don’t plan for future change; if you write the shortest possible program, future change will be optimally prepared.
• Ignore common practices if they produce redundancy, and thus violate the razor. For example, multi-tier design frequently results in huge redundancy of data definitions. A multi-tier execution environment is fine, so long as the tiers are generated from and share common definitions, so that the program at the source code level is Occam-optimal.
• Frequently, software is complex because the requirements are needlessly extensive and complex. Reducing and/or standardizing the requirements is a key part of Occamal software.
• Apply Occamal principles to the whole process involving the software, from requirements through to training and support. Cases that seem marginal when looking at the software in isolation often become clear when taken in full context.
• Copy, use, ignore or emulate everything that is not uniquely required to be successful
• Avoid over-determination, i.e., putting in things that sound good or say more than you mean to say. For example, there is frequently no mechanism to deliver the results of error checks in a database to the people or programs that need them, and they are therefore useless. Again, some people use screen design programs that specify design elements in far more detail than they really mean, therefore saying too much.

A great deal could be written about this. For an explanation of the "post-hoc design" approach to Occamality, see this. It's a way of speeding the way to a goal.

Understanding Occam-optimality technically

I am not aware of a set of terms I can use to make a technically clear definition of “Occam optimality” for software, so I’ll just re-use some existing ones. I suspect there’s an exact, more mathematical way to do this. I’m hoping someone will pull it off. My goal here is to express the most basic relevant concepts in rough terms.

By “program” I mean the entire source text that is required to build the executable program. If the program uses a DBMS, the schema definition is part of the program. If there are “include” files or resource files, they are part of the program. If the program is a multi-tier one with an ASP GUI, a component-based application and a database layer with stored procedures, it’s all part of the program. If there is information that is part of the program that is not immediately available as text, then for these purposes we convert it to text. While we normally think of these things as being in totally separate categories, for these purposes, we will construct a single source text that has everything required to build the program.

Here’s a tricky, vague but essential definition. Without this definition, an occamal program would be just a version of the original compressed by mechanical means, kind of like a compiler optimizer. The “intention” of a program is its true goals, without technical specification or constraint. The “expression” of a program is a “program” as defined above, i.e., completely determined by its text. A single program intention can have a very large number of possible expressions. The purpose of the normal program design process is, at the very early stages, to clarify the intention, but most of the process is designed to narrow down among all possible expressions to get close to a narrow range of possible expressions. The final determination of program expression is, of course, made by the coders.

The effort of creating Occamality comes during the process of turning “intention” into “expression.” Of all the evaluation criteria that have been used to drive and evaluate this process, and there have been many, the occamal criterion says: express only what has been intended (don’t over-determine), and express what has been intended in the least possible number of “semantic units” possible, with no redundancy among them.

I use the awkward phrase “semantic unit” to indicate a programmer’s thought, and to avoid getting caught up in one language vs. another, the number of characters involved in the keywords and syntax, comments, the length of labels and variable names, etc. A semantic unit is any combination of code, data and meta-data that is required to express a thought. Semantic units can be primary, composite, and have any number of connections and relationships among them. Good definition and use of connections and relationships are how redundancy is eliminated, along with careful attention to avoid “distinctions without a difference” of substance.

Semantic units can have “primary” information in them and/or “reference” information. “Primary” information in a semantic unit is an actual definition or program statement, for example a data type or a conditional expression. “Reference” information is a reference to one or more other semantic units. For example, you would define that a date is a month, day, and year, with these data types, lengths, labels, compute rules, etc. This would all be “primary” information about date. When you defined “birth date,” its most important characteristic would be the “reference” information that it is a type of date. A semantic unit could easily contain a mixture of primary and reference information. For example, “birth date” would have as primary information its unique label and the fact that it is a separate piece of information; everything else about “birth date” would be a reference to “date.” The reference should be a single reference to the “date” semantic unit.

References, as defined in this broad sense, are the key to creating occamal programs. Every reference is something that could have been given a separate expression, which might have been literally identical, but more likely would have had the same intention but differ in trivial detail as expressed. In the example of the Y2K problem, each reference to a central date definition replaces a separate definition of date. Without references, all the separate instances of date would have to be found and changed; with references to a single central definition, all that needs to change is the central definition, and the references cause the change to be rippled to all uses of date.

A “program” is Occam-optimal when it has the minimum possible number of “semantic units” in its expression, and when there is no redundancy among the semantic units, while still fairly expressing the original intention of the program.

Obviously, this only scratches the surface of the technical side of Occam-optimality. I hope it’s enough to convey the general idea.

Classifications of Software Programs

Anyone who has programmed for a while knows that all code is not alike. There are important broad categories of code. I wish there were a generally agreed to categorization, but there doesn’t seem to be. Many years ago, there were these categories (from lesser to greater in terms of perceived expertise and prestige):

• Operator (your hands actually touched a physical computer!) – low status!
• Applications programmer (many variations here, from business analyst to debugging and maintenance). And even they are not all the same! While I was working in a COBOL shop, I discovered that there is an intense, sharp difference among COBOL programmers:
  • Batch COBOL. There is no direct human interaction here.
  • Interactive COBOL, the kind that paints the infamous “green screens.”
  • Guess which is more prestigious?
• Systems programmer (these got the problems when the application programmers were stumped)
• Computer vendor employee (these people wrote the operating systems, compilers, and other tools that systems programmers “maintained = got blamed for” at user sites)

When I worked at a computer vendor, there were clear distinctions among those who were concerned with:

• The operating system kernel (and then there are the drivers, the people who deal with them have to suffer from hardware dependencies)
• The operating system utilities
• Compilers
• Database systems (“simple” file systems fell under quasi-core OS; add a serious index or mention “b-tree” and you moved here)
• Networks

These distinctions were necessary and real. The coding disciplines, received knowledge, experience, and so on were different enough for these things that people could really be good at one and not at another. Unlike in the dreaded and despised world of end users, there was no clear hierarchy. While the kernel guys felt themselves at the center of the world, as in some sense they were, there was no denying the fact that compilers involved all sorts of special knowledge, and this got really annoying once OS’s started getting written in languages requiring compilers.

I suspect there are other reality-based classifications of code. But for these purposes, I would like to break all programming into some categories that cut across these and others I have heard of:

• True algorithms
• Abstract realizations of generic functions
• Application and device specific definitions and logic

Algorithms

Algorithms are the blocks of code studied so well in Knuth’s multi-volume series, The Art of Computer Programming. They are things like sorting, searching, managing lists, etc. There are lots of other sources, but he focuses on the key factors for code that implements algorithms. In spite of the impression that may be given by his many volumes, there are a relatively small number of algorithms, and it is important that a relatively small number of people like Knuth study them intensely, understand everything about them, and make them available (I’ll gladly pay!) in standard, canonic, guaranteed implementations. Most of the metrics in this paper are not relevant to algorithms.

Why is this? Because, for most algorithms, we continue to value the amount of space they use and the amount of time they take to execute. In increasing numbers of business applications, sub-optimal algorithms could be used without ill effect, at least in some places. But if an optimal algorithm is available, why not use it? Why make life complicated (not to mention error-prone) by introducing two classes of algorithms, the optimal ones and the good-enough ones – what’s to gain?

The life and analysis of algorithms in practice is by no means complete. There are important advances to be made. But most algorithms are advanced by someone solving a previously unsolved problem (like Dijkstra and the shortest path) or by taking advantage of a newly altered constraint (disk memory is now much cheaper than it used to be, therefore we can reasonably do X, which was a terrible idea in the days of expensive disk). Because of their different goals (minimization of space and time within constraints), algorithms play by different rules than other programs.

Abstract generic functions

Abstract generic functions implement things useful to many applications, but not specific to any one of them. Application “frameworks” are a mother lode of such functions. The Rails framework in Ruby is a reasonable modern example.

Any programmer who has written a variety of programs knows what I’m talking about here, and has his own candidates for inclusion. This has its origins in the FORTRAN FORMAT statement from Paleolithic times. Major categories of software products have emerged that started as application-specific code, moved on to be abstract generic functions, and then finally hardened into products. CRM and Content Management systems are recent examples. However, software products as currently typically designed and delivered aren’t nearly as useful or flexible as unified collections of abstract generic functions. The closest thing we have is application frameworks.

Application-specific definitions and logic

This is where everything else goes. This is the code and meta-data that makes your application your application, different from others.

Even this can and should be broken into layers of generality! For example, there should be one place for the common aspects of your UI design, and one place that defines what is unique to each particular interface.

The important thing is that, to the largest extent possible, you are mostly selecting and marshaling the general mechanisms available, and making them particular to the problem at hand.

Of course, your problem may have some highly unique elements. These elements may involve writing some serious code. The code may resemble in depth and complexity that of abstract generic functions. But you’d better just write it to solve the problem at hand, and leave generalization to another day.

Occam-optimality applies to all stages of the software life cycle

It’s natural that Occam-optimality is mostly focused on software. But it applies to everything from requirements to architecture and design, QA and testing, documentation, support and the endless round of changes software tends to have.

Requirements occamality

Are Occamal principles just being applied by the programmers who code up the application? If they are, that’s good, but there are steps in the software cycle before the programmers start their work, and if those steps are conducted without Occamality in mind, even if the programmers program a perfectly Occamal program, there will end up being many more lines of code than there could have been. This is easiest to see in the requirements process. Suppose one group writes up requirements for an on-line credit card application and another group writes up requirements for an on-line application for a term loan. The requirements are likely to be similar because the same kind of data needs to be collected in both cases. But what if one group takes a very matter-of-fact approach, while the other emphasizes fancy graphics, visuals and user interaction? Once those requirements come to the programming group, unless that group is amazingly perceptive and aggressive, the two sets of requirements will be treated as wholly unrelated projects. In other words, business as usual.

Imagine, on the other hand, that the people thinking about requirements are familiar with Occamality and want to apply it. Rather than starting from scratch, they may look for places where the data they need is already being collected – maybe this new project is an opportunity to unify some existing code, and bring it into the Occamal world. In any case, the two groups will decide that they’re really one group with a couple of minor differences between the two applications, and they’ll also realize that the styles of user interaction will change over time, so they’ll take some effort to separate that out from the actual data gathering. The net effect is likely to be a set of requirements that are “pre-occamized,” that should not take much transforming for the software group to implement in an Occamal way.

If software requirements aren’t Occam-optimal, the software may be occamal in some abstract way, given the requirements, but the requirements themselves may demand a solution that feels like there are all sorts of special widgets and do-hickey’s that could just as well be dispensed with. In other words, Occamality is best achieved if it is a conscious goal from the very start of the requirements process to the end.

Here are some reasons why it’s important to think Occamally in the requirements:

• Redundant requirements or ones with trivial variations normally result in needless redundancy in all the following steps.
  
• A set of requirements that are similar in principle but different in detail are frequently over-defined and over-refined, creating work that delivers no value.
• People who create requirements frequently know a great deal about their subject and have thought about it long and hard – and the more they know and think, the more extensive their requirements are likely to be, which is often a problem.

Software requirements people frequently resemble the washing machine designers I have described who optimize what they do over a very narrow domain. The result is screens and functions that are highly customized to the job for which they’re intended. Every bit of customization adds that much more to what has to be programmed, tested, documented, taught, learned, used and maintained. This is typically a very large cost for an assumed benefit that often isn’t even there, because people find uniform programs easier to learn and use.

The consequences of applying Occamality as broadly as possible in the requirements will roll down the line of testing, documentation, roll-out, training and support, saving time and money each step of the way.

Software scope occamality

If you look at 100 lines of code, you may be able to find some redundancy, but probably not too much. If you look at 1,000 lines, 10,000 lines or 100,000 lines, there are likely to be whole gobs of redundancy.

When you cross the line from one officially-designated “program” to another, you greatly increase the odds of finding redundancy. Within a single program or project, there is only so much you’re like to find. But particularly within a single business, the same things come up over and over, and get solved over and over again, each time in slightly different ways.

Things begin to get interesting when you cross traditional software boundaries, like from user interface to application programming to network to database. In each case, there are frequently specialists who do things in their own way. If, as is likely, they have caught the “object orientation” bug, they each attempt to create their own little worlds of nicely designed classes/objects. But when you look at the project as a whole, and you think of something like “account number,” think of the number of times and places and ways that concept has been defined. Even within one of those domains, like the application, an account number data element will be defined as part of each class, and will appear every time account number crosses to or from the database, and again every time account number crosses to or from the user interface. You’ll find that “account number” is all over the place! If you have any question about this, just try to change the data type, label and error checking for it, and see how many places you have to touch to make it work.

Software architecture Occamality

Occamal thinking is crucial in the architecture and design of the software:

• Achieving Occamality frequently involves introducing a strong but appropriate amount of abstraction into the program design. Simply translating requirements with minimal abstraction into objects, schema designs, etc. typically leads to huge amounts of redundancy.
• If you just focus on the text of the code, you may find little redundancy. But you may also be able to achieve substantial code reductions by taking a whole different approach.

Software QA Occamality

The usual approach to QA is highly redundant and un-optimal. When QA automation is normally performed, it is highly redundant, and leads to a system that doesn’t have better quality but is even harder to change. Typical script-based automation ends up having the effect of copying variables already present in the program in a different environment (the QA one), and creating logic that reflects and is derived from the base program. This approach to quality automation clearly reduces the quality of the overall program, and certainly reduces its occamality of the overall software effort The ideal approach is highly automated, fully supports occamal software, and indeed is occamal itself. See this for more details.

Later stage Occamality

Post-coding software steps benefit indirectly from smaller code and less redundancy, but in addition, Occamal thinking can benefit them directly.

• Occamal thinking applies to documentation and help text. The arguments about redundancy making documentation more difficult and error-prone to change apply fully.
• Let’s not forget the users! Chances are excellent that additional code is the direct result of input from some users, but most users will suffer as a result. Every concept, field and command you eliminate is one more thing users don’t need to worry their little heads about.

It is important to focus on Occamality in the code. Extending it to encompass the entire life cycle of software is what delivers big benefits. Post-hoc design as described here is a practical approach to accomplish Occamality in waves, without going too crazy trying to do everything perfectly.

Occamality and other design principles

People use and defend the use of all sorts of methods and tools for building software. The number and range of methods and tools is overwhelming. Is Occam optimality yet another entrant to be added to the already-too-long list? No. I hope that the notion of Occam optimality will be understood to be independent of all those methods and principles, and will provide a way to both improve each of them and, in some cases, select among them.

If the notion of Occam-optimality is correct, it stands above and cuts across all methods of building software. It is a principle that enables you to take two efforts to build a program in the same environment and tools, and say objectively which is better.

It is certainly true that some tools and methods make it easier to achieve Occam-optimality than others. But it is important to understand that Occam-optimality is not primarily about judging tools and methods – it is a notion of optimality that can be applied given any selection of tools and methods. Programs can be written in assembler language (assuming there’s a good macro pre-processor) that are Occam-optimal, and in fact it is easy to understand the concept in that environment, since everything is laid bare.

If most current ways of thinking about programs are like algebra, then the analysis and understanding of Occam-optimality is like calculus. Just as calculus doesn’t replace or criticize algebra but is built on it and extends it, so is Occamal analysis built on existing methods and tools of program construction. Just as calculus provides tools that didn’t exist before, like methods for calculating the area under a curve or the zero or minima of a curve, so does occamal analysis enable us to understand what “optimal” really means for most programs, and gives us guidelines and a way of thinking to enable us to construct programs that approach occamality.

Unlike the myriad algebra-like methods of program design and construction, an occamal program that meets the business intentions is also very likely to be optimal in terms of all the other goals that business people have: it will be as efficient as possible to define, build, test, document, deploy, learn and support. Most important, it will be possible to modify and enhance it more quickly, safely and inexpensively than any other kind of program. To the extent that the program fails to meet important expectations on any other grounds, e.g. performance, availability or attractiveness, a program that is occamal lends itself to alteration to meet those expectations better than a program which is sub-occamal.

Once this criterion is understood and accepted, our tools and methods will evolve rapidly to achieve the maximum possible productivity in software design, construction, support and evolution.

Focusing on the occamality of programs is a way of focusing on what we most care about in software. It doesn’t eliminate our other concerns – we still care that programs perform reasonably, that databases avoid corrupting data, etc. But an amazing number of those other concerns will be taken care of by-the-way if our programs are occamal, and if they’re not, we know that occamal programs are almost by definition the easiest to change to address any need or fill any gap.

I worked with a company having horrible problems with the administrative workstation of their storage product. It took a long time to load and many seconds to respond to reasonable-seeming requests. Its user interface was inconsistent and it took a very long time to update to reflect changes and new features in the underlying product of which it was a part. Looking under the covers, it turns out there were more than 200,000 lines of what turned out to be java code, and it was really hard to say what useful function large chunks of it were performing. Java was selected because the people thought it was modern, object-oriented, and would let them build re-useable components and other good things. Then they just started writing code, and here we were.

There were lots of arguments, pro and con, about whether java was any good, all of which were beside the point – wrong discussion. After a great deal of struggle, the whole thing was gutted and replaced with less than 20,000 lines of code and definitions. The people doing the work did not concentrate on performance, changeability, or other architectural concerns – their main goal was to make it small. When they were done, and they did it quickly, they had a complete replacement, a consistent user interface, and it magically became really fast and easy to change!

Occamality in Software History

Occamal concepts have naturally emerged in software efforts from nearly the beginning of software, and limited formulations of Occamal concepts have been promoted and valued in software. The purpose of this section is to identify those early expressions for what they are; they show the widespread, ever-springing nature of the desire to come to some principle that will yield optimal programs.

Software tools

One of the earliest pure-software efforts, the work to build the early language FORTRAN, was clearly a step in an Occamal direction. By creating a machine-independent language in the first place, programmers could concentrate on expressing their programs in a single language, FORTRAN, instead of the multiple machine languages that came to exist. Since FORTRAN is closer to the problem domain of most scientific programming, it usually takes fewer statements to express a program in FORTRAN than in assembly language; therefore, writing in FORTRAN eliminates redundancy and is thus better in Occamal terms. The relationship between the statements of the language and the machine language are all centralized in one place, the compiler. The compiler’s run-time library was just as important in this respect – it provided a set of commonly used functions that only needed to be written once, and then used whenever needed. In the early days of FORTRAN, for example, machines typically did not provide native support for floating point operations. The programmer could write his floating-point formulas and calculations without thinking about the machine, knowing that if the machine his program eventually ran on had no support for floating point, the compiler would generate the right calls and the run-time library would do the work.

All computer languages take programs in an Occamal direction. Just as FORTRAN abstracted and centralized the floating point operations commonly needed in scientific computing, so did COBOL abstract and centralize processing records of data and the BCD arithmetic needed for financial calculations.

Similarly, the concept of subroutine and shared subroutine libraries arose very early. One of the important functions performed by early software societies (IBM’s SHARE, the ACM and IEEE) was to collect, refine and standardize libraries of functions widely used by groups of people. Early software companies owed their existence to the time and money it took to build major functions, and built a business on selling many copies of software that was expensive to build, and less expensive for the user to buy than build for himself.

The use of macros is clearly driven by Occamal goals. The idea and motivation of a macro is simple: when there are repeating examples of a string in the text of your program that you suspect you may want to change in the future, you define the text in a macro and use the macro instead of the text itself. When the need for a change arises, you go to the single macro definition, change it, and you’re good to go, regardless of how extensively the macro is used.

The concepts of inheritance, templates and component re-use are clearly Occamal in nature. The idea is you write the common parts of a function in a master class or template, and then sub-class or apply the template to handle special cases or variations. You have the master class or template which gives you the single place to go to make changes, while benefiting from only needing to spell out the variation in each particular application. Using these things tends to reduce the number of lines of code and reduce the redundancy, and is therefore Occamal.

Software applications

Occamality can be clearly seen in the evolution of software products.

Over time, software efforts in a particular field naturally tend to the Occamal, model-based ideal, because such bodies of code are the easiest to maintain and enhance, while providing maximum flexibility to their users. Early attempts at building Customer Relationship Management (CRM) systems, for example, tended to be bodies of code written in some supposedly easy-to-change 4GL. Every CRM system needs customization. When one of these early CRM systems was installed, the source code would be opened up, programmers would hack away, and sometimes an application vaguely appropriate to the customer’s needs would emerge stumbling from the dust. The awfulness of this approach quickly became evident as customers found that upgrading to the vendor’s latest release meant re-entering the coding war zone and having to fight a battle on two fronts, i.e., their customizations of the old release and the vendor’s “improvements” to the old release in the form of the “upgrade.”

The vendors who survived this nightmare were the ones who, through natural selection and the survival of the fittest, minimized the damage to their customers through the installation and customization process, not unlike the way parasites evolve to avoid killing their hosts. Invariably, this meant increasing degrees of Occamality (not “perfect” Occamality, mind you, just “increasing degrees” of Occamality), nearly always expressed as a more model and template-based approached to application definition.

Engineering product management systems, usually called PLM systems, are an example of a set of software that has long-since evolved to be more Occamal than not, simply because engineering products and processes are so highly individualized that one of the main functions of modern PLM products is to make it easy to express product and process definitions without requiring modification of the source code of the product.

Here is further explanation of the evolution towards increasing application in software applications.

These examples make it clear that Occamality is a thread that weaves through the software industry and underlies many of its developments, although the people who created and promoted the developments generally did not think of them in these terms.

 

Occamality in Databases

The accepted practice of database schema design is a good application of Occamal principles. In fact, in a broad sense, Occam optimality takes the concepts accepted in schema design and applies them to programs and the software lifecycle as a whole.

Someone at a bank would write a system for automating checking accounts. Naturally, the account information would include the name and address of the person who owns the account. Some else would write a system for automating savings accounts. Naturally, the account information would be essentially the same; it may be the same because someone knew all about the checking system and copied it, or it might be the same because of “convergent evolution.” The account holder moves, and somehow tells the bank. The person at the bank has to look everywhere the person’s address might be stored and change it there. They might miss a place, or they may enter something incorrectly. This places a burden on the bank, makes extra work, is a source of errors, and may lead to customer dissatisfaction. It’s far better to have a single place where a person is identified at the bank, and where the things we know about that person (name, address, phone number, etc.) are stored. In the database world, this is the concept of “normalization,” which essentially means store each piece of unique data exactly once.

The database world also has the concept of “reference” in a couple of ways. If the person actually sets up a checking account, the checking account system needs the account information for the customer. This is done by identifying each account by a unique identifier, known as a “primary key.” The primary key is associated with the master copy of the account-holder’s information. Then, in each place where the customer uses a bank service, for example a checking account, a “foreign key” to the account information is placed. This is actually a copy of the primary key. So if you are customer 123, your primary key is 123, and the number 123 is also made the foreign key for your savings account, your checking account, etc. Calling it a “foreign key” says that it’s a reference to the one place where the information is stored, and not the master copy itself.

The database also uses references to eliminate redundancy in data types. When you need multiple fields that have to be distinct but actually represent the same kind of data, you define the type information once in something called a “domain,” and then each use of the domain is actually a reference to the master definition. If you change the domain, all the uses of the domain automatically change.

Databases represent Occamal principles in another important respect: statements in the data manipulation language (today, that means statements in SQL, for example SELECT data FROM tables WHERE conditions) are limited to what needs to be done, not how it is to be done. For example, if there is a JOIN between two tables, which should be used first? There is an excellent answer to that question, for example if one table is much smaller than the other, use the smaller table, etc. By abstracting how the join is implemented from the fact that you want a join, you get to define the how of joins in exactly one place, while requesting joins in many places. In earlier data manipulation languages, you could get the job done, but you would explicitly go first to the table you thought best to start from, and then go to the next. The trouble is, this puts the knowledge of how to navigate tables to get information in many places! This is bad because everyone has to learn it and apply it correctly, and if you want to change the method because you’ve figured out a better way to do it, you have to go through all the DML in the program and make individual decisions about how and what to change. Separating what we want from how to get it was definitely an advance, because now SQL could benefit from ever-improving execution routines without having to be changed! The point here is that exactly those benefits are the ones we would expect, because the change also made all programs that use SQL closer to being Occam-optimal!

It is true that many database implementations are Occam-suboptimal when considered in isolation. Because DBMS schemas are typically completely isolated from the programs that use them, programs that use databases (taking the database schema and stored procedures to be part of the program) are typically far from Occam-optimal. This is recognized in the RAILS framework of the Ruby language, which unifies the data definitions of the database and the program; this is the reason why programming is so much more efficient when using RAILS. Apart from RAILS, it is wonderful to have modern databases as an example of an island of software in which Occamal principles are accepted and valued, and the benefits widely enjoyed. By understanding the examples of good database practice in terms of Occam optimality, we can get an idea of how the principle can be extended to the rest of software.

 

Why should you build Occamal Programs?

There is a good deal of background and analysis to understand just how the concept of Occamality applies to the practical details of building programs. But before we get to that, perhaps it would be good to review the benefits we can expect. The core benefit of paring down a program to its bare minimum information content is pretty simple:

To the extent that any semantic concept is repeated anywhere, in any form, in the specification of the program, it is a redundancy that, if discarded, would reduce the cost of implementing the concept, the cost of one or more of the downstream components of the program, and the cost of subsequent modifications.

In other words, an Occamal program is one that is quicker than any other program that does the same thing to build, and is overwhelming the easiest to change, even though you did not think you were building “flexibility” into the program!

Let’s review briefly the lifecycle of software. At the beginning of the process for a whole new program or a change to an existing one, there is the need to create or change embodied in a set of requirements. From there, with variations depending on the programming shops’ methodology, we have roughly specifications (business, functional, high level, low level, etc.), design (various levels of focus and detail), code (perhaps with prototypes, code walk-throughs, etc.), test (various levels, sometimes performed by different groups of people using different tools), document (internal and external), train (the people who operate, administer, install, support and use), learn and use the software. This sequence may be linear or it may have iterative cycles. Once the program is in operation, there is an extended support cycle, with support, maintenance and bug fixing. There is typically a demand for new or altered features; these go through a version of a similar chain from requirements through building and use. It is generally recognized that (1) most efforts to build new programs crash and burn prior to roll-out, and (2) most of the money that is spent on a program is spend during its extended “tail,” namely the support and modify cycle. The difficulty of building brand-new programs increases the extent to which existing programs are modified, sometimes through multiple technology cycles.

Today, because occamality is not a familiar concept, it may take longer in practice to build programs that are occamal. Even when it is understood, redundancy frequently does not cost much to build into a program. In fact, it may be quicker to build a program with extensive redundancy (think copy and modify) than with less. But every superficial gain from redundancy is paid for, over and over again. Once we’re past the learning curve, building occamal programs should cost about the same as non-occamal ones (ideally, they would cost less to build), and the majority of the cost of software ownership should experience dramatic benefits. The benefits would mostly be due to the lower cost and lower risk of testing, documenting, learning, using, maintaining, and changing programs.

What exactly is it that makes programs hard to change? Once you understand basically what you have to do, it comes down to finding all the places in the program that will be affected by your change. The side-effects are always the toughest.

What if the change you wanted to make could be accomplished by changing exactly one place in the program? What if, to take a trivial example, you wanted to change the classic

Print (“Hello, world”);

By printing “bubba” instead of “world?” This would be easy, because you could just go to the single program line and make the change.

Now let’s take a well-known horrible example – the Y2K problem. This was the problem of modifying programs so they would continue to work when the year changed from 1999 to 2000. This was a problem because many programs did not use 4 digits to represent the year. For various reasons, usually involving an attempt to save space or time, year was represented as a two digit number in many programs, for example 99, with the leading two digits, 19, understood. If those programs had a single place where the number of digits in a year was represented, solving the Y2K problem would have been trivial – you would just go to each program, find the place where year was defined, change it, re-compile, check for problems, and you’re done. A quick change, low risk, no big deal. So Y2K was a problem not because of anything difficult or mysterious about dates – in fact, it could have been about any similar program change. Y2K was a problem simply and solely because the knowledge of the number of digits in the year of dates was expressed in many ways in many places, and it took a long time to find and fix them all with a substantial risk that some places were missed.

I completely admit, and I remember because I participated in the madness, that there were bizarre variations that made Y2K particularly gruesome. One example is “overloading,” which means that sometimes a programmer would use the value of “99” in the year field to mean “no date was given.” Y2K was a problem before the year 2000 because some programmers used “9999” (which could mean September 9, 1999) to mean “no more records in this sequence.” But all such cases just multiplied the basic fact: the number of digits in a year was expressed in many ways in many places, and in some of those places, other values or indicators were also stored.

So we want to have exactly one place in which the number of digits in date is defined. Notice that doesn’t mean we are limited to a single date. The key concept here is distinguishing between definition and use. Let’s add another level to make it clear. We want to define date in one place. Then we define, say, a couple of dates: birth date, marriage date, date of loan start, date of loan end. Each of these would be defined as types of dates. “Date” would have how many digits are in a year, and “birth date” would be a kind of date. We would define with “date” those things that are true of all dates, like number of digits in the year. We would define in “birth date” only those things that were true of “birth date” in particular, for example, the label to use when displaying it.

This is an elementary concept as I have illustrated it so far. What is different is the extent of the application of the concept that I propose. While it is (I hope) common practice for dates to be defined in this way within a program, it is certainly not common practice for all dates used anywhere in a program for any purpose to be defined in this way, as we saw big time in Y2K. For example, we have one way of defining dates in the database, another one for local storage in application programs, and yet another one for display and reports. While it is reasonable for there to be aspects of dates that some parts of programs care about that others don’t, for example the display label, it is not reasonable that all aspects that are common be defined redundantly – they should be defined exactly once.

More importantly, this concept applies to all aspects of the software lifecycle, not just the building phase. It even applies to documentation, training and using programs, because it means that there is less to document, train and learn, and when changes are made, the user’s common-sense notion that “something changed” actually applies – for example, if zip code is changed from 5 numbers to 5 numbers optionally followed by 4 numbers, that change is universal. The change, made in a single place, ripples though screens, applications, temporary variable definitions and databases.

Occamality directly addresses a lament frequently heard from programmers, about not being given enough time to do a job “right,” to design it for future flexibility. One of the many reasons good managers refuse to support a “good, flexible design” is that you always pay the price of the additional effort, and you rarely actually experience the benefit. That is simply because the designer tries to anticipate the nature of the potential future change, and make provisions for it. But of course, what usually happens is the anticipated change doesn’t happen and something else does, and the fact that the program is longer than it could have been because of the additional code that provides for un-needed flexibility makes the unanticipated change harder to make than it had to be.

Now that we understand what makes change hard – the fact that what we want to change is expressed in many ways in many places, all of which have to be found, understood, and correctly changed – we can design programs that are both as quick as possible to build and as quick to change in the future, regardless of the new requirement; we know that if we need to change something, there’s always one place to go to make the change.

The answer to why you should build Occamal programs is simple:

Occamal programs cost less to specify, design and build (once you are past the learning and acceptance curves), cost much less in the later stages of the software lifecycle where most of the costs are incurred, and enable changes and other forms of support and maintenance to be made with the greatest speed and the lowest cost and risk. And this is true regardless of language, operating system, application environment, and anything else about your software environment and tools.

 

How do you know if a given piece of software is good?

Software has a real problem. Let me explain.

While a huge gulf separates the novices from the experts in every field, I like to think that the widespread simple knowledge in most fields is like writing as taught to elementary school students. Fifth graders use the same alphabet that I use; while my vocabulary is more extensive and my use of grammar more elaborate, in both cases what I do extends and builds upon what the kids do. Fifth graders don’t need to unlearn the “bad” letters when they get to high school. Books for children can be well-written, and understood and enjoyed by both children and adults. Books for adults are supersets of children’s books in terms of vocabulary, sentence structure and experience. And while professional writers may not make extensive use of notes written on 3 by 5 cards the way English teachers used to make kids do in high school, the principles of organization are the same.

In software, however, the doctrines and received methods taught and practiced in the grades and, worse, among mainstream professionals, are simply inadequate to support doing a really good job; the best people don’t extend them, they ignore them and in various ways violate their principles. The typical methods for performing the quality assurance and testing functions, for example, are counter-productive. They don’t need correction – they need complete replacement. They focus on the wrong issues, they have the wrong concepts, and the most widely used methods and tools simply don’t get the job done!! The relationship between programs and data. Same. By reference and by value, implicit and explicit. Other books in this series go into detail for a couple of these subjects.

Things are so bad in software that the most experienced and accomplished people don’t even agree on what makes a program “good.” Given a set of programs that don’t crash and meet a set of requirements, how do you rank them in order of goodness? Is goodness proportional to how deep the inheritance trees are? How well documented they are? To what extent the evil “go to” mechanism is used? What is the test coverage, or the number of unit tests? How few machine resources they consume? Whether they’ve been written in this language or that?

I propose that there is, with a few common-sense qualifications, a measure of software goodness we can all agree to. I suggest this isn’t a new idea, but one that many people have sensed and acted on. The principle even makes common sense! The only thing it lacks is articulation and discussion. Here is a non-technical explanation of it. Here is one of my attempts to state the principle in a historic and more technical context. Here is a deeper explanation of the context and background.

This is not an abstract, nice-to-have concern. Unless you know what “goodness” is in a program, how can you measure whether you’re going in the right direction? Think about target shooting: without a target, how can you decide how close you are, and therefore how accurate a shooter you are? How can you compare two different shots?

Having an explicit agreement of what makes a program “good” is indispensable to making good software, and making it effectively and efficiently.

Occam’s Razor: the key to optimal software development

William of Occam (ca. 1285 to 1349) was an English logician and Franciscan friar. He is credited with formulating a principle that has been applied to various aspects of computer systems. In those areas of computing to which it has been applied, it reigns supreme – it supplies optimal solutions to the relevant problems.

There are large areas of computing to which Occam’s razor has not been applied. Worse, it is not even one of the candidates under consideration. As a result, those aspects of computing are fractured, inefficient, unpredictable, and driven by fashion and politics.

Everyone involved knows that the whole process of specifying, designing, building, testing and supporting software is hopelessly inefficient, unpredictable and error-prone. The leading methodology that concentrates on the process of building software, “project management,” is theoretically bankrupt and in any case has an empirical track record of failure. In terms of the content of good software, there are fierce battles among competing approaches, none of which is anything but a collection of unsupported and unfounded assertions, and which in practice don’t contribute to building good software.

Occam’s razor leads to the principles on which good software may be built, and supplies a single simple, widely applicable theme that, when applied, cuts away (as a razor should) all the inefficiency and generally what we don’t like about software. Once the principle is understood and widely applied, It should be the undisputed standard for how software is built, just as it has in the other areas of computing to which it has been applied.

Here is a shorter attempt to explain the Razor and its application to software. Here is an approach to the same problem from a common-sense layman's point of view.

The razor itself

“Occam’s razor” is a famous principle of thinking from the European Middle Ages. Occam’s razor is applied in situations where there is more than one reasonable explanation for a phenomenon, and basically says that you should pick the simplest explanation consistent with the phenomena you observe. From Wikipedia:

Leonardo da Vinci (1452–1519) lived after Occam's time and has a variant of Occam's razor. His variant short-circuits the need for sophistication by equating it to simplicity.

Simplicity is the ultimate sophistication.

Occam's Razor is now usually stated as follows:

Of two equivalent theories or explanations, all other things being equal, the simpler one is to be preferred.

As this is ambiguous, Isaac Newton's version may be better:

We are to admit no more causes of natural things than such as are both true and sufficient to explain their appearances.

In the spirit of Occam's Razor itself, the rule is sometimes stated as:

The simplest explanation is usually the best.

Other applications of Occam’s razor

While it plays little overt role in intellectual debates, in fact Occam’s razor and concepts derived from it are central to scientific thinking. So while you may not have ever heard of it, or may only vaguely remember having heard it in the past, you shouldn’t think that it’s an obscure little Medieval tidbit that has no relevance to the present that this guy is just pulling out for some weird reason. If you go through the Wikipedia reference, you find that:

Occam's Razor has become a basic perspective for those who follow the scientific method. … without the principle of Occam's Razor science does not exist. The primary activity of science, formulating theories and selecting the most promising theory based on analysis of collected evidence, is not possible without some method of selecting between theories which do fit the evidence. This is because, for every set of data, there are an infinite number of theories which are consistent with those data (this is known as the Underdetermination Problem).

You can find Occam’s razor in detail in astronomy, physics, biology, and medicine. It even appears explicitly in statistics.

There are various papers in scholarly journals deriving versions of Occam's Razor from probability theory and applying it in statistical inference, and also of various criteria for penalizing complexity in statistical inference. Recent papers have suggested a connection between Occam's Razor and Kolmogorov complexity.

It should be evident that Occam’s razor is an important underpinning of the whole scientific enterprise.

Applications of Occam’s razor in computing

We already understand Occam’s razor clearly in information theory, in which we measure the information content of a transmission. Information theory has provided the theoretical foundation of all communications and data transmission since its invention by Claude Shannon in 1948.

Occam’s razor is particularly applied in minimum message length (MML) theory, which focuses on the least number of bits required to encode a given amount of information. 

MML has been in use since 1968. MML coding schemes have been developed for several distributions, and many kinds of machine learners including: unsupervised classification, decision trees and graphs, DNA sequences, Bayesian networks, Neural networks (one-layer only so far), image compression, image and function segmentation, etc.

While the terminology is not normally used, and I am not aware that Occam’s razor was explicitly used to formulate it, it is clear that the principles are clearly expressed in modern relational database theory, and in the practice of schema design. I suggest that is one of the reasons for the success of the DBMS approach to data storage and access.

Shannon’s information theory

Shannon’s application of the concept to information theory provides a good springboard to seeing how it applies to software design. So let’s understand information theory a little better. I don’t think I can do better than Wikipedia. Here is a snapshot of the article on the history of information theory:

Claude E. Shannon (1916–2001) founded information theory with his classic paper "A Mathematical Theory of Communication," published in the Bell System Technical Journal in July and October of 1948. At the beginning of his paper, Shannon asserted that "The fundamental problem of communication is that of reproducing at one point, either exactly or approximately, a message selected at another point." His theory for the first time considered communication as a rigorously stated mathematical problem in statistics and gave communications engineers a way to determine the capacity of a communication channel in terms of the common currency of bits. This problem is called the channel coding problem. The transmission part of the theory is not concerned with the meaning (semantics) of the message conveyed.

A second set of ideas in information theory relates to data compression. Using a statistical description for data, information theory quantifies the number of bits needed to describe the data. There are two formulations for the compression problem -- in lossless data compression the data must be reconstructed exactly, whereas lossy data compression examines how many bits are needed to reconstruct the data to within a specified fidelity level. This fidelity level is measured by a function called a distortion function. In information theory this is called rate distortion theory. Both lossless and lossy source codes produce bits at the output which can be used as the inputs to the channel codes mentioned above.

This division of information theory into compression and transmission is justified by the information transmission theorems, or source-channel separation theorems that justify the use of bits as the universal currency for information in many contexts. …

Communications channels have a fixed capacity for sending information per unit of time. To use that capacity as efficiently as possible, you distinguish between the message that is presented for sending and the actual information content of that message. What this amounts to is saying that the “information content” of a message is the smallest number of bits required to exactly reproduce the message after transmission. In plain language, the “information content” of a message is the message compressed as much as possible, with all repeats and redundancy removed, except for redundancy purposely introduced for error identification and correction.

Like any application of Occam’s razor, the focus on information content comes out of knowing what you care about; you include everything required to get what you care about and throw out everything else. What I care about in communications is accuracy and efficiency. Efficiency means that I should transmit the most information through a communications channel of given capacity as possible. This can be achieved by using the smallest possible number of bits to encode the message. 

The application of Information theory to software

How is all this relevant to building software? Aren’t we supposed to use structured design, use cases, components, or whatever to design software? Most of those techniques focus on the process of a good design, or a supposed ideal structure for a program. If we apply the same thinking that we see in information theory to software design, we will focus instead on the results of the process, and once we are satisfied we know how to judge whether a computer program is optimal or not, we will be able to find ways to build it.

Software specification and design has clearly been an island of technical art, separated from science and technology as a whole, except for those portions that have already come under the sway of “Occamal” thinking such as information and MML theory. Like any relatively isolated island of theory and practice, a wide variety of techniques and practices arise to fill the gap left by a completely baseless, ad-hoc approach. There are innumerable approaches to software design and building and no clear way to decide among them.

When designing software, we care most about things that are a very close analogy to what we care about in communications. In communications, we have a channel of given capacity, and want to squeeze the most information through it we can; therefore, we eliminate all redundancy from the original message, and transform it to contain only its pure information content. Anything extra would take transmission capacity and add nothing. In software design, we normally have capacity to build a representation of the program (source code) that is as long as we would like in terms of the capacity of the computer to execute the code. However, every line of code adds a burden to one or more stages of the whole chain of specify, design, code, test, document, train, learn, use, maintain, modify, and repeat. This chain of effort, the entire lifecycle of the program and everyone who touches it, is like the communications channel in information theory. The idea is that most program specifications are highly redundant, like messages in their original form. We want to define the “information content” of programs just like we define the information content of messages, so that every redundant or repeating group is eliminated, and what is left is everything required to make the program operate and absolutely nothing else. We focus on information content in communications because we want to make the best use of fixed communications capacity; we focus on information content in software because we want to make the best use of all the resources that are involved with the software in any way.

We don't just want to make programs short for general reasons. We also want to make them easy to change, with as little effort and error as possible. By eliminating all redundancy from a program, we make it so that there's exactly one place to go to make a change.

Getting back to Occam (though I admit Shannon and fancy formulas are more impressive and intimidating), if Occam’s razor is:

Entia non sunt multiplicanda praeter necessitatem.

No more things should be presumed to exist than are absolutely necessary.

Occam’s razor applied to software would be:

No more software entities should be created than are absolutely necessary.

Why? Just as sending bits beyond the information content of the message has a cost but doesn’t improve the message that is received, so do additional entities in the specification or expression of the program add to the cost to build and the cost to change without increasing the value of the program.

This may not sound dramatic or exciting. But as someone who has lived with the process of building software for a long time, and has struggled extensively with the question of what makes software “good,” it’s very satisfying to have an objective criterion that enables you to judge the “goodness” of a program, and to say when a program is “optimally” good. It gets exciting when you realize that this abstract notion of optimality has consequences that are extremely practical and down-to-earth. In particular, it shows you a path to building programs more quickly than using any other method; doing less to build the programs than you thought you had to do; and having the resulting programs be as easy and safe to change as it is possible for programs to be.