The Black Liszt

Flowcharts and Workflow in Software

The concept of workflow has been around in software from the beginning. It is the core of a great deal of what software does, including business process automation. Workflow is implicitly implemented in most bodies of software, usually in a hard-coded, ad-hoc way that makes it laborious and error-prone to implement, understand, modify and optimize. Expressing it instead as editable declarative metadata that is executed by a small body of generic, application-independent code yields a huge increase in productivity and responsiveness. It also enables painless integration of ML and AI. There are organizations that have done exactly this; they benefit from massive competitive advantage as a result.

Let’s start with some basics about flowcharts and workflow.

Flowcharts

Flowcharts pre-date computers. The concept is simple enough, as shown by this example from Wikipedia:

The very earliest computer programs were designed using flowcharts, illustrated for example in a document written by John von Neumann in 1947. The symbols and methods became standardized. By the 1960’s software designers used templates like this from IBM

to produce clean flowcharts in standardized ways.

Flowcharts and Workflow

Flowcharts as a way to express workflows have been around for at least a century. Workflows are all about repeatable processes, for example in a manufacturing plant. People would systematize a process in terms of workflow in order to understand and analyze it. They would create variations to test to see if the process could be improved. The starting motivation would often be consistency and quality. Then it would often shift to process optimization – reducing the time and cost and improving the quality of the results. Some of the early work in Operations Research was done to optimize processes.

Workflow is a natural way to express and help understand nearly any repeatable process, from manufacturing products to taking and delivering orders in a restaurant. What else is a repeatable process? A computer program is by definition a repeatable process. Bingo! Writing the program may take considerable time and effort, just like designing and building a manufacturing plant. But once written, a computer program is a repeatable process. That’s why it made sense for the very earliest computer people like John von Neumann to create flowcharts to define the process they wanted the computer to perform repeatedly.

What’s in a Flowchart?

There are different representations, but the basic work steps are common sense:

• Get data from somewhere (another program, storage, a user)
• Do something to the data
• Test the data, and branch to different steps depending on the results of the test
• Put the data somewhere (another program, storage, a user)
• Lots of these work steps are connected in a flow of control

This sounds like a regular computer software program, right? It is! When charted at the right level of detail, the translation from a flowchart to a body of code is largely mechanical. But humans perform this largely mechanical task, and get all wrapped up in the fine details of writing the code – just like pre-industrial craftsmen did.

Hey, that's not just a metaphor -- it is literally true! The vast, vast majority of software programming is done in a way that appears from the outside to be highly structured, but in fact is designing and crafting yet another fine wood/upholstery chair (each one unique!) or, for advanced programmers, goblets and plates made out of silver for rich customers.

Workflow

In the software world, workflow in general has been a subject of varying interest from the beginning. It can be applied to any level of detail. It has led to all sorts of names and even what amount to fashion trends. There is business process management. Business process engineering. And re-engineering. And business process automation. A specialized version of workflow is simulation software, which led early programmers to invent what came to be called "object-oriented programming." To see more about this on-going disaster that proved to be no better for simulating systems than it has been for software in general, see this.

When document image processing became practical in the 1980’s, the related term workflow emerged to describe the business process an organization took to process a document from its arrival through various departments and finally to resolution and archiving of the document. The company that popularized this kind of software, Filenet, was bought by IBM. I personally wrote the workflow software for a small vendor of document image processing software at that time.

Workflow in practice

There has been lots of noise about what amounts to workflow over the years, with books, movements and trends. A management professor in the 1980's talked about how business processes could be automated and improved using business process re-engineering. He said that each process should be re-thought from scratch -- otherwise you would just be "paving the cow paths," instead of creating an optimal process. As usual, lots of talk and little action. Here's the story of my personal involvement in such a project in which the people in charge insisted they were doing great things, while in fact they were spending lots of money helping the cows move a bit faster than they had been.

The Potential of Workflow

The potential of workflow can be understood in terms of maps and driving from one place to another. I've explained the general idea here.

Most software design starts with the equivalent of figuring out a map that shows where you are and where you want to get to. Then the craftsmanship begins. You end up with a hard-coded set of voluminous, low-level "directions" for driving two blocks, getting in the left lane, turning left, etc.

When the hard-coded directions fail to work well and the complaints are loud enough, the code is "enhanced," i.e., made even more complex, voluminous and hard to figure out by adding conditions and potential directional alternations.

Making the leap to an online, real time navigation system is way beyond the vast majority of software organizations. You know, one that takes account of changes, construction, feedback from other drivers on similar routes about congestion, whether your vehicle has a fee payment device installed, whether your vehicle is a truck, etc. Enhancements are regularly made to the metadata map and the ML/AI direction algorithms, which are independent of map details.

When software stays at the level of craftsmanship, you're looking at a nightmare of spaghetti code. Your cow paths aren't just paved -- they have foundations with top-grade rebar, concrete and curbs crafted of marble.

Conclusion

Metadata-driven workflow is the next step beyond schema enhancement for building automated systems to perform almost any job. It's a proven approach that many organizations have deployed -- literally for decades. But all the leaders of computing, including Computer Science departments at leading universities, remain obsessed with subjects that are irrelevant to the realities of building software that works; instead they stay focused on the wonders of craftsman-level low-level software languages. It's a self-contained universe where prestige is clearly defined and has nothing to do with the eternal truths of how optimal software is built.

 

Cartoons and Video games evolved into Bitcoin and NFT’s

Bitcoin and other cryptocurrencies are in the news. NFT’s (non-fungible tokens) have exploded onto the scene, with people spending large amounts of money to acquire unique rights to digital images. The explosion of invention and innovation is amazing, isn’t it?

Except that it's all just minor variations of things that were created decades ago, grew into huge markets with the participation of a good part of the world's population, and continue to grow today. Invention? Creativity? How about minor variations of proven ideas, giving them a new name and slightly different context, and getting super-rich?

From Drawing to Cartoons to Video Games

Drawing, sculpting and otherwise creating artificial images of the reality we experience has a long history.

For example, here’s a painting of a bovine from a cave created by early humans over 40,000 years ago:

Drawings that suggest reality but are purposely different from real things are called cartoons, and go back hundreds of years, becoming more widespread in the 1800’s in print media.

Then there was a breakthrough: animation. Leveraging early movie technology, artists worked enormously hard to create a fast-changing sequence of images to create the illusion of motion. Along with sound, you could now go to a theater and watch and hear a whole cartoon movie, filled with characters and actions that could never happen in real life. Characters like Mickey Mouse and Bugs Bunny became part of modern culture.

The next big step took place after computers were invented and got video screens. Of course the computers transformed the process of creating animation. But animation was always like watching a movie: the human could only watch and listen. With computers, the possibility first arose for actions of the person to directly and immediately change what happened on the screen. The video game was born.

The video game has gone through an extensive evolution from the primitive, simple Space War to immersive MMORPG's (massively multiplayer online role-playing games), enabling players to interact with each other in evolving shared animated worlds, often with fighting but also including other activities.

World of Warcraft (WoW) wasn't the first, but became the most popular of the MMORPG's.

Similar to other MMORPGs, the game allows players to create a characteravatar and explore an open game world in third- or first-person view, exploring the landscape, fighting various monsters, completing quests, and interacting with non-player characters (NPCs) or other players. The game encourages players to work together to complete quests, enter dungeons and engage in player versus player (PvP) combat, however the game can also be played solo without interacting with others. The game primarily focuses on character progression, in which players earn experience points to level up their character to make them more powerful and buy and sell items using in-game currency to acquire better equipment, among other game systems.

World of Warcraft was a major critical and commercial success upon its original release in 2004 and quickly became the most popular MMORPG of all time, reaching a peak of 12 million subscribers in 2010.^[4] The game had over one hundred million registered accounts by 2014^[5] and by 2017, had grossed over $9.23 billion in revenue, making it one of the highest-grossing video game franchises of all time. The game has been cited by gaming journalists as the greatest MMORPG of all time and one of the greatest video games of all time

The industries creating hardware and software for these artificial worlds has grown to be huge. In 2020 video gaming generated over $179 billion in global revenue, having surpassed the film industry years before.

Video games aren’t just for kids. There are an estimated 3.24 billion gamers across the globe.

In the US the numbers are huge. “Three out of every four, or 244 million, people in the U.S. play video games, an increase of 32 million people since 2018." Gamers spend lots of time on their games: “... gamers average 14 hours per week playing video games.”

Game World and Virtual Economies

Huge numbers of people go to a screen or put on a headset and "enter" the world of a video game, where they often spend hours at a time. While in that world, they can move from place to place as an observer, or as the controller of their personal avatar. They can interact with others, as shown by this scene from the virtual world of Second Life in 2003.

Long before Bitcoin was created, video games had virtual economies with digital currencies.

The currency used in a game world can be called different things. For example in World of Warcraft it's called -- big shock coming up here -- Gold. Gold can be earned by players accomplishing things in the game world, and can be spent for skills or in-game objects. Players can buy and sell items among themselves using such currencies. Many games enable players to buy in-game currencies using real money. In some cases, in-game virtual "land" is also for sale.

Long before Bitcoin, markets arose to enable in-game currencies to be traded (exchanged) for real-world currencies. It is now a multi-billion dollar industry. "In 2001, EverQuest players Brock Pierce and Alan Debonneville founded Internet Gaming Entertainment Ltd (IGE), a company that offered not only the virtual commodities in exchange for real money but also provided professional customer service." The company was the largest such on-line exchange and accounted for hundreds of millions of dollars of transactions.

Video Games, Bitcoin and NFT's

The first Bitcoin was sent in 2009. It wasn't much used or valued until 2013. Ethereum first went live in 2014. By this time there were already MMORPG's with many hundreds of millions of players earning, spending and exchanging digital currencies involving virtual objects in their game worlds.

Let's see how the things used by literally billions of gamers compares to Bitcoin (and other crypto-currencies) and NFT's.

• Games have digital currencies with no real-world value.
  • Sounds like Bitcoin and other crypto-currencies
    
• In-game virtual objects can be bought and sold using in-game currencies
  • Sounds like buying crypto-world NFT's with Bitcoin
    
• New units of the digital currency are created by the game software
  
  • New crypto is created by Bitcoin mining software
    
• Game currencies can be used and exchanged among gamers
  • Same with Bitcoin
• Game currencies can be exchanged for and bought with real-world money
  
  • Same with Bitcoin
• There are exchanges outside the game that enable buying/selling
  • Same with Bitcoin
• The exchange price can vary greatly
  • Same with Bitcoin
• Teams create new games with currencies and virtual objects
  
  • Teams create new crypto-currencies and NFT's
    

Still think there's no relationship between gaming and crypto? How about, as mentioned above, the fact that Brock Pierce and a partner founded the game currency exchange IGE in 2001, and the same Mr. Pierce was active in crypto-currency by 2013 and became a "Bitcoin billionaire" by 2018.

Of course, the new worlds of crypto and NFT's are different in some important ways from the gaming worlds. Games along with the objects and currencies are created and managed by the game company. While there's more control than is generally recognized, crypto-currencies have a large degree of self-management with their built-in miners. Similarly, NFT's are created independently

Conclusion

First Bitcoin came seemingly out of nowhere in 2009. A few years later, variations of Bitcoin appeared on the market. An astounding explosion of crypto followed, along with digital objects that "live" in the crypto world.

Like many other "brand new" things, the worlds of crypto and NFT's have remarkably close relations to the world of gaming, from which they appear to have evolved. Compared to the gaming world, the number of people invested in crypto is truly tiny, hundredths of a percent. But the inflation and amount of real-world currency that has been converted to crypto dwarfs the amounts in the gaming world.

As with many other tech trends, the history and evolution of the elements of the trend reward study.

Note: this was originally published on Forbes.

How to Fix Software Development and Security: A Brief History

As the use of computers grew rapidly in the 1960’s, the difficulty of creating quality software that met customer needs became increasingly evident. Wave after wave of methods were created, many of them becoming standard practice – without solving the problem! This is a high-level survey of the failed attempts to solve the problem of software development and security -- all of which are now standard practice in spite of failing to fix the problem! There are now software auditing firms that will carefully examine a software organization to see in which ways it deviates from ineffective standard practice – so that it can be “corrected!”

Houston, We've Had a problem

The astronauts of the Apollo 13 mission to the Moon radioed base control about the problem that threatened the mission and their lives.

The astronauts were saved after much effort and nail-biting time.

Should the people in charge of software make a similar call to mission control? Yes! They have made those calls, and continue to make them, thousands of times per day!!

Getting computers to do what you want by creating the kind of data we call software was a huge advance over the original physical method of plugs and switches. But it was still wildly difficult. Giant advances were made in the 1950’s that largely eliminated the original difficulty.

As the years went on into the 1960’s, the time, cost and trouble of creating and modifying software became hard to ignore. The problems got worse, and bad quality surfaced as a persistent issue. There were conferences of experts and widely read books, including one from a leader of the IBM 360 Operating System project, one of the largest efforts of its kind.

The Apollo 13 astronauts were saved, but the disaster in software development has resisted all the treatments that have been devised for it. With the invention and spread of the internet, we are now plagued with cybercrime, to the extent that ransomware attacks now take place (as of June 2021) ... get ready for it ... 149,000 times per week!! I think a few more exclamation points would have been appropriate for that astounding statistic, but I'm a laid-back kind of guy, so I figured I'd stick with a mild-mannered two. Here's my description from more than four years ago on the state of ransomware and the response of the "experts."

Following are highlights of the methods that have been devised to solve the problem of software development and security. None of which have worked, but have nonetheless become standard practice.

Programming Languages

The assembled experts in 1968 decided that "structured programming" would solve the problem of writing software that worked. Here's how that turned out. A group of experts recently convened to survey 50 years of progress in programming languages. They were proud of themselves, but had actually made things worse. Highlights here. One of those early efforts led to object-oriented languages, which are now faith-based dogma in the evidence-free world of Computer Science. New languages continue to be invented that claim to reduce programmer errors and increase productivity. What always happens is that the "advances" are widely publicized while the large-scale failures are concealed; here are a couple of juicy examples. Above all, the flow of new languages provides clear demonstration that programmers don't have enough useful work to do to keep them busy.

Outsourcing

While programmers babbled among themselves about this language or that method, company managers couldn't help but noticing that IT budgets continued to explode while the results were an avalanche of failures. Service companies emerged that promised better results than company management could achieve because they claimed to be experts in software and its management.

One of the pioneers in computer outsourcing was Ross Perot's Electronic Data Systems (EDS), which had $100 million in revenue by 1975. By the mid-1980's it had over 40,000 employees and over $4 billion in revenue. It continued to grow rapidly. along with a growing number of competitors including services branches of the major computer vendors. In a typical deal, a company would turn over some or all of its hardware and software operations to the outsourcer, who would add layers of management and process. They succeeded by lowering expectations and hiding the failures.

Off-shoring

As communications and travel grew more cost effective, basing outsourced operations in another country became practical. Off-shoring had the huge advantage that while the results were no better than regular outsourcing, the fact that employees were paid much less than US wages enabled marginally lower prices and a huge infrastructure of management and reporting which further disguised the as-usual results.

US-based outsourcers started doing this fairly early, while non-US vendors providing software services grew rapidly. Tata Computer Services first established a center in the 1980's and now has revenue over $20 billion and has been the world's most valuable IT company, employing over 500,000 people. Infosys is another India-based giant, along with hundreds of others.

Project Management

Deeper investment in the growing and evolving collection of project management methods has been a key part of software organizations. The larger the organization, the greater the commitment to widely accepted, mainstream project management methods. As new methods enter the mainstream, for example Agile, new managers and outsourcing organizations sell their embracing of the methods as a reason for hiring and contracting managers to feel confident in the decisions they are making. Project management has definitely succeeded in forming a thick layer of obfuscation over the reality of software development, quality and security. The only method of project management that has ever delivered "good" results for software is the classic method of wild over-estimation. See this for one of the conceptual flaws at the heart of software project management, see this for an overview and this for a comprehensive look.

Education and Certification

Nearly all professions have education and certification requirements, from plumbers and electricians to doctors. Software organizations grew to embrace imposing education and certification requirements on their people as a sure method to achieve better outcomes. This came to apply not to just the developers themselves but also to the QA people, security people and project managers. Unlike those other professions, computer education and certification has been distinctly divorced from results.

Standards, Compliance and Auditing

Standards were developed early on to assure that all implementations of a given software language were the same. As software problems grew, standards were increasingly created to solve the problem. The standards grew to include fine-grained detail of the development process itself and even standards that specified the "maturity level" of the organization.

Just as a new house is inspected to assure that it complies with the relevant building code, including specialists to inspect the plumbing and electricity, the software industry embraced a discipline of auditing and compliance certification to assure that the organization was meeting the relevant standards. A specialized set of standards grew for various aspects of computer security, with different measures applied to financial transactions and healthcare records for example. The standards and compliance audits have succeeded in consuming huge amounts of time and money while having no positive impact on software quality and security.

Suppress, deflect and Ignore

Everything I've described above continues apace, intent on its mission of making software development successful and computer security effective. People who write code spend much of their time engaged in activities that are supposed to assure that the code they write meets the needs in a timely manner and has no errors, using languages that are supposed to help them avoid error, surrounded by tests created by certified people using certified methods to assure that it is correct. Meanwhile highly educated certified computer security specialists assure that security is designed into the code and that it will pass all required audits when released.

How is this working out? In spite of widespread information blackouts imposed on failures, enough failures are so blatant that they can't be suppressed or ignored that we know that the problems aren't getting better.

Conclusion

We're over 50 years into this farce, which only continues because the clown show that sucks in all the money is invisible to nearly everyone. The on-the-scene reporters who tell all the spectators what's happening on the field can't see it either. What people end up hearing are a series of fantasies intended to deliver no surprises, except in those cases where the reality can't be hidden. Mostly what people do is things that make themselves feel better while not making things better. The time for a major paradigm shift to address these problems has long since passed.

Software Programming Language Evolution: Structures, Blocks and Macros

In prior posts I’ve given an overview of the advances in programming languages, described in detail the major advances and defined just what is meant by “high” in the phrase high-level language. In this post I’ll dive into the additional capabilities added to 3-GL’s that brought them to a peak of productivity.

History

Let’s remember what high-level languages are all about: productivity! They are about the amount of work it takes to write the code and how easy code is read.

The first major advance, from machine language to assembler, was largely about eliminating the grim scut-work of taking the statements you wanted to write and making the statements “understandable” to the machine by expressing them in binary. Ugh.

The second major advance, to 3-GL’s like FORTRAN and COBOL, was about eliminating the work of translating from your intention to the assembler statements required to express that intention. A single line of 3-GL code can easily translate into 10 or 20 lines of assembler code. And the 3-GL line of code often comes remarkably close to what you actually want to “say” to the computer, both writing it and reading it.

FORTRAN achieved this goal to an amazing extent.

“with the first FORTRAN compiler delivered in April 1957.^[9]:75 This was the first optimizing compiler, because customers were reluctant to use a high-level programming language unless its compiler could generate code with performance approaching that of hand-coded assembly language.^[16]”

“While the community was skeptical that this new method could possibly outperform hand-coding, it reduced the number of programming statements necessary to operate a machine by a factor of 20, and quickly gained acceptance.”

The reduction in the amount of work was the crucial achievement, but just as important was the fact that each set of FORTRAN statements were understandable, in that they came remarkably close to expressing the programmer’s intent, what the programmer wanted to achieve. No scratching your head when you read the code thinking to yourself “I wonder what he’s trying to say here??” This meant easier to write, fewer errors and easier to read.

Enhancing conditional branching and loops

The very first iteration of FORTRAN was an amazing achievement, but it’s no surprise that it wasn’t perfect. At an individual statement level it was nearly perfect. When reading long groups of statements there were situations where the code was clear, but the intention of the programmer not clearly expressed in the code itself – it had to be inferred from the code.

The first FORTRAN had a couple of the intention-expressing statements: a primitive IF statement and DO loop. Programmers soon realized that more could be done. The next major version was FORTRAN 66, which cleaned up and refined the early attempts at structuring. Along with the appropriate use of in-line comments, it was nearly as clear and intention-expressing as any practical programmer could want.

The final milestone in the march to intention-expressing languages was C.

It’s an amazing language. While FORTRAN was devised by people who wanted to do math/science calculations and COBOL by people who wanted to do business data processing, C was devised to enable computer people to write anything – above all “systems” software, like operating systems, compilers and other tools. In fact, C was used to re-write the first Unix operating system so that it could run on any machine without re-writing. C remains the language that is used to implement the vast majority of systems software to this day.

I bring it up in this context because C added important intention-expressing elements to its language that have remained foundational to this day. It enhanced conditional statements, creating the full IF-THEN-ELSE-ENDIF that has been common ever since. This meant you could say IF <condition is true> THEN <a statement>. The statement would only be executed if the condition was true. You could tack on ELSE <another statement> ENDIF. This isn’t dramatic, but the only other way to express this common thought is with GOTO statement – which certainly can be understood, but takes some figuring. In addition, C added the ability to use a delimited block of statements wherever a single statement could be used. When there are a moderate number of statements in a block, the code is easy to read. When there would be a large number, a good programmer creates a subroutine instead.

C added a couple more handy intention-expressing statements. A prime example is the SWITCH CASE BREAK statement. This is used when you have a number of conditions and something specific to do for each. The SWITCH defines the condition, and CASE <value> BREAK pairs define what to do for each possible <value> of the SWITCH.

Lots of following languages have added more and more statements to handle special cases, but the cost of a more complex language is rarely balanced with the benefit in ease and readability.

The great advance that's been ignored: Macros

C did something about all those special cases that goes far beyond the power of adding new statements to a language, and vastly increases not just the power and readability of the language but more important the speed and accuracy of making changes. This is the macro pre-processor.

I was already very familiar with macros when I first encountered the C language, because they form a key part of a good assembler language – to the extent that an assembler that has such a facility is usually called a macro-assembler. Macros enable you to define blocks of text including argument substitution. They resemble subroutine calls, but they are translated at compile time to code which is then compiled along with the hand-written code. A macro can do something simple like create a symbol for a constant that is widely used in a program, but which may have to be changed. When a change is needed, you just change the macro definition and – poof! – everywhere it’s used it has the new value. It can also do complex things that result in multiple lines of action statements and/or data definitions. It is the most powerful and extensible tool for expressing intention and enabling rapid, low-risk change in the programmer’s toolbox. While the C macro facility isn’t quite as powerful as the best macro-assemblers, it’s a zillion times better than not having one at all, like all the proud but pathetic modern languages that wouldn’t know a macro if it bit them in the shin.

The next 50 years of software language advances

The refrain of people who want to stay up to date with computers is, of course, "What's new?" Everyone knows that computers evolve more quickly than anything else in human existence, by a long shot. The first cell phones appeared not so long ago, and they were "just" cell phones. We all know that now they're astounding miniature computers complete with screens, finger and voice input, cameras and incredible storage and just about any app you can think of. So course we want to know what's new.

The trouble is that all that blindingly-fast progress is in the hardware. Software is just along for the ride! The software languages and statements that you write are 99% the same as in the long-ago times before smart phones. Of course there are different drivers and things you have to call, just as in any hardware environment. But the substance of the software and the lines of code you use to write it are nearly the same. Here is a review of the last 50 years of "advances" in software. Bottom line: it hasn't advanced!

Oh, an insider might argue: what about the huge advances of the object-oriented revolution? What about Google's new powerhouse, Go? Insiders may ooh and ahh, just like people at a fashion catwalk. The come-back is simple: what about programmer productivity? Claims to this effect are sometimes made, but more often its that the wonderful new language protects against stupid programmers making errors or something. There has not been so much as a single effort to measure increase of programmer productivity or quality. There is NO experimental data!! See this for more. Calling what programmers do "Computer Science" is a bad joke. It's anything but a science.

What this means is simple: everyone knows the answer -- claims about improvement would not withstand objective experiments -- and therefore the whole subject is shut down. If you're looking for a decades-old example of "cancel culture," this is it. Don't ask, don't tell.

Conclusion

3-GL's brought software programming to an astounding level of productivity. Using them you could write code quickly. The code you wrote came pretty close to expressing what you wanted the computer to do with minimal wasted effort. Given suitable compilers, the code could run on any machine. Using a 3-GL was at least 10X more productive than what came before for many applications.

A couple language features were incorporated into the very first languages, like conditional branching and controlled looping, that were good first steps. The next few years led early programmers to realize that a few more elaborations of conditional branching and controlled looping would handle the vast majority of practical cases. With those extra language features, code became highly expressive. All subsequent languages have incorporated these advances in some form. Sadly, the even more productive feature of macros has been abandoned, but as we'll see in future posts, their power can be harnessed to an even greater extent in the world of declarative metadata.

The Crisis in Software is over 50 years old

Everyone knows there’s a problem in software. No one likes to talk about it. Waves of new tools and techniques are introduced, hyped and fade away. Are there waves of new tools and techniques in accounting? During interviews, are accountants asked if they practice the currently “hot” methods? Of course not! What about cars? Hah, you might say – look at the wave of new electric cars! Yes, let’s look at them – do electric cars crash more than normal ones? If cars crashed or mis-performed at the same rate as software, we would all be afraid to ride in them!

The existence of a software crisis was first prominently identified in 1968. The results of the crisis were clear at that time, and haven’t changed a bit since then. Nothing – no wonderful new language, paradigm, project management or quality method, architectural technique or anything else has made a dent in the crisis.

The solutions to the crisis have largely been identified and repeatedly proven in practice by small groups of programmers who desperately need to get things done. Their methods are ignored or suppressed by the ruling elites in academia, government and corporations – including Big Tech. Sadly, this is unlikely to change any time soon. Happily, effective methods for building software that are ignored and suppressed provide a way for small, ambitious groups to do new stuff and win!

What is the software crisis?

The term “software crisis” was coined during the 1968 NATO Software Engineering Conference. Many computer science experts attended. There was general agreement that there was a serious problem. Part of the solution was identified by creating a discipline of “computer engineering,” which would structure and formalize the techniques for building software. Papers identifying the problems and solutions were generated as a result of the conference and a similar one held the following year.

The problems are summarized in the Wiki article:

Look familiar?

How do we know there’s a crisis?

When things are really bad, it’s typical for people to avoid talking about it. When there’s a whiff of a solution, though, it’s likely to receive attention. When people engage in something that’s likely to go wrong, most of them will get the best advice and be sure to follow widely accepted practices in order to avoid trouble. If something goes wrong anyway, having followed the standard accepted techniques provides a good defense – I did what people in my position are supposed to do! When hiring people, the hiring organization will often trumpet their use of these authorized and accepted methods to assure avoidance of blame. And finally, when things go wrong, great care is taken to assure that as few people know about the problem as possible.

When you hire someone in physics, do you advertise that the position requires knowledge and acceptance of relativity theory? When you hire a doctor, do you need to make sure that the doctor adheres to the infectious theory of disease?

We know there’s a crisis because, in spite of all the effort to keep it quiet, disasters happen that are so public, widespread and annoying that they get talked about. The trouble is that the highly publicized disasters are just the visible tip of a truly gigantic iceberg that takes up most of the space in the ocean of software.

We know there's a crisis because the academic field devoted to the subject, Computer Science, isn't a "science." Not even close. Check out these.

We know there's a crisis because the famous big tech companies have amazing reputations, like they're all geniuses -- while the facts show that they're pretentious bumblers with astounding salaries.

We know there's a problem because of the fashion-driven nature of the field. Here is a discussion of specific fashions, and see these for examples.

Conclusion

Look again at the list of problem identified over 50 years ago: software that's late, over budget, not what you wanted, poor quality, and sometimes has to be thrown out. This is one of the reasons managers take estimates from programmers and multiply the time by 3X before passing them on up the chain. Higher managers pad even more. People try to avoid publicizing the wonderful thing that's coming real soon because all too often, days before completion, it blows up.

There have been decades of widely-touted solutions to the problem that never solve the problem. Here is a review of 50 years of "progress" in programming languages, for example.

There are solutions to this 50 year problem. Here is one of those solutions. Small groups of programmers, pressured to produce great results in short period of time, re-invent the solutions. There are underlying errors of thought that are the primary factors behind the problem and its incredible persistence. How else can you understand the near-universal resistance to winning methods-- proven in practice! -- to be ignored?

 

Software programming language evolution: Functional Languages

Once computers were invented and started being used, people discovered that writing programs for them was brutally hard. It didn’t take long before some smart folks figured out ways to make it easier, taking two giant steps forward in ease and productivity in quick succession.

After that wonderful start, people kept on inventing new languages, but somehow never made things better – in fact the new languages often made programming harder and less productive, a situation that the experts and exalted Professors studiously ignored. They never even bothered to spell out what makes one language better than another, which you'd think would be at the heart of promoting a new language.

I this post I’ll discuss a major category of new languages that keep getting created, receive passionate kudos, are rarely used but refuse to die: functional languages. In a subsequent post I'll describe the truly good idea that lurks inside the madness of the functional language world.

The core idea: time and timeless, music and math

I learned math as a kid. In high school I took the most accelerated and advanced courses available, finishing with a course in calculus, which I aced while mostly sitting in the back of the classroom teaching myself differential equations from my father’s grad school text book. I liked it. At the same time I got into music of multiple kinds but with a strong preference for classical. The strong connection between music and math has often been noted.

During my junior year of high school I took a course in computer programming, with FORTRAN as the language. Our “teacher,” scrambling to keep a chapter ahead of us in the text book, had arranged machine time for the class on Saturdays at a computer at a nearby rocket firm, Reaction Motors. After lots more programming, by my first year of college, it was clear that math took a distinct second place to both music and software in my heart.

In this post I explain where software comes from and its relation to music. The key concept is that, while you can read a page of music or software or math, math usually exists in a world “outside” of time – at most, time is a dimension like space; think for example of a proof in geometry. Music and software, by contrast, can only be understood as flowing through time. When you look at a page of music or software you naturally start at the beginning and hear/see it flow through time in your mind, just like reading a page of text.

What this means is that there is a fundamental dis-connect between math and computing. I explore this disconnect here. The math types are always trying to turn software into math, for example plodding for decades on a fruitless pursuit for ways to “prove” the correctness of programs.

This is the best way to understand both the failure and the refusal to admit defeat of the math types to develop a practical math-like way to write software, an effort and category of languages that is usually called “functional.”

The history of functional languages

Wikipedia's introduction to functional languages is accurate:

Turing had nothing to do with making software programming practical. That amazing job fell to others, who created the two giant advances in software language development. But since the math types were hanging around computers in the early days, particularly because doing math calculation was one of the earliest tasks to which computers were applied, finding a way to make programming more math-like, essentially eliminating the factor of time from software was a top priority.

I had already done serious programming, including working on a giant FORTRAN project to optimize oil refinery operation, when a friend introduced me to functional programming for the first time. The language was APL. It came a bit closer to being able to handle practical math than many of the other non-functional "functional languages" because of its focus on the matrix, but it was as easy to read and understand as advanced math. I could and can imagine that it would be appropriate for representing certain math transformations in a compact way, but why anyone who wasn't being tortured would agree to use it for general programming remains beyond me -- assuming of course that what you want is to ... radical idea here, be warned! ... get stuff done. As opposed to demonstrate and revel in your ideological and mathematical purity and exalted status in the pantheon of Computer Science greatness.

None of this is any concern to the cultists, who march on inventing an endless stream of "new" functional languages. Among the names you might see if you look into this are  OCaml, Scheme, Haskell, Clojure, Scala. There are even a few that are object-oriented on top of being functional, like Erlang. Articles continue to pop their heads up out of the underground insisting that functional languages really are making inroads in commercial applications, better for getting a job, more effective to use to create your ground-breaking start-up, etc. I predict no end to the flow.

The fact that functional languages are mostly of interest to a cult of fanatical core believers minimizes their widespread, on-going pernicious impact on software development, fueled by the near-universal math orientation of academic and corporate Computer Science. Functional concepts keep getting introduced by language fanatics into otherwise-sane normal languages.

Conclusion

Computer software is rarely driven by purely practical considerations much less objective knowledge and definition of what constitutes "good" in a language. But when an approach like functional programming is pushed that leads to results that are often 10X worse on multiple dimensions than "normal" programming, the problem is so large that people who aren't already committed members of the cult notice the difference and refuse to put up with the nonsense.

In spite of all this, there is an amazing valuable insight at the core of the functional language movement that leads to highly productive, real-world-valuable results when embodied in the right way. I will spell this insight out and illustrate its use in a subsequent post. The heart of the insight is taking a declarative approach instead of an imperative one when and to the extent that is appropriate.

 

Micro-Services: the Forgotten History of Failures

Micro-services are one of today's leading software fashions; see this for more on software fashions in general. I have treated in depth the false claims of micro-services to make applications "scalable," and separately treated in depth the false claim that micro-services enhance programmer productivity.

Where the heck did such a crappy idea that has taken such a large mind-share in the software and management community come from?

I don't know. Who knows why skirts are short or long this year? Or whether shirts are tucked or not tucked? What I do know is that bogus fashion trends infect the software industry, some of them delivering decades of perniciousness. Looking at the spectrum of software fashions, I've noticed similarities among some of them. Some fashions well up, spread and peter out, only to recur again later with minor changes and a new name. The new name appears to be important, so that everyone is fooled into thinking the hot fashion is truly "new," not tarnished by the abject failures of its prior incarnations. I describe a clear example of an old idea with new name here.

The History of Micro-services

Why do we have micro-services? How did this bad idea get to be so popular? Was it someone's recent "great idea?" There are probably people who think it's their idea.

Do micro-services have a history? Of course they do! Nearly every major practice in software has a history of some kind. The number of software practices that are newly invented is truly tiny and going down rapidly. So why do we hear about software advances and the "new thing" so often? The whole field of software ignores its own history in truly radical ways. It even ignores what other software people in other fields and domains are doing.

Nonetheless, anyone who knows a broad range of software practices over a period of decades can easily recognize similarities and patterns -- patterns that surely constitute "history" whether or not the actors recognize the precedents of what they do.

What are Services?

Let's step back and understand what "micro-services" are. By the name, it's obvious that a micro-service is an itty-bitty "service." So what's a service? Once you peel back all the layers, a service is a plain old subroutine call with a bunch of fancy, time-consuming stuff stuck in between the code that calls and the code that is called. See this for my explanation of the fundamental concept of the subroutine.

Subroutines (or functions or methods or whatever) are an essential aspect of programming. Every programming language has its syntax, statements and other things that people get all wrapped up in. Every programming language also has an associated library -- as in function/subroutine library. If you look at a manual for a language and one for the associated library, the library is nearly always larger. The richness of a library is a key factor in the utility of a language. In modern terms, an associated framework serves essentially the same purpose; an example is the RAILS framework for Ruby, without which Ruby would just be one line on the ever-growing list of mostly-forgotten languages.

When you're writing a program, knowing and making great use of the associated subroutine library is essential. But what if you see that you're doing the same kind of thing over and over in your program? It makes sense to create a subroutine to perform that common function as part of the overall application you're building. As time goes on, most well-structured applications include a large portion of nested subroutines.

What if there's a program that can only run on another computer and you want to call it as a subroutine -- give it some data to work on and get the results back? This problem was addressed and solved in many variations decades ago. It's most often called an RPC -- Remote Procedure Call. To make a call to a distant subroutine you implement a version of it locally that looks and acts the same, but does some networking to send the fact of the call and the parameters to a cooperating routine on the distant computer. That routine gets the network call, retrieves the data, and makes a local call to the subroutine. When the return happens, the data is sent back by the same mechanism. It's just like writing a memo and putting in your out-box. The clerk who picks up the outgoing memos delivers the ones that are local, and puts the ones that aren't into an envelope, addresses it and puts the memo with return instructions into the mail. And so on. The calling code and the called code act like they normally do and the RPC makes it happen.

Along comes the internet. People notice that http is supported all over the place. Why not leverage it to implement the mechanics of the RPC? Voila! Now we have SOAP -- Simple Object Access Protocol, which uses http and XML to send and return the data. A subroutine with a SOAP interface was called a "service." This all got standardized roughly 20 years ago. Then along came a simpler version to use that had even more overhead called Restful, which is still in widespread use.

If you need to call a subroutine that can't or shouldn't run on your computer but on some other computer, having some form of RPC is extremely useful. The time penalty can easily amount to a factor of thousands or even millions compared to a normal local subroutine call, but you'll gladly pay the price if there's no other way.

Now let's recall what all this amounts to. A "service" is a subroutine that has a HUGE amount of overhead. There is no reason to ever take on the overhead unless, in the vast majority of cases, the service you want to call is on a different computer than the one doing the calling, a computer that can only be accessed by some kind of networking.

Prior Incarnations of Service Insanity

There have been a couple waves of service fashion mania. One of the waves took place roughly twenty years ago during the internet bubble. People were very concerned to be able to support growth in the use of their applications by unprecedented factors. Expert opinion rapidly agreed that building a "distributed application" was the way to accomplish this. The idea was that the load on the computer would greatly exceed the capacity of even the largest, most capable computer to handle it. To anticipate this, the application should be architected to be able to run on multiple computers that could be changed at will. Instead of a "central" application, the application would be "distributed." This was further elaborated by replacing the simple RPC mechanism with queues called "service buses," as in a bus to route large numbers of service calls from one place to another. The bus itself had to be robust so that messages weren't dropped, so it evolved into an "enterprise service bus," something which exists today. A huge, complex mechanism for accomplishing this became part of java, J2EE (Java 2 Enterprise Edition). See this for the complex, reincarnating history of the transaction monitor.

It turned out that building a "distributed application" was vastly more expensive and time-consuming than just building a plain old application, the kind that today is sneeringly dismissed as being "monolithic." The computational and elapsed time running cost of such an application was also huge. Hardly any applications had loads so large that they needed to be distributed. Even worse, the few applications that ended up with huge loads that required many computers found vastly simpler, more effective ways to get the job done. It's worth noting that articles with titles like "Why were we all fooled into thinking building distributed applications was a good idea" never appeared. The whole subject simply faded away. Here's a description from another angle of the distributed computing mania.

Another incarnation of this nutty idea took place largely inside large enterprises. Many of these places had highly diverse collections of applications running the business, accumulated by acquisition, multiple departments building applications for themselves, etc. Instead of doing the sensible thing of reducing the total number of applications by evolving the best versions to be able to handle more diverse tasks, the idea of a SOA, service-oriented architecture, became the focus. All these applications could be turned into services! Instead of building new applications, people would now build services. Tools were built to manage all these new services. There were directories. There was tracking and control. All sorts of new demands came latching onto the IT budget.

SOA never got to the fever pitch of distributed applications, but it was big. It went the same way -- fading as it turned out to be a lot of time and trouble with little benefit.

Now we come to the present. Remember that the original motivation of the RPC in any form is that a subroutine you want to call is ONLY running on some other computer. An RPC, whatever the form or cost, is better than nothing. Sensible then and sensible now. Then came ways of building programs so that their parts COULD BE run on different computers, if the computational requirements became huge. This also turned out to be rarely needed, and when it was needed, there were always better, faster, cheaper ways than a service approach. Now, with micro-services, we have reincarnated these proven-to-be-bad ideas, claiming that not only will micro-services effortlessly yield that wonderful virtue "scalability," but it will even make programmers more productive. Wrong and wrong.

Conclusion

I like the phrase "oldie but goodie." Sometimes it's applicable. In computing when a hot new tech trend comes along that "everyone" decides is the smart way to go, it rarely is. It most often is an "oldie but baddie" with a new name and new image. You would think that software people were so facts-oriented that they wouldn't be subject to this kind of emotionally-driven, be-part-of-the-group, get-with-the-program-man kind of thinking. But most people want to belong and want increased status. If believing in micro-services is the ticket to membership in the ranks of the software elite, a shocking (to me) number of people are all in.

Software Evolution: Functionality on a New Platform

When a new “platform” emerges (UNIX, Windows, Web, Apps), if you look at any application area and see how it evolved on prior platforms, the application’s functionality will emerge on the new platform in roughly the same order, though often on a compressed timescale. The functionality that is relevant depends on the particular application area. This concept applies both to system and application software.

The pattern is: functionality emerges on a new platform in roughly the same order as it emerged on earlier platforms. The timescale of the emergence may be compressed; the important aspect of the pattern isn’t the timing but the order. The pattern means that functional steps are rarely skipped – what was next last time is also next this time. The pattern also means that when someone tries to introduce functionality too soon, before the functionality that preceded it on prior platforms is generally available, the market will not accept it.

While this pattern takes a good deal of knowledge and judgment to notice and apply, I have consistently been impressed by its predictive power. By following the pattern, you can be pretty confident that you’re building proven functionality, and that you’re following a pattern of success.

I have noticed a couple of danger points here. When the company is too aware of the pattern, it is easy for them to “get ahead of themselves” and more importantly ahead of the market, by solving problems that certainly will become important, but problems that the market doesn’t know it has yet. The “same order” part of the pattern is important; building ahead of the market appears to win few business benefits.

On the other hand, without knowledge of the pattern, it is easy to make up all sorts of things you think people might want, and build them, only to find out later that you wasted time and money, because the functionality you built is never part of the accepted function set.

Really great companies who have lots of creative people, the ability to execute, and listen closely to market reactions to what they’re doing, don’t “need” to know about this pattern. However, for the rest of us mortals down here, having a “cheat sheet” to what important features the market will be demanding next can prove awfully helpful.

Basic Example: operating systems

IBM famously created an operating system for their mainframe line of computers, OS/360. It had the capability of running multiple jobs at once. Its multi-tasking abilities grew and became more sophisticated through the 1970’s.

Eventually a transaction monitor, CICS, was written and became a de facto part of the operating system for applications with lots of interactive users. As the operating system was used, it became evident that various access methods for storage and communications needed to be separate from the core, and so clear interfaces were created, and the notion of completely self-contained access methods (for example, a file system) as replaceable units was supported. A strong security system was not part of the early versions of the operating system, and the need for one became critical, and so strong external modules were written and support for security was added to the core. While there was a “main” operating system, alternative operating systems were written for various purposes, and a virtual operating system was written to actually run on the “bare metal.” With VM (the virtual OS), you could devote most of a machine’s resources to the production users, while letting some of the users running on a completely different operating system.

While all this was taking place, people were studying and experimenting in university environments, deciding just what an operating system was and was not, what the best ways to build one were, and so on.

Before very long, mini-computers were invented; these were basically mainframes on the cheap, with all sorts of features and functions missing – but they were cheap. And, since each had a unique instruction set, each minicomputer needed an operating system. Programmers were hired, and those programmers, of course, ignored the mainframe operating systems, and built simple, cheap OS’s to go along with the cheap hardware. Surprise, surprise, those cheap OS’s resembled nothing as much as – the first generation of mainframe operating systems! But people quickly discovered the limitations, just as they had before, and set about making the same set of enhancements that the previous generation of pioneering programmers had made. Within ten years, they had re-invented many of the important mainframe OS concepts, and were on the way to building the rest.

With all this knowledge of operating systems floating around and pretty easily available, what do you suppose happened when people took the early micro-processor chips and made them into micro-computers? Naturally, they understood the state of the art of operating systems theory and practice and adapted an existing OS (which were starting to be built in high level languages) or built one that took all this knowledge and applied it with appropriate adjustments to the new environment, right? Bzzzzt! Of course not!

What the kids who were faced with the task did was start from scratch, not only in terms of code, but also in terms of knowledge. They didn’t stand on the shoulders of giants; they didn’t learn from the experiences of the many that preceded them; they built OS’s as though they were the first programmers who ever tried to do such a thing. And the result was pretty much like early mainframe (even pre-360!) operating systems. There was no serious memory protection or address mapping; there was no real concept of multiple users, multiple levels and types of users, or any real security; no multi-tasking; the access methods were hard-wired in, and so on. The limitations and problems emerged pretty quickly, and so did add-on rubber band and baling wire patches, just like in earlier days.

It’s a good thing that IBM came along at this point, and brought commercial order and education to the emerging microcomputer market. When they came out with the IBM PC, they not only legitimized the market, they had deep history with mainframes and minicomputers. They employed true experts who knew operating systems inside and out. They had a research division, where there were people who could tell you what the operating systems of the future would look like. So it makes sense they would get those experts together, and they would create a small but efficient micro-tasking kernel, common interfaces for installable access methods, and many other appropriate variations on all the modern operating systems concepts. The last thing such a smart, educated and astute major company like IBM would do was make an exclusive deal with a small company that had never built an operating system, who had just bought the rights on the cheap to a half-baked excuse for a primitive first-generation OS, and make that the IBM-blessed … Wait! … that’s what they did do! Arrrgggghhh!

Explanation

One might well ask, how can a pattern like this continue to have predictive power? Why wouldn’t the people who develop on a new platform simply take a little time to examine the relevant applications on the older platforms, and leap to the state of the art? Why wouldn’t customers demand it?

It is hard to know for sure, but I think there are a couple main factors at work, and there is evidence for the relevance of each of the factors.

The first factor is the developers. It is well known that most developers learn a platform and then stick with the platform they’ve learned for an extended period of time, basically as long as they can. The reason is simple: they are experts on the platform they already know, and therefore have prestige and make more money than they would as novices on a platform they’re just learning. I speculate that this is one of the many contributing factors to the rapid migration of ambitious programmers into management, where they can advance without being tied to a platform, at least as much. So who learns the new platforms? With few exceptions, new people. If you’re just entering the industry, you are counseled to learn the hot new languages; you tend to be sensitive to where the demands and rising salaries are. Still, you expect and are paid entry-level wages, along with most other people (except managers). Why should the experienced programmers jump to the new platform? They would have to compete with hard-working young people, their knowledge of the older platform will be considered a liability, and on top of everything else, they’d have to take a pay cut.

The result is that no one working on the new platform has an in-depth, working knowledge of the applications on the older platform, and at least in part because of this, everyone considers knowledge and use of the platform to be vastly more important than knowledge of an old application on an obsolete platform. So they ignore it! As a result, they dive in and attempt to automate the application area “from scratch.” Their attempts are usually quite close to every first generation program for that application on past platforms, because it turns out that the determining factor isn’t the platform, it’s the business problem. They proceed to re-discover, step by step, the reasons why the first generation was inadequate and had to be supplanted by a second generation, etc.

The second factor is the buyers. When a new platform emerges, most buyers simply ignore it. Why pay attention? It’s a toy, there are no good applications, etc. The few buyers who do pay attention tend to be crazy, early adopter types who just love the experience of trying new things. Like the programmers, they also tend to care about the platform more than the application – otherwise, they wouldn’t even consider buying what is typically a seriously immature application on the new platform. But they can only buy what’s being sold, and so they choose among the inadequate applications. Because they don’t care about applications as much as platforms, they don’t even ask for features they know could only be present in the mature applications for older platforms – they press the applications vendors for the “next” obvious cool feature, in the narrow universe of the new platform.

The reason why application evolution repeats itself on the new platform, then, is that nearly everyone involved, builder and buyer, is ignorant of the past and wears what amounts to blinders. It’s as though they are building and buying the application for the first time. Therefore, they respond most strongly to the same business pressures that people involved in computer automation tend to see first, and then next, and so on. It’s as though there’s a “natural” most climb-able path up a mountain, and successive waves of climbers approach the climb from the foot of the mountain, completely ignorant of the experiences of those who came before them, but confronted with the same options they tend to make the same choices, and so everyone takes roughly the same route up the mountain.

Why don’t smart groups who know all this leap-frog their way to the most advanced application functionality? I have seen this happen, and it’s the buyers who tend to rain on this kind of parade. The buyers tend to be familiar with the range of applications in a category, and those applications tend to address a highly overlapping set of problems in ways that vary only slightly. The “far-seeing” builder then comes along with all sorts of solutions to problems that the buyers don’t even know they have! The buyers look around in confusion – why isn’t anyone else talking about this? I kind of understand what you’re talking about, but I feel silly thinking that something’s important when no one else does. I think I’ll wait. And they do. So getting too far ahead of the buyers is just as much of a problem as being too far behind the competition. The result: the application evolution repeats itself on the new platform, in roughly the same order each time, and no “cheating” or “running ahead” is allowed.

This all sounds incredibly common-sense when I read it written down, but I have to admit that this particular piece of common sense is not only uncommon, it took me personally decades and multiple failures to finally get this simple thought into my thick skull. The key thought is this one: you may think you know a person has a problem; the problem might be a severe one, and cost the person a great deal of money and trouble; you may even be entirely right in this judgment. However, if the person in question does not think he has a problem, why should he pay for a solution – in fact, why would he go to the trouble of implementing a solution even if it were free?  Even worse, why would he even waste time talking with you, once he got the idea that you thought he had a problem he didn’t think he had? Why would he listen to a chimney sweeper’s pitch if he lives in a house without chimneys? The air quality in Los Angeles in 1970 was terrible. But there was no market for catalytic converters on automobile exhaust systems at that time. The problem that converters solve existed, and was getting worse. It was obvious for anyone to see. It was even talked about. But in peoples’ minds at the time, having a car that contributed to air pollution was not a problem most people accepted they had (even though we know that, objectively speaking, they did have it).

Elizebeth Smith Friedman: The Cancelled Heroine of Cryptography

The unheralded Elizebeth Smith Friedman is a textbook example of the vast gulf that too often separates achievement in a field from getting credit for the achievement.

She was a true pioneer of cryptography and code-breaking, leading multiple efforts against the international criminal mob and the Axis in World War II. Unlike most people called “leaders,” she was actually the best at what she did, personally cracking “uncrackable” codes and personally pioneering new methods. She was a leader in the true sense: the manager/boss of the long distance runners AND the runner far in front of everyone else who gets there first AND helps all her fellow runners speed up.

Technical History Issues

Making an advance in technology is hard. Not many people try to do it, and a tiny fraction of those who try seem to succeed. Many of those apparent successes burn out – they weren’t advances after all. Sometimes a true advance, for various reasons, is never adopted. When it is adopted, there is often a race to claim credit for the advance. The race isn’t so much a race as it is a no-holds-barred war. To win the war, you usually need the support of loads of people who have no idea what the advance is about. These ignorant people create history, along with its winners and ignored achievers.

Is this cynical? Yes. Is it an accurate description of what happens? In all too many cases, sadly yes. Here are examples of from the war for credit for inventing the computer.

In most cases, technical invention isn’t like a giant comet streaking to earth and creating a big boom. It’s more like a sequence of parallel, overlapping efforts to solve a problem or make something better. Often an advance is made by more than one person or group without involvement with the other. What in retrospect is described as the big advance is usually a step forward, one of many, building on earlier work. Sometimes the advance isn’t an advance so much as a commercialization. Matt Ridley describes this with many examples in his mostly excellent book on Innovation. Elizebeth stands out as being a true innovator on multiple dimensions.

Elizebeth Smith Friedman

Getting to the truth about inventors and technology innovation is a problem in general. In the case of Ms. Friedman, the problem was made worse by the credit-taking actions of government leaders. The truth has only emerged recently with the release of previously concealed documents and the ending of secrecy periods.

Here are some highlights of her career:

In the 1930s, Elizebeth Smith Friedman became America’s and indeed the world’s best-known codebreaker. She inflicted severe damage on the interests of organized crime and at times needed to be protected by bodyguards. The evidence she gave in criminal trials describing how she cracked encrypted messages passing between mobsters made her a newspaper sensation.

Later, during World War 2, she broke coded messages sent on Germany’s Enigma machines. These messages revealed a plot by the Argentinian government to help Germany replace South American governments with Nazis, giving Germany bases from which to attack America. Her discoveries allowed the western allies to thwart the Argentinian and German plans.

Elizebeth Smith Friedman’s wartime codebreaking work was so secret that she was forbidden to mention it in public. She died many years before government archives were brought to light showing what she had done. During and after World War 2, J. Edgar Hoover and the FBI took the credit for work Elizebeth and her U.S. Coast Guard team had carried out.

Her whole story is fascinating. Among other things she is a wonderful example of the power of bureaucracies (education, government and corporate) to control and often suppress outstanding talent, and how sometimes, when the bureaucracy is desperate for results, it will break its own rules to achieve a goal – and then claim credit. It is particularly striking in Elizebeth’s case because she was NOT trained in math or STEM of any kind; her fascination was with literature and philosophy.

There is a book about her life which is includes previously classified and/or ignored documents about her career:

If you’re at all interested in people like Alan Turing and Grace Hopper and/or computing history, it’s worth reading. She was completely “unqualified” to do what she did – and she became the best at it, for example cracking the German Enigma machine without the huge staff and machines at Bletchley Park that have become famous.

Likewise, I had never heard of her husband William Friedman, who was also accomplished as a code-breaker both by himself and working with Elizebeth. Here are a couple links that give some highlights, though I still recommend the book.

The giant advances in software programming languages

There have been two gigantic advances in programming languages, each contributing huge advances in programmer productivity. Since then, thousands of times more attention has been given to essentially trivial changes than has been given to the two giant advances. I described these 50 years of non-advances here. In this post I’ll go back in history to describe the two giant advances that were made in the early days of computing, back in the 50’s, and I’ll tack on the one important corrective that’s been made since then.

From machine language to assembler

Every machine has one and only one “machine language.” This is a stream of binary data that the machine’s processor “understands” as instructions. Each instruction causes the machine to perform a single action on a single piece of data.

I learned to read and write machine language for a couple different machines. It’s not easy. From early on, machine language was usually shown as hexadecimal instead of raw binary, so that instead of looking at “0100101000101111” you see the hexadecimal equivalent: 4C2F.  A big plus but you’re looking at a hexadecimal pile of data. A nice editor will at least break it up into lines of code, but you still have to read the hex for the instruction, registers and storage locations

Moving from machine language to assembler language was HUGE. Night and day in terms of readability and programmer productivity. Assembler language is a body of text that closely corresponds to the machine language. Each line of text typically corresponds to exactly one machine language instruction. Assembler language transformed programming. If you understand any machine language, you can easily write and read the corresponding assembly language. For example, here's the binary of a simple instruction:

Here's the equivalent in hexadecimal:

And here it is in assembler language, including a comment:

Just as important as making instructions readable was using text labels for locations of data and addresses of other instructions. In machine language, an address is expressed as a “real” address, for example as a hexadecimal number.  If you inserted a command after a jump command and before its destination, you would have to change the address in the jump. You can see that with lots of  jumps this would quickly become a nightmare. By labeling program lines and letting the assembler generate the “real” addresses, the problem disappears.

From assembler to high-level languages

Assembler made writing code in the language of the machine incredibly easier. It was a huge advance. But people quickly noticed two big problems. The solution to both problems was the second giant advance in software programming, high-level languages.

The first problem was that writing in assembler language, while worlds easier than machine language, still required lots of busy-work. For example, adding B and C together, dividing by 2 and storing the result in A should be pretty easy, right? In nearly any high-level language it looks something like this:

                A = (B+C)/2

Putting aside messy details like data type, this would be the following in pseudo-assembler language:

                Load B into Register 1

                Load C into Register 2

                Add Register 2 to Register 1

                Load 2 into Register 2

                Divide Register 1 by Register 2

                Store Register 1 into A

Yes, you can quickly become proficient in reading and writing assembler, but wouldn’t the 1 line version be easier to write and understand than the 6 line version?

Expressions were the core advance of high level languages. Expressions turned multi-line statement blocks that were tedious and error-prone into simple, common-sense things. Even better, the use of expressions didn’t just help calculations – it helped many things. While assembler language has the equivalent of conditional branches, many such conditionals also involve expressions, for example, the following easy-to-read IF statement with an expression

                If ((B+C)/2 > D) THEN … ELSE …

Would turn into many lines of assembler – tedious to write, taking effort to read.

The second problem emerged as more kinds of computers were produced, each with its own machine and assembler language. A guy might spend months writing something in assembler and then want to share with a friend at another place using a different type of computer. But the friend’s computer was different – it had a different assembler/machine language! The whole program would have to be re-written. Total bummer!

Wouldn’t it be nice to write a program in some language that was like the 1 line version above, and that could be translated to run on any machine??

Easier to write by a good 5X, easy to read and can run on any machine, present or future. Hmmmm… There must be something wrong, this sounds too good to be true.

The concept of high level languages was good and wasn’t too good to be true. Everyone agreed, and it wasn’t long before FORTRAN and similar compute-oriented languages arose. Even performance-obsessed data nerds wrote programs in FORTRAN rather than assembler because FORTRAN compiled into machine language was just as fast, and sometimes even faster because of optimizations those clever compiler guys started inventing! And the programs could run anywhere that had a FORTRAN compiler!

The accounting and business people looked at what was happening over in heavy-compute land and grew more and more jealous. They tried FORTRAN but it just wasn’t suitable for handling records and transactions, not to mention having nice support for financial data. So business processing languages got invented, and COBOL emerged as the best of the bunch.

If you look at a compute statement in FORTRAN and COBOL, they’re nearly identical. But in the early days, FORTRAN had no support for the money data type and no good way to representing blocks of financial data that are core to anything involving accounting.

Once they got going, people just kept on inventing new languages for all sorts of reasons. The obvious deficiencies of FORTRAN and COBOL were fixed. Languages could handle intense calculations and business data processing about as well. But back around 1970, 50 years ago, there remained a gap. That gap was that an important category of programs could not be written in any of the high-level languages. Essential programs like operating systems, device drivers, compilers and other “systems” software continued to be written in the assembler language of the machine for which it was intended. There were attempts to fill this gap, but they failed. Then a couple super-nerds at Bell Labs, building on an earlier good shot at solving the problem, invented the language C.

They rewrote the UNIX kernel in C, and the rest is history – C became the high-level language of choice for writing systems software and still holds that place.

Conclusion

Two giant advances were made in programming languages. Each of these happened early in computing history, in the 1950’s. The first giant advance was the biggest: with assembler language, machine language was reasonable to write, read and change. The second giant advance was the category of high level languages. Two minor variants on the concept, COBOL and FORTRAN, established the value of having a language that enabled expressions and could be compiled to run efficiently on any machine. In spite of the on-going tumult of language invention that has taken place since, the fact that huge bodies of working code written in those two languages continue to perform important jobs makes it clear that they embodied the giant advance that was high level languages. The only substantial omission in those languages – the ability to directly reference computer memory – was filled by the invention of the language C.

Most of what’s happened since in programming languages is little but sound and fury. For a review of those non-advances see this.

I do appreciate some of the subtle changes that have been made in language design. Some modern languages really are better – but mostly not because of the details of the language! I’ll treat this issue in a future post. Meanwhile, it’s important to understand history and appreciate the giant steps forward that high level languages as a category represent.

Software Programming Languages: 50 years of Progress

I was inspired to examine 50 years of progress in computer languages by an article in the Harvard alumni magazine written by one of the leading figures in Harvard computer activity, Harry Lewis. Lewis graduated from Harvard College the year I entered, and he stayed on for a graduate degree while I was a student there. He says:

Thirty veterans of Harvard’s Aiken Computation Lab reunited on January 19, 2020, some 50 years after each of us had a role in creating today’s networked, information-rich, artificially intelligent world. Rip van Winkles who had never fallen asleep, we gathered to make sense of what had evolved from our experience as undergraduates and Ph.D. candidates during the decade 1966-1975. One thing was clear: we hadn’t understood how the work we were doing would change the world.

I interacted with many of the people and projects he describes in the period he focuses on. For this post I decided to focus just on one of his major topics, programming languages.

50 years of progress

You might think a book would be required to adequately describe the advances that have been made in programming languages in the last 50 years. In fact, many experts seem to think so. Programming languages continue to advance and improve, even with everything that’s been achieved in the last 50 years – and more!

There is no doubt that many programming languages have been introduced in decades past. The pace of new language introduction does not seem to have slowed. But has progress been made? Yes – if you count dead-ends, bad ideas, going in circles and trivial variations as progress.

While describing the true progress in programming languages is easy, explaining why all the widely-touted “advances” aren’t in fact progress would indeed take a book. Which someone other than me is welcome to write. It would be boring! What could you expect from a gruesomely detailed exposition that amounts to Idea 1 is stupid because …; Idea 2 isn’t any better because …; idea 3 just another attempt with slightly varied syntax to go after the same fruitless goal as Idea 4, decades before …

I gave a talk that included this subject to a large group of programmers in India 25 years ago. Here is the slide illustrating the mighty progress of languages as illustrated by what COBOL calls a Compute statement:

I hope that gives you a basic idea of the massive advances that had been achieved by the mid 1990’s.

The same progress has continued unabated until the present. Google is promoting the language Go, often called Golang. Go must be one terrific language if the geniuses at Google are behind it, right? Here’s that same simple assignment statement I illustrated decades ago, updated to show how Google has made it even better:

A:=B*C+D

This illustrates the exquisite judgment expressed in new language design: The assignment statement has the colon before the equals sign, like PL/1, but leaves off the ending semicolon. Genius!

Uncommon common sense about programming languages

The inventors and promoters of new programming languages nearly always talk about how the wonderful new language is "better" than all other languages. Frequently mentioned are that the new language helps programmers be more productive and commit fewer errors.

In my experience and subjective judgment, all such claims are false and simply not proven out in practice. More important is the fact that the inventors/promoters of new language have no real arguments beyond their own enthusiasm to back their claims: there is NO evidence-backed science to support ANY of the claims that a given language is superior on some measure than others. There isn't even any substantial agreement on the appropriate measure of goodness, much less an evidence-based hierarchy of languages against the measure of goodness!

That should be the end of it. Once you realize how devoid of evidence, experimentation and science the whole endeavor is, it becomes clear that anyone promoting the virtues of a new language is either an ignorant buffoon or self-promoting cult leader. Or some of each.

Following are some observations to help put this into context.

I do understand how the enthusiasm for a new language can catch hold. Being a programmer requires a powerful set of blinders and the ability to focus on an incredibly narrow field for long periods of time. It's essential to being a good programmer! But you have to step back and look at your enthusiasm objectively. Above all you have to have objective measures to compare your wonderful new language to the bad old ones. This is almost never done in software, though it's commonplace in other fields. Would Marie Curie have been a better scientist had she used German instead of French as her primary language at work? See this for more.

There is a hierarchy of skills in software. Knowing a programming language is an important part of the hierarchy but it's near the bottom of the stack! Think about learning a new human language as an adult, ignoring for the moment the huge advantage you have being proficient in a language. You start with basic vocabulary and grammar. How long and hard is it to get the accent a native speaker has? Now think about vocabulary, the basic subset you've learned vs. the huge set of the native speaker. Think about using the basics of the language to get things done compared to using the language as an excellent writer or speaker. Now think about idiomatic expressions, of which any native speaker has a huge set -- without them, you're not fluent. Finally, think about a profession like law or medicine. Those proficient are using the base language, but with a vocabulary,, knowledge and sophistication that take years to learn. These things are like the subroutine library that is a key part of any language though not strictly speaking part of the language itself, and the professional specialty is like networking or database. Yes, the core language is a necessary foundation, but most of the real skill is in the rest. See this for more depth on this important subject.

It turns out that the "goodness" of a software effort isn't really dependent on the language used! When you dig into what really matters, it turns out that factors outside the strict confines of language design are overwhelming important, though details that new-language enthusiasts tend to ignore can play a role. See this for details.

One more key factor in the explosion of languages that are touted as advances but are really just a variation on the same old thing is the core human behavior of creating a language among people who work together on the same things for long periods of time. But no one has found that any of these languages are superior in some real way than others -- they're just the languages that happen to have evolved, each with some unique vocabulary reflecting unique concerns, like the words that people who live in the arctic have for different kinds of snow and ice. This is a well-known behavior, and why shouldn't it also be seen when people "speak" computer languages together? See this for examples and details.

When you lay out the spectrum of languages, you can see some patterns. The most obvious of the patterns I've observed is the extent to which data is defined as part of the language itself or outside the language proper. Data can be fully accessed and manipulated in all cases -- the difference is the extent to which the data definitions are made part of the language itself, or as part of a core library attached to the language. Language mavens are passionate about each of the variations, and convinced that theirs is best! See this for details.

What among other things enables large groups of people to hail the latest "advance" in computer languages without being laughed out of town is the shocking ignorance and lack of interest in computer and software history everyone involves exhibits. With the least bit of software history knowledge, no one would have the nerve to invent a new language, much less brag about it. See this for more.

What does this mean?

If you're a techie, you should get a grip, some perspective and knowledge of history and science. It will help you see, for example, that the fact that Marie Curie's native language was Polish and that she worked largely among French speakers was incidental to her great mind and achievements. It will help you see that Shakespeare wouldn't have been a much better playwright if only he had written in Latin (or whatever), or would have been much worse had he written in Norse (or whatever).

If you're a manager, you should respect the enthusiasm of your employees but understand the fact that their enthusiasm is baseless. I saw a software re-write into a new language using the Neo4j database. Everyone was pumped. Disaster. I have seen a couple cases in which huge enthusiasm was behind a less radical choice. The results were no better in any way than using a more standard language -- the engineers worked harder than they would have because of their enthusiasm, but were less productive because they were less skilled in the new environment. And of course, new hires were hampered with the same problem.

Conclusion

Computer Science is a profoundly unscientific field. The fashion-driven nature of it is exhibited starkly in the decades-long march of revolutionary new languages that keep getting invented -- and having no impact on anything that matters.

If you want to be the best possible programmer, it's incumbent on you to use a sensible, mainstream language and spend your effort producing excellent results -- including bringing the whole notion of excellence to new levels. The language you use will not be the limiting factor -- what's important is the way you use the language, just as it was for Marie Curie.

Computing in 1968: Gender, Cards, etc.

I grew up in Denville, a little town in northern New Jersey. I had a friend whose father knew someone who worked at what was then called EMSI, Esso Mathematics and Systems Inc., housed in a big complex on Park Ave. in Florham Park, NJ. I don’t think I even was interviewed, but I ended up with a job there the summer after graduating from the local public high school in 1968!

EMSI was a service company, one of over 100 companies in the Standard Oil of NJ group of companies. The purpose of the company was to apply advanced math and computer techniques to make the various Esso companies work better. The company had a large computer center on-site, a large room full of card keypunch machines, and extensive offices.

There was a strict division of work by sex. There were secretaries, all women. The men, all dressed in white shirts and ties, were the math and software design people. Software design was done using graph paper and plastic sheets that had flowchart cut-outs in it, so you could make uniform branch triangles, etc.  Here’s a picture of what it looked like (not mine, I never had or used one):

The programming was all done by women, who wrote lines of code onto coding paper, with lines and columns that matched up with Hollerith cards. This was supposed to be a mechanical affair, driven by the graphical designs. Of course it was more than that. When a sheet was ready, it would be sent for keypunching, all done by women using keypunch machines. I think we used the newer IBM 026 machines. The programmers, all women, would get the cards and sort them in a rigid box for storing and transporting to the machine room, where male machine operators would stack them into card readers for loading into the computer. Often there were errors, of course, so the machine would print the error listing on wide, 132 column paper using a mechanical printer. The paper had a string of square holes along each side that fit into gears in the printer. The printers were “line printers,” printing a single line of characters and numbers on the paper. Once the line was printed, the gears would advance the paper one line, and the next one would be printed. The pages were all joined end-to-end, fed to the printer from a big box sitting beneath it. When a print job was complete, the last page would be advanced out. An operator would tear the print jobs apart from each other and put them on a table for people to pick up, along with any related cards to the programmer. The printout could be a list of error messages from the compiler, maybe along with a printout of the program itself. You could ask for the assembler language version of the program to be printed along with the original source code in FORTRAN, which was important when debugging. If the program compiled properly, i.e., had no syntax errors, it would be run. It would either run to completion and print out the desired results, or it would crash, in which case a listing would be printed including a memory dump, which was a hexadecimal listing of the contents of the memory at and around the cause of the crash. I spent many totally absorbed hours poring over and deciphering memory dumps.

The physical embodiment of a program was an ordered set of cards. You would carry the cards for a program around in a cardboard box, all lined up in a neat stack. The box was as wide and high as a card, and at least a foot long, long enough to carry 2,000 cards.  Every once in a while someone carrying a box of cards would stumble and drop the box, scattering the cards on the floor. A fun time would then ensue, figuring out the right order of the cards and putting them back. This is one of the reasons that the last handful of columns in many card decks contained the sequential number of the card in the program, or set of data, or whatever; without such numbers, putting the deck back together would take even longer. I dropped a deck once. It’s the kind of thing that, once you did it, you took great care never to do again, quite apart from the people around you in the hallway laughing at you.

I don’t know how it happened, but I was the exception – a tall, skinny, male high school grad who was bright but entirely without qualifications. Everyone seemed amused by my existence. Perhaps that’s why I got away with things. It’s probably also why I kept getting asked to do things that cut through the otherwise strict divisions of labor. For example, I skipped the usual two-step of getting programs written, the design part by tie-wearing men using plastic guides to make neat-looking diagrams of program operation and flow, followed by women “translating” the design into actual lines of code. What a waste of time, I thought. Just work out what you wanted to do in your head, and then write the code. Before long I skipped the write-the-code-on-paper part, and just typed it into a vacant keypunch machine in the sea of lady keypunchers.

Looking back on the experience, I realize how lucky I was, and how unusual the experience I had working for a large division of a giant company. I now know that there must have been projects with goals and timelines that had to be met. People were judged based on whether they hit the pre-determined goals. While I was stuck into that environment, I was there as an un-planned-for extra resource who fit into none of the pre-defined categories. People threw me bits of work to keep me busy, without ever telling me how and when it had to be done. Each task involved something I had to learn, often a lot I had to learn, so I just tore into it. When I was done, I presented it to the person who gave me the work, much like a dog who dashed off to fetch a ball someone had thrown, perhaps into a lake. I imagine I looked as eager and pleased with myself as a dog, ready to dash after another ball. I never focused on the learning; I just drove towards the ball, overcoming obstacles (mostly things I didn’t know, sometimes the strict hierarchies and divisions of labor) as quickly as I could along the way. It was a formative experience, and one that was at odds with the normal group working experience. I knew I was lucky to have the job, and my appreciation for how lucky I was has grown over the years.

This is an edited excerpt from the draft of my Computer-related Memoir.

Luddites Smashed Looms then and Resist Automation today

The few people who are familiar with the term "luddite" think that a luddite is an unfortunate but stupid person who fights against advancements that make things better, while clinging bitterly to their crappy, low-end jobs. "Luddites" in this common view, are uneducated, progress-preventing people who need to be moved to the side so that society can be improved.

The reality is that, in most cases, luddites were highly skilled craftsmen performing difficult and challenging jobs. It's not that much different today. Luddites are often highly educated professionals and managers who are convinced they bring value to their complex jobs every day The reality is that there are luddites everywhere. Who wants to have their job disrupted? Who wants to be told that their expertise is no longer needed, and that a machine or software system can do a better job?

Stepping back

This notion of what software was REALLY about struck me quite early in my career. The thought was simple: the ultimate purpose of most of the software I wrote was to replace humans. The thought made me uncomfortable. But after a while, I connected the thought with all the rest of the mechanization and industrialization in society for hundreds of years. A new machine (or program) is valuable only to the extent that it somehow reduces the total amount of human labor to reach a given result, everything taken into account!

The replacement of humans by machines was extremely clear to the luddites, the secret society of workers whose jobs were being eliminated or reduced to unskilled labor by the increased use of Jacquard Looms. The movement was named after one of the first people to smash a loom, Ned Ludd:

The core of the movement was actions to destroy the job-killing looms that automated and de-skilled the worker's jobs:

This is particularly interesting and relevant to computers eliminating human labor. No one thinks of what could be the modern equivalent of workers smashing the looms that threaten their jobs. An ironic twist is that the revolutionary Jacquard looms were mechanical computers -- they executed a "program" that was encoded physically, enabling them to execute flawlessly elaborate patterns in the woven cloth.

Luddites today

Yes, there are luddites today. Lots of them. But they're not so crass as to pick up big sledge hammers and smash the looms that threaten their livelihoods. They're more subtle, and far more supported by elite society than the original luddites ever were. They're not even seen as resisting change -- they're seen as highly trained professionals that we're so grateful to have. Very much the way that the skilled craftsmen displaced by machines were seen by themselves and most of their contemporaries!

So who exactly are these modern luddites? They include many of the elite, highly-paid professions that ambitious young people strive for.

• Lawyers. A growing fraction of lawyers are under attack by automation. The pain is shown by declines in law school graduate hiring, the growth of firms such as Legalzoom, and growing outsourcing to low-wage locations. The trend has been growing for at least 20 years. I have personally seen the resistance of lawyers to automation efforts.
• Doctors. The ready-to-be-luddite feelings are strong here, though automation is still in its early stages. Doctors increasingly feel like they work in a non-stop assembly line, with patients questioning their statements because of Doctor Google. With insurance companies increasingly questioning and challenging their actions and clinical decision-tree software chomping at the bit, doctors can feel like targets. While hiring remains strong, automation efforts continue to gain strength.
• Software engineers. It seems only fair that the people who drive automation should have their jobs automated. This isn't in the news, but in fact there have been many waves of software job elimination -- masked by strong growth in newly emerging technologies. I have run into many programmers over the years who jobs were eliminated; some of them have left the field, and others slide into management positions, managing people doing jobs they barely understand. For example, during the 1970's there was an explosion of jobs building operating systems for minicomputers. Minicomputers are long gone, along with the diverse operating systems needing to be built. A long trend is skills degrading -- today you can build a functioning web site without programming, while years ago building software with a sophisticated user interface required serious software effort.
• Actors and musicians. We have more visual and musical entertainment available than at any time in human history. But the vast, vast majority of it is recorded and replayed! The vast majority of musicians, for example, were employed making adequate, best-available performances at local places. The number of such jobs is a tiny fraction of what it once was because of recordings. All the supporting jobs are greatly reduced as well. Even jobs in movie theaters have cratered, since people watch on personal and home devices.

What will become of all those people and jobs? The answer has been the same for hundreds of years: there will be great pain for the individuals involved, but overall, the wealth of the population will grow. A clear example is agriculture. At the time of the American revolution, over 90% of the population was involved in agriculture. One step at a time, those jobs were automated, so that in 2019, only about 1% of the US population was employed in agriculture. Yet there's food enough for everyone, and jobs that didn't exist back then.

Conclusion

As a person who has been directly involved in automation for over 50 years, I have seen the jobs I've had and the skills I've acquired become obsolete over and over again. I've also seen or been part of eliminating through automation many jobs, a surprising number of them highly skilled jobs. An early one was the capable engineers who ran the oil refinery in Venezuela in the late 1960's whose jobs were eliminated by the software I worked on at the time -- the software just did a better job than the engineers could possible do! I describe this here.

Luddites can and often do postpone their replacement by automation. But they lose in the end. I don't see an end to this process any time soon.

Laser Disks and Workflow Illustrate the Insane, Fashion-Driven Nature of Software Evolution

Computers are so exact and numbers-based that we tend to think of the people in charge of them as white-jacketed scientists with esoteric knowledge driving towards calculated optimal results, much the same way we imagine that the scientists and engineers calculated the path of the Apollo capsule to the Moon. If the Apollo program were run the way most computer projects are, it would have been a miracle if the capsule made it off the launch pad, much less traced an incredibly exact path from Cape Canaveral to the chosen landing spot on the Moon, a journey of over a quarter million miles.

The reality is that the application of computers and software to automation is painfully slow, usually failing to achieve optimal results by large margins, and often lagging behind what is achievable by decades. The reasons appear to be a semi-random mixture of commercial interests, technology fashion trends, ignorant and risk-averse managers, and technical specialists rooted in the past and wearing blinders. That’s all!

Here’s the story of a typical example.

Microfilm, document imaging and laser disks

Long before computers, microfilm was the preferred medium of librarians and archivists to preserve paper documents for future use. Microfilm was more compact than paper, and lasted much longer. What happened when computers came along is a curious and educational story, a prime example of how computer use and software evolve.

Disclosure: I lived and worked as a participant in this history during the late 1980’s and early 1990’s. The story I will tell is not systematic or comprehensive, but like a traveler’s tale of what he did and saw.

A side show in the long evolution of computer storage is the laser disk. A laser disk has data recorded onto it either as a whole, during manufacturing, or incrementally, as part of a computer system. The laser disk had quite a run in the consumer market in the form of CD’s, and then DVD’s. They were an improvement on existing methods for distributing digital content of all kinds.

In the computer industry, they took the form of WORM (write-once-read-many) drives, and were used to record archival data that should not be updated, but could be read as often as needed. The technology, as often happens, went searching for problems it could solve. In an effort to reduce human labor more than microfilm could, jukeboxes were invented. Each jukebox had a disk reader and many shelves for disks, with a mechanism to load a selected disk into the reader. FileNet, now part of IBM, built one of the first of these systems in the mid-1980’s.

At around the same time that WORM drives became practical, paper document scanners were evolved from systems to capture images of the paper onto microfilm. In each case, light was shown onto paper and focused onto a target; a scanner replaced the film with an optical sensing device.

How could we get people to buy WORM drives and jukeboxes, the creative marketing people wondered? The departments that put documents onto microfilm weren’t going for it – the new systems were much more expensive, microfilm wasn’t accessed often enough to make it worth connecting it to computers, and the records retention people were understandably skeptical that any computer disk would be readable 100 years from now, as microfilm will be.

Adding workflow to document imaging

I have no idea who made the audacious leap, but some creative person/group came up with the idea of doing computer document scanning at the start of the process, when paper arrived at a location for processing, instead of microfilmed at the end for archival purposes. But WHY??? The creative types scratched their heads until they were losing hair rapidly. Why would anyone make responding to loan requests or whatever even longer than it is today by introducing the step of scanning?

GENIUS TIME: LET’S INVENT “WORKFLOW”!!!

Everyone has an image for what “workflow” is. For documents, it’s the path of processing steps from arrival to filing. In larger organizations, there are often multiple departments involved in processing a document, so there are in-boxes and out-boxes and clerks transporting documents from one to the other.

Rubbing their hands and cackling, the scammers worked out their strategy: We’ll scan the documents and convert them to images so that we can move images from place to place electronically instead of piles of paper! We’ll call it “document imaging workflow.” It will deliver massive benefits to the whole organization, to everyone who touches the paper, not just to the tiny group who files and archives at the end – and storing the scanned documents onto WORM drives will do their job too! Hooray! We will leap-frog the old-fashioned, paper-flooded back office into the modern electronic age. What’s not to like?

It was audacious and brilliant. It was a scam that would wither if confronted with just a little common sense or cost accounting. Which, true to the pattern of such fashion-driven trends, it never had to confront!

Implementing workflow

A typical target environment for implementing workflow was large offices with large numbers of documents, often forms of some kind, arriving every day. The forms would go from the mail room to the relevant people’s desks for processing. The person handling the form would often have an IBM 3270 terminal for interacting with a mainframe computer. When the person was done with the form, they would place it in one or more outboxes for further processing at other desks, or for filing. Sometimes documents would go into file cabinets for a period of time, but all would end up being archived, and sometimes microfilmed for permanent storage.

Implementing workflow first of all required scanning the documents as a first step, and storing them immediately on normal computer disks, and ultimately on WORM drives, thought to be the equivalent of microfilm. Once scanned, the documents would be managed by workflow software, which would route them to the appropriate desk in the appropriate department for processing. Every desk needed to have the old computer terminal replaced or augmented with a large, high-resolution image terminal, capable of handling at least the display of the document, and sometimes also the computer screen. The old, slow connections between terminals and mainframe needed to be replaced with a fast local area network, and there needed to be substantial servers for handling the local image handling and workflow routing.

Of course, lots of analysis and programming was involved in addition to the equipment. Workflows had to be analyzed and programmed, and all the management, reporting and exception handling taken care of.

The benefits of workflow

When a movement like document imaging workflow has a head of steam up, no one seems to ask whether it’s a good idea, and if so, how good of a good idea it is – quantitatively. When I was involved, everyone threw around there would be at least a 30% productivity improvement due to workflow, and that would make all the expense and disruption worth it. I never encountered a single study that measured this.

Think about it for a minute. Would looking at an image of a document make you work faster than you would looking at the original paper? Would picking the next paper from the in-box be slower than getting the system to show you the next image? What about the labor of the clerks moving stacks of paper from one person’s outbox to the next one’s inbox? It would certainly be saved, but chances are a single clerk could handle the paper movement for many dozens of people. And remember, there’s the scanning and indexing that’s been added and the massive computer and software infrastructure that has to be justified.

It’s obvious why no one EVER did a cost-benefit analysis of workflow – the back-of-the-envelope version easily shows that it’s a couple zeros away from making sense.

I remember a couple of out-of-it, tech-fashion-disaster know-nothings mildly thinking out loud about how workflow could possible be worth it financially. The immediate response from workflow fashion leaders was upping the ante – don’t you know, the wonderful workflow tool from (for example) FileNet lets you program all sorts of efficiencies for each stage of work; it’s something we really need to dive into once we get the basic thing installed. End of subject!

So who fell for this stuff? Just a who’s-who list of major organizations. Each one that fell for it increased the pressure for the rest to go for it. None of them did the numbers – they just knew that they couldn’t fall too far behind their peers.

Conclusion

It’s hard to think of a field that’s more precise and numbers-oriented than computers. Who has a clue what actually goes on inside those darn things? And the software? There are millions of lines of code; if a single character in any of those lines is wrong, the whole thing can break The impression nearly everyone has, understandably, is of a field that is impossibly precise and unforgiving of error. When the software experts in an organization say that some new technology should be adopted, sensible people just agree, particularly when there’s lots of noise and other major organizations are doing it. It can be even more gratifying to get known as pioneering, as one of the first organizations to jump on an emerging tech trend!

Somehow, the smoke and noise from the software battlefield is so intense that the reality is rarely seen or understood. The reality, as I’ve illustrated here, should result in everyone involved being sent to the blackboard by a stern teacher, and being made to write, over and over: “I pledge to try harder to rise above the level of rank stupidity next time.”

The Evolution of Software

There is strong interest in the latest developments in software. No one wants to be left behind.

At the same time, there is a peculiar (to me) lack of interest in whatever the latest thing grew out of or evolved from. What are the new thing's predecessors? What did that thing grow out of. Have similar things appeared in the past? Are there patterns we can observe here, or are the new software things that explode onto the scene just a meaningless sequence of random events? Are they things that we obsess about while they're here without questioning or wondering where they came from, without asking what else they may resemble in some way? Are they things we just forget about and ignore once they depart the stage of software fads?

The questions have obvious answers. Answers that no one seems to be interested in.

So long as we fail to ask these crucial questions, we will remain ignorant of software in deep ways, and will continue to stumble from fad to fad without improvement or understanding. History is important!

Evolution in Science

If you study different sciences and their history, you notice that, at some point, scientists begin to pay attention to evolution, i.e., change over time of the thing being studied.

The most obvious thing is evolution of an individual item over time. In biology, you have the cycle from birth to death. In geology, you have the change of mountains, rivers, beaches and glaciers. In materials science you have the change of objects as they are subjected to various conditions over time, for example the way that iron rusts.

Evolutionary studies reach a breaking point and get really interesting when you study groups and types and see the patterns of change. In biology, this was the revolution that happened with Darwin and his study of the evolution of species. Something similar happened in geology with plate tectonics.

When you put the patterns together, things get totally amazing and transformative. While no longer fashionable to talk about and study, the parallels between the growth stages of an individual animal and the evolution of species are obvious, though not literal at a detail level.

Studying evolution is an important stage in the evolution of most sciences!

It's long-since past time to study software evolution, as an integral part of software science.

Software Evolution

Software evolves at every level.

At a basic level, software languages evolve. I have given an introduction to this subject here. The evolution of software languages resembles biological evolution in remarkable ways, as species (languages) emerge and evolve in response to new conditions, including other species. Similarly to biology, languages evolve; sometimes many species descend from them; and sometimes a species goes extinct. See this snapshot, for example:

(Credit)

Consistent with the determination of nearly everyone involved with software to ignore its history, this chart and anything like it are ignored, and the evolutionary principles behind it are not studied. When you start to study them, you learn amazing things. Suddenly some of the random change in the world of software languages starts to make sense.

Software languages are the very most basic things about software. Just as interesting and vastly more important are the programs written in software languages. Everyone involved in writing or maintaining a software program (application) just thinks about that program. It definitely occurs to business people what the program can do compared to its immediate competitors, if any. Yes, there is a kind of "survival of the fittest" in software evolution.

What about the "insides" of the program compared to other program insides? Sometimes people are aware of gross differences, like the language the program is written in. But the curiosity normally stops there.

The program "insides" have huge practical consequences. Depending on the details of how a program is written, it can be more or less:

• reliable/buggy
• easy to hack (secure)
• expensive to run/operate
• dangerous/costly to change
• able to respond quickly to changing loads
• speedy or slow

People commonly talk about structure or architecture, but those things are just the tip of the iceberg.

When you dig in, you find the equivalent of islands and continents of software. I've treated this subject here, for example. For example, you may have spent some number of years after getting your CS degree working on website design, and interacted with a community of people similarly engaged. Not everyone uses the same tools or does things the same way, of course, and you're likely to think there's quite a variety of approaches.

Then you go wild, dive into hospital management systems, and your mind is blown. What you thought was the wide variety of software and tools used by website designers turns out to be a whole continent apart from what you find in hospital automation. You learn that the flagship product of the company that has more than a third of the market (Epic) is written in a language you've never even heard of! It's like growing up in New Jersey among what you think is a very wide variety of people and going to Chennai -- a huge city in India you've never heard of, whose population is larger than your whole state, where the main language is one you've never heard of, written in a script you've never before seen.

Even more ignored are the obvious (to those who trouble to look) repeating patterns that can be observed in applications as they are first written and then evolve over time.

The Study of Software Evolution

To put it bluntly, the study of software evolution has barely begun. A few isolated souls, hardy or foolhardy as you like, have dipped their toes into the deep waters of software evolution. Those few have found worlds to explore, oceans whose depths have yet to be plumbed.

This is particularly the case because software is a "the only reality is the present" kind of field, as it now stands. It doesn't have to be this way!

Very few people study the history of math. They don't need to: anyone who learns math starts with math as it existed thousands of years ago. If they do well in math, they get all the way up to ... the late 1600's, when Newton and/or Leibnitz discovered calculus! Each step in math is based on and often depends on the earlier steps.

In some sense, this is also true of software. Except not really.

First, knowing nothing about software, you can dive right into the latest language and tools and use them to make something that works. Anyone think you could do that with differential equations, I mean without already knowing how to count, add and multiply? The difference is that in software, the "accomplished" new-comer really is standing on generations of older software ... of which s/he is blissfully unaware! The new programmer, of course, uses an incredible array of operating systems, drivers, networking, file and database systems, and on and on. These form the "land" on which the new programmer "stands" to work. None of the wonderful new stuff would work without it. The big difference, of course, is that using the underlying software and understanding the underlying software are two rather different things. This makes software totally different from not just math, but most of the other legitimate sciences (leaving out the fake ones like political "science" and the rest).

Second, when you're building a program, chances are excellent that you're not the first person to try building something very much like that program. Lots of details may be different, but even the details are probably not as unique as you might think. Suppose you need a program that sets up an account for a person, and then when the person does stuff, the details are checked with the person's account, and if OK are added to the account, and if not are rejected but recorded. This high-level description applies to a sales operation with customers, a medical office with patients, a manufacturing company with customers, and a bank. Given that programs like this have been built tens of thousands of times, don't you think that the ways of building programs like this would have evolved? That there would be pre-built parts, specialized tools for using the parts, building new ones and gluing them together? Special methods that have evolved and been refined to make sure it's done in the optimal way? Did any of you with any level of degree in Computer Science learn any of that -- other than, of course, some boiler-plate about the virtues of object-oriented languages and maybe some other fashionable stuff? Of course not! Anyone with any sense of fashion and/or status knows that these are not career-advancing subjects. Anything involving software history or comparative study of software projects is career-killing.

Conclusion

Software evolves. But it evolves differently than other things studied by science. Virtually no one studies software history in any way, much less the patterns of evolution that become apparent when you study that history. There have been a couple of attempts to learn why software doesn't get better in the way many other things do, most notably Fred Brooks with his book The Mythical Man-Month and other writings. I have dissected this flawed analysis in my work on Software Project Management and Wartime Software. See this for a start.

I have worked for years trying to identify some notable patterns in software evolution, and will be releasing some of that work in the coming months.

Recurring Software Fashion Nightmares

Computer software is plagued by nightmares. The nightmares vary.

Sometimes they are fundamentally sound ideas that are pursued in the wrong way, in the wrong circumstances or at the wrong time. Therefore they fail – but usually come back, sometimes with a different name or emphasis.

Sometimes they’re just plain bad ideas, but sound good and are cleverly promoted, and sound like they may be relevant to solving widely acknowledged problems. Except they just don’t work. Sometimes these fundamentally bad ideas resurface, sometimes with a new name.

Sometimes they’re a good idea for an extremely narrow problem that is wildly applied beyond its area of applicability.

The worst nightmares are the ones that slowly evolve, take hold, and become widely accepted as part of what “good programmers” believe and do. Some of these even become part of what is ludicrously called “computer science.” Foundation stones such as these, accepted and taught without proof, evidence or even serious analysis, make it clear to any objective observer that computer “science” is anything but scientific, and that computer “engineering” is a joke. If the bridges and buildings designed by mechanical engineers collapsed with anything close to the frequency that software systems malfunction, the bums and frauds in charge would have long since been thrown out. But since bad software that breaks is so widespread, and since after all it “merely” causes endless delays and trouble, but doesn’t dump cars and trucks into the river, people just accept it as how things are. Sad.

Following is a brief review of sample nightmares in each of these categories. Some of what I describe as nightmares are fervently believed by many people. Some have staked their professional lives on them. I doubt any of the true adherents will be moved by my descriptions – why should they be? It was never about evidence or proof for the faithful to start with, so why should things change now?

Sound Ideas – but not here and now

• Machine learning, AI, OR methods, most analytics.
  • These things are wonderful. I love them. When properly applied in the right way by competent people against the right problem at the right time, amazing things can be accomplished. But those simple-sounding conditions are rarely met, and as a result, most efforts to apply these amazing techniques fail. See this.
    

Bad ideas cleverly promoted

• Microservices.
  • Microservices are currently what modern, transformative CTO’s shove down the throats of their innocent organizations, promising great things, including wonderful productivity and an end to that horror of horrors, the “monolithic code base.” While rarely admitted, this is little but a re-incarnation of the joke of Extreme Programming from a couple decades ago, with slightly modified rhetoric. A wide variety of virtues are supposed to come from micro-services, above all scalability and productivity, but it’s all little but arm-waving. If software valued evidence even a little, micro-services would be widely accepted as the bad joke it is.
• Big Data
  • There's nothing wrong in principle with collecting data and analyzing it. But in practice, the whole big data effort is little but an attempt to apply inapplicable methods to random collections of data, hoping to achieving generic but unspecified benefits. See here.
    

Great ideas for a narrow problem

• Blockchain.
  • My favorite example of an excellent solution to a problem that has been applied way beyond its area of legitimate application is blockchain. Bitcoin was a brilliant solution to the problem of having a currency that works with no one in charge. How do you get people to “guard the vault” and not turn into crooks? The way the mining is designed with its incentives and distributed consensus scheme brilliantly solves the problem. However, the second you start having loads of crypto-currencies, things get weaker. Once you introduce “smart contracts,” there’s a fly in the ointment. Take away the currency and make it blockchain and you’ve got a real disaster. Then “improve” it and make it private and you’ve got yourself a full-scale contradiction in terms that is spectacularly useless. See here.

Bad ideas that are part of the Computer Science and Engineering canon

• Object-orientation.
  • Languages can have varying ways of expressed data structures and relating to them. A particular variant on this issue called “object-orientation” emerged decades ago. After enjoying a heyday of evangelism during the first internet bubble, O-O languages are taught nearly universally in academia to the exclusion of all others, and adherence to the O-O faith is widely taken to be a sign of how serious a CS person you are. For some strange reason, no one seems to mention the abject failure of O-O databases. And no one talks about serious opposing views, such as that of Linus Torvalds, the creator/leader of the linux open source movement, which powers 90% of the world's web servers. Programmers who value results have long since moved on, but academia and industry as a whole remains committed to the false god of O-O-ism.
• Most project management methods.
  • I’ve written a great deal about this, including a book. Whether it’s the modern fashion of Agile with its various add-ons such as SCRUM, project management methods, ripped from other disciplines and applied mindlessly to software programming, have been an unmitigated disaster. The response? It ranges from “you’re not doing it right,” to “you need to use cool new method X.” Yet, project management is a profession, with certifications and course on it taught in otherwise respectable schools. A true nightmare. See these.
    
• Most software QA methods.
  • All software professionals accept what they’ve been taught, which is that some kind of software automation is an essential part of building software that works and keeping it working. Except the methods are horrible, wasteful, full of holes and rarely make things much better. See this.
    

Conclusion

I make no claim that the list of items I’ve discussed here is comprehensive. There are things I could have included that I have left out. But this is a start. Furthermore, if just half the items I’ve listed were tossed and replaced with things that actually, you know, worked, much better software would be produced more quickly than it is today. I don’t call for perfection – but some progress would sure be nice!

Social Media Has a Long History

It seems like the whole world is in an uproar about social media, with frequent revelations of awfulness and malfeasance. The uproar is about social media such as Facebook, Twitter and the rest. The trouble is these issues have existed in one form or another in social media going back ... hundreds of years. What we are seeing is ignorance of the present combined with ignorance of the past. In other words, business as usual.

The Core Drivers of Social Media

Most people care about their place in society -- their status. They care about how they're perceived, who knows who they are and how others relate to them. While this drive seems to come to a kind of perverse peak in middle and high school, it persists for most peoples' lives.

Intimately related to caring what others think is the drive to express what you think and what you've done. While this is related to influencing others' thoughts, it seems to be a kind of innate drive as well.

In short, you want to tell people what you've done, what you think, and you want to hear about other people, particularly those you somehow are involved with or even just similar to in some way.

Closely related to this are the core concepts of status and fashion. It's a basic urge to want to see your status reflected publicly if it's high, and many people have strong interest in what high-status people do and how they do it. Particularly as fashions of various kinds wax and wane, from clothing to activity to speech, people who want to increase their status have an intense interest in learning what the new things are.

Key Characteristics of Social Media

What makes something Social media vs. some random other kind of media? It's pretty simple: social media mostly consists of media (words and pictures) that is about and either written/created by the person or sourced from that person. It's about what a person says, thinks or does.

Now let's get to the other key characteristic: money. Who pays for social media? After all, it costs quite a bit to produce it, and that money has to come from somewhere. Historically, the people who consume the media pay a little, while ... get ready ... advertisers pay a lot. Today, the incremental cost of delivering social media to the person who consumes it is so little that no one bothers to charge for it -- the whole cost is borne by advertisers.

Social media is an amazing phenomenon, deeply rooted in human drives and emotions. People produce the content for it for free -- they are glad to have things about themselves distributed at little effort of their own to those they may like to know about it. And they read about themselves and people to whom they are socially connected with, paying to do so if necessary. They can't help but knowing that the ads that are intermixed with the "content" are going a long way (in the electronic world, all the way) to paying for their reading pleasure, but it rarely bothers them. They also know that ads are targeted to particular groups of readers. It makes common sense, after all. No big deal -- if I were an advertiser, of course I'd want to show my ads to people who are likely to buy what I'm selling!

Earlier versions of Social Media

People like to imagine that social media are strictly electronics-age things. Mark Zuckerberg invented it, didn't he? No, sorry! Social media have been around for a looooong time. I could go all the way to ancient Sumer and Egypt, but I think the point will be clear enough with more recent examples.

Here is a notice in the Pittston Pa Gazette from 1928 about a function attended by my grandmother, Agnes Black:

Here is an ad that helped pay for that information to be printed:

More recently, here is a notice in the same paper from 1955 about a visit made by my parents:

Here is one of the ads that helped pay for the notice.

The notice was actually fake news! My parents visited with their two children, David Bruce Black and Douglas John Black -- not David and Bruce. The advertisers don't care a whit -- they just want the eyeballs to persuade them to buy some hot new technology:

I could give examples from many other places and centuries. Things have evolved, but since the principles are rooted in human in human nature, not as much has changed in principle as you might think.

Conclusion

Everything is about people. A great deal about people is relationships and status. The experiences we had in middle school and high school didn't disappear into nothingness. They just evolved as each of us entered new groups of people, each with its own pecking order and rules for engagement. One of the most ironic things about modern social media is that certain groups of people are really upset about what gets published, and want to make sure that only the "truth" is published. They, of course, want to be in charge of defining what "truth" is. Sorry, guys, in the world of social relations and much else, "truth" is nothing but a pretty veneer on top of raw power. Yes, your grace, your honor. Why should it be different now that we're staring into little screens and swiping while we walk?

 

Uber’s Evolution from the inside

Uber is a great success; you can tell because competition has sprung from all corners, and scads of new companies say they’re going to be “the Uber of X.” Along with great success comes great myths, which inform a set of assumptions about how Uber got to be Uber. The myths are all derivatives of the broadly accepted narrative of how tech companies start and grow. In every case I’ve examined, fact-based history contradicts these myths and explodes what “everyone knows” about Uber and every other tech company. I’ve explored this theme in a couple of cases, and I’ve just had the opportunity to question an early Uber driver, whose story provides fascinating insight on just a small part of what really happened.

Recruiting the first drivers

My driver was one of the first handful of drivers in the New York City area. Uber is a network-effect company, i.e., customers need access to a large pool of drivers, while drivers need business from a large pool of customers. How do you get that started? Of course I don’t know the whole story, but I did hear how my driver informant was recruited.

He was sitting in his car near Madison Square Park in Manhattan with a half hour to go before his ride would appear. Someone knocked on his window. He rolled it down, and it was two people who wanted to talk with him about a new way to get rides. It started to rain, and he invited the people to sit in his car, for which they were very grateful. He ended up signing up for their service, encouraged by a $1,000 signing bonus, and further encouraged by being paid $35 an hour just for keeping the app on his phone. He was then paid $1,000 for each additional driver he referred who signed up, and a further $1,000 when that driver took their first ride. That’s how a critical mass of drivers was recruited even when there was little work for them to do.

Here’s the part of the story I found most impressive: the two people who recruited him were … the head of Uber NYC and Travis Kalanich, the co-founder and CEO of Uber.
Yes, the big boss himself, personally walking the streets of NYC and hitting on strangers. Wow. For every 1,000 bosses who puts up one of those stupid pyramid-on-its-head charts
during a company talk to say how he’s just there to serve the important people on the front lines, there might be a couple like Mr. Kalanich who act on it and get into the trenches themselves.

Recruiting the first customers

Uber today is an easy way to use an app to call for an on-demand car service. It’s all “get me a ride now. As in, now.” At the start, it wasn’t that way at all! It was mostly for really long rides, like from NYC to Boston or Philadelphia, and the rides were often round trips with a big wait in the middle. This is a tiny minority of all rides, but one that was typically arranged way in advance by phone. With Uber, you could get someone to pick you up in half an hour or less, which was great compared to the alternatives. It also meant that having just dozens of drivers available was OK.

It was only as the population of drivers and customers grew that shorter trips with shorter lead times made sense – and the long-trip buyers were an ideal starting population as customers for that, so it evolved naturally.

Evolution of the service

Here are some tidbits about how the service has evolved.

At the beginning, drivers were screened thoroughly. For example, every candidate was shown a map of Manhattan with no street names. A pin was placed on the map, and the driver had to name the streets. If you didn’t get them, you were out. The screening has gone way down over time.

The cut taken by Uber has ratcheted up sharply. It started at just 10%, moved in a few steps and is now up to 38%. A huge difference.

As the number of drivers and customers grew, Uber provided a “heat map” in the driver’s app to show where drivers were needed and where there was an excess, which helped drivers position their cars to best effect. A subsequent version of the app dropped this feature and replaced it with congestion pricing. My driver misses it, and thinks a new version is coming out with the heat map restored.

The policies and standards for drivers have evolved quite a bit, something which is also visible to customers with the addition of Uber-X and evaluations of drivers – I learned that drivers can also evaluate customers! My informant says the evaluations are on the whole helpful – one incident reminded him he always has to inspect the passenger seat when someone leaves, for example, something he had been neglecting.

In spite of all the changes, my driver/informant loves Uber. The main reason is he has control. He can start and stop any time. He can work more and make more money, or take time off any time he needs to, without asking anyone’s permission.

Observations

It appears that Uber did a very smart thing I have seen other companies do: pick a sub-market in which they could get traction, and then evolve into the actual, larger market they probably had in mind all along. The sub-market enabled them to get to a critical mass of drivers and riders with less friction.

It's easy to focus on Uber as enabled by an app, or Uber as a marketplace. Those things are true. I think of them as being good choices (the app) and reasonable thoughts (marketplace), but somehow not essential.

For the consumer, Uber provides the convenience of hailing a cab (no advance planning, scheduling, committing) with cars that you can't see. It would work just as well if you used your phone to dial a number, gave your location, and then were told how long you'd have to wait for the driver. An on-demand car service.

For the driver, Uber provides control over time and money, as my informant made clear. When you drive a cab, you sign up for a shift. With a car service, someone schedules you. With Uber, the driver can opt in and opt out whenever it meets his needs, whatever those may be. Whether he got rides from an app or from someone calling him, the result would be the same.

Conclusion

Every company has a different story. Uber is no different. Even a narrow and limited glimpse into the factual history of Uber reminds us that every company ends up having a myth-based story that explains its success, and that if you’re going to learn something useful from their success, you have to find and pay attention to the facts. AND, the facts of similar companies that failed.

Innovations that aren't: data definitions inside or outside the program

There are decades-long trends and cycles in software that explain a great deal of what goes on in software innovation – including repeated “innovations” that are actually more like “in-old-vations,” and repeated eruptions of re-cycled “innovations” that are actually retreating to bad old ways of doing things. Of the many examples illustrating these trends and cycles, I’ll focus on one of the most persistent and amusing: the cycle of the relationship between data storage definitions and program definitions. This particular cycle started in the 1950’s, and is still going strong in 2015!

The relationship between data in programs and data in storage

Data is found in two basic places in the world of computing: in “persistent storage” (in a database and/or on disk), where it’s mostly “at rest;” and in “memory” (generally in RAM, the same place where the program is, in labelled variables for use in program statements), where it’s “in use.” The world of programming has gone around and around about what is the best relationship between these two things.

The data-program relationship spectrum

At one end of the spectrum, there is a single definition that serves both purposes. The programming language defines the data generically, and has commands to manipulate the data, and other commands to get it from storage and put it back.
At the other end of the spectrum, there is a set of software used to define and manipulate persistent storage (for example a DBMS with its DDL and DML), and a portion of the programming language used for defining “local” variables and collections of variables (called various things, including “objects” and “structures.” At this far end of the spectrum, there are a variety of means used to define the relationship between the two types of data (in-storage and in-program) and how data is moved from one place to the other.

For convenience, I’m going to call the end of the spectrum in which persistent data and data-in-program are as far as possible from each other the “left” end of the spectrum. The “right” end of the spectrum is the one in which data for both purposes is defined in a unified way.

People at the left end of the spectrum tend to be passionate about their choice and driven by a sense of ideological purity. People at the right end of the spectrum tend to be practical and results-oriented, focused on delivering end-user results.

Recent example: Ruby and RAILS

The object-oriented movement tends, for the most part, to be at the left end of the spectrum. Object-oriented people focus on classes and objects, which are entirely in-program representations of data. The problem of “persisting” those objects is an annoying detail that the imperfections of current computing environments make necessary. They solve this problem by various means; one of the most popular solutions is using an ORM (object-relational mapper), which does the work of moving objects between a DBMS and where they “should” be nearly automatic.  It also can automate the process of creating effective but really ugly schemas for the data in the DBMS.

A nice new object-oriented language, Ruby, appeared in 1995. It was invented to bring a level of O-O purity to shell scripting that alternatives like PERL lacked. About 10 years later, a guy was working in Ruby for Basecamp and realized that the framework he'd created for it was really valuable: it enabled him to build and modify things his company needed really quickly. He released it to open source and became known as the RAILS framework for Ruby, the result being Ruby on RAILS. Ruby on RAILS was quickly adopted by results-oriented developers, and is the tool used by more than half a million web sites. While "pure" Ruby resides firmly at the left end of the spectrum, the main characteristic of the RAILS framework is that you define variables that serve for both program access and storage, putting it near the right end of the spectrum.

Rhetoric of the ends of the spectrum

Left-end believers tend to think of people at the other end as rank amateurs. Left-enders tend to claim the cloak of computer science for their style of work, and claim that they're serious professionals. They stress the importance of having separation of concerns and layers in software.

Right-end practitioners tend to think of people at the other end as purists, ivory-tower idealists who put theory ahead of practice, and who put abstract concerns ahead of practical results. They stress the importance of achieving business results with high productivity and quick turn-around.

It goes in cycles!

There have always been examples of programming at both ends of the spectrum; neither end disappears. But what happens is that the pendulum tends to swing from one end to the other, without end up to now.

In 1958, Algol was invented by a committee of computer scientists. Algol 60 was purely algorithmic -- it only dealt with data in memory, and didn't concern itself with getting data or putting it back. Its backers liked its "syntactic and semantic purity." It was clearly left-end oriented. But then practical business people, in part building on earlier efforts and in part inspired by the need to deliver results, agreed on COBOL 60, which fully incorporated data definitions that were used both for interacting with storage and for performing operations. It was clearly right-end oriented. All the big computer manufacturers, anxious to sell machines to business users, built COBOL compilers. Meanwhile, Algol became the choice for expressing algorithms in academic journals and text books.

There was a explosion of interest in right-end languages during the heyday of minicomputers, with languages like PICK growing in use, and niche languages like MUMPS having their day. The rise of the DBMS presented new problems and opportunities. After a period of purism, in which programmers struggled with using left-end languages that were supposed to be "easy" like BASIC to get jobs done, along came Powersoft's Powerbuilder, which dominated client-server computing because of the productivity win of integrating the DBMS schema with the language. The pendulum swung back with the rise of the internet and the emergence of "pure" java, with its arms-length relationship to persistence, putting it firmly in the left-end camp. Since Java wasn't pure enough for some people, "pure" Ruby got out there and grows -- and some Ruby user is under pressure to deliver results and invents RAILS, which then spreads to similarly-minded programmers. But "amateurs" who like Java but also like productivity invented and are spreading Grails, a derivative of Java and RAILS that combines advantages of each.

So is RAILS an innovation? Java? I don't think so. They're just minor variants of a recurring pattern in programming language design, fueled by the intellectual and emotional habits of groups of programmers, some of whom push towards the left end of the spectrum (with disdain for the amateurs) and others of whom push towards the right end of the spectrum (with disdain for the purists).

Conclusion

This spectrum of the relation between data and program has persisted for more than 50 years, and shows no signs of weakening. Every person or group who, finding themselves in an environment which emphasizes the end of the spectrum they find distasteful, tries to get to the end they like. Sometimes they write a bunch of code, and everyone thinks they're innovative. Except it would be more accurate to say they're "inno-old-ive." This is one of the many patterns to be found in software history, patterns that account for so much of the detail that is otherwise hard to explain.

Don't Know Much about History

Software people generally know very little about software history, and that's OK with them. It's too bad. There's a lot to learn from software history. It can help you now!

Wonderful World

In 1960, Sam Cooke released a single called "Wonderful World."

Here are some of the lyrics:

I sure hope you can win that girl or boy you're after in spite of all that not-knowing!

The Wonderful World of History

Politicians study history in general and the last election in particular. Fiction writers frequently read fiction, current and historic. Generals study old battles for their lessons; even today at West Point, they read about the Civil War. Learning physics is like going through the history of physics, from Galileo and Newton and through Planck and Einstein to the present. Even the terms used in physics remind you of its history: hertz, joules and Brownian motion. Math is the same way. Whatever you're learning was first established at some point in history, and remains as valid and applicable to the present as when first discovered.

Software, by contrast, is almost completely a-historical. Not only are most people involved uninterested in what happened ten years ago, even the last project is unworthy of consideration – it’s “history.”

History isn't just for historians

How did we learn about biological evolution? By observing species and trying to figure out their history. How did we learn about genes and DNA? By trying to figure out the mechanisms that make organisms work through time. Geology? Gee, I wonder how those mountains got there? And what happened so that I'm finding fossils of creatures that lived in the ocean up there?

A good deal of science is historical in nature. We try to construct theories that explain how things got to be the way they are; and then we run tests or make lots of observations.

Software History is for the Birds

Or so it appears, from the way that the vast majority of software people act. We're about to embark on a new project. How did similar projects work out in the past? What are we doing differently? The uniform response to questions like these? Crickets.

One thing I've realized is that our determined effort to ignore history in software is a completely understandable defense mechanism. Suppose you're starting an hours-long road trip. At the end is near-certain disaster. Would you like to know that at the beginning of the trip, so that every second is miserable, building to a crescendo of terror? Or would you rather blissfully cruise along, and then be blind-sided at the end, leading to a mercifully quick death? Apparently, pretty much everyone agrees that blissful ignorance is the way to go.

A Wonderful World

Here's what I think would be a wonderful world:

1. They both love each other, AND
2. They know lots of software history together, leading not only to A's in school, but great jobs and successful projects.

Math and Computer Science vs. Software Development

In a prior post, I demonstrated the close relationship between math and computer science in academia. Many posts in this blog have delved into the pervasive problems of software development. I suggest that there is a fundamental conflict between the perspectives of math and computer science on the one hand, and the needs of effective, high quality software development on the other hand. The more you have computer science, the worse your software is; the more you concentrate on building great software, the more distant you grow from computer science.

If this is true, it explains a great deal of what we observe in reality. And if true, it defines and/or confirms some clear paths of action in developing software.

A Math book helped me understand this

I've always loved math, though math (at least at the higher levels) hasn't always loved me. So I keep poking at it. Recently, I've been going through a truly enjoyable book on math by Alex Bellos.

It's well worth reading for many reasons. But this is the passage that shed light on something I've been struggling with literally for decades.

When we learn to count, we're learning math that's been around for thousands of years. It's the same stuff! Likewise when we learn to add and subtract. And multiply. When we get into geometry, which for most people is in high school, we're catching up to the Greeks of two thousand years ago.

As Alex says, "Math is the history of math." As he says, kids who are still studying math by the age of 18 have gotten all the way to the 1700's!

These are not new facts for me. But somehow when he put together the fact that "math does not age" with the observation that in applied science "theories are undergoing continual refinement," it finally clicked for me.

Computers Evolve faster than anything has ever evolved

Computers evolve at a rate unlike anything else in human experience, a fact that I've harped on. I keep going back to it because we keep applying methods developed for things that evolve at normal rates (i.e., practically everything else) to software, and are surprised when things don't turn out well. The software methods that highly skilled software engineers use are frequently shockingly out of date, and the methods used for management (like project management) are simply inapplicable. Given this, it's surprising, and a tribute to human persistance and hard work, that software ever works.

This is what I knew. It's clear, and seems inarguable to me. Even though I'm fully aware that the vast majority of computer professionals simply ignore the observation, it's still inarguable. The old "how fast do you have to run to avoid being eaten by the lion" joke applies to the situation. In the case of software development, all the developers just stroll blithely along, knowing that the lions are going to to eat a fair number of them (i.e., their projects are going to fail), and so they concentrate on distracting management from reality, which usually isn't hard.

What is now clear to me is the role played by math, computer science and the academic establishment in creating and sustaining this awful state of affairs, in which outright failure and crap software is accepted as the way things are. It's not a conspiracy -- no one intends to bring about this result, so far as I know. It's just the inevitable consequence of having wrong concepts.

Computer Science and Software Development

There are some aspects of software development which are reasonably studied using methods that are math-like. The great Donald Knuth made a career out of this; it's valuable work, and I admire it. Not only do I support the approach when applicable, I take it myself in some cases, for example with Occamality.

But in general, most of software development is NOT eternal. You do NOT spend your time learning things that were first developed in the 1950's, and then if you're good get all the way up the 1970's, leaving more advanced software development from the 1980's and on to the really smart people with advanced degrees. It's not like that!

Yes, there are things that were done in the 1950's that are still done, in principle. We still mostly use "von Neumann architecture" machines. We write code in a language and the machine executes it. There is input and output. No question. It's the stuff "above" that that evolves in order to keep up with the opportunities afforded by Moore's Law, the incredible increase of speed and power.

In math, the old stuff remains relevant and true. You march through history in your quest to get near the present in math, to work on the unsolved problems and explore unexplored worlds.

In software development, you get trapped by paradigms and systems that were invented to solve a problem that long since ceased being a problem. You think in terms and with concepts that are obsolete. In order to bring order to the chaos, you import methods that are proven in a variety of other disciplines, but which wreck havoc in software development.

People from a computer science background tend to have this disease even worse than the average software developer. Their math-computer-science background taught them the "eternal truth" way of thinking about computers, rather than the "forget the past, what is the best thing to do NOW" way of thinking about computers. Guess which group focusses most on getting results? Guess which group would rather do things the "right" way than deliver high quality software quickly, whatever it takes?

Computer Science vs. Software Development

The math view of history, which is completely valid and appropriate for math, is that you're always building on the past, standing on the shoulders of giants.

The software development view of history is that while some general things don't change (pay attention to detail, write clean code, there is code and data, inputs and outputs), many important things do change, and the best results are obtained by figuring out optimal approaches (code, technique, methods) for the current situation.

When math-CS people pay attention to software, they naturally tend to focus on things that are independent of the details of particular computers. The Turing machine is a great example. It's an abstraction that has helped us understand whether something is "computable." Computability is something that is independent (as it should be) of any one computer. It doesn't change as computers get faster and less expensive. Like the math people, the most prestigious CS people like to "prove" things. Again, Donald Knuth is the poster child. His multi-volume work solidly falls in this tradition, and exemplifies the best that CS brings to software development.

The CS mind wants to prove stuff, wants to find things that are deeply and eternally true and teach others to apply them.

The Software Development mind wants to leverage the CS stuff when it can help, but mostly concentrates on the techniques and methods that have been made possible by recent advances in computer capabilities. By concentrating on the newly-possible approaches, the leading-edge software person can beat everyone else using older tools and methods, delivering better software more quickly at lower cost.

The CS mind tends to ignore ephemeral details like the cost of memory and how much is easily available, because things like that undergo constant change. If you do something that depends on rapidly shifting ground like that, it will soon be irrelevant. True!

In contrast, the Software Development mind jumps on the new stuff, caring only that it is becoming widespread, and tries to be among the first to leverage the newly-available power.

The CS mind sits in an ivory tower among like-minded people like math folks, sometimes reading reports from the frontiers, mostly discarding the information as not changing the fundamentals. The vast majority of Software Development people live in the comfortable cities surrounding the ivory towers doing things pretty much the way they always have ("proven techniques!"). Meanwhile, the advanced Software Development people are out there discovering new continents, gold and silver, and bringing back amazing things that are highly valued at home, though not always at first, and often at odds with establishment practices.

Qualifications

Yes, I'm exaggerating the contrast between CS and Software Development. Sometimes developers are crappy because they are clueless about simple concepts taught in CS intro classes. Sometimes great CS people are also great developers, and sometimes CS approaches are hugely helpful in understanding development. I'm guilty of this myself! For example, I think the fact that computers evolve with unprecedented speed is itself an "eternal" (at least for now) fact that needs to be understood and applied. I argue strongly that this fact, when applied, changes the way to optimally build software. In fact, that's the argument I'm making now!

Nonetheless, the contrast between CS-mind and Development-mind exists. I see it in the tendency to stick to practices that are widely used, accepted practices, but are no longer optimal, given the advances in computers. I see it in the background of developers' preferences, attitudes and general approaches.

Conclusion

The problem in essence is simple:

Math people learn the history of math, get to the present, and stand on the shoulders of giants to advance it.

Good software developers master the tools they've been given, but ignore and discard the detritous of the past, and invent software that exploits today's computer capabilities to solve today's problems.

Most software developers plod ahead, trying to apply their obsolete tools and methods to problems that are new to them, ignoring the new capabilities that are available to them, all the while convinced that they're being good computer science and math wonks, standing on the shoulders of giants like you're supposed to do.

The truly outstanding people may take computer science and math courses, but when they get into software development, figure out that a whole new approach is needed. They come to the new approach, and find that it works, it's fun, and they can just blow past everyone else using it. Naturally, these folks don't join big software bureaucracies and do what everyone else does. They somehow find like-minded people and kick butt. They take from computer science in the narrow areas (typically algorithms) where it's useful, but then take an approach that is totally different for the majority of their work.

Continents and Islands in the World of Computers

The vast majority of people appear to think that the world of computers and software is pretty uniform. While everyone recognizes that there are differences between the systems used by consumers and ones in business, it makes sense that pretty much the same thing is going on inside.

The reality is that vast cultural and practical differences separate the various clusters of computer and software applications. I look forward to the first anthropological studies that are devoted to this subject, illustrating and spelling out the untravelled oceans that separate the diverse lands of computer and software practice. Meanwhile, there are both obstacles and opportunities that arise from these facts.

A Diversity of Tongues

The Bible gives a vivid explanation of how the various languages arose. In the post-flood world, there was said to be a single people with a single tongue. They built a city with a tower that reached to the sky, to make a name for themselves.
As a single people with a single tongue, "the sky was the limit" for how far they could go. God didn't like this. He reached down and confounded their speech, and scattered them over the face of the earth.

Whether it's the fault of God or humans, it's well understood that groups of people develop their own languages, which then evolve and splinter. In fact, by studying the relationship of various languages, you gain insight into how humans migrated over the earth. The Indo-European family of languages is an excellent example of this.

A Diversity of Software Languages

Early computer programmers in the 1950's clearly saw the advantage of having a single language for software. They tried hard to create a universal software language, FORTRAN being the most well-known and successful such early language. While FORTRAN (short for "formula translator") was great for math people, people working with business records weren't impressed. Thus COBOL ("COmmon Business-Oriented Language") was invented. Things were still pretty simple in the mid-1950's.

But of course it didn't stop there. People migrated to different "lands," confronted new issues, and created new languages suitable for the new problems. Here's a snapshot of some of the major developments in the mid-1970's.

By now, literally thousands of general-purpose languages have been created. Not including "esoteric" languages and thousands more languages for narrow problem domains.

Idioms and Layers

As anyone who has learned a new human language as an adult is well aware, learning a language is one thing -- but learning all the incidental aspects of the language, particularly the idioms, is a task that never seems to end. This is because there are lots of them -- an estimated 25,000 of them in English, for example.

There are equivalents of idioms in software languages, in multiple categories. I'll just give a basic example: the run-time library, whose documentation frequently exceeds that of the base language, and without which you can't write practical programs. A more complex example: a "framework," for example the RAILS and Sinatra frameworks for the Ruby language.

Let's say you know English pretty well. Are you qualified to write or even to read and understand a legal brief? Unless you're a lawyer, probably not. Yet there's no denying that the brief is written in English, with a whole pile of idioms and other special things that are not in common use.

It's the same with computer languages. You may know the language C pretty well, but when you first look at the C code in a driver or an operating system kernel, it's got a strange idiom -- "straight" C would stand out as obviously as BBC English would in Brooklyn. The C that's in a compiler is even more rife with idioms and unfamiliar constructs, so much so that a person who is otherwise fluent in C would have trouble figuring out what was going on, just as a normal fluent English-speaker would have trouble following a discussion by two doctors of a difficult medical case.

Different Continents, Different Cultures

Everyone knows that human languages are part of overall human culture. Cultures differ at least as much as languages. It's not widely appreciated that this is also true in the world of computers. While of course there are commonalities, you actually think differently in different languages, whether human or software. Lots of incidental things tend to get wrapped up in the cultures as well. To take a simple example: in the US, when you walk into a store, prices tend to be marked on the goods. If you like that thing at that price, you buy it; otherwise you don't. It doesn't work that way in much of South Korea. Prices may not be marked at all; and negotiation is assumed and expected. There are a whole set of cultural norms that have to be understood in order to thrive.

Differences in software cultures are just as strong as differences in human cultures. For example, one of the major forces in the world of hospital automation is Epic. Epic is written in a language originally called MUMPS. While few write programs from scratch in MUMPS anymore, you have to use the language to customize the Epic system, just as you have to use ABAP to customize the SAP manufacturing system. In either case, learning the peculiar language is the tip of the iceberg -- your programs "live in" the Epic system, and so knowing that system inside out is the key to success, far more important than utilization of the language itself. This is perhaps comparable to the importance of knowing all the relevant prior case law in writing a legal brief in support of your position, vs. simply knowing the vocabulary and syntax of English.

Contact Between Distant Cultures

We know that in human life, cultures developed in isolation from each other for many thousands of years. Migrating peoples would clash, and the cultures with superior instruments and warrior culture tended to decimate the others. A culture that develops superior methods of war, usually with distinct technology, tends to expand. This was true, for example of the Commanche in North America and the Mongols, each of which developed unique competence and technology with horses, and used that to rapidly expand their sphere of influence.

This is an area of both similarity and difference with software cultures. Within a company, there are frequently members of different software "tribes," who usually neither understand nor like each other. They are constantly at war to establish primacy, the winning culture grudgingly conceding resource-poor reservations to which the losers are confined.

It's different with software cultures that are separated by "oceans," usually different problem domains or industries. You might like to think that everyone who does software has all the information available, and therefore is at a similar cultural level, the equivalent of, say, the different countries in Europe. They may speak different languages and like different foods, but they all have cars, telephones, and electrical appliances. In software, this is not the case! In software, there are cultural differences that are the equivalent of cars being widely available in one place, while ox carts are the standard mode of transport in another. It's that extreme.

What's even more shocking is that the denizens of these culturally isolated software continents are comfortable and secure in what outsiders see as their "backwardness" or "ignorance," and find ways to denigrate and disparage outsiders who dare to suggest there might be a better way of doing things.

How big are these culturally retarded continents? Just a few distant places, the equivalent of Australia? If you have the opportunity to see a wide swath of software culture, what you find is that the vast majority of software groups have cultures that are dramatically inferior to the best places. Moreover, the variance in just how primitive things are is huge.  In any given place, it is likely that practices that are unknown but would dramatically enhance results are already standard practice in other places. Judging software isn't like auditing the financial books of a company, in which you either pass or fail an audit. It's more like figuring out which part of which software continent the place is, and how many years or decades behind the best known methods the place is, and to what extent their near neighbors are slightly ahead or behind.

Conclusion

It's useful to compare human language and culture to software language and culture. Just as with humans, language is an important part of culture, but thriving involves a whole lot more than just grasping the basics of a language. Just like with humans, there are varying levels and there are conflicts. But what's most interesting are the differences, which are the equivalent of humans living in environments that are physically next to each other, but using tools and methods that are hundreds of years different in terms of evolution. This fact has huge implications on many levels.

Lessons for Software from the History of Scurvy

Software is infected by horrible diseases. These awful diseases cause painfully long gestation periods requiring armies of support people, after which deformed, barely-alive products struggle to be useful, live crippled existences, and are finally forgotten. Software that functions reasonably well is surprisingly rare, and even then typically requires extensive support staffs to remain functional.

Similarly, sailors suffered from the dread disease of scurvy until quite recently in human history. The history of scurvy sheds surprising light on the diseases which plague software. I hope applying the lessons of scurvy will lead to a world of disease-free, healthy software sooner than would otherwise happen.

Scurvy

Scurvy is caused by a lack of vitamin C. It's a rotten disease. First you get depressed and weak. Then you pant while walking and your bones hurt. Next your skin goes bad,

your gums rot and your teeth fall out.

You get fevers and convulsions. And then you die. Yuck.

The Impact of scurvy

Scurvy has been known since the Egyptians and Greeks. Between 1500 and 1800, it's been estimated that it killed 2 million sailors. For example, in 1520, Magellan lost 208 out of a crew of 230, mainly to scurvy. During the Seven Years' War, the Royal Navy reported that it conscripted 184,899 sailors, of whom 133,708 died, mostly due to scurvy. Even though most British sailors were scurvy-free by then, expeditions to the Antarctic in the early 20th century were plagued by scurvy.

The Long path to Scurvy prevention and cure

The cure for scurvy was discovered repeatedly. In 1614 a book was published by the Surgeon General of the East India company with a cure. Another was published in 1734 with a cure. Some admirals kept their sailors healthy by providing them daily doses of fresh citrus. In 1747 the Scottish Naval Surgeon James Lind proved (in the first-ever clinical trial!) that scurvy could be prevented and cured by eating citrus fruit.

Finally, during the Napoleonic Wars, the British Navy implemented the use of fresh lemons and solved the problem. In 1867, the Scot Lachlan Rose invented a method to preserve lime juice without alcohol, and daily doses of the new product were soon standard for sailors, which is how "limey" became synonymous with "sailor."

Competing Theories and Establishment Resistance

The effective cures that had been known and used by some people for centuries were not in a vacuum. There were competing theories. Cures included urine mouthwashes, sulphuric acid and bloodletting. As recently as 100 years ago, the prevailing theory was that scurvy was caused by "tainted" meat. How could this be?

We've seen this movie before. Over and over again. I told the story of Lister and the discovery of antiseptic surgery -- and the massive resistance to the new method by the leading authorities at the time.

Software Diseases

This brings us back to software. However esoteric and difficult it may be, software is a human endeavor: people create, change and use software and the devices it powers. Like any human endeavor, some of what happens is because of the subject matter, but a great deal is due to human nature. People are, after all, people, regardless of what they do. Patients were killed for lack of antiseptic surgery -- and the surgical establishment fought it tooth and nail. Millions of sailors were killed by scurvy, when a cure had been known, practiced and proved for centuries. Why would we expect any other reaction to cures for software diseases, when the "only" consequence of the diseases are explosive growth in the time, cost and risk to build and maintain software, which is nonetheless crappy and late?

Is there a general outcry about this dismal software situation? No! Why would anyone expect there would be? Everyone thinks it's just the way software is, just like they thought scurvy in sailors and deaths after surgery were part of life. Government software screws up,

software from major corporations is awful,

software from cool new social media companies is inexcusably bad. Examples of bad software can be listed for endless, boring, tedious, like forever lengths.

Toward Healthy Software Development

If I had spent my life in the normal way (for a software guy), I wouldn't be on this kick. But I didn't and I am on this most-software-sucks kick. Early on, I had enough exposure to large-group software practices to convince me that I wanted none of it. I'd rather actually get stuff done, thank you very much. Now, looking at many young software ventures over a period of a couple decades, the patterns have emerged clearly.

I have described the main sources of the problems. I have described the key features of disease-free software development. I have explained the main sources of the resistance to a cure, for example in this post. And I have no illusion that things will change any time soon.

It will sure be nice when the pockets of healthy software excellence that I see proliferate more quickly than they are, and when an anti-establishment consensus consolidates and gains visibility more quickly than it is. In the meantime, there is good news: groups that use healthy, disease-free software methods will have a massive competitive advantage over the rest. It's like ninjas vs. a collection of retired security guards. It's just not fair!

Fundamental Concepts of Computing: Speed of Evolution

Nothing we encounter in our daily lives changes or evolves as quickly as computing. All our habits of thinking are geared towards things that evolve slowly, compared to computing. This is a simple concept. It is disputed by no one. But it has implications that are vast and largely undiscussed and unexplored. It clearly deserves to be a fundamental concept of computing, along with a few correlaries.

Normal evolution speeds

Our planet evolves. Most of the changes are slow, with peak events mixed in. Life forms evolve slowly, over tens of thousands of years. Human culture evolves more quickly -- too fast for some people, and not nearly fast enough for others. Human capabilities also evolve, but for something like speed to double over a period of decades is astounding.

Take people running as an example. Here are the records for running a mile over the last 150 years or so.
The time has been reduced by roughly 25% over all those years. Impressive for the people involved, and amazingly fast for human change.

Once you shift to things made by humans, the rate increases, particularly as science and technology have kicked in. We've invented cars and they've gotten much faster since the early ones. Here is a lady on a race car in 1908. She was called the "fastest lady on earth" for driving at 97 mph in 1906.

in 2013, Danica Patrick won pole position during qualifying rounds at the Daytona Speedway by averaging over 196 m.p.h.

In this case, speed roughly doubled over about a hundred years.

Computer evolution speeds

Computers are different. Moore's Law is widely known: power doubles roughly every 18 months. And then doubles again. And again. And again. Every time someone predicts an end to the doubling, someone else figures out a way to keep it going. This is a fact, and it's no secret. It's behind the fact that my cell phone has vastly more computational power and storage than the room-sized computer I learned on in 1966.

This chart should blow anyone's mind, even if you've seen it before. It shows processor transistor count (roughly correlates with power) increasing by a factor of 10, then another, then another. In sum, it shows that power has increased about one million times over the last forty years.

It would be mind-blowing enough that we have something in our lives that increases in speed at such an incomprehensible rate. But that's not all! Everything about computers has also gotten less expensive! For example, the following chart shows how DRAM storage prices have gone down over the last twenty years, from over $50,000 per GB to around $10. In other words, to about 2% of 1% of the price twenty years ago.

That's faster and cheaper in a nutshell.

So what? Everyone knows that things change quickly with computers, what's the big deal? It's just the way things are!

Here's what: this simple fact has profound implications.

Everything about human beings is geared for people and things that evolve at "normal" speed. Our patterns of thought, the things we do, most of our behaviors were developed for "normal-speed" evolvers, i.e., everything but computers (EBC). A surprising number of them break, are wrong or yield crappy results when applied to computers. There is no reason at all to be surprised by this; in fact, anything else would be surprising. What is surprising and interesting is that the implications are rarely discussed.

Computers evolve more quickly: it matters!

When you apply patterns of thought and behavior that may be appropriate for EBC to fast-evolving computers, those thoughts and behaviors typically fail miserably. This "impedance mis-match" explains failure patterns that persist for decades. Of course everyone knows that computers are different from other things -- just as they know that Indian food is different from Chinese food. What they tend not to know is that computers are different in a different way (because of the speed of evolution) from EBC.

Here are a couple examples.

The mainstream "wisdom" for software project management is essentially the same, with minor modifications, as managing anything else, from building a house to a new brand of toothpaste. It's not! That's one of the many reasons why the larger organizations that depend on those techniques fail to build software effectively.

We treat software programming skills like any other kind of specialized knowledge, like labor law. They're not! Thinking that they are is one of the many reasons why great software people are 10 times better than really good ones, who are themselves 10 times better than average ones.

The normal ways people go about hiring software engineers is crap. They think that hiring folks who deal with a subject matter that changes so dramatically is the same as hiring anyone else. It's not! They also think that extended, in-depth experience with a particular set of technologies is really valuable. It's not! It's actually a detriment!

Software engineers tend to learn to program in a certain way, using a given set of tools, techniques and thought patterns. Those tools were designed to solve the set of problems that existed with computers at a certain point in their evolution. But computers evolved from that point! Quickly! The programmers are doing the equivalent of hunting for rabbits with weapons designed for hunting Mastodons, blissfully unaware of what's appropriate for computers the way they are today!

Software is hard and isn't getting easier

Software isn't a problem that gets solved -- oh, now we know how to do it, finally. It doesn't get solved because the underlying reality (computers) evolves more quickly than anything else. The examples I mentioned are things that "everyone" is sure must apply to software. How could they not? The fact that they yield consistently horrible results seems not to break the widespread faith in the mainstream approach to software. 

These and many other broadly accepted falsehoods explain why so many things about software are broken and (worse) don't seem to get fixed, decade after decade, when superior methods have been proven in practical application. Why is everyone so resistant to change? Is everyone stupid?

Aw shucks, I admit it: "everyone is stupid" was my working hypothesis for explaining things like this. But I now have a more satisfying hypothesis: everyone is so used to dealing with "normal speed things," EBC's, that they just can't help themselves from applying to computers the methods and patterns of thought and behaviors that work reasonably well in most of their lives. Since nearly everyone gets the same lousy results, the conclusion everyone draws is that there's something about computers that's just miserable.

Conclusion

There is nothing comparable to computers in the rest of our human experience. Nothing evolves at anywhere close to the speed of computers, getting more powerful while getting cheaper at hard-to-comprehend rates. We apply methods and patterns of thought that work well with practically everything, and those methods fail when applied to computers. But they fail for everyone! The conclusion everyone draws from this is that computers are just nasty things, best to stay away from them and avoid blame. It's the wrong conclusion.

Computers are understandable. The typical failures are completely explained by the mis-matched methods we bring to them, like trying to catch butterflies with a lasso. When people use methods that are adapted to the unprecedented evolutionary speed of computers, things go well.

The Big Data Technology Fashion

Where there are people, there are fashions. Why should technology be immune? The current fashion of "big data" is a classic exemplar of the species.

The Books

Books are a good place to observe the common themes of technology fashions. You'll see patterns that resemble the ones I previously pointed out for project management.

I think it's fair to say it's not a legitimate technology trend if it's not covered in an "X for Dummies" book.

Similarly, it's got to be big. Be Revolutionary. Transform lots of stuff.

Its got to be a big, scary thing that needs taming.

For any fashion trend, its important to make sure that other things are hitched to its wagons.

Let's not forget that, if it's worth paying attention to, there's got to be a way to make money from it.

It's never too soon to start adding layers of process and paranoia to it, to assure that costs skyrocket and that hardly anything ever gets done; in other words, governance.

Finally, anything but anything has to have a human side.

I swear, I sometimes think there's a central planning committee for technology fashions. They plan when the next new label on something old and not all that interesting is going to come out, grab their standard set of titles, and pass them out to people to write the books.

But then, I guess it can't really be that organized, because there are usually so very many books, each of them covering the same small set of themes over and over and over, with slightly different language. The themes always seems to include:

• X is revolutionary; it will change lots of important stuff.
• X is big and scary, and you need help to tame it or bring it under control
• There are lots of ways to screw up doing X, so you need to pay lots of money for Y to get it right
• You're a Dummy, but I'll help you understand what you need to know about X anyway.
• X has a human side

The Conferences

Things aren't that different with conferences. They take the themes established in the books and embellish them a bit.

There are conferences for people who work in particular sectors.

 

You can't pass up an opportunity to learn from the very best.

Who can resist going to a conference which cuts through all the crap and helps you do stuff?

Anyway, you get the idea -- there are lots of conferences. The themes are predictable, even without the aid of big data or predictive analytics. Because they apply to any technology fashion trend.

Conclusion

Technology fashions -- they are forever in fashion!

What Can Software Learn from Steamboats and Antiseptic Surgery?

Software is among the most advanced, rapidly changing fields of technology. Only the "kids" who grew up with the latest techniques seem to be able to master them. At the same time, really bad ideas spread through software groups like the plague; they take hold and resist cure, in spite of producing terrible results. How can we make sense out of a field that advances rapidly and resists change at the same time?

History

As I've pointed out, software people are strongly averse to learning about computer history. In some fields (e.g. physics), the very terms used are named after historical figures; in others, history is treated with reverence (e.g., Santayana: "Those who cannot remember the past are condemned to repeat it."); in software, by contrast, we use the phrase "that's history" to dismiss anything that happened in the past as obviously irrelevant to the present.

I think studying history is the only way to understand the present, software included. I think we can understand the strange software phenomenon of rapid change combined with resistance to change by taking two examples from history: one in which new methods in technology were rapidly accepted by all concerned parties, and the other in which clearly superior new methods were resisted for many long years by the leading people in the field.

Steamboats

It would be great if software advances were adopted quickly, like the way steam technology rapidly overcame wind as a method for moving boats.

The displacement of wind by steam is clearly laid out in T.J. Stile's excellent biography of one of the major figures in the transformation, Cornelius Vanderbilt.

Vanderbilt started in business by running a sailing-boat "taxi" service from Staten Island to Manhattan. He transitioned into the rapidly emerging steam boat transportion business, not only as a captain and owner, but (surprisingly to me) as an engineer.

The public took to the new steamboats quickly. The reason is clear: speed. There was, at the time, no quicker way to get from point A to point B if there were a water route between them. The speed of the boat was immediately obvious to the simple observer, and easy to verify by noting departure and arrival times. To prove whose boat was the fastest, there were races.

Vanderbilt's steamboats were judged by a clear standard: whose was the fastest? The criteria were easy to measure.

Antiseptic Surgery

The benefits of antiseptic surgery, as introduced by Joseph Lister, were clear: instead of a large number of patients dying of infection after surgery, they would live. Ego clearly played a role in resisting the adoption of the new method. But, to be fair, there is another important factor.

What made surgery different from steamboats? They were both major technical advances. They both involved major changes in what you did and how you did it -- more so with boats than with surgery! So why did steam catch on quickly, even though they required whole new boats of radically different design and operation, while the antiseptic method was resisted for decades, even though it was subsidiary to the surgery itself, which was left largely unchanged?

Boats and Surgery

The fact that steamboats were faster than sailboats was easy and unambiguous to measure, while the surgery outcomes were difficult and ambiguous to measure.

The time of each boat trip is easy to measure. It's just a time duration. When you watch two boats, anyone can see which one moves more quickly. By contrast, every surgery is different. The patient is different, the trouble being fixed is different, and the ultimate outcome may not be determined for weeks. Many surgery patients continued to die with antiseptic methods because it wasn't the only factor influencing the outcome. Furthermore, excellent surgeons who were dirty could save patients that would have been killed by crappy surgeons who happened to use antiseptic methods, since after all not every patient got infected.

In retrospect, it's completely maddening that surgeons failed to be swayed by the arguments and evidence in favor of Lister's carbolic acid methods, and ego certainly played a role. But the case of the rapid acceptance of the more radical change to steam in boats makes it clear that something more than ego is at work here. Simply put, it is how comparable and measurable are the outcomes of the new technology? With steamboats, you can tell the difference in seconds with the naked eye, and verify it with a stopwatch. No arguments. With surgery, the cases are not clearly and unambiguously comparable, statistics are needed, and there is major variability. There is room for arguments.

Software, steamboats and antiseptic surgery

Is any given advance in software like moving from sailboats to steamboats, or is it more like adding antiseptic methods to surgery?

That's easy: unlike straightforward competitions like races, every software project is different. In a race, the competitors take off from a starting line at the same time, and whichever crosses the finish line first is the winner. Simple! But in the real world of software, every project is different; you can always point to differences in requirements, conditions, deployment, or other things to explain why this project took more time and resources than that project. It sounds like software is kind of like surgery!

Conclusion

It is my personal experience and judgment that ego can play a significant role in explaining why many software groups stay mired in the same old methods, getting the same lousy results, year after year. But I think that if software projects were as comparable as transportation schedules, the evidence would simply force more rapid change, like it or not, on intransigent software groups. But because of how genuinely challenging it is to compare software projects to each other, it is at least understandable how only the most enterprising and eager-to-be-the-best software groups seek out and adopt the very best methods. 

Computer History

In software, history is ignored and its lessons spurned. What little software history we are taught is often simply wrong. Everyone who writes or uses software pays for this, and pays big.

But we know about history in software -- there's Babbage, the ENIAC, etc.

Yes, we've all heard about various people who are said to have invented modern computing. A  shocking amount of what we are taught is WRONG.

Babbage is a case in point. People just love to go on and on about him. There are  problems, though. I'll just mention a couple.

One problem is that his machines simply didn't work, even after decades of work, and huge amounts of skilled help and money. He must have known they wouldn't; although he was personally wealthy, it was other people's money he spent on his famous dalliance.

Another problem is that his best idea wasn't his. The idea of using punched cards

to contain the program was invented in France and was a key aspect of the Jacquard Loom -- a machine that pre-dated all his work, and a machine that actually worked and was in widespread use.

The ENIAC is another good example of what appears to be the typical pattern in computing, which is someone invents a good thing, makes it work, and then someone else steals it, takes credit for it and tries to cover up the theft, often without delivering results as good as the original.

If you only read the standard literature, you would still be convinced that the ENIAC and its inventors were giants of the field. Once you read everything, you discover that reality is more interesting. It turns out that the inventors of the ENIAC were "inspired" by prior inventions, much like Babbage and the Jacquard Loom. In this case, the inspiration was the Atanasoff-Berry Computer.

Here is an excerpt from the ruling in the patent dispute that settled the issue:

Judge Larson had ruled that John Vincent Atanasoff and Clifford Berry had constructed the first electronic digital computer at Iowa State College in the 1939-1942 period. He had also ruled that John Mauchly and J. Presper Eckert, who had for more than twenty-five years been feted, trumpeted, and honored as the co-inventors of the first electronic digital computer, were not entitled to the patent upon which that honor was based. Furthermore, Judge Larson had ruled that Mauchly had pirated Atanasoff's ideas, and for more than thirty years had palmed those ideas off on the world as the product of his own genius.

Other fields don't need history -- why should software?

Not true. Other fields are saturated with history.

Politicians study history in general and the last election in particular. Fiction writers frequently read fiction, current and historic. Generals study old battles for their lessons; even today at West Point, they read about the Civil War. Learning physics is like going through the history of physics, from Galileo and Newton and through Planck and Einstein to the present. Even the terms used in physics remind you of its history: hertz, joules and Brownian motion.

Software, by contrast, is almost completely a-historical. Not only are most people involved uninterested in what happened ten years ago, even the last project is unworthy of consideration – it’s “history.”

Consequences of the lack of history

War colleges study past wars for the highly pragmatic purpose of finding out how they were won or lost. What was it the winner did right? Was it better weapons? Better strategy? Better people? Some combination? And how exactly did the loser manage to lose? Was it a foregone conclusion, or was defeat snatched from the jaws of victory? People who conduct wars are serious about their history -- they want to win!!

In software, no one is interested in history. Everyone thinks they know the "right" way to build software, and thinks that the only possible source of loss is failing to do things the "right" way -- the requirements weren't clear; the requirements were changed; I wasn't given enough time to do a proper design; there was no proper unit testing; the lab for testing was insufficently realistic. The list of complaints and excuses is endless, and their net effect is always the same: crappy software and whining: I need more people, more time and more money. Because studying history is so rare, few are exposed to the software "wars" that are fought and won by teams that didn't follow their rules.

There is only one conclusion to be drawn: software people would rather lose with lots of excuses than win by doing things the "wrong" way. Ignoring history is a great way to stay in this comfortable cocoon.

When software history becomes as important a part of computer science education as physics history is of physics, we'll know it's approaching credibility. Until then, everything about computer science, education and practice will continue to be a cruel joke.

The Name Game of "Moving to the Cloud"

The most recent in a long string of technology fashion trends, "the cloud" is hot. Like its hot technology fashion predecessors, it mostly consists of old ideas with a little spicy sauce on top and fresh packaging. If you mindlessly follow the fashion and just "go to the cloud," you are likely to end up in the same unhappy place where most mindless followers of fashion trends end up. What is "the cloud?" Simple. Clouds are belated enterprise IT implementations of the consumer internet.

What is the Cloud?
Is the "cloud" a new development? Well, it is a new name...

Ancient Clouds
I first encountered the cloud more than 40 years ago. Before that fateful meeting, my only experience with computers had been up close and personal. You had to get in the room, push buttons, flick switches, feed card decks or punched paper tape, listen to whirring sounds and watch blinking lights. Like with this computer:


But The Cloud! Ahhh, the Cloud...I remember it vividly.

Of course, things were a bit different then. We only had "fat clients," as in "takes two guys to lug it" fat. The principle was identical, however: I was in one place, with a "fat client" like this:


...and in another place was a computer like this:


...and everything worked.
Remember the importance of labelling, however: what we did 40 years ago wasn't "cloud computing," which hadn't been "invented" yet -- it was merely "telecommunications."

Creating the Modern Cloud
Between then and now, lots of iterations of Moore's Law have come and gone. All the hardware has gotten smaller, cheaper and faster, while the software has gotten larger, more expensive and slower -- but, fortunately for all of us, the rate of hardware evolution is greater than the rate of software devolution, giving the impression of net progress.

Where this leaves us, 40 years later, is faster remote computers talking with lighter remote clients over incredibly faster networks, all at lower cost. Sprinkle with a little extra software and drizzle with some marketing hoo-haw, and -- kazaam! -- you've got today's hottest technology fashion trend, Cloud Computing.

Clouds aren't always friendly
When we think of clouds, we're likely to think of this kind of cloud:


friendly, fluffy shapes floating in an otherwise sunny sky. When people think about cloud computing, this is the kind of cloud they seem to have in mind. But as we know, there are other kinds of clouds. There are dark, oppressive clouds that make everyone depressed. And there are really mean clouds, that wreck things horribly, creating the situations for which "disaster recovery plans" are made. Is this just a metaphor? Of course not. But just like financial fraud, the big, juicy examples are usually hushed up in order to protect the guilty.

OK, so What is "the Cloud"

There is a little-discussed trend that is deeply embarassing to IT professionals: there is a wide and growing gap between the use of computing technology in the consumer world and in the corporate, data center world. When the average user of corporate data systems is home, he works in a very advanced computing environment. His local machines and devices are amazingly capable and pretty easy to use. When connected to the internet, he can access a nearly limitless world of cloud computing resources -- which are themselves largely run out of data centers that are remote from the people who set up and administer the software in them, and which contain an ever-evolving mix of dedicated and shared resources and services. The consumer internet has been based on a cloud computing model for a long time.

The corporate world is a whole different thing. The corporate world has been consumed with consolidating their diverse data centers. They are finally beginning to confront the extreme flexibility and ease of use that consumers enjoy every day, and are finding it increasingly difficult to explain why the computing they run with such high capital and operating costs are so cumbersome, error-prone and inflexible.

In this context, there is no way that anyone associated with corporate computing is ever going to plainly admit that what they are basically doing is trying to catch up with the consumer internet. So they must be doing something else. Oh, yeah -- they're evolving to the latest, smartest trend in corporate computing, adopting the latest technologies and being really leading-edge: they're "moving to the cloud," but of course in a "smart" way, with large doses of "private cloud" technology along the way.

Summary
What's a "private cloud?" A corporate data center with a fancier name.

What's a corporation "moving to the cloud?" A corporate IT group trying to play catch-up with the consumer internet, and desperately trying to make it look like something else.

What's new about "cloud computing?" Very little; mostly naming and marketing fluff.

Is anything real happening when a corporation "moves to the cloud?" Sometimes yes! Sometimes, they really are copying a couple proven techniques of the consumer internet, slowly and at great cost and trouble, but nonetheless creeping towards a 21st century computing model.

Why Computer Software is So Bad

There are aspects of the theory and practice of computer software that drive me nuts.

I look at a widely accepted theory or practice, and am aghast that the cancer spread and became part of the mainstream. I look at a genuinely good practice and am shocked at the way everyone lies and slithers to make it sound to the unsuspecting that whatever lousy thing they do is a shining example of a good thing. I look at a nearly-universal approach to problem solving that may have made sense many Moore's-Law generations ago, but has long since been rendered irrelevant by advances in hardware. And I look at proven, understandable techniques that improve productivity by whole-number factors that are spurned and/or ignored by most people in the field.

How significant is this stuff I'm complaining about? A minimum of a factor of 10. Sometimes a great deal more. So it matters! This is theoretical, but it has real-world, practical implications. It's the only way, for example, that small companies can beat large, established ones that have software staffs 10 to 1,000 times larger.

I think I understand some of the causes of this deplorable state of affairs. But that doesn't make me feel better. And it definitely doesn't cure the sick or empower the unproductive.

A number of my private-distribution papers go into these subjects in considerable depth. But I thought it might be interesting to summarize a couple of the main observations here.

Among the causes of bad software are:

The blind and deaf leading the blind. In the majority of cases, the people in charge have little to no personal experience of creating software, and no interest in how it's done. This is about as sensible as having someone in charge of a baseball team who not only can't play baseball, but can't see onto the field where it's being played; all he can do is see the score at the end and hear reports from the players about how things are going. So everything turns into politics -- convincing the blind, clueless boss that you're the great contributor, everyone else is a chump. The larger the organization, the more likely it is to be led by such a genius.

Process over substance. The larger and older the organization, the more likely it is to elevate the importance of process, and the more elaborate and all-consuming that process is likely to be. The more process there is, the less code gets written, and the productivity of the innocent few who actually want to work gets ground down to the abysmal norm.

Lawyers. Shoot the lawyers! Shoot them! They and their legal ways are a plague. When lawyers see a problem, they want to write a law or create a regulation that makes the problem go away. Except that instead of saying this in simple, results-oriented terms (e.g. "programmers should not be allowed to die unnecessarily while writing code"), they say it in terms of incredibly elaborate, micro-managing, this-is-how-thou-shalt-do-it regulations (e.g., "programmers should be offered a nutrional meal no later than 12 noon local time each day consisting of no less than..." and 538 similar instructions, constantly growing). If regulations of this kind actually led to good results it would be one thing. All I should have to say at this point is "credit card theft" and PCI.

Fashion over function. Programmers are supposed to be nerds, aren't they? How can programmers let their decisions be made by "fashion," whatever that's supposed to be in the world of software? I refer you to the first point, about software teams being led by people who are clueless. These people want to seem smart in the eyes of their bosses and peers, who know even less than they do. So they make decisions that are guided by C-office fashion trends, which are usually laughably out of step with true optimal decisions.

A preference for bad new ideas over good older ones. How do bad ideas catch on? Some company promotes them. Books are written. Appealing rhetoric is created and repeated. Suddenly a fashion trend is born. Someone can appear to be smart and modern just by advocating the cool new stuff and sounding smart, without actually knowing anything or doing anything. Everyone involved thinks they're helping advance the state of software, when in fact they're digging the hole of despair deeper. It's remarkably like when the dumbest guy in Scotland moves to England, thereby increasing the average intelligence of both places.

A preference for bad old ideas over good newer ones. There aren't many good new ideas; mostly there are old good ideas that are still good. But because conditions change (like processors getting fast and memory getting cheaper), ideas emerge to respond intelligently to the new state of affairs (like this one), instead of blindly and stupidly continuing on as though nothing had changed. Like most people do.

Wrong scope; usually too narrow. This is classic optimization over too narrow a range. Much bad software results from compartmentalism or simply failing to look at things through the eyes of the ultimate consumer. This is an incredibly common mistake. The trouble gets really bad when is the practices that result from the little "silos of excellence" get elevated into industry "best practices." (Whenever I hear the phrase "best practice," it's really hard for me to avoid getting a serious look and pronouncing in deep tones "you folks had best practice a bit longer before you inflict yourselves on the world of paid, professional programming.")

I'm sure there are many more causes of bad software than the ones I've listed here -- we haven't even gotten into bad programmers in their innumerable incarnations! But every item on the list above thoroughly deserves inclusion, even more because most of them are not listed by most people as a cause of software malaise.

Databases and Applications

When databases were invented, they solved a huge problem that couldn't be solved any other way. Anyone who cares to look can see that the original problem that caused us computer programmers to invent databases has largely gone away. So why is it exactly that application programmers reflexively put their data in a database? In a surprisingly wide range of cases, it sure isn't because of necessity. Could it, perhaps, perhaps, be nothing but habit and the little-discussed fact that change happens in software at roughly the same rate that change happens in glaciers?

From the beginning of (computer) time, instructions have needed to be in memory to be executed, and data has needed to be in memory to be operated on by instructions. The memory in which instructions execute and fiddle with data has always been way faster and way more expensive than the large, slow but cheap places they are put when they're not in memory (call it storage, whether the storage is punch cards or tape or disk). It was this way at the beginning of time and it's true now.

Think of memory as your work table. Eons ago, your work table was really tiny, like this:

You can hardly fit anything at all on it! So you'd better have a really big storage place to keep all your stuff, like a pantry:

OK, that's cool. You've got all your stuff in storage, but you can only work on when it's in memory (on the table). What do you need? You need to get the stuff you want to work on now from the pantry, and you need to put the stuff you're done with back in the pantry. In other words (if you're in the world of computers)...you need a database!

The very most basic function of a database is pretty simple: its job is to shuffle your data between memory and disk. It's also nice if it keeps everything straight, avoids dropping bits on the floor, and cleans everything up when something goes wrong during the shuffling.

That was then. But things have changed. Remember Moore's Law? The amount of memory available to us at surprisingly reasonable prices has grown hugely. Exponentially. Our work tables now look more like this:

And our pantries? Well, they've grown a bit too:

So how much data do you have? Run the numbers. It goes without saying that it's going to fit in storage. But how about that work table? Here's the question you have to think about:

Will all your data fit on the work table (i.e., in memory)?

With memory available at reasonable prices for 64GB and even 256GB, the answer is often YES! It will!

Hmmmm. What was it the database does? If all my data fits in memory, why was it I needed that database???

I know, I know. Databases can be nice for reporting and data analysis and "persistence" and a few other things. I'm not saying you never use them. But for your real application, the one that takes user requests and responds to them, if you don't need to have the database shuttling stuff between the work table and the pantry... Hmmmm.