The Black Liszt

Should a Software Engineer aspire to become a manager?

Software is a strange field. Its rules and patterns for career growth and advancement are unlike most others. It's thought of as a professional occupation, requiring a college degree and offering advanced degrees. At the same time, it's treated much like a manufacturing job, with no one showing any interest in how it's done other than wanting assurance that modern standard methods are used, and wanting quality results on budget and on time.

Being a software manager is better in many ways than being a software engineer. Dilbert, as usual, pretty much sums it up:

Surely there are other ways to advance a career in software than becoming a manager like pointy-hair-boss above. There are indeed. Here is a detailed description of how to climb up the ladder of status in software. Bottom line: the farther away you are from customers and actually producing working code, the higher your status. It's kind of like being a field hand on a farm. Necessary work, but everyone would rather someone else did it.

To explain what you do to people who know nothing about software, it's hard to avoid using physical metaphors to explain things -- metaphors that are widely accepted but deeply misleading. And then it's easy to slip into  lying about the software, because it's invisible to anyone who doesn't look at the code -- in other words, practically everyone.

Partly because of the invisibility, the compensation system is totally nuts. The star players in sports get the best salaries -- everyone watches them do their thing and win! Makes sense. But in software, it's as though no one can see what happens on the field. And worse, they've never picked up a ball or bat themselves, and have no idea, visual or otherwise, how it all works. So the pay is worse.

That's part of why software is so awful. It's no secret. There have been over 50 years of attempts to make the process of building software work the way that managers imagine it should. The attempts continue to this day. They fail. Here's a bit of history on the subject.

Why keep on keeping on as a programmer? It's invisible. You are pressured to lie to get ahead. You're constrained by awful methods that are supposed to make things better but in fact make them worse. Far easier to slog through the ten seconds of guilt described by pointy-hair-boss above, and make the move.

Software Management and Business School

While I was student at Harvard College, I worked at several jobs writing production software. I also audited a course at Harvard Business School. By wild coincidence, one of the jobs I had was with a small company that was the subject of a new case study used in the HBS course I took. What I learned from the contrast between the reality of the software company and the fantasy of the case study was the seed from which grew my current understanding of the application of standard management techniques to software. Bottom line: HBS-taught management techniques result in software that is expensive and bad.

How HBS case studies are written

Early in my four years at Harvard College I worked part time at a software company in Central Square in Cambridge called CPSI, Computer Programming and Systems, Inc. It was a programming services company, doing jobs for clients. I knew the languages they needed. I remember little about it or what I did there now; I remember not being impressed or excited by the work; with the campus chaos in the spring of 1969 I dropped it. Nonetheless the experience led to things that strongly shaped my career.

Not long after working at the Cambridge software company, I walked across the river to audit a course at the Harvard Business School, HBS.

The course was pretty new, called “Management of Small Enterprises: Operating Problems and Strategies.” It was a typical giant HBS course run on case studies taken from real life. The students competed to “nail the case,” by finding what HBS said was the best answer. The course was pioneering in concentrating on small business, since mostly HBS cranked out students to fill the big consulting companies and public companies. I attended to learn about starting and running a company.

Doing some research now, I found that the engaging professor, Arch R. Dooley, was a Yale and HBS grad. He never actually worked for a company, but nonetheless became an esteemed professor of manufacturing management, and started the programs for entrepreneurs, which have remained a small part of what HBS offers.

Part way through the course, which was interesting because of the type-A students fighting and clawing each other, a case study about a small software company became the focus. By the end of the case, I realized that the case was about CPSI, the company I had worked for during the prior semester. While a couple things had been changed to anonymize it, the company was easy for me to recognize. Even so, the case got important things about the company, its software and the market for software wrong. The HBS students were triumphant when they arrived at Prof Dooley’s approved approach to managing the company, which had absolutely nothing to do with what the software actually did. This blew me away.

After I graduated and stayed around Cambridge, along with a fair number of my classmates, I discovered that some of the super-bright, verbal people I knew were hired by HBS to … write case studies! They were given a bunch of existing material and case studies, with descriptions of generic “right answers” and thrown at a company. They would quickly figure out what they could and write up a cogent, compelling study. Maybe this worked for companies that did physical things they could easily understand, but none of them had the slightest exposure to software, and of course their managers were equally clueless. The result was the case study of the company I had worked for, which everyone involved thought was good material, but was in fact deeply flawed.

This HBS case study was my first exposure to the massive moat separating software reality and entrepreneurship from management and academia. Dooley was convinced that, although he’d never written a line of code and couldn’t even see it or read it, he could teach about software. He was equally convinced that, even though he’d never worked at a company, much less an entrepreneurial one, he could teach others how to do it. To this day, his colleagues think the same, and all the students and those who hire them are equally convinced. And much of it is based on case studies written by talented Harvard English majors fresh out of school.

This is stupidity, but it’s so common there has to be an explanation. I believe it’s one of the widespread manifestations of our class system, which nearly everyone implicitly understands but few people discuss. While the classes evolve, there’s always a clear hierarchy. One of the long-standing aspects of class is that people who do stuff in the real world generally are lower class than those who direct, observe or think. Attending HBS is clearly a way to enter the directing class. Engineers are highly technical, but usually other people build what they design. Programmers, by contrast, are people who use tools to build things, the way a craftsman might build a house. This puts them levels below the people who command what is to be built, just like the people who are served a meal in a grand hall are levels above the people who serve them and the cooks in the kitchen who make the food. All the grand professors at HBS and their students are far above the minions who write code in terms of class. Grand as they are, it’s clearly beneath them to stoop to dirty themselves with details of how things are built. They hire people for that. The fact that this attitude results in inefficiency, chaos and often disaster somehow isn’t their problem.

There are good reasons to teach management

The whole idea of a school teaching "management" is that there is knowledge about managing a group effort that is independent of the details of the effort and the products and services being created. This is a reasonable thought and generally valid. Think about finance, for example. Nearly every organization -- even non-profits -- have revenue and expenses, assets and liabilities. Expenses naturally fall into multiple categories, including payroll and capital. Different industries have different patterns of these things and the details matter, but the common elements are large and important.

Other things that are taught in business school have substantial elements that are similar across different industries. All industries have people who need to be hired, directed, and organized. Even software fits into the pattern. Suppose you're a construction company and want to build a bridge. The overall requirements of the bridge first need to be determined (number of lanes, etc.), followed by overall and detailed architectural and engineering design, and finally building. Just like software, right? It makes common sense and is widely accepted.

Why standard management fails for software

The widely accepted common sense notion that building software is pretty much like building a bridge now has a track record of many decades of failure.

Software development, security and management is an ongoing disaster, with all attempts to fix it failing. If bridges failed anywhere close to the rate of software failures, there would be revolution. See this. See this also.

One big reason is that software is different from all those things that management techniques work with. Food is cooked quickly, and you can taste the results. You can see buildings in progress and most other things. Software, by contrast, is invisible. How can you manage something you can't see?

Not being able to see it is bad, and not being able to do it yourself compounds the problem. At least with cooking and building you have a little idea of how it's done. In many fields, the best managers were good at doing what they're managing. This excellent principle is rarely followed in software.

Because of the invisibility, managers demand huge piles of documentation about the stuff they can't see. Makes things worse.

Everyone involved makes big efforts to track software as it's being built. It's often the case that glowing reports are made of software up until days prior to its being done. And then disaster ensues.

Because of its invisibility, managers tend to understand how software works in terms of metaphors -- metaphors that are inappropriate and make things worse. They want to create a software factory. They think of software as being like a building, with an architect and the rest. Even the notion of "building" software, as though it were a passive thing, is wrong. It's actually extremely rare that software is "built" -- mostly software is changed.

Because they're trained in business management, they think their methods apply to everything, including software. They use standard project management and variants created specifically for software like Agile to keep things under control. Without realizing the damage they cause, they insist on optimizing for predictability, while nimble startups beat them by optimizing for speed.The difference is like the difference between building bridges in peacetime or war.

Conclusion

Things have changed little since my bright college friends were churning out case studies to teach generations of future MBA's at HBS. Software is still invisible to the vast majority of managers who are nonetheless confident in the certified management methods they use, endorsed by all the Experts and high-paid people at the top of our class system. The methods that work to enable small teams to produce top-quality software quickly and well aren't secret; they are simply beneath the dignity of the people who dominate the upper ranks of all large organizations, including Big Tech.

 

How to Fix Software Development and Security: A Brief History

As the use of computers grew rapidly in the 1960’s, the difficulty of creating quality software that met customer needs became increasingly evident. Wave after wave of methods were created, many of them becoming standard practice – without solving the problem! This is a high-level survey of the failed attempts to solve the problem of software development and security -- all of which are now standard practice in spite of failing to fix the problem! There are now software auditing firms that will carefully examine a software organization to see in which ways it deviates from ineffective standard practice – so that it can be “corrected!”

Houston, We've Had a problem

The astronauts of the Apollo 13 mission to the Moon radioed base control about the problem that threatened the mission and their lives.

The astronauts were saved after much effort and nail-biting time.

Should the people in charge of software make a similar call to mission control? Yes! They have made those calls, and continue to make them, thousands of times per day!!

Getting computers to do what you want by creating the kind of data we call software was a huge advance over the original physical method of plugs and switches. But it was still wildly difficult. Giant advances were made in the 1950’s that largely eliminated the original difficulty.

As the years went on into the 1960’s, the time, cost and trouble of creating and modifying software became hard to ignore. The problems got worse, and bad quality surfaced as a persistent issue. There were conferences of experts and widely read books, including one from a leader of the IBM 360 Operating System project, one of the largest efforts of its kind.

The Apollo 13 astronauts were saved, but the disaster in software development has resisted all the treatments that have been devised for it. With the invention and spread of the internet, we are now plagued with cybercrime, to the extent that ransomware attacks now take place (as of June 2021) ... get ready for it ... 149,000 times per week!! I think a few more exclamation points would have been appropriate for that astounding statistic, but I'm a laid-back kind of guy, so I figured I'd stick with a mild-mannered two. Here's my description from more than four years ago on the state of ransomware and the response of the "experts."

Following are highlights of the methods that have been devised to solve the problem of software development and security. None of which have worked, but have nonetheless become standard practice.

Programming Languages

The assembled experts in 1968 decided that "structured programming" would solve the problem of writing software that worked. Here's how that turned out. A group of experts recently convened to survey 50 years of progress in programming languages. They were proud of themselves, but had actually made things worse. Highlights here. One of those early efforts led to object-oriented languages, which are now faith-based dogma in the evidence-free world of Computer Science. New languages continue to be invented that claim to reduce programmer errors and increase productivity. What always happens is that the "advances" are widely publicized while the large-scale failures are concealed; here are a couple of juicy examples. Above all, the flow of new languages provides clear demonstration that programmers don't have enough useful work to do to keep them busy.

Outsourcing

While programmers babbled among themselves about this language or that method, company managers couldn't help but noticing that IT budgets continued to explode while the results were an avalanche of failures. Service companies emerged that promised better results than company management could achieve because they claimed to be experts in software and its management.

One of the pioneers in computer outsourcing was Ross Perot's Electronic Data Systems (EDS), which had $100 million in revenue by 1975. By the mid-1980's it had over 40,000 employees and over $4 billion in revenue. It continued to grow rapidly. along with a growing number of competitors including services branches of the major computer vendors. In a typical deal, a company would turn over some or all of its hardware and software operations to the outsourcer, who would add layers of management and process. They succeeded by lowering expectations and hiding the failures.

Off-shoring

As communications and travel grew more cost effective, basing outsourced operations in another country became practical. Off-shoring had the huge advantage that while the results were no better than regular outsourcing, the fact that employees were paid much less than US wages enabled marginally lower prices and a huge infrastructure of management and reporting which further disguised the as-usual results.

US-based outsourcers started doing this fairly early, while non-US vendors providing software services grew rapidly. Tata Computer Services first established a center in the 1980's and now has revenue over $20 billion and has been the world's most valuable IT company, employing over 500,000 people. Infosys is another India-based giant, along with hundreds of others.

Project Management

Deeper investment in the growing and evolving collection of project management methods has been a key part of software organizations. The larger the organization, the greater the commitment to widely accepted, mainstream project management methods. As new methods enter the mainstream, for example Agile, new managers and outsourcing organizations sell their embracing of the methods as a reason for hiring and contracting managers to feel confident in the decisions they are making. Project management has definitely succeeded in forming a thick layer of obfuscation over the reality of software development, quality and security. The only method of project management that has ever delivered "good" results for software is the classic method of wild over-estimation. See this for one of the conceptual flaws at the heart of software project management, see this for an overview and this for a comprehensive look.

Education and Certification

Nearly all professions have education and certification requirements, from plumbers and electricians to doctors. Software organizations grew to embrace imposing education and certification requirements on their people as a sure method to achieve better outcomes. This came to apply not to just the developers themselves but also to the QA people, security people and project managers. Unlike those other professions, computer education and certification has been distinctly divorced from results.

Standards, Compliance and Auditing

Standards were developed early on to assure that all implementations of a given software language were the same. As software problems grew, standards were increasingly created to solve the problem. The standards grew to include fine-grained detail of the development process itself and even standards that specified the "maturity level" of the organization.

Just as a new house is inspected to assure that it complies with the relevant building code, including specialists to inspect the plumbing and electricity, the software industry embraced a discipline of auditing and compliance certification to assure that the organization was meeting the relevant standards. A specialized set of standards grew for various aspects of computer security, with different measures applied to financial transactions and healthcare records for example. The standards and compliance audits have succeeded in consuming huge amounts of time and money while having no positive impact on software quality and security.

Suppress, deflect and Ignore

Everything I've described above continues apace, intent on its mission of making software development successful and computer security effective. People who write code spend much of their time engaged in activities that are supposed to assure that the code they write meets the needs in a timely manner and has no errors, using languages that are supposed to help them avoid error, surrounded by tests created by certified people using certified methods to assure that it is correct. Meanwhile highly educated certified computer security specialists assure that security is designed into the code and that it will pass all required audits when released.

How is this working out? In spite of widespread information blackouts imposed on failures, enough failures are so blatant that they can't be suppressed or ignored that we know that the problems aren't getting better.

Conclusion

We're over 50 years into this farce, which only continues because the clown show that sucks in all the money is invisible to nearly everyone. The on-the-scene reporters who tell all the spectators what's happening on the field can't see it either. What people end up hearing are a series of fantasies intended to deliver no surprises, except in those cases where the reality can't be hidden. Mostly what people do is things that make themselves feel better while not making things better. The time for a major paradigm shift to address these problems has long since passed.

If You Care About Good Software You Should Care About Documentation

I recently posted what I thought would be my least-read post ever. It's about the subject that has by far the greatest spread by far between "something tech people say is important" and "something tech people avoid thinking about or doing."

There are many reasons why this is the case. High on the list is the fact that documentation occupies "below zero" status on the list of things programmers and their managers actually care about. I'm also guilty of this. Look at this detailed post I wrote about the hierarchy of status in programming and you'll see that documentation is nowhere mentioned!

Here's a metaphor to help you understand the importance of documentation. Suppose an important election is on the horizon and for various reasons an unprecedented number of people are likely to vote by mail instead on in person. Suppose that there's a large region in which USPS mailboxes have been installed in scattered places. Suppose that a security failure has been discovered in the design of the mailboxes that makes it quick and easy for a trouble-maker to open it and remove all the mail, including of course any ballots. Word of this vulnerability seems likely to leak out, tempting activists to raid mailboxes where ballots  cast for the party they want to lose are likely to be found. It's a bug! It has to be fixed immediately!

In the real world, there are probably lists of the locations of all such public mailboxes. Even if there aren't, there are USPS employees who visit the boxes regularly and know where they are. Failing everything else, the public can be asked to register the location of any boxes they know about. No problem.

In the wonderful world of software, things are entirely different. The mailboxes and everything around them are invisible to normal people. Members of the public don't "go there." USPS employees don't go there. The original contractor who installed each box at various times may have been required to provide documentation of his work, but the original documentation was incomplete and full of errors, and was never updated for subsequent additions and changes. What's worse, the boxes are darned hard to find. Practically no one has the right kind of eyes and training to even be able to correctly recognize a box when driving slowly down a road looking for them.

If you're a modern, with-it programmer you might be thinking to yourself at this point "hah! That wouldn't happen with my code -- I use modern micro-services, so I'd just have to go to the right service." Uh huh. What if the bug had to do with some data that was defined incorrectly? Do you have a DBMS? Do you have a UI? Does every piece of data really appear and get used exactly once in exactly once place? Is it really so easy to not only find each instance of the data but all the downstream uses and consequences of the data that are impacted by the error?

Regardless of how your code is organized, finding and fixing a bug can take a depressing amount of time, and part of the time is often the result of not having comprehensive, accurate low-level documentation.

In the relatively simple and visible-to-everyone real world of paper ballots and mailboxes, we know there are errors and faults, some of them extensive. Bad things can and do still happen. Because of the simplicity and visibility of the normal world the faults are often noticed quickly, like when bad ballots are sent. In the incredibly complex and visible-to-few world of software, the faults can go for long periods without even being noticed, and when they finally surface it can take the tiny number of super-specialists with the right training, vision and persistence a long time to find and fix the bugs lurking in the vast spaces of the largely invisible, undocumented oceans of software we all depend on.

Documentation is unlikely to improve in the world of software because practically no one, despite the sounds that may come from their mouths, really cares. The good news is that better approaches to building software than are fashionable today go a long way to minimizing the trouble. The more we move towards Occamality, the more we'll get there. The reason is simple: if everything in a program is in exactly one place instead of scattered redundantly all over the place, at least you'll know that there's just one fierce monster bug out there and not an army of clones.

The Universally Ignored Disaster of Software Documentation

Nearly everyone knows what “click bait” is – an article title that strongly tempts readers to click and read the article. Part of being a good writer/editor these days is developing a facility for click bait.

My proposal for the most click unbaitable or click repulsion title is anything that includes the words “software documentation.” Is there a more boring subject on the planet? Even to software people? Or I should say especially to software people?

It’s too bad, because software documentation is a genuinely important subject. Every ounce of effort put into it is not just an ounce of wasted effort, but many ounces of waste by side effect. Software documentation is the subject of near-universal complaints, but few dispute its irreplaceable importance in the cycle of “responsible” software development, from requirements all the way through testing and support. We should be talking about software documentation – and documenting (heh…) its odious nature.

Software Documentation

Documentation is the backbone – or ball-and-chain – of software from start to finish. The time, effort and resulting bulk of documentation normally vastly exceeds that of the software itself. The documentation is supposed to define both generally and precisely WHAT is to be built, HOW it is to be built and tested, and then what was actually built.

If you are in a normal corporate environment, this all ends up being HUGE. If you’re in a regulated environment, it’s so big it’s amazing anything gets done. Here’s a little snippet of just some of the documentation required for medical devices by the FDA.

As you can see, you even need a document that has the plan for documents! See this for more detail and explanation.

Documenting the problems with documentation is an immense job, and given that no one cares about documentation one way or the other, I’m not going to do it. But here are a few of the lowlights.

• Requirements documentation is supposed to be the foundation of a software project, defining in MBA-understandable terms the what, why and how of the software effort to be undertaken.
• In reality, most requirements documents fall somewhere in the range between useless and ignored.
• Requirements documents are kind of like the plan to discover a new sea route to Asia in the Europe of 1490, which was basically using a lot of words to say “sail west.”
• Requirements documents are never updated to reflect what actually happened, beyond brief memos saying triumphant things like “we sailed the ocean blue and hit land in 1492!”
• Architecture documents are typically produced by people who won’t end up writing the code.
• Architecture documents resemble someone on a roof top in Kathmandu pointing mostly east and a bit north, waving and jumping around, saying excitedly with lots of words – go there and keep climbing up and up and up, and you’ll get to the peak of Mount Everest!
• Often the climb-Everest plan contains huge amounts of gruesome detail -- all written by someone who's never been to base camp, much less climbed.  
  
• The people who write the code may or may not pay lip service to the document. They certainly won’t waste time updating it with what actually happened.
• More detailed design documents may be written by people who sit in the same general area as the people who write the code.
• In the end you have to make the code sort of work, and there are time pressures. Never has a manager ever said “and I want the updated version of the design document BEFORE you release the working code.”
• Everyone cares about quality, of course. So there are test documents of all kinds, even more in a test-driven-development environment.
• Is automated testing ever actually comprehensive in the real world? Has any manager insisted that updated test plans be created before the automated test is revised? I mean, of course, a manager who keeps his job?
• Code itself needs to be documented. How are programmers who need to dive in and fix things and/or make changes supposed to figure things out without it?
• Every programmer complains about the lack of relevant, accurate documentation when they dive in to a new code base to get something done. That same complaining programmer, of course, takes extra time at the end to carefully document what he’s done and the things he learned along the way. Sure.
• One of the reasons why having code in multiple layers and scattered among loads of little microservices is so attractive to modern-thinking people who think of themselves as Computer Scientists (or at least Engineers) is that when you’re looking for something about the handling of the data you’re trying to change, the code (and related documentation – hah!) is in lots of different places with no good way to figure out what’s where – or even what language it’s in! I bet you never knew the perversity behind this modern thinking, did you?
  

Responses to the Documentation Disaster

It’s been a long time since I’ve seen a new response to the ongoing disaster that is documentation. Here are some of the typical responses.

• Real programmers like to complain about documentation. Regularly.
• They complain when documentation they think should be available to them doesn’t exist.
• When it exists, they complain about how much time they wasted in a failed attempt to get anything useful from it.
• When someone tells them they have to write documentation, they have a list of compelling reasons why higher priorities won’t let them.
• When they lose and have to write, they complain about what a waste of time it is, and how useless their work will be after the next round of major changes to the software.
• Ambitious programmers learn to promote documentation
• Most important promotion decisions are made by people who are completely clueless about software
• One of the traits most highly valued by these ignorant managers is the ability to “communicate.”
• Communication with the clueless normally means having reports, project plans and other documents with enough plain language to seem relevant with lots of acronyms to convey credibility.
• Ambitious programmers emphasize adherence to standards to convey the appearance of being risk-averse; standards tend to include piles of documentation
• Since it’s all a charade in the end, ambitious programmers learn to take short cuts and make it clear to underlings that document content quality, accuracy and completeness are weights that can be shed in the race to completion.
• Has any top manager ever done more than glance at the thick pile of project documentation, assured themselves that it was appropriately thick, and nodded "good?"
• Managers nearly always have documentation on every project plan – otherwise they’d look like they didn’t know what they were doing.
• Experienced managers know it’s a sham, and that it’s documentation that will be squeezed out when time is running short.
• Managers working on projects that fall under government regulation know they might have to pass audits, and so make sure to produce voluminous documents with all the right titles, headings and sections.
• Since auditors don’t actually know anything about software, all they do is make sure the documents look like the sort that normally passes, and grade accordingly.

In the end, real programmers know that there is no such thing as good documentation. The only thing that matters is the code. Of course, “good” code is supposed to have comments embedded in it. Some programmers do this. But the second the code is changed...

Conclusion

 All this craziness stems from the fact that software is invisible. When things are visible, we're not documentation-crazed; for example, If you're having a house built, when the project was underway would you ask to see the updated plans or would you visit the job site? Along with project management and other standard aspects of software development, documentation is one of the reasons why software takes 10 to 100 times more effort to write than it could take. There is a simple solution. For better or worse, the solution is never taught, but is re-discovered on a regular basis by small groups who write great software quickly and well. Typically the groups who discover the solution have little time and less money, but are motivated and desperate to build a good solution quickly. So they avoid project management and don’t write more than a couple pages of what-are-we-doing type documentation at the start – and don’t write anything else. Really! NO documentation!

This makes perfect sense.  If you were running away from a crazed killer, would you stop to make sure your tie was properly tied?

For more constructive suggestions about how to build software better, see my books, particularly Software Business and Wartime Software.

The Iowa Caucus Software Problem Exemplifies Widespread Software Management Problems

Everyone who’s even vaguely in touch with the headlines knows there was a problem getting results from the Iowa caucuses – a problem blamed on a software app built to automate reporting voting results. No one, including the maker of the app, disputes there was a serious software problem.

Most of the commentary has focused on steps that should have been taken to assure that the app worked better than it did. The Wall Street Journal quoted various “experts” who stated – authoritatively, no doubt – that “testing could have prevented” the problems.

What’s not being said is the emperor-has-no-clothes “secret” – our society is irrationally obsessed with software, and acts convinced that getting new software to perform some function that is now getting done with older software or without software will make things better, be worth the trouble and have no downside. Otherwise sensible and experienced people continue to ignore the ongoing, rolling disaster of software building and deployment, and blissfully welcome new software nightmares, as though there’s no chance that anything could go wrong. The Iowa Democrat caucus face-plant is just the latest of a decades-long parade of software disasters. To make it explicit: I'm using this highly visible event as an occasion to illustrate a general, widespread problem; it has nothing to do with Democrats or politics.

The 2016 Caucus Software

Both Democrats and Republicans hold Caucuses in Iowa. Each party sets its own rules and changes those rules as it sees fit. They even do the voting a bit differently. Nonetheless, the parties cooperated to have a single piece of software that would service all their needs. They jointly announced this in the Jan 31, 2016 Des Moines Register:

They also had phone backup in case of software problems. The net result is that the Democrat Party chairman announced final results:

The software wasn’t perfect, of course – software never is. But it got the job done:

The 2020 Caucus Software

In the summer of 2019, the Iowa Democrats decided that, instead of upgrading the software that worked for them last time, they would pay for brand-new software to be built by a tiny, brand-new firm run by a non-programming History major who’d had a role in the tech of the 2016 campaign for Hillary Clinton. New firm. New software. Has to work out of the box, with over 1,700 people using it for the first time. No problem, right? That’s what the chair of the DNC proclaimed just hours before the caucuses opened.

Many people at the caucuses had trouble using the software, and in the end, it just didn’t work – it wasn’t able to load the results it captured from the people who were finally able to use it into the DNC reporting database. An analysis shows the kind of amateur-hour flaws you'd expect from the kind of group that built it.

Of course they forgot or ignored, like most people do, the train wreck of so many high-visibility new software unveilings, illustrated well by the ACA software releases of 2013 such as the one in Oregon I discuss here.

The Curse of Technology

Technology can be a wonderful thing. I’m grateful for the good things it delivers. But even relatively simple, physical technology can have massive problems. Did you know that 3,613,732 people have died in motor vehicle accidents in the US alone between 1899 and 2013? In 2016, 37,461 people died, as the carnage continues unabated. Just to put that number in context, the total number of US military deaths in all wars starting with the Revolution is well under half that number.

Does this surprise you? Here’s an example: during the entire period of the Vietnam war, the US had a total of 47,424 military deaths. During EACH year in the period 1966 to 1973, there were more US traffic deaths than the total for the war, with a high of 53,543 traffic deaths in 1969. You know which category of numbers people paid attention to.

The problems with computer and software technology massively dwarf those of physical technology, by huge factors.

Physical bank and stage coach robbers make for good visuals and press. One of the most famous was Willie Sutton, who spent decades in jail for having gotten about $2 million from all his hold-ups over many years. Let’s introduce computers and software to make things better! Sure. Banks lose over $10 Billion a year due to computer-driven credit card fraud alone! Banks vs. banking software? No contest.

Loads of people in the industry are promoting the latest software fad, for crypto-currency, like Bitcoin. It’s supposed to be super-secure. Sure. It turns out to be even worse, as the mounting software failures and losses demonstrate.

What all of this means is that once a piece of software does what it’s supposed to do, in production and at scale, no sensible person even thinks about replacing it with a whole new application. The reality is that most of our bedrock software systems took years to build, are often decades-old – and continue to work solidly and reliably.

An excellent example of this is the body of COBOL code produced by the small software company Paysys. When I was CTO of the company, in the late 1990’s, it ran over 150 million credit cards, including for Citibank, GE Capital and other name brands. The company was bought by the world’s largest credit card processor, First Data, and an incrementally evolved version of the same code now runs over 600 million cards, more than any other piece of software. The code is nearly 30 years old! It’s written in “obsolete” COBOL! There have multiple major attempts to re-create it using “modern” technology, sometimes with efforts taking years and costing tens of millions of dollars. Failures, every one.

The reality is that software is hard to build and get right, even today, after decades of time to figure it out. If we were as good at building bridges as we are at software, no one would dare drive over one. The big tech companies, for all their glittering reputations, are lousy software builders. When they try to build something new, they usually take years and end up failing – which is one of the reasons they acquire so many software companies, companies that in many cases have somehow managed to build software that they couldn’t. Here are details about Facebook’s ineptitude. If it's so hard for tech giants, chock full of super-bright nerds, why should anyone expect brand-new software written by a tiny group of amateurs to perform well the first time, with no testing or training, and in a national spotlight?

Building Software is different than building other technologies

People make decisions about software that, if it were any other technology, they would never make. In fact, they would consider those decisions to be beyond stupid.

Suppose you needed to get 1,700 people to an important public event from scattered places. While they didn’t all have to arrive at exactly the same, they had to be there within a couple hour window. Without fail. What would you do? You might consider asking a couple of established limo companies with fleets of reliable cars and experienced drivers to get it done. If a couple people insisted on driving themselves, you’d make sure their cars were OK and they had a backup plan. You might arrange for some car-pooling, or bring some people to a central spot and arrange for an experienced bus service with professional drivers. You might do any number of things.

Let’s further suppose you’d done something like this once, but when you did it again, you wanted to do some important things differently. Suppose you needed to make two trips on the same day. You would probably make some changes to what you did last time, while adjusting for whatever didn’t work perfectly before, right?

There are lots of things you wouldn’t even think of doing. Would you do any of the following?  Design, build and distribute brand-new vehicles for doing the transporting? Contract with a tiny, new company with no real track record to design the vehicles from scratch, not even modifying an existing model that works? Distribute the vehicles with no real test-rides? Give them to each of the 1,700 people to drive themselves? Given that the vehicles are new, not built according to standards, expect the new drivers to drive themselves with no training and no help? When the people have trouble driving, have no help available? When the vehicles break down, have no back-up plan to get the job done? When the disaster drags on for days, still be unable to fix it?

No, I don’t think any person would consider making these kinds of decisions with physical-world technology. The very thought would be ludicrous. But this is exactly what happened with the 2020 Iowa Caucus software! Endorsed and supported from the top of the DNC, and not questioned by any of the Iowa leaders!

Conclusion

There is a problem with how we think about and build software. A big problem. It’s nearly universal, and shared by normal people and supposed software experts. What happened at the 2020 Iowa Caucus technology is exceptional only in that it was so highly visible, which is why I use it as an example. The insane decision-making process that took place there takes place every day, from the way normal people and organizations work with software to the way tech giants and software so-called experts work.

The small groups of people who build software effectively and well are largely ignored by experienced managers and organizations. But these are the ones who are most likely to create amazing new bodies of software that change industries, before their software is absorbed and subsumed into the larger organizations who try to avoid breaking it too badly.

The mainstream thinking about software, both by managers and by software experts, is badly flawed. Things won't get better until fundamental things are changed.

Barriers to Software Innovation: Radiology 2

Value-creating innovations are rarely the result of a bright new A-HA moment, though an individual may have that experience. A shocking number of innovations are completely predictable, partly because they've already been implemented -- but put back in the vast reservoir of ready-to-use innovations, or implemented in some other domain. This fact is one of the most important patterns of software evolution.

Sometimes the innovation is created, proven and fully deployed in production, like the optimization method Linear Programming, which I describe here. In other cases, like this one, the innovation is built as a functioning prototype with the cooperation of major industry players -- but not deployed.

In a prior post I described how I went to the San Francisco bay area in the summer of 1971 to help a couple of my friends implement a system that would generate a radiology report from a marked-up mark-sense form. We got the system working to the point where it could generate a customizable radiologist's report from one of the form types, the one for the hand. Making it work for all the types of reports would have been easy -- we demonstrated working software, and wrote a comprehensive proposal for building the whole system. It was never built.

True to the nature of software evolution, the idea probably pounded on many doors over the years, always ignored. But about 10 years ago, a pioneering radiologist in Cleveland came up with essentially the same idea. Of course, instead of paper mark-sense forms, the radiologist would click on choices on a screen, and would usually look at the medical image on the computer screen. This enabled the further benefit of reducing the work, and letting doctors easily read images that were taken in various physical locations. Tests showed that doctors using the system were much more productive than those who worked in the traditional way. Finally, they decided that mimicking the radiologist's normal writing style was a negative, and that the field would be improved by having all reports follow a similar format, with content expressed in the same order in the same way. This was actually a detail, because the core semantic observations would be recorded and stored in any case, enabling a leap to a new level of data analytics. It also, by the way, made the report generation system much easier to build than the working prototype we had built decades earlier, which enabled easy customization to mimic each radiologist's style of writing.

The founding radiologist was a doctor, of course, and knew little about software. He did his best to get the software written, got funding, and got the system working. Professional management was hired. My VC group made an investment. Many people saw the potential of the system; it was adopted by a famous hospital system in 2015. But in the end, the company was sold off in pieces.

Nearly 50 years after software was first written that was able to produce medical imaging diagnostic reports quickly and reliably while also populating a coded EMR to enable analytics, the system is sitting in the vast reservoir of un-deployed innovations. It can be built. It saves time. It auto-populates an EMR.

Many people have opined on why this particular venture failed to flourish. It's a classic example of the realities of software innovation and evolution. The reasons for failure were inside the company and outside the company. For the inside reasons, let's just say that the work methods of experienced, professional managers in the software development industry lead to consistently expensive, mediocre results. Nonetheless, the software worked and was in wide production use, delivering the advertised benefits. For the outside reasons, let's say that, well, the conditions weren't quite right just yet for such a transformation of the way doctors work to take place.

The conditions that weren't right just yet for this and uncountable other innovations add up to the walls, high and thick, behind which a reservoir of transformative innovation and "new" software awaits favorable conditions. In other words, the reservoir of innovations wait for that magic combination of software builders who actually know how to build software that works, with a business/social nexus that accepts the innovation instead of the standard no-holds-barred resistance.

Corporations promote what they call innovation. They are busily hiring Chief Innovation Officers, creating innovation incubation centers, hanging posters about the wonders of innovation, etc. etc. They continue to believe the standard-issue garbage that innovation needs to be invented fresh and new.

The reality is that there is a vast reservoir of in-old-vations that are proven and frequently deployed in other domains. All that's needed is to select and implement the best ones. HOWEVER, a Chief Innovation Officer is STILL needed -- to perform the necessary function of identifying and breaking down the human and institutional barriers that have prevented the in-old-vations from being deployed, in many cases preventing roll-out for -- literally! -- decades!!

Managing Software that's Invisible to You

Many things follow from the fact that software is literally invisible to most people. If you can't see it, how can you manage it? As I've detailed before.

In some cases, normal people can, at least, see some evidence of the software, like for example when it's an app or a web site. They may not choose (!) to try it themselves, but it's not too much of a stretch to think that they could.

But for some important classes of software, that's not really an option.

As usual, Dilbert explains it to us:

Note that the big boss has no easy way of knowing whether the software is "done." If it were a new web site, perhaps he could navigate to it. But what if it's a new internal process management system? Even a new customer service routing system? You could walk into a service center and see lots of people on the phone. Just like before. Is the new software done? How would they know?

Is the operating system upgrade done? How about the migration of certain applications to a virtualized environment? No one except the internal people would know the difference, unless of course a disaster happened! Hmmm... Given the track record of these things, maybe the fact that everything continued to run smoothly, with no break at all, means that nothing was changed!

There is a solution. It's pretty simple. The solution is to acknowledge that knowledge of the substance of software is more important -- by far! -- than supposed "knowledge" of generic management techniques as taught in business schools.

Coal mining

Knowledge of substance trumps knowledge of "management" even when you can see what's going on!

My great-great-grandfather was James Law McMillan. He was born in Wanlockhead, Scotland in the 1830's. After he came to this country, he went to work in the anthracite coal mines in the Pittston area of eastern Pennsylvania. He knew how to read and write, but had essentially no schooling. Here's part of his 1907 obituary:

He knew mining -- because he was a miner. He was there, deep in the pits. No one can make the argument that mining and coal is invisible in the way software is, but James was promoted because he could get it done.

Perhaps we could learn a little from the experience of the Pennsylvania company. Get people who can do the work to do it. When you find one like this: "with such energy did he labor and such an intelligent interest did he take in his work, that he soon acquired an excellent practical knowledge of mining," that person should take on management.

In software, most managing is done by people to whom the software is invisible. And what happens too often is that the people who aren't good at programming become managers so they don't have to do it any more!

Maybe those people who ran coal mines are worth emulating, at least in this respect.

How to Get A Software Job

There is lots of advice from authoritative places about how to get a job. Any job! Let's apply those principles to software jobs. If you're a great programmer and the techniques work -- you don't want to work there! If you're a software manager and the techniques work on you -- I hope you work for the government or a big corporation, you'll do well.

How to Interview Well

There are lots of places where you can find essentially the same advice, but a recent article in the Wall Street Journal did it well. Here's the key -- and note, it's hot off the press!

Rapport, huh? Sounds suspiciously like what a car salesperson does, your instant new best friend. Who are experts in this rapport thing? Somehow, I suspect it's not top programmers:

Wow. And "researchers" have found this. So it must be true!

Here's a lady in a professional job. Where'd she learn her key skills? Tending bar. So much for the value of computer science!

One thing that's recommended is using humor in the interview. Here are a couple models to follow:

The scary part is the study referenced in the WSJ article, in which ratings for competence were strongly influenced by the extent to which rapport was built in the first couple minutes of the interview -- all other things being equal, you got higher marks for competence by the interviewer if you were an effective schmoozer. Sad.

There are Alternatives. Winning Groups use them!

I strongly suspect that the author's advice is excellent. Aside from my snark about being proven, it's true that most software interviewing is mostly about interpersonal relationships. Groups that famously try to do it differently need to try again. Here's my analysis.

It doesn't have to be this way. Do it the Joe Torre way! It's all about the substance, people! Just get your head out of software and look at other substance-centric fields. All will be clear.

Software Management and Relationships

The worldly and wise among us repeat without end: relationships and getting along are everything in the world of work. I grudgingly admit that they are sort of right. But I also insist that groups of highly technical people with highly technical jobs, like programmers, outshine the competition by 10X or more by following a different set of rules.

Yes, relationships are important. But high-performing software people build real relationships based on the common substance on which they work -- software -- rather than interpersonal junk.

Sick Sheep in England

This isn't just about software. It's about knowledge and substance vs. interpersonal relationships. There's a section of Thomas Hardy's novel Far From the Madding Crowd which illustrates the point nicely.

Bathsheba has inherited a farm, and she knows little about farming. She is totally focused on her changed circumstances and her status relationships with the people around her. Because of his own troubles, a neighbor named Gabriel Oak has been working for her. Gabriel has deep knowledge of sheep. Bathsheba disapproves of the way he relates to her, and dismisses him. Then her sheep break through a fence and start eating young clover, which makes them very sick. They start to die.

Her men tell her that only Gabriel knows how to pierce their stomachs and cure them, but she refuses to ask for his help. Then another sheep dies, and she asks him to come. He refuses until asked "properly," with respect. She asks, he comes, he operates on all the sheep. The day is saved.

Bathsheba has learned the important lesson that -- substance and knowledge matter! -- and so asks him back and he accepts, now that his technical skill is respected, and front-and-center.

Sick Sheep in Software

I have seen far too many situations in software groups run by Bathshebas who have yet to go through a dying-sheep episode. I have also seen far, far too many situations in software groups in which software sheep are dying by the day ... and the Bathsheba in charge simply refuses to change her or his attitudes and reconcile with the equivalent of Gabriel Oak.

Conclusion

Do you want to win an award for most congenial software group? Or do you want to have a software group that bonds over the work it's doing, and hits it out of the park? This isn't a binary choice, but you definitely do have to decide your priorities. If someone is marginally productive and starts getting nasty, it's time for them to go. If someone is a great producer, you find ways to make the nerd happy and fulfilled as well. By doing great work. This is a tough subject and an important one. For more, see this. For lots more on software people, see this.

Managing What you Can't See

How can you manage something you can't see? Simple: you can't.

Can you see software? It depends. If you're a programmer, you can definitely see it -- assuming you know the language. Can you see written human language? Only if you know the language. Yes, you can literally "see" it, but what difference does it make if the page with the language is completely meaningless to you? What if you couldn't read it, but couldn't even speak it -- not just be unable to speak that language, but any language? You'd be in deep trouble, wouldn't you? Suppose you were a manager; would you be able to "manage" a bunch of writers, using the best management techniques taught at Harvard Business School? What a ridiculous thought!

But that is exactly the situation that people who are unable to write or read computer programs in any language find themselves when called upon to "manage" programmers. Here's what's truly amazing: it doesn't seem to phase these genius MBA types at all! Of course I can manage programmers, they say -- I can manage anything!

There's something about software that brings out the inner stupid in managers who can't program. Here are some details and examples.

There are innumerable consequences of attempting to manage something that is invisible to you. Here's a good illustration:

It's the classic problem of setting expectations. It's the reason why I assert that, in software, dates are evil.

At the same time, I fully acknowledge that programmers, at least good ones are a different breed. It's such a big deal that I've written a whole book about it.

Just to take a trivial illustration: take a look at the Dilbert cartoon again. Notice anything? Anything jump out at you? Well, something in that cartoon jumps out to programmers. Every Dilbert cartoon is on-line and accepts comments. Here is one of the comments written by a programmer, who clearly goes right to the heart of the matter:

You think it's trivial? If so, I advise you to continue avoiding learning anything about programming. You won't make it if you try. And get some humility on the subject. Maybe you should try out some concepts that are actually appropriate for software, as opposed to jammed onto software because they're general management techniques that "work for everything." See this for more.

Software people: You get what you pay for

What's more important: programmers who are really good, really productive and consistently turn out high-quality code that helps the business? Or software managers, whose technical skills are not at issue, and whose job is to hire and manage programmers? Or, if they're really important, they hire managers of managers. Based on where most put their money, it's the managers we value most.

If you want things to get better in software, you may want to consider rewarding the people who actually, you know, do the work.

Management bias

Who's in charge of your organization? Let me guess ... a manager. Am I right? So when it comes down to the important things, who do you think gets valued the most highly ... other mangers, perhaps?

Most organizations have a pay scale that rises with increasing "responsibility," i.e., how "high" you are in the management hierarchy. I use quotes here because I'm deeply cynical about those things. Since when do the top people truly take responsibility for things, instead of blaming some poor underling? And height in the hierarchy implies goodness and value. 

Look at this fascinating list of the best-paying jobs:

This list is clearly incomplete because it doesn't have things like CEO on it. But still, you'll notice that the people at the very top are individual contributors, doctors and lawyers. Right behind them are a couple kinds of technology managers. Then a "strategy manager." What about the next one, Software Architect? Sadly, in most organizations, the person with that title may not manage many people, but is considered way too "important" to actually write code, you know, like the peons do. Then we've got our first actual value-producing job category, an engineer! Not software, though. Of the last four jobs listed, three are explicitly management, and the fourth, "solutions architect," is a person who wouldn't be caught dead lowering himself to writing code.

It doesn't have to be this way

Many interesting organizations don't work this way. One example I've explored in detail is baseball, where the players are valued most highly, and where the vast majority of the managers were themselves excellent players, usually for many years.

In more detail, here are some of most highly paid managers:

Here are some of the most highly paid players:

Notice a difference in pay scale? Baseball has clearly decided that while managers are important, it's the players who win games. What an idea!

Conclusion

If what you think your organization really needs is lots of managing, and you figure the software will somehow get itself written, then you should pay your people like everyone else seems to. But if you actually need high-quality software that works and meets your needs, you may want to consider a different strategy.

p.s.: If you want to learn more about software people, see my book on the subject.

Software Business and Business School

What they teach you in business school does NOT help you running a software business. In fact, it probably cripples you.

The good news, sort of, is that most businesses are run by b-school principles and values, if not actually by b-school grads. So as a b-school grad running a software business, you won’t be a stand-out; you’ll be limping slowly along with the rest of them.

But what if you want to excel? Hit it out of the park? Then, I suggest, it would be worth your while to go against the grain, ignore the mainstream “wisdom,” learn what makes a software business different from other businesses, and execute on it.

Software CEO’s

Forget about what academics say. Let’s look at some prime examples.

Larry Ellison

Larry Ellison is a co-founder and CEO of Oracle. He’s one of the richest men in America. He didn’t graduate from college, but worked as a programmer. No formal business training.

Bill Gates

Bill Gates is a co-founder of Microsoft. He’s the richest person in the world. He started programming in high school, and didn’t graduate from college. No formal business training.

Mark Zuckerberg

Mark Zuckerberg is a co-founder and CEO of Facebook. He’s very, very, seriously rich. He started programming in middle school, and dropped out of college. No formal business training.

And so on

I could go on and on.

Enter the MBA

For some of these companies and many others, MBA types eventually enter and run the company. If you’ve gotten really big and successful and just want to coast along, it’s not a bad idea to hire a fancy-pants nanny to come in and keep your children from getting into too much trouble. Why not? You’ve earned it!

But to start the company? Do you hire the nanny to sleep with your husband to get pregnant so “you” can have kids? Not on purpose, anyway! Starting the company, innovating, doing the heavy lifting of hiring and leading the team and building an exciting new company is not something you hire out to an MBA. What’s worse, the MBA has all the wrong skills and applies inappropriate “management techniques” to getting this crucial job done.

The anti-MBA techniques

It’s far easier to be a programmer and figure out the best methods for building a software company on your own than to somehow un-learn, one at a time, all the stuff you learned in business school that just doesn’t work. In addition, you have to have the discipline to just shelve all the lessons you’re dying to apply that are probably good for most businesses – but are somewhere beyond the left-field fences in terms of a software business.

This is a BIG subject … and I’ve written a book about it! Which I’m going to publish! Soon! Yes, I’ll let you know…

Update: It's available. See here for a description, and here for the book.