The Black Liszt

The Government wants to Help Uber's Software Quality

It's reported that New York City's Taxi and Limousine Commission (TLC) wants to pre-approve new software releases by ride companies like Lyft and Uber. Since the TLC is well-known to be heavily staffed with software experts, what can be bad about this idea? Other than just about everything, that is?

The proposal

Here's what they're saying:

Uber and Lyft have to buy smartphones and give them to the TLC because the Commission runs such a tight budget that there's no way it could afford the required thousands of dollars. Oh, wait ... the planned 2015 revenue of the TLC is projected to be $545.6 million, with expenses of $61,045,000. That leaves just $480 million or so, which is undoubtedly already committed to something or other, which is probably terribly important.

Let's assume it happens. How is it going to work? Uber gives a release to the TLC, which takes exactly how long to test it how rigorously by what means? By the time it gets around to organizing to test one release, another will have arrived. So the pressure will immediately come to have fewer, larger releases. Then will come the time when the TLC approves a release and there's a bug. There will be commissions, reviews, and a big operation will be set up to implement industry best-practices, government-style. Things will get even slower and longer, and government tentacles will start weaving their way into Uber's software development organization. In the end, New York will end up getting a small number of releases, way after the rest of the world has them, buggier than everyone else, and the costs will be passed on to the drivers and riders.

Why?

Right. Sure.

The Reality

Governments can't build software that works in any reasonable time. See this.

No matter how hard they tried, software testing in the lab just doesn't work. See this.

They will press to have fewer releases, when more frequent releases are the key to good software quality. See this.

Finally, most important of all, we don't need to be protected, thank you very much. If it doesn't work, people will stop using it, and the company will either fix its problems or go out of business. That's the way the greatest wealth-creating and poverty-eliminating system ever invented works.

The Distributed Computing Zombie Bubble

Distributed computing is a trend whose time has come ... and gone. Well, not completely. If my computers have to ask your computers a question, that's best done using something like "distributed computing." But to be used by a single software group to serve their organization's needs? Fuhgeddabouddit.

The early days of distributed computing

In earlier days, there were lots of computing problems that were too large to be solved in a reasonable period of time on a single computer. If it was important to cut the time to finish the job, you had to use more than one computer, sometimes lots of them. This was frequently the case during the first internet bubble period, for example, when the concept of “distributed computing” really got traction. The idea was simple: in order to serve lots and lots of people with your application, a single computer couldn’t possibly get the job done without making everyone wait too long. So you wrote your application so that it could use lots of computers to serve your users; you wrote a “distributed” application.

It’s always been harder to write distributed applications than non-distributed ones, and of course there’s lots of overhead in moving data from one computer to another. But if you can’t serve your users with a single-computer application, you bite the bullet and go distributed.

Distributed computing today

The most common form of distributed computing lives on today, more often called "multi-tiered architecture." This is when you have, for example, computers that are web servers, front-ending computers that are application servers, front-ending computers that run a database. That's a simple, three-tier architecture. The idea is that, except for the database tier, it's easy to add computers to handle more users, and by doing much of the computing on something other than the database server, you make it handle a higher load than it otherwise would be able to.

There's a more elaborate form of distributed computing that also has a strong fan base, sometimes centered around a service bus. Other people call it SOA (a service-oriented architecture). These are slightly different flavors of distributed computing, often found together in the same application.

Like most ways of thinking about software, the people who love distributed computing learned to love it and think it's right. Period. Just plain better, more advanced, more scalable, more all good things than the stuff done by those amateurs who run around being amateurish.

The impact of computer speed evolution

As I've mentioned a few times, computers evolve more quickly than anything else in human experience. Do you think that the computers of today can handle more than computers could at the time distributed computing took its present form? Is it just possible that, for most applications, a simpler approach than distributed computing in any of its forms would get the job done?

Multi-core processors

We all know about Moore's Law, I hope. But people don't think so much about the impact of multi-core processors. Simply speaking, "cores" put more than one computer on the chip. Physically, you still have a single chip. But inside the chip, there are really multiple computers, one per core, each running completely independent of the others. And the way they've built the cores, you actually get two threads per core -- each thread can be considered a execution of a program. So, in a sense, you’ve got “distributed computing” inside the chip!

Let's take a quick look at one of those chips. Here's one of the latest from Intel.

This is one awesome chip! It's got

• 15 cores, supporting
• 30 threads, and can support
• 1.5TB of RAM
• 85GB/s memory speed, plus
• over 32MB of on-chip cache

This is incredible. In the past, you might have 3 computers on each of 3 tiers, each with a robust 16BG of RAM (who would ever need more??), for a total of 9 computers with about 150GB of RAM. Connected by dirt-slow (by comparison) ethernet. Here, you've got 2-4 times the number of threads, 10X the amount of total RAM, all in a single chip, no bopping around on the ethernet slow lanes required. Who needs distributed computing when you've got one of these babies?!

Conclusion

Clearly, all the folks who regularly attend services at the Church of Distributed Computing didn't get the memo. This is not new news -- except to the SOA and enterprise bus enthusiasts! There's no way mere facts are going to cause them to stray from their life-enhancing faith!

But for the rest of us, it's clear. Use those cores. Use those threads. Make sure there's lots of RAM. And enjoy the numerous, multi-dimensional benefits of the simpler life.

Math and Computer Science in Academia

Math and music are incredibly inter-related, as has been understood at least since Pythagoras. But they are never studied in a single academic department. Math and music are arguably more intimately bound than math and computer science. But math and music are never in the same department, while math and computer science frequently are. Hmmm....

Math and Computer Science are joined at the hip in Academia

Math and Computer Science are so intimately related in academia that they are frequenty part of the same department. This is true at elite institutions like Cal Tech.

Math and Computer Science are in the same department at private liberal arts schools, too, like Wesleyan.

They're a single department at major state universities, like Rutgers.

Same thing as lesser state schools. Here's how it goes at Cal State East Bay.

I make no argument that this is universal. Don't need to. If you search like I did, you'll find that putting math and computer science in a single department is a common practice.

Why are Math and Computer Science so Academically Intimate?

Most people seem to think that math and computer science are pretty much the same thing. Consider this:

• Most "normal" people who try either of them don't get very far.
• The people who are way into either of them are really nerdy.
• If you're good at one of them, there's a good chance you'll do well at the other.
• They are incredibly detail-oriented. They're full of symbols and strange languages.
• What you do doesn't seem to be physical at all. What are you doing while programming or doing math? Mostly staring into space or scribbling strange symbols, it seems.
• You can write programs that do math, and math applies broadly to computing.

Meanwhile, there are other remarkably similar things that don't end up in the same department. Consider the "life sciences." They all have loads of things in common. Everything they all study starts life, develops, lives for awhile, maybe has offspring, and dies. DNA is intimately involved. Oxygen and carbon dioxide play crucial roles. But since when have you ever seen a department of botany and zoology? Like never, right? In the humanities it's just as extreme. Ever hear of a department of French and German? Academics already fight enough among themselves without that...

Academics clearly think that math and computer science aren't just similar or highly related. If so, they'd treat them the way they do languages or life sciences. A broad spectrum of academics think they're so interwoven that there are compelling reasons for studying them together. Thus a single department that has them both.

Math and Computer Science, a Marriage made in ????

It's a common practice for math and computer science to be studied together. Obviously, most people have no trouble with the concept. Of all the things to question or worry about in the world, this seems pretty low on the list.

I would like to change this. I'd like to cause trouble where there is none today -- or rather, I'd like to EXPOSE the deep-seated, far-reaching, trouble-causing consequences of the fact that everyone thinks it's quite alright that math and computer science are thought of as pretty much two halves of the same coin. In fact, I will argue that the math-computer-science-marriage is just fine for math -- but the root cause of a remarkable variety of intractable problems that plague software development.

Note that I did a quick shift there. I have no problem with math and computer science being together. They kinda belong together. My problem is that everyone thinks that you study computer science in school so that you're qualified to do software development after graduating. And that software development shops require CS degrees, and pay more for advanced degrees in CS, on the theory that if some is good, more must be better.

I will flesh this out and explain why it's the case in future posts. But I thought throwing down the gauntlet was worth doing. Or at least fun!

High-IQ Programmers: the Problems

You've got really smart programmers. Problem solved, right? NOT! They can have issues, too. Just different ones.

Smart Programmers

Everyone knows that programming is hard. Everyone knows that really smart programmers can be many times more productive than average programmers. Everyone knows that your project's chance of success go way up if you have smart programmers working on it. But not everyone knows that there is a collection of flaws to which really smart programmers are particularly susceptible.

The reason is pretty simple. Programmers are people, and people have problems! But different kinds of people often have different kinds of problems, and truly exceptional people may have problems many of us are not familiar with. A person who's really tall can have problems with bonking his head on entries that most of us don't have. A person who's really famous can have problems having a quiet meal in a restaurant most of us don't have. And there can be a dark side of a really smart programmer's most admirable qualities.

The "Problem" of being good at solving really hard problems

My new book on Software People (here's a description, and here's the book on Amazon) has a whole section on the problems endemic to high-IQ programmers. Here is an excerpt:

Are good at solving hard problems. The ability to solve hard problems distinguishes them from other people. They ignore simple problems. They disdain working on them. When a simple problem can’t be avoided, they go to great lengths to turn it into a hard problem. People who are good at solving hard problems like hard problems, and can find them in places where other people see no problems at all. Sometimes this is a good thing, like when you encounter a genuinely hard problem that can’t be avoided. Smart people get bored easily. Smooth, straight roads are boring. Some smart people will actively change directions and seek out a problem they suspect will be hard because it is hard. Often, smart people seeking hard problems overlook sophisticatedly simple solutions because they are simple, or spend loads of time solving a really hard problem that actually didn’t need to be solved.

If you've got a hard problem, you darned well better have people who are good at solving problems that are hard. Otherwise, you're screwed. But it happens often enough that programmers who are good at solving hard problems are really proud of that fact (why shouldn't they be?). Their self-identity is tied up in that ability. 

What you really want is a programmer who is capable of solving really hard problems, but feels no need to demonstrate that ability unless it's really needed. I've definitely met people like this, but boy are they rare! You're talking about a super-nerd who is amazingly humble.

Climbing a Mountain

Suppose there's a mountain your team has to ascend. There's only one good mountain-climber in your group, and he's an amazing one -- famous for his ability to tackle near-impossible climbs. You and your team are standing at the foot of a mountain. Naturally, you turn to your expert.

Your expert, being an expert, scopes out the mountain. He sees lots of things that the normal people in your group miss. He spots a hard-to-see, tricky path that avoids the tough parts and makes the ascent a piece of cake. He also sees a route that starts out looking smooth, but has a pulse-pounding section that no one could make without his expert knowledge, experience and guidance. And then there are the other routes.

The expert has to choose between these two reactions at the end of the climb:

1. Boy, what an easy climb! That mountain wasn't so tough after all!
2. We got to the top, but we almost died on the way. If it hadn't been for X's amazing skills, we would be calling for helicopters to remove the injured and the dead at this point. Thanks, X!

Choice number 1: X's amazing skills are nearly invisible, because it "wasn't so hard after all" -- but only because X uniquely saw the hard-to-see route that avoided the difficulties.

Choice number 2: X's amazing skills are on full display, demonstrated in vivid 3-D to the team members, as he accomplishes something no normal mortal could pull off.

Hmmmm. Choice 1: make a hard thing simple, which only I could do, but in the end, everyone is left with the impression of how simple and easy it was. Choice 2: take a tough-but-possible route, in which my amazing powers are on full display. The smart person may not even be aware of how his guts and ego are pulling him to Choice #2. It's just human nature.

Conclusion

There's lots more in the Software People book, where this came from. On the one hand, outstanding software people are people. On the other hand, they have issues that are unique to their smartness. You want smart people on your team. Definitely. But you also want to help your smart people be even better than they already are by confronting and overcoming their unique problems.

Internet Driver's Licenses Needed for Users

We give kids sex education. We give them driver education, and require a driver test and license before driving. But we let any fool onto the internet to wreak whatever havoc they can on themselves and others without a second thought. It's time for a change!

Education for Meaningful Use

Education on the basics of how the internet and associated technologies work and how to control, respond to and interpret what you see is totally neglected. There are no significant efforts that I know of to make people educated consumers of this important, ubiquitous service that is so widely used. But there is a more important issue...

Education for Safety

By far the most important subject for internet education is safety. Maintaining internet safety has some similarities to general safety, but is different in important ways.

Internet "driving" safety

The most important aspect of safety while driving is avoiding driving while impaired in any way, and paying sharp attention to the road and other vehicles at all times. Driving while impaired by drugs or alcohol or while engaged in texting or talking
are recognized factors.

So imagine how hazardous internet driving must be when people don't even know how to read the road signs (the URL's) and can't tell that they've wandered onto a road constructed by criminals specifically for the purpose of enabling them to steal your car, drive it to your bank and take out a big withdrawal! But that's exactly what it is! Here's an example of a more brazen attack (image from a good guy, Yoo Security), demanding that you send the money yourself:
Unfortunately, there are criminals out there who have grown far beyond simple smash-and-grab operations. These sophisticated criminals with a long-term view trick you to "drive" onto their criminally-constructed "road" for the sole purpose of making your car an instrument for stealing from other people or organizations. They can make your computer into a zombie to participate in botnets. It can serve that purpose for minutes or years without your awareness. Is the problem big? You betcha. There are more computers that have been hi-jacked into botnets (maybe yours!) than most people are aware of:

Sometimes, of course, the criminals are stupid, greedy or malicious -- I guess those are the drop-outs from the "criminals should be good citizens" certification program. So your hi-jacked device could slow to a crawl, do weird things, look over your shoulder as you type until they get the information needed to drain your bank account or max out your credit card, or even (just because it's fun!) wipe out your machine while leaving some cute "it was me! Have a nice life!" Message on your screen.

Internet E-mail fraud

How often do you get a letter purporting to be from your bank asking you to send them a letter containing your account number just so they can verify that everything's OK? If you got one, do you think you'd respond as requested? Apparently you're not alone -- criminals are the supreme capitalists, and abandon efforts that are unprofitable before long.

But how about letters on the internet, i.e., e-mail? Along with everyone I know, I get an amazing number of criminal solicitations, ranging from the laughable (at least to me) to the amazingly credible every day. Data-driven capitalists that they are, the only explanation for the persistence of these efforts is that more than enough of them work to cover the costs and trouble of running the schemes, certainly more than getting a legal job. I've seen fewer solicitations from Nigeria lately, but the slack has been taken up by Libya.

Here's one of the new breed from Libya:

Here is a somewhat more plausible one from a place that really could be your bank:

Conclusion

Uneducated internet users cause billions of dollars of harm to themselves and others every year. You think this would result in outcry by those users and people who know them for education. You might think this might merit a bit of attention from the institutions who so assiduously and expensively educate, authorize, license and otherwise keep us on the straight and narrow. When I'm in Central Park in New York, there are rangers watching my every move; they set me straight when I ride my bike where I'm not supposed to, or walk in one of the ever-changing restricted areas. The conclusion is obvious: every move I make in the Park is more worthy of watchful restriction by people in uniforms than the millions of actions on the internet that seem, at least to me, far more destructive. I must be missing something.

Software Problems: the Role of Incentives

When lots of human beings work at something for a long time, they tend to figure out how to do it. Building software appears to be a huge exception to that rule. With decades of experience under our belt, why is it that we still can't build good software?

One of the reasons software projects so often fail and improved methods aren't used appears to be that the people involved have perverse incentives.

Incentives

Everyone knows about incentives. They work. Even when we know someone is using incentives to get us to do something, we're more likely to do the thing with incentives than without them.

Perverse Incentives

Whether an incentive is perverse or not is in the eye of the beholder. From the incented person's point of view, an incentive is an incentive, and as we know, incentives work. But we normally call incentives "perverse" when they incent people to do something that most other people would agree is a bad thing.

Perverse Incentives: Mortgages

The housing boom leading up to the financial crash of 2007 was clearly driven by perverse incentives on multiple fronts. Borrowers were tempted to take what seemed to be easy money. Mortgage companies could make piles of money in fees by packaging up risky mortgages and passing them on. Rating agencies could collect loads of fees by not looking too closely. And the bankers at the top of the food chain made themselves lots of money by creating and selling fancy instruments that ignored the underlying realities and the ultimate consequences of their actions. Then it all came crashing down. Many were hurt, the big guys who made the most money least of all.

Perverse Incentives:The VA System

It has recently come out that more than 120,000 veterans are experiencing long waits for care at VA hospitals, even while official reports showed minimal wait times, enabling managers to collect incentive payments. If there ever was a case of perverse incentives leading to bad behavior, this is it.

Perverse Incentives in Software

Software is so rational, so organized, the people involved are so smart and well-educated -- surely perverse incentives aren't driving behavior in software, are they?

Sorry, sweetie, perverse incentives are a human issue. Humans respond to incentives, perverse or otherwise. And as it turns out, there is a rogue's gallery of perverse incentives operating in software -- I will only scratch the surface here!

Estimates

Estimates are perverse all by themselves.

They are also a GIANT BILLBOARD incenting EVERYONE involved in the process to make any estimate as long as they can possibly get away with; and since very few people (often including the programmer involved!) has any idea how long something *should* take, the estimates are typically accepted as is; but then, manager often double the estimates before passing them on. Why is this perverse?

The organization probably would like to get something done in the shortest reasonable time. But the programmers and project people are measured on whether they beat or miss the estimate. The longer the estimate, the better the chances of avoiding failure. It's that simple. It just makes it all the more maddening that, even with inflated estimates, things still go wrong!

Requirements

The whole modern software development process starts from requirements. Gamesmanship around requirements is therefore front-and-central. Estimates are based on requirements, and therefore controlling and fixing the requirements is central to the effort of creating "success." The system may fail, the users may hate it, but if it meets the "requirements," the people running the project get to declare "success." What you'd like is for the project to succeed when the needs of the business are met. The perverse incentive is for the people delivering the system to define "meeting the requirements" and then control the requirements to assure that they're met, regardless of what disasters happen to the business.

False reporting

Just like at the VA, project managers are highly incented to avoid reporting problems -- typically using big fancy reports that are chock full of meaningful-seeming stuff but are in fact just garbage. Just like in the mortgage-driven financial crisis, everyone involved is incented to declare success, take their rewards, and kick the can down the road for the next guy. Eventually, with shocking speed, it all comes crashing down, just like the financial system, and just like the mere 4 days between the laudatory article about how great Cover Oregon was going to be and the admission of total failure.

False Assessments

Here's where the rubber meets the road. Who is incented to blow the whistle on a failing software project? How, when and by whom is a software project judged to have failed? Most importantly: what are the consequences of having failed?

We all know the answer. Who has even heard of a software engineer who was fired for failure to deliver? And the people in charge? Never. It wasn't their fault! And the project didn't fail anyway! The requirements changed every month, the target kept moving, and blah, blah, blah.

Conclusion

Your kid comes up to you and asks, "can I play my video game now?" You briefly think about how your question when you were that age was "Can I go out and play now," but the kid isn't interested, and is bouncing around waiting for your "sure." Being the aspiring adult you are, you act responsibly and ask "Have you done your homework?" There's a brief pause. The kid is doing a quick risk-reward ratio calculation. If he says "yes," he probably gets to do what he wants. But you might ask to check. Hmmm.

This is the breeding ground of perverse incentives. We all learn to balance honesty, openness and getting what we want. Some of us go for honesty and openness, deciding that anything else just isn't worth it. But loads of people make an informed judgment on a case-by-case basis, much like the kid and his homework.

Whatever the morality of the case, the facts are clear: software projects fail left and right, and perverse incentives are a significant factor in making them fail. Without changing the incentives, we're unlike to abandon the Bad Old Way of building software and achieve success.