The Black Liszt

The Name Game of "Moving to the Cloud"

The most recent in a long string of technology fashion trends, "the cloud" is hot. Like its hot technology fashion predecessors, it mostly consists of old ideas with a little spicy sauce on top and fresh packaging. If you mindlessly follow the fashion and just "go to the cloud," you are likely to end up in the same unhappy place where most mindless followers of fashion trends end up. What is "the cloud?" Simple. Clouds are belated enterprise IT implementations of the consumer internet.

What is the Cloud?
Is the "cloud" a new development? Well, it is a new name...

Ancient Clouds
I first encountered the cloud more than 40 years ago. Before that fateful meeting, my only experience with computers had been up close and personal. You had to get in the room, push buttons, flick switches, feed card decks or punched paper tape, listen to whirring sounds and watch blinking lights. Like with this computer:


But The Cloud! Ahhh, the Cloud...I remember it vividly.

Of course, things were a bit different then. We only had "fat clients," as in "takes two guys to lug it" fat. The principle was identical, however: I was in one place, with a "fat client" like this:


...and in another place was a computer like this:


...and everything worked.
Remember the importance of labelling, however: what we did 40 years ago wasn't "cloud computing," which hadn't been "invented" yet -- it was merely "telecommunications."

Creating the Modern Cloud
Between then and now, lots of iterations of Moore's Law have come and gone. All the hardware has gotten smaller, cheaper and faster, while the software has gotten larger, more expensive and slower -- but, fortunately for all of us, the rate of hardware evolution is greater than the rate of software devolution, giving the impression of net progress.

Where this leaves us, 40 years later, is faster remote computers talking with lighter remote clients over incredibly faster networks, all at lower cost. Sprinkle with a little extra software and drizzle with some marketing hoo-haw, and -- kazaam! -- you've got today's hottest technology fashion trend, Cloud Computing.

Clouds aren't always friendly
When we think of clouds, we're likely to think of this kind of cloud:


friendly, fluffy shapes floating in an otherwise sunny sky. When people think about cloud computing, this is the kind of cloud they seem to have in mind. But as we know, there are other kinds of clouds. There are dark, oppressive clouds that make everyone depressed. And there are really mean clouds, that wreck things horribly, creating the situations for which "disaster recovery plans" are made. Is this just a metaphor? Of course not. But just like financial fraud, the big, juicy examples are usually hushed up in order to protect the guilty.

OK, so What is "the Cloud"

There is a little-discussed trend that is deeply embarassing to IT professionals: there is a wide and growing gap between the use of computing technology in the consumer world and in the corporate, data center world. When the average user of corporate data systems is home, he works in a very advanced computing environment. His local machines and devices are amazingly capable and pretty easy to use. When connected to the internet, he can access a nearly limitless world of cloud computing resources -- which are themselves largely run out of data centers that are remote from the people who set up and administer the software in them, and which contain an ever-evolving mix of dedicated and shared resources and services. The consumer internet has been based on a cloud computing model for a long time.

The corporate world is a whole different thing. The corporate world has been consumed with consolidating their diverse data centers. They are finally beginning to confront the extreme flexibility and ease of use that consumers enjoy every day, and are finding it increasingly difficult to explain why the computing they run with such high capital and operating costs are so cumbersome, error-prone and inflexible.

In this context, there is no way that anyone associated with corporate computing is ever going to plainly admit that what they are basically doing is trying to catch up with the consumer internet. So they must be doing something else. Oh, yeah -- they're evolving to the latest, smartest trend in corporate computing, adopting the latest technologies and being really leading-edge: they're "moving to the cloud," but of course in a "smart" way, with large doses of "private cloud" technology along the way.

Summary
What's a "private cloud?" A corporate data center with a fancier name.

What's a corporation "moving to the cloud?" A corporate IT group trying to play catch-up with the consumer internet, and desperately trying to make it look like something else.

What's new about "cloud computing?" Very little; mostly naming and marketing fluff.

Is anything real happening when a corporation "moves to the cloud?" Sometimes yes! Sometimes, they really are copying a couple proven techniques of the consumer internet, slowly and at great cost and trouble, but nonetheless creeping towards a 21st century computing model.

Why Computer Software is So Bad

There are aspects of the theory and practice of computer software that drive me nuts.

I look at a widely accepted theory or practice, and am aghast that the cancer spread and became part of the mainstream. I look at a genuinely good practice and am shocked at the way everyone lies and slithers to make it sound to the unsuspecting that whatever lousy thing they do is a shining example of a good thing. I look at a nearly-universal approach to problem solving that may have made sense many Moore's-Law generations ago, but has long since been rendered irrelevant by advances in hardware. And I look at proven, understandable techniques that improve productivity by whole-number factors that are spurned and/or ignored by most people in the field.

How significant is this stuff I'm complaining about? A minimum of a factor of 10. Sometimes a great deal more. So it matters! This is theoretical, but it has real-world, practical implications. It's the only way, for example, that small companies can beat large, established ones that have software staffs 10 to 1,000 times larger.

I think I understand some of the causes of this deplorable state of affairs. But that doesn't make me feel better. And it definitely doesn't cure the sick or empower the unproductive.

A number of my private-distribution papers go into these subjects in considerable depth. But I thought it might be interesting to summarize a couple of the main observations here.

Among the causes of bad software are:

The blind and deaf leading the blind. In the majority of cases, the people in charge have little to no personal experience of creating software, and no interest in how it's done. This is about as sensible as having someone in charge of a baseball team who not only can't play baseball, but can't see onto the field where it's being played; all he can do is see the score at the end and hear reports from the players about how things are going. So everything turns into politics -- convincing the blind, clueless boss that you're the great contributor, everyone else is a chump. The larger the organization, the more likely it is to be led by such a genius.

Process over substance. The larger and older the organization, the more likely it is to elevate the importance of process, and the more elaborate and all-consuming that process is likely to be. The more process there is, the less code gets written, and the productivity of the innocent few who actually want to work gets ground down to the abysmal norm.

Lawyers. Shoot the lawyers! Shoot them! They and their legal ways are a plague. When lawyers see a problem, they want to write a law or create a regulation that makes the problem go away. Except that instead of saying this in simple, results-oriented terms (e.g. "programmers should not be allowed to die unnecessarily while writing code"), they say it in terms of incredibly elaborate, micro-managing, this-is-how-thou-shalt-do-it regulations (e.g., "programmers should be offered a nutrional meal no later than 12 noon local time each day consisting of no less than..." and 538 similar instructions, constantly growing). If regulations of this kind actually led to good results it would be one thing. All I should have to say at this point is "credit card theft" and PCI.

Fashion over function. Programmers are supposed to be nerds, aren't they? How can programmers let their decisions be made by "fashion," whatever that's supposed to be in the world of software? I refer you to the first point, about software teams being led by people who are clueless. These people want to seem smart in the eyes of their bosses and peers, who know even less than they do. So they make decisions that are guided by C-office fashion trends, which are usually laughably out of step with true optimal decisions.

A preference for bad new ideas over good older ones. How do bad ideas catch on? Some company promotes them. Books are written. Appealing rhetoric is created and repeated. Suddenly a fashion trend is born. Someone can appear to be smart and modern just by advocating the cool new stuff and sounding smart, without actually knowing anything or doing anything. Everyone involved thinks they're helping advance the state of software, when in fact they're digging the hole of despair deeper. It's remarkably like when the dumbest guy in Scotland moves to England, thereby increasing the average intelligence of both places.

A preference for bad old ideas over good newer ones. There aren't many good new ideas; mostly there are old good ideas that are still good. But because conditions change (like processors getting fast and memory getting cheaper), ideas emerge to respond intelligently to the new state of affairs (like this one), instead of blindly and stupidly continuing on as though nothing had changed. Like most people do.

Wrong scope; usually too narrow. This is classic optimization over too narrow a range. Much bad software results from compartmentalism or simply failing to look at things through the eyes of the ultimate consumer. This is an incredibly common mistake. The trouble gets really bad when is the practices that result from the little "silos of excellence" get elevated into industry "best practices." (Whenever I hear the phrase "best practice," it's really hard for me to avoid getting a serious look and pronouncing in deep tones "you folks had best practice a bit longer before you inflict yourselves on the world of paid, professional programming.")

I'm sure there are many more causes of bad software than the ones I've listed here -- we haven't even gotten into bad programmers in their innumerable incarnations! But every item on the list above thoroughly deserves inclusion, even more because most of them are not listed by most people as a cause of software malaise.

Databases and Applications

When databases were invented, they solved a huge problem that couldn't be solved any other way. Anyone who cares to look can see that the original problem that caused us computer programmers to invent databases has largely gone away. So why is it exactly that application programmers reflexively put their data in a database? In a surprisingly wide range of cases, it sure isn't because of necessity. Could it, perhaps, perhaps, be nothing but habit and the little-discussed fact that change happens in software at roughly the same rate that change happens in glaciers?

From the beginning of (computer) time, instructions have needed to be in memory to be executed, and data has needed to be in memory to be operated on by instructions. The memory in which instructions execute and fiddle with data has always been way faster and way more expensive than the large, slow but cheap places they are put when they're not in memory (call it storage, whether the storage is punch cards or tape or disk). It was this way at the beginning of time and it's true now.

Think of memory as your work table. Eons ago, your work table was really tiny, like this:

You can hardly fit anything at all on it! So you'd better have a really big storage place to keep all your stuff, like a pantry:

OK, that's cool. You've got all your stuff in storage, but you can only work on when it's in memory (on the table). What do you need? You need to get the stuff you want to work on now from the pantry, and you need to put the stuff you're done with back in the pantry. In other words (if you're in the world of computers)...you need a database!

The very most basic function of a database is pretty simple: its job is to shuffle your data between memory and disk. It's also nice if it keeps everything straight, avoids dropping bits on the floor, and cleans everything up when something goes wrong during the shuffling.

That was then. But things have changed. Remember Moore's Law? The amount of memory available to us at surprisingly reasonable prices has grown hugely. Exponentially. Our work tables now look more like this:

And our pantries? Well, they've grown a bit too:

So how much data do you have? Run the numbers. It goes without saying that it's going to fit in storage. But how about that work table? Here's the question you have to think about:

Will all your data fit on the work table (i.e., in memory)?

With memory available at reasonable prices for 64GB and even 256GB, the answer is often YES! It will!

Hmmmm. What was it the database does? If all my data fits in memory, why was it I needed that database???

I know, I know. Databases can be nice for reporting and data analysis and "persistence" and a few other things. I'm not saying you never use them. But for your real application, the one that takes user requests and responds to them, if you don't need to have the database shuttling stuff between the work table and the pantry... Hmmmm.

The Xiotech ISE and Technology Fashion

We all know what fashion is. Think of Vogue Magazine, or impossibly tall, thin women walking in that special way down the elevated runway, wearing something no normal person would be able to wear, or would want to wear if they could.

But fashion extends way beyond women's clothing. Let's start with men's clothing: how many guys wear suits and ties to the office today? Then cars -- how many modern cars have those giant fins that were popular in the '60's?  The kind of popular music you like dates you at least as much as wrinkles on your skin. The more you think about it, the more you realize how pervasive fashion is.

"Technology is a counter-example," perhaps you say. "It's bits and bytes and silicon, no fashion there!" Well, that's true, except that it's people who buy the technology, and people are fashion-driven creatures. Let's face it: the cool kids who once drove sporty cars now pull out their iPhones at the slightest excuse. Waiting on line to see the Beatles; waiting on line to get the latest iPhone -- what's the difference? They're both fashion-driven fads.

"I concede that consumer technology is fashion-driven," perhaps you admit. "But hard-core computing technology, where nerds are building things for nerds; how can that possibly be driven by fashion?" I fully concur that no nerd techie would ever admit that his choices, selections and designs are driven by fashion, not even to himself. But all too often, that's exactly what's happening. The techie nerd who comes up with a design approach for solving a problem almost always prides himself on originality and foresight, without any awareness of how fashion-determined his most important decisions are. These decisions are often not made consciously; they are assumptions. "It's not worth discussing, of course we'll take approach X," the techie would respond in the unlikely event that the assumption is questioned -- by some "ignorant" (which is tech-talk for "unfashionable") person. Just to be clear: we're not talking about how nerds dress; we're talking about how nerds think.

There are examples in every field of computing technology. The Java/J2EE fad during and after the internet bubble is an obvious example, and before it client/server computing was a huge fad.

There is a clear example in storage technology today. The fashion is as clear and obvious as short skirts, and moreover is explicitly stated by its adherents: the fashion is that storage functionality should be provided as a body of software, independent of any hardware embodiment, and without regard to any particular storage hardware. Companies that previously sold storage hardware no longer have real hardware design functions -- all they do is bundle their software with hardware provided by others and sell the combination. The most popular form of this approach is to buy drive bays from an OEM and connect them to controllers consisting of off-the-shelf specially configured processors; 3Par and many others do this, for example. IBM's xiv implements a variation on this theme, using all IBM commodity server hardware. While there are still loads of dollars being spent on old-style, hardware-centric storage systems (think EMC), engineers building new storage systems are uniformly following the software-centric fashion.

In this sense, the Xiotech ISE is decided unfashionable. The ISE was invented at Seagate, in response to the CEO, Steve Luczo, wanting to create a storage product that was higher value than spinning magnetic disk drives. The idea was simple: build a fixed-format super-disk, with many Seagate drives, intelligence, etc. It would be bigger than a disk, but smaller than a SAN. It would emphasize basic storage functions (write, protect, read) and leave the "high level" storage functions to the SAN vendors.

What is interesting is that Steve Sicola and a group of other storage industry veterans ended up working side-by-side with Seagate engineers, something they never would have done at a SAN vendor. Sicola and his team knew the evolving fashions in storage quite well: ignore the details of the drives, that's "just storage." Build fancy high-level functions.

But since they were stuck with the drive engineers, they did something unusual: they actually listened to them! They learned about the amazing functions the engineers embedded in the drives that all the SAN vendors ignore. They learned how annoyed the Seagate engineers were at all the drives marked "bad" by SAN's, the vast majority of which are actually good; they learned about error codes and performance details that all the other storage engineers in the industry were studiously ignoring.

Before long, they got absorbed in what you could really do once you really knew the hardware. And, being good nerds, they invented a bunch of stuff, like how to virtualize over a fixed number of heads so that top performance was maintained even when the disks are filled up. They also invented a bunch of stuff that provides major, persisting advantages as new drives with higher capacities come out.

Since I know a fair amount about Xiotech's ISE, I want to go on and on about it. But I won't, because the point of this post is technology fashion. The purpose of bringing up the ISE is that it's a great illustration of the power of fads and fashion in technology. Any normal group of self-respecting storage nerds would have built a completely hardware-independent storage system. As such, it may have had nice features, but it would be pretty much like all the others in terms of its basic functions of reading and writing disks. But because these storage engineers were sequestered with hardware types and had a unique mission imposed from above, they did something very rare: they built a leading-edge storage system that is decidedly unfashionable. Because the engineers actually paid attention to the hardware, the ISE does things (performance, reliability, density, scalability, energy use, etc.) that no other storage system on the market today does, even though it uses the same disks available to others.

Fashion is, of course, a relative term. Fashion is one thing at diplomatic receptions, and quite another hiking in the wilderness or in a war zone. What is appropriate for one doesn't work for the other. Shoes that are appropriate for a salon can cripple you in the woods.

Well, it turns out that the modern storage fashion of ignoring the storage hardware may be acceptable in salon-type environments (where appearance and style is important but there's no heavy lifting to be done), but is as crippling as high heels in the I/O-intensive environments that are increasingly found in virtualized, cloud data centers. The ISE is like storage fashion for war zones of data, for data-intensive applications like virtualized servers, where the applications are concentrated in a small number of servers, all fighting to get their data. Most storage systems know how to hold their tea cups and conduct refined discussions and other things that matter when getting your data sometime today would be nice, thanks.

But when you've got a crowd of rowdy, tense applications all of whom are demanding their data NOW, perhaps more of a war-time style is appropriate; that's what the "unfashionable" nerds at Xiotech created in the ISE.

Paleolithic Mainframes Discovered Alive in Data Center!

A recent article on forbes.com quotes me on what many people find to be the surprising longevity of mainframe computers.

Don’t things in computing just get better and better – not to mention faster, smaller and less expensive? Which implies that after a few years of use, it’s just not worth keeping the old stuff around anymore? So we throw out (oops, please excuse me, we meticulously recycle…) the useless old stuff and bring in the cool, cost-effective new stuff, right?

Like most common wisdom in modern computing, this contains elements of truth, but isn’t quite right.

The element of truth behind this thought is the astounding continued progress of Moore’s Law, which posits that electronics gets smaller and faster at a rate that boggles the mind. This is what gives us iPhones and portable computers that have more speed and capacity than the room-sized mainframes of the past.

But there is more to computing than electronics. First of all, there is this little matter of physical devices that have mass and inertia, that no amount of Moore’s-Law-driven advance will free from the tyranny of the laws of physics. This leads to growing storage problems that Moore’s Law actually makes worse. See here for a description of the issue.

Second of all, there is this thing called software. Yes, software, the invisible-to-the-human-eye “stuff” that makes all that amazing electronics actually do something. Software is really hard, complicated stuff, like most things that are essentially mental, conceptual and invisible (think math). Once some software actually gets working well enough, sensible people are loath to change it. Even worse, the amazing increases in speed and capacity of electronics mask simply awful problems in software.

Building most real, practical production software tends to be a nightmare that rarely ends. Re-building software that more-or-less works is a nightmare in hell that visits all the circles of hell in round-robin. So if the credit card companies can process their transactions, and the software that gets the job done happens to be written in totally-out-of-fashion-squared COBOL that runs best on a mainframe – that’s a great reason for IBM to build a new implementation of the mainframe instruction set out of modern electronics (thus getting most of the benefits of all the advances), just so it can run the code. It’s kind of like a horse and buggy built out of modern materials and powered by a fuel cell – it looks funny, but it’s modern and efficient and gets the job done.

So, yes, the electronic part of computers get faster, better and cheaper. And the software seems to get better because it’s along for the ride, but it actually tends to get worse, which is why Paleolithic mainframes have been discovered, alive and working, in otherwise modern data centers.