The Black Liszt

Innovation with Computers and Slow Things

People have theories about innovation. Increasingly, they think it's important to innovate. Fine. I'm all for it. Given a choice between "innovation" (whatever that is) and the alternative, which I assume is something like "sitting and rotting," I'll take some of the former, thanks very much.

Whatever people end up saying "innovation" is (which kinda doesn't matter, because before long it will fade away, eclipsed by the next fashionable thing), it's clear to me that there is a huge difference between innovation that is based on using computers (which evolve quickly) and all other kinds of innovation.

For purposes of this post, I'll define innovation simply: innovation is doing something differently than you did before.

Physical Innovation

Physical innovation is hard. It doesn't happen very often. The reason is simple: over time, everyone pretty much figures out the best way to do things, and figuring out something new is hard and rare. A typical example of this is the gradual shift from wrought iron to steel.

Here is the famous iron pillar at Qtub Minar in Delhi, as it was when I visited it a few years ago.

This pillar was created no less than 1,000 years ago, and perhaps longer than 1,500 years. Wrought iron was created in many parts of the world, from China:

to Europe.

Steel is closely related, but different in important ways. The very earliest steel is about 4,000 years old. A form of steel, Wootz steel, was made in India more than 2,000 years ago. This steel was shipped to the Middle East, where it became the raw material for Damascus steel swords. But none of it was a practical innovation over wrought iron until the introduction of the Bessemer Process in the 1850's.
Then and only then could we have wonderful modern things like steel cables, structural steel for buildings and bridges and many other things.

Physical innovation, like the replacement of wrought iron by modern steel, is tough and long, punctuated by invention while still requiring endless baby-step innovations.

Process innovation

Process innovation is a whole different animal. Process is what the concerned human beings agree it should be, even if a bunch of machines are involved. The only limit is concepts. Opportunities for process innovation are all around us. In all too many cases, it seems more appropriate to call a process innovation something more like "stop doing it the obviously stupid way."

Here's an example. Not long ago, a delayed flight I was waiting for at JFK airport was finally cancelled at 1am. A whole lot of people went to the terminal entrance and got in the roped-off line for cabs. I waited about 20 minutes to get to the front of the line, and there were loads of people still waiting after me. "It's real late," I thought, "I guess most of the cabbies were sensible and are home sleeping in bed." Nope! There was a looooong line of cabs waiting to pick up the loooooong line of exhausted rejected passengers.
What was the problem? Process, of course. There was a single person who had to find out where you were going and give you the right piece of official paper before you could get into a cab.
And instead of walking up the line of waiting people, the "dispatcher" insisted on performing his duty as you were getting into the cab, which serialized the whole process.

Process "innovation" is, more often than not, simply "stupidity elimination."

Conceptual innovation

Conceptual innovation is a pretty big deal. It is limited only by the powers of the human mind. One of my favorite examples is one I encountered around the time I graduated from high school, George Danzig's Simplex algorithm for solving a linear programming (as in math programming, not software programming) problem. It's cool; it's been called one of the top ten algorithms of last century.

Computer Innovation

I know there's lots of physical innovation involved in creating the unprecedented, awesome speed with which computers evolve. There has been nothing comparable to it in human history. It also know it's accompanied by and partly enabled by lots of true conceptual innovation and some process innovation. But let's take all that as a given. What do we have?

We have a set of tools that can control, automate and communicate faster than anything in history, and that improve at a hard-to-comprehend rate. As soon as we get something working with one generation of the things -- BOOOM! -- everything concerned has just gotten better by 2X or more in speed, cost and size.

Steel took over from wrought iron when the process of making it got faster and cheaper, and when the results were superior. It took decades. Well, that happens every year or two with computers -- the question is, how are you going to take advantage of the improvement?

That's computer innovation. What's possible now that wasn't a year ago? What can I do now to create a product or service that, on next year's devices and networks, will make sense? The people who jump on this and make it happen are the innovators.

The themes are clear: we move from slow transmission of small amounts of data to big, expensive devices (think teletype) to near-instant transmission of huge amounts of data to small, affordable devices (think smart phones). This happened in small steps. Each step was a massive technology and business disruption. Fortunes were made and lost at each step. Fortunes will continue to be made (and lost) as some people see the possibilities and take advantage of them, while most learn the current state of computing and networking, and -- amazingly -- act as though it won't change. That's actually what the vast majority of people and companies do!!

Computer innovation is different than the other kinds -- not better, just different. If you understand the rules and act accordingly, you can accomplish amazing things. It almost feels like cheating to call it "innovation," but technically it is, so let's go with it.

Replication: Good Idea! Storage replication? Nah!

Everyone knows losing your data is a bummer. If you're in charge of your organization's data, you know that losing data is the shortest path to "don't let the door hit you on the way out."

All the ways to assure your data is still available when you wake up tomorrow share a common theme: "make a copy." This is such a popular theme that it has turned into a theme-and-variations: "make a copy; make another copy; copy the copy; etc."

This sounds simple, but we all know that in computing, stuff is supposed to be complicated. Sure enough, this simple "just copy it" theme has gotten mired in hotly competing ways to get it done. And of course, there are politics -- whose responsibility is it to assure against loss?

So let me boil it down: there are two basic ways to do the copy:

1. The guys in charge of the data, the storage guys, should copy the data from the original bunch of storage to a second bunch of storage.
2. The guys who write the data, the applications or systems guys, should get their applications or systems to talk to each other and write the data twice.

The only reason this is hard is that politics and history are involved. If you had fresh, educated people starting from scratch, it would be no contest: way number 2 wins, almost every time. It's faster, cheaper and easier than way number 1. But since when can we wave a magic wand and eliminate politics and history? The reality is, storage guys own the data, they want to protect it, and so they (usually) really, really, REALLY want to be in charge.

Here's why they shouldn't be.

You've got two sites, number 1 and 2. Each one of them has a database and a bunch of storage. Transactions come into site 1 and get written to storage.

Here's a simple transaction that might be written to the database.

It's a SQL statement that says the DBMS should write the transaction into the transaction table. The transaction contains the usual fields, things like the unique ID for the tranaction, the account number it's applied to, the amount of the transaction, etc. This is usually a simple string, a line or two long.

When the database processes the transaction, it gets complicated, of course.

When the Insert statement goes to the DBMS, the DBMS has to write the transaction itself, but it also has to write at least a couple of the fields to index tables, kind of like card catalogs in old-style libraries that let you find where things are. Indices typically use well-know things called b-trees, which may require a couple of writes to create a multi-level index, for the same reason you put related files into sub-folders so you have some chance of finding them later. There will certainly be an index for the account ID and one for the account number. Finally, there's a log to enable the DBMS to figure out what it did in case bad things happen.

All this happens when the Insert transaction comes in. One simple request to the DBMS, many writes and updates to the storage, usually involving reading in big blocks of data, modifying a smal part of the block, and writing the whole thing out again.

Now we come to the crux of the matter: how do we get the data over to site 2? Does the DBMS at site 1 talk with his buddy at site 2 to get it done, or are the relevant storage blocks in site 1 copied over to site 2?

In the diagram, I show the DBMS doing the job in green and the storage doing the job in red.

You'll notice that the DBMS only has to send a tiny amount of data over to site 2, essentially the insert statement. Once it's there, DBMS #2 updates all the storage, something it's really good at doing.

To replicate the data once it's been stored (in red), HUGE amounts of data need to be sent over the network to site #2. It's not unusual for the ratio to be hundreds or thousands to one.

Sending data between sites is a relatively slow and expensive operation. That's why, if you want replication that's fast, reliable and inexpensive, you want the application to do the job, not the storage.

The storage replication people don't like to talk about the things that go wrong, but of course they do. What happens if some of the blocks get over but others don't. Or they're out of order. Or syncing with the database doesn't happen. Or any number of other bad outcomes.

Other applications

I'm using a database application to illustrate the principle, but similar dynamics work out with other applications. All major databases can replicate (Oracle, MySQL, SQLServer, MongoDB, etc.), the major file systems can replicate (for example Microsoft has VSS), and all the hypervisors can replicate.

The hypervisors are amazing. The first thing the storage guys will come back with is how many different applications you have to fiddle with to protect their data. The answer of substance is that the incremental effort for each application is truly trivial, well under 1%. The quick answer is that hypervisors (VMware, Hyper-V, etc.) are universal, and their replication is superior to storage replication. This is exactly why, as organizations move their data centers to the cloud, they are abandoning expensive, inefficient storage vendor-lock-in features like replication in favor of doing it in the hypervisor.

Conclusion

You have to protect and preserve your data. Non-negotiable. The storage guys used to have a monopoly on it. But their high-priced, inefficient copy methods are rapidly giving way to more effective, modern ways that save money and are nearly standard in the SLA-centric world of cloud computing.

 

How to Evaluate Programming Languages

Programmers say language A is "better" than language B. Or, to avoid giving offense, they'll say they "like" language A. Sometimes they get passionate, and get their colleagues to program in their new favorite language.

This doesn't happen often; inertia usually wins. When a change is made, it's usually passion and energy that win the day. If one person cares a WHOLE LOT, and everyone else is, "whatever," then the new language happens.

Sometimes a know-nothing manager (sorry, I'm repeating myself) comes along and asks the justification for the change. The leading argument is normally "the new language is better." In response to the obvious "how is it better?" people try to get away with "It's just better!" If the manager hangs tough and demands rationality, the passionate programmer may lose his cool and insist that the new language is "more productive." This of course is dangerous, because the rational manager (or have I just defined the empty set?) should reply, "OK, I'll measure the results and we'll see." Mr. Passion has now gotten his way, but has screwed over everyone. But it usually doesn't matter -- who measures "programmer productivity" anyway?

But seriously, how should we measure degrees of goodness in programming languages? If there's a common set of yardsticks, I haven't encountered them yet.

The Ability of the Programmer

First, let's handle the most obvious issue: the skill of the programmer. Edgar Allan Poe had a really primitive writing tool: pen (not even ball-point!) and paper. But he still managed to write circles around millions of would-be writers equipped with the best word processing programs that technology has to offer. I've dealt with this issue in detail before, so let's accept the qualification "assuming the programmer is equally skilled and experienced in all cases."

Dimensions of Goodness

Programmers confined to one region of programming (i.e., most programmers) don't often encounter this, but there are multiple dimensions of goodness, and they apply quite differently to different programming demands.

Suppose you're a big fan of object-orientation. Now it's time to write a device driver. Will you judge the goodness of the driver based on the extent to which it's object-oriented? Only if you're totally blindered and stupid. In a driver you want high performance and effective handling of exception conditions. Period. That is the most important dimension of goodness. Of all the programs that meet that condition, you could then rank them on other dimensions, for example readability of the code.

Given that, what's the best language for writing the driver? Our fan of object orientation may love the fact that his favorite language can't do pointer arithmetic and is grudgingly willing to admit that, yes, garbage collection does happen, but with today's fast processors, who cares?

Sorry, dude, you're importing application-centric thinking into the world of systems software. Doesn't work, terrible idea.

It isn't just drivers. There is a universe of programs for which the main dimensions of goodness are the efficiency of resource utilization. For example, sort algorithms are valued on both performance and space utilization (less is better). There is a whole wonderful universe of work devoted to this subject, and Knuth is near the center of that universe.

Consistent with that thinking, Knuth made up his own assembler language, and wrote programs in it to illustrate his algorithms. Knuth clearly felt that minimum space utilization and maximum performance were the primary dimensions of goodness.

The largest Dimension of Goodness

While there are other important special cases, the dimension of goodness most frequently relevant is hard to state simply, but is simple common sense:

The easier it is to create and modify programs, quickly and accurately, the more goodness there is.

• Create. Doesn't happen much, but still important.
• Modify. The most frequent and important act by far.
• Quickly. All other things being equal, less effort and fewer steps is better.
• Accurately. Anything that tempts us to error is bad, anything that helps us to accuracy is good.

That, I propose, is (in most cases) the most important measure of goodness we should use.

Theoretical Answer

Someday I'll get around to publishing my book on Occamality, which asks and answers the question "how good is a computer program?" Until then, here is a super-short summary: among all equally accurate expressions of a given operational requirement, the best one has exactly one place where any given semanic entity is expressed, so that for any given thing you want to change, you need only go to one place to accomplish the change. In other words, the least redundancy. What Shannon's Law is for communications channels, Occamality is for programs. Given the same logic expressed in different languages, the language that enables the least redundancy is the best, the most Occamal.

Historical Answer

By studying a wide variety of programs and programming languages in many fields over many years, it's possible to discern a trend. There is a slow but clear trend towards Occamality, which demonstrates what it is and how it's best expressed. The trend is the result of simple pressures of time and money.

You write a program. Sombody comes along and wants something different, so you change the program. Other people come along and want changes. You get tired of modifying the program, you see that the changes they want aren't to different, so you create parameters that people can change to their heart's content. The parameters grow and multiply, until you've got loads of them, but at least people feel like they're in control and aren't bugging you for changes. Parameters rule.

Then someone wants some real action-like thing just their way. You throw up your hands, give them the source code, and tell them to have fun. Maybe they succeed, maybe not. But eventually, they want your enhancements in their special crappy version of your nice program, and it's a pain. It happens again. You get sick of it, analyze the places they wanted the changes, and make official "user exits." Now they can kill themselves customizing, and it all happens outside your code. Phew.

Things keep evolving, and the use of user exits explode. Idiots keep writing them so that the whole darn system crashes, or screws up the data. At least with parameters, nothing really awful can happen. The light bulb comes on. What if I could have something like parameters (it's data, it can't crash) that could do anything anyone's wanted to do with a user exit? In other words, what if everything my application could do was expressed in really powerful, declarative parameters? Hmmm. The users would be out of my hair for-like-ever.

What I've just described is how a new programming "level" emerges historically. This is what led to operating systems, except that applications are one giant user exit. UNIX is chock full of things like this. This history is the history of SQL in a nutshell -- a powerful, does-everything system at the level of declarative, user-exit-like parameters!

The Answer

In general, the more Occamal the language (and its use), the better it is. More specifically, given a set of languages, the best one has

• semantics that are close to the problem domain
• features that let you eliminate redundancy
• a declarative approach (rather than an imperative one)

Let's go through each of these.

Problem Domain Semantics

A great example of a such a language is the unix utility AWK. It's a language whose purpose is to parse and process strings. Period. You want an accounting system, don't use AWK. You want to generate cute-looking web pages, don't use AWK. But if you've got a stream of text that needs processing, AWK is your friend.

From the enterprise space, ABAP is an interesting example. While it's now the prime language for writing SAP applications, ABAP was originally Allgemeiner Berichts-Aufbereitungs-Prozessor, German for "general report creation processor." In other words, they saw the endless varieties of reporting and created a language for it; then having seen the vast utility of putting customization power in the hands of users, generalized it.

Features that let you eliminate redundancy

This is what subroutines, classes and inheritance are supposed to be all about. And they can help. But more often, creating another program "level" is the most compelling solution, i.e., writing everything that the program might want to do in common, efficient ways, and having the application-specific "language" just select and arrange the base capabilities. This is old news. Most primitively, it's a subroutine library, something that's been around since FORTRAN days. But there's an important, incredibly powerful trick here. In FORTRAN (and in pretty much all classically-organized subroutine libraries), the library sits around passively waiting to be called by the statements in the language. In a domain-specific language, it's the other way round!

A related approach, which has been implemented over and over, is based on noticing that languages are usually designed independent of databases, but frequently used together. The result is monster redundancy! Wouldn't it be a hoot if database access were somehow an integral part of the language! Well, that explains how and why ABAP evolved from a reporting lanaguage (necessarily intimate with the database) into "Advanced Business Application Programming." And it explains why Ruby, when combined with the database-leveraging RAILS framework, is so popular and highly productive.

A declarative approach

In the world of programming, as in life, the world divides between imperative (commands you, tells you how to do something, gives you directions for getting to point B) and declarative (tells you what should be done, identifies point B as the goal). In short, "what" is declarative and "how" is imperative. A core reason for the incredible success of SQL is that it is declarative, as Chris Date has described in minute detail. Declarative also tends to be less redundant and more concise. As well as doesn't crash.

Conclusion

I'm sorry if you're disappointed I didn't name Erlang or whatever your favorite is as the "best programming language," but I insist that it's far more novel and useful to decide on what basis we are to judge "goodness" in programming languages, and in programs. In general and in most domains (with important exceptions, like inside operating systems), non-redundancy is the reigning virtue, and languages that enable it are superior to ones that are not. Non-redundancy is nearly always best achieved with a declarative, problem-domain-centric approach. And it further achieves the common-sense goal of fast, accurate creation and modification of programs.

Occamality is rarely explicitly valued by programmers, but the trend to it is easy to see. There are widespread examples of building domain-specific languages, meta-data and other aspects of Occamlity. Many programmers already act as though Occamality were the primary dimension of goodness -- may they increase in numbers and influence!