The Black Liszt

Human and Inhuman Analytics

While people talk about analytics in general, there are really two distinct varieties: human analytics and inhuman analytics. First, there is analytics for and by humans, i.e., numbers, tables and graphs designed by humans for human consumption and consideration. Second, there is algorithmic analytics, originally designed by humans but then set off to make observations, decisions and perhaps actions on its own. I dub this "inhuman analytics," because that's what it is. It is incredibly important to understand the differences between these two things, related in name but little else.

Human Analytics

When most people think about analytics, they're usually thinking about things like Data Warehouse (DW), Online Analytic Processing (OLAP), Business Intelligence (BI), and related subjects.

This is a subject that is broad and deep, with many products and vendors that have evolved over time. But there is a simple unifying theme: these are tools intended to provide information to people, often in the form of graphics, so that those people can understand what's going on and take any action that may be appropriate.

Oracle, for example, has a wide variety of such tools:

Microsoft also has a variety of such tools.

Note that both companies illustrate their approach using screens and people. That's what this type of analytics is all about.

There are a wide variety of BI tools from many vendors, in addition to open source.

Inhuman Analytics

Inhuman analytics, a terms that no one else uses, so far as I am aware, is a whole different thing. This is also a subject that is broad and deep and undergoing constant innovation. It includes such diverse subjects as machine learning (ML), advanced statistics, operations research (OR) and related subjects.

In general, inhuman analytics are far more specialized than human analytics. They are nearly impossible for anyone but a specialist to understand. There is often lots of math involved. They are not primarily about presenting information so that it makes sense to human beings -- they are about figuring stuff out that most humans wouldn't be able to figure out at all, or figure it out with a precision that exceeds human capability.

Because of this, there aren't great pictures to illustrate inhuman analytics. But here's an illustration of the ML process from one company's ML toolkit:

Inhuman analytics are behind a large number of modern innovations, though they rarely get credit for it, since the way they work is essentially like magic to most people This is a vibrant subject with a rich history. I suspect I will come back to this in some future post.

Conclusion

Human analytics has many uses and is a good thing. The visual tools it emphasizes enables knowledgeable and motivated people to explore and understand a data set, and to track it over time. Sometimes you can even discover new things, particularly in the early stages of understanding and optimization

However, inhuman analytics are the serious, heavy-duty tools to help derive value from data. They can and regularlly do figure things out and solve problems that are beyond human capability, even with the aid of human analytics.

Human analytics has its place. But it's no substitute for inhuman analytics for serious value creation.

 

Fundamental Concepts of Computing: Closed Loop

Like other fundamental concepts of computing, this is pretty simple: closed loop is better than open loop. But I concede that it's a touch more complex than other basic concepts, like counting.

Closed and Open Loop

A closed loop system is one that operates with feedback. By contrast, an open loop system operates without feedback.

This of course is a universal concept, not specific to computing. It's how living things operate in general, for example; it's the central characteristic that makes them successful and enables them to improve.

The concept applies to mechanical things as well. When steam engines, for example, were run on the open loop principle (i.e., with no feedback), they were difficult to manage and liable to unpleasant things like blowing up. Then James Watt made his steam engine into a closed loop system by applying a centrifugal governor to it to control the pressure of the steam.

Moving from open to closed loop

The classic open loop process is simple, like driving a car with your eyes closed. You have a goal, which is to drive on the road, not crashing into anything, not going too fast or too slow, until you reach your destination. With your eyes closed, i.e., without the main feedback loop in operation, it's kinda hard. Most people wouldn't try.

The closed loop process is just slightly more complicated, like driving a car with your eyes open. Everything is like driving with your eyes closed, except that you adjust the speed and direction based on the feedback you get from your eyes.

I think it's fair to say that most people prefer closed loop driving. They find it safer and easier on the nerves.

Closed loop applied to the process of building software

The usual way of building software involves working out a plan and then building the software according to the plan. This is like figuring out your route on a map, getting into the car, and driving to your destination with your eyes closed. The good news is, you know for sure when something goes wrong.

Classic waterfall is like checking at the end of a closed-eye drive whether you reached your destination. Since this is obviously insane, all waterfall processes add elaborate checks along the way to see whether you've driven off the road, etc. Never helps much.

Agile is like checking on a regular basis whether you've crashed and setting a new goal based on the latest disaster. Agile is just like waterfall except that the crashes are more spread out.

What is writing software using a closed loop process? Simple. It's like growing a baby.

Closed loop applied to system design

Just as feedback can work in the time dimension during the software building process, it can work in the conceptual dimension during the software design process. You may break down a system in layers, from UI down to storage, and design each in isolation. This is typical, and not a great idea.

It's far better to take repeated conceptual passes through the system design, as though you were driving through them in time, and apply the feedback of what each layer has to do to the other layers, optimizing the whole rather than the individual parts. This simple exercise can yield astounding results.

Closed loop applied to the software that you build

Most programmers seem to think that part of their job is creating work for systems administrators; in other words, they create software that requires care and feeding to keep running. This is strange, because software is all about automation. Why would you create software that is, in effect, open loop? Like creating a steam engine without a governor?

Just as you should apply the concept of closed loop to the process of building software, so should the software you end up building have feedback loops incorporated into it, so that to the greatest extent possible it is self-managing. Without such built-in feedback loops, we're doing the equivalent of building a steam engine with no governor. Making our software liable to blow up, like this steam engine did:

Conclusion

Sensible people keep their eyes open when driving a car. But a shocking number of otherwise sensible people do the equivalent of keeping their eyes closed when programming or designing systems, and they build software that is, in effect blind. What's this about? Open your eyes, already!!

 

The Big Data Technology Fashion

Where there are people, there are fashions. Why should technology be immune? The current fashion of "big data" is a classic exemplar of the species.

The Books

Books are a good place to observe the common themes of technology fashions. You'll see patterns that resemble the ones I previously pointed out for project management.

I think it's fair to say it's not a legitimate technology trend if it's not covered in an "X for Dummies" book.

Similarly, it's got to be big. Be Revolutionary. Transform lots of stuff.

Its got to be a big, scary thing that needs taming.

For any fashion trend, its important to make sure that other things are hitched to its wagons.

Let's not forget that, if it's worth paying attention to, there's got to be a way to make money from it.

It's never too soon to start adding layers of process and paranoia to it, to assure that costs skyrocket and that hardly anything ever gets done; in other words, governance.

Finally, anything but anything has to have a human side.

I swear, I sometimes think there's a central planning committee for technology fashions. They plan when the next new label on something old and not all that interesting is going to come out, grab their standard set of titles, and pass them out to people to write the books.

But then, I guess it can't really be that organized, because there are usually so very many books, each of them covering the same small set of themes over and over and over, with slightly different language. The themes always seems to include:

• X is revolutionary; it will change lots of important stuff.
• X is big and scary, and you need help to tame it or bring it under control
• There are lots of ways to screw up doing X, so you need to pay lots of money for Y to get it right
• You're a Dummy, but I'll help you understand what you need to know about X anyway.
• X has a human side

The Conferences

Things aren't that different with conferences. They take the themes established in the books and embellish them a bit.

There are conferences for people who work in particular sectors.

 

You can't pass up an opportunity to learn from the very best.

Who can resist going to a conference which cuts through all the crap and helps you do stuff?

Anyway, you get the idea -- there are lots of conferences. The themes are predictable, even without the aid of big data or predictive analytics. Because they apply to any technology fashion trend.

Conclusion

Technology fashions -- they are forever in fashion!

"Big Data:" Some Little Observations

"Big Data" is everywhere. If only because of this, it is important, like the way Paris Hilton

is famous for being famous.

What's included in "Big Data?"

If your concern is storing, serving or transmitting it, you don't care what kind of data it is -- data is data, a pile of bits.

But not all data is created equal. The easiest way to understand this is to break all the bits into relevant buckets. By far, the largest bucket is for image data (including both still and moving pictures, videos). While the ratios vary, it's not unusual for there to be 100 bits of image data for each bit of other data.

While there's not a commonly accepted terminology, all the rest of the data can be understood as "coded" data. This again falls into two categories. The larger portion is "unstructured" data, things like documents, blogs, e-mails and most web pages (except for the images and videos on them). The smaller portion is "structured" data, which includes all databases, forms and anything else that can show up in a report.

When people talk about "big data," they could be talking about any of the above, but mostly people talk about it because they want to extract actionable information from it, and the source of most actionable information is structured data. So in the vast majority of cases, when people talk about "big data," they're talking about structured data.

Did Data used to be Small and Now it's Big?

Think about a bank statement. There's a little information about you at the top, but most of the statement is probably taken up by the transactions -- money moving into and out of the account. In general terms, this is the action log, the transaction history. This pattern of having an account master and detail records is a common one.

Now think about a web site. The site itself is like the bank statement, and the record of people visiting and intereacting with it is like the transaction history, generally known as a web log.

People generate far more transaction records when interacting with the web than other human activities; for example, you probably click on hundreds of pages for each bank transaction you make. So the amount of data can be pretty big.

The simple answer is: before the web, transaction data wasn't very big, and with the web, there's a lot more of it than there was before. Of course big data isn't just about the web; but the web has certainly gotten people to pay attention.

So where did "Big Data" come from?

It would be interesting to do a cultural history, but I suspect that the current interest in "big data" stems from the following factors:

• Companies that pay attention to web logs get information about visitor behavior that can be used to make more money.
• Internet advertising companies have done exactly this for years, and are getting really good at it.
• Shockingly, most people don't analyze their data to improve their behaviors.
• A closed loop system in which the results of your actions are used to enhance future actions is the clear winning strategy.
• This requires (gulp) collecting and analyzing the relevant data, which is far larger than most people are used to dealing with.

Thus the term "big data," which currently applies to just about any body of transaction data.

What's "Big" about "Big Data?"

Let's start by applying one of the fundamental concepts of computing to the question: counting. One of the first disk drives I got to use was a twelve inch removable pack developed by IBM:

Its capacity was about 1MB. While that may sound small by today's standards, let's put it in perspective. Each byte is the equivalent of a character that you can type. Using a generous measure of 30 wpm and 5 cpw, that's 9,000 characters in an hour of continuous typing with no breaks, so the disk above has a capacity of more than 100 hours of continous typing. That's one reason I thought the disk's capacity was huge -- it easily held the source code for the FORTRAN compiler I wrote at the time, which was about a year's worth of work!

Now let's get modern. Drives have gotten smaller while holding more and more. Here's a good visualization of the progression:

We're now at the point where truly small drives (1 to 2.5 inches) hold massive amounts of data; 1TB or more is common.

How much is that? Remember, it would take 100 hours of continuous typing to fill up the large disk pictured earlier. How much space would those drives fill if you had 1TB to store? That's about 1 million of the older disks; if you packed them tightly, they would fill a room that was about 100 feet long, 100 feet wide and 10 feet high. And I would have to type for 100 million continuous hours to fill them up. Now, that's big data.

Now that we've got a sense of how big a TB is, let's get real.

On a good day, this blog might have 100 page views, each generating a server log record. Such records vary in length, but let's say they average 100 bytes in length each, or 10K bytes a day. Not much.

Let's say I caught up to the Washington Post, a site which is in the top 100 in the US. It gets about 1 million page views a day. That would be a mighty 100 million bytes a day of raw server log data. 10 days would add up to a GB of data, which means that ten thousand days, about 30 year's worth of data would fit on one of those physically little drives pictured above that holds just 1TB of data.

The Washington Post is a major site; top 100. Their web transaction logs are the biggest data for analysis they've got. And here's what 30 year's worth of their data will fit on:

That's what they call "big data." This is why I instinctively drop into cynical mode when the subject of "big data" comes up. It just isn't usually very big!

How much data do you need?

It depends on context. If you're a website like Facebook offering a free service holding user's data, the answer is simple: you keep as much of the user's data as you feel like. You can (and if you're Facebook, regularly do) throw out data any time you feel like it, or just drop it on the floor and lose it because your programmers weren't up to dealing with it.

If you're a money-making business that depends on data, you could probably run your business better if you

1. Kept all the data
2. Analyzed it
3. Came up with useful observations, and
4. Changed your behaviors accordingly.

But most businesses don't do this very well, if at all. And they are feeling increasingly guilty about it. Thus the marketing drum-beat for selling everything that can possibly be labelled "big data."

Sarcasm aside, the fact is that most businesses don't need much data in order to perform wonderfully useful analyses. The reasons are simple:

• The things that matter the most are things you're not doing yet. The data you've got is historic. It's like if you're a comedian and the audience doesn't laugh much; no amount of big data analysis of audience reaction will help you come up with better laughs.
• The impact of big potential changes will be seen in lots of your data. Go back to statistics 1.01. How much data do you need to see that the coin you're flipping isn't a fair one? Only enough to prove that the 2 out of 3 times it comes up heads isn't a fluke.
• In the end, how many changes can you realistically make? Hundreds? How about rank ordering them, finding the most important ones first, then moving on from there? You'll quickly get to diminishing returns.

Finally, more important than anything else, is getting into an experimental, data-driven, closed-loop system. This is always the key to success. It how organizations become successful, get more successful, recover from trouble, and stay on a winning path.

Conclusion

For better or worse, "big data" is likely to be with us for awhile, at least as a technology fashion trend. Like all such fashion trends, it's a useful occasion for getting us all to check if we're putting our transaction data to its most optimal use in keeping us on the track we're on and getting us onto improved ones.