The Black Liszt

Emails and Customer Respect: HP again

In an earlier post, I complained about software quality at big companies, illustrated by how they get the "little" things wrong, which of course sometimes builds up into getting really big things wrong.

HP is definitely on a roll in that regard. Just to pick an example, HP decided to buy Autonomy for $11.7 Billion in 2011, and about a year later, was forced to write off $8.8 Billion. In that context, little things like email marketing practices are trivial, things of no concern at all.

But that's the point! Quality starts at the top, it starts at the bottom, and attitudes about it pervade an organization. If no one is particularly concerned about quality, then stuff happens. And seems to be happening at HP (among others).

The importance of email

Some people may think that email is a tiny little unimportant thing. That's odd, because email is an important way that companies communicate with and interact with their customers. The fact that a company's many divisions may send out millions of emails tempts people inside the company to think how insignificant emails are. We send them out by the million!

But in the experience of any one customer, email may have a large role to play in forming their image of the company. Does it make good products that I want to buy? Does it listen to me? Does it respect my views and respond to my requests? When it makes an offer and I respond, what is the company's follow-up?

Email is like a salesperson, or a customer service representative, having an important role to play in the customer's future relationship with the company.

HP and email

Someone in power at HP badly needs to read this. The way HP currently handles their email is a sad example of dissing current and potential customers.

I recently received a couple more spam emails from HP, after many diligent attempts to follow their rules for unsubscribing.

Here's the bottom of the email:

Sounds good, huh? HP doesn't just protect my privacy, they're committed to it. I can choose (hah!) whether HP may communicate with me!

So, yet again, I clicked here to unsubscribe. This time HP put up a new hurdle.

They know what my email address is. All they have to do is what every responsible e-mail marketer does and put it in the unsubscribe URL in order to deliver what customers want, which is one-click unsubscribe. Not exactly a new idea!

But HP, that organization capable of rebuffing double-digit numbers of pleas from me to unsubscribe, has decided to up the ante. Now they demand that I type in my email address! Which they know! Just to show who's boss, and to show how much they respect me, and to hope that maybe I won't bother to type it in, forcing them to find another excuse to keep pounding me with spam.

The importance of email in customer relations and branding

An email interaction with a customer is like a sales person's interaction with a customer, i.e., an opportunity to show the company at its best, to make a good impression and to act in a way that inclines the customer to spend money buying the company's products and services, now or in the future. If the customer says "no, I'm not interested right now," the effective sales person bows out in a way that maximizes the chances that, in the future when the customer has a need, they will be favorably inclined to the company.

What if the sales person says "I really respect you and I respond to your needs, but if you want me to leave your office, you have to write on this piece of paper your exact name and job title. If what you write fails to match the information I have in any way, even a single letter, I will continue to walk into your office uninvited and demand your time and attention." That's the physical equivalent of the email interaction I had with HP. How do you think this would work out if physical sales people did it? Is there any reason to believe that the quality of the reaction is different when the HP representative is email?

More and even worse

HP has just announced that it's splitting into two companies. Naturally, having ignored all my requests to be dropped from their email lists, HP emailed me about it. Not just some division; central, home-office HP. Here's what I received:

Note that the subject of the email is "separation information pertaining to email subscription." Thus, the home office of HP is still convinced I'm an email subscriber. And this email is about being a legitimate, opted-in subscriber to HP email.

Naturally, I was curious to see how they'd handle the unsubscribe request. I clicked on the link above and got ... nothing! For several hours I periodically tried, and the link led to a site that simply didn't respond. The next day I tried again, and got this page:

 Again, I got that worst-practice of email unsubscription methods, the "we're making believe we don't have the email address that you clicked on to get here, so you have to guess to which of your potentially several email addresses we sent this and enter it correctly, otherwise we deem you insufficiently skilled to deserve being excused from being spammed by HP, and we will continue our periodic spam until you get it together."

That and a lawyerly "privacy" policy and five bucks will get you a coffee at the nearest Starbucks.

To all companies in the world anywhere that send email: take this as an example of what not to do. Your present and potential future customers sincerely request it, and will respond accordingly.

Big Data and Data Warehouses

There’s a rapidly growing movement to take all the data that’s scattered throughout an organization, rationalize it, bring it together, and make it available for analytics that will help management to understand and ultimately to transform the business.

This movement is taking place today. It’s explosive. It’s called Big Data and Analytics.

This movement also started in the 1970’s, took root in the 1980’s, exploded in the 1990’s and is with us today. It’s called Data Warehousing.

The fact that attention is slipping away from the thing called “data warehouse” and moving towards “Big Data” is a typical IT industry phenomenon. The problems are the same, the obstacles are the same and the solutions are the nearly the same – but the rhetoric and software are entirely different. Only a few savvy industry insiders are aware of the game that’s being played.

The Enterprise Data Warehouse

The Enterprise Data Warehouse, EDW, is the industry’s holy grail. It’s the place where all an organization’s data is stored for reading and analysis. All the data from the various operational and transaction databases is extracted, transformed as required and loaded into this database. Once there, it provides a “single source of truth” for the enterprise. Since the EDW is not running transactions, reports and analytics can be run against it at will, without harming ongoing operations. Single-purpose extracts can easily be made from it to support various projects.

The EDW makes common sense. It became a major goal for many organizations, and many are still marching towards that goal. There’s just this one little problem: getting there. Then there’s a second problem: realizing the potential value.

There are lots and lots or organizations that don’t have to worry about having an EDW that fails to fulfill its promise, because they just get bogged down along the way and never really get there.

Why don’t you read much about this? Simple: who wants to admit it? And if the road to the EDW ends up trapping those marching down it in impassable mud, who outside the organization is ever going to know it?

There’s a simple little acronym in EDW that is the tip of the mud-trap in which EDW gets bogged: ETL, which means Extract, Transform and Load. That’s what you do to get the data from where it starts to where it needs to be, in the EDW. Simple, right? Oh, if only it were…

Extract, Transform and Load

Before you even get to the E of ETL, you have to find the data. Then you have to get access to the data, with a properly jealous operational management group anxious that you avoid screwing them up. You have to get the whole thing to start with, and then a stream of updates.

The “database” could be nearly anything. It could be a set of ISAM files running under CICS on an IBM mainframe. In which case, you need to get your hands on the source copy books that contain the data definitions to have any hope of making sense out of them.

It could be something nice and modern, like Oracle. But you’d better start by getting a full dump of the schemas to have any hope of navigating among what could be many hundreds of tables. Then, without an E-R diagram that’s up to date, you’ll have little chance of making sense out of the tables. Then when you get down to it, you may discover a world of stored procedures initiated by access triggers, so that your innocent “just let me read the tables” turns out to have side effects. And then, getting the updates? You’ll soon find yourself either crawling to the DBA and begging for a change log to be shipped to you, or pleading to be allowed to program in some trigger-initiated stored procedures yourself, so you can get the updates with killing the performance of the DBMS, and avoid getting set upon by a mob of angry users.

Phew! Now you’ve gotten through the E part of one source. What if there are dozens, or hundreds?

And then the real fun begins. The also-innocent-sounding T phase of ETL. Because T doesn’t just mean simple, no-big-deal transforms, it also means take the customer names that are represented in different ways in different places, some of which have been updated or changed independently of the others, and make it so that when you end up with a customer in the EDW, that customer represents all of your relations with exactly one customer. Having three customer rows in the EDW for one customer kind of defeats the purpose of the EDW, after all.

I’m just scratching the surface here, but perhaps you can get a feel for why the glowing promise of the Enterprise Data Warehouse so often ends up with the participants hungry and wounded in various ditches along the path to the promised land.

Big Data

Forget I ever mentioned “data warehouse”, ETL or any of that other stuff. Bzzzzzzttt! New subject!! Brand new!!! NOT related to anything else in computing, completely WITHOUT history, stemming from this brand-new EXPLOSION of data that’s just EVERYwhere. It’s the Big Data movement! Where we take all these mountains of data that are just piling up useless and turn them into business GOLD. You’re already late – everyone else is already with it. There are books, conferences, experts, the whole nine yards!

You’ve got to get a Data Lake, and fill it up with data. Then you’ve got to rev up your Hadoop cluster and start cranking out those nuggets of business gold from all that data.

Except, hmmm. I’ve got to find the data. Get access to it. Get it once and then get a feed of the updates. All this data from different places, it doesn’t match up well, I’ve got to clean it up. Well, maybe I’ll just dump it into the Data Lake and let the Hadoop nerds worry about it. They’ve got all these servers at their disposal, maybe the servers can work at night cleaning everything up.

Gulp. I just looked at my nice, fresh, clean Data Lake. It’s a Data Swamp! There are snapping turtles and water moccasins swimming in there. Don’t. Like. This. Maybe I can get a transfer.

Conclusion

If “data warehousing” were a big success, it would have kept its name and would now handle what we now call “big data.” But no. “Data warehousing” projects are often classic IT projects that drift on forever, confronting obstacles and rarely producing results. Big Data is the new kid on the block. There aren’t (yet) decades of frustration and broken promises associated with it. Give it time. Every obstacle that DW projects encounter also rear up to challenge Big Data projects, and until solutions are found, returns will be equally elusive. And even then, there are conceptual flaws in most Big Data efforts.

Building Better Software More Quickly

How can you build software more quickly, without just cutting corners, taking risks and turning out crap? The prevailing view in the industry is that building software is a tough job, and it just plain takes a long time. Different factions heavily promote methods that are supposedly better, but in the end, the fancy new methods are either minor variations on standard practice or make things worse.

We don't need to speculate to decide that great software can be built much more quickly than nearly everyone thinks -- there are people who are doing it today. I've written about this extensively. But in most environments, including government contracting and large corporations, the speed methods are forbidden. Yes, forbidden, as unthinkable as trying to walk in the office barefoot. The people who get it done don't just add some secret sauce to the standard recipes -- they have whole new ways of framing the problem.

Getting from the start to the finish

Most programming jobs are like getting from Point A (where your software is today) to Point B (where you'd like it to be).
All the widely accepted methods for getting from Point A to Point B take what they sincerely believe to be the shortest path.

Standard ways to travel

There are standard, authorized, certified and audited ways to make the trip. Some people feel passionately that their way is better than the others, but they all take a long time, and there are frequent wrecks on the road. The journey is always an arduous one, and has many phases, each requiring specialists with lots of experience and training to get the job done while avoiding disaster. Which nonetheless happens often enough. Even when you make it safely to the destination, all too often you either arrive late or without all the stuff you expected to arrive with.

Everyone agrees that there are lots of steps in getting software done, steps that require education, experience and skill of different kinds, and each step builds on successful completion of earlier steps.

In the early stages of the trip, you do lots of planning, because you know how tough trips like this tend to be. You go back and forth over nearly the same ground, never making lots of progress. It’s climbing a mountain, and the switchbacks are just a necessary part of what you have to do to get to the peak. Once you’re there, you can finally see your goal at the bottom of the mountain. There’s a clear path to get there. The path continues to have lots of risks, but at least you have a definite plan, you’re know exactly where you’re going (you can see it!) and how you’re going to get there. It feels more like you’re going down the mountain.

When you’ve got point B in sight, it’s important to move deliberately and carefully. If you skip steps, you can easily wreck everything and fall off a cliff. But still, it’s easier going because everything is spelled out and written down. You’ve brought new members onto the team, specialists who have gone down mountains like this many times.

Even though you're going in a straight line, it turns out it's not a straight line on level ground! It's a straight line that goes over a very high mountain, one with vertical walls, gaps and all sort of progress-killing obstacles. There are all sorts of people promoting ways of getting to point B that are supposedly faster or less error-prone. But in the end, you still have to climb the mountain and then go down to the destination. All the same work ends up getting done, but just in different combinations and arrangements and orders.

Speed-optimizing methods

There are a couple wacko’s out there who talk about nothing but “speed.” Right. Going faster is definitely the fastest way to end up in the ditch.

Yes, that's what nearly everyone thinks, and especially people with experience, people who have been part of or been subjected to the wide variety of disasters caused by train-wreck software development efforts (oops, sorry, I'm being redundant here).

When you look at what the people who get things done quickly actually do, you find that they reject the whole paradigm of project-management-oriented software development. They think about the whole process in different terms. There's still a Point B, but instead of accepting the common delusion that you can figure out how comfortable a bed is before you lie on it, they realize that you have to get close to where Point B probably is, and then look around and refine it, maybe by quite a bit! Maybe Point B sounded wonderful based on the speculate reports received while standing on Point A, but maybe once you're in the general region of Point B, nearby Point B1 is clearly superior. So you shift to it. Look around. Figure something else out, and shift to Point B2. Wow, how great is Point B2 -- let's build some comfy chairs and a roof so we can stay here a while -- but be sure to build the roof this way, and with a skylight over there, which is pretty obvious while sitting on a camp chair in the right place, but simply couldn't have been known way over there at Point A.

Conclusion

The speedy programmers who want to get to a destination find ways to avoid making arduous, risky climbs over mountains; instead they zoom to the general place quickly, learn stuff and refine. The detailed methods that speed-oriented developers use can seem mysterious, until you see that they're looking at things from an entirely different point of view.

Secrets of Software Super-Developers

Outstanding software development groups are truly outstanding. In many sporting events, the winner is often determined by a margin of a fraction of a percent. In software, the best teams are at least 10X more productive than the competition, and sometimes 100X. Here's the most amazing thing of all: the way the super teams do it is not secret!

Discovering the secrets

For most of my software development career I was only vaguely aware of productivity differences. My job was rarely to judge and evaluate -- it was to get stuff done, quickly and well. I figured out some basic stuff to the extent it helped me get stuff done, and hired people who would contribute. I quickly realized that education was irrelevant, as was experience in a particular technology or tool set. What mattered most was raw horsepower, ability to concentrate, willingness to work and the drive to learn.

I think my first wake-up call was a consulting project I did for Sallie Mae during the early 1990's, when I was one of three groups advising the company on a massive re-engineering project using document imaging and workflow technologies. One of the groups was me; the second was a local group with 5 people on the project full time; the third was a national consulting firm that had at least 15 people on the project full time with various senior people and "experts" floating around. I was the only person involved with any of the groups who had not only used the technologies, but created them.

To make a long story short, the other groups performed standard analysis and vendor evaluation, and proposed an expensive process that would result in a 30% productivity gain, if all went well. I dove into the actual work Sallie Mae was doing, noticed some things about the work, and proposed an inexpensive process that would result in a 5X productivity gain. I called the method I used "zero-based re-engineering," combining two fashionable-at-the-time concepts, "zero-based budgeting" and "process re-engineering." They put it to the test and demonstrated that it worked! I tell the whole story here.  Did it catch on? Did the dramatic advantages of the approach lead to widespread use? Nope.

It took years for me to figure out that what I had experienced with Sallie was the tip of the iceberg, both with giant bureaucracies like Sallie Mae and with the "technology expert" consulting firms.

It's developer's gold -- but you can't give it away

About ten years later (yes, I know, I'm slow), I started noticing similar stark contrasts among groups of technology companies and the way they built software. The things I noticed were at heart simple -- but compared to existing practice, radical. And I now finally realize, revolutionary. Man, I'm really getting the formula for the secret sauce here! I've got to be really careful, and make sure I only whisper these secrets to our portfolio companies. I can't let them get out; if one of our company's competitors got wind of the methods and rolled them out before we did, we'd get killed.

So for the first couple of years, I was really careful with these "secrets." It was only spoken, never written, and only on a need-to-know basis. Some of the companies I would visit wouldn't say they had "secrets" -- but they did things differently from most groups, and I eagerly learned from then. I'd visit other companies which could really benefit by employing one or more of these "secrets." I whispered to the smart, motivated people who are leaders in companies in which we've invested where the gold is -- and it's really close, right over there!! -- and they're ho-hum, thanks a lot, so glad you let me know, please tell me when you plan to visit again, good bye. In other cases, there was apparent enthusiasm, followed by nothing. I thought the problem might be understanding. So I followed up the verbal with e-mails to solidify the ideas.

Learning more and revealing the secrets

Meanwhile, I'm getting a broader knowledge of how the super-developers get it done. It appears there's a spectrum of techniques. Some people use lots of one and little of another, but they all optimize for speed and quality instead of expectations.

I did notice that putting the ideas in writing helped. So I graduated from over-long emails to PDF papers. I told people the ideas, then gave them the papers. Or the other way round. Having the papers seemed to increase the chances that our companies would take the ideas seriously. I kept augmenting and editing the papers as I learned things, and peppered them with real-life examples. Some of the papers went through a couple dozen editions. By about 5 years ago, I had over ten papers comprising over 800 pages.

Why won't people grab the gold?

As I saw more examples of great results, I became ever-more mystified about the laggards who just don't get it.

Now, to be clear, these "laggards" are so only in relative terms. Most of them are way smarter and more accomplished than their compatriots are in the giant legacy companies. In the eyes of the world, the vast majority are leading edge. I started noticing that what I saw as laggards were nearly universally practicing methods that the software industry as a whole has labelled "leading edge," things like unit testing, Extreme programming or Agile. So on a bell curve of established practices, they were definitely leading edge.

My dim light bulb finally started flickering on. Almost no one decides important things like software methods on their own! They pick whatever brand of recognized external authority they're most comfortable with and which best matches their self-image, and try to follow the established method.

Understanding the resistance

But these are programmers, darn it! The methods aren't exotic, requiring a thorough grounding in quantum string theory -- in most cases, they're simpler than the standard stuff. So where does the reluctance come from?

This is a BIG subject. There are lots of aspects of it, and I'll return to the subject in future posts. But the bottom line is simple: if programmers want to advance in their careers, for the most part they have to accede to the demands of non-technical managers. The non-technical managers have no clue what programming is about, except that it's full of incomprehensible gobbledygook. Understandably, they want to feel confident of what's going on in the department they're in charge of, most of whose people are engaged in activities that are opaque to them. So they absolutely require the use of standard methods, measures and goals that are entirely consistent with other disciplines in which things get built, like buildings and cars. Anything else feels like a "lack of discipline."

Most of the "secret" methods did not follow the "makes managers comfortable" formula. So the vast majority of organizations would reject their use, regardless of their provable power and effectiveness. For me, keeping things secret was not the problem -- it was getting the ideas acted on. There are indeed amazing methods used by software super-developers to get things done 10X or more better than anyone else. But because the methods are not among those talked about by the public "experts" in the industry and do not resemble how houses or cars are built, they aren't a candidate for being used by the vast majority of developers, who do not make decisions for themselves based on substance.

Going fully public

That's when I decided there was no harm and possible benefit in going fully public with the things I'd been noticing over the years. That's when I decided to fix up the papers, turn them into books, and publish them in paperback and on Kindle. Here is a little more background, some illustrative blog posts and links to the books that I've already published.

I am working on book versions of the remaining papers, some of which are more revolutionary than what I've already published. Going through the process of crystallizing what I've learned over the years has taught me a great deal about knowledge itself and innovation, since part of what I've done is observe dramatic advances and seen how the knowledge is acquired and spread.