The Black Liszt

Using Advanced Software Techniques in Business: Fashion or Real Value?

Using advanced software techniques can make a dramatic positive impact on business. It’s important for everyone to assure that your software efforts aren’t stuck in out-moded, last-generation tools and techniques.

Nearly everyone, including me, agrees with this simple statement. Nearly everyone also agrees on at least the top members of a list of reasonable candidates of “advanced software techniques” that are not “out-moded” or “last-generation.” That last statement is where the best technical people part ways with the crowd.

No, I’m not talking about hard-to-understand weird-o’s babbling about esoterica in some corner. The best technical people understand and support using the best and most appropriate techniques for solving a given problem, regardless of the recency of the technique or its prevalence. Sadly, it is often the case that the most-talked-about hot trends in software should not be used in any business that actually wants to spend money widely and get stuff done, quickly and well.

This is a BIG subject. It’s important. It’s deep. And it’s extensive. So let’s start with a big, fat, juicy example, one that was hot, hot, HOT but is now fading away, so it’s possible to talk about it somewhat more rationally. Maybe. I hope.

Big Data and Hadoop

There is little doubt that Big Data is a huge trend in software, though at least talking about it with that name appears to be undergoing a typical slow fade. Here is a review I wrote more than 5 years ago of the Big Data fashion trend as it existed at that time. It was everywhere you looked! Magazine covers! Ads! Conferences! Books! If you weren't somehow doing Big Data you were nothing and nobody.

I've been working with data my entire professional life. The data has been small, medium, large, big, huge, totally awesomely huge and even gi-normous. Since I've been faced with space and time constraints, I have long since settled on a fundamental concept of computing, simple but rarely done: Count the data! It sounds ridiculous, but it's almost a secret weapon, and appears to be rarely done. Here is some analysis of data sizes in the context of the big data trend. Here is a more detailed example of a big data set that Harvard bragged about. Hint: the data isn't very big.

I shouldn't have to say this, but here it is: For anything that people say is "big data," the very first step should be to ... count the data. Sounds simple, but apparently it's not. It's also not common to dig into the data a bit. I guess it's uncommon because in most cases of data that starts out looking big after you count -- this by itself is rare -- is that you find that most of the data is just not needed or not relevant. Which makes it not big anymore. Which means you don't need Hadoop!

Hadoop

I know I'm being silly here. Hey, we're talking Big Data! Surely we've got some somewhere. We've got to get in an expert and crunch away so we can get those virtuous, business-enhancing juices flowing through our company.

In this situation, at least until recently, what that meant was that you dove into the next level of detail and found out that the go-to tool was Hadoop. Dig in some more, and it sounds great. It's scalable without limit. You build your Hadoop cluster, script up some calculations, tap into the ocean of data you've got somewhere, and hear about how the Hadoop spins those computers up and down, and crunches all the data, using the computers that are available, and even working around ones that fail without anyone having to respond to some old-style beeper or something. No wonder Hadoop is the go-to tool for Big Data!

In the vast majority of situations, it's "decision made" time at this point. You get in your experts, they build their Hadoop cluster and away you go, climbing the Hadoop stairway to Big Data Heaven, with a glow of virtue surrounding everyone involved.

Very few people seem to dive in and understand what Hadoop and its main programming paradigm MapReduce are all about. The Hadoop "experts" don't seem to know what the reasonable alternatives are, and when they might be applicable.

Here's an example. In 2011, one of our large web companies had a huge problem caused by Google's move to a new search algorithm. The CTO grabbed a massive web log file, wrote some code to boil the terrabytes (Big Data for sure!) of data down to the key data elements, and then loaded them into the 512GB of DRAM of his powerful laptop computer and ran some advanced machine learning against it. You can see the CTO doing the work here. A few days later he had figured out Google's algorithm, reflected it in the company's website family, and traffic increased back to nearly the pre-change norm. If he had taken the Hadoop path, he would have worked for months, spent huge amounts of money, and found that the cluster and Hadoop thing would have basically been irrelevant to the problem.

Here are a couple things to consider:

• Hadoop, by definition, spreads its computing over the many machines available to it in the cluster, using HDFS (the Hadoop file system) for reading and writing data.
  • It is literally thousands of times faster to get data from local memory than it is to get it from a disk-based file system. The fewer file reads and writes needed to perform a computation, the faster it will be. Hadoop doesn't care.
    
  • Using more computers in a cluster means that there will be more I/O than using fewer. Many important calculations can be performed on a single properly-configured machine!
• MapReduce, the key processing engine of Hadoop, is one of those cool-sounding ideas whose job can be done perfectly well with normal code, which can do way more, vastly more efficiently.
• Why would anyone consider such an insanely wasteful approach? Once you know the origins, it makes sense.
  • If you're a big search engine company, you have to have loads of servers, enough to hold all the data and handle all the search queries at peak traffic times.
  • As is typical in situations like this, loads of servers will be under-used a large fraction of the day. Why not write some code that sucks up these "free" cycles and put them to work? Why not build a framework so you can just specify what you want done, without worrying about what resources from what machine where is used? Who cares if it's inefficient? It gets stuff done with the computers I already have. Brilliant!
  • Now it makes sense that Hadoop started and grew at Yahoo, copying some ideas about a narrowly-applicable (MapReduce) system and framework built at Google.
  • Except that at Yahoo, they somehow decided to make the Hadoop machines dedicated! Last I heard, they were up to, get ready ... 40,000 servers. Wow.

• With such an investment in getting value out of Big Data, Yahoo must be booming, just sky-rocketing with all the juice that has come out of the investment. Not. Why would anyone want to use an expensive, strange tool that generated no value for its originator? One word: fashion.
• Yes, there are some narrow situations in which Hadoop might be applicable. But in the vast majority of cases, you'll spend too much time getting way too many computers to do too little processing on not all that much data, and taking way too much time to get it done.

Conclusion

There is no doubt -- none! -- that you should use advanced software techniques in your business, because it will give you a competitive edge over everyone else.

The trouble is telling the difference between (1) value-adding advanced software techniques and (2) hype and software fashion. Even most software people have trouble telling the difference! In fact, software people who insist that there is a difference between value-adding software techniques and the latest thing that everyone is talking about run the serious risk of being marginalized, and categorized as being old farts who are unable or unwilling to do the work to learn the new methods.

In sharp contrast to the general thinking, software is a pre-scientific, fashion-driven field that resists holding new ideas to reasonable standards of proof and evidence. This makes it tough for business executives to know what to do. There is only one approach that works: roll up your sleeves, put ego and pride to the side, and figure it out using evidence and common sense.

Here's what we can learn from the shift to smart credit card terminals

I’ve been involved in computer software for decades. Lots has changed over that time. One thing that hasn’t changed is the question people most like to ask me. It’s this: “What do you see that’s new and interesting?” It’s a perfectly reasonable question, though one for which I rarely have a ready-made answer.

A question I never hear goes something like this: “What do you see that’s touted as the newest new thing, but is mostly old stuff, and was completely predictable?” Now that’s an interesting question. And the un-helpful but honest answer is “Practically everything that’s touted as a new thing is mostly old stuff, with a little bit of ‘obvious next step’ thrown in for variety.”

Still, there are some unpredictable aspects of the fancy new things: it’s really hard to know WHEN the new thing will happen and WHO will make it happen.

A case in point is … [I’m not sorry about the pun] … the new smart card terminal company Poynt. (Disclosure: my VC fund, Oak HC/FT, is an investor.) I can see eager marketing people at Poynt are raising their hands in the back at this … ahem … point, all anxious to point (groan…) out that Poynt is a pioneer in the market, arguably the inventor of the smart terminal, an amazing device that not only takes card payments, but also rings up items just like a POS terminal and hosts endless numbers of third-party apps. True! I happily concede the point. But I hasten to point out that there are robust competitors in the market, notably including Square and Clover.

The smart terminal is a new thing, and the market is glad to have it, but it’s hardly a NEW new thing, or something where you’d knock your head and say “now who’d-a thought-a that!?”

It’s natural for consumers of technology to look at the new devices and appreciate them for what they are. That’s like being a tourist, driving on a road through the country-side, appreciating the nice new views. That’s nice for the tourist, but are there patterns here, patterns that would enable an educated person to expect something like a smart terminal to appear, and womdering when it would happen?

Yes there are. The main pattern at work here is the rate of change of the underlying hardware. Today’s hardware is something like 1,000 times faster than much larger, more expensive hardware at the turn of the last century, less than 20 years ago. That number may not seem like much, but think of this: the average human walking speed is about 3 mph. The speed of a commercial jet while flying is less than 600 mph, about 200 times faster. Now imagine a human evolving so quickly that the human could walk at the speed of a jet -- and increase in computer speed is 5 times greater than that, in less than 20 years!

What’s the point? Or Poynt? Here it is: there are underlying geology-like forces in the world of computing that make it highly likely that something very much like Poynt would be invented – though as I said, predicting who will do it and when they’ll do it is a whole other thing.

The first step in creating a technology solution to a problem is often building problem-specific hardware. Then the technology evolves, getting faster, cheaper and more capable. Then there’s a tipping point, at which the purpose-specific hardware is replaced by general-purpose hardware, and most of the specific features of the device are implemented in sofware that runs on the general-purpose hardware. Then a new era begins. The general pattern is that special-purpose devices are supplanted by general-purpose ones.

In the case of card processing technology, first we see imprints of cards made on paper, with the physical paper being sent to a central place for processing. Then the big jump to computer technology and networking: a series of increasingly-better charge terminals, specifically made for processing card charges. The terminals evolved from dial-up networking to the internet, and from stand-alone to connected to a point-of-sale system. Wonderful devices!

Now think about cell phones. If you’ve been around for a little while, you remember big phones getting better and smaller and finally evolving into flip phones. Great phones, … but they’re phones. Then came the big shift, to a next generation of phones that were really small, portable, general-purpose cmputers that can run a myriad of applications … with cell phone hardware and software built in. Yes, it’s a phone. But it runs Facebook, email, and any of thousands of applications avialable in the app store. It’s a “smart phone!”

You know this. The reason I’m reminding you of that history is that it’s exactly the transition that card-charging “terminals” are going through right now – as they become “smart terminals,” i.e., small, portable, general-purpose computers that can run a myriad of applications … with card charging hardware and software built in. Yes, it’s a terminal, but a smart one.

How often do you see “I’m just a phone” devices? Flip phones? Yup! The old card-charge terminal will become just as rare a sight in the next couple of years.

So are the new “smart terminals” new? Yes! But hardly unexpected, at least to those who see the clearly repeating patterns of the underlying technology.

A less Poynted version of this post was previously published at Forbes.

Computer Security Breach Response Excellence

Here's what the experts do for computer security:

• Hire security experts to implement best-in-class security.
• Follow all the regulations.
• Pass all the audits.
• Spend lots of money.

Then, of course, you get breached, because in spite of doing the above, you have no idea what you're doing...

Here's how you respond:

• Get more experts to find what happened.
• Establish a carefully-thought-out strategy to recover from the breach and minimize damage to your reputation.
• Alert the public and your users about the event and your concerned, respectful response.

Then, of course, you change your website, put lots of money into attractive graphics, while making it hard for users to login or reset their passwords.

The share-your-expertise website Quora is surely in the running for best-in class when it comes to computer security; they have followed the above plan with true excellence.

The Quora Story

I got this email from Quora, of which I'm an occasional user, on December 3, 2018:

Dear David B. Black, We are writing to let you know that we recently discovered that some user data was compromised as a result of unauthorized access to our systems by a malicious third party. We are very sorry for any concern or inconvenience this may cause. We are working rapidly to investigate the situation further and take the appropriate steps to prevent such incidents in the future. What Happened On Friday we discovered that some user data was compromised by a third party who gained unauthorized access to our systems. We're still investigating the precise causes and in addition to the work being conducted by our internal security teams, we have retained a leading digital forensics and security firm to assist us. We have also notified law enforcement officials. While the investigation is still ongoing, we have already taken steps to contain the incident, and our efforts to protect our users and prevent this type of incident from happening in the future are our top priority as a company. What information was involved The following information of yours may have been compromised: Account and user information, e.g. name, email, IP, user ID, encrypted password, user account settings, personalization data Public actions and content including drafts, e.g. questions, answers, comments, blog posts, upvotes Data imported from linked networks when authorized by you, e.g. contacts, demographic information, interests, access tokens (now invalidated) Non-public actions, e.g. answer requests, downvotes, thanks Questions and answers that were written anonymously are not affected by this breach as we do not store the identities of people who post anonymous content. What we are doing While our investigation continues, we're taking additional steps to improve our security: We’re in the process of notifying users whose data has been compromised. Out of an abundance of caution, we are logging out all Quora users who may have been affected, and, if they use a password as their authentication method, we are invalidating their passwords. We believe we’ve identified the root cause and taken steps to address the issue, although our investigation is ongoing and we’ll continue to make security improvements. We will continue to work both internally and with our outside experts to gain a full understanding of what happened and take any further action as needed. What you can do We’ve included more detailed information about more specific questions you may have in our help center, which you can find here. While the passwords were encrypted (hashed with a salt that varies for each user), it is generally a best practice not to reuse the same password across multiple services, and we recommend that people change their passwords if they are doing so. Conclusion It is our responsibility to make sure things like this don’t happen, and we failed to meet that responsibility. We recognize that in order to maintain user trust, we need to work very hard to make sure this does not happen again. There’s little hope of sharing and growing the world’s knowledge if those doing so cannot feel safe and secure, and cannot trust that their information will remain private. We are continuing to work very hard to remedy the situation, and we hope over time to prove that we are worthy of your trust. The Quora Team

What a bunch of careful, responsible people, those folks at Quora are! So appropriate for a share-your-expertise site!

After this notice, I kept getting the occasional teaser email from Quora, tempting me to click and answer a question or see an answer someone else gave. For example I got this one a couple weeks before the breach:

I know, it's not click-bait for the general public, but definitely a good one for me.

Yesterday I got the first teaser I'd gotten since the breach email reproduced above. Here's the lead:

Not a killer issue, but I clicked out of mild curiosity about the answer, and also to see whether Quora was up and running normally. What I got was a lesson in how to respond to a security breach by driving your customers off. It's true, after all, that if there aren't any users, there won't be any meaningful security breaches -- problem solved!!

Here's the landing page -- a new thing in itself, because clicking on an email used to be enough to identify you.

The cute graphics are all new. I put in my password and got the box in red above, telling me I had to reset the password by responding to the email they sent. OK.

I got a typical password reset email:

I clicked on the link. I got to see even more wonderful new graphics! These guys are really trying! Then I put in my old password, because I wanted to; it's my password, I should be able to pick any one I want, unless they tell me there are rules.

Can't use my old password, huh? If you're so sensitive and caring, you could just possibly have warned me about that up front. Oh well. Here's a new one:

I put it in. It's new. They match. I click on the Reset Password button. Nothing. I change the password and click again. Nothing. Again. Nothing again.

They just don't want me, it's clear. If I were a normal user, it would have been game over. But I'm not, so I went back to the password reset email and clicked again. This time I put in a brand-new password. Then, clicking worked -- it got me to the login page, where I had to enter my email and new password yet again.

Quora has a big, fat, ugly, super-obvious, BUG in their "we're taking responsibility for this breach and hoping to win back the trust of our users" new entry door to their site, not bothering to perform super-elementary QA on one of the main pathways of the new code. Not some obscure condition. Software QA 1.01.

So just who are these geniuses at Quora? Are they the super-smart, rich, cool kids that have such a track record of excellence at other tech sites? Like Facebook and Twitter and the rest? It takes a bit of looking, but the simple answer is: yes. Super-smart. Beyond cool. Rich. And still can't get the most elementary details right!

Business as usual in software. Whether it's government, big corporation or cool young hip tech company the story is the same: getting stuff to actually, you know, old-fashioned WORK is beneath, beyond, above or whatever for whoever's involved. Not to mention make software that protects customer data.