The Black Liszt

Who is Censoring US Intelligence Agency Criticism on Amazon?

A critical review I wrote of a book on Amazon was censored/suppressed. Was it profane or otherwise deserving deletion? No. It simply gave one star and pointed out errors and omissions, while acknowledging that fans of US Intelligence Agencies would admire the book. The review appeared briefly, then the link to it went bad. I interacted with customer service, who said they would submit a trouble report. I wrote another review, which appeared briefly then vanished. I tried once more. The last review was up for months. I got a warning from Amazon warning me that an unspecified review I had written failed to meet community standards, and if I did it again I would be banned. The review stayed up for many more months. And then it disappeared!

Spies, Lies and Algorithms was published by the Princeton University Press. It was written by Amy Zegart, a prominent, well-connected Professor at Stanford. I'm writing up this little experience because I suspect it's not an isolated incident; It's a step towards a society run like Orwell's 1984. Here's the story.

US Intelligence Agencies and Cyber-security

The grossly misnamed US "Intelligence" Agencies continue on the path of proclaiming their wonderfulness while US cybersecurity continues its free fall into unmitigated disaster. I describe the details of the sad state of cybersecurity in both government and private sectors in these posts.  For government cybersecurity, see this. Here is a detailed analysis of the DNC server hack declared by Intelligence agencies to be a Russian hack. They failed utterly to prove it, and ignored the clear evidence that the most prominent emails never resided on the DNC server!

It should come as no surprise that they are doing what all bureaucracies do, i.e., protect themselves and their reputations by hiding the facts and suppressing criticism. The author of the book clearly and explicitly takes the side of the Agencies on this issue and similar ones. While she has given courses to congressional staffers on cyber issues, she clearly does little but echo the propaganda of Agency Experts.

The Book

Amazon, always serving up books they think I might buy, showed me Spies, Lies and Algorithms: The History and Future of American Intelligence by Amy Zegart.

What a cleverly designed cover! I checked it out. The author seemed to be well connected and educated about the Agencies and recent events. Take a gander at her background:

Amy Zegart is the Morris Arnold and Nona Jean Cox Senior Fellow at the Hoover Institution and Professor of Political Science (by courtesy) at Stanford University. She is also a Senior Fellow at Stanford’s Freeman Spogli Institute for International Studies, Chair of Stanford’s Artificial Intelligence and International Security Steering Committee, and a contributing writer at The Atlantic. She specializes in U.S. intelligence, emerging technologies and national security, grand strategy, and global political risk management.

Zegart has been featured by the National Journal as one of the ten most influential experts in intelligence reform. She served as a commissioner on the 2020 CSIS Technology and Intelligence Task Force and has advised the National Security Commission on Artificial Intelligence. She served on the Clinton administration’s National Security Council staff and as a foreign policy adviser to the Bush 2000 presidential campaign. She has also testified before the Senate Select Committee on Intelligence and advised senior officials on intelligence, homeland security, and cybersecurity matters.

Skipping my usual practice of carefully scanning the reviews, I bought the book.

I then read the book. The author was nearly everything you could want: highly educated, knowledgeable, articulate, connected. There was just one little problem: she has completely bought in to the deep state, to the extent of denying that there is such a thing. She faithfully ignores what they ignore, reflects their lies and omissions and shows near-total ignorance of computers and the "algorithms" of her title. That's all! In fact, her book scared me, because it was such a compelling read to anyone who was ignorant of cybersecurity (i.e., practically everyone) or hadn't dug into the big lies.

The Reviews

I've written a number of reviews of all sorts of products since joining Amazon in early 1998. Most have been positive, all of them constructive. I thought this book needed a review from me; while my immersion in the Intelligence community was trivial compared to the author's, my knowledge of cybersecurity and related subjects clearly eclipsed hers.

I went to the review section and found a handful of authoritative-person reviews, all highly complementary, and about a dozen reviews by readers, all but one very positive.

The sole negative review, 1 star, was by a self-described retired Intelligence officer who titled his review "bias detracts too much." 57 people found it helpful. He didn't go into detail but said:

She tries to deny it but the Deep State is real and is our biggest challenge. ... She tipped her hand with selective alarm and omission of key events rampant in the first chapter.

The next most popular review, granting 5 stars and titled "There is some bias: here's why it isn't important" mostly demonstrates the power of being in the bubble:

I am a politically conservative individual studying intelligence and the IC for a number of years. Amy’s summary is accurate, thorough, and well-written. ...
I couldn’t find any factual inaccuracies that would render the book a decorative piece or kindling.

These reviews demonstrate the problems of the Intelligence community and its image. It isn't about bias so much as it is about ignorance and denial.

Writing a Review

I dove into writing a review. My goal was to praise the author for her evident virtues, but to give a few facts to illustrate the basis for my giving it 1 star. I typed it into the box Amazon provided without keeping a copy of what I gave -- silly me. I posted it and waited. Three days later I got a notice that my review was posted.

Hooray! I clicked to see the full review. This is what I got:

Cute dog, but what's going on here?

I went back once a day for many days and clicked again. The review never showed up.

Next I got online with Amazon customer service in a chat. After a bit of effort, the person agreed that the error about the Zegart review was a problem. I asked him to submit the issue for resolution, and he said he would. I waited until he said it was submitted.

Writing a Review Again

After a number of days with no change, I decided to simply post another review, this time saving what I had written. My submission of the second try was confirmed:

I never received the confirming email from Amazon saying that my second try at a review had been posted. But I kept checking, and to my surprise, my second review was posted. The bad link to the original review (Feb 12) now led to the new review (Feb 19). Hooray, it's posted! Here's a screen shot of the first part of it:

There's nothing bad going on here -- I'm just being paranoid! And the review has already gotten several votes for being Helpful.

The Second Review Disappears

I went to the page again, and noticed that the total number of reviews had gone down. I look for my review. I was thorough. It's not there. Not there.

I had a thought. I had saved the original confirming email I got from Amazon pointing to my original review (which never showed up), which mysteriously DID point to my second try at a review. I click, and here's the result:

At least they're different dogs...

The Second Review Magically Reappears

I went to the page again and found, to my great surprise, that the elusive review was back!!

Just as a test, I wrote a positive 5-star review of socks I had bought on Amazon.

The Warning

A day or two later after the reappearance of the review, this showed up in my inbox:

Notice the careful wording. Notice that the exact review was not referenced, and that no specific violation of the guidelines was identified. I carefully read the community guidelines. Nothing that I wrote could be construed as violating them. The only offense of the review was to point out the deep issues of the Intelligence Community and to give a couple specific examples that illustrate it -- one of them the supposed Russian hacking of the DNC server.

The second review stays up for many months, and then...

I checked the review occasionally over the following months. It stayed up and received dozens of "helpful" votes. It was second or third on the list of "critical" reviews.

Then, more than 6 months later, I got this.

Nothing about which content I posted that violates the Guidelines. No links. Nothing about removing the awful content. I'm just supposed to know, I guess. Naturally, I suspected that the Zedaris book review was the issue, so I went to the book's page. The book is doing well -- it's now an editor's pick in nonfiction. I went down to the reviews. The "Bias detracts too much" 1-star review is still up and went from 57 up votes to 102. Sure enough, mine is gone.

Boy, those misinformation police at Amazon are really on the job, giving proper notice and telling you what law you violated. And prompt! And careful to only take down evidence-based reviews that might change some minds. Or something.

The last chapter

I wrote all of the above. Before posting it, I checked once more and found to my surprise that the review was up again! It wasn't on the top reviews list, even though it had over 50 "helpful" votes, but it was there. Oh well, much ado about nothing. The review is visible, my post is irrelevant.

Many months later, I had an impulse to check again. The review is gone again!! I carefully checked, looking at all the 1-star reviews, which takes some clicking just to get to them. The "bias detracts too much" review is still there, still with exactly 102 up votes, unchanged from at least 6 months ago. I bet that somehow the review isn't shown to many people. The handful of other critical reviews each had just a couple votes at most.

My review -- gone again!

Conclusion

Somewhere operating deep within Amazon there is an operation to suppress criticism of the CIA/FBI/NSA/whatever. I guess it could be Amazon employees who feel the same way, without official blessing. Or maybe, as we now know was the case with Twitter, there was an explicit back-channel connection to one of the 3LA's (3-letter agencies). After being allowed to be viewed for many months, my review garnered dozens of "helpful" votes. During that time I thought that the "image protecting" operation was walking a fine line between total suppression and letting low-attention items like my review alone to avoid attracting notice. But then it was visible again, just hard to find. And now, roughly a year after it was originally posted, it's gone.

FWIW and to be explicit about it, my review was NOT politically motivated. It was driven by facts and competence, consistent with these posts. Specifically see this. The job of the agencies is to enforce the law and protect us all from criminal and foreign attacks, a job for which, sadly, they have a track record of failure. And the job of prestigious professors who teach people about cybersecurity is to objectively assess the quality of the work, not ignore failures and incompetence.

Yelp is Big Tech Incompetent and Corrupt

Yelp is a Big Tech company that isn't listed when people talk about "big tech," but it's nonetheless all about tech and it's unarguably big, with well over 100 million unique visitors per month to their site and a valuation in the $ billions.

How good is their software? Are they a company from which non-tech corporate America should poach talent to upgrade their in-house tech? I recently had occasion to spend a little time with Yelp, and found that they are indeed an exemplar of the Big Tech tradition of tech bumbling as I have documented.

Harvard Business School famously teaches according to the case study method. Students are presented the case and challenged with figuring out what to do. Yelp would make an excellent case study. Any of the bright, aggressive b-school students who declared that the "solution" to the case was to fire all the executives and replace the entire product and tech team with talented high school grads, would get at least a B. Anyone who further suggested avoiding hiring anyone with a Computer Science degree would get an A.

Yelp and Local Business

Yelp is an advertising company that features listings of local businesses and user-generated reviews of those businesses.I recently visited my local dentist and received excellent care on the way to getting a crown put on one of my teeth. She knows I'm a computer guy and I've helped her out in the past, so she brought up her concern about a couple of malicious false reviews that were posted to her Yelp page. I'd said I'd check it out. I did. The results were illuminating. I sometimes complain about bumbling big corporate companies and their painful but fixable software problems. Yelp provides a great example of how second-tier Big Tech companies are no better at building software than the super-famous first tier ones are.

Yelp and a Local Dentist Business

I checked out Yelp's page for my dentist, Doctor Vu. It had a couple long negative reviews. So I posted a review of my experience, along with a selfie of me at her office.

I went back after a couple days to check out the page in more detail.

At the top of the page:

I see that she has 5 reviews with an average score of what looks like 3.5.

I scrolled down and found the photo section with the photo I had posted:

I took the selfie in her office holding the coaster I had given her last year that I couldn't resist buying at my favorite store in Cape May. She's had it in her waiting room ever since; I hear it often elicits amusement. It just seemed right to take the picture at the end of a visit to get a crown. :-)

After scrolling through ads and other information I got to the reviews. Mine was first:

I scrolled through the rest, including the two 1-star slams. Something didn't seem right. I went back and carefully counted the reviews. I double-checked. There were 6 reviews! The header at the top said 5! Then I made note of the star ratings. The two super-baddies were 1 star and the other 4 were 5 stars each. That's 22 points divided by 6 reviews = 3.67. Looking carefully at the points at the top, the dividing line is clearly at the top of the star, so Yelp thinks it's 3.5.

But then Yelp said there were only 5 reviews in spite of displaying 6.  Is it 22/5? Nope, that's 4.4. A complete mystery. Maybe I'll figure it out later.

At the end I found this:

What's this about? In faint type 16 other reviews? I check it out. At the top of the page that shows up there's a video I can watch and this long, serious-sounding thing about how Yelp cares and they're smart and they're doing the best for everyone:

Impressive! Finally I scroll down to check out these malicious reviews and here's the first thing I see:

Wow. Billions of data points! That software must be good, right? So I scroll down to see these scurrilous reviews. Imagine my surprise when I see this:

Wow. My review. Again!

At least the mystery of the rating average is solved. Even though my review was shown among the legit reviews -- first, no less -- it appears that their ludicrously bad software had marked it as not recommended and so some piece of software decided there were just 5 reviews. So 3 5-star plus 2 1-star is 17 stars, divided by 5 is 3.4. Close enough to the 3.5 average rating they displayed.

Even better -- I received a congratulatory email from Yelp after posting the review with the picture, saying how great it was and I should post more pictures along with my excellent reviews. My picture is indeed featured. And my review is both first on the list of legit reviews AND first on the list of those not recommended. Score! This is what you get by employing thousands of Silicon Valley's best super-programmers...

Those billions of data points must have somehow confused the Yelpers. And their software.

Finally I read through the rest of the reviews that weren't recommended. There was nothing fake about them.

I visited Dr. Vu again to get actually crowned -- the first visit was just preparation. She told me she had contacted Yelp about the situation. First she learned that the Yelp algorithms were state-of-the-art great, and in any case there was nothing a Yelp employee could do to alter their results. Second she learned that she really should pay Yelp lots of money so she could fix up her page, post pictures, etc. Maybe if she paid something could be done about those bad reviews...

My conclusion from this wasn't hard to figure. Yelp is corrupt (pay to play!). Yelp is incompetent (bad algorithms, double-posting, bad arithmetic). Sensible people should ignore Yelp.

Conclusion

I'm tempted to say that Yelp fits right in with its first tier Big Tech buddies, and of course it does. If you can stand it, check out the summary of their issues on Wikipedia. I did and also went to some sources to make sure it wasn't a Wiki hit-job. I also checked out the disastrous business and personal judgment and behavior of the leaders. Pathetic. Yelp is now on my blocked list of websites.

Software Giants: Image and Reality at Facebook

I am perpetually amazed at the flood of reverent articles about the wonderful big software companies that are inflicted on us. How great are their leaders! How wonderful it is to work there! Everyone should emulate their business practices! Their products are awesome!

The reason why the sycophantic flood continues is based in simple economics, but why most people appear to buy the b.s. is beyond me. You don't have to be a hardened cynic to see past the image.

This subject is worth a book or two. I've contributed just a couple blog posts. This post is another one on the wonderful Facebook, which (supposedly) does so much to demonstrate software excellence and contribute to our betterment.

Why Facebook is Wonderful

An article has just appeared about the wonderfulness of Facebook. The article is an interview by John Battelle with Lori Goler, who is "VP of People" at Facebook, leading the company's growth from 500 employees to 15,000. Here she is:

She sounds like a really nice person. I've worked with the interviewer, John Battelle, at one of his prior ventures, and he's a great guy.

The whole article is worth scanning. But the subhead gives you the idea: Facebook is "the world's most admired employer." Here are a couple quotes from Ms. Goler:

We are really looking for builders...What goes along with that is a learning mindset.

Being a strengths-based organization is a place where you are really looking to put people in roles where they are doing work they enjoy that plays to their strengths. ... It’s where you get the best teamwork. It’s where the people are able to do the best work of their lives.

For us, the mission is, “To make the world more open and connected,” so it makes sense that our culture is open and connected. Then internally, we reflect that culture.

What we find is that what it really means is that people have all the context they need to be able to work with great autonomy in the organization, which of course leads to greater innovation and greater impact. It’s been a virtuous cycle for us.

According to the article, Facebook is a great place with a socially uplifting mission, populated by great people who are always learning and in roles where their strengths are tapped and their work has impact is fulfilling, in a completely open and supportive environment. Wow. Who wouldn't want to work there?

Another view of Facebook

For a contrasting view of Facebook, I recommend reading this book:

Warning: I had to force myself to get through the book; the author's self-described behavior was distasteful, to put it mildly. But it made the rest of his descriptions the more credible, and he said nothing that contradicted my inside knowledge.

The book has gotten lots of attention. It's been reviewed by major media, for example the New York Times:

And by tech journals, for example Tech Crunch, which declared it was the "year's best non-business book about business"

The book is #1 in several categories at Amazon. The top-rated review is telling:

....

Perhaps you can see that there is a contrast, shall we say, between the wonderfulness of Facebook as presented by its leaders and the reality. But this makes sense. What was the job of the VP People before getting that job? Marketing! In other words, telling stories to get you to buy stuff. She is continuing to do her job well, i.e., selling Facebook as a great place to work.

Facebook's Product

Well, maybe it isn't such a great place to work in spite of all the propaganda, but at least those 15,000 people turn out a great, high-quality product, right?

Here's a post about software quality issues with a section on Facebook, and here are details about the inability of those 15,000 engineers to turn out a product that has reasonable quality, even after many attempts -- as judged by their own users.

It's not just Facebook. It's Google and the rest. Think about this: with such wonderful employees and huge cash reserves, why can't they make their own products work, much less innovate? If they're so innovative, why does so much of what they "innovate" come from acquisitions? See this for details.

You might ask, if their software quality is so awful, how did they become so big and valuable? Good question. Zuckerberg made some world-class smart business strategy moves to get it going. See here for details.

Why this matters

These observations about image and reality at the big famous software companies have huge practical implications for small companies, managers and programmers.

I have often observed that when board members want to hire a top executive, or when managers want to fill an important software position, they often value highly a candidate's having done a stint at one of these famous giants. They'll think something like "Facebook has a product that nearly everyone uses; I want to build a product that nearly everyone uses; therefore, I'll hire people from Facebook, and I'll get a product that everyone uses."

Of course rarely will someone come out and say something like that, but the Facebook (or Google or whatever) aura is so strong, people often act as though they believe it. On the other hand, if you really get the perspective about the inept software giants described here and confirmed widely, you will tend to avoid hiring people from Facebook (or wherever) because you know you're likely to drag down your company to its abysmal level!

Lots more detail on this and related subjects is in my book on Software People. Or for an illustration from a whole different direction, consider the incompetent doctor and nurse in the PBS civil war hospital drama Mercy Street.

Nurse Hastings frequently brings up the fact that she served with Florence Nightingale as proof that her opinions are the best.

Conclusion

Nothing is going to change. Major corporations of all kinds, even more so the big tech ones, will beat the drums of self-promotion, selling themselves to customers and potential employees. It's in the interest of groups that hunger for money and attention from the big companies to make nice, and trumpet the self-congratulations. The big companies will continue to be unable to innovate, and will instead buy innovative companies. Sometimes the contrast between the image that is widely promoted and the reality gets to me. At minimum, my hope is that you're working at a place that's far better than Facebook, and that you avoid error of attribution I have described here.

Organizing for Successful Innovation: Recent History

One of today's hottest trends is fostering innovation. It's real important! There are books, conferences, certified experts and all sorts of things. Let's do two things: (1) look at the origins of today's acknowledged tech leaders; and (2) see how those tech leaders innovate today.

The origins of today's tech leaders

What would we find if we looked at the origins of some important organizations that took the market by storm, grew rapidly and became part of the modern landscape? Did they come from people following the popular methods for fostering innovation? Let's look at some big, successful tech companies, and find out how they got started. There are two possibilities:

1. It came out of some large organization that followed modern innovation methods. Or its founders were avid readers of books on innovation, certified innovation trainers, attendees of innovation conferences, or otherwise showed that they were nurtured by innovation thinking.
2. It was started by one or two people who set out on a mission without any of #1 and kept marching forward until they got it done, perhaps with the help of VC's (venture capitalists).

I've already discussed the cases of Microsoft, Facebook and Oracle here. Their founders not only lacked training in "innovation," they were all college drop-outs!  How could they have possibly founded three of the largest, most successful and valuable tech organizations? Must have been luck, I guess.

Maybe they're the exceptions! Maybe most of the rest fit comfortably into the mold of #1! Let's look at a few:

• Apple. Jobs and Wozniak. College drop-outs.
• Amazon. Bezos. Princeton grad, worked in finance, hedge funds. Jumped on a vision.
• Dell. Michael Dell. Started in his dorm room, dropped out.
• Google. Stanford grad students, dropped out. VC backing.

Why all the college drop-outs? In each case, a founder who was already obsessively good at something saw a related opportunity (sometimes with a buddy), dove in to make sure he didn't miss the opportunity, and made it more important than anything else. Including "education."

A pattern seems to be emerging here. It's not looking good for theory #1.

How today's tech leaders foster innovation

Now that these companies are big and successful, surely they are great at innovation, right? They must have certified innovation experts just crawling the halls, and a good fraction of the staff out attending innovation conferences, right? They must be just cranking out innovations left and right!

They try, in various ways. But it turns out that these great companies aren't any better than any other large organization at innovating -- and you can see it by all the acquisitions they do! They have HUGE resources -- how could some scrappy bunch of nobodies possibly come up with something they couldn't invent themselves?

Well it happens. It's happening right now, in AI. Look at who's doing the acquiring:

Facebook alone has made more than 50 acquisitions, most of them since 2010.

How about Google, who are supposed to be the smartest and most innovative of all? They've acquired more than 180 companies. Someone figured out the ten most expensive buys:

And the grand total is ...

Now, that's innovation for you. Just ask your boss for a multi-billion dollar budget, and you'll be able to innovate like Google!

Until then, remember, there really is a better way to approach making your company better, even if it is unlikely to win any awards for "innovation."

Large Organization Software Fails: the case of Microsoft Windows

Large organizations have trouble building software. This has been true since the dawn of software history, and shows no signs of changing. The decades-long, rolling disaster of Microsoft Windows is a great example of this. I've been hit personally with this. Recent experiences with Windows 8 have renewed my appreciation of the breadth and depth of the on-going awfulness of Windows.

Windows Screen Saver

I got a new computer. It had Windows 8. I was setting up my new machine and I wanted to do something simple. I had remembered that in some earlier version of Windows, you could get the screen saver to display the file name of the photo it was showing. This was useful if you wanted to get your hands on the photo that just flashed by. It's a pretty small feature, but one anyone who stores photos on their PC could find it useful.

So I drilled in to the screen saver.

I went into the settings, and didn't see the control I was hoping would be there.

So I clicked on Help, something I rarely do, but what the heck, that's what it's there for. Here's what I got: The content is missing!

It's a little thing. It's not like my computer crashed. In the world of books, it's like a footnote was missing -- hey, that's an idea, let's compare the new edition of Windows to the new edition of a book!

Software and Books

Most of us know how to judge books. If a book is poorly produced, like the pages tear easily and the type is hard to read, most of us will toss it aside -- it may have great content, but it's not worth reading. If we get past the first impression, we'll dive in and start reading. The next potential barrier is how well the book has been edited. If the book is full of spelling, usage and grammatical errors, many of us will think poorly of the author, the editor and the publishing house -- the author shouldn't have made the mistakes in the first place, the editor should have caught and corrected them, and the publishing house shouldn't have put sloppy trash in print. Then and only then do we get to the style and substance of the book.

I read a lot of books from many publishers in many genres -- fiction, history, science, etc.  -- and I'm happy to report that I rarely encounter a published book that has editing errors.

And by the time a particularly timeless book gets to later editions? There are never errors.

In that context, how is Windows 8?

I've got the latest version of Windows, 8.1, running on a new machine. It's hardly a first edition. Microsoft pours out updates, and I'm up to date. Here's a snapshot:

Note the scroll bar -- there were hundreds more updates that had been applied.

The lovely option that lets you see the file name along with the picture was in an earlier version of Windows. Making a new edition of software isn't that much different than making a new edition of a book -- basically, unless you add or change something, it stays the same. In this case, someone had to make a conscious decision to drop an isolated, harmless feature that gave value to many customers.

Why would someone do that? It's more trouble to drop a feature than just let it ride along on the next edition, so someone had to actively remove it. There is no conceivable objection to the feature. While not everyone would want it, since it's an opt-in feature, it harms no one. It's like someone deciding to drop a short appendix from a book -- not everyone will want it, but those who do value it. In the paper publishing world, dropping it might save a page or two. But in the electronic world? There's no conceivable reason.

I don't claim for a second that displaying the file name on the screen saver is important. I simply claim that the decision to drop it exemplifies the pervasive anti-customer attitude of the Microsoft organization, which unfortunately is typical of large software-building organizations in general.

It's the missing Help file though, that really set me off. Again, it's a trivial error, like dropping a footnote. But why would you do it? How could it possibly slip though what should be a totally automated editing/QA process?? It may somehow be complicated in the labyrinthine world of Windows development, but it's a fixable thing. You have a program that assures that for each instance of Help there's a corresponding piece of content, and for each piece of content there's a way to reach it. There either is no such program or it's broken. In the overall scheme of things (Windows remains horrifically slow, it freezes and crashes, etc.) it's a small thing, but surely by the edition of Windows 8 I am suffering with it would have been found and fixed?

Conclusion

Software is all about productivity, attention to detail and automation. Unless you've got a de facto monopoly, software is also about meeting customer needs. Large organizations in general (for example government, big corporations) and Microsoft in particular don't get that, in spite of the billions they spend on development and (supposedly) quality. I would love to be able to say it's getting better, but most of the evidence is on the other side. Which is why, among other things, good software will continue to be produced mostly by organizations that are small and willing to do things the "wrong" way.

Facebook's Software Quality: the Implications

I have pointed out Facebook's lack of desire or ability (who cares which?) to deliver software that actually works. I've pointed out that they're hardly alone in this respect. It's important to accept this observation as true, so that you can change behaviors that may have been unconsciously predicated on the supposition that Facebook delivers great software, effectively and efficiently. They don't. So don't hire their people and expect great things to happen, and don't mindlessly emulate their methods or use their tools!

The Unspoken Assumption

Facebook is a wildly successful company, worth over $200 billion. I'd like my company to be worth even 1% of Facebook. So I better find out what Facebook did, and learn from it. Facebook is a software company, so their engineers must be smart and effective. I better get some of them in so they can teach us the "Facebook way." And their tools -- wow. If Facebook uses something, what an endorsement that is. My guys had better have a real good reason to use something else; I look at what FB's worth and what we're worth -- don't we want to be like them? If a tool or method is good enough for FB, it should be plenty good enough for us.

The role played by software in FB's success

Here's the logic:

FB is wildly successful.

FB is built on software.

Therefore, FB software must be wildly excellent.

We already know by examining the quality of FB software that it's crappy. So we have reason to suspect that the virtues of FB software may NOT be a driver of FB's success. Consider this thought: What if FB is wildly successful IN SPITE OF its crappy software? If that's true, the LAST thing you'd want to do would be to infect your reasonably healthy engineers with disease vectors from FB.

Explaining FB's Success

There are lots of reasons software companies can become very successful other than having great software. In fact, by the time a company gets large, bureaucracy and mediocrity normally take over, and any great qualities in the software are normally eliminated. The most common reason a software company gets and stays successful is the network effect, the self-validating notion that "everyone" is using the software, therefore I should too.

The network effect becomes even more powerful when there's a marketplace. E-Bay is a great example. If you're a seller, you want to sell in the place that has the most buyers. If you're a buyer, you want the greatest choice of things to buy. Similarly, if FB is where all your friends are, you'd better sign up -- which makes the network effect even stronger.

FB, by chance or plan, leveraged the network effect for growth brilliantly. Harvard already had a physical book with everyone's pictures in it, called the Facebook by students. The basic education and promotion problem was solved out of the gate: Harvard students knew what a "facebook" was; they all had a physical one, and used it, if only because their own information was there. For example, here's me in the 1968 edition:
However straight-laced those Harvard freshman looked, a fair number of them were hackers and troublemakers. Here's the very last page of the 1968 FB. Look at the last guy listed.

There's a similar entry, with a different photo, at the start of the book.

Zuckerberg was solidly in the long-standing Harvard hacker tradition. He had already illictly grabbed student photos for a prior application, which both got him in trouble and made him famous on campus. So when he launched "thefacebook," of course all the Harvard students would check it out. He did this in January. It was used by about half of all Harvard undergrads within a month.

His next smart move was to open it just to students at a couple more elite schools, and then Ivy League schools. Once established there, he expanded. He did NOT open the doors and let anyone join -- he moved from one natural community to the next, letting the network effect do its magic before moving on. Finally, alumni were allowed to join, but only if they had a .edu address proving their affiliation. That's when I joined. Only after a whole generation of students had made it the standard did FB allow their parents to join.

The quality of the software had nothing to do with this. If people had to pay for it, FB would have flopped. Feature after feature came pouring out of the self-declared brilliant minds of the top people at FB, many of them flops, mixed in with scary experiments with privacy. But it was "good enough" most of the time, it's free, it's where your friends are, what can you do?

The conclusion is clear: FB grew to be a huge success IN SPITE OF having rotten software quality and development methods that are just horrible.

The FB environment and yours

Facebook software development methods and tools are NOT something a small, fast-moving, high-quality software shop should want to emulate. Their quality methods in particular are not only trashed by their users, but also by a fair number of ex-employees. The same thing goes for the computing and server environment.

If you find a talented ex-FB-er, by means hire him or her -- but only after verifying that they're sick of how things are done at FB and want to work at a high-quality place.

Above all, don't emulate the actions of FB's leadership. It's the network-effect flywheel that continues to bring eyeballs to their applications, NOT their great software.

And think about this: if they're so brilliant and such great developers, why have they done about 50 acquisitions in their short life, a couple of which are important to their growth?

Facebook's Software Quality: the Facts

Facebook is an incredibly successful company, one of the most valuable on the planet. It is natural to assume that a main reason for this is that they've got a boatload of great programmers who produce code that users love. This assumption is wrong. In fact, the widespread adoption of Facebook masks deep, long-term quality issues that are not getting better.

Facebook Success

Facebook recently passed $200 billion in market value. Amazing! It has billions of users world-wide and has no serious competition. No one can question FB's success in user count and market capitalization.

Facebook Mobile App

Mobile device use is going through the roof. We are in the middle of a massive, rapid migration from workstations and laptops to tablets and smart phones. This trend impacts FB just like everyone else. At the recent Money2020 conference, a top FB executive laid out the numbers, which are stunning; in short, FB mobile use nearly equals normal web use. If anything is important at FB, it's got to be getting the mobile app right.

Facebook Mobile App Quality

So how is FB doing, this premier, ultra-successful company with no lack of resources to do an excellent job? They've got to be doing way better than the rest of the industry, right?

Let's start by looking at user reviews:

Not too bad, 4 stars out of 5, right? But out of more than 22 million reviews, more than a quarter gave 1, 2 or 3 stars, more than 6,000,000 reviewers! Let's look at a few of those reviews. (I didn't scan for exceptionally bad reviews; I just picked off ones that were near the top of the Play store.)

Here are a couple reviews. Cindy gave 1 star because the app doesn't work at all, and Johnny gave 2 because he suddenly can't avoid being buried in notifications.

Here are a couple more reviews. The third reviewer gave 3 stars even though the app is basically disfunctional.

These are educational:

The 3 on the left describe things that worked on a prior release that no longer work, which is the cardinal sin of quality testing. Look at Bratty's review awarding 4 stars, even though he/she can't use the app at all. Makes you wonder if anything but 5 stars is good for FB. Jeremy's review sums it up: "you're still not listening to your users." If only 5 stars represents satisfied users, the ratings mean that about half of FB app users have a serious bone to pick. Which is quite a statement.

FB App Quality in Context

Compare the performance of the FB app to the performance of your car. Getting a new release of the app is similar to getting your car back from the repair shop, only with little trouble on your part and no expense. Most cars run pretty well -- they start in the morning, run through the day, and rarely break down. When you get your car back from the repair shop, it's even better, even less likely to break down.

Not true for FB. Even though it's "in the repair shop" pretty frequently, the FB "mechanics" all too often find a way to break things that used to work, and fail to fix things that didn't work when it went into the "shop." FB programmers and managers think they're way smarter than auto mechanics, but if the car people performed even a little bit like the FB crew, they'd be out of business. The reality is that, with all their oh-so-highly-educated-and-smart mountains of cool (mostly) dudes, the FB crowd can't come close to delivering the quality that nearly every corner-garage mechanic delivers every day.

FB quality stinks, and it stinks for their fastest-growing, flagship product. In saying so, I'm simply summarizing the expressed experiences of literally millions of their users. There are ways to achieve high quality software. FB does not lack the resources. The fact that they don't deliver quality and aren't even embarassed about it tells us that they just don't care.

The Bogus Basis of "Trending on Twitter"

People write and talk about what's "trending on Twitter" as though the trend meant something. It doesn't. It's based on deeply flawed Twitter search software that gives random, widely varying results. I know the weatherman is often wrong, but what if he said it was going to be sunny in the 70's tommorow and as often as not there was a blizzard -- would you keep listening? It's the same with Twitter, only worse.

Trending on Twitter is everywhere

It's amazing how widespread this useless stuff is. New York Times editors are in on the game.

It's even now got a prominent place on Wall Street!

You can not only follow what's trending in general, but you can narrow it down to different locations.

When a Twitter account is hacked, bad things happen.

And sure enough, the markets react.

We seem to care not only about what the Boston bomber says on Twitter:

But we also pay attention to the useless Twitter trends about it:

We've really got to stop this. It's not as though we've got reliable data here. It's just not. Twitter has been a technical joke for years, and there are no signs of improvement.

Trending on Twitter is meaningless garbage

I don't have the access to perform a universal test. But I did perform a test, and anyone else can reproduce my results. I did searches over a couple week period for the same term and saved the results. Sometimes the results were correct, but most of the time, items that were there before disappeared, only to pop up again on a subsequent search. Sometimes just a couple things were missing, and sometimes the gap was massive. Here is the evidence.

Then I took the search that appeared to have the most gaps, and performed the identical search about a week later. As I documented, one search had just 5 items and the other had 32, when they should have been identical. About 85% of the search results had been dropped by Twitter!

"Trending on Twitter" is based on comparing results of a search performed on one day to the same search performed on other days. If the number of results goes up or down, you've got a trend. Or so you think. But what if the results are really as bad as I have documented? I found that "blackliszt" went up or down by a factor of 6, like 600%! Wow!

Conclusion

Twitter software has always been bad. Management has learned to disguise the awfulness by suppressing the appearance of the "fail whale," but they clearly haven't actually, you know, made the software better. Anyone who takes its results as actually meaning something is depending on bogus data.

 

Twitter Software Quality: An Oxymoron

Twitter software quality Stinks. As I've demonstrated. On revisting and updating the facts, I've decided that "Twitter Software Quality" should be promoted to the status of oxymoron, joining the august company of terms such as "southern efficiency," "northern hospitality," and "government worker."

A Brief History of Random Awfulness

I took samples of searches for "blackliszt" on these dates: Apr 18, 19, 20, 22, 24, 25, May 1, 8. A total of 8 samples.

All searches were done as "All" to tell Twitter I wanted, you know, all the results, not just the ones Twitter felt like disclosing at the moment.

I only grabbed the first page from each search. I've shown the results in another post. Of the 8 searches, the one on May 1 is the most extreme. Here's a copy of the May 1 search for "blackliszt:"

You can see there are 5 tweets in the list of results, from Apr 11 to Oct 13. I decided to try to find out how many tweets there actually were between Oct 13 2012 and May 1, 2013, the date of the search pictured above.

I did this research on May 8. At least on May 8, Twitter was willing to admit that there were a total of 32 tweets in the same date range, although one of them (Feb 27) appears twice. Here they are:

A Twitter search for "blackliszt" performed on May 1 resulted in a list of 5 tweets going back to Oct 13. The same search for "blackliszt" performed on May 8 (above) resulted in a list of 32 tweets that should have been returned by the May 1 search. Maybe there are more! Given that one is double-counted (Feb 27), who the &*() knows?? What I do know is that on May 1, Twitter decided to discard 27 out of 32 potential results of a search. Roughly 85% of the tweets were gone!

Summary

I already knew that Twitter software quality was bad. It turns out that it's worse than I ever imagined. It's "Twitter-quality"-is-an-oxymoron bad.

You know all those "trending on Twitter" items you're seeing now that seem so modern and cool? They all assume that getting more or fewer results from a search means something. We now know that the results can easily go up by a factor of six, or drop by the same factor, just because of Twitter "quality." It's obvious that "trending on twitter" deserves to be the punchline of a joke, not something that anyone pays attention to.

Twitter Software Quality Stinks

There are big problems with software quality. The problems range from social apps to corporate to academia, include "mission critical" software, and everywhere in between. The social apps in particular have decided it's embarassing. But instead of actually, you know, fixing the problems, they seem to have decided to mask the problems! Twitter is a great example of this disease.

Two ways of Responding when you don't know the Answer

Suppose you're a kid and someone is demanding answers from you. Either you know the answer or you don't. If you know the answer, it's simple:  just give the answer!

Q: When did Columbus sail the ocean blue?

A: 1492

If you don't know the answer, there are two ways to respond: the right way and the wrong way. The right way to respond is simple: Just say you don't know!

Q: When did Columbus sail the ocean blue?

A: I don't know.

The wrong way to respond is a little more complicated. You have to guess at an answer, state it as though you knew the answer, and hope no one cares or that the person asking doesn't know either so you can get away with it.

Q: When did Columbus sail the ocean blue?

A: 1542.

When the question you're asked has several answers, you can be wrong in a different way. For example:

Q: Name the ships in Columbus' voyage to the New World.

A: The Nina and the Santa Maria.

Q: Is that all of them?

A: Yes.

Twitter's Response when it doesn't know the answer

I never thought it would happen, but now I have fond feelings for Twitter's Fail Whale, which I haven't seen recently. You would think that the fail whale not showing up as often would be a good sign. It's not. It's a sign that Twitter has decided that it's better to lie than to admit it doesn't know the answer to the question you're asking. Instead of forthrightly saying "I don't know," Twitter now brazenly gives the wrong answer. Even worse, it gives a different wrong answer from one day to the next!

Twitter's Bogus Search results

Here are some screen shots of the results of the identical query, for "blackliszt," over a couple of weeks. I always selected "All results" to remove any excuse that Twitter was selecting the "top" results to help me out.

Let's go through time. Here's the result from the first day, Apr 18:

I tried again the following day, Apr 19, and was quite surprised with the result: the Rebelmouse tweet simply disappeared, pulling an older one into the results!

On Apr 20 I added a tweet and did the search again. My new tweet was there, and RebelMouse came back!

On Apr 22 I tried yet again and got another brand-new variation: this time Cadencia's tweet disappeared!

The results were unchanged on Apr 24 and 25. I gave Twitter a couple days to lose some data, and had my patience rewarded when I searched again on May 1. The first result was Rebelmouse; the most recent posts, my post on ballet, Cadencia and Rob Majteles, were all gone! Here's May 1:

Finally, look at this simple list of my tweets taken Apr 23, not a search:

Note that I had tweets on Apr 10 and Mar 25, both of which included "blackliszt," neither of which appeared in any of the search results!!

Sadly, I can't even claim that the folks at Twitter have it out for me. It's just the way things work there ... uhhh, I mean, the way things don't work there...

Conclusion

Social Media software quality stinks. It's worth every cent you paid for it. Oh, you didn't pay anything for it, you say? Well, that's my point. When a program like Twitter gives you an interface, lets you do a search, gives you a result that's even worse than my "Nina and Santa Maria" answer, brazenly implies that it's the right answer and everyone just ignores the issue, something is wrong. 

Q to Twitter exec: Why does your software randomly leave out results from searches? Why should anyone look at "trending tweets" or anything else when the data is randomly bogus?

A: I've never been asked that question before. The answer is simple: I do it because I can, because I don't care, because no one else seems to and because I'm worth a great deal of money and you're not. Next question please.

Thanks to MaryAnn Bekkedahl for inspiring me to write this up.

Internet Software Quality Horror Shows

Whether the software is a cool social app, an academic website or a real business, there is a common theme: the software is poorly designed and, even worse, it just breaks. As in falls flat on the floor, waves its arms in surrender, and just gives up. And not just once -- it keeps breaking! As I've said before, we really need a revolution in software quality.

Cool Social Apps

Hey, social is where it's at -- how can billions of Facebook users be wrong? Before long, there will be as many FB users as MacDonald's has sold hamburgers (billions and billions)!

Those guys must be great programmers, huh? I mean, just look at their office:

Here's one of them giving a talk at a conference:

See how cool he is? He's just wearing a t-shirt, not even "business casual."

The other social media are just as cool. Here's a "chill" Twitter office:

And Jack Dorsey, the Twitter CEO -- quite the opposite of a buttoned-down financial guy, huh?

It's perfectly obvious that these guys must write just the coolest, most awesome code ever. There's no way people this cool could make elementary programming mistakes, particularly when their application is so very dead-simple, and hardly ever changes -- they could spend practically all their time being cool and polish up some already-faultless code a couple times a day, and still be OK.

Except this little detail, which I scraped from my own screen, and which I personally have seen countless times:

Yes, the famous Twitter fail whale. I think Twitter got tired of all the publicity their "cute" failure message was getting them, so they reverted to something more discrete; here's an example:

FB is just as bad, of course, and they've always tried to minimize the message when they screw up:

Apparently, FB is incapable of keeping even the most recent day's worth of updates on-line -- you should try going back in history and seeing how far you get. Oh, you thought the stuff you wrote was your data, did you?

Naturally, it makes sense to consider that you get what you pay for; all these cool social apps are, after all, free. You can hardly complain when something you didn't pay for is flakey -- return it and demand a full refund!

So let's turn to a more promising field. Everybody's supposed to go to college and learn stuff, so...

Academia

Let's see if the universities do any better. I was just on a local college's website, and it was even worse than Twitter -- Twitter's code knew it was screwing up and put up the fail whale. In this case, any number of links I hit encountered badly broken code:

Oh, alright. The colleges are perpetually underfunded, and putting up a website that works isn't a high priority compared to ... all the other things they spend money on. I guess.

Probably a real business does it better, right?

Profit-making Big Company

Even more so, an essential public service, like the cable company! Those guys have the money, the funding, the experience and the mandate to do it right. Let's pick the case where their motivation is the highest: collecting money.

Oops.

Just a few days ago, I was on my local cable provider's site trying to access my account. Here's what I got:

Not just once, but repeatedly, for hours!

But maybe it's just TW that's got problems -- surely all the other big companies do things great, with their huge staffs and policies and procedures and all, right?

Sadly, no. Here's just one personal example from Verizon:

Summary

There's no getting around it. Software is just bad. Everywhere. We can speculate about why this is the case, but let's agree on the facts: it's bad, and not getting better.