The Black Liszt

Does Vaccine Efficacy of 95% mean I won't get sick?

The Moderna and Pfizer Covid vaccines have 90-95% efficacy, but the studies submitted for their approval showed they helped only about 1% of the people who took them. This is news to most people. How can this be?

We are constantly told that vaccines are safe and highly effective, for example by the CDC. Numbers like 90% efficacy are thrown around, which most people understand to mean that getting vaccinated means there's only a 1 chance in 10 that you'll get sick. You're really protected!

What the CDC and major authorities fail to disclose is that standard statistical methods applied to the vax vendors' own data shows that only about one in a hundred people who get the jab would be protected from getting covid! The tests did indeed show 90% or better "efficacy" (relative risk improvement), but what's more relevant is "absolute risk" (AR), which their own data showed was around 1%.

Read on to understand these industry-standard measures that are mostly ignored; if widely understood and acted on, they would transform not just vaccines, but pharma and public health in general.

Winter Coats and Vaccines

Winter coats are a standard solution to protect people from getting cold when the weather outside is cold. Kind of like when the air is suffused with invisible vaccine particles, you want to help your body defend itself.

There are a wide variety of coats available to protect against the cold. What would happen to a new coat vendor that promoted its coats as being highly effective against the cold, protecting most people who wear them, but it turned out that the maker and seller knew that 99% of the people who wear them on a cold winter day wouldn't be helped by them -- word would get out quickly and the coat maker's reputation would be in the cellar.

What would happen if major authorities had subsidized the coat making, regulated their testing, and then promoted them as "safe and effective?" And then what would happen if all the authorities demanded that you buy and wear the coats, to the point refusing to let you enter a football stadium on a cold day unless you were wearing one of the approved coats? There would be mass revolt. Which is what would have happened with covid if people knew the facts that were so carefully concealed from them.

When locations like restaurants and performance halls opened, authorities in places like New York City declared that only people with proof of vaccination would be admitted. People were eager to eat out and be entertained, so this was another reason to get the jab. Vaccination cards were checked on entry so that everyone could be "safe."

While covid is the most current example of this grotesque propaganda/misinformation, it is all too common in healthcare and pharma, as I have shown for example here for saturated fat, here for cholesterol and here for hypertension. What's new in covid is the level of coercion involved.

Relative risk, absolute risk and Number Needed to Treat (NNT)

The widely used number for a vaccine called "efficacy" is technically "relative risk" (RR). In scientific papers, it's typically a number like .05, which means that compared to the number of people who got sick without the vax, just .05 of the vaxed got sick. This is translated to saying 95% of the vaxed avoided sickness compared to the unvaxed who got sick. While technically true, it is NOT about your chances of getting sick or staying well. It means relative risk, which is how much better the vax is compared to those who had no vax and got sick, independent of the number of people in the study.

Let's go back to winter coats. When people go out in the cold, they put something on to keep warm. Sometimes the coat doesn't keep some of them warm enough. Suppose the august health authorities got real worried about people dying of the cold without adequate protection. Huge amounts of time and money were spent developing what the developers thought was a great winter coat. Never mind that, for various reasons, the vast majority of people weren't getting cold. They went to a northern football stadium near the end of play-off season (winter). They got everyone entering at half the entrance gates to wear their wonderful coat and everyone who entered at the other half to wear a fake, ineffective version of the coat (the placebo) on top of whatever they were already wearing. At the end of the game, they briefly interviewed and temperature-measured everyone who left, noting which version of the coat they wore.

Let's suppose that 20,000 people went to the football game, with 10,000 getting fancy new coats and the other 10,000 getting fake coats. Suppose 10 people wearing the fancy new coat got cold, while 100 people in the fake coat group got cold.

First let's calculate the number everyone talks about, efficacy, technically known as Relative Risk (RR). RR in this case is 100 minus 10 divided by 100 = 90% efficacy. The wonderful coat did much better when added to what people were already wearing, about ten times better than the fake coat (placebo)! This is the number everyone thinks means that 90% of the people who take the vax won't get sick. Except it doesn't mean that. The key to understanding that is that RR has NOTHING to do with the size of the group, the number of people getting poked.

So let's calculate Absolute Risk (AR). In this case, of the 10,000 in the fake coat (placebo) group, 100 got cold, which is 1 in 100, for an AR of 1.0%. Your chances of avoiding getting cold without the fancy coat were excellent -- 99 out of 100! For the 10,000 people in the fancy coat group, just 10 got cold, which is 1 in 1,000, an AR of 0.1%. The relative difference between the fake and real coats was truly big -- ten times! But the absolute difference means that 10,000 people had to get the fancy coat in order to avoid just 90 of them getting cold. The reduction in absolute risk was 1.0% - 0.1% = 0.9%.

How many people have to get the fancy coat in order for one to benefit? Scientists have a name for this. It's NNT: Number Needed to Treat, sometimes called NNTV (Number Needed To Vaccinate) when a vax is involved. While "efficacy" focuses on "relative" risk, NNT turns the absolute risk (AR) into a more relevant number -- of those getting the treatment, how many will benefit? In this case, all 10,000 football fans would have to wear the fancy coat so that about 100 wouldn't be cold, ignoring the 10 who got cold anyway. In other words, in order for one person to benefit, 100 people have to get the treatment, an NNT of 100. For the other 99, the fancy coat made no difference -- they would have been warm without it.

Getting back to reality, this means that the coats most people choose to wear protect them from getting cold remarkably well. Anyone surprised? What's the normal reaction to being in the stands and getting cold? Doing something to warm up! Jump up and down. Wave your arms. Drink a cup of hot cocoa. Get hugged. Sit on someone's lap, get wrapped in their coat. If worse comes to worse, leave for someplace warm. There are "treatments" that work just fine.

Why would anyone bother accepting and wearing the authorized coat on top of what they already have? In the vast majority of cases, they'll be fine without it, and there are things they can do if they start to feel cold. Not to mention the risk of side effects of the fancy new thing. Here and here are more detailed explanations with examples.

ARR and NNT for Covid

I used round numbers above to make sure the concept was clear. But the whole point is the real world. There is a wonderful scientific website that provides NNT's for many treatments, based completely on scientific studies. For example, here is their article on cholesterol-reducing statins. which makes it clear that no one should be taking these widely used but destructive drugs.

Let's turn to the NNT for covid. What's amazing about this is that the information about NNT for covid is hidden in plain sight. Let's look at the FDA's announcement of their EUA (Emergency Use Authorization) for the Pfizer covid vaccination. The FDA states:

The FDA has determined that Pfizer-BioNTech COVID-19 Vaccine has met the statutory criteria for issuance of an EUA. The totality of the available data provides clear evidence that Pfizer-BioNTech COVID-19 Vaccine may be effective in preventing COVID-19. The data also support that the known and potential benefits outweigh the known and potential risks, supporting the vaccine’s use in millions of people 16 years of age and older, including healthy individuals.

Later in the same announcement, the FDA gives the details about how good the vaccine is. Here is the start of the key paragraph:

FDA Evaluation of Available Effectiveness Data 

The effectiveness data to support the EUA include an analysis of 36,523 participants in the ongoing randomized, placebo-controlled international study, the majority of whom are U.S. participants, who did not have evidence of SARS-CoV-2 infection through seven days after the second dose. Among these participants, 18,198 received the vaccine and 18,325 received placebo. The vaccine was 95% effective in preventing COVID-19 disease among these clinical trial participants ...

This gives the key point of (relative) effectiveness: it's 95% effective! Hooray, we've got it! See what happens when you keep reading:

... with eight COVID-19 cases in the vaccine group and 162 in the placebo group. Of these 170 COVID-19 cases, one in the vaccine group and three in the placebo group were classified as severe. At this time, data are not available to make a determination about how long the vaccine will provide protection, nor is there evidence that the vaccine prevents transmission of SARS-CoV-2 from person to person. 

First, let's look at the chance of getting covid without getting vaccinated; it's 162/18,325 = 1 in 113. Fewer than 1% of the placebo group got covid! And of those 162 cases, just 3 were classified as severe, so just 1 in over 6100 unvaxed people got severe covid. The numbers to achieve the benefit of vaccination aren't much different. The NNT is over 1 in 110 -- over 110 people had to take the vaccine for one person to avoid getting covid! Yes, the relative benefit is huge, but in absolute terms, less than 1% of people are actually helped by getting jabbed.

Note also that there was zero evidence that the vaccine prevents an infected person spreading the infection.

Is this the case only for Pfizer? A group of French scientists calculated ARR and NTT for the leading Covid drugs, based solely on the published studies of the trials of those drugs. Here is a summary and here is the study published in a scientific journal. It deserves much more attention than it seems to have gotten because of its focus on NNT.

Let's jump right to the key table.

The first drug, Pfizer, has a terrific efficacy (RR), listed there as 0.05, but normally reported as 95%. Everyone (including me, when I first saw it), thinks that means that taking the Pfizer vax means there's only a 5% chance of getting covid, right? It works great! Now look at the NNT, 141. That means that for each 141 people who are vaxed, just one benefits by not getting covid!! It makes common sense: there were 21,728 people in the control group (people who got shots that were placebos), and only 162 of them got covid,

You might think that relative and absolute risk are related, but the third drug, AstraZeneca, makes clear that they're not. AstraZeneca had efficacy (RR) of 0.30, normally reported as 70%, which is dramatically worse than Pfizer's -- why would anyone choose it? But AstraZeneca has an NNT of 83, which means that your chances of the AstraZeneca vax helping prevent covid were much better than the Pfizer vax. But even with the better NNT, chances are extremely high that you wouldn't get covid, with or without the vax.

The issues I describe here are not radical or new. The paper above was notable only in that it covered all the major covid vaccines; other doctors and scientists have publicly pointed out the same facts. For example, here is a note by a doctor published in the BMJ shortly after the trial results were first published.

Conclusion

After learning about efficacy, absolute risk and NNT, your understanding of what it means for a treatment to be "effective" changes radically. Absolute risk and NNT are at least as important. Authorities should discuss all these number prominently.

 

Data Humor Book by Rupa Mahanti

There's a new book out about Data Humor.

If you like data, you will be amused by this book. If you feel tortured by data, join the crowd -- and read this book, it will relieve some of the stress. If you were wondering what nerd humor was all about, read this book -- better to learn about nerd humor by getting the giggles.

The author searched far and wide for data humor. She stumbled upon a blog that had some bits she thought were funny -- this blog, yes, the one you're reading now!

She contacted me to ask permission to quote me. After thinking hard about whether I should grant permission -- for about a microsecond -- I gave it. She asked me to check out a draft of the book. I guess she liked what I said because my quote went on the back cover and was the first of the quotes on Amazon.

"This is a brilliant book. The title says it's humorous. It's hilarious! But even more valuable is the sad-but-true insights it conveys about humans, lost and wandering in uncharted forests of data, anxious to escape."—David B. Black, Technology Partner, Oak HC/FT Partners

It's quite amazing how widely she searched for quotes, from places I never would have thought to look:

... book containing a collection of more than 400 funny and quirky quotes, puns, and punchlines related to data, big data, statistics, and data science, from different sources and a wide array of cultural figures, thought leaders and key influencers across the world- William Edwards Deming, Charles Wheelan, Brené Brown, David B. Black, Tim O’Reilly, Jill Dyché, Evan Levy, Gwen Thomas, George Mount, David Shenk, James Gleick, Jim Barksdale, Vincent Granville, Cathy O’Neill, Dale Carnegie, Martyn Richard Jones, Timo Elliott, Mark Twain, Phil Simon, Lewis Carroll, Oscar Wilde, Thomas H. Davenport, DJ Patil, Damian Mingle, Thomas C. Redman, Cassie Korykorv, Brent Dykes, Guy Bradshaw, Scott Taylor, Susan Walsh, Winston Churchill, Ronald Reagan, Arthur Conan Doyle, and many more.

Here's just one:

Data isn't information, any more than fifty tons of cement is a skyscraper.
Clifford Stoll (Stoll 1996)

Don't you need some light in your life? I promise, it's lighter in every way than fifty tons of concrete...

The Hierarchy of Software Skills and Status in Data Science

There is a striking hierarchy of skills in software, as I've explained here. When you dive into any particular aspect of software, you usually find that it's got a hierarchy all its own. Data science is a subject of intense interest these days, so in this post I'll explain some of the basics of the data science skills hierarchy.

A skills hierarchy is very much an insider's game. What most people care about is status. I talk about the basics of software status here. Remember, the skills hierarchy is a whole world away from the status hierarchy that most people care about. Don't confuse the two!

Data Science skills

First and foremost, it's important to understand the incredibly broad range of subjects covered  by the term "data science." I attempt to explain the basics of the range in this blog post. You can be just amazing in one of those subjects, while being a neophyte in one that the outside world may consider to be "related," but which in practice is not just down the hall, but in a different building on a different campus.The general understanding of this range is SO pathetic that "data science" is typically managed as a completely independent, free-standing group. Which makes about as much sense as believing that a sous-chef belongs in something that isn't a kitchen. Or that everything that everyone does in a kitchen is basically the same thing.

Here's one cut at the hierarchy in data science, starting from the base:

1. Tool users.These are people who have learned how to use some software tool, or maybe a couple. Most "data scientists" fall into this category.
   • They don’t understand how the tools are built or any of the underlying software
   • They may know their tool, but aren't real clear on what the other tools are about, much less when you might consider using one.
2. Some have broader knowledge of tools
3. Very few have the sophistication to understand real data analysis, per my series of AI/ML; see this and the links in it: http://www.blackliszt.com/2018/04/getting-results-from-ml-and-ai-3-closed-loop.html
4. Of those, even fewer can understand the underlying algorithms and follow the latest research literature
5. Of those, even fewer can make real algorithmic advances and implement them as tools and deliver practical value.
6. Of anyone who can do all of the above, it is rare to meet anyone who can address and solve deep tool-level problems in other domains needed to make their code practical in the real world.
7. Finally, it is extremely rare to meet people who, in addition to their deep and broad prowess in data science and relevant skills needed to make it real-world practical, have similarly deep skills in an associated domain needed to make the data science fully effective for a business.

The issues of this hierarchy are compounded by the usual over-selling by people who are good promoters but little else, and corporate/government big-wigs who don't want to be bothered with details, but are keen to be seen as "doing something" on such a high-visibility topic. Getting results that make a difference is pretty low on the typical priority list.

And then of course there are the "data scientists" themselves, who most often are sincere people who are trying to do a good job as they've been taught to do it -- mostly by professors and others who have no idea what real-world success looks like, much less how to bring it about.

Finally, there is the usual "manage something that's invisible to you" phenomenon I have often discussed in this blog, which leads to so much dysfunction and so many wonderful Dilbert cartoons.

Conclusion

People talk as though "data science" were a thing, with the usual kind of hierarchy based on level of management and/or "experience." Those typical patterns of hierarchy just don't cut it for understanding what's going on in data science, just like they don't cut it for understanding software development. We will continue to see waste and dead-end efforts until we at least make a start at making our understanding of data science more sophisticated, and aligned with the facts on the ground.