Vaccine Effectiveness-time to recalibrate?

A look at COVID-19 infection data from the UK…

Recently it was reported in a substack article:

Vaccine Failure - Across the Board.

based on UK data, that vaccine efficacy had become negative for all age groups above 18. Similar trends have been seen in other countries, leading to research and discussion on vaccine effectiveness in the context of the Omicron variant.

I decided to check this out myself to understand what is going on. The UK has amongst the best reporting of data during the pandemic, albeit with limitations reported by researchers (see Discrepancies and inconsistencies in UK Government datasets compromise accuracy of mortality rate comparisons between vaccinated and unvaccinated from Professor Norman Fenton’s group at Queen Mary University of London, UK.

In Australia, where I am from, this data, on rate of infections against vaccination status, is not reported.

We should be careful about use of the term “efficacy”, as that term is used in the context of a formal trial. Effectiveness is a more appropriate term to use in this context. A positive effectiveness means that the intervention, in this case vaccination, improves the desired outcome, ie rate of infection decreases. To be generic we can use the term Relative Risk percentage increase (or decrease).

The Relative Risk or Risk Ratio (RR) is a way to compare risks for two groups. For vaccines it is assumed that the vaccine improves things. RR is the relative risk of developing the disease for vaccinated people compared to unvaccinated people. We divide the incidence (eg rate per 100,000) of disease in the vaccinated group by the incidence in the unvaccinated group. The effectiveness as a percentage is calculated as (1- RR) x 100%. It’s the difference in rate between no intervention (unvaccinated) and intervention (vaccinated) divided by the default rate (unvaccinated).

100% effective means that no one who has the intervention gets infected. As it decreases the ratio of the rate of those vaccinated to those unvaccinated who are getting infected increases. Zero% effective means that there is basically no beneficial effect. If the ratio turns around it means that the intervention is performing worse than doing nothing. -100% means that the intervention makes things twice as bad as doing nothing. It’s a little bit confusing when described as a percentage, compared with thinking about numbers of cases in a population. One way the percentage limits at 100 and the other way it can go below 100.

What was the promised Effectiveness? A report on the study, in the medical journal Nature, concludes that “Pfizer is around 92% effective at stopping people from developing a high viral load 14 days after the second dose. Nature reports the study as showing the vaccine’s efficacy declined to 90% after 30 days, 85% after 60 days and 78% after 90 days. AstraZeneca’s efficacy began at 69% a fortnight after the second dose, falling to 61% after 90 days”.

What does 90% effective mean? If we have two groups, with 100 people vaccinated and 100 people unvaccinated, and 1 vaccinated person gets infected (rate 0.01) and 10 unvaccinated people get infected (rate 0.1) then Vaccine Effectiveness (VE) is (1 – 0.01/0.1) x 100% = 90%. It means 10 times more unvaccinated are infected than vaccinated.

The reason this measure is important is because governments and companies have used Vaccine Passports to limit access of unvaccinated persons to various services, in the interest of protecting society. The premise was that vaccinated persons have much lower risk of being infected and therefore transmitting the virus. The Nature study “shows that vaccinated people who become infected with the Delta variant carry high peak levels of virus”. So being vaccinated and infected is presumably just as much risk to society as being unvaccinated and infected.

I downloaded the UK report from Week 1, 2021 and indeed the ratios reported in the first link in this post are correct for over 18s (but there appears to be an error in the under 18 group).

There are numerous disclaimers to the tables providing the raw data:

Comparing case rates among vaccinated and unvaccinated populations should not be used to estimate vaccine effectiveness against COVID-19 infections. Vaccine effectiveness has been formally estimated from a number of different sources ... going on to reference various trials. I note that it would be nice if, once in a while, trial results were not always significantly better than what is found in real life. In other words that the product being offered performed better than specified. In my experience it rarely happens, especially in medical trials.

It is clear that the desired narrative is that the intervention improves everything. There are various other disclaimers to these results. They include:

• People who are fully vaccinated may be more health conscious and therefore more likely to get tested.

• People who are fully vaccinated and people who are unvaccinated may behave differently, particularly with regard to social interactions and therefore may have differing levels of exposure to COVID-19.

• People who have never been vaccinated are more likely to have caught COVID-19 in the weeks or months before the period of the cases covered in the report. This gives them some natural immunity to the virus for a few months and may have contributed to a lower case rate in the past few weeks.

These caveats appear to be desperate. A full review of these caveats deserves another post. These are all worrying to me as a Data Scientist, as there is clearly a desire to reduce the confidence in this particular data, presumably because it does not point in the desired direction.

My understanding is that in the UK the restrictions on the unvaccinated, apart from travel overseas, were not severe. Unlike in Australia where unvaccinated were restricted access to all but non-essential services like food. Vaccinated mandates meant people lost jobs across all sectors.

One has to try to be dispassionate about the way the numbers go. Whether or not there are biases in the data I have always found that there can still be useful information to glean.

So I decided to investigate what was happening leading up to Week 1 in 2022?

I downloaded reports from Week 39 in 2021 onwards. Each of the reports actually cover multiple weeks. Week 39 is stated as data between Week 35 and 38. That’s roughly the start of September till 26 Sep 2021. Any errors in graphs I provide here may be due to my manually entering data from tables provided.

I plot the Vaccine Effectiveness (or Risk Ratio % Increase) from Week 39 in 2021 till Week 2 2022. I plot from ages 18 up. Taking a slice at Week 1 you can see the Negative Effectiveness occur for all ages. Below the dashed line means infection rate is higher in vaccinated compared to unvaccinated, ie Negative Effectiveness.

There are a lot of features in this graph to consider, in particular the significant fluctuation over time.

It is understood that the roll-out of the third booster dose started in September 2021 in the UK for the 80+ age group. This was from actual Week 38, and therefore first reported at Week 41 in the reports (ie for weeks 38-40). Boosters were then progressively rolled out for other age groups. The 80+ age group third dose coverage reached 80% for that age group by Week 46 (ie first reported at Week 49).

It appears that the introduction of the booster improves effectiveness, but this is temporary, falling again after the majority of the population are boosted and the onset of Omicron. I note that in this graph I include Week 3 data where the report only provided infections for vaccinated with 3 dose coverage (more on this later).

Similar trends are seen for other age groups.

Even if all the disclaimers in the UK reports are valid, such that the rate of infection for unvaccinated should be much higher, these temporal variations in the data warrant further consideration. From mid-December we can see the effect of the Omicron variant leading to a significant change. There is a rapid drop in effectiveness across all age groups.

It is instructive to look at the underlying absolute rates of infection for the unvaccinated and vaccinated.

The unvaccinated rate per 100,000 is on the left and the vaccinated rate on the right. Rates for some age groups on the right have gone above the y axis limit set for these graphs. The unvaccinated rate trends steadily. The Omicron onset can be seen at the end of 2021 with rapid increase from Week 51, remembering that the data for a week, shown on x axis, is based on the previous 3 weeks. On the other hand, the vaccinated rate (noting this is for those with at least 2 doses) goes up and down before the onset of Omicron. Presumably this is due to waning effects of the vaccination and improvements after boosters. For some age groups the rate of infection doesn’t get back to where it was previously.

As I went through the reports, and trends became apparent, it is clear that there editorial influence was applied as public criticism occurred. At Week 42, when Vaccine Effectiveness appeared to be negative for all age groups above 30, there was apparently outrage expressed by an eminent statistician that the UK Health Security Agency had “put out absurd statistics showing case rates higher in vaxxed than non-vaxxed … feeding conspiracy theorists worldwide” (see link to discrepancies article above). It is worth noting that this actually occurred a few weeks earlier (Week 40) and it probably took a few weeks for the trend to become obvious.

It’s interesting to go through the reports and see where more and more disclaimers are provided when numbers do not follow the accepted narrative. At Week 43 (the week after outrage was expressed) graphs that had been included in the report highlighting case rates of vaccinated versus unvaccinated for that week were removed.

At Week 47 the format of the infections table was changed and the unvaccinated rate numbers are greyed out as if something must be wrong with them (ie they are too low). A week later additional disclaimers (some shown earlier in this article) are added to the table as to why the numbers must be wrong.

In Week 3 2022 (not shown on two of the graphs here) there is a change in the reporting, quietly introduced, where the vaccinated rate is changed from at least 2 doses to 3 doses only. This manages to drop the infection rate for some age groups. For older age groups where the effect of the third dose has possibly worn off there is minimal change (see the 80+ graph above). In a future post I’ll track what happens further and it may be possible to estimate numbers for 2 dose and 3 dose separately.

As a Data Scientist, trying to extract information out of raw data, it’s always concerning when you see this type of reporting, where an accepted narrative is operating. There is always something to learn even if there are biases and discrepancies in data.

Further, when decisions are made based on reports, as a basis to restrict freedom in society, there needs to be strict ethical oversight.

In the end the ultimate aim is improved healthcare, is it not?