All Excess Mortality Estimates are Wrong but Some are Useful

This is a paraphrase of a quote by one of the most famous statisticians of the 20^th century, George Box:

“All models are wrong but some are useful”.

Statisticians can discuss the meaning of this quote for hours on end. Some even argue about it.

It has renewed meaning in the modern era of machine learning where we get the computer to construct a model based on data fed into an algorithm.

Before we start, if you read my previous article, you would recall how I felt sick watching a recent Australian Senate Committee, with the pathetic answers provided to Senators by the TGA, Pfizer and Moderna. I see Dr John Campbell was similarly made sick:

Observe the pained expression on his face.

He warns viewers beforehand about how painful it is to watch. I see the term “Pfizer Bot” has been used on social media for the rote answers provided by these zombies. The guy on the right continually refused to answer properly. He may have been putting the questions into ChatGPT to provide responses to the Senator.

There has been a lot of criticism, mainly from overseas, of the responses provided to the Australian Senators. However, there is no coverage in Australia’s lazy legacy media, who are still scratching their heads about the formerly fringe lab leak theory.

On the other hand, I saw criticism of the interview with Senator Rennick on John Campbell’s youtube, regarding Australia’s excess mortality, by the lead actuary of the Actuaries Institute’s COVID mortality working group.

BTW there is commentary (see jikkyleaks on twitter) regarding the Actuaries Institute work on COVID excess mortality. The “independent” lead actuary, from what I read, allegedly now works for the government. I can’t confirm this.

The Senators have constituents come to them personally to tell them of experience and concerns. Their job is to listen and take the people’s concerns to parliament and act on behalf of the people. In Australia we only have about half a dozen elected politicians who are expressing these concerns on behalf of the population.

I’d suggest the Pfizer bots, or people angry with the Senator for asking questions, do not interact with ordinary working-class people.

An aside:

We have just had a Royal Commission into a scandal called Robodebt where the government completely screwed up a program which tried to claw back social service payments. It led to much harm, including deaths of despair, in a vulnerable sector of community. The government executive who led it was on a salary of $900,000. She was going on to a senior role in the proposed nuclear submarine project. This didn’t sit well with the public and she has since been stood down.

The community has every right to be skeptical of government mandated programs.

Excess Mortality Estimates

Rebekah Barnett recently reported on updated excess mortality estimates from the ABS.

JUST IN: Excess deaths Australia 2020-2023

The data of the ABS report went up to the first quarter of 2023. I have just gone through this report in detail. I have mentioned previously that I have found the ABS reporting to be of good quality. Their original excess mortality estimates did a simple averaging of previous years to come up with a baseline. They were clear about this limitation. This doesn’t take into account population changes, trends, or age distribution. However, it still gives a “useful” estimate of the excess. There are questions about which years should be used to make up a baseline.

The updated model basically fits a sine wave pattern to death counts. The method is called cyclic regression. It also takes into account trend. If population is getting larger we expect more deaths over time and in a simple model that can be a component of trend. A good thing about the way they have generated the model is to acknowledge that seasons such as the bad 2017 influenza season should not be considered normal. So, in the fitting process they use it takes this into account and weights extreme values less.

The ABS note that they only take into account a linear trend because they found using extra terms (ie second order and above) made the baseline too low in 2021, 2022. We will look into whether this assertion is valid. This is the main graph from the report:

Observe the 2017 flu season peak. The magnitude is of comparable magnitude, above the baseline, as the Winter 2022 COVID wave.

I implemented the same method and I basically get the same result. I didn’t do any of the fine tuning as discussed in the ABS paper.

Visually you can see it is similar.

Senator Malcolm Roberts questioned the Moderna representatives on excess mortality. He asked how they explained 30,000 excess deaths per year? This number may be an overestimate, but it is in the ballpark. I think this number came from early ABS estimates. I found the short clip of the interaction here from his telegram channel. You would also be able to find it in the full recording I linked to in my last article.

The Moderna rep says he reviewed the Q3 2023 ABS report referenced above. He claimed there were either less deaths, or they were within range, for 2020 and 2021. He also says that during 2022 the excess deaths coincided with COVID waves, implying all excess deaths were due to COVID deaths.

At least this guy has done some homework (unlike the Pfizer bot) even though the answer is bullsh*t.

It is true that deaths look low in 2020. There are various possible reasons:

• lockdowns keeping flu or COVID virus out. This is questioned by some here, here and here and substack summary article by Martin Neil.

In Australia, see webpage here, for Queensland. Mysteriously 2021 is skipped in reporting. Look at the 2020 results from report found here. No flu in 2020.

Oh remember, that’s right no one went to the doctors which was where samples for test were collected. So we wouldn’t have known if there was any flu.

• There could be a deficit of deaths from previous years of excess (2019 was a bad flu season),

• and I wonder if hospitals being closed for elective surgery had an effect?

The escape clause used by the Moderna rep for 2021, when deaths start going above trend, is that while deaths were larger than baseline in 2021 they were within the calculated 95% confidence intervals.

Wait, what?

Those confidence intervals only apply to the week-by-week deaths. They mean nothing with respect to a sum of all deaths. When you think about it a week is an arbitrary measure of time. If the mortality tracked along the top confidence interval over the whole year (see the ABS graph above), the increase in number of deaths would be huge! It would be about 250 deaths per week extra. That would be catastrophic, so the inference that the deaths stay within the blue confidence interval lines is OK is rubbish.

There is actually some commentary on this in the ABS report. They note that once excess is greater than limits for 2 or more weeks then there is cause for concern. So this part of the answer from Moderna is crap.

Yes, there were peaks in death in the January 2022 and Winter 2022 waves caused by COVID. But there is still an overall underlying excess, even if all COVID deaths are subtracted. I am going to go into this in the discussion on COVID deaths below.

The company reps also went to lengths to say that, while their injections didn’t stop infection or transmission, they stopped severe disease and deaths. But they didn’t, did they? The majority of the deaths were elderly people, who in Australia are a population with almost 100% injection coverage. With some older age groups in Australia vaccinated at a rate greater that 100%! That’s right - if you go to the aggregator sites and look at age breakdown you’ll see more first doses than people. This was covered up by only showing >95% so it didn’t look stupid.

Now COVID deaths are only a small portion of the total number of deaths apart from at the two large COVID peaks. Weekly deaths for Australia are around 3,500 per week, so 14,000 per month. The largest number of COVID deaths in a month was 1,600 for January 2022 in the big Omicron wave which started after we opened up to vaccinated travelers. The next largest month is 1,400 for the Winter wave in July 2022.

OK now let’s compare the excess deaths, according to the ABS data, to the number COVID deaths.

Here’s the cumulative excess computed over the whole period. It should track above and below the zero line. See it go up in 2017 due to the bad flu season. It starts tracking down in mid 2020. In the second Quarter of 2021 it starts an upwards trajectory that shows no sign of leveling off.

Let’s just look at 2021. From the ABS graph above mortality is nicely following the prediction at the start of the year, our summer. There is assumed no excess. I also show the cumulative number of COVID deaths. I also plot the COVID deaths accumulating per month. They only really started taking off by October 2021.

Oh-Oh. The trend is up, up and up. And it is not accounted for by COVID deaths.

I won’t spend more time here because we know the limitations of simply using the total number of people in the country. It doesn’t take into account the age distribution of the population. If over the years there are more old people in the population (ie due to living longer) and less young people (due to less births) we expect this to affect the number deaths. It will sort of come out in the trend. It’s very confusing because it’s a complex system of many components. I find it hard myself to have any intuition over this. You have to look at raw numbers in age categories. There could also be underlying patterns for different age groups that get averaged out. For example, one age group having many more deaths than normal and another less.

This statement in the ABS report caught my attention:

Deaths are significantly lower than expected from the week beginning 1 June to mid-July 2020, dropping below lower thresholds. Winter months are typically associated with higher mortality. These decreases provide insights into how public health measures put in place to manage the COVID-19 pandemic impacted mortality.

The “public health measures” at this time were lockdowns. The implication is that this “measure” improved health in some way. The fact that straight after this is probably the worst trajectory of excess mortality in our history seems to contradict this.

How many examples can we find where a measure is put in place that improves things and then goes on to lead to a disaster? Australians know of this one:

Cane toads were introduced to Australia in 1935 as a biological control method against the Greyback cane beetle that was destroying sugar cane crops. The Cane toad is native to South and Central America and had been used successfully as a biological control agent against beetles in Hawaii. This method of pest management went horribly wrong in Australia.

The ABS Model

The ABS method was originally used to detect increases in influenza. See the Methodology section of the ABS paper which is towards the end. This model served a “useful” purpose.

Is the seasonal trend really a perfect sine wave shape? The answer is obviously, no. Other models can come up with a more complex shape. From this thread by the Actuaries Institute:

“The main reason for the difference in the March quarter comes down to choice of the model shape. The ABS has used a model that has a pre-defined, lovely, smooth shape, while we have chosen something a bit uglier that we think more closely reflects the actual shape of the data.”

This graph was provided in the tweet for comparison:

“We think more closely reflects the actual shape of the data”.

Really? Do we really think there are little bumps that always happen each year at the same time?

Of course, we have no idea what the true underlying pattern is. The number of deaths are what they are. We are averaging small sets of numbers of a distribution that is ever changing. All we can do is make an estimate. This is another way of saying our best guess.

What does Excess Mortality mean? It’s a number above which might be considered normal. All we can do is make an estimate of what we think is normal. Alternatively, it could be what we would like to be normal or what we consider acceptable.

And in doing this we are probably going to be wrong.

Trends

However, while we might never know the true number for the excess we do want to know when things are changing, particularly if mortality is getting worse and especially if it changes suddenly.

Through doing extensive analysis in the area, I have found intuition difficult to find. It is really complicated. One has to dig into the details, there is no way around it. Looking at the top-level aggregated data of all deaths can hide what is going on underneath.

Ecological Fallacy

There is a fallacy called the Ecological Fallacy. It’s a bit like Simpson’s Paradox. You can’t tell what is really going on from aggregated data.

The ‘ecological fallacy’ consists of ascribing to individuals the characteristics of groups to which they belong, even though the relevant individuals may not share such characteristics.

This is a subject for another article.

Digging Deeper

Let’s look at the Week-to-Week differences in number of deaths. Even though the pattern goes up and down the difference week to week should be within some limit.

There are two clear outliers (circled). The largest is the week ending 16 January 2022. This is deaths from COVID Omicron wave in Australia after the country opened up to vaccinated travelers late in 2021 and the majority of the population were vaccinated. COVID took off.

The other outlier is of concern. It is for the week ending 18 April 2021. There was a jump of 302 deaths from the previous week. The standard deviation of the series is 93. So, this outlier is over 3 standard deviations away from the mean.

We know there were practically no COVID deaths in Australia at this time early 2021. The vaccination rollout in Australia started at the end of February 2021. Now refer back to the excess graph I calculated from the ABS data. Keep this date in mind.

COVID Deaths

The ABS and Actuaries Institute come up with estimates of the total excess mortality:

They are fairly close. Then an inference is made that most of this excess is due to the abnormal COVID virus. However here is a problem. We unfortunately have to expect that many of the COVID deaths are deaths that would have occurred in any case. People who have died of COVID have a median age greater than the median age of all deaths. They have multiple comorbidities. The most recent ABS COVID Mortality Report shows:

Annual virus lead to some of the deaths in the pattern of rise and fall of the mortality graph. COVID is worse than usual. Just like the flu was in 2017. But some of the COVID deaths are part of the natural mortality.

But there is an aspect of this we will never know. Did a particular individual die sooner than they might have otherwise? Only God knows.

Another model

I better wrap this article up. But I’ll provide another model. I mentioned before an aspect of the ABS model where they assumed a linear trend. I won’t show all the workings but let’s look at the cumulative deaths from the start of the ABS data series in 2013.

On this scale it looks like a straight line. I made the line red from 1 March 2021. A linear trend line (dotted) is shown fitted to data pre-March 2021. We can subtract the trend line. The advantage of the cumulative sum from the beginning of the data series is that we can pick up the general trend over the period of 10 years. It turned out that a second order fit was better. We subtract the trend:

That’s all I’ve got to say about that.

Summary

All excess mortality estimates are wrong but some are useful.