October 12, 2006

Lancet study observations

I was poleaxed as much as anybody by the recent Lancet study on Iraq deaths, but it's really harder to argue with its conclusions than it first seems.

The actual statement of claim is pretty simple when you reduce it to the nub. The Iraqi survey team claims to have randomly interviewed households totalling 12,801 individuals, and found that exactly 300 of them had died violently between the U.S invasion and July, 2006. Survivors produced approximately 240 death certificates that confirmed this.

That math (even if you use only the certified deaths) works out to 5.5 violent deaths per thousand people per annum in Iraq. Extrapolate that to the entire population of Iraq and you get a number in the 450,000 range. Assume the other 60 undocumented violent deaths were truthful reports, as well, and you're up to 600K.

The methodology as defined in the study is as sound as any other scientific study (more on this later). The simplest statistical sample size calculations tell us that if the real number of violent deaths so far in Iraq had been, say, 60,000, then there should have been around 30 certifiable violent deaths in a sample of this size, not 240 (or 300).

Converting to pollster speak from science speak is not a perfect science, but if were to put these results in pollster terms, the equivalent statement for a random polling sample of this size is that the results are going to be accurate to within 1.1%, 99 times out of 100. If you concede that the middle of the range of the claim derived from a random sample this size (again, using just the certified deaths in the study) is that 1.8% of the sample population was certified as dying violently in those first 40 months of occupation, that means there is no longer any reasonable possibility that the total violent deaths nationwide certified by death certificate could be any less that 0.7% of the population, and is probably much higher. (NOTE: the actual study's confidence intervals are actually much tighter, with a lower bound in the vicinity of 400,000: I'm just doing the most basic statistical sample size calculation here, as well as confining myself to the certified deaths).

0.7% of Iraq's population is over 200,000 dead. There is no scientifically acceptable explanation short of massive deceit for the true number of violent deaths to be any lower than that anymore.

Is that lower bound so implausible? Remembering that it includes indigenous insurgent and Iraqi security forces deaths may help. Realistic estimates of fatalities (insurgent plus civilian) at the two battles of Fallujah in 2004 were in the 20,000 range at the high end: is 10 or more Fallujahs possible? Probably.

One striking thing about the deaths is that they are almost all (78%) adult males (aged 15-59), with correspondingly few women or children. This is also consistent with our idea of the kind of fighting that has been going on in Iraq.

I don't think you can seriously counter with Iraqi government denials, or their own attempts at record-keeping, which are obviously going to be deficient on a national scale, especially so recently. Reliable figures on the issuing of death certificates by local authorities are difficult to come by in a timely fashion even in Western countries.

If one had had to do a thumbnail estimate before the Lancet study came out, it probably would have gone like this. The highly reputable Carl Conetta study on total Iraqi fatalities, combat and non-combat in April-May 2003 had an upper limit of 15,000 dead, so start there. Before they were ordered by the government to stop publishing figures, the Iraq health ministry was documenting 400 civilian deaths from coalition forces alone per month, or another 15,000 in this three-year period. The Brookings Institute said in the middle of last year that it estimated 27,000 insurgents had been killed or imprisoned in the first two years of the war: it is unlikely that the total number of non-foreign insurgents killed over the last three years is much less than 40,000. Finally, Iraq's own security forces and police have certainly incurred somewhere close to 10,000 fatal casualties to date (see Donald Rumsfeld's frequent remark that they have been incurring fatalities at "twice the rate" of coalition forces).

So, even without counting all the other kinds of violent death (Shia-on-Sunni, terrorist attack, etc.), there was no way that we were looking at less than 80,000 fatal casualties in Iraq in this period. Add, say, another entirely plausible 40,000 per year for the ongoing Shia-Sunni strife, terrorism, extrajudicial killings, murders, car accidents, and the like (Baghdad morgue admissions alone have never been much less than 1,000 per month, and have recently ramped up quite sharply), and you're already up to the 200,000 fatality number I've been talking about, and beginning to approach the 400,000 lower limit of the actual Lancet study.

At the very most, I would have said the Lancet study's result could only have ever been reasonably off by a factor of two or three, which isn't bad for epidemiological work. Still a shocking conclusion, though, and I'll admit some difficulty still in reconciling it with my own previous thumbnail assumptions, above. But the idea that it's off by anything like an order of magnitude is nothing more than untutored denial.

That kind of denial, however, is pervasive, this week. For what we're seeing here, in the blogs, in the statements of the American president yesterday, and in the newspaper coverage (case in point: this morning's editorial in the Globe and Mail*) is an increasing freedom on the part of rightist critics not to challenge scientific work on its merits, but instead launch an outright attack on basic statistical assumptions that can only be described as an attack on scientific objectivity. Rather than discussing the study on its merits, or lack thereof, the largest portion of the response so far has been that the study has produced a politically relevant result, so it cannot be the truth. (See, as an example, prominent John Hinderaker, effortlessly dismissing one of the most prestigious journals in medical science as "bozos.")

This should be extremely disturbing to people. To assert that all science is political, that basic mathematical methodologies like statistical sample size are discreditable if they are displeasing, the media and politicians are now truly leading us to a place where nothing can ever be known, other than what the self-appointed leaders tell you is your truth.

*The Globe editorial (subscription required) referred to the death total as being the total dead Iraqi "non-combatants". The actual Lancet study makes no such claim about combatant status. It would have been nice if Canada's leading newspaper could have read the study before pronouncing upon it.

Posted by BruceR at 12:53 PM