More on the numbers game

If there’s anyone left who still thinks the numbers in the Lancet study are valid, he should read Steven Moore’s op-ed explaining why the study’s methodology is invalid by any standard:

After doing survey research in Iraq for nearly two years, I was surprised to read that a study by a group from Johns Hopkins University claims that 655,000 Iraqis have died as a result of the war. Don’t get me wrong, there have been far too many deaths in Iraq by anyone’s measure; some of them have been friends of mine. But the Johns Hopkins tally is wildly at odds with any numbers I have seen in that country. Survey results frequently have a margin of error of plus or minus 3% or 5%–not 1200%.

The group–associated with the Johns Hopkins Bloomberg School of Public Health–employed cluster sampling for in-person interviews, which is the methodology that I and most researchers use in developing countries. Here, in the U.S., opinion surveys often use telephone polls, selecting individuals at random. But for a country lacking in telephone penetration, door-to-door interviews are required: Neighborhoods are selected at random, and then individuals are selected at random in “clusters” within each neighborhood for door-to-door interviews. Without cluster sampling, the expense and time associated with travel would make in-person interviewing virtually impossible.

However, the key to the validity of cluster sampling is to use enough cluster points. In their 2006 report, “Mortality after the 2003 invasion of Iraq: a cross-sectional sample survey,” the Johns Hopkins team says it used 47 cluster points for their sample of 1,849 interviews. This is astonishing: I wouldn’t survey a junior high school, no less an entire country, using only 47 cluster points.

Read the rest here.

I’ll add a couple of other reasons why I doubt the study. The study is based, not on provable facts, but on interviews. The first problem with this interview approach is that we have no way of knowing whether the people interviewed had a particular axe to grind, and were therefore purposefully giving false information.

The second problem lies with the validity of oral interviews in a Middle Eastern culture. The Western concept of relaying facts that are as accurate as possible is not a Middle Eastern concept. This is not to say Middle Easterners lie, which would mean they intentionally tell a falsehood. (In any event, I mentioned the risk of out-and-out lying in the preceding paragraph.) Rather, Middle Easterners have a cultural bias towards telling the listener what they think he wants to hear. An example would be the situation in which someone asks an Arab “Is your father an old man?” Assuming that the questioner is intrigued by age, the response would be “Yes, he’s very, very old.” Had the interlocutor come and said “Wow, your father looks so young,” the response would have been “He’s very, very young.” Should birth certificates exist, the father’s would undoubtedly show him to be an average age of 45.

In other words, even though they’re not deliberately lying, Arab respondents may still not recite factual truth as we understand that concept. Culturally, they’re more into factual accommodation. I would therefore expect that, if a surveyor showed up lusting after information about the dead and dying, that’s precisely the type of information that would be showered on him. It is, after all, giving him what he wants.