On deworming

GiveWell’s Alexander Berger just posted a more in-depth blog review of the (hugely impactful) Miguel and Kremer deworming study. Here’s some background: the Cochrane reviewGivewell’s first response to it, and IPA’s very critical response.

I’ve been meaning to blog on this since the new Cochrane review came out, but haven’t had time to do the subject justice by really digging into all the papers. So I hope you’ll forgive me for just sharing the comment I left at the latest GiveWell post, as it’s basically what I was going to blog anyway:

Thanks for this interesting review — I especially appreciate that the authors [Miguel and Kremer] shared the material necessary for you [GiveWell] to examine their results in more depth, and that you talk through your thought process.

However, one thing you highlighted in your post on the new Cochrane review that isn’t mentioned here, and which I thought was much more important than the doubts about this Miguel and Kremer study, was that there have been so many other studies that did not find large effect on health outcomes! I’ve been meaning to write a long blog post about this when I really have time to dig into the references, but since I’m mid-thesis I’ll disclaim that this quick comment is based on recollection of the Cochrane review and your and IPA’s previous blog posts, so forgive me if I misremember something.

The Miguel and Kremer study gets a lot of attention in part because it had big effects, and in part because it measured outcomes that many (most?) other deworming studies hadn’t measured — but it’s not as if we believe these outcomes to be completely unrelated. This is a case where what we believe the underlying causal mechanism for the social effects to be is hugely important. For the epidemiologists reading, imagine this as a DAG (a directed acyclic graph) where the mechanism is “deworming -> better health -> better school attendance and cognitive function -> long-term social/economic outcomes.” That’s at least how I assume the mechanism is hypothesized.

So while the other studies don’t measure the social outcomes, it’s harder for me to imagine how deworming could have a very large effect on school and social/economic outcomes without first having an effect on (some) health outcomes — since the social outcomes are ‘downstream’ from the health ones. Maybe different people are assuming that something else is going on — that the health and social outcomes are somehow independent, or that you just can’t measure the health outcomes as easily as the social ones, which seems backwards to me. (To me this was the missing gap in the IPA blog response to GiveWell’s criticism as well.)

So continuing to give so much attention to this study, even if it’s critical, misses what I took to be the biggest takeaway from that review — there have been a bunch of studies that showed only small effects or none at all. They were looking at health outcomes, yes, but those aren’t unrelated to the long-term development, social, and economic effects. You [GiveWell] try to get at the external validity of this study by looking for different size effects in areas with different prevalence, which is good but limited. Ultimately, if you consider all of the studies that looked at various outcomes, I think the most plausible explanation for how you could get huge (social) effects in the Miguel Kremer study while seeing little to no (health) effects in the others is not that the other studies just didn’t measure the social effects, but that the Miguel Kremer study’s external validity is questionable because of its unique study population.

(Emphasis added throughout)

 

December

07

2012

1 Comments Add Yours ↓

The upper is the most recent comment

  1. April Harding #
    1

    Brett,
    You are right. The Kremer/Miguel study has virtually no external validity – driven not only by the unique study population.

    I outlined the range of factors which make the results unreliable for drawing insights for any other setting (including for other districts in Kenya, or even the same districts at a different point in time) in a comment on the World Bank Development Impact blog here
    http://blogs.worldbank.org/impactevaluations/node/850

    Text follows:
    JPAL’s deworming “synthesis” seems more misleading

    The JPAL crew appears to be the pot calling the proverbial kettle “black” in saying the Cochrane review is misleading.

    JPAL has published in several places their considered take on school-based deworming – it’s a “best buy” for health AND school attendance.
    See here: http://www.povertyactionlab.org/publication/deworming-best-buy-development
    They base this conclusion on evaluations of one PROJECT – which is not a program, and as such has very limited external validity. That is, implementation of the intervention was largely done by an NGO and delivered through NGO- supported schools. This means the the conditions for implementing the intervention were mostly controlled – so the evaluations are in essence efficacy not effectiveness studies.
    Putting this aside, even as an efficacy trial, the findings are not generalizable because of several context factors which clearly enhance the effect of the intervention: unusually high school attendance, and, unusually high worm infection rate.
    Because of the long-standing NGO support to the schools, the Project schools had 97-98% attendance, the average for Kenya’s schools at the time was 62% (the figures come from the Edu for All database). The average in other countries with high worm infection is LOWER.
    And, as the Cochrane review rightly noted, the worm infection rate in the Kenyan study area was also unusually high (92%) during the period of the program. Even if you implemented the exact same intervention in the exact same place with the exact same NGO implementation arrangements in subsequent years you would expect lower impact.

    Given such low (infinitesimal?) external validity – the thoughtful policy analyst would want to examine more studies before drawing conclusions or making recommendations. Alas, JPAL’s publications don’t show such restraint. They are comfortable making recommendations on how to spend aid dollars based on these studies alone. Some might call this…misleading.

    For sure there are interpretations I might quibble with here and there in the Cochrane review. Givewell’s write up covered most of the quibbles that came to my mind.
    Honestly, I feel the JPAL team should be glad that a synthesis has been done. It is certainly a sounder foundation for drawing conclusions about deworming interventions than the Kenya studies alone. Let’s see if they will update their “best buy” aid spending recommendations.