Archive for the ‘evidence-based’Category

Randomizing in the USA

The NYTimes posted this article about a randomized trial in New York City:

It has long been the standard practice in medical testing: Give drug treatment to one group while another, the control group, goes without.

Now, New York City is applying the same methodology to assess one of its programs to prevent homelessness. Half of the test subjects — people who are behind on rent and in danger of being evicted — are being denied assistance from the program for two years, with researchers tracking them to see if they end up homeless.

Dean Karlan at Innovations for Policy Action responds:

It always amazes me when people think resources are unlimited. Why is “scarce resource” such a hard concept to understand?

I think two of the most important points here are that a) there weren’t enough resources for everyone to get the services anyway, so they’re just changing the decision-making process for who gets the service from first-come-first-served (presumably) to randomized, and b) studies like this can be ethical when there is reasonable doubt about whether a program actually helps or not. If it were firmly established that the program is beneficial, then it’s unethical to test it, which is why you can’t keep testing a proven drug against placebo.

However, this is good food for thought for those who are interested in doing randomized trials of development initiatives in other countries. It shows the impact (and reactions) from individuals to being treated as “test subjects” here in the US — and why should we expect people in other countries to feel differently? That said, a lot of randomized trials don’t get this sort of pushback. I’m not familiar with this program beyond what I read in this article, but it’s possible that more could have been done to communicate the purpose of the trial to the community, activists, and the media.

There are some interesting questions raised in the IPA blog comments as well.

Results-Based Aid

Nancy Birsdall writes “On Not Being Cavalier About Results” about a recent critique of the UK’s DFID (Department for International Development):

The fear about an insistence on results arises from confusion about what “results” are. A legitimate typical concern is that aid bureaucracies pressed for “results” will resort, more than already is the case, to projects that provide inputs that seem add up to easily measured “wins” (bednets delivered, books distributed, paramedics trained, vehicles or computers purchased, roads built) while neglecting “system” issues and “institution building”. But bednets and books and vehicles and roads are not results in any meaningful sense, and the connection between these inputs and real outcomes (healthier babies, better educated children, higher farmer income) goes through systems and institutions and is often lost….

Let us define results as measured gains in what children have learned by the end of primary school, or measured reductions in infant mortality or deforestation, or measured increases in the hours of electricity available, or annual increases in revenue from taxes paid by rich households in poor countries – or a host of other indicators that ultimately add up to the transformation of societies and the end of their dependence on outside aid. For a country to get results might not require more money but a reconfiguration of local politics, the cleaning up of bureaucratic red tape, local leadership in setting priorities or simply more exposure to the force of local public opinion. Let aid be more closely tied to well-defined results that recipient countries are aiming for; let donors and recipients start measuring and reporting those results to their own citizens; let there be continuous evaluation and learning about the mechanics of how recipient countries and societies get those results (their institutional shifts, their system reforms, their shifting politics and priorities), built on the transparency that Secretary Mitchell is often emphasizing.

(Emphasis added)

I’d also like to note that Birdsall is the founding director of the Center for Global Development, a nonprofit in DC that does a lot of work related to evidence-based aid. I relied fairly heavily on their report on “Closing the Evaluation Gap” on a recent dual degree app. The full report is worth the read.


12 2010