Archive for the ‘prose’Category

Stesheni kumi na moja

I’m a bit late to the “social science bloggers love Station Elevenparty. Chris Blattman put it in his 2014 favorite novels list, and Jay Ulfelder shared a nice excerpt. I loved it too, so I’ll try to add something new.

Station Eleven is a novel about what happens after – and just before, and during – a flu pandemic wipes out 99% of the human population. The survivors refer to that event as the Collapse, and mostly avoid talking about or thinking about the immediate aftermath when all was a fight for survival. But Station Eleven is not just derivative post-apocalyptica. The book avoids a garish focus on the period just after the Collapse, but instead focuses on the more relatable period just as things are beginning to unravel, and much later, as bands of survivors who made it through the roughest bits are starting to rebuild. The main characters are a band of musicians and thespians who are trying to retain some of the cultural heritage and pass it on to the next generation, who have no memory of the world before the Collapse.

It’s also a novel about loss, both personal and societal. One of my favorite passages:

…No more ball games played out under floodlights. No more porch lights with moths fluttering on summer nights. No more trains running under the surface of cities on the dazzling power of the electric third rail. No more cities.… No more Internet. No more social media, no more scrolling through the litanies of dreams and nervous hopes and photographs of lunches, cries for help and expressions of contentment and relationship-status updates with heart icons whole or broken, plans to meet up later, pleas, complaints, desires, pictures of babies dressed as bears or peppers for Halloween. No more reading and commenting on the lives of others, and in so doing, feeling slightly less alone in the room. No more avatars.

Since I was reading this novel while traveling for work in Tanzania and Zimbabwe and Liberia, I was struck by its focus on Canada and the US. Nothing wrong with this: the author is Canadian* and the presumed audience is probably North American. But I kept wondering what the Collapse would have been like elsewhere. It was global, but would it have been equally catastrophic elsewhere? Urban centers like Manhattan are ludicrously unworkable in the absence of the electricity and cars and subways and other bits of the massive, distributed, and – to casual eyes – largely invisible infrastructure working to constantly feed them with supplies and people and information.

The novel implies that these urban centers fared worse, and focuses on suburbia and rural areas, where survivors re-learn how to farm, how to make things for themselves. We see nothing of the global “periphery” where the fall from wealth might be less great, where the collective psychological trauma of losing 99 out of 100 people might dominate the loss of technology. Of course, the periphery is defined by the observer and the writer, and isn’t the periphery at all to those who live in it. Maybe things would fare better, or maybe not.

Imagine the same novel, but set in Tanzania, or some other country where the majority of people are small-holder subsistence farmers. Maybe it would use the device of following two relatives, one living ‘upcountry’ or in ‘the village’ (i.e., poor rural parts) and the other living in Dar es Salaam. Relationships are established in an early chapter when the successful urban relative visits the village, or the rural relative visits the big city, and both marvel at their differences.

Then the flu hits, and things start to break down. Narrative chapters are intersperse with transcripts of SMS (text message) exchanges, demands for mPesa transfers, the realization that money doesn’t matter anymore, and finally the realization that the networks aren’t getting anything through anymore. Some city dwellers flee for the countryside but find themselves shunned as bearers of contagion. The urban protagonist makes her way, over the course of months or years, to the rural area where her relative once lived, hoping to find things are better there. Her belief that the village will be the same mirrors the readers’ belief – and common trope in writing about developing countries – that subsistence farmers today somehow live just as they did centuries or millenia ago. Bullshit, of course.

As the urbanite nears the village, her encounters reveal all the ways the modern fabric of village life was related to society and technology and has likewise broken down with the Collapse. Perhaps the power vacuum set off struggles amongst survivors and led to some new social order, where none of her skills are that useful. Nearing the village, she finds that the rural relative is now leader, revealing his situation has been reversed by the Collapse just as the once successful urbanite finds her way into his village with her last shilling.

Maybe this novel already exists. Or something else using the post-apocalyptic form to explore somewhere that’s not Canada or the US or Europe and not reliant on mechanized agriculture. Pointers, please, as I’d love to read it.

*originally I wrote the author was American. Oops. Apologies, Canada!

10

04 2015

Data: big, small, and meta

When I read this New York Times piece back in August, I was in the midst of preparation and training for data collection at rural health facilities in Zambia. The Times piece profiles a group called Global Pulse that is doing good work on the ‘big data’ side of global health:

The efforts by Global Pulse and a growing collection of scientists at universities, companies and nonprofit groups have been given the label “Big Data for development.” It is a field of great opportunity and challenge. The goal, the scientists involved agree, is to bring real-time monitoring and prediction to development and aid programs. Projects and policies, they say, can move faster, adapt to changing circumstances and be more effective, helping to lift more communities out of poverty and even save lives.

Since I was gearing up for ‘field work’ (more on that here; I’ll get to it soon), I was struck at the time by the very different challenges one faces at the other end of the spectrum. Call it small data? And I connected the Global Pulse profile with this, by Wayan Vota, from just a few days before:

The Sneakernet Reality of Big Data in Africa

When I hear people talking about “big data” in the developing world, I always picture the school administrator I met in Tanzania and the reality of sneakernet data transmissions processes.

The school level administrator has more data than he knows what to do with. Years and years of student grades recorded in notebooks – the hand-written on paper kind of notebooks. Each teacher records her student attendance and grades in one notebook, which the principal then records in his notebook. At the local district level, each principal’s notebook is recorded into a master dataset for that area, which is then aggregated at the regional, state, and national level in even more hand-written journals… Finally, it reaches the Minister of Education as a printed-out computer-generated report, complied by ministerial staff from those journals that finally make it to the ministry, and are not destroyed by water, rot, insects, or just plain misplacement or loss. Note that no where along the way is this data digitized and even at the ministerial level, the data isn’t necessarily deeply analyzed or shared widely….

And to be realistic, until countries invest in this basic, unsexy, and often ignored level of infrastructure, we’ll never have “big data” nor Open Data in Tanzania or anywhere else. (Read the rest here.)

Right on. And sure enough two weeks later I found myself elbow-deep in data that looked like this — “Sneakernet” in action:

In many countries a quite a lot of data — of varying quality — exists, but it’s often formatted like the above. Optimistically, it may get used for local decisions, and eventually for high-level policy decisions when it’s months or years out of date. There’s a lot of hard, good work being done to improve these systems (more often by residents of low-income countries, sometimes by foreigners), but still far too little. This data is certainly primary, in the sense that was collected on individuals, or by facilities, or about communities, but there are huge problems with quality, and with the sneakernet by which it gets back to policymakers, researchers, and (sometimes) citizens.

For the sake of quick reference, I keep a folder on my computer that has — for each of the countries I work in — most of the major recent ultimate sources of nationally-representative health data. All too often the only high-quality ultimate source is the most recent Demographic and Health Survey, surely one of the greatest public goods provided by the US government’s aid agency. (I think I’m paraphrasing Angus Deaton here, but can’t recall the source.) When I spent a summer doing epidemiology research with the New York City Department of Health and Mental Hygiene, I was struck by just how many rich data sources there were to draw on, at least compared to low-income countries. Very often there just isn’t much primary data on which to build.

On the other end of the spectrum is what you might call the metadata of global health. When I think about the work the folks I know in global health — classmates, professors, acquaintances, and occasionally thought not often me — do day to day, much of it is generating metadata. This is research or analysis derived from the primary data, and thus relying on its quality. It’s usually smart, almost always well-intentioned, and often well-packaged, but this towering edifice of effort is erected over a foundation of primary data; the metadata sometimes gives the appearance of being primary, when you dig down the sources often point back to those one or three ultimate data sources.

That’s not to say that generating this metadata is bad: for instance, modeling impacts of policy decisions given the best available data is still the best way to sift through competing health policy priorities if you want to have the greatest impact. Or a more cynical take: the technocratic nature of global health decision-making requires that we either have this data or, in its absence, impute it. But regardless of the value of certain targeted bits of the metadata, there’s the question of the overall balance of investment in primary vs. secondary-to-meta data, and my view — somewhat ironically derived entirely from anecdotes — is that we should be investing a lot more in the former.

One way to frame this trade-off is to ask, when considering a research project or academic institute or whatnot, whether the money spent on that project might result in more value for money if it was spent instead training data collectors and statistics offices, or supporting primary data collection (e.g., funding household surveys) in low-income countries. I think in many cases the answer will be clear, perhaps to everyone except those directly generating the metadata.

That does not mean that none of this metadata is worthwhile. On the contrary, some of it is absolutely essential. But a lot isn’t, and there are opportunity costs to any investment, a choice between investing in data collection and statistics systems in low-income countries, vs. research projects where most of the money will ultimately stay in high-income countries, and the causal pathway to impact is much less direct.  

Looping back to the original link, one way to think of the ‘big data’ efforts like Global Pulse is that they’re not metadata at all, but an attempt to find new sources of primary data. Because there are so few good sources of data that get funded, or that filter through the sneakernet, the hope is that mobile phone usage and search terms and whatnot can be mined to give us entirely new primary data, on which to build new pyramids of metadata, and with which to make policy decisions, skipping the sneakernet altogether. That would be pretty cool if it works out.

A more useful aid debate

Ken Opalo highlights recent entries on the great aid debate from Bill Gates, Jeff Sachs, Bill Easterly, and Chris Blattman.

Much has been said on this debate, and sometimes it feels like it’s hard to add anything new. But since having a monosyllabic first name seem sufficient qualification to weigh in, I will. First, this part of Ken’s post resonates with me:

I think most reasonable people would agree that Sachs kind of oversold his big push idea in The End of Poverty. Or may be this was just a result of his attempt to shock the donor world into reaching the 0.7 percent mark in contributions. In any event it is unfortunate that the debate on the relative efficacy of aid left the pages of journal articles in its current form. It would have been more helpful if the debate spilled into the public in a policy-relevant form, with questions like: under what conditions does aid make a difference? What can we do to increase the efficacy of aid? What kinds of aid should we continue and what kinds should we abolish all together? (emphasis added)

Lee Crawfurd wrote something along these lines too: “Does Policy Work?”  Lee wrote that on Jan 10, 2013, and I jokingly said it was the best aid blog post of the year (so far). Now that 2013 has wrapped up, I’ll extend that evaluation to ‘best aid blog post of 2013’. It’s worth sharing again:

The question “does policy work” is jarring, because we immediately realise that it makes little sense. Governments have about 20-30 different Ministries, which immediately implies at least 20-30 different areas of policy. Does which one work? We have health and education policy, infrastructure policy (roads, water, energy), trade policy, monetary policy, public financial management, employment policy, disaster response, financial sector policy, climate and environment policy, to name just a few. It makes very little sense to ask if they all collectively “work” or are “effective”. Foreign aid is similar. Aid supports all of these different areas of policy….

A common concern is about the impact of aid on growth… Some aid is specifically targeted at growth – such as financing infrastructure or private sector development. But much of it is not. One of the few papers which looks at the macroeconomic impact of aid and actually bothers to disaggregate even a little the different types of aid, finds that the aid that could be considered to have growth as a target, does increase growth. It’s the aid that was never intended to impact growth at all, such as humanitarian assistance, which doesn’t have any impact on growth.

I like to think that most smart folks working on these issues — and that includes both Sachs and Easterly — would agree with the following summaries of our collective state of knowledge:

  •  A lot of aid projects don’t work, and some of them do harm.
  • Some aid, especially certain types of health projects, works extremely well.

The disagreement is on the balance of good and bad, so I wish — as Ken wrote — the debate spilled into the public sphere along those lines (which is good? which is bad? how can we get a better mix?) rather than the blanket statements both sides are driven to by the very publicness of the debate. It reminds me a bit of debates in theology: if you put a fundamentalist and Einstein in the same room, they’ll both be talking about “God” but meaning very different things with the same words. (This is not a direct analogy, so don’t ask who is who…)

When Sachs and Easterly talk about whether aid “works”, it would be nice if we could get everyone to first agree on a definition of “aid” and “works”. But much of this seems to be driven by personal animosity between Easterly and Sachs, or more broadly, by personal animosity of a lot of aid experts vs. Sachs. Why’s that? I think part of the answer is that it’s hard to tell when Sachs is trying to be a scientist, and when he’s trying to be an advocate. He benefits from being perceived as the former, but in reality is much more the latter. Nina Munk’s The Idealist — an excellent profile of Sachs I’ve been meaning to review — explores this tension at some length. The more scientifically-minded get riled up by this confusion — rightfully, I think. At the same time, public health folks tend to love Sachs precisely because he’s been a powerful advocate for some types of health aid that demonstrably work — also rightfully, I think. There’s a tension there, and it’s hard to completely dismiss one side as wrong, because the world is complicated and there are many overlapping debates and conversations; academic and lay, public and private, science and advocacy.

So, back to Ken’s questions that would be answered by a more useful aid debate:

  • Under what conditions does aid make a difference?
  • What can we do to increase the efficacy of aid?
  • What kinds of aid should we continue and what kinds should we abolish all together?

Wouldn’t it be amazing if the public debate were focused on these questions? Actually, something like that was done: Boston Review had a forum a while back on “Making Aid Work” with responses by Abhijit Banerjee, Angus Deaton, Howard White, Ruth Levine, and others. I think that series of questions is much more informative than another un-moderated round of Sachs vs Easterly.

22

01 2014

Formalizing corruption: US medical system edition

Oh, corruption. It interferes with so many aspects of daily life, adding time to the simplest daily tasks, costing more money, and — often the most frustrating aspect — adding huge doses of uncertainty. That describes life in many low-income, high-corruption countries, leading to many a conversation with friends about comparisons with the United States and other wealthy countries. How did the US “solve” corruption?

I’ve heard (and personally made) the argument that the US reduced corruption at least in part by formalizing it; by channeling the root of corruption, a sort of rent-seeking on a personal level, to rent-seeking on an institutional level. The US political and economic system has evolved such that some share of any wealth created is channeled into the pockets of a political and economic elite who benefit from the system and in turn reinforce it. That unproductively-channeled share of wealth is simultaneously a) probably smaller than the share of wealth lost to corruption in most developing countries, b) still large enough to head off — along with the threat of more effective prosecution — at least some more overt corruption, and c) still a major drain on society.

An example: Elisabeth Rosenthal profiles medical tourism in an impressive series in the New York Times. In part three of the series, an American named Michael Shopenn travels to Belgium to get a hip replacement. Why would he need to? Because health economics in the US is less a story of free markets and  more a story of political capture by medical interests, including technology and pharmaceutical companies, physicians’ groups, and hospitals:

Generic or foreign-made joint implants have been kept out of the United States by trade policy, patents and an expensive Food and Drug Administration approval process that deters start-ups from entering the market. The “companies defend this turf ferociously,” said Dr. Peter M. Cram, a physician at the University of Iowa medical school who studies the costs of health care.

Though the five companies make similar models, each cultivates intense brand loyalty through financial ties to surgeons and the use of a different tool kit and operating system for the installation of its products; orthopedists typically stay with the system they learned on. The thousands of hospitals and clinics that purchase implants try to bargain for deep discounts from manufacturers, but they have limited leverage since each buys a relatively small quantity from any one company.

In addition, device makers typically require doctors’ groups and hospitals to sign nondisclosure agreements about prices, which means institutions do not know what their competitors are paying. This secrecy erodes bargaining power and has allowed a small industry of profit-taking middlemen to flourish: joint implant purchasing consultants, implant billing companies, joint brokers. There are as many as 13 layers of vendors between the physician and the patient for a hip replacement, according to Kate Willhite, a former executive director of the Manitowoc Surgery Center in Wisconsin.

If this system existed in another country we wouldn’t hesitate to call it corrupt, and to note that it actively hurts consumers. It should be broken up by legislation for the public good, but instead it’s protected by legislators who are lobbied by the industry and by doctors who receive kickbacks, implicit and explicit. Contrast that with the Belgian system:

His joint implant and surgery in Belgium were priced according to a different logic. Like many other countries, Belgium oversees major medical purchases, approving dozens of different types of implants from a selection of manufacturers, and determining the allowed wholesale price for each of them, for example. That price, which is published, currently averages about $3,000, depending on the model, and can be marked up by about $180 per implant. (The Belgian hospital paid about $4,000 for Mr. Shopenn’s high-end Zimmer implant at a time when American hospitals were paying an average of over $8,000 for the same model.)

“The manufacturers do not have the right to sell an implant at a higher rate,” said Philip Boussauw, director of human resources and administration at St. Rembert’s, the hospital where Mr. Shopenn had his surgery. Nonetheless, he said, there was “a lot of competition” among American joint manufacturers to work with Belgian hospitals. “I’m sure they are making money,” he added.

It’s become a cliche to compare the US medical system to European ones, but those comparisons are made because it’s hard to realize just how systematically corrupt — and expensive, as a result — the US system is without comparing it to ones that do a better job of channeling the natural profit-seeking goals of individuals and companies towards the public good. (For the history of how we got here, Paul Starr is a good place to start.)

The usual counterargument for protecting such large profit margins in the US is that they drive innovation, which is true but only to an extent. And for the implants industry that argument is much less compelling since many of the newer, “innovative” products have proved somewhere between no better and much worse in objective tests.

The Times piece is definitely worth a read. While I generally prefer the formalized corruption to the unformalized version, I’ll probably share this article with friends — in Nigeria, or Ethiopia, or wherever else the subject comes up next.

05

08 2013

Advocates and scientists

A new book by The Idealist: Jeffrey Sachs and the Quest to End Poverty. The blurbs on Amazon are fascinating because they indicate that either the reviewers didn’t actually read the book (which wouldn’t be all that surprising) or that Munk’s book paints a nuanced enough picture that readers can come away with very different views on what it actually proves. Here are two examples:

Amartya Sen: “Nina Munk’s book is an excellent – and moving – tribute to the vision and commitment of Jeffrey Sachs, as well as an enlightening account of how much can be achieved by reasoned determination.”

Robert Calderisi: “A powerful exposé of hubris run amok, drawing on touching accounts of real-life heroes fighting poverty on the front line.”

The publisher’s description seems to encompass both of those points of view: “The Idealist is the profound and moving story of what happens when the abstract theories of a brilliant, driven man meet the reality of human life.” That sounds like a good read to me — I look forward to reading when it comes out in September.

Munk’s previous reporting strikes a similar tone. For example, here’s an excerpt of her 2007 Vanity Fair profile of Sachs:

Leaving the region of Dertu, sitting in the back of an ancient Land Rover, I’m reminded of a meeting I had with Simon Bland, head of Britain’s Department for International Development in Kenya. Referring to the Millennium Villages Project, and to Sachs in particular, Bland laid it out for me in plain terms: “I want to say, ‘What concept are you trying to prove?’ Because I know that if you spend enough money on each person in a village you will change their lives. If you put in enough resources—enough foreigners, technical assistance, and money—lives change. We know that. I’ve been doing it for years. I’ve lived and worked on and managed [development] projects.

“The problem is,” he added, “when you walk away, what happens?”

Someone — I think it was Chris Blattman, but I can’t find the specific post — wondered a while back whether too much attention has been given to the Millennium Villages Project. After all, the line of thinking goes, the MVP’s have really just gotten more press and aren’t that different from the many other projects with even less rigorous evaluation designs. That’s certainly true: when journalists and aid bloggers debate the MVPs, part of what they’re debating is Sachs himself because he’s such a polarizing personality. If you really care about aid policy, and the uses of evidence in that policy, then that can all feel like an unhelpful distraction. Most aid efforts don’t get book-length profiles, and the interest in Sachs’ personality and persona will probably drive the interest in Munk’s book.

But I also think the MVP debates have been healthy and interesting — and ultimately deserving of most of the heat generated — because they’re about a central tension within aid and development, as well as other fields where research intersects with activism. If you think we already generally know what to do, then it makes sense to push forward with it at all costs. The naysayers who doubt you are unhelpful skeptics who are on some level ethically culpable for blocking good work. If you think the evidence is not yet in, then it makes more sense to function more like a scientist, collecting the evidence needed to make good decisions in the longer term. The naysayers opposing the scientists are then utopian advocates who throw millions at unproven projects. I’ve seen a similar tension within the field of public health, between those who see themselves primarily as advocates and those who see themselves as scientists, and I’m sure it exists elsewhere as well.

That is, of course, a caricature — few people fall completely on one side of the advocates vs. scientists divide. But I think the caricature is a useful one for framing arguments. The fundamental disagreement is usually not about whether evidence should be used to inform efforts to end poverty or improve health or advance any other goal. Instead, the disagreement is often over what the current state of knowledge is. And on that note, if you harbor any doubts on where Sachs has positioned himself on that spectrum here’s the beginning of Munk’s 2007 profile:

In the respected opinion of Jeffrey David Sachs…. the problem of extreme poverty can be solved. In fact, the problem can be solved “easily.” “We have enough on the planet to make sure, easily, that people aren’t dying of their poverty. That’s the basic truth,” he tells me firmly, without a doubt.

…To Sachs, the end of poverty justifies the means. By hook or by crook, relentlessly, he has done more than anyone else to move the issue of global poverty into the mainstream—to force the developed world to consider his utopian thesis: with enough focus, enough determination, and, especially, enough money, extreme poverty can finally be eradicated.

Once, when I asked what kept him going at this frenzied pace, he snapped back, “If you haven’t noticed, people are dying. It’s an emergency.”

—-

via Gabriel Demombynes.

If you’re new to the Millennium Villages debate, here’s some background reading: a recent piece in Foreign Policy by Paul Starobin, and some good posts by Chris Blattman (one, two, three), this gem from Owen Barder, and Michael Clemens.

The greatest country in the world

I’ve been in Ethiopia for six and a half months, and in that time span I have twice found myself explaining the United States’ gun culture, lack of reasonable gun control laws, and gun-related political sensitivities to my colleagues and friends in the wake of a horrific mass shooting.

When bad things happen in the US — especially if they’re related to some of our national moral failings that grate on me the most, e.g. guns, health care, and militarism — I feel a sense of personal moral culpability, much stronger when I’m living in the US. I think having to explain how terrible and terribly preventable things could happen in my society, while living somewhere else, makes me feel this way. (This is by no means because people make me feel this way; folks often go out of their way to reassure me that they don’t see me as synonymous with such things.)

I think that this enhanced feeling of responsibility is actually a good thing. Why? If being abroad sometimes puts the absurdity of situations at home into starker relief, maybe it will reinforce a drive to change. All Americans should feel some level of culpability for mass shootings, because we have collectively allowed a political system driven by gun fanatics,  a media culture unintentionally but consistently glorifying mass murderers, and a horribly deficient mental health system to persist, when their persistence has such appalling consequences.

After the Colorado movie theater shooting I told colleagues here that nothing much would happen, and sadly I was right. This time I said that maybe — just maybe — the combination of the timing (immediately post-election) and the fact that the victims were schoolchildren will result in somewhat tighter gun laws. But, attention spans are short so action would need to be taken soon. Hopefully the fact that the WhiteHouse.gov petition on gun control already has 138,000 signatures (making it the most popular petition in the history of the website) indicates that something could well be driven through. Even if that’s the case, anything that could be passed now will be just the start and it will be long hard slog to see systematic changes.

As Andrew Gelman notes here, we are all part of the problem to some extent: “It’s a bit sobering, when lamenting problems with the media, to realize that we are the media too.” He’s talking about bloggers, but I think it extends further: every one of us that talks about gun control in the wake of a mass shooting but quickly lets it slip down our conversational and political priorities once the event fades from memory is part of the problem. I’m making a note to myself to write further about gun control and the epidemiology of violence in the future — not just today — because I think that entrenched problems require a conscious choice to break the cycle. In the meantime, Harvard School of Public Health provides some good places to start.

17

12 2012

On deworming

GiveWell’s Alexander Berger just posted a more in-depth blog review of the (hugely impactful) Miguel and Kremer deworming study. Here’s some background: the Cochrane reviewGivewell’s first response to it, and IPA’s very critical response.

I’ve been meaning to blog on this since the new Cochrane review came out, but haven’t had time to do the subject justice by really digging into all the papers. So I hope you’ll forgive me for just sharing the comment I left at the latest GiveWell post, as it’s basically what I was going to blog anyway:

Thanks for this interesting review — I especially appreciate that the authors [Miguel and Kremer] shared the material necessary for you [GiveWell] to examine their results in more depth, and that you talk through your thought process.

However, one thing you highlighted in your post on the new Cochrane review that isn’t mentioned here, and which I thought was much more important than the doubts about this Miguel and Kremer study, was that there have been so many other studies that did not find large effect on health outcomes! I’ve been meaning to write a long blog post about this when I really have time to dig into the references, but since I’m mid-thesis I’ll disclaim that this quick comment is based on recollection of the Cochrane review and your and IPA’s previous blog posts, so forgive me if I misremember something.

The Miguel and Kremer study gets a lot of attention in part because it had big effects, and in part because it measured outcomes that many (most?) other deworming studies hadn’t measured — but it’s not as if we believe these outcomes to be completely unrelated. This is a case where what we believe the underlying causal mechanism for the social effects to be is hugely important. For the epidemiologists reading, imagine this as a DAG (a directed acyclic graph) where the mechanism is “deworming -> better health -> better school attendance and cognitive function -> long-term social/economic outcomes.” That’s at least how I assume the mechanism is hypothesized.

So while the other studies don’t measure the social outcomes, it’s harder for me to imagine how deworming could have a very large effect on school and social/economic outcomes without first having an effect on (some) health outcomes — since the social outcomes are ‘downstream’ from the health ones. Maybe different people are assuming that something else is going on — that the health and social outcomes are somehow independent, or that you just can’t measure the health outcomes as easily as the social ones, which seems backwards to me. (To me this was the missing gap in the IPA blog response to GiveWell’s criticism as well.)

So continuing to give so much attention to this study, even if it’s critical, misses what I took to be the biggest takeaway from that review — there have been a bunch of studies that showed only small effects or none at all. They were looking at health outcomes, yes, but those aren’t unrelated to the long-term development, social, and economic effects. You [GiveWell] try to get at the external validity of this study by looking for different size effects in areas with different prevalence, which is good but limited. Ultimately, if you consider all of the studies that looked at various outcomes, I think the most plausible explanation for how you could get huge (social) effects in the Miguel Kremer study while seeing little to no (health) effects in the others is not that the other studies just didn’t measure the social effects, but that the Miguel Kremer study’s external validity is questionable because of its unique study population.

(Emphasis added throughout)

 

Someone should study this: Addis housing edition

Attention development economists and any other researchers who have an interest in urban or housing policy in low-income countries:

My office in Addis has about 25 folks working in it, and we have a daily lunch pool where we pay in 400 birr a month (about 22 USD) to cover costs and all get to eat Ethiopian food for lunch every day. It’s been a great way to get to know my coworkers — my work is often more solitary: editing, writing, and analyzing data — and an even better way to learn about a whole variety of issues in Ethiopia.

addis construction

Addis construction site (though not probably not government condos)

The conversation is typically in Amharic and mine is quite limited, so I’m lucky if I can figure out the topic being discussed.  [I usually know if they’re talking about work because so many NGO-speak words aren’t translated, for example: “amharic amharic amharic Health Systems Strengthening amharic amharic…“] But folks will of course translate things as needed.  One observation is that certain topics affect their daily lives a lot, and thus come up over and over again at lunch.

One subject that has come up repeatedly is housing. Middle class folks in Addis Ababa feel the housing shortage very acutely. Based on our conversations it seems the major limitation is in getting credit to buy or build a house.

The biggest source of good housing so far has been government-constructed condominiums, for which you pay a certain (I’m not sure how much) percentage down and then make payments over the years. (The government will soon launch a new “40/60 scheme” to which many folks are looking forward, in which anyone who can make a 40% down payment on a house will get a government mortgage for the remaining 60%.)

When my coworkers first mentioned that the government will offer the next round of condominiums by a public lottery, my thought was “that will solve someone’s identification problem!” A large number of people — many thousands — have registered for the government lottery. I believe you have to meet a certain wealth or income threshold (i.e., be able to make the down payment), but after that condo eligibility will be determined randomly. I think that — especially if someone organizes the study prior to the lottery — this could yield very useful results on the impact of urban housing policy.

How (and how much) do individuals and families benefit from access to better housing? Are there changes in earnings, savings, investments? Health outcomes? Children’s health and educational outcomes? How does it affect political attitudes or other life choices? It could also be an opportunity to study migration between different neighborhoods, amongst many other things.

A Google Scholar search for Ethiopia housing lottery turns up several mentions, but (in my very quick read) no evaluations taking advantage of the randomization. (I can’t access this recent article in an engineering journal, but from the abstract assume that it’s talking about a different kind of evaluation.) So, someone have at it? It’s just not that often that large public policy schemes are randomized.

06

12 2012

"As it had to fail"

My favorite line from the Anti-Politics Machine is a throwaway. The author, James Ferguson, an anthropologist, describes a World Bank agricultural development program in Lesotho, and also — through that lens — ends up describing development programs more generally. At one point he notes that the program failed “as it had to fail” — not really due to bad intentions, or to lack of technical expertise, or lack of funds — but because failure was written into the program from the beginning. Depressing? Yes, but valuable.

I read in part because Chris Blattman keeps plugging it, and then shortly before leaving for Ethiopia I saw that a friend had a copy I could borrow. Somehow it didn’t make it onto reading lists for any of my classes for either of my degrees, though it should be required for pretty much anyone wanting to work in another culture (or, for that matter, trying to foment change in your own). Here’s Blattman’s description:

People’s main assets [in Lesotho] — cattle — were dying in downturns for lack of a market to sell them on. Households on hard times couldn’t turn their cattle into cash for school fees and food. Unfortunately, the cure turned out to be worse than the disease.

It turns out that cattle were attractive investments precisely because they were hard to liquidate. With most men working away from home in South Africa, buying cattle was the best way to keep the family saving rather than spending. They were a means for men to wield power over their families from afar.

Ferguson’s point was that development organizations attempt to be apolitical at their own risk. What’s more, he argued that they are structured to remain ignorant of the historical, political and cultural context in which they operate.

And here’s a brief note from Foreign Affairs:

 The book comes to two main conclusions. First is that the distinctive discourse and conceptual apparatus of development experts, although good for keeping development agencies in business, screen out and ignore most of the political and historical facts that actually explain Third World poverty-since these realities suggest that little can be accomplished by apolitical “development” interventions. Second, although enormous schemes like Thaba-Tseka generally fail to achieve their planned goals, they do have the major unplanned effect of strengthening and expanding the power of politically self-serving state bureaucracies. Particularly good is the discussion of the “bovine mystique,” in which the author contrasts development experts’ misinterpretation of “traditional” attitudes toward uneconomic livestock with the complex calculus of gender, cash and power in the rural Lesotho family.

The reality was that Lesotho was not really an idyllically-rural-but-poor agricultural economy, but rather a labor reserve more or less set up by and controlled by apartheid South Africa. The gulf between the actual political situation and the situation as envisioned by the World Bank — where the main problems were lack of markets and technical solutions — at the time was enormous. This lets Ferguson have a lot of fun showing the absurdities of Bank reports from the era, and once you realize what’s going on it’s quite frustrating to read how the programs turned out, and to wonder how no one saw it coming.

This contrast between rhetoric and reality is the book’s greatest strength: because the situation is absurd, it illustrates Ferguson’s points very well, that aid is inherently political, and that projects that ignore that reality have their future failure baked in from the start. But that contrast is a weakness too, as because the situation is extreme you’re left wondering just how representative the case of Lesotho really was (or is). The 1970s-80s era World Bank certainly makes a great buffoon (if not quite a villain) in the story, and one wonders if things aren’t at least a bit better today.

Either way, this is one of the best books on development I’ve read, as I find myself mentally referring to it on a regular basis. Is the rhetoric I’m reading (or writing) really how it is? Is that technical, apolitical sounding intervention really going to work? It’s made me think more critically about the role outside groups — even seemingly benevolent, apolitical ones — have on local politics. On the other hand, the Anti-Politics Machine does read a bit like it was adapted from an anthropology dissertation (it was); I wish it could get a new edition with more editing to make it more presentable. And a less ugly cover. But that’s no excuse — if you want to work in development or international health or any related field, it should be high on your reading list.

Why we should lie about the weather (and maybe more)

Nate Silver (who else?) has written a great piece on weather prediction — “The Weatherman is Not a Moron” (NYT) — that covers both the proliferation of data in weather forecasting, and why the quantity of data alone isn’t enough. What intrigued me though was a section at the end about how to communicate the inevitable uncertainty in forecasts:

…Unfortunately, this cautious message can be undercut by private-sector forecasters. Catering to the demands of viewers can mean intentionally running the risk of making forecasts less accurate. For many years, the Weather Channel avoided forecasting an exact 50 percent chance of rain, which might seem wishy-washy to consumers. Instead, it rounded up to 60 or down to 40. In what may be the worst-kept secret in the business, numerous commercial weather forecasts are also biased toward forecasting more precipitation than will actually occur. (In the business, this is known as the wet bias.) For years, when the Weather Channel said there was a 20 percent chance of rain, it actually rained only about 5 percent of the time.

People don’t mind when a forecaster predicts rain and it turns out to be a nice day. But if it rains when it isn’t supposed to, they curse the weatherman for ruining their picnic. “If the forecast was objective, if it has zero bias in precipitation,” Bruce Rose, a former vice president for the Weather Channel, said, “we’d probably be in trouble.”

My thought when reading this was that there are actually two different reasons why you might want to systematically adjust reported percentages ((ie, fib a bit) when trying to communicate the likelihood of bad weather.

But first, an aside on what public health folks typically talk about when they talk about communicating uncertainty: I’ve heard a lot (in classes, in blogs, and in Bad Science, for example) about reporting absolute risks rather than relative risks, and about avoiding other ways of communicating risks that generally mislead. What people don’t usually discuss is whether the point estimates themselves should ever be adjusted; rather, we concentrate on how to best communicate whatever the actual values are.

Now, back to weather. The first reason you might want to adjust the reported probability of rain is that people are rain averse: they care more strongly about getting rained on when it wasn’t predicted than vice versa. It may be perfectly reasonable for people to feel this way, and so why not cater to their desires? This is the reason described in the excerpt from Silver’s article above.

Another way to describe this bias is that most people would prefer to minimize Type II Error (false negatives) at the expense of having more Type I error (false positives), at least when it comes to rain. Obviously you could take this too far — reporting rain every single day would completely eliminate Type II error, but it would also make forecasts worthless. Likewise, with big events like hurricanes the costs of Type I errors (wholesale evacuations, cancelled conventions, etc) become much greater, so this adjustment would be more problematic as the cost of false positives increases. But generally speaking, the so-called “wet bias” of adjusting all rain prediction probabilities upwards might be a good way to increase the general satisfaction of a rain-averse general public.

The second reason one might want to adjust the reported probability of rain — or some other event — is that people are generally bad at understanding probabilities. Luckily though, people tend to be bad about estimating probabilities in surprisingly systematic ways! Kahneman’s excellent (if too long) book Thinking, Fast and Slow covers this at length. The best summary of these biases that I could find through a quick Google search was from Lee Merkhofer Consulting:

 Studies show that people make systematic errors when estimating how likely uncertain events are. As shown in [the graph below], likely outcomes (above 40%) are typically estimated to be less probable than they really are. And, outcomes that are quite unlikely are typically estimated to be more probable than they are. Furthermore, people often behave as if extremely unlikely, but still possible outcomes have no chance whatsoever of occurring.

The graph from that link is a helpful if somewhat stylized visualization of the same biases:

In other words, people think that likely events (in the 30-99% range) are less likely to occur than they are in reality, that unlike events (in the 1-30% range) are more likely to occur than they are in reality, and extremely unlikely events (very close to 0%) won’t happen at all.

My recollection is that these biases can be a bit different depending on whether the predicted event is bad (getting hit by lightning) or good (winning the lottery), and that the familiarity of the event also plays a role. Regardless, with something like weather, where most events are within the realm of lived experience and most of the probabilities lie within a reasonable range, the average bias could probably be measured pretty reliably.

So what do we do with this knowledge? Think about it this way: we want to increase the accuracy of communication, but there are two different points in the communications process where you can measure accuracy. You can care about how accurately the information is communicated from the source, or how well the information is received. If you care about the latter, and you know that people have systematic and thus predictable biases in perceiving the probability that something will happen, why not adjust the numbers you communicate so that the message — as received by the audience — is accurate?

Now, some made up numbers: Let’s say the real chance of rain is 60%, as predicted by the best computer models. You might adjust that up to 70% if that’s the reported risk that makes people perceive a 60% objective probability (again, see the graph above). You might then adjust that percentage up to 80% to account for rain aversion/wet bias.

Here I think it’s important to distinguish between technical and popular communication channels: if you’re sharing raw data about the weather or talking to a group of meteorologists or epidemiologists then you might take one approach, whereas another approach makes sense for communicating with a lay public. For folks who just tune in to the evening news to get tomorrow’s weather forecast, you want the message they receive to be as close to reality as possible. If you insist on reporting the ‘real’ numbers, you actually draw your audience further from understanding reality than if you fudged them a bit.

The major and obvious downside to this approach is that people know this is happening, it won’t work, or they’ll be mad that you lied — even though you were only lying to better communicate the truth! One possible way of getting around this is to describe the numbers as something other than percentages; using some made-up index that sounds enough like it to convince the layperson, while also being open to detailed examination by those who are interested.

For instance, we all the heat index and wind chill aren’t the same as temperature, but rather represent just how hot or cold the weather actually feels. Likewise, we could report some like “Rain Risk” or “Rain Risk Index” that accounts for known biases in risk perception and rain aversion. The weather man would report a Rain Risk of 80%, while the actual probability of rain is just 60%. This would give us more useful information for the recipients, while also maintaining technical honesty and some level of transparency.

I care a lot more about health than about the weather, but I think predicting rain is a useful device for talking about the same issues of probability perception in health for several reasons. First off, the probabilities in rain forecasting are much more within the realm of human experience than the rare probabilities that come up so often in epidemiology. Secondly, the ethical stakes feel a bit lower when writing about lying about the weather rather than, say, suggesting physicians should systematically mislead their patients, even if the crucial and ultimate aim of the adjustment is to better inform them.

I’m not saying we should walk back all the progress we’ve made in terms of letting patients and physicians make decisions together, rather than the latter withholding information and paternalistically making decisions for patients based on the physician’s preferences rather than the patient’s. (That would be silly in part because physicians share their patients’ biases.) The idea here is to come up with better measures of uncertainty — call it adjusted risk or risk indexes or weighted probabilities or whatever — that help us bypass humans’ systematic flaws in understanding uncertainty.

In short: maybe we should lie to better tell the truth. But be honest about it.