Archive for September, 2012
My Hunger Games survival analysis post keeps getting great feedback. The latest anonymous comment:
Nice effort on the analysis, but the data is not suitable for KM and Cox. In KM, Cox and practically almost everything that requires statistical inference on a population, your variable of interest should be in no doubt independent from sample unit to sample unit.
Since your variable of interest is life span during the game where increasing ones chances in a longer life means deterring another persons lifespan (i.e. killing them), then obviously your variable of interest is dependent from sample unit to sample unit.
Your test for determining whether the gamemakers rig the selection of tributes is inappropriate, since the way of selecting tributes is by district. In the way your testing whether the selection was rigged, you are assuming that the tributes were taken as a lot regardless of how many are taken from a district. And the way you computed the expected frequency assumes that the number of 12 year olds equals the number of 13 year olds and so on when it is not certain.
Thanks for the blog. It was entertaining.
And there’s a lot more in the other comments.
A long time ago (in years, two more than the product of 10 and the length of a single American presidential term) John Siegfried wrote this First Lesson in Econometrics (PDF). It starts with this:
Every budding econometrician must learn early that it is never in good taste to express the sum of the two quantities in the form: “1 + 1 = 2”.
… and just goes downhill from there. Read it.
(I wish I remembered where I first saw this so I could give them credit.)
Nate Silver (who else?) has written a great piece on weather prediction — “The Weatherman is Not a Moron” (NYT) — that covers both the proliferation of data in weather forecasting, and why the quantity of data alone isn’t enough. What intrigued me though was a section at the end about how to communicate the inevitable uncertainty in forecasts:
…Unfortunately, this cautious message can be undercut by private-sector forecasters. Catering to the demands of viewers can mean intentionally running the risk of making forecasts less accurate. For many years, the Weather Channel avoided forecasting an exact 50 percent chance of rain, which might seem wishy-washy to consumers. Instead, it rounded up to 60 or down to 40. In what may be the worst-kept secret in the business, numerous commercial weather forecasts are also biased toward forecasting more precipitation than will actually occur. (In the business, this is known as the wet bias.) For years, when the Weather Channel said there was a 20 percent chance of rain, it actually rained only about 5 percent of the time.
People don’t mind when a forecaster predicts rain and it turns out to be a nice day. But if it rains when it isn’t supposed to, they curse the weatherman for ruining their picnic. “If the forecast was objective, if it has zero bias in precipitation,” Bruce Rose, a former vice president for the Weather Channel, said, “we’d probably be in trouble.”
My thought when reading this was that there are actually two different reasons why you might want to systematically adjust reported percentages ((ie, fib a bit) when trying to communicate the likelihood of bad weather.
But first, an aside on what public health folks typically talk about when they talk about communicating uncertainty: I’ve heard a lot (in classes, in blogs, and in Bad Science, for example) about reporting absolute risks rather than relative risks, and about avoiding other ways of communicating risks that generally mislead. What people don’t usually discuss is whether the point estimates themselves should ever be adjusted; rather, we concentrate on how to best communicate whatever the actual values are.
Now, back to weather. The first reason you might want to adjust the reported probability of rain is that people are rain averse: they care more strongly about getting rained on when it wasn’t predicted than vice versa. It may be perfectly reasonable for people to feel this way, and so why not cater to their desires? This is the reason described in the excerpt from Silver’s article above.
Another way to describe this bias is that most people would prefer to minimize Type II Error (false negatives) at the expense of having more Type I error (false positives), at least when it comes to rain. Obviously you could take this too far — reporting rain every single day would completely eliminate Type II error, but it would also make forecasts worthless. Likewise, with big events like hurricanes the costs of Type I errors (wholesale evacuations, cancelled conventions, etc) become much greater, so this adjustment would be more problematic as the cost of false positives increases. But generally speaking, the so-called “wet bias” of adjusting all rain prediction probabilities upwards might be a good way to increase the general satisfaction of a rain-averse general public.
The second reason one might want to adjust the reported probability of rain — or some other event — is that people are generally bad at understanding probabilities. Luckily though, people tend to be bad about estimating probabilities in surprisingly systematic ways! Kahneman’s excellent (if too long) book Thinking, Fast and Slow covers this at length. The best summary of these biases that I could find through a quick Google search was from Lee Merkhofer Consulting:
Studies show that people make systematic errors when estimating how likely uncertain events are. As shown in [the graph below], likely outcomes (above 40%) are typically estimated to be less probable than they really are. And, outcomes that are quite unlikely are typically estimated to be more probable than they are. Furthermore, people often behave as if extremely unlikely, but still possible outcomes have no chance whatsoever of occurring.
The graph from that link is a helpful if somewhat stylized visualization of the same biases:
In other words, people think that likely events (in the 30-99% range) are less likely to occur than they are in reality, that unlike events (in the 1-30% range) are more likely to occur than they are in reality, and extremely unlikely events (very close to 0%) won’t happen at all.
My recollection is that these biases can be a bit different depending on whether the predicted event is bad (getting hit by lightning) or good (winning the lottery), and that the familiarity of the event also plays a role. Regardless, with something like weather, where most events are within the realm of lived experience and most of the probabilities lie within a reasonable range, the average bias could probably be measured pretty reliably.
So what do we do with this knowledge? Think about it this way: we want to increase the accuracy of communication, but there are two different points in the communications process where you can measure accuracy. You can care about how accurately the information is communicated from the source, or how well the information is received. If you care about the latter, and you know that people have systematic and thus predictable biases in perceiving the probability that something will happen, why not adjust the numbers you communicate so that the message — as received by the audience — is accurate?
Now, some made up numbers: Let’s say the real chance of rain is 60%, as predicted by the best computer models. You might adjust that up to 70% if that’s the reported risk that makes people perceive a 60% objective probability (again, see the graph above). You might then adjust that percentage up to 80% to account for rain aversion/wet bias.
Here I think it’s important to distinguish between technical and popular communication channels: if you’re sharing raw data about the weather or talking to a group of meteorologists or epidemiologists then you might take one approach, whereas another approach makes sense for communicating with a lay public. For folks who just tune in to the evening news to get tomorrow’s weather forecast, you want the message they receive to be as close to reality as possible. If you insist on reporting the ‘real’ numbers, you actually draw your audience further from understanding reality than if you fudged them a bit.
The major and obvious downside to this approach is that people know this is happening, it won’t work, or they’ll be mad that you lied — even though you were only lying to better communicate the truth! One possible way of getting around this is to describe the numbers as something other than percentages; using some made-up index that sounds enough like it to convince the layperson, while also being open to detailed examination by those who are interested.
For instance, we all the heat index and wind chill aren’t the same as temperature, but rather represent just how hot or cold the weather actually feels. Likewise, we could report some like “Rain Risk” or “Rain Risk Index” that accounts for known biases in risk perception and rain aversion. The weather man would report a Rain Risk of 80%, while the actual probability of rain is just 60%. This would give us more useful information for the recipients, while also maintaining technical honesty and some level of transparency.
I care a lot more about health than about the weather, but I think predicting rain is a useful device for talking about the same issues of probability perception in health for several reasons. First off, the probabilities in rain forecasting are much more within the realm of human experience than the rare probabilities that come up so often in epidemiology. Secondly, the ethical stakes feel a bit lower when writing about lying about the weather rather than, say, suggesting physicians should systematically mislead their patients, even if the crucial and ultimate aim of the adjustment is to better inform them.
I’m not saying we should walk back all the progress we’ve made in terms of letting patients and physicians make decisions together, rather than the latter withholding information and paternalistically making decisions for patients based on the physician’s preferences rather than the patient’s. (That would be silly in part because physicians share their patients’ biases.) The idea here is to come up with better measures of uncertainty — call it adjusted risk or risk indexes or weighted probabilities or whatever — that help us bypass humans’ systematic flaws in understanding uncertainty.
In short: maybe we should lie to better tell the truth. But be honest about it.
These photos are of the construction site next to my office in Addis — the quality isn’t that great, but I still think they’re interesting. Some observations on this site:
- progress is slow
- manual labor is substituted for capital-intensive technology wherever possible
- the scaffolding is made by hand on site
- there’s absolutely no protective gear (no hard hats, no harnesses while hanging off the flimsy handmade scaffolding), and
- women are surprisingly well-represented (at least at this site).
Guernica: Was it important to you to stay in the vicinity of the community?
Katherine Boo: Quite the contrary. It was important to me, in the course of my reporting in Annawadi, day after day, night after night, to leave and get a sense of the city as a whole. It is a city that until eleven years ago was unknown to me, and is changing all of time, so I really had to explore it, learn about it. I certainly did a lot reporting around the five-star hotels as well as Annawadi. I did my whole anthropology of five-star bathrooms, each one more lavish than the next. (Laughs.)
Even if I were to stay in Annawadi or something like it, it wouldn’t be the same. After Hurricane Katrina, for instance, I did stay in the shelter [when] I did reporting for The New Yorker. But me staying in a shelter is not the same as someone who’s been evacuated to that shelter. This whole thing of, “I’m walking a mile in their shoes by living this certain way.” Well, I’m not living that way. I can turn around and leave. We can do the best we can to get to the core of people’s circumstances, but it’s ludicrous to think that my being in Annawadi all of that time is walking in their shoes. It’s not.
The quote in the title of this post is from a section on the feelings of guilt that haunt Boo when she thinks about how her work exploits people, especially poor people. The interview’s a great read. (Found via LongForm.org, a great source for creative nonfiction / narrative journalism.)
I read a bunch of articles for my public health coursework, but one that has stuck in my mind and thus been on my things-t0-blog-about list for some time is “Mystification of a simple solution: Oral rehydration therapy in Northeast Brazil” by Marilyn Nations and L.A. Rebhun. Unfortunately I can’t find an ungated PDF for it anywhere (aside: how absurd is it that it costs $36 USD to access one article that was published in 1988??) so you can only access it here for free if you have access through a university, etc.
The article describes the relatively simple diarrhea treatment, ORT (oral rehydration therapy), as well as how physicians in a rural community in Brazil managed to reclaim this simply procedure and turn it into a highly specialized medical procedure that only they could deliver. One thing I like is that the article has (for an academic piece) a great narrative: you learn how great ORT is … then about the ridiculous ritualization / mystification of a simple process that keeps it out of reach of those who need it most … then about the authors’ proposed solution … and finally a twist where you realize the solution has some (though not certainly not all) of the same problems. Here’s one of their case studies of how doctors mystified ORT:
Benedita, a 7-month-old girl with explosive, watery diarrhea of 5 days duration, weakness, and marked dehydration, was brought by her mother to a government hospital emergency room. White-garbed nurses and physicians examined the baby and began ORT. They meticulously labeled a sterilized baby bottle with millimeter measurements, weighed the child every lSmin, mixed the chemical packet with clean water, gave the predetermined amount of ORT, and recorded all results on a chart, checking exact times with a wristwatch. The routine continued for over 3 hr, during which little was said to the mother, who waited passively on a wooden bench. Later interviews with the mother revealed that she believed the child’s diarrhea was due to evil eye, and had previously consulted 3 [traditional healers]. Despite her more than 5 hr at the clinic, the mother did not know how to mix the ORT packet herself. When asked if she thought she or a [traditional healer] could mix ORT at home, she replied, “Oh no, we could never do that. It’s so complicated! I don’t even know how to read or write and. I don’t even own a wristwatch!”
They then discuss how they trained a broader group of providers (including the traditional healers) to administer the ORT themselves. But the traditional healers end up ritualizing the treatment as much or more than the physicians did:
… Dona Geralda cradled the leaves carefully so as not to spill their evil content as she carried them to an open window and flung them out. The evil forces causing diarrhea are believed to disappear with the leaves, leaving the child’s body ‘clean’ and disease-free…Turning to the small altar in a comer of the healing room, Dona Geralda offered the ORS ‘holy water’ to the images of St Francisco and the folk hero Padre Cicero there.
There’s more to both sides of the ritualization in the body of the article, and the similarities are striking. I’m not sure the authors intended to make it this way (and this likely speaks more to my own priors or prejudices), but the traditional healers’ ritualization sounds quite suspicious to me, full of superstition and empty of understanding of how ORT works, while at the same time the physicians’ ritualization, as unnecessary as it is, is comfortingly scientific. The crucial difference though is that the physicians’ ritualization seriously impedes access to care, while the healers’ process does not — in fact, it even makes care more accessible:
Clearly, our program did not de-ritualize ORTdelivery; the TH’s administration methods are highly ritualized. The ceremony with the leaves, the prayer,the offering of the ORS-tea to the saints, are all medically unnecessary to ensure the efficacy of ORT.They neither add to nor detract from its technologial effectiveness. But in this case, the ritualization, insteadof discouraging the mother from using ORT and mystifying her about its ease of preparation andadministration, encourages her to participate actively in her child’s cure.
- Go read Austin Frakt on the importance of counterfactuals in policy debates. Somewhere deep in the WordPress interface for this blog I have a draft post on the most important lessons I learned in my year of coursework at Hopkins public health — numbers one, two, and three on the list are “the counterfactual,” “thinking about the counterfactual,” and “mo really, the counterfactual is everything.”
- Two cool upcoming events: in New York City, a big data conference on September 13-14. And in Stockholm, a panel discussion featuring Sachs, Easterly, Duflo, Collier, and Rodrik. Wow. h/t Tom Murphy
- Here’s an in-the-weeds description of how two groups of researchers came up with fairly different child mortality estimates — the pointer came from David Spiegelhalter on Twitter.
- Flowing Data describes how researchers store petabits of data with DNA.
- A friend posts on Awra Amba, a non-religious community with gender equality in rural Ethiopia.
- On a completely off-topic front: a diatribe against splitting the Hobbit into three different movies, at TheOneRing.net. I remain deeply skeptical that the least serious of Tolkien’s Middle Earth canon will make one good movie, let alone three…
- Lastly: I met someone recently in Addis who had read my Hunger Games survival analysis in the spring, and the nerdy side of me (which is most of me) found this deeply gratifying.