Trusting Secret Data: Dunedin edition

Unless you've run the regressions yourself, it's often hard to trust empirical results. A lot of results are fragile - small changes in specifications, either changing date ranges or adding seemingly irrelevant variables, can change results. And endogeneity always makes inference hard in social sciences.

First best is running things yourself. If the data and code are available for replication and extension, then it's impossible to maintain outright fraud. And fragile results will eventually be commonly known to be fragile. Boing Boing recently ran a very nice summary in two parts on why we see so many contradictory results on guns and crime: all the specifications are fragile. On some questions, teh science isn't strong enough to give much policy direction.

When the data is not publicly available, we really have to trust the people running the specifications. There can be good reasons for keeping data private. Most self-interestedly, if the researchers did a ton of work in putting together the dataset, they might well want to get a few papers based on it before letting everyone else pile on. It's not an edifying reason, but even if the data collection involves public funding, it doesn't seem unreasonable to give the survey team the first few kicks at the can. On the other hand, if the data cannot be shared without violating the privacy of individuals who answered the surveys, and if there aren't clever ways of anonymising the data so that it cannot be used to identify individuals, then that's a pretty good reason to be stingy with the data. 

But, there potentially being good reasons for secret data certainly isn't sufficient reason to trust the consequent analysis. And so we come to Dunedin vs Rogeberg.

The Dunedin Longitudinal Survey is secret data, likely for the good reason of wishing to maintain the anonymity of survey respondents. We really have to trust the people running regressions on secret data. The Dunedin group recently found that early marijuana use predicts IQ decline. Ole Rogeberg wondered whether the results were due to cohort selection effects: the kids most likely to try marijuana early could well have had different results even had marijuana not existed. Rogeberg provided Dunedin a list of tests that might sort things out. He summarises things here:

I’ll start with a short recap: Researchers published article august 2012 arguing that adolescent-onset cannabis smoking harms adolescent brains and causes IQ to decline. I responded with an article available here arguing that their methods were insufficient to establish a causal link, and that non-cognitive traits (ambition, self-control, personality, interests etc) would influence risks of adolescent-onset cannabis use while also potentially altering IQs by influencing your education, occupation, choice of peers etc. For various reasons, I argued that this could show up in their data as IQ-trends that differed by socioeconomic status (SES), and suggested a number of analyses that would help clarify whether their effect was biased due to confounding and selection-effects. In a reply this week (gated, I think), the researchers show that there is no systematic IQ-trend difference across three SES groups they’ve constructed. However, as I note in my reply (available here), they still fail to tell us how different the groups of cannabis users (never users, adolescent-onset users with long history of dependence etc) were on other dimensions, and they still fail to control for non-cognitive factors and early childhood experiences in any of the ways I proposed. In fact, none of the data or analyses that my article asked for have been provided, and the researchers conclude with a puzzling claim that randomized clinical trials only show “potential” effects while observational studies are needed to show “whether cannabis actually is impairing cognition in the real world and how much.”

I'll be interested in seeing the Dunedin group's reply when it comes out. Rogeberg points to evidence the Dunedin group themselves have provided showing reasonable cohort differences between users and non-users. It's not implausible that these kinds of differences are responsible for at least some of their measured marijuana effect. It would be simplest if Dunedin would send Rogeberg the data and make him sign a data confidentiality waiver; it's not particularly plausible that a Norwegian labour econometrician wants the data for identifying NZ individuals. But if they're not willing to do that, they should at least be willing to put his code to their data. 

I'll be downgrading my trust in all the Dunedin Longitudinal results if they don't handle this well; secret data requires trust.

There is away around all of these kinds of problems though.

The General Social Survey in the US would have almost as many privacy concerns as the Dunedin survey. And yet they're able to make available an online resource letting anybody with a web browser run analysis on the data, for free. I regularly set assignments in my undergrad public choice class where students without much in way of metrics background have to go and muck about in the data.

As I understand things, the Health Research Council funds the Dunedin surveys. There's a worldwide movement toward open data in government-funded projects. HRC could fund a Dunedin Longitudinal equivalent of the GSS browser analytics. Anything personally identifying, like census meshblock, could be culled. Nobody would see the individual observations. And any cross-tab that would reduce to too-small a number of observations could return nulls. But folks with concerns about Dunedin studies could do first-cut checks without having access to the bits that might cause legitimate worries about privacy. Somebody like Rogeberg should be able to run a t-test on whether those who later go on to report marijuana use differed on other important variables prior to their starting consumption.

We all could trust Dunedin results more if we could check things like this. And it shouldn't be hard to set this up either: see how SDA coded things for GSS, put in the Dunedin data instead, then host it. If HRC did this, one small bit of funding could have a whole ton of researchers publishing useful studies based on the already-funded survey instead of having to fund Dunedin for every new regression. Let the Dunedin group keep any new wave of the survey under wraps for a couple of years before releasing it to the online analytics so they get a fair kick at the can. After that, why not open things up?

Uncultured

It's not easy being an anti-nationalist cultural-syncretist rationalist atheist quasi-trans-humanist singulatarean when your kid's school is looking for volunteers for the international cultural day.

We're one of the many international families sending a NZ-born child to Ilam School. I'm Canadian, Susan is American, the kids are Kiwis who are eligible for the other citizenships. While there exist things that are certainly Canadian and American, they're not really part of our culture, even if they're part of the culture of the places where we were born. And the things that we do do are really close enough to the things done by other Kiwi households that it would be pretty boring. Presents at Christmas, pancakes with maple syrup, sledding when there's snow... hohum. There's more curling in Canada, but Kiwis do it too. Same for hockey, which here you must preface with the word ice to distinguish it from the silly field variant with the tiny tiny sticks. Bad weak coffee is Canadian culture, but not my culture. And a cultural demonstration of arresting people for selling wheat or milk outside the marketing board doesn't really show the best of Canada, isn't my culture, and is a bit too didactic for my taste anyway.

It's not like our family doesn't have a culture. The kids get more Canadian stories and songs at home than would most Kiwi kids, but I expect that they also get a lot more of the Wotanic and Greek myths too. The kids have heard and enjoyed Anne of Green Gables, but it's no more a part of our culture than is the Wizard of Oz, Alice in Wonderland, TinTin, Lord of the Rings, Dungeons & Dragons, Little House on the Prairie, Wagner's Ring Cycle, The Iliad & Odyssey, Tennyson, Coleridge, Poe, Kipling, Freakazoid, Pinky & The Brain, or any of the international fairytales from the kids' audiobooks to which we listen on the morning commute. Inuit artwork is overrepresented at our house relative to a Kiwi norm, but so too are the amate paintings of San Agustin Oapan.

And so I haven't a clue what to fill in for the international day questionnaires. Our at-home culture is chaotic good / rationalist-geek-science / pluralist libertarian. When they're a bit older, they'll get the Yudkowski treatment, but it's kinda built into a lot of stuff already.

Our kids play Paper-Rock-Scissors-Lizard-Spock. Maybe we could share that as being broadly representative. It's closer to right than making poutine and having him wear his Winnipeg Jets sweater.

Morning roundup

Item the first: New York Supreme Court Judge Milton Tingling blocked Bloomberg's ban on Big Gulps. I agree with the Judge here:

Judge Tingling found the Board of Health's mission is to protect New Yorkers by providing regulations that prevent and protect against diseases. Those powers, he argued, don't include the authority to "limit or ban a legal item under the guise of 'controlling chronic disease.' "

The board may supervise and regulate the city's food supply when it affects public health, but the City Charter clearly outlines when such steps may be taken: According to Judge Tingling, the city must face imminent danger due to disease.

"That has not been demonstrated," he wrote.

Judge Tingling also suggested that Mr. Bloomberg overstepped his powers by bringing the sugary drink rules before the Board of Health, which is solely appointed by him. The City Council, he wrote, is the legislative body "and it alone has the authority to legislate as the board seeks to do here."

But a malicious part of me really wishes this had gone ahead. The ban looked to be an intractable nightmare that New York, having elected Bloomberg, really deserved to endure. They could have served as example unto others. Instead, the lesson is that judges will bat back things that are entirely too crazy, so there's no harm in electing Bloombergs. Sometimes, the electorate does deserve to get what it wants, and that right hard.

Item the Second: Rasmussen polls show 36% of Americans think it's at least somewhat likely that Fukushima radiation "did significant harm to the United States." I would bet that a survey of scientists specialising in radiation would find no more than 1-2% thinking it did significant harm to the US, and those would be saying that the negative effects were through reduced public acceptance of nuclear power. I really really wish that polls of this sort, where there is a right scientific answer, or at least a likely strong consensus among scientists, would mention it. The survey doesn't mean that there's a 36% chance that it did harm, it means that 36% of Americans are scientifically illiterate.

EQNZ Lesson 1 for Wellington: Fix EQC

It's too late for policy changes to do much for Christchurch. But the Alpine Fault's pointed straight at Wellington. The quake's hitting there will be bad enough; a few policy changes could avoid exacerbating the damage.

Lesson One from Christchurch should be that EQC really should never have had the job of assessing house damage. It should have no role in repairs, in project management, or, really, anything other than writing cheques.

When #EQNZ Wellington hits, from the homeowner's perspective, it should be as though EQC doesn't exist. Instead, the private insurer should simply send its assessment to EQC for coverage of the below-cap damages: EQC then effectively simply covering a high deductible. EQC could do some random audits to check if any of the private insurers were making a habit of getting all their clients a house tidy-up where there wasn't really any damage, but it really oughtn't otherwise be involved.

Here in Christchurch, because EQC and Fletchers are severely competence-constrained, they decided it was too complicated to let homeowners add at the homeowners' own cost insulation into the walls when the wall linings were already being torn out for repairs. In the last month they've reversed that decision, rightly outraging those homeowners lucky enough to have completed their repairs but who missed out on getting other repairs done.

Asbestos isn't uncommon* in houses of a particular age here. Where the quakes damaged the asbestos, the home repairs have it taken out as part of the insured claim. But where homeowners wish to have the rest removed at the homeowners' own cost, they're being barred from having it done with the rest of the repairs. EQC is right that while the material is undamaged, it is entirely harmless. But the marginal cost of removing this stuff is much lower while everything else is opened up, some future quake could release asbestos fibres, and having any asbestos in a house makes other home repairs more complicated - you don't particularly want to drill into an asbestos sheet by accident.

The constraint seems to be EQC's worry that asbestos contractors' taking an extra day or two on minor jobs holds them up from hitting other jobs. Alternatively, letting asbestos contractors' wages be bid up with demand induces more asbestos contractors' to move into town to get the job done.

This kind of nonsense was one reason we went for an opt-out builder; we're still hoping to get our work done in May-June.

It's much too late to get this part fixed for Christchurch. But hopefully somebody's doing it before Wellington gets its shake.

* I'm still hoping 3-d bioprinted lungs are available in 40 years' time. I'd be pretty surprised if there weren't an #eqnz mesothelioma hit waiting for folks who were here through the dusty times post-quake.

Blind to others' ends

Marion Nestle's op-ed in the New York Daily News is breathtaking.

She begins:

If we want Americans to be healthy, we are going to have to take actions like this - and many more - and do so soon. It's long past time to tax sugar soda, crack down further on what gets sold in our schools, tackle abusive marketing practices, demand a redesign of labels - and extend the soda cap, no matter how controversial it may seem. This must be the beginning, not the end, of efforts toward a healthier America.

In short, we need a series of serious changes to make the healthy choice the easy choice. The soda size cap is a nudge in that direction. You will still be able to drink all the soda, and down all the sugar, that you want. The cap on soda size makes it just a tiny bit harder for you to do so.

...Most people eat whatever size is in front of them - the "default," in public health-speak - and are content with that amount. So a reasonable goal of public health intervention is to change the default drink to a smaller size. Hence: Bloomberg's 16-ounce size cap. From my nutritionist's perspective, a 16-ounce soda is still generous. Just one contains the equivalent of 12 packets of sugar. Just one provides 10% of the daily calorie needs of someone who typically eats 2,000 calories a day. Just one contains the upper limit of sugar intake that health officials recommend for an entire day. Once you down a 16-ounce soda, it's best to stop right there.

From my perspective, as someone who regularly shares a soda with his wife at the movies, having to leave a movie for a refill is more than a minor inconvenience and no longer being able to share a drink with your wife adds more than a minor cost. But Nestle either can't see the diversity of ends sought by those wishing larger sodas, or considers them irrelevant collateral damage.

So-called "nanny-state" measures - like bans on driving while drunk, smoking in public places and, now, selling absurdly large sugary drinks - help to level the playing field. Such measures are about giving everyone an equal opportunity to live a safer and healthier life.

I'd thought the point of drink-driving laws was the protection of other drivers. And that's how bans on smoking in public were sold, though that always seemed really rather a stretch when they started pushing for bans in outdoor areas.

The column doesn't get better.

At the moment, it is up to you to make healthier choices, but that's not easy in the face of relentless soda marketing. Governments have a responsibility to provide healthier environments for their citizens.

Yes, we're all helpless in the face of marketing. I think she needs some theory explaining why ads for soda are that much more persuasive than ads for milk and kale.

Here are some additional actions New York City should take, if only it were allowed to. Close the loopholes. The city does not have jurisdiction over sales of sodas in convenience stores and supermarkets. The state does. Gov. Cuomo denied Mayor Bloomberg's request to extend the size cap to those stores, not on principle but because he hadn't thought about it. He should, right now. Let's keep all sugary drinks to 16 ounces or less.

And what of larger families who like to share larger-sized drinks? Or buying soda for parties?

Fix the price differential. A 7.5-ounce can of soda costs twice as much per ounce as a two-liter bottle, and you can't buy just one; it comes in an 8-pack. Price determines sales. If a 16-ounce soda costs a dollar, a 32-ounce soda should cost two dollars.

Hey, let's extend her price control regime to everything. If a single toilet paper roll costs $0.25, there's clearly no reason that a 24-pack should cost anything other than $6.00. If a half-dozen eggs costs $4, then a dozen should cost $8. If a single drumstick at KFC is $1, then a 20-pack should be $20. If a night at a hotel is $100, then a week should be $700. There's no reason anybody might provide volume discounts except to foster addiction and overconsumption. If a motorbike with two wheels costs $10,000, then a car should cost $20,000 because it has twice as many wheels. Just put Marion Nestle in charge of the "Setting the price of everything" committee. It'll be great.

Tax sodas. Most people wouldn't dream of eating candy all day, but soda companies have made it seem normal to drink sodas from morning to night. Raising the price of sodas would discourage sales, especially among young people most susceptible to marketing efforts and most vulnerable to weight gain. A one-cent tax per ounce should do the trick and raise plenty of needed revenue besides.

I rather doubt that a one-cent-per-ounce tax would satisfy her thirst.

It goes on in this vein for a while. She concludes:

Actions like these will evoke ferocious opposition from the soda industry, and it will spare no expense to make sure such things never happen. We would surely hear more and more howls of "nanny-state" from those who insist Bloomberg has led us to the brink of a public health police state. Polls say that many New Yorkers oppose the 16-ounce cap and would oppose measures like this, too.

But I can't tell whether the opposition comes from genuine concern about limits on personal choice or because soda companies have spent millions of dollars to protect their interests and gin up histrionic, misinformed opposition.

Oh, those illegitimate howls of "nanny state" against a woman who wants to ban advertising of soda (if it looks to her like the marketing targets kids), ban vending machines in schools, mandate great big calorie warning labels on the front of containers, and ban 2-litre bottles from supermarkets.

Valuing amenities

A few things I tell my Economics & Current Policy Issues Class:

• It's double-counting to weigh the direct costs of some disamenity as well as the disamenity's effects on land values because the latter incorporates the former (and the same for positive amenity affects);
• We can weigh the value of hard to value things like mortality risk by looking for effects on priced markets. Differential on-the-job accident risk can give us measures of the value of a statistical life. The value of a neighbourhood park can be approximated by housing price effects, and the same for the value of neighbourhood disamenities.

And so I'll have to remember this rather nice NBER working paper by Currie, Davis, Greenstone, and Walker. They pick a relatively hard case: plants that emit toxic chemicals that are sometimes hard to smell or see. They find emissions decline exponentially with distance from the plant, reaching baseline levels a mile from the plant. When a plant opens, housing prices in a half mile radius drop by 1-2%; when it closes, they go up by 2-3%. Within a mile, the price drop is 1.5%. A plant within a mile also increases the likelihood of having a low birthweight child by two percent (that percent, not percentage points, and the base rate incidence is low). 

When they add everything up, they find that a plant's opening reduces average housing market capitalisation in the surrounding mile by $1.5 million on average,* and that the costs of low-birthweight births in the surrounding mile are about $700k. So the disamenity cost, as measured by housing prices, is about twice the biggest likely measurable health effect. And that's entirely consistent with housing markets capitalising the real imposed disamenities. The biggest price effects are closest to the plant, where health disamenities combine with the noise and sight of the plant, then decline towards the one mile mark, where there are still real health effects but the other disamenities are negligible relative to those in the reference category of homes 1-2 miles away. 

They also find that neighbourhoods around plants pick up in demographic characteristics after a plant's opening: people move there for jobs. They wonder whether this is consistent with fully informed decision-making, but it's rather plausible that, given the low absolute risk imposed, households simply value being within walking distance of work more than they dislike the disamenties. 

On the whole I take this as pretty supportive of our base-line theory on this stuff and reasonable evidence against the "Oh, but people are stupid and irrational and they would never live there if they only knew and we have to ban a pile of stuff" alternative hypothesis.

* By way of comparison, they note that a typical plant - a small coal-power plant - costs $280m to build. The disamenity effect, though real, isn't some big huge cost that would tip the balance against approving the plant were it incorporated.

Unintended consequences?

Richard Thaler points to a lovely piece of university regulation. We cannot afford such a regulation at the post-earthquake University of Canterbury, but it is beautiful.

Don't get Euro rule that bonus<salary. Won't it just raise salaries? Once worked at U that had rule alcohol<food at dinners-> more desserts!
— Richard H Thaler (@R_Thaler) March 6, 2013

Lots of universities get tetchy about alcohol being on dinner bills where staff are taking distinguished guests and speakers to dinner after an event. Some of it is puritanism, some of it is worries about how it looks, and some of it is pure misanthropy from university regulatory czars.

This rule consequently is a beautiful thing. It has all the appearance of working to constrain alcohol spending. But it encourages a move to a far better equilibrium: dinner with dessert and with wine, while still prohibiting really outrageous wine expenditures. If this were an intended consequence with a deliberately opaque regulation, would that academia had more administrators of this sort.