Causality anyone? [updated]

[Updated below]
The New Zealand press is all a-twitter with reporting on the latest study by the folks at the University of Otago finding that marriage is good for mental health. Neither the stories nor the University's press release mentions the word causality.

There's one simple question that ought to be required for any journalist interviewing any social scientist on any issue like this: "How did you address causality?" It's a simple question. If the researcher says "Oh, we used a panel design to get within-subject effects" or "Oh, we instrumented by using...", that's a great start! If not, all you've got is correlation.

Digging up the paper, we find that they've run a hazard model on the likelihood of first onset of each of several mental illnesses. Every person-year is an observation, where each person in the survey is asked to remember back for each prior year and state whether they've had any of a list of mental illnesses. Single and divorced people are more likely to start having their first instance of a mental illness than currently married people. Or, rather, persons of any current marital status are more likely to report having had their first instance of a particular mental illness while they were single or divorced or during their second marriage than while in a first marriage.

Of course, if folks in the dating market are saying "I don't want to marry her because she seems likely to be crazy" or "I like him, he seems really stable", or "I've gotta divorce her because she's showing signs of going nuts and I can't deal with the hassles I can see coming down the track", then causality is entirely wrong: marriage doesn't prevent mental illness, rather, folks are more likely to marry and less likely to divorce folks who seem less likely to become mentally ill. All we need is potential partners to be roughly right in their guesses, and it's then screening. Male-female differences are then just measures of how able females and males are to detect (prospective) partners' mental stability crossed with how much they care about it.

In the paper's conclusion:

Finally, the limitations to the control of selection bias also need to be acknowledged. The survival analysis that we employed reduces the effects of selection bias by excluding situations where the prior existence of the focal disorder has an influence either on reduced chances of becoming married or increased chances of marriage dissolution. However, it cannot eliminate the possible influence of factors that may both decrease the likelihood of getting married and increase the likelihood of mental disorder onset, such as personality or history of sexual abuse.[emphasis added] The fact that we found that marriage was associated with reduced onset of disorders that typically occur well before marriage (the phobias), is suggestive of some residual selection bias of this sort, though this would only apply to the contrast between the married and the never married.

Yes, using first onset timing helps. But if screening happens prior to first onset by folks looking prospectively and being on average right, then there's still a causality issue.

Worse, it's unclear in the paper whether each hazard regression has "the first onset of this mental illness" or "the first onset of this mental illness, which is the first mental illness experienced by the respondent" as the dependent variable. If the latter, then the selection bias is far worse: if the likelihood of experiencing any mental illness is increased by having experienced some other mental illness, and if any mental illness reduces the probability of marriage and increases the probability of divorce, then a method that looks at each illness separately without considering the selection effects induced by comorbid prior mental illnesses... would have problems.

Just ask yourself: when you've been dating, has the relative mental stability of the potential partner made you more likely to want to keep seeing them? Sure, we can get things wrong, and sure, some folks who seem really stable can go on to develop mental illness. I'm sure lots of folks can tell stories about a partner who just snapped out of nowhere. But if on average over lots of people folks guess these things about right, then there's still a big causality problem in the paper.

None of the news reports even asked about causality. I'm not sure a lot more could have been done about causality in the paper given the data. But I'd certainly be nervous about staking bold causality claims on it, and all of the journalists are suggesting the relationship is causal.

I sometimes wonder whether a one day session for journalists on the basics of statistics, inference and economics would be worthwhile or if it would only attract the folks who already know about it. Nothing controversial -- just the stuff on which the vast majority of economists and statisticians agree. Like the difference between percentages and percentage points, nominal and real, cumulative effects of percentage changes over time, external versus internal costs, correlation and causation....

Update: Dr. Scott kindly reports that survey respondents code the frequency of various symptoms of mental illness, looking back retrospectively, then an algorithm determines whether a set of symptoms and their frequency adds up to an instance of mental illness. The hazard is the first onset of the focal disorder (each disorder taken in turn) regardless of prior onsets of other disorders. I'm still then more than a bit nervous about selection driving things: if you're less likely to get married (more likely to divorce) if you've exhibited one disorder (or symptoms thereof) and if the likelihood of another disorder is increased by having exhibited a prior one, then the hazard of any particular disorder's onset is increased by being single because you're less likely to have selected into marriage.