Wednesday, 12 June 2013

These 32 hours have 20 falsifiable forecasts

Phillip Metaxas and Andrew Leigh sat through 32 hours of Australian talking-head TV.* In all that talking, they were able to find only 20 falsifiable forecasts from Australian pundits. The rest were so qualified that they were unfalsifiable.

They suggest that talk show hosts demand falsifiable predictions:
A nice start would be for Insiders host Barrie Cassidy to insist on ‘falsifiable predictions’ in his final segment, rather than the current ‘observations or predictions’ from the pundits. While such an innovation would be helpful, assessing whether a prediction has proven true or false is likely to provoke objections from the pundits involved. As Gardner (2010) notes, when predictions go awry, pundits often invoke an array of excuses to exonerate themselves. Such excuses include the ‘I was almost right’ defence, and the ‘wait and see’ defence. In our analysis, had David Marr and Lenore Taylor been incorrect about public opinion changing on the mining tax, they could easily have invoked the ‘wait and see’ option to defend themselves. 
For this reason, we advocate a Tetlock-style experiment involving Australia’s leading political pundits. Such an experiment substantially removes the ability of pundits to invoke the common defence mechanisms when their predictions go astray. An appropriate study would require our pundits to make falsifiable predictions about the political world in the future. Predictions could cover a range of topics such as the party likely to win the next federal election, the next leaders of each party, and the passage of key legislation. There would be a sufficient number of questions to ensure that lucky hits do not skew the results. We need more evidence than our analysis could provide to properly separate flukes from prescience. Furthermore, an anonymous survey like Tetlock’s would help determine whether our pundits as a group are worth the airtime they are afforded. An open survey could also give us the ability to rank our political pundits based on their predictive powers.
I'd love to see a talk show partner up with one of the betting markets to refine the opposing pundits' claims into a single falsifiable test prior to the show. Launch the contract at the start of the show, unless it's a contract that already exists, like "Who will win the next election", then have them argue about the appropriate price for the contract while the audience submits bids and asks. Let the ticker move over the course of the show, then insist that both of them put, say, $1000 on it at the conclusion of the debate if their argued price differs from the market price.

This would be really easy for things like "Who will win the next election?" or "Will Julia Gillard lead Labor into the next election?". And it isn't that much harder for combinatorials like "Labor would win more (less) seats under Gillard than under Rudd".

There would be nothing banning such things in Australia or New Zealand. There are political betting markets in both places, even if Senator Xenophon keeps trying to ban them in Oz. The constraint seems more likely to be that viewers aren't really all that interested in getting falsifiable predictions and accurate odds. Theatre and that cherished fantasies never ever be shown to be wrong - that's where it's at.

* Ok, they read transcripts.


  1. Poor researchers, sifting through all that garbage for so little. For anyone who's looking for a better airtime-to-predictions ratio, I suggest sports. At least football (soccer) pundits routinely make precise predictions about the next champion or the exact result of an upcoming match. Also, those are results I'd really like to see. And I'm not going to do it myself.

  2. I like the idea but would love to see it extended to academics and politicians as well. How accurate are Andrew Leigh's predictions? Why not start by subjecting himself to the standard of falsifiability, now that he's a politician?

  3. I'm pretty sure that he'd be willing to bet folks were it to come up.