The talking heads on TV give such a convincing story about what the future holds that its hard not to believe them. But pinning them down to a time-frame and discrete future event is often next to impossible, so you can never determine whether they were right or wrong. When such talking heads are pinned down, their track-record turns out to be very poor, according to Philip Tetlock in his latest book “Superforecasting: The Art and Science of Prediction“. This makes sense as they are picked more for their entertainment value, than their track-records.
The real super-forecasters are people like Doug Lorch (ex-IBM, retired, lives in Santa Barbara), Mary Simpson (independent financial consultant, formerly regulatory affairs at utility Southern California Edison) and Devyn Duffy (welfare case worker, Pittsburgh). What they lack in TV appeal they make up for in their approach to forecasting.
Tetlock and team found them as part of an IARPA, which is part of US Intelligence, project to improve forecasting of political and other events. He found that the way you approach forecasting rather than your credentials or political leanings matter most. If you’re a one-big idea person (i.e. a hedgehog) you will struggle, while if you’re a pragmatic tinkerer (i.e. a fox) you will do well. He listed the eleven commandments for aspiring super forecasters:
- Focus on questions where your handwork will pay off. So don’t bother with a question like “who will win the US election in 2028?”. It’s too far out to bother predicting.
- Break big problems into smaller ones.
- Strike the right balance between inside and outside views. There is nothing new under the sun. So first calculate the odds of an event happening based on similar ones in the past (i.e. top-down or outside view), then work-out the bottom-up/inside view of the specific event.
- Strike the right balance between under- and overreacting to evidence. Careful incorporate new evidence and update beliefs accordingly. The best forecasters are incremental belief updaters.
- Acknowledge the counter-arguments. If you believe military action never works, be open to the possibility that it might. Take both views and synthesise.
- Distinguish as many degrees of doubt as the problem permits. Your uncertainty dials needs more than 3 settings (certain, maybe, impossible). Translate vague hunches into numeric probabilities.
- Strike the right balance between under- and over-confidence. Understand the risks of rushing to judgment and of dawdling too long near “maybe”.
- Don’t justify or excuse your failures. Conduct unflinching postmortem.
- Bring out the best in others and let others bring out the best in you. Master team management: perspective taking (being able to reproduce the others argument to their satisfaction), precision questioning (clarify arguments so there is no misunderstanding), and constructive confrontation (learning to disagree without being disagreeable).
- Master the error-balancing bicycle. Learning requires doing, with good feedback that leaves no ambiguity about whether you are succeeding.
- Don’t treat commandments as commandments!
You can practise your prediction skills by joining Tetlock’s Good Judgment Project