The Tetlock Forecast

James Thompson in Psychological Comments:

I have admired Philip Tetlock since, almost 30 years ago, he reviewed a book I had just written which contained one big and so far untested prediction, and gave it by far the most detailed, insightful and helpful assessment it had received among many warm but perfunctory reviews, mildly adding references to a few papers which, when I followed them up, showed me exactly how much I had missed out. His kindness made his critical points far more effective. (In a subsequent lecture tour I met up with one of the international affairs experts he had mentioned, who offered to work with me, though in the end I went on to other things, and consequently made no revision of the book).

Now the Press are picking up his work on super-forecasting, which has major implications for how we go about anticipating and planning for future events, supposedly one of the features of high intelligence. Bright people should be particularly good at forecasting, shouldn’t they?

What has Tetlock found? First, that most pundit forecasts are unfalsifiable. Even time travel would not help you know if the predictions of these commentators had been met. They are at the low level of Nostradamus and contemporary journalism. Second, if you run a proper forecasting contest (not “will there be a stock market correction sometime soon” but “what will the Standard and Poor index stand at on 31 December 2015”) most commentators are “too busy” to participate. They do the broad brush stuff which gets well paid, not the nitty-gritty testable stuff that nerds do for fun.

More here.