How to Predict the Unpredictable

Steven Poole in The Guardian:

“Prediction is very difficult,” the great Danish physicist Niels Bohr was fond of saying, “especially if it's about the future.” This book doesn't in fact claim to teach you how to predict what is really unpredictable – such as the weather in a month's time, or the next turn of the roulette wheel. It might more honestly, but less seductively, have been titled How to Predict the Sort-of Predictable Behaviour of People Who Are Trying to Act Randomly. This is actually much more interesting than the bland paradox of the given title. When they want to act unpredictably, it turns out, people deviate from true randomness in ways that can be recognised. According to Poundstone's vivid account, this was first rigorously demonstrated by a family of “outguessing machines” created by mathematicians and engineers at Bell Labs in the 1950s.

The outguessing machines played a very simple game. Every round, both machine and human player pick one of two choices: heads or tails, left or right. It is decided beforehand that if the choices match, one player scores a point, whereas if they are different, the other player scores. What happens is that, over dozens of rounds, humans fall into unconscious patterns that a computer can recognise, and therefore anticipate. In this way, with only 16 bits of memory (16 ones or zeroes), a machine by the information theorist Claude Shannon was able to beat all comers. To call this “outsmarting” the humans is perhaps a bit of a stretch, but it is what Poundstone means by the term when he goes on to apply it to different areas. Indeed there are a surprising number of areas where a similar kind of “outguessing” strategy can be fruitful. Rock, Paper, Scissors is a random game, but because most people deviate from true randomness, it is possible to have a strategy. (“A player who loses is more likely to switch to a different throw the next time,” Poundstone explains as an example.) In tennis, too, most players alternate their directions of serve too regularly, so Poundstone recommends using a wristwatch or heart-rate monitor to properly randomise them.

More here.