Ed Yong in Not Exactly Rocket Science:
Here’s David Attenborough, chilling out on a rock in the middle of Africa, with four lumps of plasticine. The smallest one on the far left represents the brain of a bushbaby, a small primate that lives on its own. The next one is the brain of a colobus monkey, which lives in groups of 15 or so. The one after that is a guenon, another monkey; group size: 25. And on the far right: a baboon that lives in groups of 50. “Were you to give a skull to a researcher who works on monkeys, even though they didn’t know what kind of monkey it belonged to, they would be able to accurately predict the size of group in which it lived,” says Attenborough.
That sequence, from The Life of Mammals, is a wonderful demonstration of the social brain hypothesis—a bold idea, proposed in the 1980s, which suggests that living in groups drove the evolution of large brains. Social animals face mental challenges that solitary animals do not: they have to recognise the other members of their cliques, cope with fluid and shifting alliances, manage conflicts, and manipulate or deceive their peers. So as social groups get bigger, so should brains. This idea has been repeatedly tested and confirmed in many groups of animals, including hoofed mammals, carnivores, primates, and birds.
What about insects? Ants, termites, bees, and wasps, also live in large societies, and many of them have unusually big brains—at least, for insects. But in 2010, Sarah Farris from West Virginia University and Susanne Schulmeister from the American Museum of Natural History showed that in these groups, large brains evolved some 90 million years before big social groups. If anything, they correlated with parasitic body-snatching rather than group-living.
More here.