The “Streetlight Effect”: A Metaphor for Knowledge and Ignorance

by Yohan J. John

There is a story that I think anyone interested in human knowledge ought to know. It comes in many forms. Here is one version, incarnated as a joke: 'A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, “this is where the light is.”'

A parable featuring the Seljuk Sufi mystic Nasrudin Hodja may be the earliest form of the story: 'Someone saw Nasrudin searching for something on the ground. “What have you lost, Mulla?” he asked. “My key,” said the Mulla. So they both went down on their knees and looked for it. After a time the other man asked: “Where exactly did you drop it?” “In my own house.” “Then why are you looking here?” “There is more light here than inside my own house.”' The Indologist Wendy Doniger quotes this parable in her book The Hindus: An Alternative History, as a way to prepare the reader for the disappointing realization that the “available light” on Hinduism — the hymns, the histories, the archaeological remains — tends to illuminate the perspectives of dominant groups, relegating to the shadows the viewpoints of women, lower castes, and other marginalized groups.

Noam Chomsky has a characteristically dry and precise version of the story: “Science is a bit like the joke about the drunk who is looking under a lamppost for a key that he has lost on the other side of the street, because that’s where the light is. It has no other choice.”

So historians, mystics, scientists and drunks have something in common: they all tend to seek the truth where the process of seeking is easy, rather than where truth is. Responses to this problem vary. The mystic is most likely trying to remind the listener of how limited human knowledge is, and how often we look for solutions in precisely the wrong places. The humanities professor Doniger uses the problem as a justification for reading between the lines: using the available light to speculate about what may lie in the darkness. And the cognitive scientist Chomsky seems to be using the problem to justify why scientists answer questions that are almost, but not quite, entirely unlike the questions they originally set out to answer.

I'm a scientist, a history buff, and also a bit of a mystic, so I tend to combine all three perspectives on the “available light” problem, also known as the “streetlight effect“. The mystic (and the teller of jokes) is right to ridicule our tendency to look for something in the wrong place, however well lit that place happens to be. It's absurd to force a problem into some intellectual framework simply because this framework happens to be convenient. But does this mean we should abandon problems that have no good framework? That seems to be what Chomsky is saying. Don't waste your time fumbling in the dark: answer the questions you can answer with the resources you have. The rest is just hand-waving. This hard-nosed approach is popular among a certain sort of scientist and engineer, but it leaves most people quite dissatisfied. 'What's wrong with a bit of hand-waving?', Doniger seems to be asking. In the case of history, we know something must have been going on. Why not use our imaginative faculties to shine light in the dark places?

Combining these three attitudes towards “available light” is easier said than done. Given a particular problem that seems to be lurking on the boundaries between light and dark, do we resign ourselves to wide-eyed wonder? Scorn humanity's hubris? Move into the light? Or try to trace shapes in the dark? I think the key is to know where you are, and also what time it is.

I like to call vision the 'master metaphor'. To understand is to see clearly. And to see we need illumination. Ignorance is darkness. The metaphor of seeing as understanding works very well with other famous metaphors, so I'd like to indulge in a metaphor mash-up. Humans require light to see what they are doing. The hours of daylight are synonymous with work, even though many of us work under artificial lighting. If light is the key to understanding, then sunlight is the Big Idea: the intellectual framework that allows us to see everything. The sunlight lets us delineate object and shadow, true and false, good and bad. Sunlight is what determines what the “true” color of an object is. Sunlight is objectivity. Sunlight is metanarrative. Sunlight is ideology.

Nighttime by contrast, is synonymous with play, mischief, music, fun, and sex. Anything other than work. Under the light of the moon and the stars, the sharp daylit boundaries become blurred. Color is drained from the world. Shades of grey abound. In candlelight, the shadows dance. Nighttime is a time of escape from the strictures of the day. Madness can come when the free-form night invades the structured day. Pink Floyd's Roger Waters conjures this image up in 'Eclipse', the litany that concludes The Dark Side of the Moon:

All that you touch
All that you see
All that you taste
All you feel
All that you love
All that you hate
All you distrust
All you save
All that you give
All that you deal
All that you buy
beg, borrow or steal
All you create
All you destroy
All that you do
All that you say
All that you eat
everyone you meet
All that you slight
everyone you fight
All that is now
All that is gone
All that's to come
And everything under the sun is in tune
But the sun is eclipsed by the moon.*

Under the sun, everything is in tune. True and false are clearly separated. At night we allow light and shadow to dance with each other. A perfect metaphor for the play of nighttime is the campfire. We gather around it, hypnotized by the flames. We cook, we eat, we sing songs. Humans are the only creatures who cook their food with fire. Fire is a central metaphor for human illumination. We use it like no other animal does. We beat back the darkness. Seated around a fire, people can see each others' faces and hear their voices. All around them, the darkness looms. And what do people typically do in such situations? Tell stories. What are stories? Shadow-puppetry. Speculations about life and death. Fantasies about what lies on the other side of that hill. Or ocean. Or galaxy. Imagination is the light that can lead us beyond the campfire.

It is tempting to think of our stories and myths as lies we tell ourselves to comfort and entertain. From this serious-minded perspective, the word “truth” should simply not apply to fiction. Truth comes from the non-fiction prose of daylight, and not the poetry of moon, star and fire. Against this boring scientism, the dreamers and poets periodically revolt. “All tribal myths are true, for a given value of 'true'.” A poem can feature “imaginary gardens with real toads in them“. If sunlight is objectivity, metanarrative and ideology, then moonlight is subjectivity, fairy tale, and ritual.

There is an alchemy in the blurring of light and shade that happens at night. The available light of day makes it seem as if the horizons of possibility are fixed. The science of day is Thomas Kuhn's “normal science“: conservative, incremental, firmly anchored in a paradigm. The science of night is Kuhn's revolutionary science: progressive, saltatory, unmoored. It is no surprise that major scientific breakthroughs can happen in dreams. August Kekulé came up with the idea for the benzene ring after he dreamed of an ouroboros — a snake eating its own tail.

A totalitarian worldview is one that considers available light as the only light there is. (Perhaps it is not a coincidence that this worldview often leads to nighttime curfews and restrictions on parties.) From this way of thinking we get austere monotheism, boilerplate Marxism, free market fundamentalism, and the various inanities of evolutionary psychology and selfish-gene biology. The opposing worldview isn't really a worldview at all. It wallows in murk and revels in fragmentary vision. It manifests in various forms of obscurantism: self-ironizing humanities-speak, incoherent conspiracy theories, New Age frippery, and superficial religious syncretism. But the two ways of seeing can each serve a purpose. The desire for a single overarching “source of illumination” is what gives us our most intricate “theories of everything”, as well as our conceptions of universal human rights. A suspicion of the Great Light Bulb in the Sky is what informs much art and music, and also countercultural resistance.

In attempting to fold so many ideas into one compound metaphor, I may be somewhat guilty of a totalizing worldview myself. But I can't resist adding more samples to my metaphor mash-up. The philosopher Isaiah Berlin wrote a popular essay called The Hedgehog and the Fox, the title of which was inspired by a fragment of ancient Greek poetry: “a fox knows many things, but a hedgehog one important thing”. According to Berlin, you can divide thinkers into two groups: hedgehogs who view the world through a single lens (such as Plato, Hegel, Dostoevsky, and Nietzsche) and foxes who view the world through many lenses (such as Aristotle, Shakespeare, Goethe and Pushkin). Berlin's hedgehogs operate in the stark light of a singular idea, whereas his foxes prowl nimbly in the shadows.

Let us now return to the campfire. The mystic is the one singing the songs with quirky lyrics. She tells tall tales about her experiences wandering in the darkness. The historian looks back and forth between darkness and light, trying to imagine what happened during the day. The “normal” scientist relaxes, recharging for the real work, which only happens in daylight. The “revolutionary” scientist dreams up new schemes for how to make the normal scientist's job a little easier.

And this is why it's important to have a sense of place and time. All the people at the campsite need to gather wood for the night's fire, and food to cook. There is a time for blur and dither, and a time for sharp boundaries. The universe has conspired to make it somewhat easy for humans to decide when to do what. During the day, our literal and metaphorical lights are brightest. At night, our lights are more varied and dynamic.

Technology adds a whole new dimension to nighttime. Thanks to the artifice of lightbulbs, we conjure up an imitation of the day at night. Apparently this has real consequences for the quality of our sleep, and therefore for the quality of our waking hours too. Technology gives us the power to let night and day bleed into each other. Perhaps Pink Floyd only capture one half of the modern malaise. Eclipses disrupt the workday, and light pollution distorts the dreamtime. Perhaps this is why the tech brigade (the canaries in the coalmine of modernity) make workplaces that look like amusement parks, while at the same time using numbers to quantify everything from baseball to burritos.

Maybe this era of whimsical work and programmed play is just a transitional phase. “Available light” is always in a state of flux. During the twilight hours the two kinds of light meet each other. It makes sense that these periods are associated with magic. Magic is nothing if not transformation. Humanity may be in a civilizational twilight period, but we can't tell yet if technological magic is leading us towards a bright new day, or towards a psychotic all-night rave.

Let's hope our sources of illumination provide us with enough light to navigate through what is shaping up to be a dark and confusing century.

_____

Notes and marginalia

• I was inspired to write about the “available light” issue because of a neuroscience-related question I was asked on Quora: How do we become “smarter”? If this 3QD essay is on the twilight side, my Quora answer is more of a daylight piece.
• The blog Quote Investigator does an excellent job of collecting various forms of the streetlight effect story, complete with references. The cartoon at the top of this post comes from there.
• Running wild with metaphors is how I got this writing gig at 3QD. I've used the vision metaphor in an essay on science fiction — Science Fiction: The Shadows Cast by Modernity.
• The 'What does the Fox Say' image comes from YouTube. All other images come from Wikipedia.

_____

* The Dark Side of the Moon ends with someone saying

“There is no dark side of the moon really.
Matter of fact it's all dark.”

_____

I purchased Wendy Doniger's book The Hindus: An Alternative History long before the controversy that lead Penguin India to withdraw it. I confess I never finished reading it. There's a great deal of food for thought in it, but I found the writing style very irritating: it's glib and giddy, and prone to unnecessary digressions, jokes, and puns. (Using the term “Zen diagram” once to talk about a Venn diagram of religious practice is bad enough. Repeating it is perverse.) But my attitude towards the book really soured after finding a minor but embarrassing error. In a footnote on India's geography, Doniger claims that “most of India and all of Japan are in the Northern Hemisphere”. Most of India? How can an Indologist not know that all of India is in the Northern Hemisphere? The southernmost tip of the Indian subcontinent is around 900 kilometers north of the equator. All of Sri Lanka is north of the equator too, as are all the islands one might associate with India. Doniger (and her editors) clearly didn't have much available light to shed on their maps. Of course, geographical ignorance is no reason to ban or withdraw a book. Nor is causing offense to Hindu fundamentalists. I should point out that Doniger's heart seems to be in the right place, and she has a genuine affection for many aspects of Hinduism. Just not the aspects that the fundamentalists prefer.