The Limits of User Research

This piece is part of an on-going series of blogposts from the frontlines of Startup Tunnel, a new incubator based in New Delhi. You might also want to check out dispatches one, two, three and four.

by Aditya Dev Sood

Akbar BirbalHaving commissioned a new suit of armor, the emperor Akbar was now in the process of inspecting it. Installed upon a stone mannequin in the armory workshop, black bell metal and brass accents gleamed back upon the badshah and his vazir Birbal. Fresh from recent campaigns, the emperor now said he wanted to be sure of the quality of protection it offered. And so he called for a lance, with which he reared back and then charged upon the mannequin. He was able, after a few tries, to pierce all the small slits of the helmet. He asked for a sword and tore apart the subtle slits between the body armor and the helmet. He asked for a mace and went at the now headless mannequin and cracked the chain metal links all around its torso. Even now that it had fallen upon the floor of the workshop, Akbar was still working out his PTSD on that prone suit of armor and the lifeless dummy within. When he was done, he looked up and declared it to be a lousy suit, practically the same as wearing nothing at all.

Perhaps you already know the end of this parable? Perhaps you have heard some other version of it? I'm not sure when I first encountered it, either at the back of an Amar Chitra Katha or else perhaps among a collection of stories from Iran. Either way, it has stuck in the mind, long awaiting the unraveling. There is something so shocking about seeing a new suit of armor being destroyed like that, something like a medieval crash test. One knows not what to make of what is going on, nor even how to respond to Akbar's judgement. Is a suit of armor really useless if it cannot survive many minutes of the untamed rage of a battle hardened king?

The badshah is about turn his fury onto his smith, when Birbal suggests that they give him a sharp warning and a week to build another prototype. The next week, when Akbar returns to the workshop to inspect the new piece he finds Birbal already there, wearing the emperor's battle armor and spoiling for sport. It is new and improved, he says, have at me and I'll show you. Akbar is eventually goaded into picking up a lance. He makes straight for Birbal, who steps lithely aside, pulls the lance forward, tripping Akbar forward and landing him on all four. Now that someone's wearing it, he grins, it's begun working pretty well.

On the face of it, this would seem to be a parable about how an artifact changes with use — an early instance of user-centered thinking about human artifacts. But there's something a bit tricky about how a suit of armor is best used and what its function really is. Birbal's response is cryptic, and it forces you to think about the whole the point of battle armor: it must not only resist onslaught, but allow its wearer to move about and conduct battle. This little fable sticks in the mind is because of the way it shifts between offence and defence, between object and agent. That little shift of the mind, between a closed and essentially reactive reference frame and a horizon of open possibilities is sudden and complete. It cannot arise gradually and it has no continuity with that earlier way of thinking.

You will remember, reader, that I've signed up to share a more prosaic kind of story, about the setting up of a new kind of business in a more prosaic time in a global city whose air is already thick with pollution and corruption and crony capitalism.

The moral of every tale will continue evolve over time as it is told and retold. What I now understand is that so long as one remains inert and unmoving, any preparation will be inadequate and one's defences liable to be demolished with a few quick jabs. But once one enters a field of action, as one bobs, weaves and improvises in many ways, one leaves preparation behind.

For many years I've been in the business of making armor, not actually of giving fight. I've built a consulting company that studies how people respond to new technologies, I've led teams that have imagined different kinds of solutions for clients, I've drafted and delivered strategic recommendations to company boards. For more than a decade, I've pursued user-centered design as a new kind of ethic and means to remake our technological world more perfectly in our own image, as individuals and as communities. If the world seems misshapen by the forces of capital and crudeness of technology, those forces and their forms could be remade through iterative design intervention. And this has also been one of the fundamental principles of our new incubation program: that when those same skills and methodological approaches were applied to startup challenges, they might yield substantially better and more successful startups.

Several interesting things happen when you move from advising large imperious organizations like Nokia or Vodafone to working with startups. First the creative budget collapses to very nearly zero. And then the time horizon in which one's research and learning must emerge and be applied also falls to something like zero. All the market frictions which needed to be resolved and mediated by a creative agency so as to earn its living all begin to fall to miniscule levels. The budget and time required for product development can approach zero when startup founders create product for end users very similar to themselves and when they themselves can bear the costs of product development, iteration and modification. Today, so many young people new spend their free time dreaming up an experience that they would enjoy and that they think others might enjoy as well.

Digital and network technologies, of course, are allowing and encouraging both these trends: while the cost of software product continues to fall lower and lower as this expertise becomes more widely available in society, young founders are following their own passions and preoccupations in the development of new product concepts. In effect, market forces are compelling them to do so, because the costs of figuring out what people might need or want if they are very different from you grow quickly prohibitive. There's a better chance of your work coming good if you code about things you already know something about.

It's quite interesting that the academic turn in favor of design research begins to emerge at almost the same moment as its refutation by startup founders. Writing in 1982, for instance, the design theorist Nigel Cross more-or-less assumes that design research is more or less the future of this discipline, but by 1985 Steve Jobs is already dissing his PC competition for having been designed by focus-group. His own Macintosh devices, by comparison, he says, are designed for the very people who made them. This makes them intuitive and superior in every way because how can people really know what they want from a computer experience when they've never seen a computer? This brash insight, which has always chafed during the many years I've worked on user research projects, was later articulated with some greater sophistication by Eric von Hippel. Lead users, he said, are members of a user population who are currently experiencing needs that will later be experienced by many other users, and who anticipate relatively high benefits from obtaining solutions for their needs. One may read back into Jobs comments the idea that Macintosh devices were made by and for lead users, who knew a great deal about what they themselves wanted out of a new computer platform, and whose needs would prove predictive of the marketplace as a whole.

With startups, moreover, there is no possibility of cross-subsidizing new product development with margins from existing product lines — there are no existing product lines. And this is why Ben Horowitz advocates shipping a lousy product even when you know it's lousy, so long as you can get someone to buy or pay for it. Shipping product ensures you live to fight another day and gives you a chance to improve it. Without that channel to the market, all product debates are rendered moot, abstract and academic. This is obviously anathema to any designer or perhaps any ethically minded person at all. But it is the difficult and bitter truth of people struggling to engage a market.

A product concept must be developed and defended, not only as an abstract idea, but also as a living, struggling, market proposition. It's useful to acquire all kinds of information about one's users by studying them and looking over their shoulder, but all of that is nothing compared to the information one acquires about them through one's own digital products, once they begin to adopt and use them. Over time one way of understanding your users may complement the other, but if you never get anyone to use anything you've made, you're nowhere and there's nothing really to talk about.

There's one final change of reference frame that might be in order, from building armor to waving one's arms around in it, and that has to do with the role of user research and design thinking in relation to the long-term goals and strategic objectives of a startups. The ethic of design research and user experience mapping is inherently reactive, operating on faith that this approach will always ensure that one lucks into the most beneficial market alignment without actually forcing that convexity into existence. Through the last decade my own ideal of a tech company was Nokia, a global organization that continuously valorized users, expressed unflagging interest in all the minute details of user behavior and promised always to try to build human technologies, none of which prevented that company from turning into a giant flaming rig. Peter Thiel's advice is rather to always look to the end game first, and decide what kind of market needs will a company fulfill going forward. A stable solution to a large market problem can define the contours and possibility of a product concept. But can user research identify such an unmet challenge? Not directly, in any case. It can only help define and characterize such a problem once it has been already identified through market and value chain analysis.

We still believe we're right to build suits of armor and teach our founders to learn to stalk about in them. We're also humbled to realize that it won't nearly be enough. They will have to rely on their wits and reflexes to survive and succeed with their startup.

Startup tunnel logo