The illusion of data as “objective” conceals that it rarely shows you the whole picture. Making decisions based on the easiest data to gather is a recipe for disaster.
Last week, the former CMO of Nike published a rare deep dive into a marketing blunder four years in the making. As a firm believer in taking inspiration from analogous domains, I think there is a great lesson here for Product and UX to take away.
Why do I think it’s relevant? Just tell me this doesn’t sound familiar:
“Nike invested billions into something that was less effective but easier to be measured vs something that was more effective but less easy to be measured.”
On the advice of McKinsey, Nike’s new CEO John Donahue decided to pivot to a “data-driven” approach, reorganizing the company towards digital direct-to-consumer sales and eliminating the former model centered on distinct categories. The allure is easy to recognize, and it’s the same trap that Boeing and other companies fell into over the preceding years.
Coming up with new ideas is difficult and requires specialist knowledge. Moreover, it requires specialist knowledge to understand what those specialists are doing and therefore manage them.
Meanwhile, slashing costs works the same way across every industry. And so Nike squared up to eliminate duplicate processes, streamline operations, improve efficiency, and increase productivity — all those phrases that are meant to say “whatever it is you do — do it harder.”
This kind of strategy was already a punchline in 1997 (Bellwether, Connie Willis)
How well did it work? Well, that depends on what the intent behind the strategy was. If Donahue intended to lose $25B in market cap and tank the stock price 32%, it was a smashing success. For those of us who don’t think that’s what he meant to do, it bears reflecting on where he went wrong.
The hard limits of data
It is wrong to suppose that if you can’t measure it, you can’t manage it — a costly myth. — W. Edwards Deming
Data isn’t worthless. Data is really, really valuable for telling you what has happened in the past. Great expense has gone into producing data that can tell you what’s going on in the present. But as the 7- and 8-figure salaries of quantitative analysts at hedge funds show us, using data to extrapolate what will happen in the future is one of the most challenging things you can try to do with it.
Typically, the way one would do that is by harvesting what’s called warm data — the qualitative data that gives the numbers their meaning — and then using that to tell a story about where the numbers are going.
Unfortunately, that’s not what online advertising does. As Phil Bastien calls out here the most common outcome of advertising data is that if you buy a couch, it must mean that you need more couches.
Phil’s assertion is easy to validate — just turn off Adblock and see if your experience matches that of the 1100 people who agree with user brain.curiosities on Threads
Human marketers and merchandisers know this — it’s why you might see paper towels next to the BBQ sauce or buns and hotdogs together at the store. Humans could enrich “you are buying bread” with the qualitative data point “toppings go on bread” to create a delightful shopping experience. But getting data — even data fed to an AI — to do this is much harder than you’d think.
Nike’s decision to eliminate individual product categories — who could marshal exactly this kind of expertise — in favor of a generic one-size-fits-all data model created a problem that you can probably predict. This model performed the equivalent of handing out flyers for your pizza shop in the pizza shop’s lobby and signaled the business to reorganize from attracting new customers to pumping more money out of the customers they already had.
The data-driven death spiral
Measuring the wrong things is a fast path to disaster. — Jared Spool
One might propose that the entire profit motive behind the genre of creative work made up by brand, marketing, and design is to drive behavior change. We enable customers to do something new — whether by informing them that an option exists, or making it more attractive and effective to choose that option over others.
And this is where the trouble starts. Nike’s incestuous strategy was bringing in data on existing customers — those whose behavior didn’t need changing.
This is an extremely common mistake in all kinds of industries — the most vocal, most frequent, loudest participants in your research are the least representative of the total addressable market. The people who don’t — or can’t — use your product barely show up on the radar.
It’s tempting to reach for the data that’s easy to harvest and satisfy yourself with the online shopping data. That’s exactly what Nike did. As a result, their product priorities rapidly diverged from the kind of things ordinary customers buy, while product with mass appeal rotted in warehouses for lack of places to sell it. And the longer Nike chased these fringe customers, the more ordinary shoppers moved on to a competitor’s product.
Letting data drive you is fixed-mindset thinking
“A fetish has been made of quantitative methods and whole laboratories have been devoted to solving, with elaborate statistical machinery, problems which had very slight importance.” — Gregory Bateson, 1944
With the benefit of time and distance, it’s easy to condemn Nike for making this mistake. It’s even easier to believe that we can avoid it — because we would simply pick the right data to follow, instead of the wrong data.
Arrested Development: The One Where Michael Leaves
The fact of the matter is that this mistake was made by experienced, intelligent professionals who also thought that they were doing the right thing. It’s not enough to identify that they were wrong; we need to understand why in that moment, the wrong thing seemed like the best way forward.
And it’s easy to see how it would be. Nike is still one of the world’s most valuable brands, and chasing whales has worked well for many lesser firms. In an environment as risk-laden as 2020 was, it feels good to lean into your strengths and get a practically-guaranteed 10% (or whatever) bump in your metrics.
It’s also very tempting to close your eyes and pretend that the 10% isn’t coming at the cost of larger losses elsewhere, because if you speak up it might be your job that’s on the line in the next round of layoffs. It takes a lot of courage and conviction to push back against the dominant narrative. It takes research skills to form an informed hypothesis and business sense to convince people with budgets that it ought to be tried.
“Validating” what everyone already believes is far easier — but it adds no value. The value of research doesn’t come from elevating people who are already shouting. It comes from finding the people who are not being heard, and adding their voices to the conversation.
Then you can make a real data-driven decision: a decision driven by all the data.
Otherwise, the best you can do is commodity-grade decision-making.
Nike’s $25B blunder shows us the limits of “data-driven” was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.
