Does experimentation encourage innovation through validation?

“Experimentation-driven innovation will kill intuition and judgement”​. Stefan Thomke

This is the first of seven myths Stefan Thomke writes about in his book, Experimentation Works.

In my last article, I argued that experimenting could do the exact opposite. That it restricts creativity and innovation just as much as it fosters it. Why? Well, I think there can be inherent bias within validation itself, given that users might not know any better or be ready for said innovation. It could be a timing thing or a “I don’t know what I don’t know”​ thing. Give it a read.

Most disagreed. Although, I do wonder what would happen if my Linkedin connections were within fields of creativity rather than fields of experimentation. And yes I know experimentation is arguably a creative field but I not here to satisfy trolls.

These guys aren’t wrong.

Experimentation fully supports innovation as a method for validation and exploration. It does not create it. I do truly believe that. And when I was writing this, I thought to myself “is this even worth writing about? People already know this. That much is clear from the comments of my last post alone”​.

But I believe in it. As much as I believe that it can, on occasion, inhibit innovation. There’s a place for intuition and a place for experimentation. What is that place?

To make it interesting, I’ve tried to back it up with a tonne of sexy examples from the commercial, economic and scientific world.

Intro.

“Experimentation supports innovation, it doesn’t create it”​. I just said that. But is it that simple to just say?

I think the complexities of this topic are within the semantics themselves; creativity and innovation. The economist, David Galenson, spent years studying this topic and came up with two distinct personas and definitions of innovators.

The first being conceptual — think Mozart or Galilei — where bold new ideas are pursued born out of genius or prodigy. The second being experimental — think Jobs or Wright Brothers — using a trial-and-error approach to innovate and create.

Interestingly, Galenson is known for saying that the latter approach of ‘experimental’​ is far more “interesting”​ and “common”​.

And there are hundreds of advocates and examples of where experimentation leads to innovation. Louis Pasteur’s discovery of artificial vaccines came by experimenting with injecting hens with mild doses of cholera. 3Ms discovery of Post-It notes came from a series of experiments developing a polymer-based adhesive.

“I had sat down and factored it out beforehand, I wouldn’t have done the experiment. If I had limited my thinking only to what the literature said, I would have stopped”​ Spencer Silver [ref the Post-It note]

But why and how does experimentation foster innovation?

Failing.

Failure breeds learnings. And learnings breed knowledge. And knowledge breeds innovation.

It’s important. In an experimentation context everyone states “there’s no such thing as a losing test”​. Yes yes, I know. You can’t get to adulthood without going through awkward teenage puberty.

And yes, there have been so many examples of note where innovation derived from failure. The lightbulb had 9,000 experiments before it worked. Slack was born out of a failed online multiplayer game called Tiny Speck. Amazon Marketplace (which now accounts for 40% of revenue by the way) was born from the Z Shops product which was born from the failed Auctions product. WD40 was born from a series of experiments — 40 of them to be exact — fancy that. Cordless vacuum cleaners from Dyson took 5,271 prototypes. Sounds like a competition, doesn’t it? Who can fail the most?

The concept of failure is a complex one. And I could cite something like the below from Andrew Stanton — which is beautiful both because it’s Pixar and authentic — but it’s not rooted in commercial environments where, I believe, stakeholders naturally fear failure. For reasons that are ego-driven. For reasons that they are human and therefore naturally avoid catastrophe. But also for reasons that commercial environments pulsate the economy and need for money is perpetual (*cough or greed).

So whereas ‘failing’​ is often cited as an aspirational state of mind, in a commercial environment, from experience, I often find it rarely is true.

“My strategy has always been: be wrong as fast as we can. We’re gonna screw up, let’s just admit that. But let’s do it as fast as we can so we can get to the answer.”​ Andrew Stanton

But failure is needed because to understand whether you’ve failed, you must validate and measure; and to do that you must experiment. So surely, innovation ‘from failure’​ only ever stems from an experiment in understanding, not necessarily failure itself. You must be willing to be wrong on your way to being right (Godin)

“If I find ten thousand ways something wont work, I haven’t failed. Every wrong attempt discarded is just one more step forward”​ Thomas Edison

Practicality.

Experiments are quicker. They get shit done. Galenson siad it himself, thinking the experimental innovator is much more common because of this level of practicality.

I got given a great article the other day (h/t to Ben Labay, which came from Simon Elsworth, which I think came from Matt Lacey), where Michael Schrage argues that a testable idea is better than a good idea. He mostly talks about the practicality of testable solutions; they’re measurable, action-orientated, deliverable, quick.

Where non-testable ideas, those moments of “I had a good idea”​, are discussed, debated, passive and placid. We beat around the bush and over-polish. Seth Godin talks about good work never seeing the light of day because finance professionals are paralysed by their fear of getting something “wrong”, herein we come back to failure.

“Testable hypotheses seemed a faster and, frankly, better gateway to innovative action and active innovation”​ Michael Schrage

Testable ideas are better both operationally and emotionally.

Operationally because of the aforementioned. Emotionally they’re better because they remove that subjectivity, pride and ego that comes with the next big idea. “Business ideaholics, not unlike meth addicts and other junkies, are always looking for the next fix.”​ say Schrage. Here we are talking about ego again. And you’re going for the jugular there, Michael. But he’s basically saying that there is a serious temptation to fall in love with the process of generating the idea itself. We’re human beings. Naturally creative; it’s what sets us apart from the primates. To proliferate these ideas endlessly and forget that something needs to be done with the ideas is all too common.

Those big ideas, too, are impractical and are often a result of what Jeff Bezos calls, a “hail-mary”​ event; a desperate attempt to save a companies corporate existence. Blockbuster. Blackberry. Nokia. The contrary, and more popularised approach of what we’re discussing today, is smaller ideas or, what Peter Sims calls in his book, Small Bets.

Validation.

And here’s the big one. Experimentation validates ideas. The illusion of rationality is a known phenomenon when ideas or assumptions seem logical, but they haven’t been validated in the real world.

Chris Rock didn’t just win 3 x Grammys for “best comedy album”​ by being ‘naturally’​ funny. The albums are tested hundreds, if not thousands, of times at smaller gigs. In a 45 minute set, Chris will ramble, lose his train of thought, and only sometimes stumble on something where he’s audibly heard as saying “this needs to be fleshed out more if it’s going to make it”​

Amazon, similarly, are known for their innovative approaches to delivery or video with Prime subscriptions or hosting with AWS. Jeff Bezos often talks about how they go down “blind alleys”​ But, he goes on to state “every once in a while, you go down an alley and it opens up this huge, broad avenue”​

Both Chris and Jeff accept that you can’t rely on intuition what works and what doesn’t work. You must experiment.

“If you follow your intuition, you will more often than not err by misclassifying a random event as systematic. We are too willing to reject the belief that much of what we see in life is random”​ Daniel Kahneman

Intuition can mislead or misrepresent.

Mislead. Intuition holds no evidence behind a claim. “I feel it in my gut”​ is as risky as putting it all on 17 Red. In fact, research suggests that our predictions of how customers behave are wrong 9 out of 10 times (Thomke and Beyersdorfer, Booking.com: Harvard Business School Case).

Going back to our mate Jeff, Amazon once launched a feature where in one click, Amazon would show you what other customers purchased allowing you to find customers with similar order histories. No one used it. And of course, there are thousands of examples of this too, aren’t there? Where people have created stuff based on pure intuition. I think it’s what the majority of the popularised entrepreneurs still do when pitching to the scary Dragons on BBC One. Pride can get in the way.

“Our history is full of things like that. Where we came up with an innovation that we thought was really cool and the customers didn’t care.”​ Jeff Bezos

Mis-represent. Intuition can mis-represent because of the relationship between A and B. If intuition is telling you A because of B, what happens if the relationship between A and B is purely coincidental. Like how eating more ice cream increases your chances of drowning. Or forrest fires. Or polio. In other words, there’s a difference between correlation and causation, and pure intuition poses a risk of the former. In simple terms, when your intuition is wrong.

Data, especially through experimentation, can verify correlation. Yet, and this should be in the section above, causal questions can never be answered from past data alone (Pearl and McKenzie, The Book of Why: The New Science of Cause and Effect, 2018). So does experimentation provide the ability to truly innovate?

Continually learning.

Maybe this is the big one actually.

“Experimentation matters because it fuels the discovery and creation of knowledge”​ Stefan Thomke

Stefan wrote that as the very first sentence in his book, Experimentation Matters in 2003.

Stefan was talking both about the exploration of experiments, as well as the exploitation. It’s not just the validation but the why. Analysed right, iterated on with further hypotheses, experimentation can be the most effective and efficient knowledge source in your organisation.

Over 75% of innovations come from your active users, a study by the economist Eric von Hippel found. So ask them and find out. Demonstrate behavioural and attitudinal shifts with them, practically, through variable changes. It’s how films tinker with their story line. Or how P&G prototype quick and dirty ideas for new products. And now we’re just talking about how good user research is, so I’m going to stop.

Whilst Steve Jobs stated that “no one wants to rent their music”​ (cue Spotify), he also stated that “people don’t know what they want until they’ve seen it”​. Thus, in a paradoxical conversation with himself proving, that practical feedback — be that through an experiment or not — does indeed support innovation, but only through learning itself.

We’ve (FKW User Conversion) drastically altered how staff speak to customers and package products for one client before — by testing something online. Not because the the experiment ‘won’​, but because of what we learned from the attitudes of those within the variant group.

That won vs loss thing, by the way, is such a red herring. It places experimentation as this binary method of being. When it’s a method for further knowledge as much as validation.

For my final semi-tenuous example, did you know Beethoven had hundreds of manuscripts with annotations and notes that clearly demonstrate his experimental approach? They highlighted a clear evolution of his musical style from, arguably, the same as everyone else — especially Mozart — to a highly distinctive and innovative style; so much so it ushered in a new era of classical romantic music. Just like Chris Rock (which I never thought I’d be comparing the two) both took a process of testing their art, understanding what did and didn’t resonate along the way, leading them to new heights.

See. Experimentation gives us knowledge we didn’t have before which allows us to create and innovate.

Summary

Here’s something which I bet you didn’t know (or perhaps care). I consider creativity to be one of my core values*.

Experimentation is also process I really genuinely believe in.

These two things are beautiful in their own right. And when you put them together? Love and music, baby. I do believe the two work together and I even advocate this on my website. “I believe creating a culture of experimentation is an enabler to learn and iterate”​. I wrote that months ago. So, nur nur nur nur.

But, I do feel there is still a place for intuition. It might mislead and misrepresent; ut it creates. If we’re going for moonshot innovation, theres going to be a point where you’ve just got to take a leap. And I’m going to talk about that in “Experimenting for iteration and experimenting for innovation”​

Having a process of experimentation, at least in some form, will support that leap though validation and extra learnings.; it just might not create the leap itself.

What’s next?

In my next series of articles, I am going to dive a bit deeper into questions like:

• Does experimentation inhibit or encourage innovation?
• Experimentation validates creativity ← that’s this one, spoilers!
• Can a hypotheses limit creativity?
• Experimenting for iteration and experimenting for innovation
• How can we be more creative within our process of experimentation?

*if you’re interested / bored / nosey in the other 4, at least for me, it’s autonomy (self-determined and independent), industry (to work hard and well in life), hope (maintain a positive outlook) and courtesy (to be considerate and polite). I did a great exercise with my wife not too long ago on our values because we are just that cool. DM me if you fancy the exercise.