Experimenting for versioning, visioning and venturing forms of innovation

I’ve really been torn recently. And found it hard to articulate my paradox.

On one hand, I discussed in my last article, I believe that AB testing encourages innovation. It can create a fountain of knowledge and lead us on a path of discovery. Perhaps that path is not the perceived intention, but that’s what creates ‘the original’.

In my musings before that though, I mentioned how experimentation held limited value for innovation. I just don’t understand how we can test and learn about user behaviours, when the users themselves are not ready for such an innovation? 🤷🏼

Sit on the fence, David.

I’ve been thinking about where this clashing of two minds is coming from.

I think it’s two fold.

1. Innovation holds many different categories and is more a spectrum, rather than a binary definition
2. Our societal perception of what experimentation is designed to do is to tweak, iterate, move stuff round on a page — rather than truly innovate

I think innovation needs a level of categorisation to understand whether experimentation is better for some categories over others. So we’ll explore that.

But I also think, that within those categories, especially in ecommerce, experimentation is pigeon-holed. It’s poster-boy’d and seen as incremental ‘improvements’, rather than innovative leaps forward. Stuck in a world of moving images from the left to the right, or making fonts bigger or small. In this world, cyclical case studies create a perception of experimenting for the tweaking and versioning, rather than the bold ‘go where no person has gone before’ venturing.

WTF is innovation

Innovation is many things; not one thing.

I’ve previously explained how the economist David Galenson came up with two different types of innovators; the conceptual and the experimental. The conceptual are the strokes of genius you come across once in a century. Where the experimental, the more common of the two, is the type of innovation that occurs over time from learning through trial-and-error. Sounds familiar doesn’t it, experimenters?

There are two very different types of artists: those I call Old Masters, who work by trial and error and tend to improve with age, and conceptual people, or Young Geniuses, who generally do their best work early in their careers.

I began to wonder if we can expand on Galenson’s categorisation of innovation. And whether experimentation plays a bigger role in one type of innovation more than others.

Have you ever seen Kevin Dame’s, Head of Product at Youtube, talks about versioning, visioning and venturing? It’s fab, watch it.

I wonder if, and how, experimentation has a place within each of these categories? Does it hold more value in versioning than venturing?

I think we think it does because of how the industry, and society, treat experimentation.

Versioning

Experimentation is complex. So to make it more accessible (and to sell their services!), those companies that advocate it — the agencies, the SaaS platforms, the communities — create case studies that resonate by making it seem….simple.

That’s needed, I get it. But in doing so, over-simplifying this methodology has led us to believe that the purpose of experimentation is one of moving shit round on a page to increase a metric. Usually a conversion rate. Hence the term “conversion rate optimisation”.

Enter versioning. The notion of incremental improvements to a behaviour or aesthetic. The Kindle is now on its seventh generation. Samsung Galaxy on its 21st. iPhone on its 13th.

There’s nothing wrong with this, but I think that we think experimentation holds the greatest place here; within versioning. I reiterate, we think it thrives here. Because of how case studies, communities, and success stories talk about it.

This type of trial-and-error approach of validation is exactly what David Galenson was talking about when he was advocating the experimental innovator. That’s fine, but there are other types of innovation of which experimentation are suited to; I just feel this is the most popular because it is, by virtue, the most accessible.

And, for reference, versioning is a necessity. And experimentation to support that is also a necessity.

Take Netflix. They have been working on the hypothesis of personalisation since 2000. They introduced Cinematch (a recommendation system where member ratings were used to predict how much a member would like a movie). But their evolution of the same hypothesis has seen multiple iterations, in fact, from (to name a few)

• 2006 — Ratings Wizard with a “Movies You’ll Heart” tab
• 2009 — collaborative filtering algorithms focussed on popularity
• 2009 — category match personalisation (because you watched X, you’ll like Y)
• 2011 — AB tested personalisation vs no personalisation (the former won)
• 2012 — Multiple profiles for individual personalised recommendations
• 2016 — floating rows for a personalised UI
• 2018 — personalised movie art

Netflix innovated by experimenting by versioning their recommendations system. Some failed, some became anachronistic. But they did it.

Ironically, I find when it comes to ecommerce, rarely do they version with their experiments. Experimentation is seen as versioning, but, in my experience, it stops at version 1.1, and rarely moves beyond that.

Despite community companies distorting our lens towards “this is how experimentation should be”, I often find that when we, as optimisers, experiment and reach a conclusion, we give it a binary “win” or a “lose” label.

This isn’t innovation.

Taking that concept, we’ve worked with Flannels before and gave a talk on the power of iteration with AB Tasty previously. We discussed how some of our experiments were only really classified as ‘successful’ once they were iterated on because we learned from the behaviours of the previous incarnation. What we really should have spoken about is how they pushed our creative thinking. Look at the below — I’ve personally never seen filters within a product card designed to encourage filter usage before. That’s versioning. That’s iteration. That’s innovation. And all thanks to validating behaviours, and learning from them, within a series of experiments.

Visioning

Visioning is moving beyond the 1% improvements for how we see our product in 2–3 years. What’s a leap in value we can create for our users?

I think there’s a different process required for visioning, then there is for versioning. And I wonder whether experimentation holds more of a common place in the former than the latter. Or, at least, again our perception is that experimentation works better in the former rather than the latter.

In versioning, we ask “what’s the quickest way we can get this out of the door? What’s an MVP to prove our hypothesis? What’s feasible? How can we seek out a 1% improvement?” These are experimentation mindsets.

Especially when we prioritise based on “time” (or effort) as an indicator, often the quickest thing to engineer is prioritised first and, adversely, that which takes the longest is prioritised last. So will we ever develop visioning ideas when our experimentation prioritisation frameworks are so biased against time?

Visioning requires someone to slow down. Take a leap for 10% improvements. And often, think more about the problem, rather than the solution. Sometimes at the cost of speed of production within versioning, comes a negligence in “am I really tackling the problem here, or am I just putting a plaster over it?”

Take the below example. YouTube creators wanted to promote donations to social causes they cared about. YouTube took a versioning mindset and quickly added information within the “i” icon that came with every video. Convergent thinking. This satisfied the hypothesis and creators saw an uplift in donations to their causes.

It wasn’t until a visioning mindset was taken where that feature really became its own and the user experience excelled. This was as a result of divergent thinking where they got a tonne of ideas together, honed in on the one they thought was best, validated that, and went with it.

The result was a 30x uplift in donations.

Experimentation is absolutely perfect for such a situation, because it validates an understanding. That understanding might be assumed or seen as reaching for the stars, but the methodology helps you understand it’s plausibility.

Venturing

Venturing are the invention breakthroughs. The moonshot projects. Those that take 5–10 years to complete.

When I wrote “does experimentation inhibit or encourage innovation”, this is the type of innovation I was hinting at. Can we test ideas on our users when the users don’t know any different? Or when they aren’t ready? Where the innovation is so new the reticence to change outweighs or dilutes the potential of the idea. Think a “if I asked people what they wanted they would have said faster horses” moment.

I could argue experimentation doesn’t work as well here. But I would be wrong.

The benefits of validation (are we heading in the right direction) and discovery (learning from our experiments) is too great that, in my opinion, it outweighs the “I don’t know what I don’t know” argument. Again, similar to the visioning debate, experimentation is extremely powerful here for the purposes of discovery and validation; arguably more the former than the latter.

Rowing harder doesn’t help if the board is headed in the wrong direction. Kenichi Ohmae

All experimentation does is show us which direction to row in.

How experimentation is perceived amongst the many

Experimentation works across all these categories of innovation.

I want to say that experimentation thrives more in versioning. But whilst I want to say it, I don’t think that’s true. I think I’m forced to say it because of the societal perception of experimentation.

The real power of experimentation is one of validation and discovery.

But, from personal experience, in 90% + cases of how I see experimentation being used, it primary use is to change web design layouts, to tweak the user experience onsite or worse still, for marketing campaigns to bypass engineering teams.

Perhaps that’s because I live more in an ecommerce world. In this world, AB testing largely sits within a marketing function, rather than an engineering function.

Yet because of this bias towards small improvements experimentation is seen as better for versioning because it’s stuck there; for most companies. Only until we really understand what AB testing can give us can we use it for the purpose it was intended; for innovative leaps forward.

“I’d spend 55 minutes defining the problem and 5 minutes solving it” Albert Einstein

When the majority of experiment value is stuck within a world of versioning, the purpose of experimentation is adversely believed to be one of a solution. Really, the main value of experimentation comes from helping to define the problem itself.

In this respect, regardless of the innovation categorisation, experimenting is vital because it helps define the problem through validation and discovery.

What’s next?

In my next series of articles, I am going to dive a bit deeper into questions like:

• Does experimentation inhibit or encourage innovation?
• Experimentation validates creativity
• Can a hypotheses limit creativity?
• Experimenting for iteration and experimenting for innovation ← that’s this one, spoilers!
• How can we be more creative within our process of experimentation?