Why do we assume experimentation is about financial gain?

  1. No, you can’t accurately attribute, nor forecast, revenue to experimentation. Here’s why.
  2. Yes. We need to attribute revenue to experimentation. Here’s how: Part 1
  3. Yes. We need to attribute revenue to experimentation. Here’s how: Part 2

Why do we think like this?

Why? Why is it so common for people to think that we can accurately attribute a revenue gain to AB tests or changes we make on site. Before the why, comes the how. How did this phenomena come about?

AB testing is over-simplified

It should be to be more widely accessible.

* those are real examples - I was going to link to them, but thought I'd be thoroughly told off by outing such examples

When did we start attributing financial gain?

Optimizely might be one of the first.

And where are we now?

The popularity of and exposure of AB testing has led to a viral effect. If one company saw a $300m uplift from the changing of the wording of a button, then can these effects be replicated on a different website or platform?

Am I being overly precious?

Sometimes I admit, I easily slip into practitioner, evangelical mode. It’s easy to do so. To fly the flag for those experimenting, rather than truly understanding how a single methodology fits into a wider business purpose, or accepting the maturity of a practice into a commercial field. That being said, I will never accept a ESL (European Super League) bid. We must be emotionally mature enough to appreciate how the piece fits into the jigsaw.

Summary

It is difficult, if not impossible, to provide an accurate ROI for experimentation. The over-simplification and virality of click-bait headlines for case studies designed for commercial gain and a wider audience has led to associating financial gain to an output.

  1. because of the preconception that an experiment is directly linked to revenue gain
  1. No, you can’t accurately attribute, nor forecast, revenue to experimentation. Here’s why.
  2. Yes. We need to attribute revenue to experimentation. Here’s how: Part 1
  3. Yes. We need to attribute revenue to experimentation. Here’s how: Part 2

Don’t miss my spurious thoughts on optimisation

I currently work just 2 days a week (and thoroughly enjoying it). But for the other 3 days, in a bid to keep me off Twitter and reading about the bull markets of Cryptocurrency, I learn, write and share. For nothing more than creating debate and enjoyment 😎

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
David Mannheim

David Mannheim

Stories and advice within the world of conversion rate optimisation. Founder @ User Conversion. Global VP of CRO @ Brainlabs. Experimenting with 2 x children