Yes. We need to attribute revenue to experimentation. Here’s how: Part 2

  1. No, you can’t accurately attribute, nor forecast, revenue to experimentation. Here’s why.
  2. Yes. We need to attribute revenue to experimentation. Here’s how: Part 1
  3. Yes. We need to attribute revenue to experimentation. Here’s how: Part 2
  1. OKRs for focus and alignment
  2. Talk in ranges
  3. Using Degradation
  4. Program quality metrics

Expected loss

What about open discussions on the alternatives to experimentation? What is the cost of not experimenting?

OKRs for focus and alignment

Want to know one of the best decisions we ever made at FKA User Conversion (in my humble opinion)?

Talk in ranges

This seemed like a fairly obvious solution to the ‘problem’ when it originally reared its head. And it’s a tactic, after speaking with a lot of practitioners, that’s well used.

Using Degradation

We spoke in the previous article about not collating experiment attribution together because, “you’re adding fuzz on top of fuzz”.

  1. …when, we are conditioned to these variables, we become desensitised to them
  2. …meaning the impact of that stimulus is reduced.

Program quality metrics

To remove the focus of revenue attribution, we must re-frame the focus towards something more attributable. Something that defines performance. Something that showcases we’re moving in the right direction, at least.

  • Time to implement
  • Success rate
  • Good execution (mainly about the planned experiment duration and the adherence to that plan)
  • Good shipping (validates that the decision is in line with the shipping criteria.)

Summary

How should you attribute revenue to experimentation? In part 1 we spoke about ‘communication’. How to best communicate

  1. Set Standards and therefore expectations
  2. Define what accuracy is acceptable
  3. Tell Stories
  1. OKRs for focus and alignment
  2. Talk in ranges
  3. Using Degradation
  4. Program quality metrics

My final final summary about this “Experimentation Revenue Attribution Series”

I really hope this series has helped, at least, spark some debate and encourage thought. I too, struggle with this and so do many others. You’re not alone. The “theory” often doesn’t match the “practice” and I’m trying to balance the two; often slipping into practitioner and evangelical mode. Sorry about that.

  1. No, you can’t accurately attribute, nor forecast, revenue to experimentation. Here’s why.
  2. Yes. We need to attribute revenue to experimentation. Here’s how: Part 1
  3. Yes. We need to attribute revenue to experimentation. Here’s how: Part 2

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store