Experimenting Everywhere

Experimentation is a methodology to help us understand better and faster. It’s really not that difficult.

In fact it’s become over-simplified with non-impactful UI tweaks. Or best practice UX that, often can be likened to, moving deck chairs on the Titanic. 🛳

The better comes in the means of measured tests designed to understand how users actually behave, not how we interpret or they say they behave.

People have told us over and over again. They don’t want to rent their music. They don’t want subscriptions. (Steve Jobs, 2003)

The faster comes in the form of learning faster and reduced risk. What is it that Mark Zuckerberg said? “Move fast and break things”. Something like that.

There’s no question that experimentation works. The paradoxical fortunate and unfortunate scenario we find ourselves in is that experimentation is inherently linked to the web – or more specifically web design. Thank you, Optimizely and Barack Obama. Both a blessing and a curse. It truly is a blessing within the optimisation community. Being able to practically understand user behaviour, to craft and continually iterate on experiences rooted in data. Sublime and exciting. 🎆

But somewhere along the regime of bad best practices and testing 41 shades of blue for the best link or button, the curse is that we have lost our way.

I unfortunately believe we have subconsciously trodden down a garden path and forgotten what experimentation is designed to achieve. Instead, we have become accustom to testing the minutia of web design layouts for the pursuit of…acknowledgement? “We’re doing it” Or perhaps the belief of attributable, unattributable revenue? “This single test generated £200m in revenue”. Let’s not go down that cynical rabbit hole.

If it’s so good, why doesn’t everyone use it in their decision making?

Experimentation is the pursuit of knowledge about cause and effect … what does and does not work (Stefan Thomke, 2019)

The antithesis of experimentation is intuition. And there’s nothing wrong with intuition. Some, including myself, would argue intuition is the greatest driver of creativity. But when it comes to decision making you are trying to obtain the best outcome avoiding shortcomings or risk.🙀

Experimentation gives you practical data and insight to make that decision, we all know that, right?

But here’s why I think it’s not used as a methodology as often as it should be. We misconstrue experimentation as another process or layer in the decision making. It’s sloooooooow. It’s not. Actually, it’s completely the opposite of slow. 🏃‍♀️

Think about your last business concept or large-scale decision, how long did that take to come to fruition? We recently introduced a sprint system over at User Conversion, a process that took 9 months and is still ongoing. I know, I know. I know of companies that take 12 months+ to hire. Or have heard of retail outfits that take years to select a ‘perfect’ store location.

Here’s the thing about decision making. And who better to cite than Barack Obama?

…being comfortable with the fact that you’re not going to get [a] 100% solution, and understanding that you’re dealing with probabilities, so that you don’t get paralyzed trying to think that you’re going to actually solve this perfectly. (Barack Obama)

He goes on to state that being just 51% confident is enough and that, along the way, your approach can be moulded or, to coin a term that we optimisers use all the time, iterated on.

That is why experimentation is so powerful. It can give you the ability to quickly iterate, mould and change course. In essence, reducing risk. 🗡️

Experimentation has been pigeon-holed into technology, specifically UI treatments on websites

One of the reasons why experimentation has become so prolific in recent years is the advent of technology. 📟

I heard a story once that the ability to manipulate DOM elements was created in the 2008 recession. This was vital to allow for what-you-see-is-what-you-get-editors (or WYSIWYG) allowing users to play about with manipulating elements on their own accord. The timing is interesting though because, seemingly, in a recessive society where people unfortunately lose their jobs, innovation is stretched, forced and heightened, right? Microsoft, WhatsApp, Groupon, AirBnb – these were all created within a recession. I’m moving beyond a point – And I’m also not sure how true the WYSIWYG editor story is considering Optimizely was only created in 2010. But, what we can say with some certainty, is that technology enables us to test; easily and a lot.

I just think testing has been more prolific on the web rather than other areas – be they digital or not – because of the ‘case-study-ification’* of web based ‘wins’. Like this. Or this. Or this. And I attribute the majority of this to what I would have thought to be the founder of AB testing case-study-ifications, the $300m button.

*hey, if McDonalisation is a word, I can use case-study-ification.

So let’s look at experimentation beyond UI and even beyond web. 🖥️

Product experimentation

The guys at AirBnB test search algorithms, pricing, discounts, and new features within their rollouts. ✈️

Those at Sky test their customer service wait times, personalised chat bot services, opening hours, engineer visit schedule.

A popularised method for testing beyond UI, is testing algorithms for merchandising rules – be they product recommendations, product merchandising or search rule testing.

In essence, anything that has code behind it can be tested. Apps are a great example. Until a few years ago, apps were unable to be tested via traditional AB testing methodologies. But with the proliferation of server-side test, or full-stack from the likes of Optimizely, anything inc. apps with an SDK can be tested, without having to go through Apple or Google app reviews.

Take Kiosks. KFC experimented with, and continue to do so, AI kiosks with facial recognition. These kiosks can recommend menu items based on a customer’s estimated age and mood. Apparently, if you’re female and in your 50s you’ll get a recommendation of “porridge and soybean milk for breakfast”. But if you’re male and in your early 20s, be prepared for a crispy chicken hamburger, roasted chicken wings and coke. Bit of a difference there.

Those Coca-Cola kiosks you see in the likes of Five Guys, too. They are personalised based on demographic, advertiser selection and popularity and showcase hierarchical options, or recommendations. 🥤

Even drive-throughs. Did you know, McDonalds iterate their approach to specific stores with licence plate recognition? Drive up. Recognise your number plate. And whilst digital menus aren’t showing personalised recommendations to you as an individual yet, they do have hierarchy behind them. All with the objective of “speed to the customer”. The weather, trending menu items, restaurant traffic, time of the day, local events and popularity are important, but none more so than operational stock. If we have more hamburgers and less nuggets, push those burgers.

New product development is vital is for product experimentation. When Albert Einstein noted that anyone who has never made a mistake has never tried anything new, I’m fairly certain he was talking about the need to experiment in the quest for discovering new things. The Post it was an experiment – the chemist Spencer Silver stated ✉️

“The key to the Post-It adhesive was doing the experiment. If I had sat down and factored it out beforehand, and thought about it, I wouldn’t have done the experiment.

Viagra was an experiment, albeit accidental. It’s cited that project teams in R&D organisations spent an average 77% of their time on experimentation and related analysis activities to remove uncertainty. For me, proof of concepts are vital to the success of a product. Determine the intent from the user and learning from that to iterate is the easiest and quickest source of innovation. With experimentation, online, you can run something known as a painted door test – something that exists as an interaction, but the product or feature isn’t available. The purpose of the interaction is to measure it to determine intent. Here are some examples:

• Buffer created a landing page with a proposition and a button to see plans and pricing before the product was ready
• Dropbox created an explainer video of Drew, the founder, using a series of prototyped designs before the product was ready – increasing the beta waiting list pre-product from 5,000 to 75,000 overnight (video is here)
• Hubble created a website to gauge demand and interest before a supplier was even selected – what they called a “demand experiment” 👓
• The same with eSalon – running a proof of concept to see if women might be interested in buying customised colour online, creating a rudimentary website and hand-mixing colours for just 50 women.

Digital experience stack

We’ve all seen the image of the explosion of Martech tools from 2007 where there were just ~150 vendors to now over 7,000.

Especially when every tool (over)promises to solve a (all the)problem(s) and add 50,000 x ROI. More and more tools have Open APIs meaning testing is possible with and within them. At User Conversion, on behalf of some of our clients, we’ve helped validate the attribution of the “tool that promises to solve all of the problems” through serving the platform as a treatment in the variation, and hidden in the control. We’ve testing adding (and removing) payment providers like Klarna, recommended product engines such as Increasingly or Nosto, merchandising products such as Foundit! Or Attraqt, customer reviews like Bazaarvoice…I could go on. All done with the aim of validating attribution to better understand ROI, as well as, understanding behaviour to iterate and improve.

Whilst the increase in technology has assisted our ability to test, it has merely clouded what experimentation can be for other concepts. And in some respects, limited our creativity and innovation to embrace it elsewhere.

Experimentation outside of digital

Testing and experimentation is most synonymous with the pharmaceutical industry. The first in-field experiment being in 1747, where James Lind, surgeon of the HMS Salisbury, hypothesised that the breakdown of bodily tissues and organs could be prevented with acids. He divided the 12 sick sailors into pairs of two, giving them a different supplement with their diet, in what is assumed to be the first clinical trial of our time.

The most sudden and visible good effects were perceived use from the oranges and lemons. (Lind, 1753)

What about movies? We see films continually experimenting and seeking feedback as they go – the making of Frozen 2 ‘Into The Unknown’ on Disney + earlier this year taught me that much. It happens all the time. The entire ending of ET changed after test audiences were upset that our little brown friend didn’t make it home. 🤯 Or, more poignantly, in the 2004 hit Dodgeball: A True Underdog Story, Ben Stiller’s GloboGym were meant to win at the end, despite audiences rooting all the way through for the Average Joes. Apologies for the spoilers. But whilst were here, let’s remind ourselves of that famous GloboGym ad

https://www.youtube.com/watch?v=pNGaCgnAeJs

David Brailsford from 2003 experimented with different massage gels for faster muscle recovery, with washing hands in the most accurate of manners to avoid illness or tested various fabrics in the bike seat for comfort and aerodynamics for the British Cycling Team. Just 5 years later, in 2008, the team won 60% of available Gold medals at Beijing and set 9 world records 4 years later in London. And having gone from not winning a Tour De France in 110 years, through continuous experimentation, Bradley Wiggins, Chris Froome and the team won 5 in the next 6 years. 🚴

In a slightly less glamourised example, but more popular, The Tim Ferris Experiment explored this concept of mastering a skill quickly using his personal methods spoken about in The 4 Hour Work Week. It’s a good watch, I must admit – check it out on iTunes.

Disney experiment with guest behaviour all the time. They’ve experimented with, and failed, on removing all turnstiles before. Or of the 46 rides currently at Disney World, taking just one – Toy Story Mania - and making a ‘queue-free’ ride where, instead of queuing, you had to make an appointment. It failed miserably, but led to what we see now, what they call a “virtual queue” at the popular Star Wars Ride of Resistance. It’s supposed to be amazing. No spoilers please.

The bike manufacturer and retailer Vanmoof identified that their bikes were arriving to their customers damaged when being shipped to the USA. As the boxes were about the same size as a big, expensive flat screen TV, they decided to put a picture of the TV on the box instead of a picture of the bike, assuming that handlers would take a little bit more care. Overnight their shipping damages dropped by 75% (credit to Steven Pavlovich for telling me about this one…)

Bringing it back round to business. Many founders experiment with their idea before launching a more developed product. In the case of Innocent drinks, the three founders fresh out of University went to a music festival, set up a stand, sold some drinks and asked festival-goers “if they should give up their advertising day jobs to start a smoothie business” with two bins to empty the cups into – “yes” or “no”. You can assume the rest given they sold the majority of their business to Coca-Cola 15 years later for £95m

The truth is that experimentation is everywhere. It’s needed. It’s needed to reduce uncertainty. To drive innovation. And to learn.

What’s the message here?

We’ve lost our way. The concept of experimentation is designed for our decision making, to fuel our creativity, and to speed our route to market. It’s not designed to move the filters on a page from the left to the top, or to make them sticky, or to change the colour of the call to action from a insipid orange to a refreshing blue.

Remember, experimentation …

• Speeds decision making up either / both through reduced risk of an unwanted event or learning to iterate moving forwards
• Does not need a control (although it helps with effective measurement). It is purely the correlation between cause and effect.
• Can easily be mistaken or merged with the idea of “research”- because that’s what it is; knowledge. Weren’t Innocent drinks founders just researching, not testing? The same with Steven Spielberg’s research on ET’s ending?
• Is designed to build from and iterate on top of. Or as John Cleese calls it in his book Creativity, “bring the critical faculties … to bear on whatever it is you have thought of. You’re now sufficiently clear about your idea to be in a position to evaluate it. If you decide it can be improved, you go into the iteration process”