Why do you keep reciting the first 2 letters of the alphabet?
A/B testing is the process of comparing two different iterations of a webpage or application so a company can discover which variation performs better (Martin, 2015). I like to explain this using a food analogy. Think about when you purchase your favourite burger from your favourite fast food chain, and you would never purchase anything else. They suddenly remove your beloved burger and replace it with a 1 week trial of a new burger, but only in your local branch; perhaps they swap mayonnaise with a chutney. You try it and my gosh…you love it! even more so than your previous go to choice. What’s even better is that it turns out there is an increase in 30% of sales as others love the burger too. The company then decides to roll out the new iteration nationwide. That’s a successful A/B test in the physical world, but now imagine that online. Figure 1 shows 50% of visitors to a site being shown variation A, and 50% being shown variation B; we then look to analyse the % of conversions from each variation with the higher figure being the winner. Companies can then choose to conduct another A/B test (why stop when you’re onto a good thing), or go live with the winning variation so all visitors can cast their eyes on the update.
Surprising results
Hubspot details fantastic examples of A/B tests that provide unexpected results. Figure 2 shows a variation that incentivises users to pre-order a new Sims game with a discount voucher; figure 3 displays a variation which removed the code entirely. The test results revealed the variation with no promotional code increased purchase rates by 43% compared to the control group. Think about how crazy that sounds? – people just wanted to purchase the game; no incentive required. This is a great example of how conventional human opinion fails to optimise your website, and exposes that letting data decide ensures an unbiased result. You can see how companies are making this a point of parity (a minimum standard) with an intriguing Netflix presentation on how the media powerhouse uses A/B testing, found here.
So is this the God particle or is there a catch?
Roy (2001) detailed the idea of having an Overall Evaluation Criterion (OEC) which can be applied to analyse the significance of an A/B test. This doesn’t have to be limited to sales conversions (as with the burger analogy earlier) but could be to increase subscriber count, or encourage users to leave a product review. Now of course there are infinite reasons why our audience behave differently with variation A compared to variation B; we’ve got to remember that correlation doesn’t necessarily mean causation (Levitin, 2016). A user might be having a good day and naturally want to leave a positive review of a product, or they might have heard through word of mouth how brilliant an Instagram account is, so subscribed purely on a recommendation from a friend; it just so happens they are helplessly caught in an A/B test. I won’t bore you by recommending statistical text books on how you can gain accuracy in your testing but Kohavi., et al (2009) summarises that a confidence level is found at 95%, implying that the final 5% “will incorrectly conclude that there is a difference when there is none”. The study also explains the limitation of A/B testing through “primacy and newness effects” which argues that experienced users may be less efficient at navigating a website due to being shown an unfamiliar webpage (primacy effect), as well as clicking on links or calls to action purely because they want to investigate the new feature (newness effect). The study suggests running a test for a few weeks rather than just a few days in order to counteract this bias.
I can work Microsoft Word, but I wouldn’t have a clue about these science experiments?
Ok, I admit, A/B testing can get quite complex but let’s not forget about the basics (think back to the burger analogy). I’ll supplement this with another to illustrate that it’s important to dive into the detail when you’ve mastered the basics. Tennis is a simple sport; you hit a ball with a racket and hope the other player either hits the net or misses entirely. But if you want to improve your game, you might want to consider purchasing a new racket string. Do you go for Natural Gut? Synthetic Gut? a Kevlar string? or hybrid? For your frame, do you choose Babolat’s Aeorobeam technology Stabilizer technology? or Wilson’s Countervail blades? You get the idea. These tools can get quite complex for a relatively simple sport, but when tools are utilised, the improvement can be massive. See how this applies to A/B testing? Fortunately for you I’ve got some tools to help your testing too…
User friendly sights such as Optimizely and A/B Tasty eliminates the need for coding and only requires basic training to conduct simple A/B tests. I’ve found a fantastic “Optimizely Fundamentals” course on Lynda.com; you get a free 10 day trial which should be more than enough to get you an A/B testing king/queen with the tool. Practice the basics and you will soon be diving into the detail as you optimise your website with advanced tools; similar to the tennis player wanting to improve their game.
References
Kohavi, R., Longbotham, R., Sommerfield, D. and Henne, R.M., 2009. Controlled experiments on the web: survey and practical guide. Data mining and knowledge discovery, 18(1), pp.140-181.
Levitin, D. (2016) A field guide to lies and statistics: A Neuroscientist on how to make sense of a complex world. United Kingdom: Viking.
Martin, E. 2015, The ABCs OF A/B TESTING, ONLINE INC, WILTON.
Roy, R.K., 2001. Design of experiments using the Taguchi approach: 16 steps to product and process improvement. John Wiley & Sons.