A/B Testing…?!

What A/B testing is.

A/B testing is the way you can compare two websites or mobile apps against each other in order to find out which one executes better (Optimizely, 2015). A/B testing is also known as bucket testing or split testing. It is used to improve your conversion rates by using data and statistics to validate new designs changes. A/B testing is an expression or phrase used for non-specific experimentation which sole purpose is to test a theory and produce statistics (Martin, 2015)



Running an A/B test that compares a variation against a current experience, allows you to ask focused questions about changes to your website or app, and after that collect data about the impact of that change. In order to sustain your business you have to be able to literally read and understand the customers which are visiting your website or app every day (McMillen, J., 2014).

How does It work?

You simply take the screen of a webpage or app and create a second version of the same page. The change could be something small, just one button or it could be something substantial such as new design on the entire page. Then your traffic is shown on the original page and also on the second version, which is with the changes.


When customer enters your webpage or app and has been served by the control (original page) or variation (modified page), then their experience of engagement is recorded, measured and analysed through statistical engines (Kohavi, 2009). After that you can have a clear view whether the changes that were made have any effect or not.


Why you should use A/B Testing?

This testing allows you to make changes to what the customer experiences when visiting your website and at the same time allows you to collect data on the results of the implemented changes. For example, a business wants to increase its sales volume by implementing a new landing page. In order to achieve such a goal, the particular business has to use A/B testing when changing headline, adding call-to-action buttons or a completely new layout. By testing each new component individually, the business is going to have clear view of which change works best.  A/B testing could also be used if there is a new product, which is about to be launched.

Some ideas for A/B testing.

Below I’m going to list some ideas of how to start A/B testing, but the testing itself depends on the industry your business belongs to. For example, a media company wants to increase the time the reader spends on their website and expand their articles with social sharing. In order to achieve its aim, the business has to test variations on: Email sign-up, suggested content and button for social media sharing.

Travel companies would have to make other changes such as faster and a more reliable booking process. The company needs to test variation of: home page, search results of the page or the way the service is presented to the customer.

A/B testing tools

There are a number of A/B testing tools available out there such as Google Analytics, KISSmetrics, Unbounce and so on.

In my previous blog I have mentioned Google Analytics, but now im going to emphasise on KISSmetrics, which In my opinion offers a pretty good and flexible service. KISSmetrics is a well know testing tool which highlights the human side of data. The testing data is attached to people who engage with your website and are monitored throughout the time spent on your website (KISSmetrics, 2014). KISSmetrics allows you to keep a constant track of what is happening from the beginning to the end of the communication funnel. But not everything is ideal, there are pros and cons.


The advantages are: very flexible reporting data options, simple to use software, can track data to real people, the funnel-based data is highly accurate and not at least work well with other paid or free softwares.

The disadvantages are: KISSmetrics is low power in comparison with other testing tools, its highly costly, more specific data requires a more in depth learning abilities.

Overall the A/B testing have a positive impact on your website or app, but bear in mind there are several disadvantages. A/B testing may lead to losing some conversions due to the experiment. This means that you could lose sign-ups in order to experiment with a bad variation and if you repeat the process with a worse variation than the control, you may end with substantial loss of your sign-ups. Another disadvantage is that things change with the time. In A/B testing you receive only quantitative data, it just tells you the better one of two given options, but we also need qualitative data, which tells you how to make a better change just given one option (Zuupy, 2011). Also NEVER test the control first and then the variation, you MUST do both at the same time, otherwise you will end up with inaccurate data.

Zuupy.com, 2011, The Disadvantages of A/B Testing, [Online], Available at: https://zuupy.wordpress.com/2011/06/18/the-disadvantages-of-ab-testing/. (Accessed 8th of May 2016).

McMilen, J., 2014, How To Chose The Right Testing Software For Your Business [Online]  Available at: https://blog.crazyegg.com/2014/06/25/best-testing-software/. (Accessed 8th of May 2016).

Optimizely, 2015, A/B Testing, [Online] Available at: https://www.optimizely.com/ab-testing/. (Accessed 8th of May 2016)

KISSmetrics, 2016, Supercharge Your Testing With The New Kissmetrics A/B Test Report, [Online] Available at: https://blog.kissmetrics.com/kissmetrics-ab-test-report/. (Accessed 8th of May 2016).

Martin, E. J. (2015). The ABCs OF A/B TESTING. EContent, 38(7), 12-17.

Kohavi, R., Longbotham, D., Sommerfield, and Henne, R., (2009)  “Controlled experiments on the web: survey and practical guide,” Data Mining and Knowledge Discovery, vol. 18, no. 1, pp. 140-181.


Leave a Reply

Your email address will not be published. Required fields are marked *