Successful deployment of continuous A/B testing in a non-digitally born organisation

Abstract

This thesis aims to analyse the applicability of the experimentation methods and practices of industry-leading organisations found in literature through a case study of a non-digital-born organisation operating in the physical and digital worlds. Organisations today focus increasingly on delivering customer value and solving customer challenges. To do so, organisations need a reliable method to assess their ideas, and during the last decades, A/B testing has become the industry standard for evaluating ideas. A/B testing is a scientific method that compares two versions of a product to each other to understand which is better based on statistics. When A/B testing is used continuously and systematically, it is referred to as continuous experimentation. Continuous experimentation is a cycle where the learnings of the previous experiment are used in the following experiment.

The critical success factors of an individual A/B test are, based on literature, statistical validity, the choice of the right metric and accurate experiment analysis. In addition to these factors, the case study highlighted the importance of reliable and trustworthy data and the need for experienced and capable professionals to conduct accurate experiment analysis and ensure statistical validity.

The main challenges in moving towards continuous experimentation, based on literature, are unsupportive architecture, the inertia of organisational culture, slow lifecycle speed, finding the right metrics, lack of priority, experience, and well-functioning tools. To move towards continuous experimentation, the critical aspects based on literature are technical evolution, organisational evolution and business evolution. While the case study did not experience all these challenges, its main challenges were slow lifecycle speed, unsupportive infrastructure, lack of experience and team self-sufficiency, and data quality issues.

The case study found that critical success factors for reliable A/B testing results are applicable to a non-digitally born organisation. The main difference is the sample size, as industry-leading organisations have millions of users to test on, while smaller organisations do not. However, the challenges faced by the case organisation were mainly technical and related to obtaining reliable and trustworthy results. In contrast, industry leaders focused on scaling up experimentation, running parallel experiments, and building an in-house experimentation platform.