Designing experiment objects for Continuous Experimentation: An exploratory multiple-case study

Abstract

Continuous experimentation relies on the design and implementation of experiment objects: artefacts that are used as the treatment or stimulus in a software product experiment. To be used in practice, experiment objects must have traits that allow drawing correct inferences regarding a new product, while simultaneously being resource-efficient to produce. In this paper, we investigate what kinds of experiment objects companies use, how they are selected, what factors influence the selection, and how they evolve over time. We present a multiple-case study based on semi-structured interviews with representatives from eight companies. Four kinds of experiment objects were found: prototypes, minimum viable products, proofs-of-concept, and wireframes. The results show that deciding what kind of experiment object to develop is a collaborative and customer-oriented process. As experimentation proceeds, experiment objects can be thrown away, modified for new experiments, or even become products. Experiment objects were understood not only as artefacts in product experiments, but blended with purposes familiar from prototyping and user interface and experience design, and they were used as vehicles for customer communication. The study shows that experiment objects are created for a variety of purposes and understandings of experimentation. Guidelines for creating and managing effective experiment objects for different variations of continuous experimentation are called for.