Automating playtest data analysis: A natural language processing approach

Abstract

This thesis implements and evaluates a new tool for automating the analysis of playtesting videos to help streamline the playtesting process.

Playtests provide relevant and consequential feedback to a game development team and are a powerful tool in a user researcher’s toolbox. However, the analysis process is a very time-consuming task; tens or even hundreds of hours of footage must be examined by hand to uncover actionable data. This time investment and a general lack of workarounds in the industry remain a notable weakness in playtesting.

However, it is now easier than ever to leverage machine learning, and new artificial intelligence tools frequently become available. In order to improve playtesting efficiency, we leverage one of these new tools to develop a system that prunes the amount of video material that must be analyzed by hand.

More specifically, the thesis investigates the use of neural language models in analyzing video voice transcripts to identify segments with relevant content, such as the user expressing dislike or confusion. Neural language models have advanced rapidly in recent years and present yet underexplored opportunities for playtest data analysis.

The system was developed and tested at HypeHype Inc for their game making platform, HypeHype. A comparative study is conducted to evaluate the automated solution, measuring the improvement over a purely manual analysis process using metrics such as time-spent, insights collected and accuracy. While the number of valuable insights per video is reduced using this system, it offers a much higher rate of discovering said insights. Ultimately, this system contributes to a much more time-efficient playtesting analysis workflow, resulting in a lessened manual workload for the researcher.