Remote user study: a reflection on remote user research methods during the COVID-19 pandemic

Abstract

At times of pandemic, traditional design ethnographic field practices do not work because of restrictions on social contact. As the prevailing situation becomes the “new normal,” design methods need to evolve to include remote user studies. Modern remote user research tends to focus on usability testing in the form of unmoderated, automated, simplified online services and tools, glorifying the need for vast amounts of quantitative data. User research, as distinguished from usability testing, is design ethnography, but the field of digital ethnography has not advanced despite its potential and necessity.

The purpose of this thesis is to critically reflect on learnings from the presented case study in the context of remote user research during the times of COVID-19 from the perspective of a design practitioner. The study focused on a publicly accessible outage map used extensively in the Finnish utility market to provide information on network disruptions. Quantitative and qualitative data was gathered through surveys to understand demographics and for selecting participants for qualitative semi-structured phone interviews. Additionally, key informant interviews were run separately.

A noteworthy sample size of 1153 responses allowed for data analysis, and further interviews gave qualitative insights. However, research methods forced by the COVID-19 pandemic produced results that represented a highly attitudinal view. In design practice, behavioral research is preferred (focusing on user actions), but approximating contextual research and ethnography in remote user research proves to be challenging. It requires more from the participant (pro-active sharing of video, pictures, texts, audio, and potentially falling victim to Zoom fatigue) and from the researcher (planning for contingencies, ethics, and data privacy rights).

With the increase in designers working fully remote, user research requires advances in practical digital ethnographic methods. Remote user study needs to become easier and more accessible because user research is too important in the human-centered design process to be skipped.