A Model Monitoring Platform for Machine Learning

Abstract

In this thesis, model registry and model monitoring services are implemented and integrated with an existing workflow to address the efficient management of the machine learning lifecycle. Utilization and integration of multiple services are aimed at addressing the challenges of machine learning applications in production. The related background knowledge, including machine learning, data science, and MLOps, is presented to provide context for the work. The thesis also covers basic visualization principles and introduces potential visualization platforms.

The entire lifecycle of a machine learning project is then discussed in detail, highlighting the difficulties that can arise in production. Model registry and model monitoring are explained in more detail, as they are central to the work of the thesis. Third-party machine learning solutions and infrastructure services are also introduced. To evaluate the monitoring visualizations, questions about what to monitor in a machine learning project are collected from staff with expertise in machine learning and data in Granlund. These serve as requirements for visualization monitoring. The extended workflow with the newly implemented model registry and model monitoring services is then introduced in detail. A machine learning project on heating energy consumption prediction is run through the extended workflow as an example to demonstrate the details and operations of each part.

Created monitoring visualizations in Grafana are evaluated based on the collected questions. Most questions can be answered in the different dashboards and the reasons are discussed. However, some questions remain unsolved due to a lack of information from the infrastructure. The thesis concludes with a discussion of potential future work, focusing on the loading of infrastructure data into the existing workflow.