A Model Monitoring Platform for Machine Learning

No Thumbnail Available
Journal Title
Journal ISSN
Volume Title
Perustieteiden korkeakoulu | Master's thesis
Date
2023-01-23
Department
Major/Subject
Data Science
Mcode
SCI3115
Degree programme
Master's Programme in ICT Innovation
Language
en
Pages
55 + 1
Series
Abstract
In this thesis, model registry and model monitoring services are implemented and integrated with an existing workflow to address the efficient management of the machine learning lifecycle. Utilization and integration of multiple services are aimed at addressing the challenges of machine learning applications in production. The related background knowledge, including machine learning, data science, and MLOps, is presented to provide context for the work. The thesis also covers basic visualization principles and introduces potential visualization platforms. The entire lifecycle of a machine learning project is then discussed in detail, highlighting the difficulties that can arise in production. Model registry and model monitoring are explained in more detail, as they are central to the work of the thesis. Third-party machine learning solutions and infrastructure services are also introduced. To evaluate the monitoring visualizations, questions about what to monitor in a machine learning project are collected from staff with expertise in machine learning and data in Granlund. These serve as requirements for visualization monitoring. The extended workflow with the newly implemented model registry and model monitoring services is then introduced in detail. A machine learning project on heating energy consumption prediction is run through the extended workflow as an example to demonstrate the details and operations of each part. Created monitoring visualizations in Grafana are evaluated based on the collected questions. Most questions can be answered in the different dashboards and the reasons are discussed. However, some questions remain unsolved due to a lack of information from the infrastructure. The thesis concludes with a discussion of potential future work, focusing on the loading of infrastructure data into the existing workflow.
Description
Supervisor
Vuorimaa, Petri
Thesis advisor
Kukkonen, Ville
Keywords
machine learning, visualization, model monitoring, grafana, azure
Other note
Citation