Optimizing Multiple Consumer-specific Objectives in End-to-End Ensemble Machine Learning Serving

Abstract

Optimizing the quality of machine learning (ML) services for individual consumers with specific objectives is crucial for improving consumer satisfaction. In this context, end-to-end ensemble ML serving (EEMLS) faces many challenges in selecting and deploying ensembles of ML models on diverse resources across the edge-cloud continuum. This paper provides a method for evaluating the runtime performance of inference services via consumer-defined metrics. We enable ML consumers to define high-level metrics and consider consumer satisfaction in estimating service costs. Moreover, we introduce a time-efficient ensemble selection algorithm to optimize the EEMLS with intricate trade-offs between service quality and costs. Our intensive experiments demonstrate that the algorithm can be executed periodically despite the extensive search space, enabling dedicated optimization for individual consumers in dynamic contexts.