Designing an Effort Estimation Process for Embedded Software Projects

Abstract

This thesis proposes a hybrid software development effort estimation approach by exploring and combining expert-based and data-driven estimation methods. Methods traditionally employed in the software development industry are analyzed and augmented by recent research in machine learning and methods for structured expert judgment. The context is embedded software development at a Finnish company, EKE-Electronics Ltd. The approach proposes using the Classical Model based on bottom-up workload estimation by several experts with three-point elicitation for each task: the lowest, highest, and most likely values. The estimation is augmented by explicitly asking the experts how confident they are in their estimates. The effort probability distributions are modeled as triangular distributions and are convoluted to create the project's workload forecast as a probability distribution. Opinions of several experts are averaged with an option for using expert-specific weights. The IDEA protocol is proposed for the most complicated projects because it supplements the Classical Model approach through a discussion round where experts can clarify their disagreements. The artificial neural network, CatBoost, and XGBoost models are developed to test data-driven estimation. The models are tuned using automatic hyperparameter tuning with Optuna, and their structure is explained using the Shapley Additive Explanations framework. Finally, the foundations of effort estimation are presented, and suitable accuracy metrics are discussed. The criteria for method selection are ranked using the Analytical Hierarchy Process with an acceptable consistency score. The implementation of the new process is framed using Kotter's eight-stage change-leading process.