aalto1 untyped-item.component.html
GRADSTOP: Early Stopping of Gradient Descent via Posterior Sampling
Loading...
Access rights
openAccess
CC BY-NC
CC BY-NC
Creative Commons license
Except where otherwised noted, this item's license is described as openAccess
publishedVersion
URL
Journal Title
Journal ISSN
Volume Title
A4 Artikkeli konferenssijulkaisussa
This publication is imported from Aalto University research portal.
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.
Date
Department
Major/Subject
Mcode
Degree programme
Language
en
Pages
8
Series
ECAI 2025 - 28th European Conference on Artificial Intelligence, including 14th Conference on Prestigious Applications of Intelligent Systems, PAIS 2025 - Proceedings, pp. 2057-2064, Frontiers in Artificial Intelligence and Applications ; Volume 413
Abstract
Machine learning models are often learned by minimising a loss function on the training data using a gradient descent algorithm. These models often suffer from overfitting, leading to a decline in predictive performance on unseen data. A standard solution is early stopping using a hold-out validation set, which halts the minimisation when the validation loss stops decreasing. However, this hold-out set reduces the data available for training. This paper presents GRADSTOP, a novel stochastic early stopping method that only uses information in the gradients, which are produced by the gradient descent algorithm 'for free.' Our main contributions are that we estimate the Bayesian posterior by the gradient information, define the early stopping problem as drawing sample from this posterior, and use the approximated posterior to obtain a stopping criterion. Our empirical evaluation shows that GRADSTOP achieves a small loss on test data and compares favourably to a validation-set-based stopping criterion. By leveraging the entire dataset for training, our method is particularly advantageous in data-limited settings, such as transfer learning. It can be incorporated as an optional feature in gradient descent libraries with only a small computational overhead. The source code is available at https://github.com/edahelsinki/gradstop.
Description
Publisher Copyright: © 2025 The Authors.
Keywords
Other note
Citation
Jamshidi, A, Seppäläinen, L, Haitsiukevich, K, Luu, H P H, Björklund, A & Puolamäki, K 2025, GRADSTOP: Early Stopping of Gradient Descent via Posterior Sampling. in I Lynce, N Murano, M Vallati, S Villata, F Chesani, M Milano, A Omicini & M Dastani (eds), ECAI 2025 - 28th European Conference on Artificial Intelligence, including 14th Conference on Prestigious Applications of Intelligent Systems, PAIS 2025 - Proceedings. Frontiers in Artificial Intelligence and Applications, vol. 413, IOS Press, pp. 2057-2064, European Conference on Artificial Intelligence, Bologna, Italy, 25/10/2025. https://doi.org/10.3233/FAIA251043