aalto1 untyped-item.component.html
Tenplex : Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections
Loading...
Access rights
openAccess
CC BY
CC BY
Creative Commons license
Except where otherwised noted, this item's license is described as openAccess
publishedVersion
URL
Journal Title
Journal ISSN
Volume Title
A4 Artikkeli konferenssijulkaisussa
This publication is imported from Aalto University research portal.
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.
Authors
Wagenländer, Marcel
Li, Guo
Zhao, Bo
Mai, Luo
Pietzuch, Peter
Date
Department
Major/Subject
Mcode
Degree programme
Language
en
Pages
16
Series
SOSP 2024 - Proceedings of the 2024 ACM SIGOPS 30th Symposium on Operating Systems Principles, pp. 195-210
Abstract
Deep learning (DL) jobs use multi-dimensional parallelism, i.e., combining data, model, and pipeline parallelism, to use large GPU clusters efficiently. Long-running jobs may experience changes to their GPU allocation: (i) resource elasticity during training adds or removes GPUs; (ii) hardware maintenance may require redeployment on different GPUs; and (iii) GPU failures force jobs to run with fewer devices. Current DL frameworks tie jobs to a set of GPUs and thus lack support for these scenarios. In particular, they cannot change the multi-dimensional parallelism of an already-running job in an efficient and model-independent way.We describe Tenplex, a state management library for DL systems that enables jobs to change their parallelism dynamically after the GPU allocation is updated at runtime. Tenplex achieves this through a new abstraction, a parallelizable tensor collection (PTC), that externalizes the job state during training. After a GPU change, Tenplex uses the PTC to transform the job state: the PTC repartitions the dataset state under data parallelism and exposes it to GPU workers through a virtual file system; and the PTC obtains the model state as partitioned checkpoints and transforms them to reflect the new parallelization configuration. For efficiency, Tenplex executes PTC transformations in parallel with minimum data movement between GPU workers. Our experiments show that Tenplex enables DL jobs to support dynamic parallelization with low overhead.
Description
Publisher Copyright: © 2024 Copyright held by the owner/author(s).
Other note
Citation
Wagenländer, M, Li, G, Zhao, B, Mai, L & Pietzuch, P 2024, Tenplex : Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections. in SOSP 2024 - Proceedings of the 2024 ACM SIGOPS 30th Symposium on Operating Systems Principles. ACM, pp. 195-210, ACM Symposium on Operating Systems Principles, Austin, Texas, United States, 04/11/2024. https://doi.org/10.1145/3694715.3695975