Titel
Limitations of neural network training due to numerical instability of backpropagation
Autor*in
Abstract
We study the training of deep neural networks by gradient descent where floating-point arithmetic is used to compute the gradients. In this framework and under realistic assumptions, we demonstrate that it is highly unlikely to find ReLU neural networks that maintain, in the course of training with gradient descent, superlinearly many affine pieces with respect to their number of layers. In virtually all approximation theoretical arguments which yield high order polynomial rates of approximation, sequences of ReLU neural networks with exponentially many affine pieces compared to their numbers of layers are used. As a consequence, we conclude that approximating sequences of ReLU neural networks resulting from gradient descent in practice differ substantially from theoretically constructed sequences. The assumptions and the theoretical results are compared to a numerical study, which yields concurring results.
Stichwort
Round-off errorsDeep neural networksGradient descentNumerical stability
Objekt-Typ
Sprache
Englisch [eng]
Persistent identifier
Erschienen in
Titel
Advances in Computational Mathematics
Band
50
ISSN
1019-7168
Erscheinungsdatum
2024
Publication
Springer Science and Business Media LLC
Projekt
Kod / Identifikator
F65
Erscheinungsdatum
2024
Zugänglichkeit
Rechte
Rechteangabe
© The Author(s) 2024
Universität Wien | Universitätsring 1 | 1010 Wien | T
+43-1-4277-0