Bereiche | Tage | Auswahl | Suche | Aktualisierungen | Downloads | Hilfe
SOE: Fachverband Physik sozio-ökonomischer Systeme
SOE 7: Focus Session: Self-Regulating and Learning Systems: from Neural to Social Networks
SOE 7.4: Vortrag
Mittwoch, 19. März 2025, 10:30–10:45, H45
Response functions in residual networks as a measure for signal propagation — •Kirsten Fischer1,2, David Dahmen1, and Moritz Helias1,3 — 1Institute for Advanced Simulation (IAS-6), Jülich Research Centre, Jülich, Germany — 2RWTH Aachen University, Aachen, Germany — 3Department of Physics, Faculty 1, RWTH Aachen University, Aachen, Germany
Residual networks (ResNets) demonstrate superior trainability and performance compared to feed-forward networks, particularly at greater depths, due to the introduction of skip connections that enhance signal propagation to deeper layers. Prior studies have shown that incorporating a scaling parameter into the residual branch can further improve generalization performance. However, the underlying mechanisms behind these effects and their robustness across network hyperparameters remain unclear.
For feed-forward networks, finite-size theories have proven valuable in understanding signal propagation and optimizing hyperparameters. Extending this approach to ResNets, we develop a finite-size field theory to systematically analyze signal propagation and its dependence on the residual branch's scaling parameter. Through this framework, we derive analytical expressions for the response function, which measures the network's sensitivity to varying inputs. We obtain a formula for the optimal scaling parameter, revealing that it depends minimally on other hyperparameters, such as weight variance, thereby explaining its universality across hyperparameter configurations.
Keywords: signal propagation; residual networks; field theory; finite-size effects