Berlin 2024 – scientific programme

Parts | Days | Selection | Search | Updates | Downloads | Help

SOE: Fachverband Physik sozio-ökonomischer Systeme

SOE 8: Machine Learning in Dynamics and Statistical Physics II (joint session DY/SOE)

SOE 8.12: Talk

Tuesday, March 19, 2024, 12:30–12:45, BH-N 243

Investigating the Evolution of Fisher Information for Neural Network Dynamics — •Marc Sauter, Samuel Tovey, Konstantin Nikolaou, and Christian Holm — Institute for computational physics, Stuttgart, Germany

Machine learning has proven to be a powerful tool with remarkable effectiveness for various physical applications. Even though Neural Networks offer great potential for solving complex tasks, their black-box nature limits the information obtainable about the underlying mechanisms they employ. Especially in physics, where the exact methodologies of solving tasks are just as important as finding solutions itself, advances in interpretable machine learning promise to aid the applicability of Neural Networks task-solving capabilities greatly. The Fisher Information Matrix (FIM) is a statistical measure known for its ability to identify second order phase transitions in physical systems. It can also be used for analyzing learning dynamics, where it expresses correlations between influences of parameters in Neural Networks. However, because of its size, computation of the FIM is currently intractable for Neural Networks used in common problem settings. As a way of obtaining parts of the information contained in the FIM, we introduce a novel mathematical relationship between the trace of the FIM and the Neural Tangent Kernel, a smaller observable of neural network training. We apply this approach for simple test models and discuss arising research topics.

Keywords: Machine Learning; Fisher Information; Neural Tangent Kernel; Phase Transitions