Berlin 2024 – scientific programme

Parts | Days | Selection | Search | Updates | Downloads | Help

DY: Fachverband Dynamik und Statistische Physik

DY 34: Poster: Machine Learning, Data Science, and Reservoir Computing

DY 34.8: Poster

Wednesday, March 20, 2024, 15:00–18:00, Poster C

Mutual information estimation in the learning process of neural networks — •Lea Melina Faber, Ibrahim Talha Ersoy, and Karoline Wiesner — Universität Potsdam, Institut für Astronomie und Physik, Potsdam, Deutschland

Deep neural networks play an increasingly important role to predict the dynamics of highly complex systems. The most prominent example is that of climate models. Nonetheless it is still unknown how the actual learning process works. To better understand such models it is crucial to open the black box of neural nets. It has been conjectured, that mutual information is a good measure of learning in neural networks. Often times the simplest of estimation techniques, like binning, are used. However, it has been shown that in some settings they are only very crude estimators. We have analyzed the MSE, bias and variance for standard mutual information estimators, using the specific settings relevant for neural network analysis. Our results show that most estimators have problems with high dimensions and high mutual information, specifically in combination, which is a standard situation in the context of neural networks. Furthermore, it has been shown, that generally estimators show the best results for Gaussian distributions. However, our numerical experiments show that non-Gaussian distributions are likely to play a significant role in the learning processes in neural networks. Hence, we require suitable estimators for these cases. We illustrate our results with the MNIST dataset.

Keywords: Machine Learning; Mutual Information Estimation; Neural Networks; Information Plane