The function space of deep-learning machines is investigated by studying growth in the entropy of functions of a given error with respect to a reference function, realized by a deep-learning machine. Using physics-inspired methods we study both sparsely and densely connected architectures to discover a layerwise convergence of candidate functions, marked by a corresponding reduction in entropy when approaching the reference function, gain insight into the importance of having a large number of layers, and observe phase transitions as the error increases.
Bibliographical note© 2018 The American Physical Society. Exploring the Function Space of Deep-Learning Machines. Bo Li and David Saad. Phys. Rev. Lett. 120, 248301 – Published 12 June 2018
- deep-learning, statistical physics, machine learning