Exploring the Function Space of Deep-Learning Machines

Bo Li; David Saad

doi:10.1103/PhysRevLett.120.248301

Exploring the Function Space of Deep-Learning Machines

Bo Li
, David Saad

Department of Physics
Hong Kong University of Science and Technology

Research output: Contribution to journal › Article › peer-review

19 Citations (Scopus)

141 Downloads (Pure)

Abstract

The function space of deep-learning machines is investigated by studying growth in the entropy of functions of a given error with respect to a reference function, realized by a deep-learning machine. Using physics-inspired methods we study both sparsely and densely connected architectures to discover a layerwise convergence of candidate functions, marked by a corresponding reduction in entropy when approaching the reference function, gain insight into the importance of having a large number of layers, and observe phase transitions as the error increases.

Original language	English
Article number	248301
Journal	Physical Review Letters
Volume	120
Issue number	24
Early online date	12 Jun 2018
DOIs	https://doi.org/10.1103/PhysRevLett.120.248301
Publication status	Published - 12 Jun 2018

Bibliographical note

Keywords

deep-learning, statistical physics, machine learning

Access to Document

10.1103/PhysRevLett.120.248301

Exploring the Function Space of Deep-Learning Machines
© 2018 The American Physical Society. Exploring the Function Space of Deep-Learning Machines. Bo Li and David Saad. Phys. Rev. Lett. 120, 248301 – Published 12 June 2018
Accepted author manuscript, 497 KB

https://arxiv.org/abs/1708.01422

Cite this

@article{3502cf15b5fb48e2a69963431caf6e67,

title = "Exploring the Function Space of Deep-Learning Machines",

abstract = "The function space of deep-learning machines is investigated by studying growth in the entropy of functions of a given error with respect to a reference function, realized by a deep-learning machine. Using physics-inspired methods we study both sparsely and densely connected architectures to discover a layerwise convergence of candidate functions, marked by a corresponding reduction in entropy when approaching the reference function, gain insight into the importance of having a large number of layers, and observe phase transitions as the error increases.",

keywords = "deep-learning, statistical physics, machine learning",

author = "Bo Li and David Saad",

note = "{\textcopyright} 2018 The American Physical Society. Exploring the Function Space of Deep-Learning Machines. Bo Li and David Saad. Phys. Rev. Lett. 120, 248301 – Published 12 June 2018 ",

year = "2018",

month = jun,

day = "12",

doi = "10.1103/PhysRevLett.120.248301",

language = "English",

volume = "120",

journal = "Physical Review Letters",

issn = "0031-9007",

publisher = "American Physical Society",

number = "24",

}

TY - JOUR

T1 - Exploring the Function Space of Deep-Learning Machines

AU - Li, Bo

AU - Saad, David

PY - 2018/6/12

Y1 - 2018/6/12

N2 - The function space of deep-learning machines is investigated by studying growth in the entropy of functions of a given error with respect to a reference function, realized by a deep-learning machine. Using physics-inspired methods we study both sparsely and densely connected architectures to discover a layerwise convergence of candidate functions, marked by a corresponding reduction in entropy when approaching the reference function, gain insight into the importance of having a large number of layers, and observe phase transitions as the error increases.

AB - The function space of deep-learning machines is investigated by studying growth in the entropy of functions of a given error with respect to a reference function, realized by a deep-learning machine. Using physics-inspired methods we study both sparsely and densely connected architectures to discover a layerwise convergence of candidate functions, marked by a corresponding reduction in entropy when approaching the reference function, gain insight into the importance of having a large number of layers, and observe phase transitions as the error increases.

KW - deep-learning, statistical physics, machine learning

UR - http://www.scopus.com/inward/record.url?scp=85048526527&partnerID=8YFLogxK

U2 - 10.1103/PhysRevLett.120.248301

DO - 10.1103/PhysRevLett.120.248301

M3 - Article

AN - SCOPUS:85048526527

SN - 0031-9007

VL - 120

JO - Physical Review Letters

JF - Physical Review Letters

IS - 24

M1 - 248301

ER -

Exploring the Function Space of Deep-Learning Machines

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this