Finite-size effects in on-line learning of multilayer neural networks

David Barber, David Saad, Peter Sollich

Research output: Contribution to journalArticle

Abstract

We complement recent advances in thermodynamic limit analyses of mean on-line gradient descent learning dynamics in multi-layer networks by calculating fluctuations possessed by finite dimensional systems. Fluctuations from the mean dynamics are largest at the onset of specialisation as student hidden unit weight vectors begin to imitate specific teacher vectors, increasing with the degree of symmetry of the initial conditions. In light of this, we include a term to stimulate asymmetry in the learning process, which typically also leads to a significant decrease in training time.
Original languageEnglish
Pages (from-to)151-156
Number of pages6
JournalEurophysics Letters
Volume34
Issue number2
Publication statusPublished - Apr 1996

Fingerprint

learning
instructors
descent
complement
students
education
asymmetry
gradients
thermodynamics
symmetry

Bibliographical note

Copyright of EDP Sciences

Keywords

  • probability theory
  • stochastic processes
  • and statistics

Cite this

Barber, David ; Saad, David ; Sollich, Peter. / Finite-size effects in on-line learning of multilayer neural networks. In: Europhysics Letters. 1996 ; Vol. 34, No. 2. pp. 151-156.
@article{52d97381056e4951b9143688617835e9,
title = "Finite-size effects in on-line learning of multilayer neural networks",
abstract = "We complement recent advances in thermodynamic limit analyses of mean on-line gradient descent learning dynamics in multi-layer networks by calculating fluctuations possessed by finite dimensional systems. Fluctuations from the mean dynamics are largest at the onset of specialisation as student hidden unit weight vectors begin to imitate specific teacher vectors, increasing with the degree of symmetry of the initial conditions. In light of this, we include a term to stimulate asymmetry in the learning process, which typically also leads to a significant decrease in training time.",
keywords = "probability theory, stochastic processes, and statistics",
author = "David Barber and David Saad and Peter Sollich",
note = "Copyright of EDP Sciences",
year = "1996",
month = "4",
language = "English",
volume = "34",
pages = "151--156",
journal = "Europhysics Letters",
issn = "0295-5075",
publisher = "IOP Publishing Ltd.",
number = "2",

}

Finite-size effects in on-line learning of multilayer neural networks. / Barber, David; Saad, David; Sollich, Peter.

In: Europhysics Letters, Vol. 34, No. 2, 04.1996, p. 151-156.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Finite-size effects in on-line learning of multilayer neural networks

AU - Barber, David

AU - Saad, David

AU - Sollich, Peter

N1 - Copyright of EDP Sciences

PY - 1996/4

Y1 - 1996/4

N2 - We complement recent advances in thermodynamic limit analyses of mean on-line gradient descent learning dynamics in multi-layer networks by calculating fluctuations possessed by finite dimensional systems. Fluctuations from the mean dynamics are largest at the onset of specialisation as student hidden unit weight vectors begin to imitate specific teacher vectors, increasing with the degree of symmetry of the initial conditions. In light of this, we include a term to stimulate asymmetry in the learning process, which typically also leads to a significant decrease in training time.

AB - We complement recent advances in thermodynamic limit analyses of mean on-line gradient descent learning dynamics in multi-layer networks by calculating fluctuations possessed by finite dimensional systems. Fluctuations from the mean dynamics are largest at the onset of specialisation as student hidden unit weight vectors begin to imitate specific teacher vectors, increasing with the degree of symmetry of the initial conditions. In light of this, we include a term to stimulate asymmetry in the learning process, which typically also leads to a significant decrease in training time.

KW - probability theory

KW - stochastic processes

KW - and statistics

UR - http://iopscience.iop.org/0295-5075/34/2/151/fulltext?ejredirect=.iopscience

M3 - Article

VL - 34

SP - 151

EP - 156

JO - Europhysics Letters

JF - Europhysics Letters

SN - 0295-5075

IS - 2

ER -