Noise, regularizers, and unrealizable scenarios in online learning from restricted training sets

Yuan-Sheng Xiong, David Saad

Research output: Contribution to journalArticle

Abstract

We study the dynamics of on-line learning in multilayer neural networks where training examples are sampled with repetition and where the number of examples scales with the number of network weights. The analysis is carried out using the dynamical replica method aimed at obtaining a closed set of coupled equations for a set of macroscopic variables from which both training and generalization errors can be calculated. We focus on scenarios whereby training examples are corrupted by additive Gaussian output noise and regularizers are introduced to improve the network performance. The dependence of the dynamics on the noise level, with and without regularizers, is examined, as well as that of the asymptotic values obtained for both training and generalization errors. We also demonstrate the ability of the method to approximate the learning dynamics in structurally unrealizable scenarios. The theoretical results show good agreement with those obtained by computer simulations.
Original languageEnglish
Article number011919
Pages (from-to)1-18
Number of pages18
JournalPhysical Review E
Volume64
Issue number1
DOIs
Publication statusPublished - 27 Jun 2001

Fingerprint

Online Learning
learning
education
Generalization Error
Scenarios
Replica Method
Multilayer Neural Network
Network Performance
Closed set
replicas
repetition
Computer Simulation
computerized simulation
Training
output
Output
Demonstrate
Learning

Bibliographical note

Copyright of the American Physical Society

Keywords

  • on-line learning
  • multilayer neural networks
  • dynamical replica method
  • network performance
  • noise level

Cite this

@article{d45af05c71e846c59ab10d49ebd9934b,
title = "Noise, regularizers, and unrealizable scenarios in online learning from restricted training sets",
abstract = "We study the dynamics of on-line learning in multilayer neural networks where training examples are sampled with repetition and where the number of examples scales with the number of network weights. The analysis is carried out using the dynamical replica method aimed at obtaining a closed set of coupled equations for a set of macroscopic variables from which both training and generalization errors can be calculated. We focus on scenarios whereby training examples are corrupted by additive Gaussian output noise and regularizers are introduced to improve the network performance. The dependence of the dynamics on the noise level, with and without regularizers, is examined, as well as that of the asymptotic values obtained for both training and generalization errors. We also demonstrate the ability of the method to approximate the learning dynamics in structurally unrealizable scenarios. The theoretical results show good agreement with those obtained by computer simulations.",
keywords = "on-line learning, multilayer neural networks, dynamical replica method, network performance, noise level",
author = "Yuan-Sheng Xiong and David Saad",
note = "Copyright of the American Physical Society",
year = "2001",
month = "6",
day = "27",
doi = "10.1103/PhysRevE.64.011919",
language = "English",
volume = "64",
pages = "1--18",
journal = "Physical Review E",
issn = "1539-3755",
publisher = "American Physical Society",
number = "1",

}

Noise, regularizers, and unrealizable scenarios in online learning from restricted training sets. / Xiong, Yuan-Sheng; Saad, David.

In: Physical Review E, Vol. 64, No. 1, 011919, 27.06.2001, p. 1-18.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Noise, regularizers, and unrealizable scenarios in online learning from restricted training sets

AU - Xiong, Yuan-Sheng

AU - Saad, David

N1 - Copyright of the American Physical Society

PY - 2001/6/27

Y1 - 2001/6/27

N2 - We study the dynamics of on-line learning in multilayer neural networks where training examples are sampled with repetition and where the number of examples scales with the number of network weights. The analysis is carried out using the dynamical replica method aimed at obtaining a closed set of coupled equations for a set of macroscopic variables from which both training and generalization errors can be calculated. We focus on scenarios whereby training examples are corrupted by additive Gaussian output noise and regularizers are introduced to improve the network performance. The dependence of the dynamics on the noise level, with and without regularizers, is examined, as well as that of the asymptotic values obtained for both training and generalization errors. We also demonstrate the ability of the method to approximate the learning dynamics in structurally unrealizable scenarios. The theoretical results show good agreement with those obtained by computer simulations.

AB - We study the dynamics of on-line learning in multilayer neural networks where training examples are sampled with repetition and where the number of examples scales with the number of network weights. The analysis is carried out using the dynamical replica method aimed at obtaining a closed set of coupled equations for a set of macroscopic variables from which both training and generalization errors can be calculated. We focus on scenarios whereby training examples are corrupted by additive Gaussian output noise and regularizers are introduced to improve the network performance. The dependence of the dynamics on the noise level, with and without regularizers, is examined, as well as that of the asymptotic values obtained for both training and generalization errors. We also demonstrate the ability of the method to approximate the learning dynamics in structurally unrealizable scenarios. The theoretical results show good agreement with those obtained by computer simulations.

KW - on-line learning

KW - multilayer neural networks

KW - dynamical replica method

KW - network performance

KW - noise level

UR - http://www.scopus.com/inward/record.url?scp=0035396879&partnerID=8YFLogxK

UR - http://journals.aps.org/pre/abstract/10.1103/PhysRevE.64.011919

U2 - 10.1103/PhysRevE.64.011919

DO - 10.1103/PhysRevE.64.011919

M3 - Article

VL - 64

SP - 1

EP - 18

JO - Physical Review E

JF - Physical Review E

SN - 1539-3755

IS - 1

M1 - 011919

ER -