Adaptive back-propagation in on-line learning of multilayer networks

Ansgar H L West, David Saad

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

An adaptive back-propagation algorithm is studied and compared with gradient descent (standard back-propagation) for on-line learning in two-layer neural networks with an arbitrary number of hidden units. Within a statistical mechanics framework, both numerical studies and a rigorous analysis show that the adaptive back-propagation method results in faster training by breaking the symmetry between hidden units more efficiently and by providing faster convergence to optimal generalization than gradient descent.
Original languageEnglish
Title of host publicationProceedings of the neural information processing systems
EditorsDavid S Touretzky, Michael C Mozer, Michael E. Hasselmo
Place of PublicationBoston
PublisherMIT
Volume8
ISBN (Print)0262201070
Publication statusPublished - 1996
EventNeural Information Processing Systems 95 -
Duration: 1 Jan 19961 Jan 1996

Conference

ConferenceNeural Information Processing Systems 95
Period1/01/961/01/96

Fingerprint

Backpropagation
Multilayers
Statistical mechanics
Backpropagation algorithms
Adaptive algorithms
Neural networks

Bibliographical note

Copyright of the Massachusetts Institute of Technology Press (MIT Press)

Keywords

  • adaptive back-propagation
  • algorithm
  • gradient descent
  • neural networks
  • statistical

Cite this

West, A. H. L., & Saad, D. (1996). Adaptive back-propagation in on-line learning of multilayer networks. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Proceedings of the neural information processing systems (Vol. 8). Boston: MIT.
West, Ansgar H L ; Saad, David. / Adaptive back-propagation in on-line learning of multilayer networks. Proceedings of the neural information processing systems. editor / David S Touretzky ; Michael C Mozer ; Michael E. Hasselmo. Vol. 8 Boston : MIT, 1996.
@inbook{e99cc3edac6b4756ac61f653b41d8291,
title = "Adaptive back-propagation in on-line learning of multilayer networks",
abstract = "An adaptive back-propagation algorithm is studied and compared with gradient descent (standard back-propagation) for on-line learning in two-layer neural networks with an arbitrary number of hidden units. Within a statistical mechanics framework, both numerical studies and a rigorous analysis show that the adaptive back-propagation method results in faster training by breaking the symmetry between hidden units more efficiently and by providing faster convergence to optimal generalization than gradient descent.",
keywords = "adaptive back-propagation, algorithm, gradient descent, neural networks, statistical",
author = "West, {Ansgar H L} and David Saad",
note = "Copyright of the Massachusetts Institute of Technology Press (MIT Press)",
year = "1996",
language = "English",
isbn = "0262201070",
volume = "8",
editor = "Touretzky, {David S} and Mozer, {Michael C} and Hasselmo, {Michael E.}",
booktitle = "Proceedings of the neural information processing systems",
publisher = "MIT",

}

West, AHL & Saad, D 1996, Adaptive back-propagation in on-line learning of multilayer networks. in DS Touretzky, MC Mozer & ME Hasselmo (eds), Proceedings of the neural information processing systems. vol. 8, MIT, Boston, Neural Information Processing Systems 95, 1/01/96.

Adaptive back-propagation in on-line learning of multilayer networks. / West, Ansgar H L; Saad, David.

Proceedings of the neural information processing systems. ed. / David S Touretzky; Michael C Mozer; Michael E. Hasselmo. Vol. 8 Boston : MIT, 1996.

Research output: Chapter in Book/Report/Conference proceedingChapter

TY - CHAP

T1 - Adaptive back-propagation in on-line learning of multilayer networks

AU - West, Ansgar H L

AU - Saad, David

N1 - Copyright of the Massachusetts Institute of Technology Press (MIT Press)

PY - 1996

Y1 - 1996

N2 - An adaptive back-propagation algorithm is studied and compared with gradient descent (standard back-propagation) for on-line learning in two-layer neural networks with an arbitrary number of hidden units. Within a statistical mechanics framework, both numerical studies and a rigorous analysis show that the adaptive back-propagation method results in faster training by breaking the symmetry between hidden units more efficiently and by providing faster convergence to optimal generalization than gradient descent.

AB - An adaptive back-propagation algorithm is studied and compared with gradient descent (standard back-propagation) for on-line learning in two-layer neural networks with an arbitrary number of hidden units. Within a statistical mechanics framework, both numerical studies and a rigorous analysis show that the adaptive back-propagation method results in faster training by breaking the symmetry between hidden units more efficiently and by providing faster convergence to optimal generalization than gradient descent.

KW - adaptive back-propagation

KW - algorithm

KW - gradient descent

KW - neural networks

KW - statistical

UR - http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=8421

M3 - Chapter

SN - 0262201070

VL - 8

BT - Proceedings of the neural information processing systems

A2 - Touretzky, David S

A2 - Mozer, Michael C

A2 - Hasselmo, Michael E.

PB - MIT

CY - Boston

ER -

West AHL, Saad D. Adaptive back-propagation in on-line learning of multilayer networks. In Touretzky DS, Mozer MC, Hasselmo ME, editors, Proceedings of the neural information processing systems. Vol. 8. Boston: MIT. 1996