Transients and asymptotics of natural gradient learning

Magnus Rattray, David Saad

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

We analyse natural gradient learning in a two-layer feed-forward neural network using a statistical mechanics framework which is appropriate for large input dimension. We find significant improvement over standard gradient descent in both the transient and asymptotic phases of learning.
Original languageEnglish
Title of host publicationProceedings of the 8th International Conference on Artificial Neural Networks
EditorsL. Niklasson, M. Boden, T. Ziemke
PublisherSpringer
Pages165-170
Number of pages6
Volume1
ISBN (Print)3540762639
DOIs
Publication statusPublished - 1 Sep 1998

Fingerprint

Statistical mechanics
Feedforward neural networks

Bibliographical note

The original publication is available at www.springerlink.com

Keywords

  • natural gradient
  • statistical mechanics
  • gradient descent
  • transient
  • asymptotic

Cite this

Rattray, M., & Saad, D. (1998). Transients and asymptotics of natural gradient learning. In L. Niklasson, M. Boden, & T. Ziemke (Eds.), Proceedings of the 8th International Conference on Artificial Neural Networks (Vol. 1, pp. 165-170). Springer. https://doi.org/10.1007/978-1-4471-1599-1_21
Rattray, Magnus ; Saad, David. / Transients and asymptotics of natural gradient learning. Proceedings of the 8th International Conference on Artificial Neural Networks. editor / L. Niklasson ; M. Boden ; T. Ziemke. Vol. 1 Springer, 1998. pp. 165-170
@inbook{ff2c0833fae942f488fc92fa2d6e363a,
title = "Transients and asymptotics of natural gradient learning",
abstract = "We analyse natural gradient learning in a two-layer feed-forward neural network using a statistical mechanics framework which is appropriate for large input dimension. We find significant improvement over standard gradient descent in both the transient and asymptotic phases of learning.",
keywords = "natural gradient, statistical mechanics, gradient descent, transient, asymptotic",
author = "Magnus Rattray and David Saad",
note = "The original publication is available at www.springerlink.com",
year = "1998",
month = "9",
day = "1",
doi = "10.1007/978-1-4471-1599-1_21",
language = "English",
isbn = "3540762639",
volume = "1",
pages = "165--170",
editor = "L. Niklasson and M. Boden and T. Ziemke",
booktitle = "Proceedings of the 8th International Conference on Artificial Neural Networks",
publisher = "Springer",
address = "Germany",

}

Rattray, M & Saad, D 1998, Transients and asymptotics of natural gradient learning. in L Niklasson, M Boden & T Ziemke (eds), Proceedings of the 8th International Conference on Artificial Neural Networks. vol. 1, Springer, pp. 165-170. https://doi.org/10.1007/978-1-4471-1599-1_21

Transients and asymptotics of natural gradient learning. / Rattray, Magnus; Saad, David.

Proceedings of the 8th International Conference on Artificial Neural Networks. ed. / L. Niklasson; M. Boden; T. Ziemke. Vol. 1 Springer, 1998. p. 165-170.

Research output: Chapter in Book/Report/Conference proceedingChapter

TY - CHAP

T1 - Transients and asymptotics of natural gradient learning

AU - Rattray, Magnus

AU - Saad, David

N1 - The original publication is available at www.springerlink.com

PY - 1998/9/1

Y1 - 1998/9/1

N2 - We analyse natural gradient learning in a two-layer feed-forward neural network using a statistical mechanics framework which is appropriate for large input dimension. We find significant improvement over standard gradient descent in both the transient and asymptotic phases of learning.

AB - We analyse natural gradient learning in a two-layer feed-forward neural network using a statistical mechanics framework which is appropriate for large input dimension. We find significant improvement over standard gradient descent in both the transient and asymptotic phases of learning.

KW - natural gradient

KW - statistical mechanics

KW - gradient descent

KW - transient

KW - asymptotic

UR - https://link.springer.com/chapter/10.1007/978-1-4471-1599-1_21

U2 - 10.1007/978-1-4471-1599-1_21

DO - 10.1007/978-1-4471-1599-1_21

M3 - Chapter

SN - 3540762639

VL - 1

SP - 165

EP - 170

BT - Proceedings of the 8th International Conference on Artificial Neural Networks

A2 - Niklasson, L.

A2 - Boden, M.

A2 - Ziemke, T.

PB - Springer

ER -

Rattray M, Saad D. Transients and asymptotics of natural gradient learning. In Niklasson L, Boden M, Ziemke T, editors, Proceedings of the 8th International Conference on Artificial Neural Networks. Vol. 1. Springer. 1998. p. 165-170 https://doi.org/10.1007/978-1-4471-1599-1_21