Natural gradient matrix momentum

Silvia Scarpetta, Magnus Rattray, David Saad

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

Natural gradient learning is an efficient and principled method for improving on-line learning. In practical applications there will be an increased cost required in estimating and inverting the Fisher information matrix. We propose to use the matrix momentum algorithm in order to carry out efficient inversion and study the efficacy of a single step estimation of the Fisher information matrix. We analyse the proposed algorithm in a two-layer network, using a statistical mechanics framework which allows us to describe analytically the learning dynamics, and compare performance with true natural gradient learning and standard gradient descent.
Original languageEnglish
Title of host publicationNinth International Conference on Artificial Neural Networks, ICANN 99
Place of PublicationEdinburgh UK
PublisherIEEE
Pages43-48
Number of pages6
Volume1
Publication statusPublished - 7 Sep 1999

Publication series

NameConference Publication
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Volume470

Bibliographical note

©1999 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

Keywords

  • Natural gradient learning
  • on-line learning
  • Fisher information matrix
  • matrix momentum algorithm
  • two-layer network

Fingerprint Dive into the research topics of 'Natural gradient matrix momentum'. Together they form a unique fingerprint.

  • Cite this

    Scarpetta, S., Rattray, M., & Saad, D. (1999). Natural gradient matrix momentum. In Ninth International Conference on Artificial Neural Networks, ICANN 99 (Vol. 1, pp. 43-48). (Conference Publication; Vol. 470). IEEE.