The dynamics of matrix momentum

Magnus Rattray, David Saad

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Newton's method, by extending a recent statistical mechanics framework to include second order algorithms. We study the efficacy of this method when the Hessian is available and also consider a practical implementation which uses a single example estimate of the Hessian. The method is shown to provide excellent asymptotic performance, although the single example implementation is sensitive to the choice of training parameters. We conjecture that matrix momentum could provide efficient matrix inversion for other second order algorithms.
Original languageEnglish
Title of host publicationProceedings of the 8th International Conference on Artificial Neural Networks
EditorsLars F. Niklasson, Mikael B. Boden, Tom Ziemke
PublisherSpringer
Pages183-188
Number of pages6
Volume1
ISBN (Print)3540762639
DOIs
Publication statusPublished - 1 Sep 1998

Bibliographical note

The original publication is available at www.springerlink.com

Keywords

  • matrix momentum
  • statistical mechanics
  • asymptotic performance
  • matrix inversion
  • Hessian

Fingerprint Dive into the research topics of 'The dynamics of matrix momentum'. Together they form a unique fingerprint.

  • Cite this

    Rattray, M., & Saad, D. (1998). The dynamics of matrix momentum. In L. F. Niklasson, M. B. Boden, & T. Ziemke (Eds.), Proceedings of the 8th International Conference on Artificial Neural Networks (Vol. 1, pp. 183-188). Springer. https://doi.org/10.1007/978-1-4471-1599-1_24