The role of biases in on-line learning of two-layer networks

Ansgar H.L. West, David Saad

    Research output: Contribution to journalArticlepeer-review

    Abstract

    The influence of biases on the learning dynamics of a two-layer neural network, a normalized soft-committee machine, is studied for on-line gradient descent learning. Within a statistical mechanics framework, numerical studies show that the inclusion of adjustable biases dramatically alters the learning dynamics found previously. The symmetric phase which has often been predominant in the original model all but disappears for a non-degenerate bias task. The extended model furthermore exhibits a much richer dynamical behavior, e.g. attractive suboptimal symmetric phases even for realizable cases and noiseless data.
    Original languageEnglish
    Pages (from-to)3265-3291
    Number of pages27
    JournalPhysical Review E
    Volume57
    Issue number3
    DOIs
    Publication statusPublished - Mar 1998

    Bibliographical note

    Copyright of the American Physical Society

    Keywords

    • learning dynamics
    • two-layer neural network
    • soft-committee machine
    • on-line gradient descent learning

    Fingerprint

    Dive into the research topics of 'The role of biases in on-line learning of two-layer networks'. Together they form a unique fingerprint.

    Cite this