Gaussian processes for classification: mean field algorithms: mean-field algorithms

Manfred Opper, Ole Winther

    Research output: Contribution to journalArticle

    Abstract

    We derive a mean field algorithm for binary classification with Gaussian processes which is based on the TAP approach originally proposed in Statistical Physics of disordered systems. The theory also yields an approximate leave-one-out estimator for the generalization error which is computed with no extra computational cost. We show that from the TAP approach, it is possible to derive both a simpler 'naive' mean field theory and support vector machines (SVM) as limiting cases. For both mean field algorithms and support vectors machines, simulation results for three small benchmark data sets are presented. They show 1. that one may get state of the art performance by using the leave-one-out estimator for model selection and 2. the built-in leave-one-out estimators are extremely precise when compared to the exact leave-one-out estimate. The latter result is a taken as a strong support for the internal consistency of the mean field approach.
    Original languageEnglish
    Pages (from-to)2655-2684
    Number of pages30
    JournalNeural Computation
    Volume12
    Issue number11
    DOIs
    Publication statusPublished - Nov 2000

    Fingerprint

    Support vector machines
    Benchmarking
    Mean field theory
    Physics
    Costs and Cost Analysis
    Costs
    Support Vector Machine
    Datasets
    Field Theory
    Computational
    Performance Art
    Model Selection
    Benchmark
    Statistical Physics
    Simulation

    Keywords

    • mean field algorithm
    • binary classification
    • Gaussian processes
    • TAP approach
    • Statistical Physics
    • disordered systems
    • estimator
    • generalization error
    • computational cost
    • support vector machines (SVM)

    Cite this

    Opper, Manfred ; Winther, Ole. / Gaussian processes for classification: mean field algorithms : mean-field algorithms. In: Neural Computation. 2000 ; Vol. 12, No. 11. pp. 2655-2684.
    @article{263575af66d74a778bd02638e95a22b6,
    title = "Gaussian processes for classification: mean field algorithms: mean-field algorithms",
    abstract = "We derive a mean field algorithm for binary classification with Gaussian processes which is based on the TAP approach originally proposed in Statistical Physics of disordered systems. The theory also yields an approximate leave-one-out estimator for the generalization error which is computed with no extra computational cost. We show that from the TAP approach, it is possible to derive both a simpler 'naive' mean field theory and support vector machines (SVM) as limiting cases. For both mean field algorithms and support vectors machines, simulation results for three small benchmark data sets are presented. They show 1. that one may get state of the art performance by using the leave-one-out estimator for model selection and 2. the built-in leave-one-out estimators are extremely precise when compared to the exact leave-one-out estimate. The latter result is a taken as a strong support for the internal consistency of the mean field approach.",
    keywords = "mean field algorithm, binary classification, Gaussian processes, TAP approach, Statistical Physics, disordered systems, estimator, generalization error, computational cost, support vector machines (SVM)",
    author = "Manfred Opper and Ole Winther",
    year = "2000",
    month = "11",
    doi = "10.1162/089976600300014881",
    language = "English",
    volume = "12",
    pages = "2655--2684",
    journal = "Neural Computation",
    issn = "0899-7667",
    publisher = "MIT Press Journals",
    number = "11",

    }

    Gaussian processes for classification: mean field algorithms : mean-field algorithms. / Opper, Manfred; Winther, Ole.

    In: Neural Computation, Vol. 12, No. 11, 11.2000, p. 2655-2684.

    Research output: Contribution to journalArticle

    TY - JOUR

    T1 - Gaussian processes for classification: mean field algorithms

    T2 - mean-field algorithms

    AU - Opper, Manfred

    AU - Winther, Ole

    PY - 2000/11

    Y1 - 2000/11

    N2 - We derive a mean field algorithm for binary classification with Gaussian processes which is based on the TAP approach originally proposed in Statistical Physics of disordered systems. The theory also yields an approximate leave-one-out estimator for the generalization error which is computed with no extra computational cost. We show that from the TAP approach, it is possible to derive both a simpler 'naive' mean field theory and support vector machines (SVM) as limiting cases. For both mean field algorithms and support vectors machines, simulation results for three small benchmark data sets are presented. They show 1. that one may get state of the art performance by using the leave-one-out estimator for model selection and 2. the built-in leave-one-out estimators are extremely precise when compared to the exact leave-one-out estimate. The latter result is a taken as a strong support for the internal consistency of the mean field approach.

    AB - We derive a mean field algorithm for binary classification with Gaussian processes which is based on the TAP approach originally proposed in Statistical Physics of disordered systems. The theory also yields an approximate leave-one-out estimator for the generalization error which is computed with no extra computational cost. We show that from the TAP approach, it is possible to derive both a simpler 'naive' mean field theory and support vector machines (SVM) as limiting cases. For both mean field algorithms and support vectors machines, simulation results for three small benchmark data sets are presented. They show 1. that one may get state of the art performance by using the leave-one-out estimator for model selection and 2. the built-in leave-one-out estimators are extremely precise when compared to the exact leave-one-out estimate. The latter result is a taken as a strong support for the internal consistency of the mean field approach.

    KW - mean field algorithm

    KW - binary classification

    KW - Gaussian processes

    KW - TAP approach

    KW - Statistical Physics

    KW - disordered systems

    KW - estimator

    KW - generalization error

    KW - computational cost

    KW - support vector machines (SVM)

    UR - http://www.scopus.com/inward/record.url?scp=0034320350&partnerID=8YFLogxK

    UR - http://www.mitpressjournals.org/doi/abs/10.1162/089976600300014881?prevSearch=allfield%253A%2528Gaussian%2Bprocesses%2Bfor%2Bclassification%255C%253A%2BMean-field%2Balgorithms%2529&searchHistoryKey=

    U2 - 10.1162/089976600300014881

    DO - 10.1162/089976600300014881

    M3 - Article

    C2 - 11110131

    VL - 12

    SP - 2655

    EP - 2684

    JO - Neural Computation

    JF - Neural Computation

    SN - 0899-7667

    IS - 11

    ER -