Regularisation of mixture density networks

Lars U. Hjorth

    Research output: Working paperTechnical report

    Abstract

    Mixture Density Networks are a principled method to model conditional probability density functions which are non-Gaussian. This is achieved by modelling the conditional distribution for each pattern with a Gaussian Mixture Model for which the parameters are generated by a neural network. This thesis presents a novel method to introduce regularisation in this context for the special case where the mean and variance of the spherical Gaussian Kernels in the mixtures are fixed to predetermined values. Guidelines for how these parameters can be initialised are given, and it is shown how to apply the evidence framework to mixture density networks to achieve regularisation. This also provides an objective stopping criteria that can replace the `early stopping' methods that have previously been used. If the neural network used is an RBF network with fixed centres this opens up new opportunities for improved initialisation of the network weights, which are exploited to start training relatively close to the optimum. The new method is demonstrated on two data sets. The first is a simple synthetic data set while the second is a real life data set, namely satellite scatterometer data used to infer the wind speed and wind direction near the ocean surface. For both data sets the regularisation method performs well in comparison with earlier published results. Ideas on how the constraint on the kernels may be relaxed to allow fully adaptable kernels are presented.
    Original languageEnglish
    Place of PublicationBirmingham
    PublisherAston University
    Number of pages62
    ISBN (Print)NCRG/99/004
    Publication statusPublished - 12 Feb 1999

    Fingerprint

    Neural networks
    Radial basis function networks
    Probability density function
    Satellites

    Keywords

    • NCRG
    • neural nets
    • Bayesian regularisation
    • maximum likelihood estimation
    • mixture density networks
    • multivalued functions
    • neural networks
    • probability

    Cite this

    Hjorth, L. U. (1999). Regularisation of mixture density networks. Birmingham: Aston University.
    Hjorth, Lars U. / Regularisation of mixture density networks. Birmingham : Aston University, 1999.
    @techreport{a4a4a56787864b1ea2e27e4b92958b7f,
    title = "Regularisation of mixture density networks",
    abstract = "Mixture Density Networks are a principled method to model conditional probability density functions which are non-Gaussian. This is achieved by modelling the conditional distribution for each pattern with a Gaussian Mixture Model for which the parameters are generated by a neural network. This thesis presents a novel method to introduce regularisation in this context for the special case where the mean and variance of the spherical Gaussian Kernels in the mixtures are fixed to predetermined values. Guidelines for how these parameters can be initialised are given, and it is shown how to apply the evidence framework to mixture density networks to achieve regularisation. This also provides an objective stopping criteria that can replace the `early stopping' methods that have previously been used. If the neural network used is an RBF network with fixed centres this opens up new opportunities for improved initialisation of the network weights, which are exploited to start training relatively close to the optimum. The new method is demonstrated on two data sets. The first is a simple synthetic data set while the second is a real life data set, namely satellite scatterometer data used to infer the wind speed and wind direction near the ocean surface. For both data sets the regularisation method performs well in comparison with earlier published results. Ideas on how the constraint on the kernels may be relaxed to allow fully adaptable kernels are presented.",
    keywords = "NCRG, neural nets, Bayesian regularisation, maximum likelihood estimation, mixture density networks, multivalued functions, neural networks, probability",
    author = "Hjorth, {Lars U.}",
    year = "1999",
    month = "2",
    day = "12",
    language = "English",
    isbn = "NCRG/99/004",
    publisher = "Aston University",
    type = "WorkingPaper",
    institution = "Aston University",

    }

    Hjorth, LU 1999 'Regularisation of mixture density networks' Aston University, Birmingham.

    Regularisation of mixture density networks. / Hjorth, Lars U.

    Birmingham : Aston University, 1999.

    Research output: Working paperTechnical report

    TY - UNPB

    T1 - Regularisation of mixture density networks

    AU - Hjorth, Lars U.

    PY - 1999/2/12

    Y1 - 1999/2/12

    N2 - Mixture Density Networks are a principled method to model conditional probability density functions which are non-Gaussian. This is achieved by modelling the conditional distribution for each pattern with a Gaussian Mixture Model for which the parameters are generated by a neural network. This thesis presents a novel method to introduce regularisation in this context for the special case where the mean and variance of the spherical Gaussian Kernels in the mixtures are fixed to predetermined values. Guidelines for how these parameters can be initialised are given, and it is shown how to apply the evidence framework to mixture density networks to achieve regularisation. This also provides an objective stopping criteria that can replace the `early stopping' methods that have previously been used. If the neural network used is an RBF network with fixed centres this opens up new opportunities for improved initialisation of the network weights, which are exploited to start training relatively close to the optimum. The new method is demonstrated on two data sets. The first is a simple synthetic data set while the second is a real life data set, namely satellite scatterometer data used to infer the wind speed and wind direction near the ocean surface. For both data sets the regularisation method performs well in comparison with earlier published results. Ideas on how the constraint on the kernels may be relaxed to allow fully adaptable kernels are presented.

    AB - Mixture Density Networks are a principled method to model conditional probability density functions which are non-Gaussian. This is achieved by modelling the conditional distribution for each pattern with a Gaussian Mixture Model for which the parameters are generated by a neural network. This thesis presents a novel method to introduce regularisation in this context for the special case where the mean and variance of the spherical Gaussian Kernels in the mixtures are fixed to predetermined values. Guidelines for how these parameters can be initialised are given, and it is shown how to apply the evidence framework to mixture density networks to achieve regularisation. This also provides an objective stopping criteria that can replace the `early stopping' methods that have previously been used. If the neural network used is an RBF network with fixed centres this opens up new opportunities for improved initialisation of the network weights, which are exploited to start training relatively close to the optimum. The new method is demonstrated on two data sets. The first is a simple synthetic data set while the second is a real life data set, namely satellite scatterometer data used to infer the wind speed and wind direction near the ocean surface. For both data sets the regularisation method performs well in comparison with earlier published results. Ideas on how the constraint on the kernels may be relaxed to allow fully adaptable kernels are presented.

    KW - NCRG

    KW - neural nets

    KW - Bayesian regularisation

    KW - maximum likelihood estimation

    KW - mixture density networks

    KW - multivalued functions

    KW - neural networks

    KW - probability

    M3 - Technical report

    SN - NCRG/99/004

    BT - Regularisation of mixture density networks

    PB - Aston University

    CY - Birmingham

    ER -

    Hjorth LU. Regularisation of mixture density networks. Birmingham: Aston University. 1999 Feb 12.