Pruning reservoirs with random static projections

J.B. Butcher, C.R. Day, P.W. Haycock, D. Verstraeten, B. Schrauwen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Reservoir Computing is a relatively new field of Recurrent Neural Networks in which only the output weights are re-calculated by the training process, removing the problems associated with traditional gradient descent algorithms. As the reservoir is recurrent, it can possess short term memory, but there is a trade-off between the amount of memory a reservoir can have and its nonlinear mapping capabilities. A new, custom architecture was recently proposed to overcome this by combining a reservoir with an extreme learning machine to deliver improved results. This paper extends this architecture further by introducing a ranking and pruning algorithm which removes neurons according to their significance. This provides further insight into the type of reservoir characteristics needed for a given task, and is supported by further reservoir measures of non-linearity and memory. These techniques are demonstrated on artificial and real world data.
Original languageEnglish
Title of host publicationProceedings of the 2010 IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2010
PublisherIEEE
ISBN (Electronic)978-1-4244-7877-4
ISBN (Print)978-1-4244-7875-0
DOIs
Publication statusPublished - 7 Oct 2010
Event2010 IEEE International Workshop on Machine Learning for Signal Processing - Kittila, Finland
Duration: 29 Aug 20101 Sep 2010

Conference

Conference2010 IEEE International Workshop on Machine Learning for Signal Processing
CountryFinland
CityKittila
Period29/08/101/09/10

Fingerprint

Data storage equipment
Recurrent neural networks
Neurons
Learning systems

Cite this

Butcher, J. B., Day, C. R., Haycock, P. W., Verstraeten, D., & Schrauwen, B. (2010). Pruning reservoirs with random static projections. In Proceedings of the 2010 IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2010 IEEE. https://doi.org/10.1109/MLSP.2010.5589251
Butcher, J.B. ; Day, C.R. ; Haycock, P.W. ; Verstraeten, D. ; Schrauwen, B. / Pruning reservoirs with random static projections. Proceedings of the 2010 IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2010. IEEE, 2010.
@inproceedings{776c47c8279a4b48bb257d647e8ddab7,
title = "Pruning reservoirs with random static projections",
abstract = "Reservoir Computing is a relatively new field of Recurrent Neural Networks in which only the output weights are re-calculated by the training process, removing the problems associated with traditional gradient descent algorithms. As the reservoir is recurrent, it can possess short term memory, but there is a trade-off between the amount of memory a reservoir can have and its nonlinear mapping capabilities. A new, custom architecture was recently proposed to overcome this by combining a reservoir with an extreme learning machine to deliver improved results. This paper extends this architecture further by introducing a ranking and pruning algorithm which removes neurons according to their significance. This provides further insight into the type of reservoir characteristics needed for a given task, and is supported by further reservoir measures of non-linearity and memory. These techniques are demonstrated on artificial and real world data.",
author = "J.B. Butcher and C.R. Day and P.W. Haycock and D. Verstraeten and B. Schrauwen",
year = "2010",
month = "10",
day = "7",
doi = "10.1109/MLSP.2010.5589251",
language = "English",
isbn = "978-1-4244-7875-0",
booktitle = "Proceedings of the 2010 IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2010",
publisher = "IEEE",
address = "United States",

}

Butcher, JB, Day, CR, Haycock, PW, Verstraeten, D & Schrauwen, B 2010, Pruning reservoirs with random static projections. in Proceedings of the 2010 IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2010. IEEE, 2010 IEEE International Workshop on Machine Learning for Signal Processing, Kittila, Finland, 29/08/10. https://doi.org/10.1109/MLSP.2010.5589251

Pruning reservoirs with random static projections. / Butcher, J.B.; Day, C.R.; Haycock, P.W.; Verstraeten, D.; Schrauwen, B.

Proceedings of the 2010 IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2010. IEEE, 2010.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Pruning reservoirs with random static projections

AU - Butcher, J.B.

AU - Day, C.R.

AU - Haycock, P.W.

AU - Verstraeten, D.

AU - Schrauwen, B.

PY - 2010/10/7

Y1 - 2010/10/7

N2 - Reservoir Computing is a relatively new field of Recurrent Neural Networks in which only the output weights are re-calculated by the training process, removing the problems associated with traditional gradient descent algorithms. As the reservoir is recurrent, it can possess short term memory, but there is a trade-off between the amount of memory a reservoir can have and its nonlinear mapping capabilities. A new, custom architecture was recently proposed to overcome this by combining a reservoir with an extreme learning machine to deliver improved results. This paper extends this architecture further by introducing a ranking and pruning algorithm which removes neurons according to their significance. This provides further insight into the type of reservoir characteristics needed for a given task, and is supported by further reservoir measures of non-linearity and memory. These techniques are demonstrated on artificial and real world data.

AB - Reservoir Computing is a relatively new field of Recurrent Neural Networks in which only the output weights are re-calculated by the training process, removing the problems associated with traditional gradient descent algorithms. As the reservoir is recurrent, it can possess short term memory, but there is a trade-off between the amount of memory a reservoir can have and its nonlinear mapping capabilities. A new, custom architecture was recently proposed to overcome this by combining a reservoir with an extreme learning machine to deliver improved results. This paper extends this architecture further by introducing a ranking and pruning algorithm which removes neurons according to their significance. This provides further insight into the type of reservoir characteristics needed for a given task, and is supported by further reservoir measures of non-linearity and memory. These techniques are demonstrated on artificial and real world data.

UR - http://www.scopus.com/inward/record.url?eid=2-s2.0-78449293605&partnerID=MN8TOARS

UR - https://ieeexplore.ieee.org/document/5589251

U2 - 10.1109/MLSP.2010.5589251

DO - 10.1109/MLSP.2010.5589251

M3 - Conference contribution

SN - 978-1-4244-7875-0

BT - Proceedings of the 2010 IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2010

PB - IEEE

ER -

Butcher JB, Day CR, Haycock PW, Verstraeten D, Schrauwen B. Pruning reservoirs with random static projections. In Proceedings of the 2010 IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2010. IEEE. 2010 https://doi.org/10.1109/MLSP.2010.5589251