Pruning reservoirs with random static projections

J.B. Butcher; C.R. Day; P.W. Haycock; D. Verstraeten; B. Schrauwen

doi:10.1109/MLSP.2010.5589251

Pruning reservoirs with random static projections

J.B. Butcher, C.R. Day, P.W. Haycock, D. Verstraeten, B. Schrauwen

Research output: Chapter in Book/Published conference output › Conference publication

Abstract

Reservoir Computing is a relatively new field of Recurrent Neural Networks in which only the output weights are re-calculated by the training process, removing the problems associated with traditional gradient descent algorithms. As the reservoir is recurrent, it can possess short term memory, but there is a trade-off between the amount of memory a reservoir can have and its nonlinear mapping capabilities. A new, custom architecture was recently proposed to overcome this by combining a reservoir with an extreme learning machine to deliver improved results. This paper extends this architecture further by introducing a ranking and pruning algorithm which removes neurons according to their significance. This provides further insight into the type of reservoir characteristics needed for a given task, and is supported by further reservoir measures of non-linearity and memory. These techniques are demonstrated on artificial and real world data.

Original language	English
Title of host publication	Proceedings of the 2010 IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2010
Publisher	IEEE
ISBN (Electronic)	978-1-4244-7877-4
ISBN (Print)	978-1-4244-7875-0
DOIs	https://doi.org/10.1109/MLSP.2010.5589251
Publication status	Published - 7 Oct 2010
Event	2010 IEEE International Workshop on Machine Learning for Signal Processing - Kittila, Finland Duration: 29 Aug 2010 → 1 Sept 2010

Conference

Conference	2010 IEEE International Workshop on Machine Learning for Signal Processing
Country/Territory	Finland
City	Kittila
Period	29/08/10 → 1/09/10

Access to Document

10.1109/MLSP.2010.5589251

Cite this

@inproceedings{776c47c8279a4b48bb257d647e8ddab7,

title = "Pruning reservoirs with random static projections",

abstract = "Reservoir Computing is a relatively new field of Recurrent Neural Networks in which only the output weights are re-calculated by the training process, removing the problems associated with traditional gradient descent algorithms. As the reservoir is recurrent, it can possess short term memory, but there is a trade-off between the amount of memory a reservoir can have and its nonlinear mapping capabilities. A new, custom architecture was recently proposed to overcome this by combining a reservoir with an extreme learning machine to deliver improved results. This paper extends this architecture further by introducing a ranking and pruning algorithm which removes neurons according to their significance. This provides further insight into the type of reservoir characteristics needed for a given task, and is supported by further reservoir measures of non-linearity and memory. These techniques are demonstrated on artificial and real world data.",

author = "J.B. Butcher and C.R. Day and P.W. Haycock and D. Verstraeten and B. Schrauwen",

year = "2010",

month = oct,

day = "7",

doi = "10.1109/MLSP.2010.5589251",

language = "English",

isbn = "978-1-4244-7875-0",

booktitle = "Proceedings of the 2010 IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2010",

publisher = "IEEE",

address = "United States",

note = "2010 IEEE International Workshop on Machine Learning for Signal Processing ; Conference date: 29-08-2010 Through 01-09-2010",

}

Butcher, JB, Day, CR, Haycock, PW, Verstraeten, D & Schrauwen, B 2010, Pruning reservoirs with random static projections. in Proceedings of the 2010 IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2010. IEEE, 2010 IEEE International Workshop on Machine Learning for Signal Processing, Kittila, Finland, 29/08/10. https://doi.org/10.1109/MLSP.2010.5589251

TY - GEN

T1 - Pruning reservoirs with random static projections

AU - Butcher, J.B.

AU - Day, C.R.

AU - Haycock, P.W.

AU - Verstraeten, D.

AU - Schrauwen, B.

PY - 2010/10/7

Y1 - 2010/10/7

N2 - Reservoir Computing is a relatively new field of Recurrent Neural Networks in which only the output weights are re-calculated by the training process, removing the problems associated with traditional gradient descent algorithms. As the reservoir is recurrent, it can possess short term memory, but there is a trade-off between the amount of memory a reservoir can have and its nonlinear mapping capabilities. A new, custom architecture was recently proposed to overcome this by combining a reservoir with an extreme learning machine to deliver improved results. This paper extends this architecture further by introducing a ranking and pruning algorithm which removes neurons according to their significance. This provides further insight into the type of reservoir characteristics needed for a given task, and is supported by further reservoir measures of non-linearity and memory. These techniques are demonstrated on artificial and real world data.

AB - Reservoir Computing is a relatively new field of Recurrent Neural Networks in which only the output weights are re-calculated by the training process, removing the problems associated with traditional gradient descent algorithms. As the reservoir is recurrent, it can possess short term memory, but there is a trade-off between the amount of memory a reservoir can have and its nonlinear mapping capabilities. A new, custom architecture was recently proposed to overcome this by combining a reservoir with an extreme learning machine to deliver improved results. This paper extends this architecture further by introducing a ranking and pruning algorithm which removes neurons according to their significance. This provides further insight into the type of reservoir characteristics needed for a given task, and is supported by further reservoir measures of non-linearity and memory. These techniques are demonstrated on artificial and real world data.

UR - http://www.scopus.com/inward/record.url?eid=2-s2.0-78449293605&partnerID=MN8TOARS

UR - https://ieeexplore.ieee.org/document/5589251

U2 - 10.1109/MLSP.2010.5589251

DO - 10.1109/MLSP.2010.5589251

M3 - Conference publication

SN - 978-1-4244-7875-0

BT - Proceedings of the 2010 IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2010

PB - IEEE

T2 - 2010 IEEE International Workshop on Machine Learning for Signal Processing

Y2 - 29 August 2010 through 1 September 2010

ER -

Pruning reservoirs with random static projections

Abstract

Conference

Access to Document

Other files and links

Fingerprint

Cite this