Advances in Randomly-Weighted Neural Networks and Temporal Gaussian Processes

Thumbnail Image

Files

isbn9789526086712.pdf (2.62 MB)
Errata_grigorievskiy_alexander_DD_144_2019_publication_P6.pdf (21.21 KB)

URL

Journal Title

Journal ISSN

Volume Title

School of Science | Doctoral thesis (article-based) | Defence date: 2019-09-20
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.

Authors

Date

2019

Department

Tietotekniikan laitos
Department of Computer Science

Major/Subject

Mcode

Degree programme

Language

en

Pages

127 + app. 83

Series

Aalto University publication series DOCTORAL DISSERTATIONS, 144/2019

Abstract

This dissertation consists of three main parts. In the first part, the existing methods of machine learning are applied to the environmental and astronomical datasets. The problems addressed in this part are the prediction of phosphorus concentration in the Pyhäjärvi lake (Finland) and the analysis of the correlation of geomagnetic storms with solar activity. For the first problem, several different models are built and the final accuracy is improved by variable selection and making an optimal ensemble. The second problem is solved by considering a correlation coefficient and estimating its uncertainty by the bootstrap method. The second part of the dissertation is devoted to studying randomly-weighted neural networks or in more narrow terminology Extreme Learning Machines (ELM). Vanilla ELM is trained by ordinary linear regression. As a consequence, ELM has reasonable accuracy but its training is much faster than the training of other neural networks. In this dissertation ELM for time series forecasting is investigated. It is shown that Optimally Pruned ELM (OP-ELM) algorithm in combination with a certain prediction strategy is better than a baseline model for time series data from different domains. Besides, the general regression algorithm (Inc)-OP-ELM is proposed which is significantly faster than the original OP-ELM but has the same performance. Finally, in the third part of the dissertation, the probabilistic models for time series data are studied. Two types of probabilistic time series models are considered: linear state-space models and temporal Gaussian processes (GP). The connections between them are studied and new Gaussian process covariance functions are derived. These new covariance functions correspond to state-space models which are popular in the literature. Temporal Gaussian processes can be converted to state-space form as well. It is shown that this conversion allows expressing the inference in temporal GPs as operations with block-tridiagonal matrices. These matrix operations can be computed in linear time with respect to the number of samples or in sub-linear time if parallel algorithms are utilized. Algorithms developed in this dissertation can serve as a basis for more complex models like spatio-temporal models and models with non-Gaussian likelihoods.

Description

Supervising professor

Vehtari, Aki, Prof., Aalto University, Department of Computer Science, Finland

Thesis advisor

Karhunen, Juha, Prof. Emeritus, Department of Computer Science, Aalto University, Finland

Keywords

randomly-weighted neural networks, extreme learning machines, Gaussian processes, time series prediction, state-space models

Other note

Parts

  • [Publication 1]: Alexander Grigorevskiy, Anton Akusok, Marjo Tarvainen, Anne-Mari Ventelä, Amaury Lendasse. Practical Estimation of Missing Phosphorus Values in Pyhäjärvi Lake Data. In Workshop on New Challenges in NeuralComputation, 2013, Saarbrücken (Germany), Machine Learning Reports, Volume 2, pages 8-16, September 2013.
  • [Publication 2]: E.K.J. Kilpua, N. Olspert, A. Grigorievskiy, M.J. Käpylä, E.I. Tanskanen, H. Miyahara, R. Kataoka, J. Pelt, Y.D. Liu. Statistical Study of Strong and Extreme Geomagnetic Disturbances and Solar Cycle Characteristics. The Astrophysical Journal, 2015, Volume 806, Number 2, pages 272.
  • [Publication 3]: Alexander Grigorievskiy, Yoan Miche, Anne-Mari Ventelä, Eric Sèverin, Amaury Lendasse. Long-term time series prediction using OP-ELM. Neural Networks, 2014, Volume 51, pages 50-56.
    DOI: 10.1016/j.neunet.2013.12.002 View at publisher
  • [Publication 4]: Alexander Grigorievskiy, Yoan Miche, Maarit Mantere, Amaury Lendasse. Singular Value Decomposition Update and Its Application to (Inc)-OPELM. Neurocomputing, 2016, Volume 174, Part A, pages 99-108,
    DOI: 10.1016/j.neucom.2015.03.107 View at publisher
  • [Publication 5]: Alexander Grigorievskiy, Maarit Mantere, Anton Akusok, Emil Eirola, Amaury Lendasse. Forecasting the Outbursts of the Photometry Light Curve of Star V363 Lyr. In International work-conference on Time Series(ITISE-2014), Granada (Spain), Proceedings of ITISE-2014, Volume 2, pages 520-531, June 2014.
  • [Publication 6]: Alexander Grigorievskiy, Juha Karhunen. Gaussian Process Kernels for Popular State-Space Time Series Models. In International Joint Conference on Neural Networks (IJCNN 2016), Vancouver (Canada), pages 3354-3363, July 2016.
    DOI: 10.1109/IJCNN.2016.7727628 View at publisher
  • [Publication 7]: Alexander Grigorievskiy, Neil Lawrence, Simo Särkkä. Parallelizable Sparse Inverse Formulation Gaussian Processes (SpInGP). In Proceedings of the Workshop on Machine Learning for Signal Processing MLSP2017), Tokyo (Japan), September 2017.
  • [Errata file]: Errata of P6

Citation

Permanent link to this item

https://urn.fi/URN:ISBN:978-952-60-8671-2

Collections

[diss] Perustieteiden korkeakoulu / SCI