Advances in Randomly-Weighted Neural Networks and Temporal Gaussian Processes

Abstract

This dissertation consists of three main parts. In the first part, the existing methods of machine learning are applied to the environmental and astronomical datasets. The problems addressed in this part are the prediction of phosphorus concentration in the Pyhäjärvi lake (Finland) and the analysis of the correlation of geomagnetic storms with solar activity. For the first problem, several different models are built and the final accuracy is improved by variable selection and making an optimal ensemble. The second problem is solved by considering a correlation coefficient and estimating its uncertainty by the bootstrap method. The second part of the dissertation is devoted to studying randomly-weighted neural networks or in more narrow terminology Extreme Learning Machines (ELM). Vanilla ELM is trained by ordinary linear regression. As a consequence, ELM has reasonable accuracy but its training is much faster than the training of other neural networks. In this dissertation ELM for time series forecasting is investigated. It is shown that Optimally Pruned ELM (OP-ELM) algorithm in combination with a certain prediction strategy is better than a baseline model for time series data from different domains. Besides, the general regression algorithm (Inc)-OP-ELM is proposed which is significantly faster than the original OP-ELM but has the same performance. Finally, in the third part of the dissertation, the probabilistic models for time series data are studied. Two types of probabilistic time series models are considered: linear state-space models and temporal Gaussian processes (GP). The connections between them are studied and new Gaussian process covariance functions are derived. These new covariance functions correspond to state-space models which are popular in the literature. Temporal Gaussian processes can be converted to state-space form as well. It is shown that this conversion allows expressing the inference in temporal GPs as operations with block-tridiagonal matrices. These matrix operations can be computed in linear time with respect to the number of samples or in sub-linear time if parallel algorithms are utilized. Algorithms developed in this dissertation can serve as a basis for more complex models like spatio-temporal models and models with non-Gaussian likelihoods.