Differentially deep Gaussian processes

Abstract

Many modern machine learning methods, including deep neural networks, utilize a discrete sequence of parametric transformations to learn complex functions. Neural network based approaches can be an attractive choice for many real-world problems especially because of their modular nature. Gaussian process based methods, on the other hand, pose function approximation as a probabilistic inference problem by specifying prior distributions on unknown functions. Further, these probabilistic non-linear models provide well-calibrated uncertainty estimates which can be useful in many applications. However, the flexibility of these models depends on the choice of the kernel; handcrafting problem-specific kernels can be difficult in practice. Recently, deep Gaussian processes, a way of stacking multiple layers of Gaussian processes, was proposed as a flexible way of expanding model capacity.

In this thesis, we propose a novel probabilistic deep learning approach by formulating stochastic differential transformations or `flows' of inputs using Gaussian processes. This provides continuous-time `flows' as an alternative to the traditional approach of a discrete sequence of transformations using `layers'. Moreover, the proposed approach can also be seen as an approximation to very deep Gaussian processes with infinitesimal increments across layers. We also derive a scalable inference method based on variational sparse approximations for Gaussian processes. The proposed model shows excellent results on various experiments on real-world datasets, as compared to the other popular probabilistic approaches including deep Gaussian processes and Bayesian neural networks.