Advances in distributed Bayesian inference and graph neural networks

Thumbnail Image
Journal Title
Journal ISSN
Volume Title
School of Science | Doctoral thesis (article-based) | Defence date: 2021-11-24
Degree programme
41 + app. 92
Aalto University publication series DOCTORAL DISSERTATIONS, 166/2021
Bayesian statistics and graph neural networks comprise a bag of tools widely employed in machine learning and applied sciences. The former rests on solid theoretical foundations, but its application depends on techniques that scale poorly as data increase. The latter is notorious for large-scale applications (e.g., in bioinformatics and natural language processing), but is largely only based on empirical intuitions. This thesis aims to i) broaden the scope of applications for Bayesian inference, and ii) deepen the understanding of core design principles of graph neural networks. First, we focus on distributed Bayesian inference under limited communication. We advance the state-of-the-art of embarrassingly parallel Markov chain Monte Carlo (MCMC) with a novel method that leverages normalizing flows as density estimators. On the same front, we also propose an extension of stochastic gradient Langevin dynamics for federated data, which are inherently distributed in a non-IID manner and cannot be centralized due to privacy constraints. Second, we develop a methodology for meta-analysis which allows the combination of Bayesian posteriors from different studies. Our approach is agnostic to study-specific complexities, which are all encapsulated in their respective posteriors. This extends the application of Bayesian meta-analysis to likelihood-free posteriors, which would otherwise be challenging. Our method also enables us to reuse posteriors from computationally costly analyses and update them post-hoc, without rerunning the analyses. Finally, we revisit two popular graph neural network components: spectral graph convolutions and pooling layers. Regarding convolutions, we propose a novel architecture and show that it is possible to achieve state-of-the-art performance by adding a minimal set of features to the most basic formulation of polynomial spectral convolutions. On the topic of pooling, we challenge the need for intricate pooling schemes and show that they do not play a role in the performance of graph networks in relevant benchmarks.
Defence is held on 24.11.2021 12:00 – 16:00 Zoom,
Supervising professor
Kaski, Samuel, Prof., Aalto University, Department of Computer Science, Finland
Bayesian statistics, graph neural networks, machine learning
Other note
  • [Publication 1]: Diego Mesquita, Paul Blomstedt, and Samuel Kaski. Embarrassingly Parallel MCMC with deep invertible transformations. In Uncertainty in Artificial Intelligence, Tel-Aviv, Israel, July 2019.
  • [Publication 2]: Khaoula el Mekkaoui, Diego Mesquita, Paul Blomstedt, and Samuel Kaski. Federated stochastic gradient Langevin dynamics. In Uncertainty in Artificial Intelligence, Online, July 2021
  • [Publication 3]: Paul Blomstedt, Diego Mesquita, Jarno Lintuusari, Tuomas Sivula, Jukka Corander, and Samuel kaski. Meta-analysis of Bayesian analyses. Submitted to a journal, 2020
  • [Publication 4]: Diego Mesquita, Amauri Souza, and Samuel Kaski. Rethinking pooling in graph neural networks. In Advances in neural information processing systems, Online, December 2020.
    Full text in Acris/Aaltodoc:
  • [Publication 5]: Hojin Kang, Jou-hui Ho, Diego Mesquita, Amauri Souza, Jorge Pérez, and Samuel Kaski. Spectral Graph Networks with Constrained Polynomials. Submitted to a journal, 2021