Trans-dimensional inference over Bayesian neural networks

Abstract

Trans-dimensional Bayesian inference for multi-layer perceptron architectures of varying size by reversible jump Markov chain Monte Carlo is developed and examined for its theoretical and practical merits and considerations. The algorithm features the No-U-Turn Sampler and Hamiltonian Monte Carlo for within-dimension moves, and makes use of a delayed-rejection sampler while exploring a variety of across-dimension moves that propose neural network models with varying numbers of hidden layers and hidden nodes. The advantages and considerations of sampling from a joint posterior distribution over model architecture and parameters are examined, and posterior predictive distributions are developed for classification and regression tasks.