Uncertainty approximation in neural networks using parameter-space proximity regularization

Abstract

A common question regarding the application of neural networks is whether the predictions of the model are reliable, in other words, what is the degree of uncertainty of our model. Generalizing a neural network into a Bayesian neural network is a frequent choice to quantify uncertainty. This means the extension of scalar weights and biases of the network to random variables. In terms of implementation, there are two main approaches: (1) the assumption of an unknown distribution for the random variables (i.e. weights) which are sampled and then utilized to infer the distribution of the output; (2) the assumption of a prior distribution for the random variables, and search for the output in the form of a similar posterior distribution. A common drawback of both approaches is that their scalability is limited, and generally require more computational resources than a simple neural network. In this paper, we introduce a new parameter sampling method, which maximizes the pairwise distance in parameter space between the models in the ensemble, therefore ensuring model diversity. Results indicate that this method generally needs less samples from the parameter space to be effective, thus it surpasses the other investigated approximation methods in terms of scalability. Finally, we apply our method to a real-life problem related to semantic segmentation of objects relevant in urban driving environments.