Principled inference pipelines

We have just begun exploring the tools of statistical inference, with much of our focus being on MCMC. Regardless of what tool we use, we should bear in mind that statistical inference is fragile. There are many failure modes.

In what follows, we will lay out an approach to help prevent us from falling into common traps. We do this in the context of MCMC-based inference, but the principles we lay out here apply to any approach, including variational and optimization approaches. The key idea is that we throw all sorts of possible data sets at our inference pipeline and investigate its performance, sniffing out aspects of data sets that make it brittle.