The prior \(π(θ)\) in a Bayesian model is the prior belief about the parameters \(θ\).
STK4021 – Applied Bayesian analysis
Lecture notes
1 The Bayesian pipeline
1.1 Introduction
Bayesian statistics
Bayesian model
Prior
Likelihood
Posterior
Marginal likelihood
Exercise
Posterior mean
Posterior variance
Exercise
Exercise
Normalization trick · Functional form trick
Functional form
Proportional Bayes theorem
The predictive distribution
Predictive distribution
Functional form
Exercise
Exercise
1.2 Bayesian decision theory
Decision theory
Action
Decision function
Loss function · Cost function
Frequentist risk · Risk
Bayes risk
Bayes estimator
Posterior expected loss
Bayes risk minimization theorem
Exercise
Exercise
Exercise
Exercise
Exercise
2 Choosing the prior distribution
Prior selection
2.1 Conjugate priors
Conjugate prior
Exercise
Exercise
Exercise
2.1.1 The multivariate Gaussian distribution
Multivariate Gaussian distribution
Symmetric matrix
Positive definite matrix
Exercise
Exercise
Exercise
Conditional Gaussian distributions
Covariance matrix
Precision matrix
Exercise
Woodbury matrix identity
Marginal Gaussian distributions
Exercise
Bayes' theorem for Gaussian variables
Linear Gaussian model
Exercise
Exercise
Exercise
Sherman-Morrison formula
2.2 Empirical Bayes
Hyperparameter
Hyperparameter selection
Empirical Bayes
Empirical Bayes
Exercise
Exercise
2.3 The Jeffreys prior
Fisher information
Jeffreys prior
Improper prior
Exercise
Exercise
Exercise
Score
3 The Laplace approximation (Lazy Bayes)
Laplace approximation derivation
Laplace approximation
MAP · Maximum a posteriori
Observed Fisher information
Exercise
Exercise
Exercise
4 Model selection and model averaging
Model space
Model prior
Model-level likelihood
Bayesian model space
Conditioning on model gives model
Model posterior
Model selection
Bayes factor derivation
Uniform model selection · Bayes factor selection
Bayes factor
Exercise
Exercise
4.1 The Bayesian information criterion (BIC)
BIC · Bayesian information criterion
4.2 Derivation of the BIC
Derivation of the BIC
5 Regression and classification
5.1 Linear models for regression
5.1.1 The frequentist solution: least squares and penalization
5.1.2 Bayesian linear regression
5.1.3 Model comparison
5.1.4 Empirical Bayes
5.2 Linear models for classification
5.2.1 Bayesian classification
6 Exchangeability and de Finetti's theorem
6.1 Exchangeability
Permutation
Exchangeable variables
Exchangeable sequence
Exercise
iid sequence is exchangeable
Exercise - Pólya's urn
De Finetti's theorem
De Finetti's theorem - Integral form
Exercise
Exercise
7 The Bernstein–von Mises theorem
Asymptotic normality of MLE
Total variation distance
Exercise
Total variation integral expression
Bernstein–von Mises theorem
Exercise
8 Markov chain Monte Carlo (MCMC)
MCMC methods
8.1 General state space Markov chains
Markov chain
Homogeneous chain
Kernel
Kernel density
Exercise
Example 2
Exercise
\(m\)-step kernel
\(m\)-step kernel density
8.2 Key properties
\(μ\)-irreducible
Exercise
Periodic chain
Aperiodic chain
Exercise
Stationary distribution · Invariant distribution
Detailed balance
Exercise
Ergodicity
Irreducible & stationary chain is ergodic
Aperiodic ergodic chain converges to stationary
8.3 The Metropolis–Hastings algorithm
Metropolis–Hastings idea
Target density
Proposal distribution
Acceptance probability
Metropolis–Hastings algorithm
Exercise
Metropolis–Hastings kernel derivation
Metropolis–Hastings kernel
Exercise
Metropolis–Hastings target is stationary proof
Metropolis–Hastings target is stationary
Metropolis–Hastings Bayesian posterior target derivation
Metropolis–Hastings Bayesian posterior target
Exercise
Exercise
8.4 Gibbs sampling
Gibbs sampling idea
Gibbs target density
Exercise
Gibbs sampling
Gibbs kernel
Exercise
Gibbs joint distribution is stationary
Proof
Partial Gibbs kernel
Exercise
Joint satisfies detailed balance for partial kernel
Composed partial kernels
Chain composition preserve stationary
Proof
Partial Gibbs kernel as Metropolis–Hastings kernel
Proof
Exercise
Exercise
8.5 Convergence diagnostics
Empirical estimate
Burn-in
Trace plot
Exercise
Autocorrelation-lag plot
\(r\)-lag autocorrelation
\(r\)-lag autocovariance
\(r\)-lag autocovariance estimator
\(r\)-lag autocorrelation estimator
Exercise
ESS · Effective sample size
Effective sample size estimator
Exercise
MCMC quality assurance checklist
Auxiliary
Fubini's theorem
Spectral theorem
The likelihood in a Bayesian model is the
...
The marginal likelihood \(π(y)\) in a Bayesian model
...
Make the optimal decision based on the (sampled) data.
An action is a function of the data
A measure of the cost of taking an action given the data.
The average cost of using a decision function.
\[R(a, θ) = \int_\mathcal{Y} L(a(y)|θ)π(y|θ)dy\]
Select the hyperparameters that maximizes the marginal likelihood \(π(y)\).
Select the model \(m\) with highest probability given the data \(y\) \(π(m|y)\).
Ergodicity ensures that large sample properties hold for the chain.
In particular, an expectation of the chain over time is equal to the expectation over the stationary distribution.
In general, computing the normalization factor of a distribution is very difficult or expensive. We might only have an unnormalized distribution.
Metropolis–Hastings allows us to sample from a distribution even if all we have is the unnormalized distribution.
Construct a Markov chain thats stationary distribution is the target distribution.
It is hard to sample from the joint, but it is easy to sample from the conditionals.
1
If a function is Lebesgue integrable on a rectangle, then the integral is equivalent to an iterated integral, and the order of integration is optional.
A real symmetrix matrix can be diagonalized. It has a set of eigenvalues and eigenvectors \((λ_i, u_i)\), where the diagonalization is the diagonal matrix with eigenvalues on the diagonal.