Priors

Conjugate Priors

Conjugate Prior

Suppose $\mathcal{F}$ is a class of sampling distributions (likelihoods) $L(x\mid \theta )$, and $\mathcal{P}$ is a class of prior distributions $\pi (\theta )$ for $\theta$, then the class $\mathcal{P}$ is conjugate for $\mathcal{F}$ if $\pi (\theta \mid x)\in \mathcal{P}\text{ for all }L(\cdot \mid \theta )\in \mathcal{F}\text{ and }\pi (\cdot )\in \mathcal{P}$

Lemma Conjugation Property of Gaussian
Suppose $x\in {\mathbb{R}}^d,{\mu }_i\in {\mathbb{R}}^d$ and ${\sigma }_i\in {\mathbb{S}}_d^{+}$ is positive semidefinite symmetric matrix. Then

$$\prod _{i=1}^n\exp \left(-\frac{1}{2}(x-{\mu }_i{)}^{\top }{\sigma }_i^{-1}(x-{\mu }_i)\right)=c\cdot \exp \left(-\frac{1}{2}(x-\hat{\mu }{)}^{\top }{\hat{\sigma }}^{-1}(x-\hat{\mu })\right)$$

where $\hat{\sigma }={\left({\sum }_{i=1}^n{\sigma }_i^{-1}\right)}^{-1}$, $\hat{\mu }=\hat{\sigma }\left({\sum }_{i=1}^n{\sigma }_i^{-1}{\mu }_i\right)$ and $c=\exp \left({\sum }_{i=1}^n{\mu }_i^{\top }{\sigma }_i^{-1}{\mu }_i-{\hat{\mu }}^{\top }{\hat{\sigma }}^{-1}\hat{\mu }\right)$.
Proof $\begin{array}{rl}& \prod _{i=1}^n\exp (-\frac{1}{2}(x-{\mu }_i{)}^{\top }{\sigma }_i^{-1}(x-{\mu }_i))\\ =& \exp \left(-\frac{1}{2}\sum _{i=1}^n(x-{\mu }_i{)}^{\top }{\sigma }_i^{-1}(x-{\mu }_i)\right)\\ =& \exp \left(-\frac{1}{2}\sum _{i=1}^n\left(x^{\top }{\sigma }_i^{-1}x-{\mu }_i^{\top }{\sigma }_i^{-1}x-x^{\top }{\sigma }_i^{-1}{\mu }_i+{\mu }_i^{\top }{\sigma }_i^{-1}{\mu }_i\right)\right)\\ =& \exp \left(-\frac{1}{2}\left(x^{\top }\left(\sum _{i=1}^n{\sigma }_i^{-1}\right)x-2x^{\top }\left(\sum _{i=1}^n{\sigma }_i^{-1}{\mu }_i\right)+\sum _{i=1}^n{\mu }_i^{\top }{\sigma }_i^{-1}{\mu }_i\right)\right)\\ =& \exp \left(-\frac{1}{2}\left(x^{\top }{\hat{\sigma }}^{-1}x-2x^{\top }{\hat{\sigma }}^{-1}\hat{\mu }+{\hat{\mu }}^{\top }{\hat{\sigma }}^{-1}\hat{\mu }-{\hat{\mu }}^{\top }{\hat{\sigma }}^{-1}\hat{\mu }+\sum _{i=1}^n{\mu }_i^{\top }{\sigma }_i^{-1}{\mu }_i\right)\right)\\ =& \exp \left(-\frac{1}{2}\left((x-\hat{\mu }{)}^{\top }{\hat{\sigma }}^{-1}(x-\hat{\mu })+\sum _{i=1}^n{\mu }_i^{\top }{\sigma }_i^{-1}{\mu }_i-{\hat{\mu }}^{\top }{\hat{\sigma }}^{-1}\hat{\mu }\right)\right)\\ =& \exp \left(-\frac{1}{2}(x-\hat{\mu }{)}^{\top }{\hat{\sigma }}^{-1}(x-\hat{\mu })\right)\exp \left(\sum _{i=1}^n{\mu }_i^{\top }{\sigma }_i^{-1}{\mu }_i-{\hat{\mu }}^{\top }{\hat{\sigma }}^{-1}\hat{\mu }\right)\end{array}$

Prop Summary of Common Conjugate Priors

Prior	Likelihood	Posterior
$\theta \sim \text{Beta}(a,b)$	$x\sim \text{Binomial}(n,\theta )$	$\theta \mid x\sim \text{Beta}(a+x,b+n-x)$
$\theta \sim \text{Gamma}(a,b)$	$x_i\sim \text{Possion}(\theta )$	$\theta \mid x\sim \text{Gamma}(a+{\sum }_i{x}_i,b+n)$
$\theta \sim \text{Gamma}(a,b)$	$x_i\sim \text{Gamma}(k,\theta )$	$\theta \mid x\sim \text{Gamma}(a+kn,b+{\sum }_i{x}_i)$
$\theta \sim \mathcal{N}(b,c^{-1})$	$x_i\sim \mathcal{N}(\theta ,{\tau }^{-1})$	$\theta \mid x,\tau \sim \mathcal{N}(\frac{cb+n\tau \bar{x}}{c+n\tau },\frac{1}{c+n\tau })$

Improper Priors

Def Improper Prior
A prior distribution $\pi (\theta )$ is improper if $\int \pi (\theta )\text{dd}\theta =\infty$.

Def Sufficient
We say that the empirical mean $\bar{x}$ is sufficient for parameter $\theta$ if the posterior depends on the data only through $\bar{x}$, rather than the each $x_i$.

Jeffreys’ Prior

Def Jeffreys’ Prior
The Jeffreys’ prior is defined as${\pi }_J(\theta )\propto \sqrt{|I(\theta )|}$where $I(\theta )$ is the Fisher information.