Exponential Family

Def Exponential Family
A parametric family of univariate continuous distributions is said to be an exponential family if the probability density function of any member of the family can be written as$f(x)=h(x)\exp (\eta (\theta {)}^{\top }T(x)-A(\theta ))$where

• $\theta \in {\mathbb{R}}^k$ is the vector of natural parameters;
• $h:\mathbb{R}\to {\mathbb{R}}^{+},x\mapsto h(x)$ is base measure;
• $\eta :{\mathbb{R}}^k\to {\mathbb{R}}^l$ is natural parameters.
• $T:\mathbb{R}\to {\mathbb{R}}^l$ is sufficient statistic.
• $A:{\mathbb{R}}^k\to \mathbb{R}$ is log-partition.
  Equivalently, we can also write it as:$f(x)=\exp \left(\mathcal{A}(x)+{\theta }^{\top }t(x)-\psi (\theta )\right)$

Thrm The natural parameter space is a convex domain, and $\psi$ is a convex function.

Def Dual Parameter
Given an exponential family $f(x)=\exp \left(\mathcal{A}(x)+{\theta }^{\top }t(x)-\psi (\theta )\right)$, $\eta =\nabla \psi (\theta )=\text{E}[t(x)]$ is the dual/mean/expectation parameter of the exponential family.

Prop The exponential family is conjugate.

Prop The maximum likelihood estimator of an exponential family is simply arithmetic average: $\hat{\eta }=\nabla \psi (\hat{\theta })=\frac{1}{n}{\sum }_{i=1}^{n}t(x_i)=\overline{t(x)}$
Prop The fisher information matrix of an exponential family $f(x)=\exp \left(\mathcal{A}(x)+{\theta }^{\top }t(x)-\psi (\theta )\right)$ is given by $I(\theta )=-{\mathbb{E}}_p\left[\frac{{\partial }^{2}L}{\partial \theta \partial {\theta }^{\top }}\right]=\frac{{\partial }^2\psi }{\partial \theta \partial {\theta }^{\top }}=\frac{\partial \eta }{\partial \theta }$