Classification of Divergences

Def Divergence
The divergence is a real-valued function with and be two probability measures defined on the same measurable space. It satisfies

• with equality holds iff .
• may not be symmetric: in general.
• may not satisfy the triangle inequality.
  Therefore, is not a metric distance.
  

Def Locally-Rao
A divergence is called locally-Rao, if

Def -Divergence
For probability distribution , we define: where with properties that and . Clearly is convex and we have , yielding that .

Thrm is convex w.r.t. .

Prop -divergence is locally-Rao.

Def -Divergence
Suppose , plug into the definition of -divergence, we get
Prop

• -divergence with is exactly the KL divergence.
• -divergence with is the reverse KL divergence.
• .

Prop KL Divergence of Exponential Family
The KL divergence of exponential family and is