Information Divergence

Def Divergence The divergence is a real-valued function with and be two probability measures defined on the same measurable space. It satisfies

• with equality holds iff .
• may not be symmetric: in general.
• may not satisfy the triangle inequality. Therefore, is not a metric distance.

Def Locally-Rao A divergence is called locally-Rao, if

Def -Divergence For probability distribution , we define: where with properties that and . Clearly is convex and we have , yielding that .

Thrm is convex w.r.t. .

Prop -divergence is locally-Rao.

Def -Divergence Suppose , plug into the definition of -divergence, we get Prop

• -divergence with is exactly the KL divergence.
• -divergence with is the reverse KL divergence.
• .

Prop KL Divergence of Exponential Family The KL divergence of exponential family and is