Divergence and Bregman Geometry

Divergence

Divergence

A divergence is a function on a manifold , it satisfies the following properties:

• if and only if ;
• is positive definite.

Given a divergence , the dual divergence is defined by

e.g. The Euclidean distance is a metric distance but not a divergence; The squared Euclidean distance is a non-metric symmetric divergence.

Proposition

Proof On the diagonal, the first order derivatives vanish, so it is constant, thus:

Divergence-Induced Statistical Manifold

Suppose is a divergence. Then is a statistical manifold with We will use to denote the such divergence-induced statistical manifold.

Proof Clearly the constructed connections are torsion-free. We need to check that the induced connections are conjugate to each other:

Bregman Geometry

Bregman Divergence

Suppose is a smooth manifold. Let be a neighbourhood of some point homeomorphic to some Euclidean space under local chart (coordinate) . Given a strictly convex smooth function called a potential function, where is an open convex domain. The Bregman divergence corresponding to this and coordinate is defined to be:

Proposition

The Bregman divergence with negative Shanon entropy as potential function is the Kullback-Leibler divergence.

Proof Let , then we have: Therefore,

Bregman Manifold

We call the statistical manifold induced by a Bregman divergence with potential a Bregman manifold. It satisfies So it is -flat.

Proof

Duality

Dual Potential

The dual potential of a potential function with chart on is defined as its Legendre-Fenchel transformation: And we call the dual coordinate. To simplify our notation, we write for , and for .

Proposition

Suppose is a potential function, where is an affine coordinate. Then the relation between the dual coordinate and coordinate is:

Proof In the definition of dual potential, achieves supremum when . Since Legendre-Fenchel transformation is involutive, we have .

Crouzeix Identity

The following identity holds, relating Hessians of the potential functions:

Corollary

On a Bregman manifold, suppose and are basis vectors for some tangent space under the original and dual coordinates, then Furthermore,

Proof Since on a Bregman manifold, . Thus by Crouzeix identity, we immediately have . Now write . Then we have Thus , it follows that . Therefore, we have .

Proposition

Dual Bregman divergence in the original coordinate is the Bregman divergence of the dual potential in the dual coordinate:

Proof

Pythagorean Theorem for Bregman Manifold

On a Bregman manifold, let and be the divergence and dual divergence, the Pythagorean theorem holds in the sense that if the geodesic from to and the dual geodesic from to are orthogonal.

Proof By definition of the Bregman divergence, we have Notice that the two curves are perpendicular at , so . Since and are geodesics and dual geodesics, and both connection and dual connection are flat, and are lines in the corresponding coordinate systems. Thus and , so the Pythagorean theorem is satisfied.