Axiomatic Characterizations of Information Measures

Abstract

Axiomatic characterizations of Shannon entropy, Kullback I-divergence, and some generalized information measures are surveyed. Three directions are treated: (A) Characterization of functions of probability distributions suitable as information measures. (B) Characterization of set functions on the subsets of {1; : : : ;N} representable by joint entropies of components of an N-dimensional random vector. (C) Axiomatic characterization of MaxEnt and related inference rules. The paper concludes with a brief discussion of the relevance of the axiomatic approach for information theory.

Keywords

• Shannon entropy
• Kullback I-divergence
• Rényi information measures
• f- divergence
• f-entropy
• functional equation
• proper score
• maximum entropy
• transitive inference rule
• Bregman distance