On Defining Expressions for Entropy and Cross-Entropy: The Entropic Transreals and Their Fracterm Calculus

Abstract

Classic formulae for entropy and cross-entropy contain operations x0 and log2x that are not defined on all inputs. This can lead to calculations with problematic subexpressions such as 0log20 and uncertainties in large scale calculations; partiality also introduces complications in logical analysis. Instead of adding conventions or splitting formulae into cases, we create a new algebra of real numbers with two symbols ±∞ for signed infinite values and a symbol named ⊥ for the undefined. In this resulting arithmetic, entropy, cross-entropy, Kullback–Leibler divergence, and Shannon divergence can be expressed without concerning any further conventions. The algebra may form a basis for probability theory more generally.

Keywords

• partial formulae
• fracterm calculus
• transreals
• entropic transreals
• peripheral numbers
• entropy