In probability theory, statistics, and machine learning, the continuous Bernoulli distribution is a family of continuous probability distributions parameterized by a single shape parameter , defined on the unit interval , by:
The continuous Bernoulli distribution arises in deep learning and computer vision, specifically in the context of variational autoencoders, for modeling the pixel intensities of natural images. As such, it defines a proper probabilistic counterpart for the commonly used binary cross entropy loss, which is often applied to continuous, -valued data. This practice amounts to ignoring the normalizing constant of the continuous Bernoulli distribution, since the binary cross entropy loss only defines a true log-likelihood for discrete, -valued data.
The continuous Bernoulli also defines an exponential family of distributions. Writing for the natural parameter, the density can be rewritten in canonical form: .
Given an independent sample of points with from continuous Bernoulli, the log-likelihood of the natural parameter is
and the maximum likelihood estimator of the natural parameter is the solution of , that is, satisfies
where the left hand side is the expected value of continuous Bernoulli with parameter . Although does not admit a closed-form expression, it can be easily calculated with numerical inversion.
The entropy of a continuous Bernoulli distribution is
The continuous Bernoulli can be thought of as a continuous relaxation of the Bernoulli distribution, which is defined on the discrete set by the probability mass function:
where is a scalar parameter between 0 and 1. Applying this same functional form on the continuous interval results in the continuous Bernoulli probability density function, up to a normalizing constant.
The Uniform distribution between the unit interval [0,1] is a special case of continuous Bernoulli when or .
An exponential distribution with rate restricted to the unit interval [0,1] corresponds to a continuous Bernoulli distribution with natural parameter .
The multivariate generalization of the continuous Bernoulli is called the continuous-categorical.