4 ms·

“Cross entropy” specifically refers to the log-likelihood function of a binary random variable, and is only used as the cost function for binary classifiers. It

“Cross entropy” specifically refers to the log-likelihood function of a binary random variable, and is only used as the cost function for binary classifiers. It does not refer to likelihood functions in general.

ansk 5y ago
Do people not google terms before trying to speak authoritatively on a topic they aren't familiar with? The original commenter is correct, cross entropy is a generic measure of two probability distributions - in the case of maximum likelihood estimation, these are the data distribution and the distribution of the learned model.
- MontyCarloHall 5y ago
  You are incorrect. For a given probability distribution parameterized by θ with probability mass/density p(x|θ), the likelihood of θ given a set of data X = {x_1, …, x_n} (assuming X is independently/identically distributed) is simply the product of independent probabilities, L(θ|X) = Π_i=1^n p(x_i|θ) Maximizing this product with respect to θ yields the maximum likelihood estimate of θ. Since sums are generally easier to work with than products, and log is a monotonic function, we generally work with the log-likelihood function log L(θ|X) = Σ_i=1^n log p(x_i|θ) since the log-likelihood will achieve its maximum for the same value of θ as the likelihood. The cross entropy of two discrete probability distributions p and q is Σ_i=1^n p_i log q_i (For continuous distributions, replace the sum with an integral.) This is completely unrelated to the generic log-likelihood function defined above. The two are only related if p happens to be the probability distribution of a binary random variable x = {0,1}, with probability π of equalling 1: p(x|π) = π^x(1-π)^(1-x) Its log-likelihood is therefore x log π + (1-x) log(1-π) which for this particular case, happens to be a cross entropy. Note that this is the log-likelihood of a single observation in a single class; for multiple observations/multiple classes, we sum across them, e.g. Σ_i=1^n x_i log π_i + (1-x_i)log (1-π_i) for a single observation across n total classes. But again, the relationship to cross entropy only holds for this particular choice of p. It is not generally the case that the generic log-likelihood function, log L(θ|X) = Σ_i=1^n log p(x_i|θ) is a cross entropy!
  - contravariant 5y ago
    You can take the cross entropy between the the probability distribution and the dirac-delta distribution for the actual data. This will equal the log-likelihood. Things get a little iffy with continuous probability distributions, but that's just because both your cross-entropy and your MLE estimate will depend on your choice of variables if you don't pick a prior. Just as for MLE you can blindly plug in the probability density and it'll work just fine.
    - MontyCarloHall 5y ago
      True! Given a ~Dirac comb~ mixture of Dirac distributions c(x) = 1/n Σ_i=1^n δ(х - x_i) and some function f, you can express the sum of f over x_i as Σ_i=1^n f(x_i) = ∫_-∞^∞ dx’ f(x)c(x - x’) If f were a log probability, this would be indeed be a (continuous) cross entropy: Σ_i log p(х_i|θ) = ∫_-∞^∞ dx’ log p(х_i|θ) c(x - x’) However, this isn’t generally how we think about likelihood functions, since there is nothing gained from expressing a simple sum of log probability densities in terms of a Dirac comb. Indeed, every ML text/paper I’ve read only ever refers to “cross entropy” in the context of the cost function for one-hot categorical random variables, since the formula for cross entropy is immediately present in the likelihood function. Cost functions involving other random variables are simply called “cost functions” or just “likelihoods” if the author comes from a stats background.
      - dkshdkjshdk 5y ago
        > Given a Dirac comb > c(x) = 1/n Σ_i=1^n δ(х - x_i) Sorry for the pedantry, but a mixture of Dirac distributions is almost always not a Dirac comb. Notice that a mixture of Dirac distributions is a Dirac comb only if you have an infinite number of equally-separated samples (and empirical distributions tend to have a finite number of samples).
    - whimsicalism 5y ago
      > your MLE estimate will depend on your choice of variables if you don't pick a prior. If you're doing MLE, then you don't have a prior (or rather, you have a uniform prior over the parameter(s) of interest).

ansk 5y ago

Do people not google terms before trying to speak authoritatively on a topic they aren't familiar with? The original commenter is correct, cross entropy is a generic measure of two probability distributions - in the case of maximum likelihood estimation, these are the data distribution and the distribution of the learned model.

MontyCarloHall 5y ago
You are incorrect. For a given probability distribution parameterized by θ with probability mass/density p(x|θ), the likelihood of θ given a set of data X = {x_1, …, x_n} (assuming X is independently/identically distributed) is simply the product of independent probabilities, L(θ|X) = Π_i=1^n p(x_i|θ) Maximizing this product with respect to θ yields the maximum likelihood estimate of θ. Since sums are generally easier to work with than products, and log is a monotonic function, we generally work with the log-likelihood function log L(θ|X) = Σ_i=1^n log p(x_i|θ) since the log-likelihood will achieve its maximum for the same value of θ as the likelihood. The cross entropy of two discrete probability distributions p and q is Σ_i=1^n p_i log q_i (For continuous distributions, replace the sum with an integral.) This is completely unrelated to the generic log-likelihood function defined above. The two are only related if p happens to be the probability distribution of a binary random variable x = {0,1}, with probability π of equalling 1: p(x|π) = π^x(1-π)^(1-x) Its log-likelihood is therefore x log π + (1-x) log(1-π) which for this particular case, happens to be a cross entropy. Note that this is the log-likelihood of a single observation in a single class; for multiple observations/multiple classes, we sum across them, e.g. Σ_i=1^n x_i log π_i + (1-x_i)log (1-π_i) for a single observation across n total classes. But again, the relationship to cross entropy only holds for this particular choice of p. It is not generally the case that the generic log-likelihood function, log L(θ|X) = Σ_i=1^n log p(x_i|θ) is a cross entropy!
- contravariant 5y ago
  You can take the cross entropy between the the probability distribution and the dirac-delta distribution for the actual data. This will equal the log-likelihood. Things get a little iffy with continuous probability distributions, but that's just because both your cross-entropy and your MLE estimate will depend on your choice of variables if you don't pick a prior. Just as for MLE you can blindly plug in the probability density and it'll work just fine.
  - MontyCarloHall 5y ago
    True! Given a ~Dirac comb~ mixture of Dirac distributions c(x) = 1/n Σ_i=1^n δ(х - x_i) and some function f, you can express the sum of f over x_i as Σ_i=1^n f(x_i) = ∫_-∞^∞ dx’ f(x)c(x - x’) If f were a log probability, this would be indeed be a (continuous) cross entropy: Σ_i log p(х_i|θ) = ∫_-∞^∞ dx’ log p(х_i|θ) c(x - x’) However, this isn’t generally how we think about likelihood functions, since there is nothing gained from expressing a simple sum of log probability densities in terms of a Dirac comb. Indeed, every ML text/paper I’ve read only ever refers to “cross entropy” in the context of the cost function for one-hot categorical random variables, since the formula for cross entropy is immediately present in the likelihood function. Cost functions involving other random variables are simply called “cost functions” or just “likelihoods” if the author comes from a stats background.
    - dkshdkjshdk 5y ago
      > Given a Dirac comb > c(x) = 1/n Σ_i=1^n δ(х - x_i) Sorry for the pedantry, but a mixture of Dirac distributions is almost always not a Dirac comb. Notice that a mixture of Dirac distributions is a Dirac comb only if you have an infinite number of equally-separated samples (and empirical distributions tend to have a finite number of samples).
  - whimsicalism 5y ago
    > your MLE estimate will depend on your choice of variables if you don't pick a prior. If you're doing MLE, then you don't have a prior (or rather, you have a uniform prior over the parameter(s) of interest).