4 ms·

Do people not google terms before trying to speak authoritatively on a topic they aren't familiar with? The original commenter is correct, cross entropy is a g

• MontyCarloHall 5y ago
  You are incorrect. For a given probability distribution parameterized by θ with probability mass/density p(x|θ), the likelihood of θ given a set of data X = {x_1, …, x_n} (assuming X is independently/identically distributed) is simply the product of independent probabilities, L(θ|X) = Π_i=1^n p(x_i|θ) Maximizing this product with respect to θ yields the maximum likelihood estimate of θ. Since sums are generally easier to work with than products, and log is a monotonic function, we generally work with the log-likelihood function log L(θ|X) = Σ_i=1^n log p(x_i|θ) since the log-likelihood will achieve its maximum for the same value of θ as the likelihood. The cross entropy of two discrete probability distributions p and q is Σ_i=1^n p_i log q_i (For continuous distributions, replace the sum with an integral.) This is completely unrelated to the generic log-likelihood function defined above. The two are only related if p happens to be the probability distribution of a binary random variable x = {0,1}, with probability π of equalling 1: p(x|π) = π^x(1-π)^(1-x) Its log-likelihood is therefore x log π + (1-x) log(1-π) which for this particular case, happens to be a cross entropy. Note that this is the log-likelihood of a single observation in a single class; for multiple observations/multiple classes, we sum across them, e.g. Σ_i=1^n x_i log π_i + (1-x_i)log (1-π_i) for a single observation across n total classes. But again, the relationship to cross entropy only holds for this particular choice of p. It is not generally the case that the generic log-likelihood function, log L(θ|X) = Σ_i=1^n log p(x_i|θ) is a cross entropy!
  • contravariant 5y ago
    You can take the cross entropy between the the probability distribution and the dirac-delta distribution for the actual data. This will equal the log-likelihood. Things get a little iffy with continuous probability distributions, but that's just because both your cross-entropy and your MLE estimate will depend on your choice of variables if you don't pick a prior. Just as for MLE you can blindly plug in the probability density and it'll work just fine.
    • MontyCarloHall 5y ago
      True! Given a ~Dirac comb~ mixture of Dirac distributions c(x) = 1/n Σ_i=1^n δ(х - x_i) and some function f, you can express the sum of f over x_i as Σ_i=1^n f(x_i) = ∫_-∞^∞ dx’ f(x)c(x - x’) If f were a log probability, this would be indeed be a (continuous) cross entropy: Σ_i log p(х_i|θ) = ∫_-∞^∞ dx’ log p(х_i|θ) c(x - x’) However, this isn’t generally how we think about likelihood functions, since there is nothing gained from expressing a simple sum of log probability densities in terms of a Dirac comb. Indeed, every ML text/paper I’ve read only ever refers to “cross entropy” in the context of the cost function for one-hot categorical random variables, since the formula for cross entropy is immediately present in the likelihood function. Cost functions involving other random variables are simply called “cost functions” or just “likelihoods” if the author comes from a stats background.
      • dkshdkjshdk 5y ago
        > Given a Dirac comb > c(x) = 1/n Σ_i=1^n δ(х - x_i) Sorry for the pedantry, but a mixture of Dirac distributions is almost always not a Dirac comb. Notice that a mixture of Dirac distributions is a Dirac comb only if you have an infinite number of equally-separated samples (and empirical distributions tend to have a finite number of samples).
    • whimsicalism 5y ago
      > your MLE estimate will depend on your choice of variables if you don't pick a prior. If you're doing MLE, then you don't have a prior (or rather, you have a uniform prior over the parameter(s) of interest).
      • contravariant 5y ago
        Yeah I suppose in the context of MLE it makes more sense to talk about your choice of variables. Which does matter unfortunately (which is kind of obvious when you note that you can convert any distribution into any other by chaging its coordinates [*]). Using a prior gives you an 'out' by picking the Radon-Nikodym derivative w.r.t. that prior, since this definition of probability density is independent of your choice of variables. In most applications of MLE people implicitly use the (improper) uniform prior in which case you end up with the usual density. However this is usually done without justification, which is a bit dangerous. [*]: For a rather extreme example consider that if X is exponentially distributed with mean 1 then so is -log(1 - e^-X) which you get by using the CDF, flipping the distribution and then using the inverse CDF, this transformation swaps 0 (the mode) with positive infinity.
  • ansk 5y ago
    You've correctly shown that maximizing the likelihood is equivalent to minimizing cross entropy in the discrete case, but frankly that is unrelated to your claim that the equivalency doesn't hold in the general case. As noted in the sibling comment, the generalization to the continuous case is evident when viewing the empirical data distribution as a mixture of dirac densities.
    • whimsicalism 5y ago
      Hm, I never really thought about it this way - but I guess it does generalize to continuous space in a pretty natural way.