3 ms·
True! Given a ~Dirac comb~ mixture of Dirac distributions c(x) = 1/n Σ_i=1^n δ(х - x_i) and some function f, you can express the sum of f over x_i as Σ_i=1^n
by MontyCarloHall 5y ago
True! Given a ~Dirac comb~ mixture of Dirac distributions
c(x) = 1/n Σ_i=1^n δ(х - x_i)
and some function f, you can express the sum of f over x_i as
Σ_i=1^n f(x_i) = ∫_-∞^∞ dx’ f(x)c(x - x’)
If f were a log probability, this would be indeed be a (continuous) cross entropy:
Σ_i log p(х_i|θ) = ∫_-∞^∞ dx’ log p(х_i|θ) c(x - x’)
However, this isn’t generally how we think about likelihood functions, since there is nothing gained from expressing a simple sum of log probability densities in terms of a Dirac comb. Indeed, every ML text/paper I’ve read only ever refers to “cross entropy” in the context of the cost function for one-hot categorical random variables, since the formula for cross entropy is immediately present in the likelihood function. Cost functions involving other random variables are simply called “cost functions” or just “likelihoods” if the author comes from a stats background.
- dkshdkjshdk 5y ago> Given a Dirac comb > c(x) = 1/n Σ_i=1^n δ(х - x_i) Sorry for the pedantry, but a mixture of Dirac distributions is almost always not a Dirac comb. Notice that a mixture of Dirac distributions is a Dirac comb only if you have an infinite number of equally-separated samples (and empirical distributions tend to have a finite number of samples).