4 ms·

erm, I think boltzman figured out entropy for thermo like 100 years before shannon's theory of communication could be totally wrong though entropy, complexity

by f00_ 9y ago

erm, I think boltzman figured out entropy for thermo like 100 years before shannon's theory of communication could be totally wrong though entropy, complexity, and evolution are words that come to mind

vitus 9y ago
Yes, Shannon explicitly stated in Mathematical Theory of Communication that the general form was the same as that in statistical mechanics: "The form of H will be recognized as that of entropy as defined in certain formulations of statistical mechanics where p_i is the probability of a system being in cell i of its phase space. H is then, for example, the H in Boltzmann’s famous H theorem." where the general form was H = -K\sum p_i * log(p_i) IIRC, this form is the only kind with the following two properties: - chain rule -- H(X,Y) = H(X) + H(Y|X) - maximized for the uniform distribution.
- f00_ 9y ago
  Could you explain KL divergence and/or cross entropy to me? from wikipedia: "In mathematical statistics, the Kullback–Leibler divergence is a measure of how one probability distribution diverges from a second expected probability distribution" Kullback-Leibler divergence is used a lot in the reinforcement learning setting "In information theory, the cross entropy between two probability distributions p and q over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set" Cross Entropy is used as a cost function in neural networks rather than least squares http://neuralnetworksanddeeplearning.com/chap3.html#the_cross-entropy_cost_function http://neuralnetworksanddeeplearning.com/chap3.html#the_cros...
- f00_ 9y ago
  Also, have you read Norbert Weiner's Cybernetics? I think he had parts of information theory there before Shannon I want to understand his work on random processes too, have just kind of scanned the book and have tried to upgrade my calculus

by f00_ 9y ago

vitus 9y ago

Yes, Shannon explicitly stated in Mathematical Theory of Communication that the general form was the same as that in statistical mechanics: "The form of H will be recognized as that of entropy as defined in certain formulations of statistical mechanics where p_i is the probability of a system being in cell i of its phase space. H is then, for example, the H in Boltzmann’s famous H theorem." where the general form was H = -K\sum p_i * log(p_i) IIRC, this form is the only kind with the following two properties: - chain rule -- H(X,Y) = H(X) + H(Y|X) - maximized for the uniform distribution.

f00_ 9y ago
Could you explain KL divergence and/or cross entropy to me? from wikipedia: "In mathematical statistics, the Kullback–Leibler divergence is a measure of how one probability distribution diverges from a second expected probability distribution" Kullback-Leibler divergence is used a lot in the reinforcement learning setting "In information theory, the cross entropy between two probability distributions p and q over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set" Cross Entropy is used as a cost function in neural networks rather than least squares http://neuralnetworksanddeeplearning.com/chap3.html#the_cross-entropy_cost_function http://neuralnetworksanddeeplearning.com/chap3.html#the_cros...
f00_ 9y ago
Also, have you read Norbert Weiner's Cybernetics? I think he had parts of information theory there before Shannon I want to understand his work on random processes too, have just kind of scanned the book and have tried to upgrade my calculus