4 ms·
In Shannon’s 1948 paper, part V deals with continuous sources. The key is to realise that you cannot measure a continuous signal exactly, and so you can define
by hanche 4y ago
In Shannon’s 1948 paper, part V deals with continuous sources. The key is to realise that you cannot measure a continuous signal exactly, and so you can define a rate of information relative to the fidelity of your measurement. (I only skimmed that part years ago, and never studied it carefully. But it makes perfect sense.)
- ogogmad 4y agoIf you mean differential entropy (which Shannon supposedly suggested as a generalisation to continuous random variables), this is not a good generalisation of entropy to continuous random variables. It lacks all the interesting properties of entropy. The "proper" generalisation of entropy to continuous random variables is something called relative entropy, or in some books it's called KL divergence. But this is now a property of how two probability distributions relate to each other, rather than a property of a single probability distribution alone. I'm not an expert in probability theory or physics, but this is what I've learnt from a brief study of these areas.
- hanche 4y agoRelative entropy? KL? Ah, found it – Kullback–Leibler divergence, it’s called. Thanks, I’ll put that on my list of stuff to learn about.