5 ms·

Do you mean that information theory in general is essential in working with ML systems, or a specific point raised by Shannon?

by davnn 3y ago

tysam_and 3y ago
I'm sure there's a specific point raised by Shannon, and I've been (very recently!) learning how Shannon himself is not the sole info theory dude (even though lots of people credit him for it!). But basically, the communication of information over a noisy channel _is_ the foundation for deep learning.
- arketyp 3y ago
  As they say, intelligence is prediction is compression is encoding...
  - tysam_and 3y ago
    Well, I agree that encoding is compression, at least, but the rest of that statement I do disagree with! It seems to be one of the more common mantras going around right now, though. Which is partially why I advocate for Shannon's theory! It's very clean and bare metal in terms of being related to intelligence, though I think one could build a decent argument for intelligence from info theory.
    - PeterisP 3y ago
      > Well, I agree that encoding is compression, at least, but the rest of that statement I do disagree with! IMHO that Shannon's paper you linked is effectively the initial work for linking prediction with compression, showing how the information transmitted (and the necessity for transmitted information) decreases as you can more accurately predict the likelihoods of the next symbol.
      - tysam_and 3y ago
        Yes, but prediction != compression, and intelligence != prediction! However, prediction can inform compression, but there's necessarily some framework needed to translate the information from prediction<->compression. Perhaps that is too nitpicky, but to me it's like thinking about raw materials (like raw food or whatnot) and how it ends up as a finished product (like cornbread. Corn farming is not cornbread, but there is a semi-clear family of paths from one to the other with its own set of optimizations, rules, and etc). Again, that could be a bit nitpicky on my end depending on how one is viewing it.
        deleted 3y ago
        [deleted]
- davnn 3y ago
  It probably is from the perspective of an information theorist. Did you read any interesting articles on the connections between deep learning and information theory to come to this conclusion? I‘m highly interested in this space, but the influence of information theory on deep learning developments appears to be negligible.
  - tysam_and 3y ago
    Aha! Yes, yes, yes! Good question. A senior scientist handed me a book form of this paper (plus the introduction by Weaver, which made things much easier for little, younger me!) From there, it's just been the process of getting to know the field better and making the connections over time. It's been about 7-8 years for me, so I've been deep into neural networks for a little while. Not everyone uses information theory to inform research into neural networks, which is a darn shame, but many of the recent advances I feel can be trivially explained with info theory, or at least derived from it. I can give a few basic examples about where it interacts with deep learning. One obvious one is the cross-entropy function. This of course is trying perform ERM use the MLE over the dataset where the neural network is the mapping function f(x) -> y. For a loss function, the cross-entropy is ideal in this case, as we want to choose a coding scheme that minimizes our regret on new data based on the statistics of some data we have (thus the 'E' in Empirical Risk Minimization, and the 'R', and the, oh gosh darn it, you get what I mean here). This is analogous to a communications process where the neural network f is emitting a target token with a likelihood determined by f(x). In this case, the softmax serves as that emission probability, and instead of optimizing a discrete system, we are optimizing a continuous relaxation of the probabilities of that discrete system in the limit. You can go a lot deeper into the rabbit hole and rephrase basically everything in ML in that light. Doing so has rather significantly helped me push the bounds of performance in my area of research, at least. There is a lot to learn, however. So I'm sure there are decent articles out there, but second-hand stuff will likely only be useful for getting a general gist of a problem, at the cost of also inheriting the common public model for a way of thinking about things. If you go off of the beaten path and try to translate the whole training process (don't forget the temporal aspect!) into this framework, I think you'll find it very challenging and interesting. Very fun! <3 :) Hope that answers the question, if not, then feel free to let me know! I think there are some learning materials that talk about the basics of info theory w.r.t. ML, but the rabbit hole goes....very deep indeedy.

Do you mean that information theory in general is essential in working with ML systems, or a specific point raised by Shannon?

by davnn 3y ago

Do you mean that information theory in general is essential in working with ML systems, or a specific point raised by Shannon?

tysam_and 3y ago

I'm sure there's a specific point raised by Shannon, and I've been (very recently!) learning how Shannon himself is not the sole info theory dude (even though lots of people credit him for it!). But basically, the communication of information over a noisy channel _is_ the foundation for deep learning.

arketyp 3y ago
As they say, intelligence is prediction is compression is encoding...
- tysam_and 3y ago
  Well, I agree that encoding is compression, at least, but the rest of that statement I do disagree with! It seems to be one of the more common mantras going around right now, though. Which is partially why I advocate for Shannon's theory! It's very clean and bare metal in terms of being related to intelligence, though I think one could build a decent argument for intelligence from info theory.
  - PeterisP 3y ago
    > Well, I agree that encoding is compression, at least, but the rest of that statement I do disagree with! IMHO that Shannon's paper you linked is effectively the initial work for linking prediction with compression, showing how the information transmitted (and the necessity for transmitted information) decreases as you can more accurately predict the likelihoods of the next symbol.
    - tysam_and 3y ago
      Yes, but prediction != compression, and intelligence != prediction! However, prediction can inform compression, but there's necessarily some framework needed to translate the information from prediction<->compression. Perhaps that is too nitpicky, but to me it's like thinking about raw materials (like raw food or whatnot) and how it ends up as a finished product (like cornbread. Corn farming is not cornbread, but there is a semi-clear family of paths from one to the other with its own set of optimizations, rules, and etc). Again, that could be a bit nitpicky on my end depending on how one is viewing it.
      - deleted 3y ago
        [deleted]
davnn 3y ago
It probably is from the perspective of an information theorist. Did you read any interesting articles on the connections between deep learning and information theory to come to this conclusion? I‘m highly interested in this space, but the influence of information theory on deep learning developments appears to be negligible.
- tysam_and 3y ago
  Aha! Yes, yes, yes! Good question. A senior scientist handed me a book form of this paper (plus the introduction by Weaver, which made things much easier for little, younger me!) From there, it's just been the process of getting to know the field better and making the connections over time. It's been about 7-8 years for me, so I've been deep into neural networks for a little while. Not everyone uses information theory to inform research into neural networks, which is a darn shame, but many of the recent advances I feel can be trivially explained with info theory, or at least derived from it. I can give a few basic examples about where it interacts with deep learning. One obvious one is the cross-entropy function. This of course is trying perform ERM use the MLE over the dataset where the neural network is the mapping function f(x) -> y. For a loss function, the cross-entropy is ideal in this case, as we want to choose a coding scheme that minimizes our regret on new data based on the statistics of some data we have (thus the 'E' in Empirical Risk Minimization, and the 'R', and the, oh gosh darn it, you get what I mean here). This is analogous to a communications process where the neural network f is emitting a target token with a likelihood determined by f(x). In this case, the softmax serves as that emission probability, and instead of optimizing a discrete system, we are optimizing a continuous relaxation of the probabilities of that discrete system in the limit. You can go a lot deeper into the rabbit hole and rephrase basically everything in ML in that light. Doing so has rather significantly helped me push the bounds of performance in my area of research, at least. There is a lot to learn, however. So I'm sure there are decent articles out there, but second-hand stuff will likely only be useful for getting a general gist of a problem, at the cost of also inheriting the common public model for a way of thinking about things. If you go off of the beaten path and try to translate the whole training process (don't forget the temporal aspect!) into this framework, I think you'll find it very challenging and interesting. Very fun! <3 :) Hope that answers the question, if not, then feel free to let me know! I think there are some learning materials that talk about the basics of info theory w.r.t. ML, but the rabbit hole goes....very deep indeedy.