4 ms·

Okay, I confess. I really didn't understand most of that post. It sounds really smart, but someone will have to vouch that it's legit, because the picture of Ka

by milesf 11y ago

Okay, I confess. I really didn't understand most of that post. It sounds really smart, but someone will have to vouch that it's legit, because the picture of Kadanoff cuddling Cookie Monster trigged my baloney detector https://charlesmartin14.files.wordpress.com/2015/04/kadanoff.jpeg https://charlesmartin14.files.wordpress.com/2015/04/kadanoff...

oneloop 11y ago
I will read the post later, but I can tell you right now that I'm a physicist and when I was first introduced to deep learning my first thought, especially in the context of visual recognition, was "this smells of renormalization group".
- contravariant 11y ago
  Do you happen to know any good introductory texts to the renormalization group?
  - selimthegrim 11y ago
    The standard one in use at Santa Barbara is http://www.amazon.com/Lectures-Transitions-Renormalization-Frontiers-Physics/dp/0201554097 http://www.amazon.com/Lectures-Transitions-Renormalization-F... Some others I have sitting beside me at my desk right now are http://www.amazon.com/Theory-Critical-Phenomena-Introduction-Renormalization/dp/0198513933/ http://www.amazon.com/Theory-Critical-Phenomena-Introduction..., http://www.amazon.com/Renormalization-Introduction-Operator-Product-Monographs-Mathematical/dp/0521311772 http://www.amazon.com/Renormalization-Introduction-Operator-... and http://www.amazon.com/Renormalization-Methods-Guide-For-Beginners/dp/0199236526/ http://www.amazon.com/Renormalization-Methods-Guide-For-Begi... The most modern treatment is probably http://www.amazon.com/Scaling-Renormalization-Statistical-Physics-Cambridge/dp/0521499593 http://www.amazon.com/Scaling-Renormalization-Statistical-Ph... (For God's sake, stay away from anything written by Zinn-Justin, Itzykson, or Zuber unless you know what you're doing)
chmartin 11y ago
I try to lighten things up because the math by be unfamiliar.
charleshmartin 11y ago
I am happy to answer questions. You can ask here or directly on the blog.
TravisDick 11y ago
I don't mean to be super negative, but because of the general tone early in the article and some sloppy notation, I never finished reading. I think the goal of an article like this should be to give a high-level intuitive explanation for some technical result, rather than sounding smart or complicated. First, it is a little weird to me to talk about "old-school ML" as learning maps from inputs to hidden features. That seems neither old, nor very representative of the field of Machine Learning as a whole. It's also weird to say that RBMs and other deep learning algorithms are formulated using classical statistical mechanics. Moreover, implying that this scary-sounding formulation is the reason they are interesting seems like an attempt at sounding smart. Typically there are many ways to motivate and derive different algorithms, and it is /useful/ to acknowledge the multiple viewpoints because they often give different insights. Second, the section about flow maps and fixed points seems to make a mess out of the notation by either being unclear or disagreeing with standard notation. What is meant by the notation "f(X) -> X"? Presumably this means something like f is a function that maps elements of the set X to elements in the set X. More standard notation for this would be something like "f: X -> X". Perhaps it means that the image of the set X under the function f is again the set X. But does that require that f be a surjective function? Confusingly, it also looks like the function f might be required to be the identity function, but given the context this is clearly not the intended interpretation. When defining the fixed point, it seems that it would be more natural to say that x is a fixed point of f if f(x) = x. That is, x is fixed or unmoved by the function f. It turns out that for contractions (and some other functions, too), that the sequence f(x), f(f(x)), f(f(f(x))), and so on is guaranteed to converge to a unique fixed point of f. The notation f^n typically refers to the function f being applied n times, which is not the usage in the article. In the article, f^1, f^2, and so on are all identical copies of the function f. Using the standard notation, the definition of f_infty would be f_infty(x) = lim_{n -> infty} f^n(x). And, in the case of a contraction, the Banach fixed point theorem gives that f_infty is well-defined, and there exists a unique x_fix in X so that f_infty(x) = x_fix for all x in X (i.e., iterating f repeatedly converges to a unique fixed point x_fix of the function f). These things do not necessarily mean that the article is uninteresting or uninformative or even technically incorrect. But if the author didn't take the time to make the simple things clear, then I'm not sure that I want to read the rest. Sorry for the rant.
- 11y ago
Animats 11y ago
I got that feeling too, but I don't understand the subject well enough. One key concept here is that you want your learning system to have some "free energy" metric which decreases as you run the training set through again and again. This insures some kind of convergence, rather than just thrashing around. Of course, the other problem is getting stuck at a local minimum, which is why you don't want to converge too fast. (I'm not up to speed on that; when I studied AI years ago, everybody was getting stuck at local minima. Now, that's less of a problem, and some of the old algorithms, run with slow learning rates over and over, get stuck less.) The connection to physics seems to be from conservation of energy. The total energy of a system must decrease. For complex systems where energy is sloshing around from one form to another (flow, turbulence, gas compression, heat, maybe combustion) an energy-based analysis is a way of looking at the problem in a simpler way. That is apparently a useful way to look at deep learning systems. I think this is where the author is coming from. Whether the connection to physical systems is useful, or merely an interesting analogy, isn't clear from the paper. It may be too soon to tell.

by milesf 11y ago

oneloop 11y ago

I will read the post later, but I can tell you right now that I'm a physicist and when I was first introduced to deep learning my first thought, especially in the context of visual recognition, was "this smells of renormalization group".

contravariant 11y ago
Do you happen to know any good introductory texts to the renormalization group?
- selimthegrim 11y ago
  The standard one in use at Santa Barbara is http://www.amazon.com/Lectures-Transitions-Renormalization-Frontiers-Physics/dp/0201554097 http://www.amazon.com/Lectures-Transitions-Renormalization-F... Some others I have sitting beside me at my desk right now are http://www.amazon.com/Theory-Critical-Phenomena-Introduction-Renormalization/dp/0198513933/ http://www.amazon.com/Theory-Critical-Phenomena-Introduction..., http://www.amazon.com/Renormalization-Introduction-Operator-Product-Monographs-Mathematical/dp/0521311772 http://www.amazon.com/Renormalization-Introduction-Operator-... and http://www.amazon.com/Renormalization-Methods-Guide-For-Beginners/dp/0199236526/ http://www.amazon.com/Renormalization-Methods-Guide-For-Begi... The most modern treatment is probably http://www.amazon.com/Scaling-Renormalization-Statistical-Physics-Cambridge/dp/0521499593 http://www.amazon.com/Scaling-Renormalization-Statistical-Ph... (For God's sake, stay away from anything written by Zinn-Justin, Itzykson, or Zuber unless you know what you're doing)

chmartin 11y ago

I try to lighten things up because the math by be unfamiliar.

charleshmartin 11y ago

I am happy to answer questions. You can ask here or directly on the blog.

TravisDick 11y ago

I don't mean to be super negative, but because of the general tone early in the article and some sloppy notation, I never finished reading. I think the goal of an article like this should be to give a high-level intuitive explanation for some technical result, rather than sounding smart or complicated. First, it is a little weird to me to talk about "old-school ML" as learning maps from inputs to hidden features. That seems neither old, nor very representative of the field of Machine Learning as a whole. It's also weird to say that RBMs and other deep learning algorithms are formulated using classical statistical mechanics. Moreover, implying that this scary-sounding formulation is the reason they are interesting seems like an attempt at sounding smart. Typically there are many ways to motivate and derive different algorithms, and it is /useful/ to acknowledge the multiple viewpoints because they often give different insights. Second, the section about flow maps and fixed points seems to make a mess out of the notation by either being unclear or disagreeing with standard notation. What is meant by the notation "f(X) -> X"? Presumably this means something like f is a function that maps elements of the set X to elements in the set X. More standard notation for this would be something like "f: X -> X". Perhaps it means that the image of the set X under the function f is again the set X. But does that require that f be a surjective function? Confusingly, it also looks like the function f might be required to be the identity function, but given the context this is clearly not the intended interpretation. When defining the fixed point, it seems that it would be more natural to say that x is a fixed point of f if f(x) = x. That is, x is fixed or unmoved by the function f. It turns out that for contractions (and some other functions, too), that the sequence f(x), f(f(x)), f(f(f(x))), and so on is guaranteed to converge to a unique fixed point of f. The notation f^n typically refers to the function f being applied n times, which is not the usage in the article. In the article, f^1, f^2, and so on are all identical copies of the function f. Using the standard notation, the definition of f_infty would be f_infty(x) = lim_{n -> infty} f^n(x). And, in the case of a contraction, the Banach fixed point theorem gives that f_infty is well-defined, and there exists a unique x_fix in X so that f_infty(x) = x_fix for all x in X (i.e., iterating f repeatedly converges to a unique fixed point x_fix of the function f). These things do not necessarily mean that the article is uninteresting or uninformative or even technically incorrect. But if the author didn't take the time to make the simple things clear, then I'm not sure that I want to read the rest. Sorry for the rant.

11y ago

Animats 11y ago

I got that feeling too, but I don't understand the subject well enough. One key concept here is that you want your learning system to have some "free energy" metric which decreases as you run the training set through again and again. This insures some kind of convergence, rather than just thrashing around. Of course, the other problem is getting stuck at a local minimum, which is why you don't want to converge too fast. (I'm not up to speed on that; when I studied AI years ago, everybody was getting stuck at local minima. Now, that's less of a problem, and some of the old algorithms, run with slow learning rates over and over, get stuck less.) The connection to physics seems to be from conservation of energy. The total energy of a system must decrease. For complex systems where energy is sloshing around from one form to another (flow, turbulence, gas compression, heat, maybe combustion) an energy-based analysis is a way of looking at the problem in a simpler way. That is apparently a useful way to look at deep learning systems. I think this is where the author is coming from. Whether the connection to physical systems is useful, or merely an interesting analogy, isn't clear from the paper. It may be too soon to tell.