12 ms·
Why Deep Learning Works II: the Renormalization Group
- albertzeyer 11y agoNote: This is about unsupervised learning and mostly about RBMs/DBNs. Most of the Deep Learning success is all about supervised learning. In the past, RBMs have been used for unsupervised pretraining of the model, however, nowadays, everyone uses supervised pretraining. And the famous DeepMind works (Atari games etc) is mostly about Reinforcement learning, which is again different.
- BenderV 11y agoWell, if I understood correctly, the RL DeepMind implementation is basically making a RL algorithm work with a supervised model.
- sieisteinmodel 11y agoThis has been done since the 90s. The Deepmind paper is about a few more tricks.
- charleshmartin 11y agoI will address the supervised vs unsupervised issue in my next post. Here, I believe the analogy would be that when a field is applied to a spin glass, it does not exhibit a glass transition to a non-self-averaging (highly non-convex) ground state. As to supervised vs reinforcement learning, its not that different. See how Vowpal Wabbit incoporates both the 2 ideas in how the SGD update is formulated.
- reader5000 11y agoIs the "group" in renormalization group the same "group" in group theory?
- oneloop 11y agoAlmost. The name "group" in renormalization group was inspired by the groups in group theory, but in reality the renormalization "group" isn't a group but a semi-group. A semi-group satisfies the same axioms as a group except for the existence of inverse. And that is the mathematical reason why you can describe big things in terms of smaller things, but you can't describe small things in terms of bigger things: the renormalization (semi-) group flows from the ultraviolet to the infrared, but not the other way around :-)
- ximeng 11y agooneloop, your comment looks helpful but you are hell-banned, you may want to email HN to be reinstated. -- Edit - he's been reinstated.
- chmartin 11y agowhat does hell-banned mean?
- ximeng 11y agoComments are marked as dead (can only be seen by people with showdead set in their profile), but appear visible to the user so that they may not realise why nobody responds to them or upvotes their comments. Normally a punishment for bad behaviour or poor commenting, but seems inappropriate in this case.
- selimthegrim 11y agoHis comment below about a semigroup is absolutely correct, if that helps his case any. Qualifications: I studied RG in classes at Santa Barbara
- chmartin 11y agoKinda. The 'group' refers to a group scale and/or conformal transformations. In the context of Deep Learning, the 'scale' transform is akin to adding layers.
- 11y ago
- milesf 11y agoOkay, I confess. I really didn't understand most of that post. It sounds really smart, but someone will have to vouch that it's legit, because the picture of Kadanoff cuddling Cookie Monster trigged my baloney detector https://charlesmartin14.files.wordpress.com/2015/04/kadanoff.jpeg https://charlesmartin14.files.wordpress.com/2015/04/kadanoff...
- oneloop 11y agoI will read the post later, but I can tell you right now that I'm a physicist and when I was first introduced to deep learning my first thought, especially in the context of visual recognition, was "this smells of renormalization group".
- contravariant 11y agoDo you happen to know any good introductory texts to the renormalization group?
- selimthegrim 11y agoThe standard one in use at Santa Barbara is http://www.amazon.com/Lectures-Transitions-Renormalization-Frontiers-Physics/dp/0201554097 http://www.amazon.com/Lectures-Transitions-Renormalization-F... Some others I have sitting beside me at my desk right now are http://www.amazon.com/Theory-Critical-Phenomena-Introduction-Renormalization/dp/0198513933/ http://www.amazon.com/Theory-Critical-Phenomena-Introduction..., http://www.amazon.com/Renormalization-Introduction-Operator-Product-Monographs-Mathematical/dp/0521311772 http://www.amazon.com/Renormalization-Introduction-Operator-... and http://www.amazon.com/Renormalization-Methods-Guide-For-Beginners/dp/0199236526/ http://www.amazon.com/Renormalization-Methods-Guide-For-Begi... The most modern treatment is probably http://www.amazon.com/Scaling-Renormalization-Statistical-Physics-Cambridge/dp/0521499593 http://www.amazon.com/Scaling-Renormalization-Statistical-Ph... (For God's sake, stay away from anything written by Zinn-Justin, Itzykson, or Zuber unless you know what you're doing)
- chmartin 11y agoI try to lighten things up because the math by be unfamiliar.
- jmount 11y agoI think a key difference is the physics renormalization structures use fairly regular or uniform weights and the deep learning plays a lot with the weights. So there are going to be pretty big differences in behavior.
- paulpauper 11y agoI think this is similar to the scalar theory of the stock market, which uses scale invariant geometric objects to represent stock market emery levels http://greyenlightenment.com/sornette-vs-taleb-debate/ http://greyenlightenment.com/sornette-vs-taleb-debate/ Sornette’s 2013 TED video, in which he predicts an imminent stock market crash due to some ‘power law’, is also wrong because two years later the stock market has continued to rally. You write on your blog: These kinds of crashes are not caused by external events or bad players–they are endemic to all markets and result from the cooperative actions of all participants. Easier said than done. I don't think the log periodic theory is a holy grail to making money in the market. There are too many instances here it has failed, but you cherry-picked a single example with bitcoin where it could have worked.
- charleshmartin 11y agoIt is easier to apply the Sornette theory to antibubbles. Bitcoin seemed like a great example. I gotta go back and see how well the predictions actually worked.
- noobermin 11y agoAnd here I thought the renormalization group had no application outside high energy physics and condensed matter. May be I should have stuck with HEP after all.
- curiousjorge 11y agoIt always depresses me when I read anything with math formulas and esoteric terms, a constant reminder of my lifelong incompetence with math and university calculus courses.
- abecedarius 11y agoI expect very few people learned from this post; I didn't, and I kinda like math. This was the sort of math writing that makes sense only given nearly the same background as the author. (Someone above posted specific complaints about the unclear notation.)
- sgt101 11y agoI don't like the assertion at all because so many techniques are held to be "deep learning" and because even when specific techniques are built on an analogy of this sort (think Simulated Annealing and Genetic Algorithms) they do not work "because" they are "like" the physical processes that served as an inspiration. Names are useful, but only as a aide to thinking. Does this help us think about these techniques?
- deleted 11y ago[deleted]
- octatoan 11y agoNo MathJax. I am disappoint.
- fizixer 11y agoOne way of think of it is that: There are connections between Deep Learning and Theoretical Physics because there are (even stronger) connections between Information Theory and Statistical Mechanics.
- ElectricFeel 11y agoincredibly impressive research article you've posted!
- make3 11y agopeople don't use unsupervised pre training in deep learning anymore afaik