3 ms·
Yeah, it's a very silly article with wrong mathematical reasoning. Hinton is quite obviously talking about a much more information-theoretic approach to the pro
by tysam_and 3y ago
Yeah, it's a very silly article with wrong mathematical reasoning. Hinton is quite obviously talking about a much more information-theoretic approach to the process, but he's phrasing it in people-friendly terms.
What's a little more concerning to me is that people are reading and upvoting it. I think, because I have hopes and aspirations about working on some very hard problems and communicating them to the public at some point. And if this is the level of 'ooh, squirrel'! that we're going at, that the work that I make might get overshadowed by something really silly.
Perhaps an odd insecurity, but there it is, I think.
- nighthawk454 3y agoYeah I agree with that for sure. It’s so strange how the majority of the research folks appear to be on this ‘new shiny’ mentality at the expense of fundamentals. Especially for how new this field is, relatively. It’s not exactly like we’re all tapped out. Probably not even of low hanging fruit.
- tysam_and 3y agoYeah, if you read Hinton's long backlog... It's mindblowing. So many fresh concepts, just left there in the dust. Highly encourage. There's a goldmine in there, I thinksies. <3 :')))))))))
- theGnuMe 3y ago>information-theoretic approach to the process Can you elaborate on this? I've studied some information theory and I don't see it.
- nighthawk454 3y agoI think the analogy is something like: if you have a simple distribution over all words, then that's just word frequency. Obviously not a good predictor. The 'information' necessary to predict the correct next word contextually is just not there if you're predicting words in a vacuum. In order to be practically useful and predict the right words _in context_, the model must be conditioning off of more of the sentence/document (aka more information). So it should not be surprising that a 'glorified autocomplete' has some degree of "understanding" as it would be impossible for it to be any good as an autocomplete-er otherwise.
- theGnuMe 3y agoThat's not information theoretic, that's just conditional probability.
- nighthawk454 3y agoit is conditional probability, but that is a fundamental concept used in information theory
- tysam_and 3y agoYou might want to take another look at Shannon's paper, lol, this statement is quite contradictory. Probability _is_ the backbone of information theory, dude! It's quite incredible.