5 ms·
>> Over the last few decades, innate priors have gone out of fashion, and today Deep Learning research prizes closely-supervised end-to-end learning (supported
by YeGoblynQueenne 9y ago
>> Over the last few decades, innate priors have gone out of fashion, and today Deep Learning research prizes closely-supervised end-to-end learning (supported by big-data and big-compute) as the dominant paradigm.
I don't get why deep learning researchers are so hung up on learning everything from scratch. The tendency for ever more compute and data is just unsustainable. For problems like natural language, where distinct evens may be infinite, you can keep throwing data at the problem and you'll never make a dent to it. There are problems that grow at a pace that cannot be matched by any computer, no matter how powerful.
What's more, as a civilisation we have nothing if not knowledge about the world. We have been accumulating it for thousands of years. It's what makes the difference between an intelligent human and an educated intelligent human. And it's a big difference. So, if we have all this background knowledge, why not use it, and make our lives easier?
- SiempreViernes 9y agoI guess they never got a visit from Minsky, and thus haven’t reached enligthenment. ======================= Sussman attains enlightenment In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. “What are you doing?”, asked Minsky. “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied. “Why is the net wired randomly?”, asked Minsky. “I do not want it to have any preconceptions of how to play”, Sussman said. Minsky then shut his eyes. “Why do you close your eyes?”, Sussman asked his teacher. “So that the room will be empty.” At that moment, Sussman was enlightened ==================== http://spetharrific.tumblr.com/post/26600309788/sussman-attains-enlightenment http://spetharrific.tumblr.com/post/26600309788/sussman-atta...
- smallnamespace 9y agoI think it's a pretty well established in learning theory that the combination of neural network topology + the training procedure sets up an implicit prior (it establishes a bias towards the type of things the network will learn more easily from the input data). In fact, this is a corollary of the No Free Lunch Theorem. The problem is that these are extremely high-dimensional spaces, and even narrowing down the space within which the prior might be defined (much less finding the prior itself) is mathematically difficult. Hence the progress is slow and halting—it's consisted of discussions like what the representations fully-trained neural networks find within the early activation layers, which means the question is being investigated empirically, without the benefit of a fully-fledged theory. Initeresting, the 'nature vs. nurture' debate can be viewed as basically a discussion of learning priors in the context of human beings. Turns out the truth is pretty subtle—for many problem domains, humans (and animals) can be made to learn a very large set of things with repeated training, but we also have strong priors, so some things can be taught much more easily and quickly than others. Our brain networks are primed to learn certain things, e.g. ducklings have a prior that the first animate object they see is their mother. It's possible to force a duckling to unlearn its maternal imprinting, but only with repeated and large amounts of negative conditioning. It seems pretty likely that this prior is embedded in both the topology and neurochemical functioning of a normally developing brain.
- rhaps0dy 9y ago> the combination of neural network topology + the training procedure sets up an implicit prior [...] That's what I got from the koan. Sussman thinks that a randomly wired NN has no prior, but that's false. Same as Minsky "thinks" that the room is empty when he closes his eyes. EDIT: actually if you read the source (spetharrific.tumblr.com) it spells this out.
- make3 9y ago[Deleted]
- Scea91 9y ago> it's gone out of fashion because it's been beaten extremely convincingly by methods that learned everything from scratch, in every single task Totally false. By every single task you probably mean "every deep learning success story" which would be tautological. There are tons of tasks where deep learning doesn't quite work yet and you still have to hand-craft your features. Example from my domain: Try learning a classifier for malware which uses just raw binaries on input.
- nl 9y agoMy understanding is that work on deep learning for malware classification is pretty early. Having said that, my understanding is that they are getting closer. https://pdfs.semanticscholar.org/f1c8/7533e628fd374b9e98f1e1ccc18a0e8f195d.pdf https://pdfs.semanticscholar.org/f1c8/7533e628fd374b9e98f1e1... https://devblogs.nvidia.com/malware-detection-neural-networks/ https://devblogs.nvidia.com/malware-detection-neural-network...
- Scea91 9y agoResults in this field usually look way better than they would be in a production environment. Regarding articles you mentioned: Using ROC curve for evaluation in this case is a red-flag because it doesn't take the data imbalance into account. Precision-Recall curve is way more suitable. You can have great AUC on the ROC curve but precision can be near zero in highly imbalanced problems such as malware detection. Precision is the probability that a positive detection is a true positive. Which is usually the measure you are most interested in. The problem is that precision changes if you change the class priors. Because of that, the results are always very dataset specific. With that said, I do not say that machine learning or neural networks do not work on this task. It just doesn't work in the end-to-end manner where you just feed raw binaries as input to some generic architecture as we can do with images in some tasks.
- 9y ago
- tanilama 9y ago>> I don't get why deep learning researchers are so hung up on learning everything from scratch. It is easier to formulate the problem when you are dealing with only one specific task. Transfer learning is useful, but in its current format it is limited, and can only borrow features from tasks that are similar. Meta-learning is trying to tackle this problem specifically, promising result is presented, but still long way to go to be useful in real-world production. In order to NOT learning from scratch, we need understand the representation of knowledge. For neural networks, the knowledge is parameters that learnt. However, the parameter is tied to the task that your trained your network with, so we are back to square one. I doubt continuous differentiable tensors are the ONLY representation of human's knowledge. It might represent the hard-to-explain intuition part of it, however, a large, probably larger part of our knowledge is better represented by category/hierarchy/graph/rules that is not easily differentiable, which current deep learning techniques struggle to model. So, IMO, it is not really that researchers are obsessed with supervised learning, it is probably that they have no better alternatives.
- sharemywin 9y ago"Do not try and bend the spoon, that's impossible. Instead, only try to realize the truth...there is no spoon. Then you'll see that it is not the spoon that bends, it is only yourself." ―Spoon Boy to Neo category/hierarchy/graph/rules are important but I suspect only our conscious/social minds works in that abstraction.
- tanilama 9y agoNevertheless, our brain is able to model such abstractions just fine. The capacity to represent such non-differentiable structure is the key.
- sharemywin 9y agoBut the representation isn't static. We can learn new symbol systems and adjust them all the time. Plus create rules of thumb to make re-processing easier. and go back and/forth. Think of an elephant. now an Asian elephant walking by a river. your image probably was pretty abstract then got more vivid. Then re-adjusted to add the river. So you went from symbols to a more detailed model like a generative model, but you probably only have had a few examples of elephants to generate from.
- singularity2001 9y ago>> I don't get why deep learning researchers are so hung up on learning everything from scratch. they are not: the technique of transfer learning is a strong prior which is now widely used.
- MaxBarraclough 9y ago> I don't get why deep learning researchers are so hung up on learning everything from scratch. It's a better approach if we want to develop generalisable learning systems, no?
- YeGoblynQueenne 9y agoIn theory, perhaps- in practice it's not feasible. I was having a similar conversation today with my thesis supervisor and he quoted something Turing had said in his original machine intelligence paper, that "There's not enough information in data to learn from nothing". This was in the context of machine learning, because Turing was wondering about the best way to create a learning machine. I think the bottom line is that he didn't think you could do it without some background knowledge, of some sort. Edit: I haven't read Turing's paper recently and I might be misquoting him, but I do think that was his general intuition, that you can't just learn the world by observing the world.
- MaxBarraclough 9y ago> There's not enough information in data to learn from nothing Given that AlphaZero just happened, we now know that isn't true, no?
- posterboy 9y agoThe alphazero paper mentions that go has especially simple rules and that chess and shogi are more complicated. But the rules are still pretty clear. To some extend, they must have at least hard coded in the training data who won a particular game; or the game rules have to be hard coded. Something has to be hard coded. That is far from "nothing" prior! But the paper is cryptic to me, overall ("Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm"). Also, the parent comment is paradox. Data is information and that is something, so learning from data is learning from something.
- YeGoblynQueenne 9y ago
- andreyk 9y agoI don't think this is actually true for most deep learning researchers. There is a lot of work now on incorporating knowledge bases and more domain info in learning, and it's likely to just get more and more common.