4 ms·
I doubt that limits to generalizing NNs will be solved by mere bayesianfication. NNs poorly generalize, as with all of ML, because the strategy is the modelling
by mjburgess 7y ago
I doubt that limits to generalizing NNs will be solved by mere bayesianfication. NNs poorly generalize, as with all of ML, because the strategy is the modelling of the data, not of the world.
Since data stands in as proxy for the environment, a NN will only generalize insofar as it has the data to do so. There is no 3rd dimension in an infinite number of 2D photographs -- an no "bayesianification" will put it there.
- jacquesm 7y ago> because the strategy is the modelling of the data, not of the world. But that is how all learning works. Even the most advanced physics models are verified using experimental data. If data is found that contradicts the model the model is deemed faulty and will be adjusted to match the insights gained from the new data. DL just automates that approach aiming for the minimum gap between 'theory' and 'data' where the theory is expressed as a number of weights, thus encoding 'insight' in a - for now - non-reversible way. Just like wetware typically has not clue how it arrived at a certain conclusion, we use the words intuition and synonyms when actually we mean 'guess based on past experience'.
- tsimionescu 7y agoThat is not how human learning works. Manuals don't contain hundreds of thousands of examples to allow students to draw their own conclusion. And for inventing new physics, there are two very different approaches. One is perhaps similar to machine learning, where people look at an existing model, find experimental data that doesn't match the model, and adjust the model slightly until it matches the new data. But another mode is where you have a model with certain mathematical properties, then you discover new mathatics that give more properties, and you only then ask for more data to check these extensions.
- Schiphol 7y agoHow does human learning work, then? Do you have pointers to literature on alternative formal learning theories? I am genuinely curious :)
- tsimionescu 7y agoThere's a field called pedagogy which explores some principles of how to teach humans things.
- smallnamespace 7y ago> That is not how human learning works. As someone who has a baby in the house, I assure you that it is indeed how a human learns and refines motor skills, at the very least - by repeated experiment and play, with thousands of permutations of position, speed, weight, and situation. They also try to infer causal models [1]. Just what do you think 'play' is for a child, if not (in part) an opportunity to safely explore its surroundings and refine mental models? [1] https://www.nsf.gov/news/news_videos.jsp?cntn_id=125575&media_id=73196&org https://www.nsf.gov/news/news_videos.jsp?cntn_id=125575&medi...
- tsimionescu 7y agoThe GP's post was not about early development, but about all human learning, including higher learning and advanced physics. Sure, there are modes of learning that are pure experimentation, but there are many other modes of human learning. Even children, after some point, start developing a model and then designing experiments to test hypotheses to refine that model, which is not at all how machine learning works.
- jacquesm 7y agoHumans were learning long before manuals.
- tsimionescu 7y agoSure, manuals were just a very obvious example. Teacher-pupil and master-apprentice relationships are similar to manuals though, and these were likely present even in primitive human societies. Even in primates, copying another individual's actions is a major mode of learning, instead of random exploration.
- MiroF 7y agoPart of the problem is that we don't have any way of translating prior "theory" information into a prior over the weights of the neural network, so we are stuck assuming some sort of prior normal distribution over the weights. It's unclear if the posterior over the weights has any particularly useful meaning.