4 ms·
>> If a model truly generalizes, why wouldn't it be possible for it to generate more real signal than was used for training it? If by "a model [that] truly gen
by YeGoblynQueenne 4y ago
>> If a model truly generalizes, why wouldn't it be possible for it to generate more real signal than was used for training it?
If by "a model [that] truly generalizes" you mean one that generalises outside its training distribution, then neural nets can't train such models. I don't reckon any machine learning approach can do that.
In any case this is ImageNet we're talking about that's been done to death many times over already. Far from learning more general models, after a certain point in time improvements in accuracy have meant that models are getting better at overfitting. Same with MNIST, where that happened a long time ago.
Obligatory reference to defend against this comment being knee-jerked to oblivion:
>> This stands in sharp contrast with what deep nets do, which I would call "local generalization": the mapping from inputs to outputs performed by deep nets quickly stops making sense if new inputs differ even slightly from what they saw at training time.
Francois Chollet, The limitations of deep learning
https://blog.keras.io/the-limitations-of-deep-learning.html https://blog.keras.io/the-limitations-of-deep-learning.html
- moyix 4y agoI genuinely don't see how this viewpoint is compatible with the grokking results.
- YeGoblynQueenne 4y agoI'm sorry, I'm not familiar with the grokking results. What are you referring to?
- MrScruff 4y agoFrom the linked blog post (2017) > Even with this data, you could not train a deep learning model to simply read a product description and generate the appropriate codebase. Would be make the same claim now I wonder?
- YeGoblynQueenne 4y agoI wonder too. I've communicated with Chollet in the past (about his ARC dataset) and he's responsive. You could email him and ask, or you could ask him on his twitter (I don't use twitter).
- deleted 4y ago[deleted]
- YeGoblynQueenne 4y agoSorry, I can't edit my earlier comment but I re-read our conversation and I realise I misread your quote. In particular: >> read a product description and generate the appropriate codebase Chollet here is talking about creating an entire application, not short code snippets or larger functions. Producing short programs from specifications was already possible when Chollet wrote the above-linked article. It is still not possible to genera the entire codebase for an application by reading a product description. Not anywhere near that.
- MrScruff 4y agoAgreed, but the question is would he still claim it's not possible to train a deep model to achieve this goal at all. Maybe not, but it feels a lot less certain than it did in 2017.