3 ms·
>I'm not arguing that just scaling it up the best approach-- but just that faulting its forgetting effect is not the most compelling criticism because that one
by Barrin92 6y ago
>I'm not arguing that just scaling it up the best approach-- but just that faulting its forgetting effect is not the most compelling criticism because that one almost certainly can be addressed by scaling.
it can't because the size of the system obviously completely explodes to practically infinity if you try to learn the past just by absorbing random factoids.
This is clearly not how human memory works. If I ask you "did you float through your bedroom on march 2nd 2015 at 11 pm?", you don't consult some sort of history you have burned in a neural net, you make an inference using the laws of physics to conclude that you didn't because that's impossible.
These neural nets don't reason this way, they don't have high level understanding of ontologies or laws and they can't make inferences like this, I haven't tested GPT-3 but I assume it can't even reliably solve basic algebra that isn't found in the dataset.
So memory is a function of combining high level understanding of the world with data, not shoving history into an encoder.
- nullc 6y ago> I assume it can't even reliably solve basic algebra that isn't found in the dataset. ::sigh:: It can, even though there are specific technical reasons that working with both numbers and exact answers is hard for it. (The way the input text is encoded combines digits in an inconsistent and formatting specific way, and the way text is generated isn't just to take the models most likely output but to randomly sample among the top couple most likely outputs, which is an obvious handicap when there is a single right answer.) https://pbs.twimg.com/media/EdHuMgsWsAEk-29?format=png&name=large https://pbs.twimg.com/media/EdHuMgsWsAEk-29?format=png&name=... https://pbs.twimg.com/media/EdII5jhXkAA8eKv?format=png&name=medium https://pbs.twimg.com/media/EdII5jhXkAA8eKv?format=png&name=... (lines starting with > are the human)