4 ms·
> somebody showed transformers could generalize I read an interesting paper recently that had a great take on this: If you add enough data, nothing is outside
by Swizec 3y ago
> somebody showed transformers could generalize
I read an interesting paper recently that had a great take on this: If you add enough data, nothing is outside training data. Thus solving the generalization problem.
Wasn’t the main point of that paper, but it made me go ”Huh yeah … I guess … technically correct?”. It raises an interesting thought that yes if you just train your neural network on everything, then nothing falls outside its domain. Problem solved … now if only compute was cheap.
- andy99 3y agoCompute and data collection. This is the long tail problem.
- sharemywin 3y agodoesn't the long tail just need to be a little outside the range of human capability for it to be AGI?
- andy99 3y agoNot sure I understand but people don't need the long tail because we don't write rules and then blindly act on them when we encounter new things. We can reason about stuff we haven't seen before.