4 ms·
Is this architecture actually able to generalize or is it mostly based on memorization? Have you tried some basic tasks that require generalization? e.g. number
by cpldcpu 6d ago
Is this architecture actually able to generalize or is it mostly based on memorization? Have you tried some basic tasks that require generalization? e.g. number addition etc?
- volotat 6d agoThe model is way too small and undertrained to make any generalization claims. I want to wait until it reads the whole corpus I gave and then test it on some simple established benchmarks to see how it will behave.
- jacquesm 6d agoWhat kind of hardware are you using for training? nm, I found it: > RTX 3070 Laptop GPU with 8 GB Super impressive.
- dinfinity 6d agoSeems a bit premature to make an HN post about then, imho. It's an interesting idea, but it doesn't really do anything interesting yet. I looked at the output in the training run and it is a far, far cry from intelligence. Worse than GPT-2 as it stands. I do hope it will perform well when scaled and trained, though; best of luck.