3 ms·
I strongly suspect it's tokenization that drives this. If we trained with ASCII or even UTF8, I think we'd have much better results.
by mikewarot 3mo ago
I strongly suspect it's tokenization that drives this. If we trained with ASCII or even UTF8, I think we'd have much better results.
- bellowsgulch 3mo agoSelf-attention is an O(n^2) operation. You don't want to train on individual characters, or bytes.