3 ms·
Karpathy himself has raised some of these issues with BPE https://twitter.com/karpathy/status/1657949234535211009 https://twitter.com/karpathy/status/1657949234
by ianand 3y ago
Karpathy himself has raised some of these issues with BPE https://twitter.com/karpathy/status/1657949234535211009 https://twitter.com/karpathy/status/1657949234535211009
There is research about getting rid of tokens, some of which is in that thread
- Der_Einzige 3y agoThe gwern article significantly predates the twitter thread
- ianand 3y agoThat is fair. I didn't mean to imply it was earlier, just that he had mentioned it and research continued in this area. The article you linked is definitely more comprehensive.