3 ms·
Hi all, it's still too early to look at this code :) I wanted to put up my alpha version to start the feedback going a bit. I'm recording a video alongside wher
by karpathy 3y ago
Hi all, it's still too early to look at this code :) I wanted to put up my alpha version to start the feedback going a bit. I'm recording a video alongside where we build minBPE and expect that to be a lot more useful, coming out in a few days.
I will say tokenization is UGLY. Really ugly. There are a ton of little, subtle, sharp edges and gotchas. Eternal glory awaits anyone who figures out a way to delete this stage.
- kleiba 3y agoDo you have a link to the original Gage paper from 1994?
- kleiba 3y agoFound it myself: http://www.pennelynn.com/Documents/CUJ/HTML/94HTML/19940045.HTM http://www.pennelynn.com/Documents/CUJ/HTML/94HTML/19940045....
- Der_Einzige 3y agoReally wish that more folks had read the last authoritative piece on why BPE causes serious issues for LLMs, particularly in creativity and also for things like math: https://gwern.net/gpt-3#bpes https://gwern.net/gpt-3#bpes
- ianand 3y agoKarpathy himself has raised some of these issues with BPE https://twitter.com/karpathy/status/1657949234535211009 https://twitter.com/karpathy/status/1657949234535211009 There is research about getting rid of tokens, some of which is in that thread
- Der_Einzige 3y agoThe gwern article significantly predates the twitter thread
- ianand 3y agoThat is fair. I didn't mean to imply it was earlier, just that he had mentioned it and research continued in this area. The article you linked is definitely more comprehensive.