3 ms·
Really wish that more folks had read the last authoritative piece on why BPE causes serious issues for LLMs, particularly in creativity and also for things like
by Der_Einzige 3y ago
Really wish that more folks had read the last authoritative piece on why BPE causes serious issues for LLMs, particularly in creativity and also for things like math: https://gwern.net/gpt-3#bpes https://gwern.net/gpt-3#bpes
- ianand 3y agoKarpathy himself has raised some of these issues with BPE https://twitter.com/karpathy/status/1657949234535211009 https://twitter.com/karpathy/status/1657949234535211009 There is research about getting rid of tokens, some of which is in that thread
- Der_Einzige 3y agoThe gwern article significantly predates the twitter thread
- ianand 3y agoThat is fair. I didn't mean to imply it was earlier, just that he had mentioned it and research continued in this area. The article you linked is definitely more comprehensive.