3 ms·
Karpathy on Tokenizers
- drcwpl 3y agosome key points... Tokenization is behind many of these issues • Why can't LLM spell words in images? • Why can't LLM do super simple string processing tasks like reversing a string? • Why is LLM worse at non-English languages (e.g. Japanese)? • Why is LLM bad at simple arithmetic? • Why did GPT-2 have more than necessary trouble coding in Python? • Why did an LLM abruptly halt when it sees the string "<lendoftext|>"? • What is this weird warning I get about a "trailing whitespace"? • Why the LLM break if I ask it about "SolidGoldMagikarp"? • Why should I prefer to use YAML over JSON with LLMs? • Why is LLM not actually end-to-end language modeling?