3 ms·
Is Karpathy lying saying that word tokenization brings such problems that can be seen in many LLMs? https://twitter.com/karpathy/status/1657949234535211009 htt
by margorczynski 3y ago
Is Karpathy lying saying that word tokenization brings such problems that can be seen in many LLMs?
https://twitter.com/karpathy/status/1657949234535211009 https://twitter.com/karpathy/status/1657949234535211009
I'm not arguing that you can't use single chars just that many of the issues parent discussed are caused by this.
- imtringued 3y agoThe easy solution is to create an additional dataset that is token aware. I.e. you take 1% of the dataset and take random tokens and split them into smaller tokens while expecting the same answer on the character level. This should force the model to learn multiple token representations of the same character strings.
- tintor 3y agoLetter-by-letter tokenization increases inference and training costs and latency (as you need more tokens)