6 ms·

The ByT5 paper shows significantly better resiliency to typos and noise in internet scale benchmarks, as well as better reasoning capabilities with tasks like p

by yunyu 4y ago

The ByT5 paper shows significantly better resiliency to typos and noise in internet scale benchmarks, as well as better reasoning capabilities with tasks like punctuation and spacing correction: https://arxiv.org/pdf/2105.13626.pdf https://arxiv.org/pdf/2105.13626.pdf The analogy doesn’t hold because while you might not parse specific characters by default, you are trained on the character level and you can switch to that parsing mode. This simply is not possible for LLMs without hacks like inserting spaces, which then suffers from poor performance due to this being rare in pretraining. The reason byte level models aren’t as popular is because inference speed is significantly slower for a given completion length, and training for the same context window in characters is significantly more expensive I say this as someone who’s deployed ByT5 vs normal T5 in prod, having found this to be true in real world use cases. No reasonable amount of fine tuning can save GPT from performing poorly on a space insertion task (“WHOLEFDSMKT” -> “Whole Foods Market”), while ByT5 just works.

sillysaurusx 4y ago
The paper doesn’t seem to show perplexity for completions of a large dataset, i.e. the standard benchmark of language models. It shows benefits for specialized tasks, but as I said, specialized task training isn’t the goal. It’s always possible to outperform a general model by choosing a sufficiently specialized task. I don’t know why people feel so strongly that the tokenization is a weakness, but ultimately there’s not much choice but to agree to disagree.
- yunyu 4y ago
  This is obviously the case if you apply much narrower criteria, most benchmarks of existing large datasets aren’t for character level tasks. That said, the synthetic noise section should be extremely interesting if not fully representative of your criteria. Agree that tokenization isn’t a weakness for most general applications, disagree that it isn’t a weakness for the specific string manipulation task that the blog post is referencing

by yunyu 4y ago

sillysaurusx 4y ago

The paper doesn’t seem to show perplexity for completions of a large dataset, i.e. the standard benchmark of language models. It shows benefits for specialized tasks, but as I said, specialized task training isn’t the goal. It’s always possible to outperform a general model by choosing a sufficiently specialized task. I don’t know why people feel so strongly that the tokenization is a weakness, but ultimately there’s not much choice but to agree to disagree.

yunyu 4y ago
This is obviously the case if you apply much narrower criteria, most benchmarks of existing large datasets aren’t for character level tasks. That said, the synthetic noise section should be extremely interesting if not fully representative of your criteria. Agree that tokenization isn’t a weakness for most general applications, disagree that it isn’t a weakness for the specific string manipulation task that the blog post is referencing