4 ms·
> This makes it much more difficult to learn how to reason about particular data that appears inside the token, because it does not ever receive that informatio
by telotortium 3y ago
> This makes it much more difficult to learn how to reason about particular data that appears inside the token, because it does not ever receive that information.
More difficult, but it shouldn't be impossible. The Chinese managed to make rhyming dictionaries over 1000 years ago despite using a non-phonetic writing system. More relevantly, GPT-4 has definitely ingested both regular and rhyming dictionaries.
- wilg 3y agoI agree it should be possible.
- vidarh 3y agoPossible, certainly, but given the difficulties it has with it, it appears it likely requires more training. Unlike with Chinese, the LLMs will not benefit from extensive training data mapping its tokens to and from sounds or rhyming with other words, but need to derive during training from data that rarely mentions these tokens at all.