3 ms·
No wonder GPT does so horribly on anything involving spelling, or the exact specifications of letters. To fix it, I'd throw a few gigabytes of synthetic data i
by mikewarot 3y ago
No wonder GPT does so horribly on anything involving spelling, or the exact specifications of letters.
To fix it, I'd throw a few gigabytes of synthetic data in the training mix before fine tuning that included the alphabets of all the relevant languages, things like.
A is an upper case a
a is a lower case A
the sequence of numbers is 0 1 2 3 4 5 6 7 8 9 10 11 12
0 + 1 = 1
1 + 1 = 2
etc.
It still amazes me that Word2Vec is as useful as it is, let alone LLMs. The structure inherent in language really does convey far more meaning that we assume. We're like fish, not being aware of water, when we use language.
- sebzim4500 3y agoWe know OpenAI trains on significant amounts of synthetic data, they probably have something like this.
- CGamesPlay 3y agoI am curious if an LLM trained using a tokenizer that renders words into IPA alphabet would make a better bot for creative writing, especially things like rhyming, assonance, puns, and other sound-based word games. It might also do better on "fringe" languages, where there is low corpus in the language but the words might have cognates in more widely known languages.
- pyinstallwoes 3y agoHumans, a host form. Language, a life form.