3 ms·

I’ll be thrilled when Siri can spell my wife’s name correctly after 13 years of continuous usage and explicitly training it in her name. Admittedly her name is

by tomcam 3y ago

I’ll be thrilled when Siri can spell my wife’s name correctly after 13 years of continuous usage and explicitly training it in her name. Admittedly her name is wildly complicated and totally unknown to the software folks at Apple: Ada Also my radio-trained voice is so generic a caller every week-ish assumes I am a bot, so I’m pretty sure the problem isn’t me enunciation or accent.

margorczynski 3y ago
LLMs can't really spell as the smallest "block" of information they operate on are tokenized whole words. Same issue with e.g. arithmetic.
- ta8645 3y ago
  But they should be able to select the correct token for homophones, which amounts to the same thing.
- jxy 3y ago
  Can we stop this kind of misinformation? Training a model to map a token to individual letters are no harder than training a model to be fluent at English. Arithmetics with small number of digits are achievable as well. You can just try a small 7B model yourself. If you don't know where to start, try the mistral instruct v0.2, and this is how it goes, > [INST] Spell out the following word letter by letter: margorczynski [/INST] m - a - r - g - o - r - c - z - y - n - s - k - i > So, the word "margorczynski" spelled out letter by letter is: m-a-r-g-o-r-c-z-y-n-s-k-i. The text between `[INST]` and `[/INST]` is the input. The text after `[/INST]` is the output.
  - margorczynski 3y ago
    Is Karpathy lying saying that word tokenization brings such problems that can be seen in many LLMs? https://twitter.com/karpathy/status/1657949234535211009 https://twitter.com/karpathy/status/1657949234535211009 I'm not arguing that you can't use single chars just that many of the issues parent discussed are caused by this.
    - imtringued 3y ago
      The easy solution is to create an additional dataset that is token aware. I.e. you take 1% of the dataset and take random tokens and split them into smaller tokens while expecting the same answer on the character level. This should force the model to learn multiple token representations of the same character strings.
    - tintor 3y ago
      Letter-by-letter tokenization increases inference and training costs and latency (as you need more tokens)
ben_w 3y ago
> Admittedly her name is wildly complicated and totally unknown to the software folks at Apple: Ada Aye. I was surprised this morning when it decided I had was talking about a "Mark of chain". 1/3rd of the time it hears "bedroom 100%" as "bedroom off". When cooking dinner today, I asked for a "ten minute timer", it responded "for how long?" then confirmed my "ten minute minute timer". Still better than Alexa, which kept telling us it couldn't find «kitchen» on Spotify even though we didn't even have Spotify. And way better than the voice control on Mac OS Classic; back in the late 90s/early 00s, it interpreted 75% of my attempts to use it as "tell me a joke" (it wasn't even a good joke), and ignored 20%.

by tomcam 3y ago

margorczynski 3y ago

LLMs can't really spell as the smallest "block" of information they operate on are tokenized whole words. Same issue with e.g. arithmetic.

ta8645 3y ago
But they should be able to select the correct token for homophones, which amounts to the same thing.
jxy 3y ago
Can we stop this kind of misinformation? Training a model to map a token to individual letters are no harder than training a model to be fluent at English. Arithmetics with small number of digits are achievable as well. You can just try a small 7B model yourself. If you don't know where to start, try the mistral instruct v0.2, and this is how it goes, > [INST] Spell out the following word letter by letter: margorczynski [/INST] m - a - r - g - o - r - c - z - y - n - s - k - i > So, the word "margorczynski" spelled out letter by letter is: m-a-r-g-o-r-c-z-y-n-s-k-i. The text between `[INST]` and `[/INST]` is the input. The text after `[/INST]` is the output.
- margorczynski 3y ago
  Is Karpathy lying saying that word tokenization brings such problems that can be seen in many LLMs? https://twitter.com/karpathy/status/1657949234535211009 https://twitter.com/karpathy/status/1657949234535211009 I'm not arguing that you can't use single chars just that many of the issues parent discussed are caused by this.
  - imtringued 3y ago
    The easy solution is to create an additional dataset that is token aware. I.e. you take 1% of the dataset and take random tokens and split them into smaller tokens while expecting the same answer on the character level. This should force the model to learn multiple token representations of the same character strings.
  - tintor 3y ago
    Letter-by-letter tokenization increases inference and training costs and latency (as you need more tokens)

ben_w 3y ago

> Admittedly her name is wildly complicated and totally unknown to the software folks at Apple: Ada Aye. I was surprised this morning when it decided I had was talking about a "Mark of chain". 1/3rd of the time it hears "bedroom 100%" as "bedroom off". When cooking dinner today, I asked for a "ten minute timer", it responded "for how long?" then confirmed my "ten minute minute timer". Still better than Alexa, which kept telling us it couldn't find «kitchen» on Spotify even though we didn't even have Spotify. And way better than the voice control on Mac OS Classic; back in the late 90s/early 00s, it interpreted 75% of my attempts to use it as "tell me a joke" (it wasn't even a good joke), and ignored 20%.