3 ms·

Why do you think that tokenization is to blame on this particular limitation?

• RicoElectrico 3y ago
  There's no inductive bias in the model for it to understand that a particular token is composed of some characters. It _does_ work for commmon words, but it's not super reliable. Transformers are powerful enough to pick up the relationship if it were in the training data, but there's like tens of thousands of tokens, right? I checked just now a simple letter counting task, it's hilarious: https://chat.openai.com/share/4d4926b1-a6fe-4815-af77-4456c7f2a6c8 https://chat.openai.com/share/4d4926b1-a6fe-4815-af77-4456c7...
  • adhesive_wombat 3y ago
    No, it's right: you might think there are 17 letters in "superconductivity" but if it's actually superconducting, r goes to zero. Hence, 16.
  • tmalsburg2 3y ago
    What you say is correct but it doesn’t explain why the model fails to capture the semantics of “is”.