3 ms·

That's a lot of words to not explain what the thing is. Any example tokens?

• PaulHoule 4y ago
  See Table 4
  • superkuh 4y ago
    Table 4's BERT/Semantic tokenizer examples are full of word splittings that only a chemist could love. The BERT/WordPiece splittings make more semantic sense to me.
    • sp332 4y ago
      The problem is that each prefix gets a separate token, so the LLM has to learn from context that they refer to the same thing. In the semantic encoding, all of the words start with the same token, so the LLM can learn that the words are related with a lot less data.
      • thomastjeffery 4y ago
        Your paragraph told me so much more than the abstract in the OP.