3 ms·

I would think if it were AI generated it would use common words. You know, like from its training dataset...

• flangola7 3y ago
  Tokens consist text both larger and smaller than whole words.
  • Kerb_ 3y ago
    The tokens that make up rare words would still likely be just as rare as the words.
    • flangola7 3y ago
      And yet GPT-4 has no problem crafting new words from whole cloth. If you're making new words rare tokens are useful and the model knows this.