4 ms·

ChatGPT doesn't even work with "words" to begin with but with vectors encoding meaning of words. At least as far as I understand it. That's why it is able to ca

• tyfon 4y ago
  It actually works with "less than words", tokens that can encode either a whole word or part of it. Example might be "you" as a single token, but "craftsmanship" might be 5-10 tokens depending on the encoder. It has absolutely no encoding of the meaning, however it does have something called an "attention" matrix that it trains itself to make sure it is weighing certain words more than others in it's predictions. So words like "a", "the" etc will eventually count for less than words like "cat", "human", "car" etc when it is predicting new text.