3 ms·
I honestly don't understand why they don't just set words as tokens. Is the dictionary really that big?
by koheripbal 4y ago
I honestly don't understand why they don't just set words as tokens. Is the dictionary really that big?
- qup 4y agoYour question belies the truth in your statement.
- cyclotron3k 4y agoYes, it really is! Because you're not actually dealing with a dictionary of English words - you're dealing with all possible combinations of characters, from all unicode character sets. Here's a really good explanation: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...