2 ms·
The fact that LLMs are usually trained on tokens and not on characters, doesn't really speak about what generative AI is going to reach or not. >most people se
by GaggiX 2y ago
The fact that LLMs are usually trained on tokens and not on characters, doesn't really speak about what generative AI is going to reach or not.
>most people selling LLM today pretend it is AGI
Who are these "most people"?
- smokedetector1 2y agoELI5 why are tokens not a single letter?
- WhitneyLand 2y agoSuch an architecture could be implemented, it could use one token per letter, or one token per word, instead of the typical 0.75 per word we see. The choice just comes with trade-offs in memory usage, compute, and effectiveness of the model in various scenarios. So what we ended up with was a pragmatic/engineering decision rather than a theoretical or fundamental constraint.