4 ms·
We can, tokenization is literally just to maximize resources and provide as much "space" as possible in the context window. There is no advantage to tokenizati
by cco 2y ago
We can, tokenization is literally just to maximize resources and provide as much "space" as possible in the context window.
There is no advantage to tokenization, it just helps solve limitations in context windows and training.
- TZubiri 2y agoI like this explanation