3 ms·
This depends on what your workflow is. There are use cases for tokenization that don't always involve immediately feeding the text into a model.
by michaelmior 2mo ago
This depends on what your workflow is. There are use cases for tokenization that don't always involve immediately feeding the text into a model.
- casey2 2mo agoThere are more usecases, for this class of tokenizer, now that it's 1000x faster as well. RAG being one example.