4 ms·
No, it’s an example that shows that LLMs still use a tokenizer, which is not an impediment for almost any task (even many where you would expect it to be, like
by Uehreka 10mo ago
No, it’s an example that shows that LLMs still use a tokenizer, which is not an impediment for almost any task (even many where you would expect it to be, like searching a codebase for variants of a variable name in different cases).
- 8note 10mo agothe question remains: is the tokenizer going to be a fundamental limit to my task? how do i know ahead of time?
- worldsayshi 10mo agoWould it limit a person getting your instructions in Chinese? Tokenisation pretty much means that the LLM is reading symbols instead of phonemes. This makes me wonder if LLMs works better in Chinese.