3 ms·
Yeah... it obviously uses the appropriate tokenizer
by weird-eye-issue 2y ago
Yeah... it obviously uses the appropriate tokenizer
- spencerchubb 2y agoSome companies like Anthropic haven't publicly released the tokenizer used in their API, thereby making it impossible for this library to use the appropriate tokenizer in all cases. Be careful about how you use the word 'obviously'