4 ms·
Some companies like Anthropic haven't publicly released the tokenizer used in their API, thereby making it impossible for this library to use the appropriate to
by spencerchubb 2y ago
Some companies like Anthropic haven't publicly released the tokenizer used in their API, thereby making it impossible for this library to use the appropriate tokenizer in all cases. Be careful about how you use the word 'obviously'