3 ms·
> and used tiktoken to approximate tokens, by dividing character count by four. This is just the type of thing that stands out. You used a tokenizer library t
by wolttam 3mo ago
> and used tiktoken to approximate tokens, by dividing character count by four.
This is just the type of thing that stands out.
You used a tokenizer library to determine the number of tokens in your text by dividing its length by 4.
That makes no sense!