4 ms·
Make sure you are testing with the tiktokenizer: https://news.ycombinator.com/item?id=35460648 https://news.ycombinator.com/item?id=35460648
by ec109685 3y ago
Make sure you are testing with the tiktokenizer: https://news.ycombinator.com/item?id=35460648 https://news.ycombinator.com/item?id=35460648
- wongarsu 3y agoAh, good catch, it's actually closer to 8 tokens per LoC with GPT4's toktoken, so about twice as good. Some quick testing suggests that's mostly down to better whitespace handling.