3 ms·
Testing some random rust code, it's about 15-20 tokens per LoC, so about 1500-2000 LoC in a 32k context. Interestingly, using 2-space indentation, as soon as y
by wongarsu 4y ago
Testing some random rust code, it's about 15-20 tokens per LoC, so about 1500-2000 LoC in a 32k context.
Interestingly, using 2-space indentation, as soon as you are about 3-4 indentation levels deep you spend as many tokens on indentation as on the actual code. For example, "log::LevelFilter::Info" is 6 tokens, same as 6 consecutive spaces. There are probably a lot of easy gains here reformatting your code to use longer lines or maybe no indentation at all.
- ec109685 4y agoMake sure you are testing with the tiktokenizer: https://news.ycombinator.com/item?id=35460648 https://news.ycombinator.com/item?id=35460648
- wongarsu 4y agoAh, good catch, it's actually closer to 8 tokens per LoC with GPT4's toktoken, so about twice as good. Some quick testing suggests that's mostly down to better whitespace handling.
- JimDabell 4y ago> There are probably a lot of easy gains here reformatting your code to use longer lines or maybe no indentation at all. Using tabs for indentation needs fewer tokens than using multiple spaces for indentation.
- mafuy 4y agoCurious how this point in the tabs vs. spaces debate re-emerges