3 ms·
A while ago I benchmarked different tokenizers with a few common C++ coding styles. Depending on the combination I was able to reduce the token usage by as much
by floriangoebel 2mo ago
A while ago I benchmarked different tokenizers with a few common C++ coding styles. Depending on the combination I was able to reduce the token usage by as much as 5% just by auto formatting the codebase with clang-format.
Of course, this doesn't necessarily mean that a coding agent would perform better, but it was a fun experiment.