4 ms·

One user is able to fit more code into LLM context window since tabs take one token and spaces take 4. https://old.reddit.com/r/ChatGPTCoding/comments/15s62yl/

One user is able to fit more code into LLM context window since tabs take one token and spaces take 4. https://old.reddit.com/r/ChatGPTCoding/comments/15s62yl/save_30_of_your_tokens_and_fit_more_into_the/ https://old.reddit.com/r/ChatGPTCoding/comments/15s62yl/save...

cap11235 1y ago
For one tokenizer, a couple years ago.
zahlman 1y ago
Did these systems really not learn to recognize a run of four spaces as a single token, despite all the Python source code fed to them?