3 ms·
Even though Python code may use more characters/LoC than say Rust in text form, it's not necessarily more token dense because LLM tokenizers are good at "compre
by nylonstrung 5mo ago
Even though Python code may use more characters/LoC than say Rust in text form, it's not necessarily more token dense because LLM tokenizers are good at "compressing" its English keywords
In contrast, langs with symbol-heavy syntax (ALP as extreme example) use fewer characters but don't tokenize well in practice so aren't as efficient as one would think