3 ms·

I wonder how many LoC that is on average. Is there an average for LoC? It’s probably based on the language…

• ethbr0 4y ago
  Finally, APL's day is here! https://en.m.wikipedia.org/wiki/APL_(programming_language) https://en.m.wikipedia.org/wiki/APL_(programming_language)
• wongarsu 4y ago
  Testing some random rust code, it's about 15-20 tokens per LoC, so about 1500-2000 LoC in a 32k context. Interestingly, using 2-space indentation, as soon as you are about 3-4 indentation levels deep you spend as many tokens on indentation as on the actual code. For example, "log::LevelFilter::Info" is 6 tokens, same as 6 consecutive spaces. There are probably a lot of easy gains here reformatting your code to use longer lines or maybe no indentation at all.
  • ec109685 4y ago
    Make sure you are testing with the tiktokenizer: https://news.ycombinator.com/item?id=35460648 https://news.ycombinator.com/item?id=35460648
    • wongarsu 4y ago
      Ah, good catch, it's actually closer to 8 tokens per LoC with GPT4's toktoken, so about twice as good. Some quick testing suggests that's mostly down to better whitespace handling.
  • JimDabell 4y ago
    > There are probably a lot of easy gains here reformatting your code to use longer lines or maybe no indentation at all. Using tabs for indentation needs fewer tokens than using multiple spaces for indentation.
    • mafuy 4y ago
      Curious how this point in the tabs vs. spaces debate re-emerges
• omega3 4y ago
  How do this even work if the code also uses external libraries?
• MacsHeadroom 4y ago
  A LoC is ~7 tokens thanks to the new toktoken tokenization in GPT-4. 32k is ~4.5k LoC