5 ms·

When you're working on something difficult that requires a model to reason intelligently, lower level and strongly typed languages often outperform on the same

When you're working on something difficult that requires a model to reason intelligently, lower level and strongly typed languages often outperform on the same problems [0]. We have a few hypotheses about why, with a moderately high correlation between performance and token density of the output program -- i.e. more token dense languages are more difficult for programs to reason about. Most models come up with the least effective solutions when writing Python. [0] https://gertlabs.com/rankings https://gertlabs.com/rankings

ajb 5mo ago
That's a very interesting page, but the language ranking is wildly different for "average percentage" (python bottom) and "success rate" (python second). Sounds like there is some subtly about this.
- gertlabs 5mo ago
  Success rate is essentially loading/compilation success + ability to adhere to the environments' rules. For one-shot responses, the majority of failures are environmental/syntax, which naturally favors interpreted languages. For longer agentic coding sessions, models solve the environment issues quickly and it becomes a fair comparison of who comes up with the smarter solution. You can filter for that here: https://gertlabs.com/rankings?mode=agentic_coding https://gertlabs.com/rankings?mode=agentic_coding
christophilus 5mo ago
I ran a little test with Go, TypeScript, Clojure, F#, Haskell, and Rust. Token count was roughly in the same ballpark, but it used the fewest for TypeScript, then Go. The rest required a bit more. Clojure always won in terms of lines of code though, generally coming in at 1/2 the size of the Go or Typescript solutions.
- iLemming 5mo ago
  I bet you have not tested it with live Clojure REPL. When you give the LLM living, breathing REPL, it stops guessing and starts empirically analyzing current state of things and produces working solution faster, costing far less tokens. This article doesn't seem to mention it either https://martinalderson.com/posts/which-programming-languages-are-most-token-efficient https://martinalderson.com/posts/which-programming-languages... even though states Clojure to be the most token efficient. I personally, honestly don't care much. In my opinion (using LLMs with multiple different languages), specifics of PLs don't matter to the point of stating a "clear winner". It's not the language that matters but the "stories you tell" with the language. And greatest stories sometimes told in the languages people long forgotten.
  - e12e 5mo ago
    Curious what that would look like - and if javascript, ruby would benefit equally? How do you work with LLM and repl?
    - iLemming 5mo ago
      When I say REPL, I specifically mean "Lisp REPL". Every step in Read.Eval.Print.Loop slightly differs in homoiconic languages like Clojure, CL, Fennel, Elisp, etc. Javascript and Ruby in that sense can only "benefit" if they have a homoiconic language on top - e.g. Clojurescript.
      - e12e 5mo ago
        How so? I don't think anyone would consider Smalltalk to be homoiconic? But would consider it to support a repl-like development process? Smalltalk has system images - which AFAIK clojure lacks (as does python, ruby). I wonder if it would be possible to pair python ZODB with storing python code alongside the pickled objects... And effectively create an unholy image-like workflow with IPython and ZODB? But at any rate, I was more curious about how you mix repl, clojure and LLMs in practice?
        iLemming 5mo ago
WatchDog 5mo ago
Your theory about token density seems reasonable, but your data doesn't seem to really match it. Very little difference between TypeScript and JavaScript, which are essentially the same language, just one has more tokens. Functional languages like Clojure and OCaml are pretty dense, I would have expected them to feature lower. Kotlin is in some ways a more token dense version of Java, yet Kotlin leads, and Java is almost last.
nylonstrung 5mo ago
Even though Python code may use more characters/LoC than say Rust in text form, it's not necessarily more token dense because LLM tokenizers are good at "compressing" its English keywords In contrast, langs with symbol-heavy syntax (ALP as extreme example) use fewer characters but don't tokenize well in practice so aren't as efficient as one would think

ajb 5mo ago

That's a very interesting page, but the language ranking is wildly different for "average percentage" (python bottom) and "success rate" (python second). Sounds like there is some subtly about this.

gertlabs 5mo ago
Success rate is essentially loading/compilation success + ability to adhere to the environments' rules. For one-shot responses, the majority of failures are environmental/syntax, which naturally favors interpreted languages. For longer agentic coding sessions, models solve the environment issues quickly and it becomes a fair comparison of who comes up with the smarter solution. You can filter for that here: https://gertlabs.com/rankings?mode=agentic_coding https://gertlabs.com/rankings?mode=agentic_coding

christophilus 5mo ago

I ran a little test with Go, TypeScript, Clojure, F#, Haskell, and Rust. Token count was roughly in the same ballpark, but it used the fewest for TypeScript, then Go. The rest required a bit more. Clojure always won in terms of lines of code though, generally coming in at 1/2 the size of the Go or Typescript solutions.

iLemming 5mo ago
I bet you have not tested it with live Clojure REPL. When you give the LLM living, breathing REPL, it stops guessing and starts empirically analyzing current state of things and produces working solution faster, costing far less tokens. This article doesn't seem to mention it either https://martinalderson.com/posts/which-programming-languages-are-most-token-efficient https://martinalderson.com/posts/which-programming-languages... even though states Clojure to be the most token efficient. I personally, honestly don't care much. In my opinion (using LLMs with multiple different languages), specifics of PLs don't matter to the point of stating a "clear winner". It's not the language that matters but the "stories you tell" with the language. And greatest stories sometimes told in the languages people long forgotten.
- e12e 5mo ago
  Curious what that would look like - and if javascript, ruby would benefit equally? How do you work with LLM and repl?
  - iLemming 5mo ago
    When I say REPL, I specifically mean "Lisp REPL". Every step in Read.Eval.Print.Loop slightly differs in homoiconic languages like Clojure, CL, Fennel, Elisp, etc. Javascript and Ruby in that sense can only "benefit" if they have a homoiconic language on top - e.g. Clojurescript.
    - e12e 5mo ago
      How so? I don't think anyone would consider Smalltalk to be homoiconic? But would consider it to support a repl-like development process? Smalltalk has system images - which AFAIK clojure lacks (as does python, ruby). I wonder if it would be possible to pair python ZODB with storing python code alongside the pickled objects... And effectively create an unholy image-like workflow with IPython and ZODB? But at any rate, I was more curious about how you mix repl, clojure and LLMs in practice?
      - iLemming 5mo ago

WatchDog 5mo ago

Your theory about token density seems reasonable, but your data doesn't seem to really match it. Very little difference between TypeScript and JavaScript, which are essentially the same language, just one has more tokens. Functional languages like Clojure and OCaml are pretty dense, I would have expected them to feature lower. Kotlin is in some ways a more token dense version of Java, yet Kotlin leads, and Java is almost last.

nylonstrung 5mo ago

Even though Python code may use more characters/LoC than say Rust in text form, it's not necessarily more token dense because LLM tokenizers are good at "compressing" its English keywords In contrast, langs with symbol-heavy syntax (ALP as extreme example) use fewer characters but don't tokenize well in practice so aren't as efficient as one would think