3 ms·

Why? I'm giving the system the same prompts either way.

rdb_ 2mo ago
is it that llms write "better" typescript than let's say elixir because it has seen more of it..? or is it that you're relying on something like effect-ts to keep llms from tripping over even small things? coincidentally, "good code" in popular lang is rarely directly attributed to only that part; and it's also about the underlying principles it tries to follow in the code... another example; is it typescript that's good, or are "types" inherently making things/feedback loops easier to reason about in llms? (only using ts here for all example because it's probably one of the most "trained on" pl)
- a2ff6eeb0 2mo ago
  The first. Training data on a problem trumps most of the other considerations.
  - magnusi 2mo ago
    On the other hand, the more mainstream a programming language, the higher proportion of the training data is going to be terrible code. I think there's an optimal ratio somewhere
    - a2ff6eeb0 2mo ago
      I haven't seen that matter. LLM code has tended to be kinda samey regardless of language. Or at least it used to be when I spent time looking at it. These days I moved up the ladder of abstraction, so I don't really look; the main criteria I have is how the LLM gets things done.
      - hatefulmoron 2mo ago
        If it's samey regardless of language, isn't that in contradiction to your original theory? " .. the most important feature for agent performance is the popularity of the language .. "
        a2ff6eeb0 2mo ago
        No, not really. It's a similar flavor of output, but there's less iterations to get a correct result. The training is mostly about reducing error rates on generation.
wild_egg 2mo ago
Because the system allows for it. Lisp is a far more powerful language which enables faster iteration and development. If you give that to an LLM, it is then also able to iterate and develop faster.
- a2ff6eeb0 2mo ago
  I don't get how. I'm not interacting with the lisp, and the agents don't really get frustrated with slow compilation times or anything, and are perfectly adept at debugging. The best that people have said about lisp is that evidence LLMs perform worse with it is weak.

rdb_ 2mo ago

is it that llms write "better" typescript than let's say elixir because it has seen more of it..? or is it that you're relying on something like effect-ts to keep llms from tripping over even small things? coincidentally, "good code" in popular lang is rarely directly attributed to only that part; and it's also about the underlying principles it tries to follow in the code... another example; is it typescript that's good, or are "types" inherently making things/feedback loops easier to reason about in llms? (only using ts here for all example because it's probably one of the most "trained on" pl)

a2ff6eeb0 2mo ago
The first. Training data on a problem trumps most of the other considerations.
- magnusi 2mo ago
  On the other hand, the more mainstream a programming language, the higher proportion of the training data is going to be terrible code. I think there's an optimal ratio somewhere
  - a2ff6eeb0 2mo ago
    I haven't seen that matter. LLM code has tended to be kinda samey regardless of language. Or at least it used to be when I spent time looking at it. These days I moved up the ladder of abstraction, so I don't really look; the main criteria I have is how the LLM gets things done.
    - hatefulmoron 2mo ago
      If it's samey regardless of language, isn't that in contradiction to your original theory? " .. the most important feature for agent performance is the popularity of the language .. "
      - a2ff6eeb0 2mo ago
        No, not really. It's a similar flavor of output, but there's less iterations to get a correct result. The training is mostly about reducing error rates on generation.

wild_egg 2mo ago

Because the system allows for it. Lisp is a far more powerful language which enables faster iteration and development. If you give that to an LLM, it is then also able to iterate and develop faster.

a2ff6eeb0 2mo ago
I don't get how. I'm not interacting with the lisp, and the agents don't really get frustrated with slow compilation times or anything, and are perfectly adept at debugging. The best that people have said about lisp is that evidence LLMs perform worse with it is weak.