4 ms·

No this isn't mindless reinventing --- this is finally having clear empiric evidence for something that we knew the entire time. This is a huge relief! Next up

No this isn't mindless reinventing --- this is finally having clear empiric evidence for something that we knew the entire time. This is a huge relief! Next up is demonstrating the AI is more productive with better programming languages.

tikhonj 2mo ago
I've already seen at least one promising experiment about how static checking helps LLMs: https://arxiv.org/abs/2606.01522 https://arxiv.org/abs/2606.01522 Key parts of the abstract: > This raises a question the programming-language community has not previously had reason to ask: should error-message detail be calibrated differently for AI agents than for humans? > We investigate this question through a controlled experiment using Shplait, an ML-style statically typed language. We construct a suite of programs containing a single deliberate type error each, and measure how often an AI agent repairs them under ablation: a detailed error context using the unification stack; a proximate error location; a minimal type error; and a dynamic (test suite) error only. An automated oracle uses a test suite to classify each repair attempt as a type error, semantically incorrect, or semantically correct. > We find concrete evidence that more detailed error messages generally improve an agent's ability to fix type errors. We also find that the presence of a type system appears to help more than only test suite failure reports.
kagevf 2mo ago
> Next up is demonstrating the AI is more productive with better programming languages. That actually would be pretty exciting! But, wouldn't AI be biased toward more popular languages, since those will by nature of their popularity provide more sources for training material? Even still, the AI could demonstrate which algorithms or maybe patterns and techniques are more productive, in the context of the popular languages.
- bobthepanda 2mo ago
  That’s not necessarily a bad thing. As an example, 110v American outlet plugs are almost certainly not the most efficient way to power devices and appliances; but, because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances in the US. Similar story with USB C.
  - mkehrt 2mo ago
    Outlets benefit from standards, however. Every device (well kind of) uses the same few plug types, and a different plug would be a disadvantage. On the other hand, having a bunch of different languages is an advantage. People can pick the best tool for the job, and experiment with new language features.
  - kagevf 2mo ago
    > That’s not necessarily a bad thing. True, and to my point, "popular" isn't necessarily "better". > because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances Yes, but LLMs can do the heavy lifting when analyzing the "better programming languages" for productivity. There will be a bias toward popularity in their training sources, though.
- gadflyinyoureye 2mo ago
  You might be surprised. I've had an AI start spitting out none functioning python code. I've rewritten it with Go. The requirements are that every change has to pass compilation, linting and tests. Same requirements as Python. Go keeps the AI on the rails better. The compilation failing provides quick and probably better feedback than a whole dictionary of tests.
  - kagevf 2mo ago
    For sure. Its code output is definitely not to be fully trusted. Oftentimes, you can point at where it made a mistake and it usually fixes it too.
keeganpoppen 2mo ago
i can't wait for that one haha
seanmcdirmid 2mo ago
> Next up is demonstrating the AI is more productive with better programming languages. This is hard to do given the current reliance on model weights in its programming language knowledge. You’ll get your best results for python or some variation of python. But
- CuriouslyC 2mo ago
  Not true. Python performance is actually pretty low, and Elixir, Kotlin and C# trounce it handily.
  - seanmcdirmid 2mo ago
    Do you have citations? Everything I’ve read have said they have an overwhelming bias to Python; eg https://arxiv.org/html/2503.17181v1 https://arxiv.org/html/2503.17181v1 and https://arxiv.org/html/2505.13004v1 https://arxiv.org/html/2505.13004v1
    - CuriouslyC 2mo ago
      Agents love writing Python, but it turns out they're not amazing at it. https://autocodebench.github.io/ https://autocodebench.github.io/
      - seanmcdirmid 2mo ago
        I’m not really sure how to read those charts, but it’s completely possible that python did not work well for the approach they were trying. I would be the first person to give up python for kotlin or some other more natively typed higher performing language. But the python bias is so far too big to ignore.
clickety_clack 2mo ago
Haha, queue a big argument over what “better” means for programming languages.
- flir 2mo ago
  They're the ones AI is more productive in.... (It's an interesting question though - my prediction would be that AI is best in the languages it consumed the most examples of, and I reckon the size of the StackOverflow site for each language would be a rough proxy for that).
  - clickety_clack 2mo ago
    If it was as simple as that we would have far fewer programming languages even for humans today.

tikhonj 2mo ago

I've already seen at least one promising experiment about how static checking helps LLMs: https://arxiv.org/abs/2606.01522 https://arxiv.org/abs/2606.01522 Key parts of the abstract: > This raises a question the programming-language community has not previously had reason to ask: should error-message detail be calibrated differently for AI agents than for humans? > We investigate this question through a controlled experiment using Shplait, an ML-style statically typed language. We construct a suite of programs containing a single deliberate type error each, and measure how often an AI agent repairs them under ablation: a detailed error context using the unification stack; a proximate error location; a minimal type error; and a dynamic (test suite) error only. An automated oracle uses a test suite to classify each repair attempt as a type error, semantically incorrect, or semantically correct. > We find concrete evidence that more detailed error messages generally improve an agent's ability to fix type errors. We also find that the presence of a type system appears to help more than only test suite failure reports.

kagevf 2mo ago

> Next up is demonstrating the AI is more productive with better programming languages. That actually would be pretty exciting! But, wouldn't AI be biased toward more popular languages, since those will by nature of their popularity provide more sources for training material? Even still, the AI could demonstrate which algorithms or maybe patterns and techniques are more productive, in the context of the popular languages.

bobthepanda 2mo ago
That’s not necessarily a bad thing. As an example, 110v American outlet plugs are almost certainly not the most efficient way to power devices and appliances; but, because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances in the US. Similar story with USB C.
- mkehrt 2mo ago
  Outlets benefit from standards, however. Every device (well kind of) uses the same few plug types, and a different plug would be a disadvantage. On the other hand, having a bunch of different languages is an advantage. People can pick the best tool for the job, and experiment with new language features.
- kagevf 2mo ago
  > That’s not necessarily a bad thing. True, and to my point, "popular" isn't necessarily "better". > because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances Yes, but LLMs can do the heavy lifting when analyzing the "better programming languages" for productivity. There will be a bias toward popularity in their training sources, though.
gadflyinyoureye 2mo ago
You might be surprised. I've had an AI start spitting out none functioning python code. I've rewritten it with Go. The requirements are that every change has to pass compilation, linting and tests. Same requirements as Python. Go keeps the AI on the rails better. The compilation failing provides quick and probably better feedback than a whole dictionary of tests.
- kagevf 2mo ago
  For sure. Its code output is definitely not to be fully trusted. Oftentimes, you can point at where it made a mistake and it usually fixes it too.

keeganpoppen 2mo ago

i can't wait for that one haha

seanmcdirmid 2mo ago

> Next up is demonstrating the AI is more productive with better programming languages. This is hard to do given the current reliance on model weights in its programming language knowledge. You’ll get your best results for python or some variation of python. But

CuriouslyC 2mo ago
Not true. Python performance is actually pretty low, and Elixir, Kotlin and C# trounce it handily.
- seanmcdirmid 2mo ago
  Do you have citations? Everything I’ve read have said they have an overwhelming bias to Python; eg https://arxiv.org/html/2503.17181v1 https://arxiv.org/html/2503.17181v1 and https://arxiv.org/html/2505.13004v1 https://arxiv.org/html/2505.13004v1
  - CuriouslyC 2mo ago
    Agents love writing Python, but it turns out they're not amazing at it. https://autocodebench.github.io/ https://autocodebench.github.io/
    - seanmcdirmid 2mo ago
      I’m not really sure how to read those charts, but it’s completely possible that python did not work well for the approach they were trying. I would be the first person to give up python for kotlin or some other more natively typed higher performing language. But the python bias is so far too big to ignore.

clickety_clack 2mo ago

Haha, queue a big argument over what “better” means for programming languages.

flir 2mo ago
They're the ones AI is more productive in.... (It's an interesting question though - my prediction would be that AI is best in the languages it consumed the most examples of, and I reckon the size of the StackOverflow site for each language would be a rough proxy for that).
- clickety_clack 2mo ago
  If it was as simple as that we would have far fewer programming languages even for humans today.