4 ms·

Training data can't be the whole answer. LLMs are really good at translating to different programming languages. This makes sense, given that they are derived f

by gmueckl 5mo ago

Training data can't be the whole answer. LLMs are really good at translating to different programming languages. This makes sense, given that they are derived from text translation systems. I'm getting great results in languages with comparatively small bodies of freely available code. The bigger hurdle is usually that LLMs tend to copy common idioms in the target language and if it is an "enterprise-y" language like Java or C#, the amount of useless boilerplate can skyrocket immediately, which creates a real danger that the result grows beyond the usable context window size and the quality suffers.

lanyard-textile 5mo ago
Very true. I have to steer models hard for C++. They constantly suggest std::variant :P
- socalgal2 5mo ago
  is that bad? Godbolt got a 2x speed improvement switching from what he thought was a good fast impl to std:variant https://www.youtube.com/watch?v=gg4pLJNCV9I https://www.youtube.com/watch?v=gg4pLJNCV9I
  - lanyard-textile 5mo ago
    More fundamentally -- it creatively finds ways to use std::variant that I don't think any person would ever naturally conceive :) And I think it's extremely fascinating, because I have absolutely no idea why. Claude can plow through my TypeScript and read my mind easy peasy, but C++? It's just a constant battle to convince it not to use std::variant. Basically if there's any two branches of possibilities, I look forward to seeing a complicated proposal for std::variant. And then I inevitably push back, and Claude gives the "You're right, I was overcomplicating it" and it does an if/else like you'd expect. Maybe something about my particular repos cause that, but none of them use std::variant a single time. I also wonder if it's far more commonly used than I realize, that also occurs to me.
chromacity 5mo ago
> LLMs are really good at translating to different programming languages. ...for which ample training data is available. > This makes sense, given that they are derived from text translation systems. ...for languages with ample training data available. Yes, LLMs can combine information in novel ways. They are wonderful in many respects. But they make far more mistakes if they can't lean on copious amounts of training data. Invent a toy language, write a spec, and ask them to use it. They will, but they will have a hard time.
- lmm 5mo ago
  That might be an argument for not using a novel homebrew programming language. But it's not an argument against, like, any top-100 or even top-1000 programming language, which will be adequately represented in the training data.
  - ambicapter 5mo ago
    It is if more training data results in better performance. In which case, GP will continue to use the language that is likely to have the most training data available.
    - lmm 5mo ago
      > It is if more training data results in better performance. Sure. But given the relation with translation systems, it seems far more likely that there are diminishing returns to larger volumes of training data.
- agentultra 5mo ago
  They are also good at generating plausible code. The kind that has no obvious bugs in it. I wouldn’t be surprised if humans in the loop over report success with these tools. Combined with decision fatigue… it’s not a good recipe for humans making good decisions. An experienced Rust developer is going to be in a better position to drive an agent to generate useful Rust code than a Python programmer with little or no Rust experience. Not sure I agree with the author that everyone should just generate reams of Rust now. At least if your get paged at 3am to fix the 300k AI-generated Django blog you’ll have a chance at figuring things out. Good luck to you if Claude is down at the same time. But still better than if it was in Rust if you have no experience with that language.
jryio 5mo ago
In higher dimensional vector space, yes it can. Dimensionality gets bizarre in 1000-D space. Similarity and orthogonality express themselves in strange ways and each dimension codes different semantic meaning. Therefore, if the training data is highly consistent you are by definition reducing some complexity and/or encoding better similarity. In Go the statement result, err := Storage.write(...) Is almost always going to be followed by if err != nil { ... } In a highly dynamic language you may not get try { Storage.write() } catch (error) { ... } Unless explicitly asked for.
- za3faran 5mo ago
  > In a highly dynamic language you may not get Being dynamic is secondary. A language that uses exceptions for errors does not always need to surround every try with a catch if the code doesn't need to. You have a top level handler that would catch everything.
- dnautics 5mo ago
  It's a little bit old, but challenge you opinions about what matters for LLM agentic coding: https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/blob/main/figures/exp_acb.png https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/blob/ma...
dnautics 5mo ago
> Training data can't be the whole answer. Absolutely correct. Anthropic showed that 250 examples can "poison" an LLM -- independent of LLM activation count.

by gmueckl 5mo ago

lanyard-textile 5mo ago

Very true. I have to steer models hard for C++. They constantly suggest std::variant :P

socalgal2 5mo ago
is that bad? Godbolt got a 2x speed improvement switching from what he thought was a good fast impl to std:variant https://www.youtube.com/watch?v=gg4pLJNCV9I https://www.youtube.com/watch?v=gg4pLJNCV9I
- lanyard-textile 5mo ago
  More fundamentally -- it creatively finds ways to use std::variant that I don't think any person would ever naturally conceive :) And I think it's extremely fascinating, because I have absolutely no idea why. Claude can plow through my TypeScript and read my mind easy peasy, but C++? It's just a constant battle to convince it not to use std::variant. Basically if there's any two branches of possibilities, I look forward to seeing a complicated proposal for std::variant. And then I inevitably push back, and Claude gives the "You're right, I was overcomplicating it" and it does an if/else like you'd expect. Maybe something about my particular repos cause that, but none of them use std::variant a single time. I also wonder if it's far more commonly used than I realize, that also occurs to me.

chromacity 5mo ago

> LLMs are really good at translating to different programming languages. ...for which ample training data is available. > This makes sense, given that they are derived from text translation systems. ...for languages with ample training data available. Yes, LLMs can combine information in novel ways. They are wonderful in many respects. But they make far more mistakes if they can't lean on copious amounts of training data. Invent a toy language, write a spec, and ask them to use it. They will, but they will have a hard time.

lmm 5mo ago
That might be an argument for not using a novel homebrew programming language. But it's not an argument against, like, any top-100 or even top-1000 programming language, which will be adequately represented in the training data.
- ambicapter 5mo ago
  It is if more training data results in better performance. In which case, GP will continue to use the language that is likely to have the most training data available.
  - lmm 5mo ago
    > It is if more training data results in better performance. Sure. But given the relation with translation systems, it seems far more likely that there are diminishing returns to larger volumes of training data.
agentultra 5mo ago
They are also good at generating plausible code. The kind that has no obvious bugs in it. I wouldn’t be surprised if humans in the loop over report success with these tools. Combined with decision fatigue… it’s not a good recipe for humans making good decisions. An experienced Rust developer is going to be in a better position to drive an agent to generate useful Rust code than a Python programmer with little or no Rust experience. Not sure I agree with the author that everyone should just generate reams of Rust now. At least if your get paged at 3am to fix the 300k AI-generated Django blog you’ll have a chance at figuring things out. Good luck to you if Claude is down at the same time. But still better than if it was in Rust if you have no experience with that language.

jryio 5mo ago

In higher dimensional vector space, yes it can. Dimensionality gets bizarre in 1000-D space. Similarity and orthogonality express themselves in strange ways and each dimension codes different semantic meaning. Therefore, if the training data is highly consistent you are by definition reducing some complexity and/or encoding better similarity. In Go the statement result, err := Storage.write(...) Is almost always going to be followed by if err != nil { ... } In a highly dynamic language you may not get try { Storage.write() } catch (error) { ... } Unless explicitly asked for.

za3faran 5mo ago
> In a highly dynamic language you may not get Being dynamic is secondary. A language that uses exceptions for errors does not always need to surround every try with a catch if the code doesn't need to. You have a top level handler that would catch everything.
dnautics 5mo ago
It's a little bit old, but challenge you opinions about what matters for LLM agentic coding: https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/blob/main/figures/exp_acb.png https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/blob/ma...

dnautics 5mo ago

> Training data can't be the whole answer. Absolutely correct. Anthropic showed that 250 examples can "poison" an LLM -- independent of LLM activation count.