3 ms·

I hate to admit it, because I don't particularly like Rust, but I'm slowly coming around to the idea it should replace things. The major change is a recent com

I hate to admit it, because I don't particularly like Rust, but I'm slowly coming around to the idea it should replace things. The major change is a recent comment I made where I was musing about a future where LLMs actually write most of the code. That isn't a foregone conclusion but the recent improvements in coding LLMs suggests it isn't as far off a future as I once considered. My thought was simple: if I am using a LLM to do the significant portion of generating code, how does that change what I think about the programming language I am using? Does the criteria I use to select the language change? Upon reflection, Rust is probably the best candidate. It has strictness in ways that other languages do not have. And if the LLM is paying most of the cost of keeping the types and lifetimes in order, what do I care if the syntax is ugly? As long as I can read the output of the LLM and verify it (code review it) then I actually want the strictness. I want the most statically analyzable code possible since I have low trust in the LLM. The fact that Rust is also, ahem, blazingly fast, is icing on the cake. As an aside to this aside, I was also thinking about modular kernels, like Minix. I wonder if there is a world where we take a userland like the one Ubuntu is trying and apply it to Minix. And then slowly replace the os modules with ones written in Rust. I think the modularity of something like Minix might be an advantage, but that might just be because I am completely naïve.

Covenant0028 2y ago
I've had similar thoughts as well. Rust or C would also be the ideal candidate from an energy consumption point of view, as they consume way less energy than Python, which is the language that LLMs most often default to in my experience. However it's unlikely LLMs will generate a lot of Rust code. All these LLMs are generating is the most likely next token, based on what they've been trained on. And in their training set, it's likely there is massively more Python and JS code out there than Rust, simply because those are way more popular languages. So with Rust, it's more likely to hallucinate and make mistakes than with Python where the path is much better trodden
- jdright 2y ago
  In practice, it is not what happens. I've been doing AI assisted Rust for some time, and it is very convincing that this is the way. I expect 6mo to 1y to be basically fully automated. Rust has tons of code out there, and quality code. Different from js or Python that has an abundance of low quality to pure garbage code.
  - m00dy 2y ago
    same here. I think rust + llm combo is unbeatable.
- phanimahesh 2y ago
  However it is much easier to statically analyse rust and rust has compile time validations compared to an interpreted language like python. This makes it easier to produce code with write compile fix loop from an agentic llm in rust than python. In my experience llms are not particularly bad with rust compared to python, though I've only toyed with them minimally.
  - saghm 2y ago
    While I'm lucky enough not to ever have had to spend a while trying to refactor/fix LLM-produced code, I've definitely had a much easier time trying to do major refactors to fix larger issues with human-written Rust code than any other language I've tried it with. The compiler obviously doesn't catch everything, but the types of things it _can_ catch seem to be very common to encounter when trying to make cascading changes to a codebase to fix the types of issues I've seen refactors attempt to address, and my instinct is that these might also be the types of things that end up needing to be fixed in LLM-produced code. Elsewhere in this thread someone mentioned LLMs producing poor performing code that does stuff like collect iterators far too often and pointed out that fixing this sort of thing often requires significant Rust expertise, and I don't disagree with that. However, my impression is that it still ends up being less work for an experienced Rust programmer to claw back some memory from poor code than for someone similarly experienced in something other than Rust working in a code base in their own language. I've seen issues when three or four Go engineers spend weeks trying to track down and reduce memory overhead of slices due to the fact that it's not always easy to tell whether a given slice owns its own memory on the heap or is referencing a different one; looking for everywhere that `collect` is called (or another type known to heap allocate) is comparatively a lot simpler. Maybe Go being a lot simpler than Rust could make LLMs produce code that's so much better that it makes it easier for humans to debug and fix, but at least for some types of issues, my experience has been the opposite with Go code written by humans, so I wouldn't feel particularly confident about making that prediction at this point in time.
_factor 2y ago
It’s interesting it took you this long to substitute “junior coder” with LLM. The implicit safety is just as applicable to teams of human error prone devs.
kortilla 2y ago
The rust produced by LLMs is quite bad. It’s overly verbose (misses combinators) and often subtly wrong (swallows errors on result types when it shouldn’t). A single errant collect or clone call can destroy your performance and LLMs sprinkle them for no reason. Unless you are experienced in rust, you have zero ability to catch the kind of mistakes LLMs make producing rust code.
- Diederich 2y ago
  Is this deficiency likely to persist long-term as LLMs grow more powerful?
  - queuebert 2y ago
    I think one issue is less extant Rust code to train on. As more Rust is written and published, LLMs should in theory get better.
    - andai 2y ago
      I heard most code today is written by LLMs. I hope that's not true, but if it is, the future improvements will be slow (barring something like, actually teaching them to read the docs).
- sroussey 2y ago
  I just had an LLM (something 3.7) propose a 100 line implementation and after I stared at things for a while, I reduced it to one. I’m sure I’m in the minority of not just accepting the 100 line addition.
  - andai 2y ago
    Could you elaborate? What were those lines doing?
    - sroussey 2y ago
      Lots of the lines were safely dereferencing things in JS where you could just do ?., putting try/catch blocks around that (each time!) and calling into things in a way that a single line would have sufficed. I almost took a screenshot for the lols.
Cloudef 2y ago
LLM truly provides the garbage in, garbage out experience
- deleted 2y ago
  [deleted]

Covenant0028 2y ago

I've had similar thoughts as well. Rust or C would also be the ideal candidate from an energy consumption point of view, as they consume way less energy than Python, which is the language that LLMs most often default to in my experience. However it's unlikely LLMs will generate a lot of Rust code. All these LLMs are generating is the most likely next token, based on what they've been trained on. And in their training set, it's likely there is massively more Python and JS code out there than Rust, simply because those are way more popular languages. So with Rust, it's more likely to hallucinate and make mistakes than with Python where the path is much better trodden

jdright 2y ago
In practice, it is not what happens. I've been doing AI assisted Rust for some time, and it is very convincing that this is the way. I expect 6mo to 1y to be basically fully automated. Rust has tons of code out there, and quality code. Different from js or Python that has an abundance of low quality to pure garbage code.
- m00dy 2y ago
  same here. I think rust + llm combo is unbeatable.
phanimahesh 2y ago
However it is much easier to statically analyse rust and rust has compile time validations compared to an interpreted language like python. This makes it easier to produce code with write compile fix loop from an agentic llm in rust than python. In my experience llms are not particularly bad with rust compared to python, though I've only toyed with them minimally.
- saghm 2y ago
  While I'm lucky enough not to ever have had to spend a while trying to refactor/fix LLM-produced code, I've definitely had a much easier time trying to do major refactors to fix larger issues with human-written Rust code than any other language I've tried it with. The compiler obviously doesn't catch everything, but the types of things it _can_ catch seem to be very common to encounter when trying to make cascading changes to a codebase to fix the types of issues I've seen refactors attempt to address, and my instinct is that these might also be the types of things that end up needing to be fixed in LLM-produced code. Elsewhere in this thread someone mentioned LLMs producing poor performing code that does stuff like collect iterators far too often and pointed out that fixing this sort of thing often requires significant Rust expertise, and I don't disagree with that. However, my impression is that it still ends up being less work for an experienced Rust programmer to claw back some memory from poor code than for someone similarly experienced in something other than Rust working in a code base in their own language. I've seen issues when three or four Go engineers spend weeks trying to track down and reduce memory overhead of slices due to the fact that it's not always easy to tell whether a given slice owns its own memory on the heap or is referencing a different one; looking for everywhere that `collect` is called (or another type known to heap allocate) is comparatively a lot simpler. Maybe Go being a lot simpler than Rust could make LLMs produce code that's so much better that it makes it easier for humans to debug and fix, but at least for some types of issues, my experience has been the opposite with Go code written by humans, so I wouldn't feel particularly confident about making that prediction at this point in time.

_factor 2y ago

It’s interesting it took you this long to substitute “junior coder” with LLM. The implicit safety is just as applicable to teams of human error prone devs.

kortilla 2y ago

The rust produced by LLMs is quite bad. It’s overly verbose (misses combinators) and often subtly wrong (swallows errors on result types when it shouldn’t). A single errant collect or clone call can destroy your performance and LLMs sprinkle them for no reason. Unless you are experienced in rust, you have zero ability to catch the kind of mistakes LLMs make producing rust code.

Diederich 2y ago
Is this deficiency likely to persist long-term as LLMs grow more powerful?
- queuebert 2y ago
  I think one issue is less extant Rust code to train on. As more Rust is written and published, LLMs should in theory get better.
  - andai 2y ago
    I heard most code today is written by LLMs. I hope that's not true, but if it is, the future improvements will be slow (barring something like, actually teaching them to read the docs).
sroussey 2y ago
I just had an LLM (something 3.7) propose a 100 line implementation and after I stared at things for a while, I reduced it to one. I’m sure I’m in the minority of not just accepting the 100 line addition.
- andai 2y ago
  Could you elaborate? What were those lines doing?
  - sroussey 2y ago
    Lots of the lines were safely dereferencing things in JS where you could just do ?., putting try/catch blocks around that (each time!) and calling into things in a way that a single line would have sufficed. I almost took a screenshot for the lols.

Cloudef 2y ago

LLM truly provides the garbage in, garbage out experience

deleted 2y ago
[deleted]