4 ms·

Many of the examples seem very easy -- I suspect that without LLMs, just simple Google searches lead you to a stackoverflow question that asks the same thing wh

by rs186 1y ago

Many of the examples seem very easy -- I suspect that without LLMs, just simple Google searches lead you to a stackoverflow question that asks the same thing which. I wonder how this performs in bigger, more complex codebase. Also, my personal experience with LLMs fixing compilation errors is: when it works, it works great. But when it doesn't, it's so clueless and lost that it's a complete waste of time to employ LLM in the first place -- you are much better off debugging the code yourself using old fashioned method.

lolinder 1y ago
Yep. This is true for all languages that I've tried, but it's particularly true in Rust. The model will get into a loop where it gets further and further away from the intended behavior while trying to fix borrow checker errors, then eventually (if you're lucky) gives up and hand the mess back over to you. Which at least with Cursor's implementation means that it by default gives you the last iteration of its attempt to fix the problem, which when this happens is almost always way worse than its first attempt.
- greenavocado 1y ago
  That's why you need to implement logical residual connections to keep the results focused over successive prompts (like ResNets do)
- _bin_ 1y ago
  Absolutely; I re-try using LLMs to debug this every so often and they just aren't capable of "fixing" anything borrow checker-related. They spit out some slop amalgamation of Rc/Arc/even UnsafeCell. They don't understand futures being send + sync. They don't understand lifetimes. The other path is it sometimes loops between two or three broken "fixes" that still don't compile. "Sure! Let me...." (writes the most ungodly garbage Rust known to man) Now, I certainly hope I'm wrong. It would be nice to enjoy similar benefits to guys doing more python/typescript work. I just doubt it's that good.
  - steveklabnik 1y ago
    This is pretty contrary to my experience, for whatever it's worth. I wonder what the differences are.
    - _bin_ 1y ago
      Well to start, you're really, really good at rust :) I'm not sure. I've begun advocating for people, when they write blogposts etc., to dump and attach their conversations (unless precluded by security/privacy) so those of us who seem to get worse results can see if there are "tricks" or styles to it we miss. If you're amenable to it, this would be doubly helpful from someone who knows the language so extremely well!
      - steveklabnik 1y ago
        I’ll think about it, and thanks for the kind words.
    - bsaul 1y ago
      senior dev, rust very recent onboarder : my personal experience is that LLM are super helpful at the very very beginning, because they show you the very common rust pattern, and help with the verbose syntax by reducing the amount of typing ( which isn't usually a problem, but in rust it can be). Then once you start reaching for a bit more complex problems (like ffi, or building your own lib), llms are actually making waste time spitting nonsense. However, once your patterns are back on track, and you start to know a bit better what you're doing, you can easily dismiss those bad advices, and immediately give the llm hints that will make them useful again. At least that's my experience.
  - OJFord 1y ago
    > It would be nice to enjoy similar benefits to guys doing more python/typescript work. No need to be envious: it doesn't give me compilation errors in python, but that ain't because it always gives correct code! (It can be helpful too, but I get a lot of hallucinated APIs/arguments/etc.)
- cmrdporcupine 1y ago
  Yeah borrow checker errors in Rust fed to an LLM inevitably lead to the thing just going in circles. You correct it, it offers something else that doesn't work, when notified of that it gives you some variant of the original problem. Usually when you come to an LLM with an error like this it's because you tried something that made sense to you and it didn't. Turns out those things usually "make sense" to an LLM, too, and they don't step through and reason through the "logic", they just vibe off of it, and the pattern they come back to you with is usually just some variant of the original pattern that led you to the problem in the first place.
nicce 1y ago
There have been cases when o1/o3 has helped me to solve some issues that I could not solve with stackoverflow or Rust forum. LLM was able to connect the dots of some more complex and rarer Rust features and my requirements. I did not know that they could be used like that. One case was, for example, about complex usage of generic associated types (GATs). What it comes to lifetime issues, usually it is about wasting time if trying to solve with LLMs.
- seanw444 1y ago
  I rarely touch LLMs (I think they're very overhyped and usually unnecessary), but I did find one use case that was handy recently. I was writing some Nim code that interfaced with Cairo, and one Cairo feature wasn't really documented at all, and the design pattern was one I was not familiar with. Asking it to write Nim code to solve my problem resulted in a mess of compilation errors, and it kept looping through multiple broken approaches when I pointed out what was flawed. Finally, I just decided to ask it to explain the C design pattern I was unfamiliar with, and it was capable of bridging the gap enough for me to be able to write the correct Nim code. That was pretty cool. There was no documentation anywhere for what I needed, and nobody else had ever encountered that problem before. That said, I could have just gone to the Nim forum with a snippet and asked for help, and I would have solved the problem with a fraction of the electricity usage.
rvz 1y ago
> Also, my personal experience with LLMs fixing compilation errors is: when it works, it works great. But when it doesn't, it's so clueless and lost that it's a complete waste of time to employ LLM in the first place -- you are much better off debugging the code yourself using old fashioned method. Or just 'learning the Rust syntax' and standard library? As you said, LLMs are unpredictable in their output and will can generate functions that don't exist and incorrect code as you use more advanced features, wasting more time than it saves if you don't know the language well enough. I guess those coming from dynamically typed languages are having a very hard time in getting used to strongly typed languages and then struggle with the basic syntax of say, Rust or C++. Looking at this AI hype with vibe-coding/debugging and LLMs, it just favours throwing code on the wall with a lack of understanding as to what it does after it compiles. This is why many candidates won't ever do Leetcode with Rust in a real interview.
mountainriver 1y ago
LLMs have made me at least twice as fast at writing rust code. I now think that more people should be writing rust as it’s been made fairly simple to do. And yes there are some errors it gets stuck in a loop on. It’s not often and generally just switching to another LLM in cursor will fix it.
onlyrealcuzzo 1y ago
> But when it doesn't, it's so clueless and lost that it's a complete waste of time to employ LLM in the first place -- you are much better off debugging the code yourself using old fashioned method. So why not automatically try it, see if it fixes automatically, and if not then actually debug it yourself?
- colonial 1y ago
  Tokens cost money, either directly (APIs) or indirectly (wall power for your GPU). I would prefer not to spend my hard-earned dollars on slop advice.
  - onlyrealcuzzo 1y ago
    It would cost less than $0.05, and it's right way more times than 1 in 1000, which would pay for itself. But everyone's entitled to their own choices.
    - colonial 1y ago
      > RustAssistant is able to achieve an impressive peak accuracy of roughly 74% on real-world compilation errors in popular open-source Rust repositories. 5 cents, for every compilation error, for a 3/4 chance of success (which is likely disproportionately weighted towards errors I could trivially fix myself.) That easily works out to multiple wasted dollars an hour even if it only applies to manual compiler invocations, not accounting for any time wasted evaluating bogus solutions.
themusicgod1 1y ago
"very easy" if you have access to the correct dependencies which outside of microsoft's walled garden, and access to a free LLM (https://elevenfreedoms.org/ https://elevenfreedoms.org/) which is not guaranteed at all all of this looks very different when you have to patch in rust dependencies by hand outside of github.

by rs186 1y ago

lolinder 1y ago

Yep. This is true for all languages that I've tried, but it's particularly true in Rust. The model will get into a loop where it gets further and further away from the intended behavior while trying to fix borrow checker errors, then eventually (if you're lucky) gives up and hand the mess back over to you. Which at least with Cursor's implementation means that it by default gives you the last iteration of its attempt to fix the problem, which when this happens is almost always way worse than its first attempt.

greenavocado 1y ago
That's why you need to implement logical residual connections to keep the results focused over successive prompts (like ResNets do)
_bin_ 1y ago
Absolutely; I re-try using LLMs to debug this every so often and they just aren't capable of "fixing" anything borrow checker-related. They spit out some slop amalgamation of Rc/Arc/even UnsafeCell. They don't understand futures being send + sync. They don't understand lifetimes. The other path is it sometimes loops between two or three broken "fixes" that still don't compile. "Sure! Let me...." (writes the most ungodly garbage Rust known to man) Now, I certainly hope I'm wrong. It would be nice to enjoy similar benefits to guys doing more python/typescript work. I just doubt it's that good.
- steveklabnik 1y ago
  This is pretty contrary to my experience, for whatever it's worth. I wonder what the differences are.
  - _bin_ 1y ago
    Well to start, you're really, really good at rust :) I'm not sure. I've begun advocating for people, when they write blogposts etc., to dump and attach their conversations (unless precluded by security/privacy) so those of us who seem to get worse results can see if there are "tricks" or styles to it we miss. If you're amenable to it, this would be doubly helpful from someone who knows the language so extremely well!
    - steveklabnik 1y ago
      I’ll think about it, and thanks for the kind words.
  - bsaul 1y ago
    senior dev, rust very recent onboarder : my personal experience is that LLM are super helpful at the very very beginning, because they show you the very common rust pattern, and help with the verbose syntax by reducing the amount of typing ( which isn't usually a problem, but in rust it can be). Then once you start reaching for a bit more complex problems (like ffi, or building your own lib), llms are actually making waste time spitting nonsense. However, once your patterns are back on track, and you start to know a bit better what you're doing, you can easily dismiss those bad advices, and immediately give the llm hints that will make them useful again. At least that's my experience.
- OJFord 1y ago
  > It would be nice to enjoy similar benefits to guys doing more python/typescript work. No need to be envious: it doesn't give me compilation errors in python, but that ain't because it always gives correct code! (It can be helpful too, but I get a lot of hallucinated APIs/arguments/etc.)
cmrdporcupine 1y ago
Yeah borrow checker errors in Rust fed to an LLM inevitably lead to the thing just going in circles. You correct it, it offers something else that doesn't work, when notified of that it gives you some variant of the original problem. Usually when you come to an LLM with an error like this it's because you tried something that made sense to you and it didn't. Turns out those things usually "make sense" to an LLM, too, and they don't step through and reason through the "logic", they just vibe off of it, and the pattern they come back to you with is usually just some variant of the original pattern that led you to the problem in the first place.

nicce 1y ago

There have been cases when o1/o3 has helped me to solve some issues that I could not solve with stackoverflow or Rust forum. LLM was able to connect the dots of some more complex and rarer Rust features and my requirements. I did not know that they could be used like that. One case was, for example, about complex usage of generic associated types (GATs). What it comes to lifetime issues, usually it is about wasting time if trying to solve with LLMs.

seanw444 1y ago
I rarely touch LLMs (I think they're very overhyped and usually unnecessary), but I did find one use case that was handy recently. I was writing some Nim code that interfaced with Cairo, and one Cairo feature wasn't really documented at all, and the design pattern was one I was not familiar with. Asking it to write Nim code to solve my problem resulted in a mess of compilation errors, and it kept looping through multiple broken approaches when I pointed out what was flawed. Finally, I just decided to ask it to explain the C design pattern I was unfamiliar with, and it was capable of bridging the gap enough for me to be able to write the correct Nim code. That was pretty cool. There was no documentation anywhere for what I needed, and nobody else had ever encountered that problem before. That said, I could have just gone to the Nim forum with a snippet and asked for help, and I would have solved the problem with a fraction of the electricity usage.

rvz 1y ago

> Also, my personal experience with LLMs fixing compilation errors is: when it works, it works great. But when it doesn't, it's so clueless and lost that it's a complete waste of time to employ LLM in the first place -- you are much better off debugging the code yourself using old fashioned method. Or just 'learning the Rust syntax' and standard library? As you said, LLMs are unpredictable in their output and will can generate functions that don't exist and incorrect code as you use more advanced features, wasting more time than it saves if you don't know the language well enough. I guess those coming from dynamically typed languages are having a very hard time in getting used to strongly typed languages and then struggle with the basic syntax of say, Rust or C++. Looking at this AI hype with vibe-coding/debugging and LLMs, it just favours throwing code on the wall with a lack of understanding as to what it does after it compiles. This is why many candidates won't ever do Leetcode with Rust in a real interview.

mountainriver 1y ago

LLMs have made me at least twice as fast at writing rust code. I now think that more people should be writing rust as it’s been made fairly simple to do. And yes there are some errors it gets stuck in a loop on. It’s not often and generally just switching to another LLM in cursor will fix it.

onlyrealcuzzo 1y ago

> But when it doesn't, it's so clueless and lost that it's a complete waste of time to employ LLM in the first place -- you are much better off debugging the code yourself using old fashioned method. So why not automatically try it, see if it fixes automatically, and if not then actually debug it yourself?

colonial 1y ago
Tokens cost money, either directly (APIs) or indirectly (wall power for your GPU). I would prefer not to spend my hard-earned dollars on slop advice.
- onlyrealcuzzo 1y ago
  It would cost less than $0.05, and it's right way more times than 1 in 1000, which would pay for itself. But everyone's entitled to their own choices.
  - colonial 1y ago
    > RustAssistant is able to achieve an impressive peak accuracy of roughly 74% on real-world compilation errors in popular open-source Rust repositories. 5 cents, for every compilation error, for a 3/4 chance of success (which is likely disproportionately weighted towards errors I could trivially fix myself.) That easily works out to multiple wasted dollars an hour even if it only applies to manual compiler invocations, not accounting for any time wasted evaluating bogus solutions.

themusicgod1 1y ago

"very easy" if you have access to the correct dependencies which outside of microsoft's walled garden, and access to a free LLM (https://elevenfreedoms.org/ https://elevenfreedoms.org/) which is not guaranteed at all all of this looks very different when you have to patch in rust dependencies by hand outside of github.