4 ms·

I’m curious how this performs against Claude Code/Codex. The “RustAssistant Algorithm” looks to be a simple LLM workflow[0], and their testing was limited to G

I’m curious how this performs against Claude Code/Codex. The “RustAssistant Algorithm” looks to be a simple LLM workflow[0], and their testing was limited to GPT-4 and GPT-3.5. In my experience (building a simple Rust service using OpenAI’s o1), the LLM will happily fix compilation issues but will also inadvertently change some out-of-context functionality to make everything “just work.” The most common issues I experienced were subtle changes to ownership, especially when using non-standard or frequently updated crates, which caused performance degradations in the test cases. Therefore I wouldn’t really trust GPT-4 (and certainly not 3.5) to modify my code, even if just to fix compilation errors, without some additional reasoning steps or oversight. [0] https://www.anthropic.com/engineering/building-effective-agents https://www.anthropic.com/engineering/building-effective-age...

woadwarrior01 1y ago
I tried Claude Code with a small-ish C++ codebase recently and found it to be quite lacking. It kept making a lot of silly syntax errors and going around in circles. Spent about $20 in credits without it getting anywhere close to being able to solve the task I was trying to guide it through. OTOH, I know a lot of people who swear by it. But they all seem to be Python or Front-end developers.
- Wheaties466 1y ago
  Do we really know why LLMs seem to score the highest with python related coding tasks? I would think there are equally good examples of javascript/c++/java code to train from but I always see python with the highest scores.
  - triyambakam 1y ago
    Could be related to how flexible Python is. Pretty easy to write bad and "working" Python code
- greenavocado 1y ago
  May I ask what you tried? I have had strong successes with C++ generation
  - woadwarrior01 1y ago
    It was a bit esoteric, but not terribly so, some metal-cpp based code for a macOS app.
    - pjmlp 1y ago
      Probably due to lack of data then, it isn't as if there are many developers rushing to use metal-cpp instead of Objective-C or Swift with Metal. Even the documentation is basically having to go through the source code.
mfld 1y ago
I find that Claude code works well to fix rust compile errors in most cases. Interestingly, the paper didn't compare against agentic coding tools at all, which of course will be more easy to use and more generally applicable.

woadwarrior01 1y ago

I tried Claude Code with a small-ish C++ codebase recently and found it to be quite lacking. It kept making a lot of silly syntax errors and going around in circles. Spent about $20 in credits without it getting anywhere close to being able to solve the task I was trying to guide it through. OTOH, I know a lot of people who swear by it. But they all seem to be Python or Front-end developers.

Wheaties466 1y ago
Do we really know why LLMs seem to score the highest with python related coding tasks? I would think there are equally good examples of javascript/c++/java code to train from but I always see python with the highest scores.
- triyambakam 1y ago
  Could be related to how flexible Python is. Pretty easy to write bad and "working" Python code
greenavocado 1y ago
May I ask what you tried? I have had strong successes with C++ generation
- woadwarrior01 1y ago
  It was a bit esoteric, but not terribly so, some metal-cpp based code for a macOS app.
  - pjmlp 1y ago
    Probably due to lack of data then, it isn't as if there are many developers rushing to use metal-cpp instead of Objective-C or Swift with Metal. Even the documentation is basically having to go through the source code.

mfld 1y ago

I find that Claude code works well to fix rust compile errors in most cases. Interestingly, the paper didn't compare against agentic coding tools at all, which of course will be more easy to use and more generally applicable.