3 ms·

I actually don't think it's that cut and dry. I expect especially that rust (due to lifetimes) will stump LLMs - fixing locally triggers a need for refactor els

I actually don't think it's that cut and dry. I expect especially that rust (due to lifetimes) will stump LLMs - fixing locally triggers a need for refactor elsewhere. I actually think a language like Clojure (very functional, very compositional, focus on local, stand-alone functions, manipulate base data-structures (list, set, map), not specialist types (~classes) would do well. That said, atm. I get WAY more issues in ocaml suggestions from claude than for Python. Training is king - the LLM cannot reason so types are not as big a help as one might think.

littlestymaar 1y ago
> fixing locally triggers a need for refactor elsewhere. Yes, but such refactors are most of the time very mechanical, and there's no reason to believe the agent won't be able to do it. > the LLM cannot reason so types are not as big a help as one might think. You are missing the point: the person you are responding expects it to be superior in an agentic scenario, where the LLM can try its code and see the compiler output, rather than in a pure text-generation situation where the LLM can only assess the code from bird eye view.
- inciampati 1y ago
  Mechanical repairs, and often indicative of mistakes about lifetimes. So it's just part of the game.
- Capricorn2481 1y ago
  No, I think others are missing the point. An "Agentic scenario" is not dissimilar from passing code manually to an AI, it just does it by itself. And if you've tried to use AI for Rust, you would understand why this is not reliable. An LLM can read compiler output, but how it corrects the code is, ultimately, a semantic guess. It can look at the names of types, it can use its training to guess where new code should go based on types, but it's not able to actually use those types while making changes. It would use them in the same way it would use comments, to inform what code it should output. It makes a guess, checks the compiler output, makes another guess, etc. This may lead to code that compiles, but not code that should be committed by any means. And Rust is not what I'd call a "flexible language," where lots of different coding styles are acceptable in a codebase. You can easily end up with brittle code. So you don't get much benefits from types, but you do have the overhead of semantic complexity. This is a huge problem for a language like Rust, which is one of the most semantically complicated languages. The best languages are going to be ones that are semantically simple but popular, like Golang. Although I do think Clojure's support is impressive given how little code there is compared to other languages.
noodletheworld 1y ago
> so types are not as big a help as one might think. Yes, they are. An agent can combine the compiler type system and iterate. That is impossible using clojure. The reason you have problems woth ocaml is that the tooling youre using is too shit to support iterating until the compiler passes before returning the results to you. …not because tooling doesnt exist. Not because the tooling doesn't work. —> because you are not using it. Sure, rust ownership makes it hard for LLMs. Faaair point; but ultimately, why would a coding agent ever suggest code to you that doesnt compile? Either: a) the agent tooling is poor or b) it is impossible to verify if the code compiles. One of those is a solvable problem. One is not. (Yes, what many current agents do is run test suites; but dynamically generating valid tests is tricky; checking if code compiles is not tricky.)
- diggan 1y ago
  > An agent can combine the compiler type system and iterate. > That is impossible using clojure. It might be impossible to use the compiler type system, but in Clojure you have much more powerful tools for actually working with your program as it runs, one would think this would be a much better setup for an LLM that aims to implement something. Instead of just relying on the static types based on text, the LLM could actually inspect the live data as the program runs. Besides, the LLM could also replace individual functions/variables in a running program, without having to restart. The more I think about it, the more obvious it becomes how well fitted Clojure would be for an LLM to iteratively build an actual working program, compared to other static approaches like using Rust.
  - michalsustr 1y ago
    I understand the point , however I think explicit types are still superior, due to abundance of data in the training phase. It seems to me to be too computationally hard to incorporate a REPL-like interactive interface in the gpu training loop. Since it’s processing large amounts of data you want to keep it simple, without back-and-forth with CPUs that would kill performance. And if you can’t do it at training time, it’s hard to expect for the LLM to do well at inference time. Well, if you could run clojure purely on gpu/inside the neural net, that might be interesting!

littlestymaar 1y ago

> fixing locally triggers a need for refactor elsewhere. Yes, but such refactors are most of the time very mechanical, and there's no reason to believe the agent won't be able to do it. > the LLM cannot reason so types are not as big a help as one might think. You are missing the point: the person you are responding expects it to be superior in an agentic scenario, where the LLM can try its code and see the compiler output, rather than in a pure text-generation situation where the LLM can only assess the code from bird eye view.

inciampati 1y ago
Mechanical repairs, and often indicative of mistakes about lifetimes. So it's just part of the game.
Capricorn2481 1y ago
No, I think others are missing the point. An "Agentic scenario" is not dissimilar from passing code manually to an AI, it just does it by itself. And if you've tried to use AI for Rust, you would understand why this is not reliable. An LLM can read compiler output, but how it corrects the code is, ultimately, a semantic guess. It can look at the names of types, it can use its training to guess where new code should go based on types, but it's not able to actually use those types while making changes. It would use them in the same way it would use comments, to inform what code it should output. It makes a guess, checks the compiler output, makes another guess, etc. This may lead to code that compiles, but not code that should be committed by any means. And Rust is not what I'd call a "flexible language," where lots of different coding styles are acceptable in a codebase. You can easily end up with brittle code. So you don't get much benefits from types, but you do have the overhead of semantic complexity. This is a huge problem for a language like Rust, which is one of the most semantically complicated languages. The best languages are going to be ones that are semantically simple but popular, like Golang. Although I do think Clojure's support is impressive given how little code there is compared to other languages.

noodletheworld 1y ago

> so types are not as big a help as one might think. Yes, they are. An agent can combine the compiler type system and iterate. That is impossible using clojure. The reason you have problems woth ocaml is that the tooling youre using is too shit to support iterating until the compiler passes before returning the results to you. …not because tooling doesnt exist. Not because the tooling doesn't work. —> because you are not using it. Sure, rust ownership makes it hard for LLMs. Faaair point; but ultimately, why would a coding agent ever suggest code to you that doesnt compile? Either: a) the agent tooling is poor or b) it is impossible to verify if the code compiles. One of those is a solvable problem. One is not. (Yes, what many current agents do is run test suites; but dynamically generating valid tests is tricky; checking if code compiles is not tricky.)

diggan 1y ago
> An agent can combine the compiler type system and iterate. > That is impossible using clojure. It might be impossible to use the compiler type system, but in Clojure you have much more powerful tools for actually working with your program as it runs, one would think this would be a much better setup for an LLM that aims to implement something. Instead of just relying on the static types based on text, the LLM could actually inspect the live data as the program runs. Besides, the LLM could also replace individual functions/variables in a running program, without having to restart. The more I think about it, the more obvious it becomes how well fitted Clojure would be for an LLM to iteratively build an actual working program, compared to other static approaches like using Rust.
- michalsustr 1y ago
  I understand the point , however I think explicit types are still superior, due to abundance of data in the training phase. It seems to me to be too computationally hard to incorporate a REPL-like interactive interface in the gpu training loop. Since it’s processing large amounts of data you want to keep it simple, without back-and-forth with CPUs that would kill performance. And if you can’t do it at training time, it’s hard to expect for the LLM to do well at inference time. Well, if you could run clojure purely on gpu/inside the neural net, that might be interesting!