5 ms·

Go is absolutely one of the best programming languages for LLMs for the reason you say, and Python is just what LLMs like to use to write short throwaway script

Go is absolutely one of the best programming languages for LLMs for the reason you say, and Python is just what LLMs like to use to write short throwaway scripts. Frontier LLMs are generally pretty good at most programming languages and can pick up new ones pretty quickly. Training data seems to mostly just increase the speed which they write code, for example, GPTs tend to write Rust and Python faster than other programming languages. For actual output quality, the main deciding factor is simply how much tooling it is there for the LLMs to check their own work, as LLMs seemed to avoid using a lot of libraries in general. That's why C# is underrated due to the tooling strength of the .NET ecosystem, as long as you tell LLMs to avoid using reflections unless absolutely necessary. C++ is also surprisingly good, but you pretty much have to tell the LLMs to treat it like Go and don't use any of the dangerous features for normal code.

bob1029 2mo ago
I think a lot of people are sleeping on the advantages of "batteries included" ecosystems. The need to select an appropriate 3rd party library represents an entire dimension of the search space that can be eliminated. Imagine having to make this choice multiple times per day when your competition is just mindlessly using System.* types. The fact that the .NET ecosystem is curated by one entity should not be underestimated. Even when we do need to import 3rd party nugets, the models seem to follow this highly structured pattern. They scan the xml docs, and failing that they will build a throwaway console app to reflect over all the unique types and build a report. The fact that we can easily do this with a simple powershell command makes a big difference. How many other ecosystems can even consider doing this? Reflection is a superpower, not something to be avoided.
- YuechenLi 2mo ago
  Reflection is good for prototyping and get something setup quickly, but if you build your architecture around it, not only do you lose access to NativeAOT, the code becomes very hard to debug, and if you code with LLM a lot, you either have to spend time trying to debug reflections or just rewrite it with source generation to begin with, which is at least honest about the metaprogramming there. Reflection is just such a dangerous feature that looks like ordinary code, which is why it is something to be avoided, and having an LLM write/analyze the code for eliminates the need to use a lot of reflective code to begin with.
- xnorswap 2mo ago
  And those batteries will soon include the very benchmark in this article. There'll be no need to build a zstd decoder, .NET will ship one in dotnet 11: https://learn.microsoft.com/en-us/dotnet/api/system.io.compression.zstandarddecoder?view=net-11.0 https://learn.microsoft.com/en-us/dotnet/api/system.io.compr... Another slept-on feature is the ability to quickly write your own static analyzers, which along with the myriad of existing analyzers out there, provide excellent guard-rails to prevent slop.
deleted 2mo ago
[deleted]
boxed 2mo ago
> Go is absolutely one of the best programming languages for LLMs for the reason you say, and Python is just what LLMs like to use to write short throwaway scripts. And yet this article has pretty strong empirical data to show that your intuition here is incorrect. You should back up your statement with something more than vibes.
- win311fwg 2mo ago
  Best to read the comments before replying. The article is about correctness, while the parent is talking about output quality.
  - boxed 2mo ago
    What is output quality without correctness? That seems like a distinction without a difference. Is the claim that LLMs produce Go code that is superficially nice looking but in fact fail to solve the stated problem? Because that's an anti-Go position I'd say.
    - win311fwg 2mo ago
      Correctness is binary, while quality is not. Correctness is a suitable property to act as a multiplier in your formula, where incorrect is 0 and correct is 1, but you also need other facets to find a quality gradient.
      - boxed 2mo ago
        In this context it's not binary. Context is everything. If it was binary there would only be 0 and 1 on one of the axis in the graph. That's not the case.
        win311fwg 2mo ago
        Correctness is binary even in context. The axis of which you speak shows distance; essentially how close the programs were to being correct. Every single sample was incorrect. Think of it as being like a road trip. Arrival is binary. You have either arrived or have not arrived, but distance can tell you how close you are to arrival. Being almost there does not imply that you have arrived, however. Same applies here. Some samples were closer to being correct than others, but none were correct. They were all incorrect. But as you alluded to earlier, I don't suppose anyone wants code that fails to solve the stated problem. Given that we have empirical evidence that LLMs cannot produce correct code within a given set of problems (and I suspect that extends to most problems), correctness is a weak signal. What is a useful is to know is how much additional effort is required to make the program correct. That is what quality has traditionally meant as it pertains to code. High quality codebases are considered high quality because the effort to reach and maintain correctness is considered to be low. We do not have enough context to know for certain if that is what was meant in the earlier comment, but it seems likely. What we do know is that the comment is about quality while the article is about correctness.
majoe 2mo ago
> C++ is also surprisingly good, but you pretty much have to tell the LLMs to treat it like Go and don't use any of the dangerous features for normal code. For existing codebases I made the experience, that LLMs are very good at replicating their style. At work most of our C++ codebases use a fairly consistent style and subset of C++ features and to my initial surprise specifying style conventions etc explicitly turned out to be mostly superfluous. Of course, we also have some legacy projects originally, written in ANSI C, which only received a few changes in the last 15 years to compile with a C++ compiler. Here a style guide is helpful, bit I consider it more like a temporary instruction for refactoring.
katzenversteher 2mo ago
If you want the opposite experience, try LLMs with an older Oracle Database. Even with the official Oracle SqlDeveloper MCP Server they struggle a lot. However, I struggle as well and the error messages of Oracle are some of the worst I've ever experienced.
- cryptos 2mo ago
  The error messages are just advertising for Oracle consulting!
nojvek 2mo ago
I find golang very verbose and unreadable. With 1000s of lines LLMs generate, it’s hard to make sense of is it good code or bad code. Add in golang doesn’t have proper types for Option<T> to have a nilable type. Its pointers and null pointer bugs are plenty, no proper stack traces. I find myself to be less productive.

bob1029 2mo ago

I think a lot of people are sleeping on the advantages of "batteries included" ecosystems. The need to select an appropriate 3rd party library represents an entire dimension of the search space that can be eliminated. Imagine having to make this choice multiple times per day when your competition is just mindlessly using System.* types. The fact that the .NET ecosystem is curated by one entity should not be underestimated. Even when we do need to import 3rd party nugets, the models seem to follow this highly structured pattern. They scan the xml docs, and failing that they will build a throwaway console app to reflect over all the unique types and build a report. The fact that we can easily do this with a simple powershell command makes a big difference. How many other ecosystems can even consider doing this? Reflection is a superpower, not something to be avoided.

YuechenLi 2mo ago
Reflection is good for prototyping and get something setup quickly, but if you build your architecture around it, not only do you lose access to NativeAOT, the code becomes very hard to debug, and if you code with LLM a lot, you either have to spend time trying to debug reflections or just rewrite it with source generation to begin with, which is at least honest about the metaprogramming there. Reflection is just such a dangerous feature that looks like ordinary code, which is why it is something to be avoided, and having an LLM write/analyze the code for eliminates the need to use a lot of reflective code to begin with.
xnorswap 2mo ago
And those batteries will soon include the very benchmark in this article. There'll be no need to build a zstd decoder, .NET will ship one in dotnet 11: https://learn.microsoft.com/en-us/dotnet/api/system.io.compression.zstandarddecoder?view=net-11.0 https://learn.microsoft.com/en-us/dotnet/api/system.io.compr... Another slept-on feature is the ability to quickly write your own static analyzers, which along with the myriad of existing analyzers out there, provide excellent guard-rails to prevent slop.

deleted 2mo ago

[deleted]

boxed 2mo ago

> Go is absolutely one of the best programming languages for LLMs for the reason you say, and Python is just what LLMs like to use to write short throwaway scripts. And yet this article has pretty strong empirical data to show that your intuition here is incorrect. You should back up your statement with something more than vibes.

win311fwg 2mo ago
Best to read the comments before replying. The article is about correctness, while the parent is talking about output quality.
- boxed 2mo ago
  What is output quality without correctness? That seems like a distinction without a difference. Is the claim that LLMs produce Go code that is superficially nice looking but in fact fail to solve the stated problem? Because that's an anti-Go position I'd say.
  - win311fwg 2mo ago
    Correctness is binary, while quality is not. Correctness is a suitable property to act as a multiplier in your formula, where incorrect is 0 and correct is 1, but you also need other facets to find a quality gradient.
    - boxed 2mo ago
      In this context it's not binary. Context is everything. If it was binary there would only be 0 and 1 on one of the axis in the graph. That's not the case.
      - win311fwg 2mo ago
        Correctness is binary even in context. The axis of which you speak shows distance; essentially how close the programs were to being correct. Every single sample was incorrect. Think of it as being like a road trip. Arrival is binary. You have either arrived or have not arrived, but distance can tell you how close you are to arrival. Being almost there does not imply that you have arrived, however. Same applies here. Some samples were closer to being correct than others, but none were correct. They were all incorrect. But as you alluded to earlier, I don't suppose anyone wants code that fails to solve the stated problem. Given that we have empirical evidence that LLMs cannot produce correct code within a given set of problems (and I suspect that extends to most problems), correctness is a weak signal. What is a useful is to know is how much additional effort is required to make the program correct. That is what quality has traditionally meant as it pertains to code. High quality codebases are considered high quality because the effort to reach and maintain correctness is considered to be low. We do not have enough context to know for certain if that is what was meant in the earlier comment, but it seems likely. What we do know is that the comment is about quality while the article is about correctness.

majoe 2mo ago

> C++ is also surprisingly good, but you pretty much have to tell the LLMs to treat it like Go and don't use any of the dangerous features for normal code. For existing codebases I made the experience, that LLMs are very good at replicating their style. At work most of our C++ codebases use a fairly consistent style and subset of C++ features and to my initial surprise specifying style conventions etc explicitly turned out to be mostly superfluous. Of course, we also have some legacy projects originally, written in ANSI C, which only received a few changes in the last 15 years to compile with a C++ compiler. Here a style guide is helpful, bit I consider it more like a temporary instruction for refactoring.

katzenversteher 2mo ago

If you want the opposite experience, try LLMs with an older Oracle Database. Even with the official Oracle SqlDeveloper MCP Server they struggle a lot. However, I struggle as well and the error messages of Oracle are some of the worst I've ever experienced.

cryptos 2mo ago
The error messages are just advertising for Oracle consulting!

nojvek 2mo ago

I find golang very verbose and unreadable. With 1000s of lines LLMs generate, it’s hard to make sense of is it good code or bad code. Add in golang doesn’t have proper types for Option<T> to have a nilable type. Its pointers and null pointer bugs are plenty, no proper stack traces. I find myself to be less productive.