3 ms·
It doesn’t matter to me how good the LLM is at writing Go if the compiler can’t stop it from accidentally leaving another part of the software with invalid stat
by beaker52 2mo ago
It doesn’t matter to me how good the LLM is at writing Go if the compiler can’t stop it from accidentally leaving another part of the software with invalid state as a result of a change the LLM is making.
What am I talking about? Nil and partially constructed structs are impossible to prevent the creation of in Go.
Sure, if you’ve got a small program with limited scope, that’s probably fine if you look through squinted eyes. But the teams I work with are working on sprawling, evolving software where the compiler saying “hey, that’s not a valid Widget” would be extremely useful and save much heartache.
An LLM does a good job of “checking” for other uses and “checking” if everything is going to work correctly, but - supposedly we’ve committed the concept to code so that the compiler can actually verify it - and Go intentionally permits invalid states of structs. This makes Go a fundamentally problematic language choice for the kind of software I work with teams on, LLM or not.
- temphaaa 2mo agoi am using nilaway from fb it's great for those
- tgv 2mo agoIt's from uber, I believe.
- baalimago 2mo agoI've found that keeping very healthy test coverage solves issues like this. LLMs are very good at validating their own implementations with TDD.
- cookiengineer 2mo agoWith unit tests you gotta be careful though, oftentimes LLMs skip implementations with mockups that just say "not implemented yet" or similar and then the unit tests become pointless because they start to only test internal structures for being set / not default values. For me it helped a lot to try to make containerized end-to-end tests and a custom TestMain for this, where I am using podman to run the integration tests. This way the end-to-end tests are forced to be on network level, and you can test protocol and API quirks much easier with LLMs. Also, never forget to write a bootstrapping docs/ folder so that you don't have to re-explain these things all the time.
- aktau 2mo ago+1, I like to test in a similar way. For example if I'm making a CLI, the test will spawn the CLI for each test-case, instead of invoking the code directly. This provides a more realistic flow, and makes it clear which user journeys one is supporting. Faking responses I often do with environment variables, like: ts := httptest.NewServer(...) cmd := exec.Command(...) cmd.Env = append(os.Environ(), fmt.Sprintf("MYCLI_PROD_ADDR=%s", ts.URL)) Of course, it's better still to go down this turtle stack (e.g. by spawning a local instance of your backend server instead of some faked handlers), but that adds more cost. I find the trade-off OK here.
- cookiengineer 2mo agoI actually had to use a similar hack there due to the limitation that go test compilates cannot spawn themselves where I needed to have an environment variable with the actual binary prebuilt before the tests run. Took me a while to understand that TestMain doesn't cover that use case...
- aktau 2mo agoOn Linux, "self" is /proc/self/exe. In general, I've seen people use `os.Args[0]`. But on checking again, I see there's even `os.Executable()` (https://pkg.go.dev/os#Executable https://pkg.go.dev/os#Executable) for this purpose. I'm quite sure go test compilates can spawn themselves, I'm doing it on many platforms. But, the test setup I was referring to was explicitly not that: the tests are spawning the main binary, not themselves. Using bazel+runfiles this is pretty easy to do. With the pure Go build tool, I'm not sure what approach I'd use to "guarantee" that I get a binary build for the same environment as the test.
- deleted 2mo ago[deleted]
- dnautics 2mo agoi think they validate less with TDD; if you tell them a bare bones spec to validate they tend to write just that. if you write the test after, then their testing is causally conditioned on what was written. if you are going to write tests, it seems like teat first is way better than test last.
- cookiengineer 2mo agoSo your problem is the default/zero values of properties? In Go the convention is kind of to have a constructor pattern with a NewStruct(...) *Struct method that initializes all properties. Also can't you build your own validator for that with the reflect package in the Add() method of your UI graph to prevent this sorta thing?
- TheDong 2mo agoThe Go std library, as well as practically all go library code, is full of things that don't fully initialize all properties and things that nil-pointer-panic if you hold them wrong, so no, no matter what you do you have to deal with this wart of Go. The go type-system is simply incapable of enforcing nil-safety without being no longer able to compile the go stdlib nor most code in the wild, so it's a quite valid criticism of the go type-system and language, and your comment doesn't hit on a valid solution.
- deleted 2mo ago[deleted]
- 0x696C6961 2mo agoYou're painting a picture where people writing Go are constantly drowning in nil pointer panics. This is not reality. You hit them occasionally and they're trivial to understand and fix.
- jchook 2mo agoGood point. It's sometimes challenging to get a Rust program to compile... but if you do, it's probably going to work.
- moomin 2mo agoThe authors make some good points: compile time and test speed matter, platforms matter. But they really dodge the whole “guardrails matter” thing. And guardrails are going to win long term. As for readability, the fact that AI-written Go closely resembles human-written Go is not necessarily a point in Go’s favour.
- m00x 2mo ago> As for readability, the fact that AI-written Go closely resembles human-written Go is not necessarily a point in Go’s favour. There's just not many ways of writing Go. It's a very dull language. It was designed to be dull and easily understandable.
- dnautics 2mo agobut there are a whole lot of ways you can mess up a dull language, like shitty variable names, bad code organization, etc. if an llm has learned dumb things from dumb users it could disproportionately cause provlems versus other languages, just by being "in a sloppy mood" when writing go.
- toolslive 2mo ago> "guardrails matter" This thing has been solved in the 70s. Basically, have a powerful type system, and a compiler that beats you into submission when you try to stray from the straight and narrow. Languages like (S/OCa)ML, Haskell and Rust will bring this to you. As a consequence, you fight the compiler, and once it submits, you have a good chance it's going to work. The compiler is also the ultimate refactoring tool here. Change the concept? Just change the type in the code and follow through by fixing the error messages the compiler spits out.
- BodyCulture 2mo agoVery good point, thanks for confirming that! What is the reasoning behind the Go design choices you described?
- Cthulhu_ 2mo agoFair criticism, it's my main gripe with Go as well - it's not strict enough when it comes to e.g. nil, enums, and type safety. Annotations are supported but they're just strings. Projects require additional tooling / linters to check for things like unchecked errors and many more "gotchas" that I think could (should?) be part of the compiler or standard tools. Trivial example, Go's compiler will error when you have an unused variable, but won't if you reuse and overwrite an error variable a dozen times and only handle it once. But on the other hand, I suppose it makes it a bit more pragmatic - less checks makes for a faster compiler, and fast compilation was/is very high up in the language's requirements and motivation. If you want / need more strictness in your language, there's Rust, Java, C#, etc.
- larodi 2mo agoHow about Swift and Erlang? Why are these always left out of comparisons?
- byward 2mo agoHaven’t you heard? Swift is the first-ever corporate programming language, and we on HN can’t take its community efforts seriously to the extent of considering its potential value in general-purpose software. /s
- larodi 2mo agoI see it as a great language being driven (fw atm) by same buy who did Rust, no? And my take really is that the fact they were no good corpo created languages (GO is not one or what?), then it is totally fine to some big corpo like the much-disloved-yet-selling-shitloads-of-phones Apple to actually do something which benefits the world and does not incur costs ojn it.
- dnautics 2mo agoi never run into these issues in >100k loc of productionized llm generated elixir (nil safety, type issues). i wonder, is there something architecturally in go that makes this a particular problem?
- apancyborg 2mo agoWith test, which LLM are good at writing, you can reduce this tremendously. Even Rust won't protect you in this case, remember the cloudflare outage. At the end it is not the language, it is you intrinsic ability to architecture well your software from there any LLM can do the work.
- dimaaan 2mo ago>> Nil and partially constructed structs C# has "nullable reference types" and "required" keyword for that