3 ms·

I've found that keeping very healthy test coverage solves issues like this. LLMs are very good at validating their own implementations with TDD.

cookiengineer 2mo ago
With unit tests you gotta be careful though, oftentimes LLMs skip implementations with mockups that just say "not implemented yet" or similar and then the unit tests become pointless because they start to only test internal structures for being set / not default values. For me it helped a lot to try to make containerized end-to-end tests and a custom TestMain for this, where I am using podman to run the integration tests. This way the end-to-end tests are forced to be on network level, and you can test protocol and API quirks much easier with LLMs. Also, never forget to write a bootstrapping docs/ folder so that you don't have to re-explain these things all the time.
- aktau 2mo ago
  +1, I like to test in a similar way. For example if I'm making a CLI, the test will spawn the CLI for each test-case, instead of invoking the code directly. This provides a more realistic flow, and makes it clear which user journeys one is supporting. Faking responses I often do with environment variables, like: ts := httptest.NewServer(...) cmd := exec.Command(...) cmd.Env = append(os.Environ(), fmt.Sprintf("MYCLI_PROD_ADDR=%s", ts.URL)) Of course, it's better still to go down this turtle stack (e.g. by spawning a local instance of your backend server instead of some faked handlers), but that adds more cost. I find the trade-off OK here.
  - cookiengineer 2mo ago
    I actually had to use a similar hack there due to the limitation that go test compilates cannot spawn themselves where I needed to have an environment variable with the actual binary prebuilt before the tests run. Took me a while to understand that TestMain doesn't cover that use case...
    - aktau 1mo ago
      On Linux, "self" is /proc/self/exe. In general, I've seen people use `os.Args[0]`. But on checking again, I see there's even `os.Executable()` (https://pkg.go.dev/os#Executable https://pkg.go.dev/os#Executable) for this purpose. I'm quite sure go test compilates can spawn themselves, I'm doing it on many platforms. But, the test setup I was referring to was explicitly not that: the tests are spawning the main binary, not themselves. Using bazel+runfiles this is pretty easy to do. With the pure Go build tool, I'm not sure what approach I'd use to "guarantee" that I get a binary build for the same environment as the test.
      - deleted 1mo ago
        [deleted]
dnautics 2mo ago
i think they validate less with TDD; if you tell them a bare bones spec to validate they tend to write just that. if you write the test after, then their testing is causally conditioned on what was written. if you are going to write tests, it seems like teat first is way better than test last.

cookiengineer 2mo ago

With unit tests you gotta be careful though, oftentimes LLMs skip implementations with mockups that just say "not implemented yet" or similar and then the unit tests become pointless because they start to only test internal structures for being set / not default values. For me it helped a lot to try to make containerized end-to-end tests and a custom TestMain for this, where I am using podman to run the integration tests. This way the end-to-end tests are forced to be on network level, and you can test protocol and API quirks much easier with LLMs. Also, never forget to write a bootstrapping docs/ folder so that you don't have to re-explain these things all the time.

aktau 2mo ago
+1, I like to test in a similar way. For example if I'm making a CLI, the test will spawn the CLI for each test-case, instead of invoking the code directly. This provides a more realistic flow, and makes it clear which user journeys one is supporting. Faking responses I often do with environment variables, like: ts := httptest.NewServer(...) cmd := exec.Command(...) cmd.Env = append(os.Environ(), fmt.Sprintf("MYCLI_PROD_ADDR=%s", ts.URL)) Of course, it's better still to go down this turtle stack (e.g. by spawning a local instance of your backend server instead of some faked handlers), but that adds more cost. I find the trade-off OK here.
- cookiengineer 2mo ago
  I actually had to use a similar hack there due to the limitation that go test compilates cannot spawn themselves where I needed to have an environment variable with the actual binary prebuilt before the tests run. Took me a while to understand that TestMain doesn't cover that use case...
  - aktau 1mo ago
    On Linux, "self" is /proc/self/exe. In general, I've seen people use `os.Args[0]`. But on checking again, I see there's even `os.Executable()` (https://pkg.go.dev/os#Executable https://pkg.go.dev/os#Executable) for this purpose. I'm quite sure go test compilates can spawn themselves, I'm doing it on many platforms. But, the test setup I was referring to was explicitly not that: the tests are spawning the main binary, not themselves. Using bazel+runfiles this is pretty easy to do. With the pure Go build tool, I'm not sure what approach I'd use to "guarantee" that I get a binary build for the same environment as the test.
    - deleted 1mo ago
      [deleted]

dnautics 2mo ago

i think they validate less with TDD; if you tell them a bare bones spec to validate they tend to write just that. if you write the test after, then their testing is causally conditioned on what was written. if you are going to write tests, it seems like teat first is way better than test last.