31 ms·

In terms of LOCs maybe, in terms of importance I think is much less. At least that's how I use LLMs. While I understand that <Enter model here> might produce t

In terms of LOCs maybe, in terms of importance I think is much less. At least that's how I use LLMs. While I understand that <Enter model here> might produce the meaty bits as well, I believe that having a truck factor of basically 0 (since no-one REALLY understands the code) is a recipe for a disaster and I dare say long term maintainability of a code base. I feel that you need to have someone in any team that needs to have that level of understanding to fix non trivial issues. However, by all means, I use the LLM to create all the scaffolding, test fixtures, ... because that is mental energy that I can use elsewhere.

epicureanideal 1y ago
Agreed. If I use an LLM to generate fairly exhaustive unit tests of a trivial function just because I can, that doesn’t mean those lines are as useful as core complex business logic that it would almost certainly make subtle mistakes in.
- andsoitis 1y ago
  > If I … generate fairly exhaustive unit tests of a trivial function … then you are not a senior software engineer
  - triyambakam 1y ago
    Neither are you if that's your understanding of a senior engineer
    - mgh95 1y ago
      I think the parent commentors point was that it is nearly trivial to generate variations on unit tests in most (if not all) unit test frameworks. For example: Java: https://docs.parasoft.com/display/JTEST20232/Creating+a+Parameterized+Unit+Test https://docs.parasoft.com/display/JTEST20232/Creating+a+Para... C# (nunit, but xunit has this too): https://docs.nunit.org/articles/nunit/technical-notes/usage/Parameterized-Tests.html https://docs.nunit.org/articles/nunit/technical-notes/usage/... Python: https://docs.pytest.org/en/stable/example/parametrize.html https://docs.pytest.org/en/stable/example/parametrize.html cpp: https://google.github.io/googletest/advanced.html https://google.github.io/googletest/advanced.html A belief that the ability of LLMs to generate parameterizations is intrinsically helpful to a degree which cannot be trivially achieved in most mainstream programming languages/test frameworks may be an indicator that an individual has not achieved a substantial depth of experience.
      - com2kid 1y ago
        The useful part is generating the mocks. The various auto mocking frameworks are so hit or miss I end up having to manually make mocks which is time consuming and boring. LLMs help out dramatically and save literally hours of boring error prone work.
        mgh95 1y ago
        Why mock at all? Spend the time making integration tests fast. There is little reason a database, queue, etc. can't be set up in a per-test group basis and be made fast. Reliable software is built upon (mostly) reliable foundations.
        lanstin 1y ago
        hmmmm. I do like integration tests, but I often tell people the art of modern software is to make reliable systems on top of unreliable components. And the integration tests should 100% include times when the network flakes out and drops 1/2 of replies and corrupts msgs and the like.
        mgh95 1y ago
        > I do like integration tests, but I often tell people the art of modern software is to make reliable systems on top of unreliable components. There is a dramatic difference between unreliable in the sense of S3 or other services and unreliable as in "we get different sets of logical outputs when we provide the same input to a LLM". In the first, you can prepare for what are logical outcomes -- network failures, durability loss, etc. In the latter, unless you know the total space of outputs for a LLM you cannot prepare. In the operational sense, LLMs are not a system component, they are a system builder. And a rather poor one, at that. > And the integration tests should 100% include times when the network flakes out and drops 1/2 of replies and corrupts msgs and the like. Yeah, it's not that hard to include that in modern testing.
        sethammons 1y ago
        Minor nit. I wouldn't call those failing systems tests integration tests. Unit tests are for validation of error paths. Unit tests can leverage mocks or fakes. Need 3 retires with exponential back off, use unit tests and fakes. Integration tests should use real components. Typically, integration tests are happy path and unit are error paths. Making real components fail and having tests validate failure handling in a more complete environment jumps from integration testing to resilience or chaos testing. Being able to accurately validate backoffs and retries may diminish, but validating intermediate or ending state can be done with artifact monitoring via sinks. There is unit-integration testing which fakes out as little as possible but still fakes out some edges. The difference being that the failures are introduced via fake vs managing actual system components. If you connect to a real db on unit-integration tests, you typically wouldn't kill the db or use Comcast to slow the network artificially. That would be reserved for the next layer in the test pyramid.
    - goosejuice 1y ago
      We're not a licensed profession with universally defined roles. It's whatever the speaker wants it to be given how wildly it varies.
fergie 1y ago
> test fixtures I'm curious- how does the AI know what you want?
- marcyb5st 1y ago
  I use CodeCompanion on neovim. My workflow for text fixtures is basically: """ Given these files: file1, file2, ... (these are pulled entirely into the LLM context) Create a test fixture by creating a type that implements the trait A and should use an in memory SQLite DB, another one that implements Trait B, ... """ Of course there is a bit of back and forth, but I find that using Interfaces/Traits/ABCs extensively makes LLMs perform better at these tasks (but I believe it is a nice side-effect of having more testable code to begin with). However, wiring with IoC frameworks are a bit of a hit and miss to be honest so often I still have to do these parts manually.

epicureanideal 1y ago

Agreed. If I use an LLM to generate fairly exhaustive unit tests of a trivial function just because I can, that doesn’t mean those lines are as useful as core complex business logic that it would almost certainly make subtle mistakes in.

andsoitis 1y ago
> If I … generate fairly exhaustive unit tests of a trivial function … then you are not a senior software engineer
- triyambakam 1y ago
  Neither are you if that's your understanding of a senior engineer
  - mgh95 1y ago
    I think the parent commentors point was that it is nearly trivial to generate variations on unit tests in most (if not all) unit test frameworks. For example: Java: https://docs.parasoft.com/display/JTEST20232/Creating+a+Parameterized+Unit+Test https://docs.parasoft.com/display/JTEST20232/Creating+a+Para... C# (nunit, but xunit has this too): https://docs.nunit.org/articles/nunit/technical-notes/usage/Parameterized-Tests.html https://docs.nunit.org/articles/nunit/technical-notes/usage/... Python: https://docs.pytest.org/en/stable/example/parametrize.html https://docs.pytest.org/en/stable/example/parametrize.html cpp: https://google.github.io/googletest/advanced.html https://google.github.io/googletest/advanced.html A belief that the ability of LLMs to generate parameterizations is intrinsically helpful to a degree which cannot be trivially achieved in most mainstream programming languages/test frameworks may be an indicator that an individual has not achieved a substantial depth of experience.
    - com2kid 1y ago
      The useful part is generating the mocks. The various auto mocking frameworks are so hit or miss I end up having to manually make mocks which is time consuming and boring. LLMs help out dramatically and save literally hours of boring error prone work.
      - mgh95 1y ago
        Why mock at all? Spend the time making integration tests fast. There is little reason a database, queue, etc. can't be set up in a per-test group basis and be made fast. Reliable software is built upon (mostly) reliable foundations.
        lanstin 1y ago
        hmmmm. I do like integration tests, but I often tell people the art of modern software is to make reliable systems on top of unreliable components. And the integration tests should 100% include times when the network flakes out and drops 1/2 of replies and corrupts msgs and the like.
        mgh95 1y ago
        > I do like integration tests, but I often tell people the art of modern software is to make reliable systems on top of unreliable components. There is a dramatic difference between unreliable in the sense of S3 or other services and unreliable as in "we get different sets of logical outputs when we provide the same input to a LLM". In the first, you can prepare for what are logical outcomes -- network failures, durability loss, etc. In the latter, unless you know the total space of outputs for a LLM you cannot prepare. In the operational sense, LLMs are not a system component, they are a system builder. And a rather poor one, at that. > And the integration tests should 100% include times when the network flakes out and drops 1/2 of replies and corrupts msgs and the like. Yeah, it's not that hard to include that in modern testing.
        sethammons 1y ago
        Minor nit. I wouldn't call those failing systems tests integration tests. Unit tests are for validation of error paths. Unit tests can leverage mocks or fakes. Need 3 retires with exponential back off, use unit tests and fakes. Integration tests should use real components. Typically, integration tests are happy path and unit are error paths. Making real components fail and having tests validate failure handling in a more complete environment jumps from integration testing to resilience or chaos testing. Being able to accurately validate backoffs and retries may diminish, but validating intermediate or ending state can be done with artifact monitoring via sinks. There is unit-integration testing which fakes out as little as possible but still fakes out some edges. The difference being that the failures are introduced via fake vs managing actual system components. If you connect to a real db on unit-integration tests, you typically wouldn't kill the db or use Comcast to slow the network artificially. That would be reserved for the next layer in the test pyramid.
  - goosejuice 1y ago
    We're not a licensed profession with universally defined roles. It's whatever the speaker wants it to be given how wildly it varies.

fergie 1y ago

> test fixtures I'm curious- how does the AI know what you want?

marcyb5st 1y ago
I use CodeCompanion on neovim. My workflow for text fixtures is basically: """ Given these files: file1, file2, ... (these are pulled entirely into the LLM context) Create a test fixture by creating a type that implements the trait A and should use an in memory SQLite DB, another one that implements Trait B, ... """ Of course there is a bit of back and forth, but I find that using Interfaces/Traits/ABCs extensively makes LLMs perform better at these tasks (but I believe it is a nice side-effect of having more testable code to begin with). However, wiring with IoC frameworks are a bit of a hit and miss to be honest so often I still have to do these parts manually.