3 ms·

Couldn’t an llm provided with the right level of logs write really good characterization tests?

by totetsu 3y ago

crowcroft 3y ago
The seems like a perfect use case. Quickly find all the foot guns you didn’t know to look for.
- wahnfrieden 3y ago
  as long as you have process to dismantle the tests and move fully over to a new system, if you are indeed migrating/upgrading. leaving a legacy thing dangling and tightly coupled tests lingering for years happens easily when going from 95% to 100% can cost too much for management and stakeholders in various ways relative to other pressing needs
  - skissane 3y ago
    > as long as you have process to dismantle the tests and move fully over to a new system, if you are indeed migrating/upgrading. leaving a legacy thing dangling and tightly coupled tests lingering for years happens easily when going from 95% to 100% can cost too much for management and stakeholders in various ways relative to other pressing needs Characterisation tests are not supposed to be tightly coupled – they are supposed to be integration/end-to-end tests not unit tests – the point is to ensure that some business process continues to produce the same outputs given the same inputs, not that the internals of how it produces that output are unchanged. Code coverage is used as an (imperfect) measure of how complete your set of test inputs is, and as a tool to help discover new test inputs, and minimise test inputs (if two test inputs all hit the same lines/branches, maybe it is wasteful to keep both of them–although it isn't just about code coverage, e.g. extreme values such as maximums and minimums can be valuable in the test suite even if they don't actually increase coverage.) They can take the form of unit tests if you are focusing on refactoring a specific component, and want to ensure its interactions with the rest of the application are not changed. But at some point, a larger redesign may get rid of that component entirely, at which point you can throw those unit tests away, but you'll likely keep the system-level tests
skissane 3y ago
A significant part of writing characterisation tests can be simply staring at a code coverage report and asking “can I write a test (possibly by modifying an existing one) which hits this line/branch”. Sometimes that’s easy, sometimes that’s hard, sometimes that’s impossible (code bases, especially crapulent legacy ones, sometimes contain large sections of dead code which are impossible to reach given any input). An LLM doesn’t have to always get it right to be useful-have it generate a whole bunch of tests, run them all, keep the ones which hit new lines/conditions, maybe even feed those results back in to see if it can iteratively improve, stop when it is no longer generating useful tests. Hopefully, that addresses most of the low-hanging fruit, and leaves the harder cases to a human. There already exist automated test generation systems which can do some of this–for example, concolic testing-but an LLM can be viewed as just another tool in the toolbox, which may sometimes be able to generate tests which concolic testing can’t, or possibly produce the same tests quicker than concolic testing would. There is also the potential for them to interact synergistically - the LLM might produce a test which concolic testing couldn’t, but then concolic testing might then use that to discover further tests which the LLM couldn’t.

Couldn’t an llm provided with the right level of logs write really good characterization tests?

crowcroft 3y ago

The seems like a perfect use case. Quickly find all the foot guns you didn’t know to look for.

wahnfrieden 3y ago
as long as you have process to dismantle the tests and move fully over to a new system, if you are indeed migrating/upgrading. leaving a legacy thing dangling and tightly coupled tests lingering for years happens easily when going from 95% to 100% can cost too much for management and stakeholders in various ways relative to other pressing needs
- skissane 3y ago
  > as long as you have process to dismantle the tests and move fully over to a new system, if you are indeed migrating/upgrading. leaving a legacy thing dangling and tightly coupled tests lingering for years happens easily when going from 95% to 100% can cost too much for management and stakeholders in various ways relative to other pressing needs Characterisation tests are not supposed to be tightly coupled – they are supposed to be integration/end-to-end tests not unit tests – the point is to ensure that some business process continues to produce the same outputs given the same inputs, not that the internals of how it produces that output are unchanged. Code coverage is used as an (imperfect) measure of how complete your set of test inputs is, and as a tool to help discover new test inputs, and minimise test inputs (if two test inputs all hit the same lines/branches, maybe it is wasteful to keep both of them–although it isn't just about code coverage, e.g. extreme values such as maximums and minimums can be valuable in the test suite even if they don't actually increase coverage.) They can take the form of unit tests if you are focusing on refactoring a specific component, and want to ensure its interactions with the rest of the application are not changed. But at some point, a larger redesign may get rid of that component entirely, at which point you can throw those unit tests away, but you'll likely keep the system-level tests

skissane 3y ago

A significant part of writing characterisation tests can be simply staring at a code coverage report and asking “can I write a test (possibly by modifying an existing one) which hits this line/branch”. Sometimes that’s easy, sometimes that’s hard, sometimes that’s impossible (code bases, especially crapulent legacy ones, sometimes contain large sections of dead code which are impossible to reach given any input). An LLM doesn’t have to always get it right to be useful-have it generate a whole bunch of tests, run them all, keep the ones which hit new lines/conditions, maybe even feed those results back in to see if it can iteratively improve, stop when it is no longer generating useful tests. Hopefully, that addresses most of the low-hanging fruit, and leaves the harder cases to a human. There already exist automated test generation systems which can do some of this–for example, concolic testing-but an LLM can be viewed as just another tool in the toolbox, which may sometimes be able to generate tests which concolic testing can’t, or possibly produce the same tests quicker than concolic testing would. There is also the potential for them to interact synergistically - the LLM might produce a test which concolic testing couldn’t, but then concolic testing might then use that to discover further tests which the LLM couldn’t.