6 ms·

This red vs blue team is a good way to understand the capabilities and current utility of LLMs for expert use. I trust them to add tests almost indiscriminately

This red vs blue team is a good way to understand the capabilities and current utility of LLMs for expert use. I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them; and if they are correct, they adds value. But often they don’t test the core functionality; the best tests I still have to write myself. Having LLMs fix bugs or add features is more fraught, since they are prone to cheating or writing non robust code (eg special code paths to pass tests without solving the actual problem).

skdidjdndh 1y ago
> I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. Having worked on legacy codebases, some of the hardest problems are determining “why is this broken test here that appears to test a behavior we don’t support”. Do we have a bug? Or do we have a bad test? On the other end, when there are tests for scenarios we don’t actually care about it’s impossible to determine if that test is meaningful or was added because “it’s testing the code as written”.
- yojo 1y ago
  I would add that few things slow developer velocity as much as a large suite of comprehensive and brittle tests. This is just as true on greenfield as on legacy. Anticipating future responses: yes, a robust test harness allows you to make changes fearlessly. But most big test suites I’ve seen are less “harness” and more “straight-jacket”
  - andrepd 1y ago
    I don't understand this. How does it slow your development if the tests being green is a necessary condition for the code being correct? Yes it slows it compared to just writing incorrect code lol, but that's not the point.
    - yojo 1y ago
      "Brittle" here means either: 1) your test is specific to the implementation at the time of writing, not the business logic you mean to enforce. 2) your test has non-deterministic behavior (more common in end-to-end tests) that cause it to fail some small percentage of the time on repeated runs. At the extreme, these types of tests degenerate your suite into a "change detector," where any modification to the code-base is guaranteed to make one or more tests fail. They slow you down because every code change also requires an equal or larger investment debugging the test suite, even if nothing actually "broke" from a functional perspective. Using LLMs to litter your code-base with low-quality tests will not end well.
    - winstonewert 1y ago
      The problem is that sometimes it is not a necessary condition. Rather, the tests might have been checking implementation details or just been wrong in the first place. Now, when tests fails I have extra work to figure out if its a real break or just a bad test.
    - threatofrain 1y ago
      It's that hard to write specs that truly match the business, hence why test-driven-development or specification-first failed to take off as a movement. Asking specs to truly match the business before we begin using them as tests would handcuff test people in the same way we're saying that tests have the potential to handcuff app and business logic people — as opposed to empowering them. So I wouldn't blame people for writing specs that only match the code implementation at that time. It's hard to engage in prophecy.
      - marcosdumay 1y ago
        > So I wouldn't blame people for writing specs that only match the code implementation at that time. WFT are you doing writing specs based on implementation? If you already have the implementation, what are you using the specs for? Or, if you want to apply this direct to tests, if you are already assuming the program is correct, what are you trying to test? Are you talking about rewriting applications?
  - sour-taste 1y ago
    An old coworker used to call these types of tests change detector tests. They are excellent at telling you whether some behavior changed, but horrible at telling you whether that behavior change is meaningful or not.
    - jrockway 1y ago
      Yup. Working on a 10 year old codebase, I always wondered whether a test failing was "a long-standing bug was accidentally fixed" or "this behavior was added on purpose and customers rely on it". It can be about 50/50 but you're always surprised. Change detector tests add to the noise here. No, this wasn't a feature customers care about, some AI added a test to make sure foo.go line 42 contained less than 80 characters.
      - groestl 1y ago
        > a long-standing bug was accidentally fixed In some cases (e.g. in our case) long standing bugs become part of the API that customers rely on.
        strbean 1y ago
        It's nearly guaranteed, even if it is just because customers had to work around the bug in such a way that their flow now breaks when the bug is gone. Obligatory: https://xkcd.com/1172/ https://xkcd.com/1172/
        giaour 1y ago
        Also known as Hyrum's Law (https://www.hyrumslaw.com/ https://www.hyrumslaw.com/), but more people know the XKCD at this point :)
        thaumasiotes 1y ago
        That comic doesn't show someone working around a bug in such a way that their flow breaks when the bug is gone. It shows them incorporating a bug into their workflow for purposes of causing the bug to occur. It isn't actually possible for fixing a bug to break a workaround. The point of a workaround is that you're not doing the thing that's broken; when that thing is fixed, your flow won't be affected, because you weren't doing it anyway.
  - materielle 1y ago
    I think a problem with AI productivity metrics is that a lot of the productivity is made up. Most enterprise code involves layers of interfaces. So implementing any feature requires updating 5 layers and mocking + unit testing at each layer. When people say “AI helps me generate tests”, I find that this is what they are usually referring to. Generating hundreds of lines of mock and fake data boilerplate in a few minutes, that would otherwise take an entire day to do manually. Of course, the AI didn’t make them more productive. The entire point of automated testing is to ensure software correctness without having to test everything manually each time. The style of unit testing above is basically pointless. Because it doesn’t actually accomplish the goal. All the unit tests could pass and the only thing you’ve tested is that your canned mock responses and asserts are in-sync in the unit testing file. A problem with how LLMs are used is that they help churn through useless bureaucratic BS faster. But the problem is that there’s no ceiling to bureaucracy. I have strong faith that organizations can generate pointless tasks faster than LLMs can automate them away. Of course, this isn’t a problem with LLMs themselves, but rather an organization context in which I see them frequently being used.
    - refulgentis 1y ago
      I think it's appropriate to be skeptical with new tools, and being appropriately, respectfully, prosocially, skeptical, point out failure modes. Kudos. Something that crosses my mind is if AI generating tests necessitates that it only generates tests with fakes and stubs that exercise no actual logic, the expertise required to notice that, and if it is correctable. Yesterday, I was working on some OAuth flow stuff. Without replayed responses, I'm not quite sure how I'd test it without writing my own server, and I'm not sure how I'd develop the expertise to do that without, effectively, just returning the responses I expected. It reminds me that if I eschewed tests with fakes and stubs as untrustworthy in toto, I'd be throwing the baby with the bathwater.
- ozgrakkurt 1y ago
  What do you think about leaning on fuzz testing and deriving unit tests from bugs found by fuzzing?
  - manmal 1y ago
    What kind of bugs do you find this way, besides missing sanitization?
    - raddan 1y ago
      You can often find memory errors not directly related to string handling with fuzz testing. More generally, if your program embodies any kind of state machine, you may find that a good fuzzer drives it into states that you did not think should exist.
      - manmal 1y ago
        That sounds a bit like using a jackhammer to drive in a nail. Wouldn’t it be smarter to enumerate edge cases and test all permutations of those?
        quacksilver 1y ago
        Would it even be possible to enumerate all edge cases and test all the permutations of them in non-trivial codebases or interconnected systems? How do you know when you have all of the edge cases? With fuzzing you can randomly generate bad input that passes all of your test cases that were written using by whatever method you have already been using but still causes the application to crash or behave badly. This may mean that there are more tests that you could write that would catch the issue related to the fuzz case, or the fuzz case itself could be used as a test. Using probability you can get to 90 or 99% or 99.999% or whatever confidence level you need that the software is unaffected by bugs based on the input size / number of fuzz test cases. In many non-critical situations the goal may not be 100% but 'statistically very unlikely with a known probability and error'
- wagwang 1y ago
  This is the conclusion I'm at too, working on a relatively new codebase. Our rule is that every generated test must be human reviewed, otherwise its an autodelete.
- bicx 1y ago
  I believe they just meant that tests are easy to generate for eng review and modification before actually committing to the codebase. Nothing else is a dependency on an individual test (if done correctly), so it's comparatively cheap to add or remove compared to production code.
  - _alternator_ 1y ago
    Yup. I do read and review the tests generated by LLMs. Often the LLM tests will just be more comprehensive than my initial test, and hit edge cases that I didn’t think of (or which are tedious). For example, I’ll write a happy path test case for an API, and a single “bad path” where all of the inputs are bad. The LLM will often generate a bunch of “bad path” cases where only one field has an error. These are great red team tests, and occasionally catch serious bugs.
- manmal 1y ago
  > Tests are the source of truth more so than your code Tests poke and prod with a stick at the SUT, and the SUT's behaviour is observed. The truth lives in the code, the documentation, and, unfortunately, in the heads of the dev team. I think this distinction is quite important, because this question: > Do we have a bug? Or do we have a bad test? cannot be answered by looking at the test + the implementation. The spec or people have to be consulted when in doubt.
  - andruby 1y ago
    What does SUT stand for? I'm not familiar with the acronym Is it "System Under Test"? (That's Claude.ai's guess)
    - dfabulich 1y ago
      It is.
    - card_zero 1y ago
      That's what Wiktionary says too. Lucky guess, Claude.
    - deleted 1y ago
      [deleted]
    - lightbendover 1y ago
      [dead]
    - deleted 1y ago
      [deleted]
  - 9rx 1y ago
    > The spec The tests are your spec. They exist precisely to document what the program is supposed to do for other humans, with the secondary benefit of also telling a machine what the program is supposed to do, allowing implementations to automatically validate themselves against the spec. If you find yourself writing specs and tests as independent things, that's how you end up with bad, brittle tests that make development a nightmare — or you simply like pointless busywork, I suppose. But, yes, you may still have to consult a human if there is reason to believe the spec isn't accurate.
- jgalt212 1y ago
  > Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. I hear you on this, but you can still use so long as these tests are not comingled with the tests generated by subject-matter experts. I'd treat them almost a fuzzers.
- SamuelAdams 1y ago
  Ideally the git history provides the “why was this test written”, however if you have one Jira card tied to 500+ AI generated tests, it’s not terribly helpful.
  - djeastm 1y ago
    >if you have one Jira card tied to 500+ AI generated tests The dreaded "Added tests" commit...
- Pxtl 1y ago
  > “why is this broken test here that appears to test a behavior we don’t support” Because somebody complained when that behavior we don't support was broken, so the bug-that-wasn't-really-a-bug was fixed and a test was created to prevent regression. Imho, the mistake was in documentation: the Test should have comments explaining why this test was created. Just as true for tests as for the actual business logic code: The code can only describe the what and the how. It's up to comments to describe the why.
- layer8 1y ago
  This is why tests need documenting what exactly they intend to test, and why.
mvieira38 1y ago
I have the exact opposite idea. I want the tests to be mine and thoroughly understood, so I am the true arbiter and then I can let the LLM go ham on the code without fear. If the tests are AI made, then I get some anxiety letting agents mess with the rest of the codebase
- _alternator_ 1y ago
  I think this is exactly the tradeoff (blue team and red team need to be matched in power), except that I’ve seen LLMs literally cheat the tests (eg “match input: TEST_INPUT then return TEST_OUTPUT”) far too many times to be comfortable with letting LLMs be a major blue team player.
  - johnisgood 1y ago
    Yeah, they may do that, but people really should read the code an LLM produces. Ugh, makes me furious. No wonder LLMs have a bad rep from such users.
    - otabdeveloper4 1y ago
      > people really should read the code an LLM produces Yeah, but that, like, requires that you know how to code. And wasn't the point of LLMs in the first place to let clueless people make software?
      - johnisgood 1y ago
        I do not know, I would hope not. The bar to entry is already too low. I do not think you will ever be able to get an LLM work flawlessly for people who do not know programming. I know how to code, and I used LLMs before. It seems to be a prerequisite to know how to code if I want useful outputs.
fpoling 1y ago
I tried a LLM to generate tests for Rust code. It was more harmful then useful. Surely there were a lot of tests, but they still miss the key coverage and it was hard to see what was missed due to the amount of generated code. Then to change the code behavior in future would require to fix a lot of tests again versus fixing few lines in manually written tests.
torginus 1y ago
There's a saying that since nobody tests the tests, they must be trivially correct. That's why they came up with the Arrange-Act-Assert pattern. My favorite kind of unit test nowadays is when you store known input-output pairs and validate the code on them. It's easy to test corner cases and see that the output works as desired.
- 01HNNWZ0MV43FF 1y ago
  "Golden snapshot testing"
bravesoul2 1y ago
AI is like a calculatorin this respect. Calculators can do things most humans can't. They make great augmentation devices. AI being a different kind of intelligence is very useful! Everyone is building AI replace human things. But the value is in augmentation.
positron26 1y ago
> prone to cheating or writing non robust code (eg special code paths to pass tests without solving the actual problem). The solution will come from synthetic data training methods that lobotomize part of the weights. It's just cross-validation. A distilled awareness won't maintain knowledge of the cheat paths, exposing them as erroneous. This may a reason why every living thing on Earth that encounters psychoactive drugs seems to enjoy them. Self-deceptive paths depend on consistency whereas truth-preservation of facts grounded in reality will always be re-derived.
theptip 1y ago
I think the more fundamental attribute of interest is how easy it is to verify the work. Much red team work is easily verifiable; either the exploit works or it doesn’t. Whereas more blue-team work is not easily verifiable; it might take judgement to figure out if a feature is promising. LLMs are extremely powerful (and trainable) on tasks with a good oracle.

skdidjdndh 1y ago

> I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. Having worked on legacy codebases, some of the hardest problems are determining “why is this broken test here that appears to test a behavior we don’t support”. Do we have a bug? Or do we have a bad test? On the other end, when there are tests for scenarios we don’t actually care about it’s impossible to determine if that test is meaningful or was added because “it’s testing the code as written”.

yojo 1y ago
I would add that few things slow developer velocity as much as a large suite of comprehensive and brittle tests. This is just as true on greenfield as on legacy. Anticipating future responses: yes, a robust test harness allows you to make changes fearlessly. But most big test suites I’ve seen are less “harness” and more “straight-jacket”
- andrepd 1y ago
  I don't understand this. How does it slow your development if the tests being green is a necessary condition for the code being correct? Yes it slows it compared to just writing incorrect code lol, but that's not the point.
  - yojo 1y ago
    "Brittle" here means either: 1) your test is specific to the implementation at the time of writing, not the business logic you mean to enforce. 2) your test has non-deterministic behavior (more common in end-to-end tests) that cause it to fail some small percentage of the time on repeated runs. At the extreme, these types of tests degenerate your suite into a "change detector," where any modification to the code-base is guaranteed to make one or more tests fail. They slow you down because every code change also requires an equal or larger investment debugging the test suite, even if nothing actually "broke" from a functional perspective. Using LLMs to litter your code-base with low-quality tests will not end well.
  - winstonewert 1y ago
    The problem is that sometimes it is not a necessary condition. Rather, the tests might have been checking implementation details or just been wrong in the first place. Now, when tests fails I have extra work to figure out if its a real break or just a bad test.
  - threatofrain 1y ago
    It's that hard to write specs that truly match the business, hence why test-driven-development or specification-first failed to take off as a movement. Asking specs to truly match the business before we begin using them as tests would handcuff test people in the same way we're saying that tests have the potential to handcuff app and business logic people — as opposed to empowering them. So I wouldn't blame people for writing specs that only match the code implementation at that time. It's hard to engage in prophecy.
    - marcosdumay 1y ago
      > So I wouldn't blame people for writing specs that only match the code implementation at that time. WFT are you doing writing specs based on implementation? If you already have the implementation, what are you using the specs for? Or, if you want to apply this direct to tests, if you are already assuming the program is correct, what are you trying to test? Are you talking about rewriting applications?
- sour-taste 1y ago
  An old coworker used to call these types of tests change detector tests. They are excellent at telling you whether some behavior changed, but horrible at telling you whether that behavior change is meaningful or not.
  - jrockway 1y ago
    Yup. Working on a 10 year old codebase, I always wondered whether a test failing was "a long-standing bug was accidentally fixed" or "this behavior was added on purpose and customers rely on it". It can be about 50/50 but you're always surprised. Change detector tests add to the noise here. No, this wasn't a feature customers care about, some AI added a test to make sure foo.go line 42 contained less than 80 characters.
    - groestl 1y ago
      > a long-standing bug was accidentally fixed In some cases (e.g. in our case) long standing bugs become part of the API that customers rely on.
      - strbean 1y ago
        It's nearly guaranteed, even if it is just because customers had to work around the bug in such a way that their flow now breaks when the bug is gone. Obligatory: https://xkcd.com/1172/ https://xkcd.com/1172/
        giaour 1y ago
        Also known as Hyrum's Law (https://www.hyrumslaw.com/ https://www.hyrumslaw.com/), but more people know the XKCD at this point :)
        thaumasiotes 1y ago
        That comic doesn't show someone working around a bug in such a way that their flow breaks when the bug is gone. It shows them incorporating a bug into their workflow for purposes of causing the bug to occur. It isn't actually possible for fixing a bug to break a workaround. The point of a workaround is that you're not doing the thing that's broken; when that thing is fixed, your flow won't be affected, because you weren't doing it anyway.
- materielle 1y ago
  I think a problem with AI productivity metrics is that a lot of the productivity is made up. Most enterprise code involves layers of interfaces. So implementing any feature requires updating 5 layers and mocking + unit testing at each layer. When people say “AI helps me generate tests”, I find that this is what they are usually referring to. Generating hundreds of lines of mock and fake data boilerplate in a few minutes, that would otherwise take an entire day to do manually. Of course, the AI didn’t make them more productive. The entire point of automated testing is to ensure software correctness without having to test everything manually each time. The style of unit testing above is basically pointless. Because it doesn’t actually accomplish the goal. All the unit tests could pass and the only thing you’ve tested is that your canned mock responses and asserts are in-sync in the unit testing file. A problem with how LLMs are used is that they help churn through useless bureaucratic BS faster. But the problem is that there’s no ceiling to bureaucracy. I have strong faith that organizations can generate pointless tasks faster than LLMs can automate them away. Of course, this isn’t a problem with LLMs themselves, but rather an organization context in which I see them frequently being used.
  - refulgentis 1y ago
    I think it's appropriate to be skeptical with new tools, and being appropriately, respectfully, prosocially, skeptical, point out failure modes. Kudos. Something that crosses my mind is if AI generating tests necessitates that it only generates tests with fakes and stubs that exercise no actual logic, the expertise required to notice that, and if it is correctable. Yesterday, I was working on some OAuth flow stuff. Without replayed responses, I'm not quite sure how I'd test it without writing my own server, and I'm not sure how I'd develop the expertise to do that without, effectively, just returning the responses I expected. It reminds me that if I eschewed tests with fakes and stubs as untrustworthy in toto, I'd be throwing the baby with the bathwater.
ozgrakkurt 1y ago
What do you think about leaning on fuzz testing and deriving unit tests from bugs found by fuzzing?
- manmal 1y ago
  What kind of bugs do you find this way, besides missing sanitization?
  - raddan 1y ago
    You can often find memory errors not directly related to string handling with fuzz testing. More generally, if your program embodies any kind of state machine, you may find that a good fuzzer drives it into states that you did not think should exist.
    - manmal 1y ago
      That sounds a bit like using a jackhammer to drive in a nail. Wouldn’t it be smarter to enumerate edge cases and test all permutations of those?
      - quacksilver 1y ago
        Would it even be possible to enumerate all edge cases and test all the permutations of them in non-trivial codebases or interconnected systems? How do you know when you have all of the edge cases? With fuzzing you can randomly generate bad input that passes all of your test cases that were written using by whatever method you have already been using but still causes the application to crash or behave badly. This may mean that there are more tests that you could write that would catch the issue related to the fuzz case, or the fuzz case itself could be used as a test. Using probability you can get to 90 or 99% or 99.999% or whatever confidence level you need that the software is unaffected by bugs based on the input size / number of fuzz test cases. In many non-critical situations the goal may not be 100% but 'statistically very unlikely with a known probability and error'
wagwang 1y ago
This is the conclusion I'm at too, working on a relatively new codebase. Our rule is that every generated test must be human reviewed, otherwise its an autodelete.
bicx 1y ago
I believe they just meant that tests are easy to generate for eng review and modification before actually committing to the codebase. Nothing else is a dependency on an individual test (if done correctly), so it's comparatively cheap to add or remove compared to production code.
- _alternator_ 1y ago
  Yup. I do read and review the tests generated by LLMs. Often the LLM tests will just be more comprehensive than my initial test, and hit edge cases that I didn’t think of (or which are tedious). For example, I’ll write a happy path test case for an API, and a single “bad path” where all of the inputs are bad. The LLM will often generate a bunch of “bad path” cases where only one field has an error. These are great red team tests, and occasionally catch serious bugs.
manmal 1y ago
> Tests are the source of truth more so than your code Tests poke and prod with a stick at the SUT, and the SUT's behaviour is observed. The truth lives in the code, the documentation, and, unfortunately, in the heads of the dev team. I think this distinction is quite important, because this question: > Do we have a bug? Or do we have a bad test? cannot be answered by looking at the test + the implementation. The spec or people have to be consulted when in doubt.
- andruby 1y ago
  What does SUT stand for? I'm not familiar with the acronym Is it "System Under Test"? (That's Claude.ai's guess)
  - dfabulich 1y ago
    It is.
  - card_zero 1y ago
    That's what Wiktionary says too. Lucky guess, Claude.
  - deleted 1y ago
    [deleted]
  - lightbendover 1y ago
    [dead]
  - deleted 1y ago
    [deleted]
- 9rx 1y ago
  > The spec The tests are your spec. They exist precisely to document what the program is supposed to do for other humans, with the secondary benefit of also telling a machine what the program is supposed to do, allowing implementations to automatically validate themselves against the spec. If you find yourself writing specs and tests as independent things, that's how you end up with bad, brittle tests that make development a nightmare — or you simply like pointless busywork, I suppose. But, yes, you may still have to consult a human if there is reason to believe the spec isn't accurate.
jgalt212 1y ago
> Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. I hear you on this, but you can still use so long as these tests are not comingled with the tests generated by subject-matter experts. I'd treat them almost a fuzzers.
SamuelAdams 1y ago
Ideally the git history provides the “why was this test written”, however if you have one Jira card tied to 500+ AI generated tests, it’s not terribly helpful.
- djeastm 1y ago
  >if you have one Jira card tied to 500+ AI generated tests The dreaded "Added tests" commit...
Pxtl 1y ago
> “why is this broken test here that appears to test a behavior we don’t support” Because somebody complained when that behavior we don't support was broken, so the bug-that-wasn't-really-a-bug was fixed and a test was created to prevent regression. Imho, the mistake was in documentation: the Test should have comments explaining why this test was created. Just as true for tests as for the actual business logic code: The code can only describe the what and the how. It's up to comments to describe the why.
layer8 1y ago
This is why tests need documenting what exactly they intend to test, and why.

mvieira38 1y ago

I have the exact opposite idea. I want the tests to be mine and thoroughly understood, so I am the true arbiter and then I can let the LLM go ham on the code without fear. If the tests are AI made, then I get some anxiety letting agents mess with the rest of the codebase

_alternator_ 1y ago
I think this is exactly the tradeoff (blue team and red team need to be matched in power), except that I’ve seen LLMs literally cheat the tests (eg “match input: TEST_INPUT then return TEST_OUTPUT”) far too many times to be comfortable with letting LLMs be a major blue team player.
- johnisgood 1y ago
  Yeah, they may do that, but people really should read the code an LLM produces. Ugh, makes me furious. No wonder LLMs have a bad rep from such users.
  - otabdeveloper4 1y ago
    > people really should read the code an LLM produces Yeah, but that, like, requires that you know how to code. And wasn't the point of LLMs in the first place to let clueless people make software?
    - johnisgood 1y ago
      I do not know, I would hope not. The bar to entry is already too low. I do not think you will ever be able to get an LLM work flawlessly for people who do not know programming. I know how to code, and I used LLMs before. It seems to be a prerequisite to know how to code if I want useful outputs.

fpoling 1y ago

I tried a LLM to generate tests for Rust code. It was more harmful then useful. Surely there were a lot of tests, but they still miss the key coverage and it was hard to see what was missed due to the amount of generated code. Then to change the code behavior in future would require to fix a lot of tests again versus fixing few lines in manually written tests.

torginus 1y ago

There's a saying that since nobody tests the tests, they must be trivially correct. That's why they came up with the Arrange-Act-Assert pattern. My favorite kind of unit test nowadays is when you store known input-output pairs and validate the code on them. It's easy to test corner cases and see that the output works as desired.

01HNNWZ0MV43FF 1y ago
"Golden snapshot testing"

bravesoul2 1y ago

AI is like a calculatorin this respect. Calculators can do things most humans can't. They make great augmentation devices. AI being a different kind of intelligence is very useful! Everyone is building AI replace human things. But the value is in augmentation.

positron26 1y ago

> prone to cheating or writing non robust code (eg special code paths to pass tests without solving the actual problem). The solution will come from synthetic data training methods that lobotomize part of the weights. It's just cross-validation. A distilled awareness won't maintain knowledge of the cheat paths, exposing them as erroneous. This may a reason why every living thing on Earth that encounters psychoactive drugs seems to enjoy them. Self-deceptive paths depend on consistency whereas truth-preservation of facts grounded in reality will always be re-derived.

theptip 1y ago

I think the more fundamental attribute of interest is how easy it is to verify the work. Much red team work is easily verifiable; either the exploit works or it doesn’t. Whereas more blue-team work is not easily verifiable; it might take judgement to figure out if a feature is promising. LLMs are extremely powerful (and trainable) on tasks with a good oracle.