4 ms·

This is a hacky joke. No sane engineer would ever sign off on this. Even for a 1-5 person team, why would I want a probabilistic selection of test execution? T

by jryio 1y ago

This is a hacky joke. No sane engineer would ever sign off on this. Even for a 1-5 person team, why would I want a probabilistic selection of test execution? The solution to running only e2e tests on affected files has been around long before LLM. This is a bandage on poor CI.

johnfn 1y ago
I have worked at large, competent companies, and the problem of "which e2e tests to execute" is significantly more complicated than you seem to suggest that it is. I've worked with smart engineers that put a lot of time into this problem to only get only middling results.
- Yoric 1y ago
  ...and I'm not confident at all that Claude can do anything at that level.
  - johnfn 1y ago
    How does that reconcile with the article, which states: > Did Claude catch all the edge cases? Yes, and I'm not exaggerating. Claude never missed a relevant E2E test. But it tends to run more tests than needed, which is fine - better safe than sorry. If you have some particular issue with the author's methodology, you should state that.
    - cerved 1y ago
      Well since it never broke for some rando on the internet, surely that means it will always work for everyone
      - johnfn 1y ago
        If you have some particular issue with the article, you should state that. Otherwise, the most charitable interpretation of your position I can come up with is "the article is wrong for some reason I refuse to specify", which doesn't lead to a productive dialogue.
        ambicapter 1y ago
        I think you're the one being uncharitable here. The meaning of what he's saying is very clear. You can't say this probabilistic method (using LLMs to decide your e2e test plan) works if you only have a single example of it working.
        johnfn 1y ago
        It's really not clear. Using probabilistic methods to determine your e2e test plan is already best practice at large tech shops, and to be quite honest the heuristics that they used to use were pretty poor and arbitrary.
        cerved 1y ago
        The author said they used Claude to decide which E2E tests to run and "Claude never missed a relevant E2E test." How many times did they conduct this experiment? Over how long time? How did they determine which tests were relevant and that Claude didn't miss them? Did they try it on more than one project? My point was that none of this tells me this will work in general
    - bgwalter 1y ago
      If the author can keep the whole function code_change -> relevant E2E_TESTS in his head, it seems to be a trivial application. We don't know the methodology, since the author does not state how he verified that function or how he would verify the function for a large code base.
    - troupo 1y ago
      Easy. The article asks us to believe. There's a handy list to check against the article here: https://dmitriid.com/everything-around-llms-is-still-magical-and-wishful-thinking https://dmitriid.com/everything-around-llms-is-still-magical... starting at "For every description of how LLMs work or don't work we know only some, but not all of the following"
      - johnfn 1y ago
        It seems to me like we have the answers to all those questions. - Do we know which projects people work on? It's pretty easy to discover that OP works on https://livox.com.br/en/ https://livox.com.br/en/, a tool that uses AI to let people with disabilities speak. That sounds like a reasonable project to me. - Do we know which codebases (greenfield, mature, proprietary etc.) people work on The e2e tests took 2 hours to run and the website quotes ~40M words. That is not greenfield. - Do we know the level of expertise the people have? It seems like they work on nontrivial production apps. - How much additional work did they have reviewing, fixing, deploying, finishing etc.? The article says very little.
        troupo 1y ago
        > The article says very little. And that's the crux, isn't it. Because that checklist really is just the tip of the iceberg. Some people have completely opposite experiences: https://news.ycombinator.com/item?id=45152139 https://news.ycombinator.com/item?id=45152139 Others question the validity of the approach entirely: https://news.ycombinator.com/item?id=45152668 https://news.ycombinator.com/item?id=45152668 Oh, don't get me wrong: I like the idea. I would trust LLMs with this idea about as far as I could throw them.
trenchpilgrim 1y ago
> The solution to running only e2e tests on affected files has been around long before LLM. This doesn't work in distributed systems, since changing the behavior of one file that's compiled in one binary can cause a downstream issue in a separate binary that sends a network call to the first. e.g. A programmer makes a behavioral change to binary #1 that falls within defined behavior, but encounters Hyrum's Law because of a valid behavior of binary #2.
- hamandcheese 1y ago
  That's easy: - avoid distributed systems at all costs - if you can't avoid them, never make breaking API changes
  - madeofpalk 1y ago
    Determining breaking API changes is the whole point of tests.
  - trenchpilgrim 1y ago
    While we're at it, give things good names and don't invalidate caches at the wrong times!
    - lukan 1y ago
      Also always keep your documentation updated and complete.
jampa 1y ago
I think you might be confusing end-to-end (E2E) tests with other types of testing, such as unit and integration tests. No one is advocating this approach for unit tests, which should still run in their entirety on every pull request. Running all E2E tests in a pipeline isn't feasible due to time constraints (takes hours). Most companies just run these tests nightly (and we still do). Which means we would still catch any issues that slip through the initial screening. But so far, nothing did.
hamandcheese 1y ago
I would sign off on it. The only evidence I would need to see is some analysis of whether the risks are worth the benefits. Risks: missing e2e tests that should have run letting bugs into production, more time spent chasing down flakes due to non determinism. Benefits: increased productivity, catch bugs sooner (since you can run e2e tests more often).

by jryio 1y ago

johnfn 1y ago

I have worked at large, competent companies, and the problem of "which e2e tests to execute" is significantly more complicated than you seem to suggest that it is. I've worked with smart engineers that put a lot of time into this problem to only get only middling results.

Yoric 1y ago
...and I'm not confident at all that Claude can do anything at that level.
- johnfn 1y ago
  How does that reconcile with the article, which states: > Did Claude catch all the edge cases? Yes, and I'm not exaggerating. Claude never missed a relevant E2E test. But it tends to run more tests than needed, which is fine - better safe than sorry. If you have some particular issue with the author's methodology, you should state that.
  - cerved 1y ago
    Well since it never broke for some rando on the internet, surely that means it will always work for everyone
    - johnfn 1y ago
      If you have some particular issue with the article, you should state that. Otherwise, the most charitable interpretation of your position I can come up with is "the article is wrong for some reason I refuse to specify", which doesn't lead to a productive dialogue.
      - ambicapter 1y ago
        I think you're the one being uncharitable here. The meaning of what he's saying is very clear. You can't say this probabilistic method (using LLMs to decide your e2e test plan) works if you only have a single example of it working.
        johnfn 1y ago
        It's really not clear. Using probabilistic methods to determine your e2e test plan is already best practice at large tech shops, and to be quite honest the heuristics that they used to use were pretty poor and arbitrary.
      - cerved 1y ago
        The author said they used Claude to decide which E2E tests to run and "Claude never missed a relevant E2E test." How many times did they conduct this experiment? Over how long time? How did they determine which tests were relevant and that Claude didn't miss them? Did they try it on more than one project? My point was that none of this tells me this will work in general
  - bgwalter 1y ago
    If the author can keep the whole function code_change -> relevant E2E_TESTS in his head, it seems to be a trivial application. We don't know the methodology, since the author does not state how he verified that function or how he would verify the function for a large code base.
  - troupo 1y ago
    Easy. The article asks us to believe. There's a handy list to check against the article here: https://dmitriid.com/everything-around-llms-is-still-magical-and-wishful-thinking https://dmitriid.com/everything-around-llms-is-still-magical... starting at "For every description of how LLMs work or don't work we know only some, but not all of the following"
    - johnfn 1y ago
      It seems to me like we have the answers to all those questions. - Do we know which projects people work on? It's pretty easy to discover that OP works on https://livox.com.br/en/ https://livox.com.br/en/, a tool that uses AI to let people with disabilities speak. That sounds like a reasonable project to me. - Do we know which codebases (greenfield, mature, proprietary etc.) people work on The e2e tests took 2 hours to run and the website quotes ~40M words. That is not greenfield. - Do we know the level of expertise the people have? It seems like they work on nontrivial production apps. - How much additional work did they have reviewing, fixing, deploying, finishing etc.? The article says very little.
      - troupo 1y ago
        > The article says very little. And that's the crux, isn't it. Because that checklist really is just the tip of the iceberg. Some people have completely opposite experiences: https://news.ycombinator.com/item?id=45152139 https://news.ycombinator.com/item?id=45152139 Others question the validity of the approach entirely: https://news.ycombinator.com/item?id=45152668 https://news.ycombinator.com/item?id=45152668 Oh, don't get me wrong: I like the idea. I would trust LLMs with this idea about as far as I could throw them.

trenchpilgrim 1y ago

> The solution to running only e2e tests on affected files has been around long before LLM. This doesn't work in distributed systems, since changing the behavior of one file that's compiled in one binary can cause a downstream issue in a separate binary that sends a network call to the first. e.g. A programmer makes a behavioral change to binary #1 that falls within defined behavior, but encounters Hyrum's Law because of a valid behavior of binary #2.

hamandcheese 1y ago
That's easy: - avoid distributed systems at all costs - if you can't avoid them, never make breaking API changes
- madeofpalk 1y ago
  Determining breaking API changes is the whole point of tests.
- trenchpilgrim 1y ago
  While we're at it, give things good names and don't invalidate caches at the wrong times!
  - lukan 1y ago
    Also always keep your documentation updated and complete.

jampa 1y ago

I think you might be confusing end-to-end (E2E) tests with other types of testing, such as unit and integration tests. No one is advocating this approach for unit tests, which should still run in their entirety on every pull request. Running all E2E tests in a pipeline isn't feasible due to time constraints (takes hours). Most companies just run these tests nightly (and we still do). Which means we would still catch any issues that slip through the initial screening. But so far, nothing did.

hamandcheese 1y ago

I would sign off on it. The only evidence I would need to see is some analysis of whether the risks are worth the benefits. Risks: missing e2e tests that should have run letting bugs into production, more time spent chasing down flakes due to non determinism. Benefits: increased productivity, catch bugs sooner (since you can run e2e tests more often).