5 ms·
- Just spin up more test instances. If the AI is as good as people claim then it's still way cheaper than extra programmers. - Write fast code. At $WORK we can
by hansvm 1y ago
- Just spin up more test instances. If the AI is as good as people claim then it's still way cheaper than extra programmers.
- Write fast code. At $WORK we can test roughly a trillion things per CPU physical core year for our primary workload, and that's in a domain where 20 microsecond processing time is unheard of. Orders of magnitude speed improvements pay dividends quickly.
- LLMs don't care hugely about the language. Avoid things like rust where compile times are always a drag.
- That's something of a strange human problem you're describing. Once the PR is reviewed, can't you just hit "auto-merge" and go to the next task, only circling back if the code was broken? Why is that a significant amount of developer time?
- The thing you're observing is something every growing team witnesses. You can get 90% of the way to what you want by giving the build system a greenfield re-write. If you really have to run 100x more tests, it's worth a day or ten sanity checking docker caching or whatever it is your CI/CD is using. Even hermetic builds have inter-run caching in some form; it's just more work to specify how the caches should work. Put your best engineer on the problem. It's important.
- Be as specific as possible in describing test dependencies. The fastest tests are the ones which don't run.
- Separate out unit tests from other forms of tests. It's hard to write software operating with many orders of magnitude of discrepancies, and tests are no exception. Your life is easier if conceptually they have a separate budget (e.g., continuous fuzz testing or load testing or whatever). Unit tests can then easily be fast enough for a developer to run all the changed ones on precommit. Slower tests are run locally when you think they might apply. The net effect is that you don't have the sort of back-and-forth with your CI that actually causes lost developer productivity because the PR shouldn't have a bunch of bullshit that's green locally and failing remotely.
- mike_hearn 1y agoThese are all good suggestions, albeit many are hard to implement in practice. > That's something of a strange human problem you're describing. Are we talking about agent-written changes now, or human? Normally reviewers expect tests to pass before they review something, otherwise the work might change significantly after they did the review in order to fix broken tests. Auto merges can fail due to changes that happened in the meantime, they're aren't auto in many cases. Once latency goes beyond a minute or two people get distracted and start switching tasks to something else, which slows everything down. And yes code review latency is a problem as well, but there are easier fixes for that.