7 ms·

Agree with this so much. One of the main things that I try to do is ensure that the turn around time between testing and writing code is as short as possible.

Agree with this so much. One of the main things that I try to do is ensure that the turn around time between testing and writing code is as short as possible. As an example I inherited a codebase about 2 years ago that would take 15 minutes to run the test suite. It was painful. It was just long enough that it would waste an entire day just trying to implement a simple feature. The project was behind schedule and no one wanted to work on it because it just took too long to do anything. So I spent a few weeks running benchmarks on the tests and figuring out where the slow down was. I did a bunch of different things, reusing SQL tables, only truncating tables ones that were used, refactored how the tests worked, removed some code so that docker was not needed to do the testing, etc. I got those tests running in 3 seconds down from 15 minutes. And that changed the whole velocity of the project. Always sharpen your ax.

switchbak 5y ago
Absolutely. What I find astonishing is how undervalued test feedback time is. I don't know what drives that, but in most jobs I've had to fight to do the right thing. Just recently I've made some optimizations to get a functional test suite from 65 minutes down to 3.6. Parallelism is my favorite lever, as it scales so well, and core count is still going up!
- HPsquared 5y ago
  Probably something to do with that XKCD about waiting for compiling, "The #1 programmer excuse for legitimately slacking off": https://xkcd.com/303/ https://xkcd.com/303/ Running tests is the same type of thing!
faraaz98 5y ago
How much time did removing docker shave off?
- vbezhenar 5y ago
  Docker helped me to significantly increase unit test speed on one project. Each test was recreating database, run dozens of DDL scripts over and over (to ensure clean environment). I reimplemented in in a way that DDL scripts were run once, then container with database was commited to an image and that image was re-used for every test. Also multiple containers were run at once, so tests could be run simultaneously. Docker is a miracle technology sometimes.
  - petters 5y ago
    Normally you'd open a transaction, run the unit test, then roll back the transaction. That makes each test start at the same, clean state.
    - vbezhenar 5y ago
      1. That wouldn't work if code under test uses multiple transactions. 2. Rolling back a transaction is not exactly free. 3. Does not allow running multiple test simultaneously.
      - bpicolo 5y ago
        > 3. Does not allow running multiple test simultaneously. It does with almost every form of transaction isolation mode. Read uncommitted in Postgres still behaves as read committed, so in Postgres you're never missing the mark here. > 1. That wouldn't work if code under test uses multiple transactions. A lot of abstractions interpret nested transactions as save points for this reason. It's not the only valid pattern, but it is a consistently behaving one that's straightforward to implement and gets you most of the UX you want from integration test design.
        vbezhenar 5y ago
        I still think that tests would be flaky without complete isolation in a different database instances. Transaction isolation is nice, but for serial isolation you would get errors when you can't commit that transaction and you'll need to repeat the test (which might cause further issues) and for lesser isolation levels you would get some phantom reads which might manifest as a rare test failures which is a terrible thing, because you can't reproduce it.
idiocrat 5y ago
For some reason not many people like the "maintenance" type of work.
- malux85 5y ago
  Not full time, but documenting interfaces, speeding up tests, generating stubs and harnesses, profiling pain points - all these I find to be meditative tasks, and I do them when I'm not 100%. I can do about 4-5 hours a day of "difficult" programming, but then I can easily do another 4-5 hours of this maintenance type work, and the cumulative effects of repeated maintenance time keeps everything running super smoothly.
  - mellavora 5y ago
    I'd add that the cumulative effects of always knowing that the maintenance backlog is under control and that you always have time for the maintenance makes it much easier to consistently deliver the 4-5 hours of "difficult" programming.
- yupper32 5y ago
  The reason is simple. Maintenance work doesn't progress your career. Why bother?
  - DarylZero 5y ago
    That's not the reason. The work itself is not fun. Even if the whole project has no career purpose, it's more fun to write new code.
    - etripe 5y ago
      Well, if it were fun, it would lose its shine if you're constantly being told it doesn't matter to the business.
    - darkwater 5y ago
      It can be fun, it just depends on who you are. Sadly most developers prefer to follow PM and churn in features instead of this precious work.
gfodor 5y ago
Another much under-utilized way to speed up test suites: delete tests
- wruza 5y ago
  Can one configure tests to run in “full”, “actual” and “scope:<modname>” modes?
- zerocount 5y ago
  This is being down-voted, but there is merit to this. Tests should be audited for continued need.
- xcambar 5y ago
  I understand the underlying message but I prefer the equivalent following phrasing: "check test suite coherence". It moves the deletion as a means towards the coherence rather than as an end in and of itself, and also includes refactoring as an option. I hope it helps the less experienced developers grab the intention more explicitly.
  - gfodor 5y ago
    I think the core point isn't about coherence, but tradeoffs. Some tests are not worth holding onto: ones that are slow, flaky, and test code that is of marginal importance. In those cases, the ideal tradeoff can be to delete the test. Perhaps to rewrite a better one, or perhaps to just 'eat' the risk incurred by removing it in favor of improved iteration times and the other long term benefits of not having to maintain it.
    - xcambar 5y ago
      As always, mentioning that Your Mileage May Vary (YMMV) puts the emphasis/decision on the people facing this or that situation rather than blindly applying advice found on the Internet. And rightfully so :)
- nijave 5y ago
  Also just deleting dead code and tests that go along with it Tend to get a boost in application boot/startup time, too
greggman3 5y ago
A few more things I'd add The OP mentioned using tests instead of the entire game. You could also consider splitting the tests. The code base I'm on now has 9 test suites. It would be much slower if all 9 were merged into 1. So, if you have a monolithic test suite consider breaking it apart. Our team also has a CI/CQ so I never run all the tests myself. I run the tests I think my change affects. When it passes I upload to the CQ and let it run all the tests. Then I go work on something else and check later if it all passd. In games there is often no need to run on the target hardware for 95% of tasks so choose the hardware that's fastest. In other words, if you're making a game that can run on PC, PS5, XBox then develop on PC and only switch to PS5,XBox when you have to (platform specific features, checking perf, etc.). For VR for example I'd test via PC on a link cable or on a Vive/Index/Rift and only test on standalone Quest when I absolutely had to. So much time saved. The same is true in other places. I work a very large project that run on many platforms (Windows, Mac, Linux, Android). Building and testing on Linux is 10x faster than Windows, 40x faster than Mac. Once I found that out I switched to Linux for my day to day work and only pull out the other devices if the stuff I'm working on is platform specific.

switchbak 5y ago

Absolutely. What I find astonishing is how undervalued test feedback time is. I don't know what drives that, but in most jobs I've had to fight to do the right thing. Just recently I've made some optimizations to get a functional test suite from 65 minutes down to 3.6. Parallelism is my favorite lever, as it scales so well, and core count is still going up!

HPsquared 5y ago
Probably something to do with that XKCD about waiting for compiling, "The #1 programmer excuse for legitimately slacking off": https://xkcd.com/303/ https://xkcd.com/303/ Running tests is the same type of thing!

faraaz98 5y ago

How much time did removing docker shave off?

vbezhenar 5y ago
Docker helped me to significantly increase unit test speed on one project. Each test was recreating database, run dozens of DDL scripts over and over (to ensure clean environment). I reimplemented in in a way that DDL scripts were run once, then container with database was commited to an image and that image was re-used for every test. Also multiple containers were run at once, so tests could be run simultaneously. Docker is a miracle technology sometimes.
- petters 5y ago
  Normally you'd open a transaction, run the unit test, then roll back the transaction. That makes each test start at the same, clean state.
  - vbezhenar 5y ago
    1. That wouldn't work if code under test uses multiple transactions. 2. Rolling back a transaction is not exactly free. 3. Does not allow running multiple test simultaneously.
    - bpicolo 5y ago
      > 3. Does not allow running multiple test simultaneously. It does with almost every form of transaction isolation mode. Read uncommitted in Postgres still behaves as read committed, so in Postgres you're never missing the mark here. > 1. That wouldn't work if code under test uses multiple transactions. A lot of abstractions interpret nested transactions as save points for this reason. It's not the only valid pattern, but it is a consistently behaving one that's straightforward to implement and gets you most of the UX you want from integration test design.
      - vbezhenar 5y ago
        I still think that tests would be flaky without complete isolation in a different database instances. Transaction isolation is nice, but for serial isolation you would get errors when you can't commit that transaction and you'll need to repeat the test (which might cause further issues) and for lesser isolation levels you would get some phantom reads which might manifest as a rare test failures which is a terrible thing, because you can't reproduce it.

idiocrat 5y ago

For some reason not many people like the "maintenance" type of work.

malux85 5y ago
Not full time, but documenting interfaces, speeding up tests, generating stubs and harnesses, profiling pain points - all these I find to be meditative tasks, and I do them when I'm not 100%. I can do about 4-5 hours a day of "difficult" programming, but then I can easily do another 4-5 hours of this maintenance type work, and the cumulative effects of repeated maintenance time keeps everything running super smoothly.
- mellavora 5y ago
  I'd add that the cumulative effects of always knowing that the maintenance backlog is under control and that you always have time for the maintenance makes it much easier to consistently deliver the 4-5 hours of "difficult" programming.
yupper32 5y ago
The reason is simple. Maintenance work doesn't progress your career. Why bother?
- DarylZero 5y ago
  That's not the reason. The work itself is not fun. Even if the whole project has no career purpose, it's more fun to write new code.
  - etripe 5y ago
    Well, if it were fun, it would lose its shine if you're constantly being told it doesn't matter to the business.
  - darkwater 5y ago
    It can be fun, it just depends on who you are. Sadly most developers prefer to follow PM and churn in features instead of this precious work.

gfodor 5y ago

Another much under-utilized way to speed up test suites: delete tests

wruza 5y ago
Can one configure tests to run in “full”, “actual” and “scope:<modname>” modes?
zerocount 5y ago
This is being down-voted, but there is merit to this. Tests should be audited for continued need.
xcambar 5y ago
I understand the underlying message but I prefer the equivalent following phrasing: "check test suite coherence". It moves the deletion as a means towards the coherence rather than as an end in and of itself, and also includes refactoring as an option. I hope it helps the less experienced developers grab the intention more explicitly.
- gfodor 5y ago
  I think the core point isn't about coherence, but tradeoffs. Some tests are not worth holding onto: ones that are slow, flaky, and test code that is of marginal importance. In those cases, the ideal tradeoff can be to delete the test. Perhaps to rewrite a better one, or perhaps to just 'eat' the risk incurred by removing it in favor of improved iteration times and the other long term benefits of not having to maintain it.
  - xcambar 5y ago
    As always, mentioning that Your Mileage May Vary (YMMV) puts the emphasis/decision on the people facing this or that situation rather than blindly applying advice found on the Internet. And rightfully so :)
nijave 5y ago
Also just deleting dead code and tests that go along with it Tend to get a boost in application boot/startup time, too

greggman3 5y ago

A few more things I'd add The OP mentioned using tests instead of the entire game. You could also consider splitting the tests. The code base I'm on now has 9 test suites. It would be much slower if all 9 were merged into 1. So, if you have a monolithic test suite consider breaking it apart. Our team also has a CI/CQ so I never run all the tests myself. I run the tests I think my change affects. When it passes I upload to the CQ and let it run all the tests. Then I go work on something else and check later if it all passd. In games there is often no need to run on the target hardware for 95% of tasks so choose the hardware that's fastest. In other words, if you're making a game that can run on PC, PS5, XBox then develop on PC and only switch to PS5,XBox when you have to (platform specific features, checking perf, etc.). For VR for example I'd test via PC on a link cable or on a Vive/Index/Rift and only test on standalone Quest when I absolutely had to. So much time saved. The same is true in other places. I work a very large project that run on many platforms (Windows, Mac, Linux, Android). Building and testing on Linux is 10x faster than Windows, 40x faster than Mac. Once I found that out I switched to Linux for my day to day work and only pull out the other devices if the stuff I'm working on is platform specific.