51 ms·

It's not super relevant, but I was poking around their CI infrastructure, and I noted the use of a temporary RAM disk to speed-up tests: https://github.com/pyp

It's not super relevant, but I was poking around their CI infrastructure, and I noted the use of a temporary RAM disk to speed-up tests: https://github.com/pypa/pip/blob/master/.azure-pipelines/scripts/New-RAMDisk.ps1 https://github.com/pypa/pip/blob/master/.azure-pipelines/scr... I'm very surprised. Is this common?

dijit 6y ago
Yes, it has secondary benefits like not burning out SSDs and wearing out HDDs. The drawbacks are that memory is kinda expensive in high amounts, and compilers love memory.
- acdha 6y ago
  These days, “high amounts” is a lot more than most projects need - even ultra-portables have 8GB or more. Most projects don’t have unit test suites limited on file I/O measured in gigabytes unless they also have a budget.
detaro 6y ago
It's a fairly straight-forward optimization if disk is a relevant part of your run time and you don't need terribly much of it. Probably not exactly common, because many people don't bother, but also not extraordinary.
- sethhochberg 6y ago
  I'd bet its becoming increasingly popular/convenient to do so with the growth of container-based test suites - really trivial to include a "postgres-ram" image for your test stack instead of one that uses disk, and you don't even need to know what tmpfs is to do so.
  - pletnes 6y ago
    Maybe even more so on cloud instances?
  - detaro 6y ago
    If the tooling you use lets you create tmpfs mounts for your container its very easy, yes. AFAIK those are RAM-backed by default. I think from inside a container it could be difficult to have the permissions necessary.
jackric 6y ago
Another cool trick - use ephemeral memory SQLite database during unit tests: conn = sqlite3.connect(":memory:") But also run against your prod-like database (Postgres/MySQL etc.) later in the CI pipe to catch idiosyncratic database problems.
- aitchnyu 6y ago
  For Postgres, there is also disabling fsync delays (20x speed claimed) and even running it all in memory. https://dev.to/thejessleigh/speed-up-your-postgresql-unit-tests-with-one-weird-trick-364p https://dev.to/thejessleigh/speed-up-your-postgresql-unit-te...
diggan 6y ago
Yeah, I've seen that pattern around and also used it myself. Useful if you're using the disk a lot, for any purpose. Let's say you have 200 JS libraries that you need to install dependencies, build and test for each commit. Making the JS dependencies install into a common cache directory into /dev/shm will easily speed up the install phase (and make the projects share dependencies cache fairly easily), and you could also make the test data (like fixtures) go through /dev/shm if you need it to.
davidbrennerjr 6y ago
Yikes! I'm surprised by the number of people who think temp RAM disk for QA testing came from using containers or the cloud. It was actually in use prior to that technology. And, it was popularized in the use of remote PXE installation which required modifications to the OS after the fact.
- detaro 6y ago
  Where do you see anybody thinking that "came from using containers or the cloud"?
  - davidbrennerjr 6y ago
    My contribution is correct. I'm not a parrot. Temp RAM disk started in Linux 2.3 (possibly older than that), it didn't come from builds inside containers or instances in the cloud.
mixmastamyk 6y ago
Was very common in the spinning rust days to speed up io bound ops, and mentioned in yesterday’s Amiga thread as built in. Could still be useful if tests use a ton of disk.
aeyes 6y ago
Pretty common if you run a lot of builds, have a big test suite or pay per minute. In my environment unpacking a 200MB .tar.gz containing node_modules from previous runs takes half the time on RAM disk vs. SSD local storage. Python doesn't gain as much, packages tend to have fewer small files but there still is a small gain. How much tests benefit from using a RAM disk depends heavily on what they do, I have seen 10% for React apps. We also use other simple tricks like running databases with disabled fsync, mv instead of rm, cache whatever can be cached between CI runs, shallow checkouts and so on. Nothing ground breaking but it all adds up. However, for pipelines which run for five minutes twice a day I don't even bother.

dijit 6y ago

Yes, it has secondary benefits like not burning out SSDs and wearing out HDDs. The drawbacks are that memory is kinda expensive in high amounts, and compilers love memory.

acdha 6y ago
These days, “high amounts” is a lot more than most projects need - even ultra-portables have 8GB or more. Most projects don’t have unit test suites limited on file I/O measured in gigabytes unless they also have a budget.

detaro 6y ago

It's a fairly straight-forward optimization if disk is a relevant part of your run time and you don't need terribly much of it. Probably not exactly common, because many people don't bother, but also not extraordinary.

sethhochberg 6y ago
I'd bet its becoming increasingly popular/convenient to do so with the growth of container-based test suites - really trivial to include a "postgres-ram" image for your test stack instead of one that uses disk, and you don't even need to know what tmpfs is to do so.
- pletnes 6y ago
  Maybe even more so on cloud instances?
- detaro 6y ago
  If the tooling you use lets you create tmpfs mounts for your container its very easy, yes. AFAIK those are RAM-backed by default. I think from inside a container it could be difficult to have the permissions necessary.

jackric 6y ago

Another cool trick - use ephemeral memory SQLite database during unit tests: conn = sqlite3.connect(":memory:") But also run against your prod-like database (Postgres/MySQL etc.) later in the CI pipe to catch idiosyncratic database problems.

aitchnyu 6y ago
For Postgres, there is also disabling fsync delays (20x speed claimed) and even running it all in memory. https://dev.to/thejessleigh/speed-up-your-postgresql-unit-tests-with-one-weird-trick-364p https://dev.to/thejessleigh/speed-up-your-postgresql-unit-te...

diggan 6y ago

Yeah, I've seen that pattern around and also used it myself. Useful if you're using the disk a lot, for any purpose. Let's say you have 200 JS libraries that you need to install dependencies, build and test for each commit. Making the JS dependencies install into a common cache directory into /dev/shm will easily speed up the install phase (and make the projects share dependencies cache fairly easily), and you could also make the test data (like fixtures) go through /dev/shm if you need it to.

davidbrennerjr 6y ago

Yikes! I'm surprised by the number of people who think temp RAM disk for QA testing came from using containers or the cloud. It was actually in use prior to that technology. And, it was popularized in the use of remote PXE installation which required modifications to the OS after the fact.

detaro 6y ago
Where do you see anybody thinking that "came from using containers or the cloud"?
- davidbrennerjr 6y ago
  My contribution is correct. I'm not a parrot. Temp RAM disk started in Linux 2.3 (possibly older than that), it didn't come from builds inside containers or instances in the cloud.

mixmastamyk 6y ago

Was very common in the spinning rust days to speed up io bound ops, and mentioned in yesterday’s Amiga thread as built in. Could still be useful if tests use a ton of disk.

aeyes 6y ago

Pretty common if you run a lot of builds, have a big test suite or pay per minute. In my environment unpacking a 200MB .tar.gz containing node_modules from previous runs takes half the time on RAM disk vs. SSD local storage. Python doesn't gain as much, packages tend to have fewer small files but there still is a small gain. How much tests benefit from using a RAM disk depends heavily on what they do, I have seen 10% for React apps. We also use other simple tricks like running databases with disabled fsync, mv instead of rm, cache whatever can be cached between CI runs, shallow checkouts and so on. Nothing ground breaking but it all adds up. However, for pipelines which run for five minutes twice a day I don't even bother.