4 ms·

Moving files from one directory to another directory on the same filesystem doesn't actually result in any data getting moved, only the directory entries are ch

by shawnz 2y ago

Moving files from one directory to another directory on the same filesystem doesn't actually result in any data getting moved, only the directory entries are changed

iwwr 2y ago
In some restricted environments (e.g. containers in CI, build servers or AWS Lambda) you may have limited disk space or no write access at all, so it is useful to have a tool that can stream to memory while decompressing on the fly, and discarding what you don't need.
maccard 2y ago
You’ve made a big assumption about being on the same file system there. We had a CI job that was spending an awful lot of time in fetching cached artifacts. We investigated and it turns out that CI was pulling to %TMP%, but we were actual executing the builds on a separate EBS volume meaning we downloaded and wrote to an EBS volume, then had to copy it to another remote volume, it was super easy to spot on windows, but I’m not sure I would have twigged it as easily if we were moving from /tmp to /data/dira
- shawnz 2y ago
  Sure, but you're probably not using PeaZip (which is primarily a GUI front-end for existing compression tools) in CI or containerized environments. For those situations there are specialized tools which are more appropriate.
  - maccard 2y ago
    I’m not in this case, but if something writes to /tmp and then copies it to /data, there is no guarantee that they are on the same device or even network.
bravetraveler 2y ago
Let's start slicing the pie, everyone. On Linux this is becoming a rarity; '/tmp' (the usual $TMPDIR) is mounted using tmpfs, a distinct memory-backed filesystem. Windows on the other hand is happy for this to reside on your physical disk, persistently. I think - this isn't my area. This almost guarantees your "yes, but" doesn't apply for anyone using Fedora Linux, for example. I dare say nearly any distribution that includes systemd, and probably/consequently, 'tmp.mount'. While it is memory backed, one can only copy/read from this as fast as the destination can write.
- edflsafoiewq 2y ago
  You shouldn't use /tmp for that for that reason. aunpack for example extracts into a temp directory in the same directory as the final destination.
  - bravetraveler 2y ago
    I'm fine with either, I think you and Andrew at the top of the thread agree - PeaZip does as you suggest. I'm just piling on :D Things that naively use temporary directories otherwise may run into an extra copy... because it's not necessarily on the same filesystem. It's rare, in fact, on modern Linux
- dataflow 2y ago
  /tmp is often not tmpfs, in fact. /dev/shm or /var/run are.
  - bravetraveler 2y ago
    Yes, and? It still is very common to find. Turns out things aren't homogeneous. People often both die forever-alone and happily married, too. I already said it's up to the distribution. I wasn't really interested in splitting it more than I have, I hoped to avoid it... but let's go. It's been upstream over a decade https://github.com/systemd/systemd/commits/main/units/tmp.mount https://github.com/systemd/systemd/commits/main/units/tmp.mo... Participation, as always, has been/remains optional. Anything using 'systemd' will likely have this be 'tmpfs', unless 'ConditionPathIsSymbolicLink' is violated. Anything not using 'systemd' is a niche that most of the industry ignores. See how uninsightful/unhelpful these statements are? More specifically, this leaves the one directive/condition to consider... and a subset of a subset that's relevant - systems using runit/s6/whatever. Anyway, that's the fun thing about 'often'. It both often is and isn't. Put simply, word games. We're both right, '/tmp' often is and isn't 'tmpfs'. It's an implementation detail. It's codified into some standards, and it turns out some distributions are more prepared/cognizant than others. The inconsistency in applying this makes the assertion that lead to this whole thread off base. edit: I originally mistook the last edit two years ago for the creation time - adjusted. This has been established for ~14 years!

by shawnz 2y ago

Moving files from one directory to another directory on the same filesystem doesn't actually result in any data getting moved, only the directory entries are changed

iwwr 2y ago

In some restricted environments (e.g. containers in CI, build servers or AWS Lambda) you may have limited disk space or no write access at all, so it is useful to have a tool that can stream to memory while decompressing on the fly, and discarding what you don't need.

maccard 2y ago

You’ve made a big assumption about being on the same file system there. We had a CI job that was spending an awful lot of time in fetching cached artifacts. We investigated and it turns out that CI was pulling to %TMP%, but we were actual executing the builds on a separate EBS volume meaning we downloaded and wrote to an EBS volume, then had to copy it to another remote volume, it was super easy to spot on windows, but I’m not sure I would have twigged it as easily if we were moving from /tmp to /data/dira

shawnz 2y ago
Sure, but you're probably not using PeaZip (which is primarily a GUI front-end for existing compression tools) in CI or containerized environments. For those situations there are specialized tools which are more appropriate.
- maccard 2y ago
  I’m not in this case, but if something writes to /tmp and then copies it to /data, there is no guarantee that they are on the same device or even network.

bravetraveler 2y ago

Let's start slicing the pie, everyone. On Linux this is becoming a rarity; '/tmp' (the usual $TMPDIR) is mounted using tmpfs, a distinct memory-backed filesystem. Windows on the other hand is happy for this to reside on your physical disk, persistently. I think - this isn't my area. This almost guarantees your "yes, but" doesn't apply for anyone using Fedora Linux, for example. I dare say nearly any distribution that includes systemd, and probably/consequently, 'tmp.mount'. While it is memory backed, one can only copy/read from this as fast as the destination can write.

edflsafoiewq 2y ago
You shouldn't use /tmp for that for that reason. aunpack for example extracts into a temp directory in the same directory as the final destination.
- bravetraveler 2y ago
  I'm fine with either, I think you and Andrew at the top of the thread agree - PeaZip does as you suggest. I'm just piling on :D Things that naively use temporary directories otherwise may run into an extra copy... because it's not necessarily on the same filesystem. It's rare, in fact, on modern Linux
dataflow 2y ago
/tmp is often not tmpfs, in fact. /dev/shm or /var/run are.
- bravetraveler 2y ago
  Yes, and? It still is very common to find. Turns out things aren't homogeneous. People often both die forever-alone and happily married, too. I already said it's up to the distribution. I wasn't really interested in splitting it more than I have, I hoped to avoid it... but let's go. It's been upstream over a decade https://github.com/systemd/systemd/commits/main/units/tmp.mount https://github.com/systemd/systemd/commits/main/units/tmp.mo... Participation, as always, has been/remains optional. Anything using 'systemd' will likely have this be 'tmpfs', unless 'ConditionPathIsSymbolicLink' is violated. Anything not using 'systemd' is a niche that most of the industry ignores. See how uninsightful/unhelpful these statements are? More specifically, this leaves the one directive/condition to consider... and a subset of a subset that's relevant - systems using runit/s6/whatever. Anyway, that's the fun thing about 'often'. It both often is and isn't. Put simply, word games. We're both right, '/tmp' often is and isn't 'tmpfs'. It's an implementation detail. It's codified into some standards, and it turns out some distributions are more prepared/cognizant than others. The inconsistency in applying this makes the assertion that lead to this whole thread off base. edit: I originally mistook the last edit two years ago for the creation time - adjusted. This has been established for ~14 years!