2 ms·
There is also the "sponge" tool from Joey Hess' moreutils. > You see, sort is too smart to read the entire file into memory, sort it, then spit it out on stdou
by jstimpfle 10y ago
There is also the "sponge" tool from Joey Hess' moreutils.
> You see, sort is too smart to read the entire file into memory, sort it, then spit it out on stdout, which is how it can manage sorting huge amounts of data without completely trashing your system memory.
This is not why the file is empty. sort doesn't need the original file for very long. Very likely it starts a merge sort by sorting smaller chunks of the input file to temporary files. As soon as all these initial-size chunks have been read, the input file is no longer needed.
The reason for the file being truncated is the semantics of I/O redirection:
anyprogram somefile > somefile
means that the file is truncated by the shell (because of the redirection) before "anyprogram" even has the chance to open that file.
- ComputerGuru 10y agoYou are right and I was wrong. I do however routinely run into cases where some chain | tee inputfile.txt > /dev/null also results in lost data. I will try to find it.
- jstimpfle 10y agoThat's a similar case, with the difference that there is a race condition here. "some chain" and "tee" are started in parallel, and it's a random process who comes first: "some chain" for opening inputfile for reading, or "tee" for opening (a new file, but with the same name) it for writing. You can provoke it by echo a > a (sleep 1; cat a) | tee a very likely "a" is empty now.
- ComputerGuru 10y agoThat makes sense. Thank you. `rewrite` should work in those cases.