4 ms·

DIRECTORY_HAUNTED_DO_NOT_OPEN = true;

2 points by patmcguire 11y ago

greenyoda 11y ago
"Guess what 'rm -rf *' does? That's right, it expands the wildcard and tries to load the gigs and gigs of names of files into memory until it hits swap hell, but recursively." Why not go up to the parent directory and do 'rm -rf dirname'? Then there'd be no wildcard to expand, and 'rm' would just walk the directory removing one file at a time.
- jepler 11y ago
  So I set up a cloud instance (20GB disk, 512MB RAM) and tried this. $ find . Killed $ ls ls: Memory exhausted but indeed, cd'ing up a level and rm-rf'ing the directory full of files seems to succeed--my 'strace rm -rf' is busy spewing unlinkat() syscalls right now.
- dozzie 11y ago
  That's right. And even if we assume `rm' does build those gigabytes, `find | xargs' should not. xargs is smart enough to build commands with chunk of the input and execute them one by one. The job can also be done with one of the several programming languages, Perl oneliner being my choice. Actually, this could be a good interview riddle, along the lines of fixing `chmod a-x /bin/chmod'.
tzs 11y ago
A simple $: ls * would load the millions of unique names into memory hit the memory max and throw the server into swap hell. I'm surprised by that. I'd expect the shell to abort wildcard expansion when the expanded argument list gets bigger than ARG_MAX, since there is no point in continuing to pile on expanded args past that point. It would be interesting to try this with different shells and see if they all behave this way. Guess what $: rm -rf * does? That's right, it expands the wildcard and tries to load the gigs and gigs of names of files into memory until it hits swap hell, but recursively. I do not believe this is correct. It does exactly what "ls *" does with regard to the wildcard. Wildcard expansion takes place in the shell before the command is executed, so the fact that rm is being asked to operate recursively has no bearing on the wildcard expansion.
- dozzie 11y ago
  > I'm surprised by that. In hindsight, you shouldn't. Shell sorts wildcard expansion lexicographically, so it needs all the filenames read at the sorting time.
  - tzs 11y ago
    I remain surprised. If the partially read, unsorted set of filenames exceeds ARG_MAX bytes the shell can infer that exec will fail (E2BIG). There is no need to finish reading the names and sorting them.

2 points by patmcguire 11y ago

"Guess what 'rm -rf *' does? That's right, it expands the wildcard and tries to load the gigs and gigs of names of files into memory until it hits swap hell, but recursively." Why not go up to the parent directory and do 'rm -rf dirname'? Then there'd be no wildcard to expand, and 'rm' would just walk the directory removing one file at a time.

jepler 11y ago
So I set up a cloud instance (20GB disk, 512MB RAM) and tried this. $ find . Killed $ ls ls: Memory exhausted but indeed, cd'ing up a level and rm-rf'ing the directory full of files seems to succeed--my 'strace rm -rf' is busy spewing unlinkat() syscalls right now.
dozzie 11y ago
That's right. And even if we assume `rm' does build those gigabytes, `find | xargs' should not. xargs is smart enough to build commands with chunk of the input and execute them one by one. The job can also be done with one of the several programming languages, Perl oneliner being my choice. Actually, this could be a good interview riddle, along the lines of fixing `chmod a-x /bin/chmod'.

tzs 11y ago

A simple $: ls * would load the millions of unique names into memory hit the memory max and throw the server into swap hell. I'm surprised by that. I'd expect the shell to abort wildcard expansion when the expanded argument list gets bigger than ARG_MAX, since there is no point in continuing to pile on expanded args past that point. It would be interesting to try this with different shells and see if they all behave this way. Guess what $: rm -rf * does? That's right, it expands the wildcard and tries to load the gigs and gigs of names of files into memory until it hits swap hell, but recursively. I do not believe this is correct. It does exactly what "ls *" does with regard to the wildcard. Wildcard expansion takes place in the shell before the command is executed, so the fact that rm is being asked to operate recursively has no bearing on the wildcard expansion.

dozzie 11y ago
> I'm surprised by that. In hindsight, you shouldn't. Shell sorts wildcard expansion lexicographically, so it needs all the filenames read at the sorting time.
- tzs 11y ago
  I remain surprised. If the partially read, unsorted set of filenames exceeds ARG_MAX bytes the shell can infer that exec will fail (E2BIG). There is no need to finish reading the names and sorting them.