4 ms·
This is a practical issue that can cause real problems. I was looking after an AWS-hosted web application where an unimportant plugin was stuck in a crash-resta
by jiggawatts 5y ago
This is a practical issue that can cause real problems. I was looking after an AWS-hosted web application where an unimportant plugin was stuck in a crash-restart loop for weeks. It had created something like tens of millions of tiny log files in the same directory over that period.
It turns out that cleaning this up is decidedly non-trivial. Worse still, this was "managed" cloud storage with strict IOPS limits. The "one op per file delete" model seems logical until you're staring down the barrel of multi-week completion times.
I tried various things, but in the end the only practical solution was to copy the valid data across to a new volume and drop the old one. There was almost no reasonable way to delete that many files. As in this article, even the most basic command line tools like "ls" were useless!
You know you're in a bad spot when you're using "kill" on "ls" multiple times per hour...
- southerntofu 5y ago> Worse still, this was "managed" cloud storage with strict IOPS limits. It almost sounds like the cloud was someone else's computer all along, and we don't have any control over it. Thanks for sharing this anecdote!