4 ms·
What was interesting to me about this was that they had decided not to enforce a deletion policy on /wrk, because they had so much space and the filesystem hadn
by sevensor 11y ago
What was interesting to me about this was that they had decided not to enforce a deletion policy on /wrk, because they had so much space and the filesystem hadn't ever failed. But a rolling deletion policy would have gone a long way to containing the damage by encouraging the users to move their data to a filesystem optimized for reliability instead of availability. Still, I appreciate the heroics involved in restoring the data.
- ople 11y agoAuthor here. We have had an automated deletion policy on our previous filesystems but opted out this time: There are users that have temporary files that they want to persist on the /wrk and we have plenty of capacity. We definitely learned our lesson, though. :)
- Natanael_L 11y agoBetter solution for the future: client side scripts that push those files back after every purge.
- jabl 11y agoThe (inevitable?) consequence of deletion policies that typically delete based on mtime, tend to be users putting touch scripts in cron, making the metadata servers even more of a bottleneck than they already are. Been there, done that. Perhaps the solution is some netflix-like chaos monkey that randomly deletes files..;) Or for each user over its soft quota, delete the oldest files until under the quota. Or something like that..