5 ms·
"The directory is intended for temporary storage of results before staging them into a more permanent location [...] During the three years that the filesystem
by zimpenfish 11y ago
"The directory is intended for temporary storage of results before staging them into a more permanent location [...] During the three years that the filesystem has been in operation, it has accumulated 1.7 Petabytes of data in 850 million objects."
There needs to be some law about how temporary directories always end up containing vitally important data.
- lmkg 11y agoIt's not just files. Anything originally designed as a temporary stop-gap has a habit of becoming permanent. I first learned this about portable classrooms, then laws, then organizational practices. By comparison, the temp directory isn't so surprising.
- Aaargh20318 11y agoThere are few things as permanent as a temporary solution. The problem with a temporary solution is that it makes the problem go away, so suddenly there is no longer any incentive to fix it properly.
- Natanael_L 11y agoHow about this formulation? "Any quick fix that suppress the feeling of emergency will become permanent"
- Natanael_L 11y ago(since I can't update the previous comment anymore) As a corollary, I wrote a few variants of a suggestion for an anti-quick fix permanence policy: A: Ugly problems gets fixes first, so make your quick fixes ugly. B: Every quick fix must be made exponentially uglier than its degree of crappiness. C: Make your fixes transparent; have your fixes let the symptoms shine through from the original problem. And if the problem is silent, have the quick fix generate noise. Definitions: * Ugliness = how bad it looks to everybody involved, including non-technical bosses. * Crappiness = how risky / exploit prone / buggy / inefficient / hard to maintain it is. Justification: to maintain an appropriate degree of a sense of emergency. Two days of unplanned downtime is more than twice as bad as one day (thus the exponent), as costs often grow exponentially. The apparent ugliness of the fix itself should drive everybody to implement a better fix, as they all want to get rid of it right away and NOT put something worse in its place. Physical analogy: a lobby's unstable ceiling held up by a temporary pillar will remain unfixed with the pillar standing if the pillar gets painted, but will be urgently fixed if the pillar stays looking ugly.
- not_kurt_godel 11y agoHere's my solution: Do it right the first time.
- Natanael_L 11y agoAbsolutely, if you have no deadline.
- sevensor 11y agoWhat was interesting to me about this was that they had decided not to enforce a deletion policy on /wrk, because they had so much space and the filesystem hadn't ever failed. But a rolling deletion policy would have gone a long way to containing the damage by encouraging the users to move their data to a filesystem optimized for reliability instead of availability. Still, I appreciate the heroics involved in restoring the data.
- ople 11y agoAuthor here. We have had an automated deletion policy on our previous filesystems but opted out this time: There are users that have temporary files that they want to persist on the /wrk and we have plenty of capacity. We definitely learned our lesson, though. :)
- Natanael_L 11y agoBetter solution for the future: client side scripts that push those files back after every purge.
- jabl 11y agoThe (inevitable?) consequence of deletion policies that typically delete based on mtime, tend to be users putting touch scripts in cron, making the metadata servers even more of a bottleneck than they already are. Been there, done that. Perhaps the solution is some netflix-like chaos monkey that randomly deletes files..;) Or for each user over its soft quota, delete the oldest files until under the quota. Or something like that..
- deleted 11y ago[deleted]