3 ms·
Agreed. We had a petabyte storage for scratch data on our cluster. We were stuck around 95-100% capacity used for the past month and some change until we recent
by mroche 8y ago
Agreed. We had a petabyte storage for scratch data on our cluster. We were stuck around 95-100% capacity used for the past month and some change until we recently added another half petabyte or so. We were constantly deleting and automating warnings to people that info would be removed and their access restricted if things weren’t removed. We ended up suspending the cluster for a day while we tried to figure out how we were exponentially losing data (one user decided to start uploading terabytes of data to scratch, which is against policy).
No one seemed to understand that if data filled up completely, the entire cluster would go offline because _we wouldn’t have the space for anything_.
- tfigment 8y agoCynically, I think its not that they don't understand its that they don't care and that it is why you are probably there to care for them and solve it. I have similar engineers and its a struggle. Didn't Douglas Adams invent the concept of the SEP Field to explain this (somebody else's problem)?
- gnufx 8y agoYes, it's part of system management to prevent users interfering with one another -- and at least detect problems that are difficult to prevent, like activity crippling a parallel filesystem, and sorting it out.