3 ms·

These sound like servers used by researchers. I've worked with higher education research computing and you might be surprised at what you would find. Researche

These sound like servers used by researchers. I've worked with higher education research computing and you might be surprised at what you would find. Researchers may be generating or churning through countless TB of intermediary data, scratch files, etc. Often, the people who actually run the it infrastructure for researchers are... grad students. Sometimes they have grants for hardware and tight budgets, and paying anything for backups isn't part of it. Sometimes, if you're lucky, the it department will be aware of the work and allowed to help. Now maybe that's not the case here, and there really is one department responsible, and that department decided against spending money on backups. Well maybe they were told by the provost or the dean of whoever that they couldn't afford to back up everything, so they should just stick to file servers. Maybe the boxes compromised here are compute only, and all code and valuable artifacts are expected to be stored safely somewhere else. And maybe the researchers heard this and understood it when they agreed to use the system. But maybe the new grad student didn't get the memo and developed his model in vim on the compute node. The point is, academic computing is kind of the wild west. Weird fiefdoms and weird restrictions, budgetary and otherwise. It's tough to guess which of these scenarios played out from the outside and we really can't know whether the CISO or CTO, or even anybody working for them, dropped the ball.

beh9540 6y ago
I'm so glad I'm not the only one who has experienced this. Only thing I'd add is the lack of understanding about how hard backups are at some of the scales of data in research. Trying to explain to researchers who run the departments that doing a full backup on 35TB+ of data on all budget drives is going to take weeks was something I could never get across, until they lost all their data. And no, snapshots weren't an option on a legacy clustered filesystem that they wouldn't migrate from.
- briffle 6y ago
  But that is only like 3x 12TB USB drives.. </sarcasm>
justinclift 6y ago
> But maybe the new grad student didn't get the memo and developed his model in vim on the compute node. Was trying to think through scenario's like that as well, but they don't really make sense. If various staff members created a few weeks (or even months) worth of work on storage that isn't backed up, then the response from mgmt would generally be "Bad luck, you'll need to redo it". But mgmt decided that work was worth paying US$1.1M+ for, and copping the reputation damage. So, it's a weird mix of potential scenarios to actually get that happening. ;) Maybe someone internal was actually responsible for that malware and is going for some kind of weird payday?
- JeremyNT 6y ago
  > If various staff members created a few weeks (or even months) worth of work on storage that isn't backed up, then the response from mgmt would generally be "Bad luck, you'll need to redo it". Thinking of them as "staff" is the thing. If the random grad student is doing the work the pi really needs done, and deadlines are approaching, then other people's reputations are suddenly on the line. And maybe there's a deadline for a grant that would bring in millions that couldn't be met without that data. Would you pay a million now for much more grant funding? And maybe that ransom money simply comes out of a different financial pot, but the purse strings are loosened for an important faculty member. Maybe the faculty member is hugely important but runs their own shop for vanity reasons, (which the it department really hates, by the way), but when the faculty member gets in trouble he can pull rank and raid the it department budget for the ransom. The incentives and power structures in these organizations are complex. And the egos are huge.
  - justinclift 6y ago
    Good points. :)

beh9540 6y ago

I'm so glad I'm not the only one who has experienced this. Only thing I'd add is the lack of understanding about how hard backups are at some of the scales of data in research. Trying to explain to researchers who run the departments that doing a full backup on 35TB+ of data on all budget drives is going to take weeks was something I could never get across, until they lost all their data. And no, snapshots weren't an option on a legacy clustered filesystem that they wouldn't migrate from.

briffle 6y ago
But that is only like 3x 12TB USB drives.. </sarcasm>

justinclift 6y ago

> But maybe the new grad student didn't get the memo and developed his model in vim on the compute node. Was trying to think through scenario's like that as well, but they don't really make sense. If various staff members created a few weeks (or even months) worth of work on storage that isn't backed up, then the response from mgmt would generally be "Bad luck, you'll need to redo it". But mgmt decided that work was worth paying US$1.1M+ for, and copping the reputation damage. So, it's a weird mix of potential scenarios to actually get that happening. ;) Maybe someone internal was actually responsible for that malware and is going for some kind of weird payday?

JeremyNT 6y ago
> If various staff members created a few weeks (or even months) worth of work on storage that isn't backed up, then the response from mgmt would generally be "Bad luck, you'll need to redo it". Thinking of them as "staff" is the thing. If the random grad student is doing the work the pi really needs done, and deadlines are approaching, then other people's reputations are suddenly on the line. And maybe there's a deadline for a grant that would bring in millions that couldn't be met without that data. Would you pay a million now for much more grant funding? And maybe that ransom money simply comes out of a different financial pot, but the purse strings are loosened for an important faculty member. Maybe the faculty member is hugely important but runs their own shop for vanity reasons, (which the it department really hates, by the way), but when the faculty member gets in trouble he can pull rank and raid the it department budget for the ransom. The incentives and power structures in these organizations are complex. And the egos are huge.
- justinclift 6y ago
  Good points. :)