4 ms·

This exact thing happened to realitychecknetwork hosting about 6 years ago (now rebranded to serverstack and digitalocean). There was 250+ dedicated servers, 2

This exact thing happened to realitychecknetwork hosting about 6 years ago (now rebranded to serverstack and digitalocean). There was 250+ dedicated servers, 2-3 weeks of restoring week-old backups (thankfully they had these weekly intervals kept offline). Mass exodus of clients. "Ex-employee" used root keys and a boot zerofill drop and rebooted every server resulting in severe data loss. Their online backup systems were also using these keys and we're not spared. They said they would have to shut down the company as a result, but ended up securing capital and eventually launching what would become digitalocean. They said it was highly probable that it was an ex employee and that the FBI was investigating buy nothing was released about it. Good cautionary tale for segregation of credentials and proper user key management.

mintplant 9y ago
Looks like DO has managed to keep this detail out of the record for the most part, save some forum posts that pop up if you search "realitychecknetwork digitalocean". It would explain the odd gap in time in their Wikipedia page history: > In 2003, Ben and Moisey Uretsky who had founded ServerStack, a managed hosting business, wanted to create a new product which would combine the web hosting and virtual servers. The Uretskys, having surveyed the cloud hosting market felt that most hosting companies were targeting enterprise client leaving the entrepreneurial software developers market underserved. In 2011 the Uretskys founded DigitalOcean, a company which would provide server provisioning and cloud hosting for software developers.
- tgtweak 9y ago
  Yeah Ben and Moisey are both incredibly competent sysadmins and shrewd businessmen. They deserve the success they've had but they were very close to desperation after this incident. I can't blame them for omitting it from the records, the reality is likely that digital ocean and serverstack are both much more secure as a result.
FussyZeus 9y ago
Problem is at some point some person or more likely group of people has to be entrusted with the Master Keys. Using different keys for production machines and that sort of thing is all well and good but they're kept somewhere, meaning if an admin decides to screw you, there isn't a lot you can do. Seems like the better option is keep your admins happy as much as possible.
- pvg 9y ago
  By that logic, all employees should be kept as happy as possible because they might commit arson.
  - DashRattlesnake 9y ago
    > By that logic, all employees should be kept as happy as possible because they might commit arson. If that's the only logic that successfully gets through to the boss, it's good logic.
  - deleted 9y ago
    [deleted]
  - defined 9y ago
    All employees should be kept as happy as possible anyway, shouldn't they? Why would anyone want unhappy employees?
    - pvg 9y ago
      The point is, it's not a trade-off between employees' job satisfaction and 'committing a crime'. These are not two sides of the same or even related issues. You don't keep employees happy because they might burn the place down if you don't and and someone's departure from a job is not a reason for them to commit a crime. Equating these things is silly.
  - chiph 9y ago
    I need to watch Office Space again...

mintplant 9y ago

Looks like DO has managed to keep this detail out of the record for the most part, save some forum posts that pop up if you search "realitychecknetwork digitalocean". It would explain the odd gap in time in their Wikipedia page history: > In 2003, Ben and Moisey Uretsky who had founded ServerStack, a managed hosting business, wanted to create a new product which would combine the web hosting and virtual servers. The Uretskys, having surveyed the cloud hosting market felt that most hosting companies were targeting enterprise client leaving the entrepreneurial software developers market underserved. In 2011 the Uretskys founded DigitalOcean, a company which would provide server provisioning and cloud hosting for software developers.

tgtweak 9y ago
Yeah Ben and Moisey are both incredibly competent sysadmins and shrewd businessmen. They deserve the success they've had but they were very close to desperation after this incident. I can't blame them for omitting it from the records, the reality is likely that digital ocean and serverstack are both much more secure as a result.

FussyZeus 9y ago

Problem is at some point some person or more likely group of people has to be entrusted with the Master Keys. Using different keys for production machines and that sort of thing is all well and good but they're kept somewhere, meaning if an admin decides to screw you, there isn't a lot you can do. Seems like the better option is keep your admins happy as much as possible.

pvg 9y ago
By that logic, all employees should be kept as happy as possible because they might commit arson.
- DashRattlesnake 9y ago
  > By that logic, all employees should be kept as happy as possible because they might commit arson. If that's the only logic that successfully gets through to the boss, it's good logic.
- deleted 9y ago
  [deleted]
- defined 9y ago
  All employees should be kept as happy as possible anyway, shouldn't they? Why would anyone want unhappy employees?
  - pvg 9y ago
    The point is, it's not a trade-off between employees' job satisfaction and 'committing a crime'. These are not two sides of the same or even related issues. You don't keep employees happy because they might burn the place down if you don't and and someone's departure from a job is not a reason for them to commit a crime. Equating these things is silly.
- chiph 9y ago
  I need to watch Office Space again...