5 ms·

The warning about RAID is well-founded, but I'm finding it hard to imagine what sort of backup plan could defeat deliberate sabotage by a former insider. I kee

by dreish 18y ago

The warning about RAID is well-founded, but I'm finding it hard to imagine what sort of backup plan could defeat deliberate sabotage by a former insider. I keep coming up with ideas, and immediately thinking of problems with them. Short of visually examining a substantial number of backed-up files to confirm they contain genuine and correct data ... but then wouldn't it be possible, with enough effort, to create data that looked superficially valid but was in fact, on much closer inspection, mixed up beyond usefulness? RAID isn't a guard against filesystem bugs, carelessness, or extreme physical server damage, but it seems to me this is more of a warning against bad blood.

olefoo 18y ago
If they had been backing their stuff up to AmazonS3, or just dragging a USB disk into the datacenter every month, they would have a copy of the data. And more than likely, if they used the S3 option, they'd have a copy current from the night before. Somebody has to eat the broccoli and do the shit work of making sure that the data is backed up; if you can't do that or you hand off the job to the sort of person who is going to sabotage your company because they don't feel appreciated... you aren't a startup, you're a hobby.
- dreish 18y ago
  The suspected culprit was the chief IT person at the company, and I think we're being led to believe he was both clever and motivated. USB disks would likely contain garbage data N days before the culmination of the sabotage, where N is whatever it takes to make sure the full rotation schedule gets ruined. On the other hand, it probably isn't a coincidence that this happened to someone who had no backups at all.
  - cconstantine 18y ago
    At a high level the way to defeat this kind of attack is to require more people to be involved. Large conspiracies most often fail. In this case that could mean having 3 USB disks, each with their own 'owner'. The owner of each disk would be responsible for making the backups and storing them in a place only they can access. This is also lesson on how to let someone go. They say it's possible that a former employee may have done the deed. When letting someone go on bad (and even good) terms you need to have IT disable all of their accounts as soon as possible, preferably while they are in-process of being fired.
scott_s 18y ago
Sure, it's possible. No backup plan is fool-proof. But if you set the hurdle high, you can make sure only a determined fool will exploit it.
iigs 18y ago
Of course if you can't trust the (creator/maintainer of the) code on a machine there's nothing you can in code on the machine to make it secure. That said, if this kind of problem is a valid concern for an installation, the simplest solution is external validation of backups. For example, one might hire someone not in control of the original system (and not in contact with them) to do restorations and provide certification that they worked according to a test plan. This is the foundation for a lot of Sarbanes-Oxley technical audit consulting. Audited backups would, of course, require a fairly elaborate disaster recovery plan, which sounds out of scope for this company. I don't want to pass judgement, but if a pivotal IT person was let go and sabotage was discovered, at least a basic pass through the systems taking backups doesn't seem out of line.
gaius 18y ago
If you regularly refresh your dev/qa enviroment from production backups, you get the verification "for free".

by dreish 18y ago

olefoo 18y ago

If they had been backing their stuff up to AmazonS3, or just dragging a USB disk into the datacenter every month, they would have a copy of the data. And more than likely, if they used the S3 option, they'd have a copy current from the night before. Somebody has to eat the broccoli and do the shit work of making sure that the data is backed up; if you can't do that or you hand off the job to the sort of person who is going to sabotage your company because they don't feel appreciated... you aren't a startup, you're a hobby.

dreish 18y ago
The suspected culprit was the chief IT person at the company, and I think we're being led to believe he was both clever and motivated. USB disks would likely contain garbage data N days before the culmination of the sabotage, where N is whatever it takes to make sure the full rotation schedule gets ruined. On the other hand, it probably isn't a coincidence that this happened to someone who had no backups at all.
- cconstantine 18y ago
  At a high level the way to defeat this kind of attack is to require more people to be involved. Large conspiracies most often fail. In this case that could mean having 3 USB disks, each with their own 'owner'. The owner of each disk would be responsible for making the backups and storing them in a place only they can access. This is also lesson on how to let someone go. They say it's possible that a former employee may have done the deed. When letting someone go on bad (and even good) terms you need to have IT disable all of their accounts as soon as possible, preferably while they are in-process of being fired.

scott_s 18y ago

Sure, it's possible. No backup plan is fool-proof. But if you set the hurdle high, you can make sure only a determined fool will exploit it.

iigs 18y ago

Of course if you can't trust the (creator/maintainer of the) code on a machine there's nothing you can in code on the machine to make it secure. That said, if this kind of problem is a valid concern for an installation, the simplest solution is external validation of backups. For example, one might hire someone not in control of the original system (and not in contact with them) to do restorations and provide certification that they worked according to a test plan. This is the foundation for a lot of Sarbanes-Oxley technical audit consulting. Audited backups would, of course, require a fairly elaborate disaster recovery plan, which sounds out of scope for this company. I don't want to pass judgement, but if a pivotal IT person was let go and sabotage was discovered, at least a basic pass through the systems taking backups doesn't seem out of line.

gaius 18y ago

If you regularly refresh your dev/qa enviroment from production backups, you get the verification "for free".