9 ms·

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually the one thing they should have backed up because all of the smaller files can be regenerated from the raw ones. I would not be surprised if all of this was running under someone’s desk.

ibeckermayer 6y ago
What’s the difference between replication and backups? Is the distinction that backups must be stored on separate infrastructure, whereas replication might still be 1 or 2 points of failure?
- lukevp 6y ago
  Replication is about having data level redundancy to protect from drive failure. Backups are about having point in time snapshots of the system state, and about having them tiered from a location perspective. The 3-2-1[1] principle says to have 3 total copies, 2 of which are local but on different devices, and 1 of which is offsite. This gives you tiers of recoverability. It’s important from a backup perspective that it’s point in time as well, otherwise as soon as you get ransomware that encrypts your file you now have replicated those changes everywhere. [1] https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
  - mumblemumble 6y ago
    I would add to that, that the offsite one should also be offline.
    - ashtonkem 6y ago
      Or at least in an append only data store.
      - mumblemumble 6y ago
        Though, tread carefully there. If the thing that makes it append only is that you're just not using its destructive update features, then the store isn't really append only. Just because everyone agreed to limit the ways they interact with the data store doesn't mean you can trust buggy code, sloppy programming, and attackers to honor that agreement.
    - skoskie 6y ago
      That’s a lot harder to pull off. What methods do you use to accomplish this?
      - randoramax 6y ago
        S3 bucket with object lock is as offline and convenient as it gets
        reaperducer 6y ago
        I don't know much about S3, but isn't S3, by definition, online, not offline?
        zxcvbn4038 6y ago
        I’ve always wondered if Amazon backs up S3. I don’t think they explicitly say but I get the impression that it is the user’s responsibility to replicate to a second region to guard against data loss so I am guessing not. Object Lock wouldn’t protect against an S3 failure.
        count 6y ago
        They say it's designed to provide significant durability : https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurabili... (99.999999999% durable over a year per object, and able to sustain data loss in 2 facilities). Given the peeks that amazon has provided into the scale of S3, I don't know if you CAN 'back it up'.
- deleted 6y ago
  [deleted]
- geerlingguy 6y ago
  Classically, backups are taken then stored offline (e.g. on archive tape, on optical media, or on drives that are disconnected and put in storage between backups). Otherwise, if they are not 'cold' backups, they are still disconnected so if the main storage blows up, it won't impact the data in the backups. Replication usually involves storage that is powered up and connected somehow to the same systems as the main storage; and any data that's corrupted on the main storage would propagate to the replicated storage.
- kgermino 6y ago
  I think the biggest factor is timing (although backups should also be on a second system). In short, if anything that happens to the files is immediately copied to the "backup" then you don't actually have a chance to recover from any software problems. Whereas, if you make a copy of the data every night and keep the last 30 copies you can find an issue like this and go back in time to retrieve the files from before it started.
- ResidentSleeper 6y ago
  Backups protect against a wider class of problems. For example, a software vulnerability/bug or a human error could result in the deletion of an object in all replicas (because deletions are replicated too), but you'd usually need another incident to occur for you to lose the backup as well.
- throwaway9d0291 6y ago
  Replication is intended to create a 1:1 copy of the data. If data is removed from the primary system, replication will ensure it's removed from the replica. A backup can be a 1:1 copy but it should be set up such that something going wrong with the primary (e.g. cryptolocker malware), can't affect it (since "something went wrong with the primary" is what the backup is intended to resolve). To do that you could take the backup offline or use features like filesystem snapshots to ensure that changes can be rolled back.
- SAI_Peregrinus 6y ago
  If your drive fails (or a single computer fails) replication saves your data, and keeps things running seamlessly. If you `sudo rm -rf / --no-preserve-root` your drive, replication deletes everything, while backups let you restore to the last time you took a backup.
- gameswithgo 6y ago
  A backup would be resistant to someone accidentally deleting all the files, for instance. Replication would not.
- jrochkind1 6y ago
  The problem is that most replication setups will replicate deletes (and other modifications) as well. So if misbehaving software deletes (or corrupts) things, those deletes (or corruptions) will be replicated, and the replication does not give you a way to actually recover from a mistaken delete.
- njharman 6y ago
  Replication includes replicating deletes (and overwrites, and subtilier changes). Replication is READ / WRITE. Backups are WRITE ONCE. They persist if the original is deleted or modified. Separate infrastructure and geographical distribution are orthogonal.
- dec0dedab0de 6y ago
  I see a bunch of explanations, but I don't think any of them really drove home the reason for why you usually need both. Imagine if you had a document that stored useful information. You had this document automatically replicated to another system in a different time zone every time there was a change. You think you're doing great, if there is an outage in the one system you just get your important document from the other system. Then one day someone accidentally copies and pastes the wrong data into the file. Now what do you do? If you goto your replicated copy it also has the bad data. The answer is you should have had backups too. so you could go back an hour,a day or a week or even much longer. Depending on how much data you have, how often it changes, how important it is, and how quickly someone would notice bad data.
chrismorgan 6y ago
The original story spoke of the rumours of malware involvement, but in the update which forms the first half of the linked article, Canon says there was no malware involved. (“Replication isn’t backup” still applies, of course.)
- benatkin 6y ago
  accidental malware (I get that the name implies malicious and deliberate but that never stops definitions from expanding)
t_minus_3 6y ago
replication is not backup and backups need to be replicated too.. and it goes on and on..
crazygringo 6y ago
Do any cloud providers create backups on top of replication though? Backing up databases (terabytes) is feasible, they're not that big. But an entire cloud storage for photo and video for millions, we're talking maybe exabytes. The notion of making "separate" backups seems cost-prohibitive. I am curious, though -- for services like Dropbox or Google Drive, how many replicas are there of your files? I know there must be redundancy in case a disk fails, but do they keep 2 instances of your data or more? And are the instances spread out geographically, some or all?
- returningfory2 6y ago
  > Do any cloud providers create backups on top of replication though? Yes - I work at one of the FAANGs for a team that's doing precisely this. We develops an internal disaster recovery tool that creates backups of data files that can't be touched by the creating application, and that can be read back in a disaster event to recover the data.
  - xyzzy_plugh 6y ago
    Are those backups for you or for your customers? The parent is referring to cloud providers, e.g. AWS S3.
    - returningfory2 6y ago
      both
- sandworm101 6y ago
  >> we're talking maybe exabytes I don't see the size as the big problem. If one cloud can handle the size, the backup storage system can too. For me, the real issue is timing. How do you backup a cloud full of constantly changing data? Do you draw a line in the sand, an image of the cloud state at a particular moment? Is that even possible? You have to do backups of smaller chunks, individual accounts, but eventually that just looks like another software-managed internal array structure rather than a true duplicate. Your backup system is just as susceptible to deletion error as the cloud it lives within. Mandatory xkcd: https://xkcd.com/1737/ https://xkcd.com/1737/
bscphil 6y ago
> It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually the one thing they should have backed up because all of the smaller files can be regenerated from the raw ones. Ironically my experience has been exactly the opposite. It's the demosaic-ed, fully developed copies of my photos that are larger and harder to preserve than the original RAWs. And these files can't be trivially regenerated from the RAWs, either. Let me explain. The most important issue is that the process of taking a raw image and turning it into an edited RGB-pixel image is not obvious, at all. Tons of steps have to happen in this process, and there's currently no way to describe this process in a way that's compatible with a single open standard or even multiple pieces of software. At all. The steps can be broken down roughly into a series of "instructions", but what those instructions mean (e.g. to Lightroom) is entirely opaque and even secret in the case of closed source programs. Even open source programs, like RawTherapee and Darktable, use entirely different and incompatible approaches, algorithms, and instructions. What this means on a practical level is that you can preserve your raw files and the instructions for replicating the edits as carefully as you like, but without the exact same piece of software (often even the same version of the program with the same settings and defaults set), your edits are as good as gone forever. As a result I've had to take fairly drastic steps to make sure my photos are safe. For my older photos, I keep an entire VirtualBox image with Windows and Lightroom backed up along with my raw images, so that I can be sure of restoring exactly the same output files if necessary. (This more-or-less has to be a hacked version of Lightroom because you can't take chances with licensing problems preventing the program from running now that Lightroom is subscription based.) And actually for newer photos, I've moved away from Lightroom to RawTherapee. Even though I feel it's usually inferior, I feel safer since I know the pipeline from raw to burned edit is essentially public. I keep a backed up copy of the RawTherapee source code, but even if that somehow failed someone could make a RawTherapee compatible raw converter from scratch. So that's why edited images are actually harder to preserve: whereas with raw you can just save them in 2-3 different physical locations and storage devices, to keep the edits you have to take several additional steps and there are more points of failure. Why not just keep the static edited images too? Well, I do that for my most essential photos. But that gets into the other issue, that the edited images are actually larger than the original raws. If the point is preserving my edits, not just having a copy that's good enough for Facebook, the images have to be lossless. This pretty much means PNG or TIFF, and in fact the latter seems to be necessary since PNG doesn't handle metadata anywhere nearly as well. Unfortunately, while the compression used on raw files tends to be pretty good (which is further aided by the images being mosaiced), the compression algorithms compatible with TIFF are pretty terrible. Add that to the fact that you almost certainly want 16 bit images, in order to preserve as much raw detail as possible (in case you want high quality prints or need to do further editing), and you end up with whopping huge TIFFs. I regularly see my TIFF output files 4-5 times larger than the corresponding raws. In short, managing backups for a high quality photography workflow is actually a good bit more difficult than it seems at first sight - and how it looks at first is not that easy either!
_nato_ 6y ago
There was an event when a startup I was at asked Basho (they were the company behind Riak db) about backing up our data. Backing was a little side-feature that was possible to rig up, but I recall they looked at this inquiry as if I had two heads -- as if to say, it's replicated, why jump the shark? There was a bug with one of the Riak releases, and all the data was lost. (When we scaled up with this buggy Riak release, the empty node assumed master roll, and all the child nodes went, ah... the new state has no data, let's all delete records 0..k. Fun times.)
- pravda 6y ago
  BTsync did that to me! One computer had a hard drive failure, BTSync deleted all the files from the computer that didn't. Doubleplus ungood.
  - nix23 6y ago
    Thats why syncronisation/replication is NOT a Backup.
danudey 6y ago
Not to mention no delayed deletion. Ideally, a "delete" should mark images as unavailable and queue them for deletion at a later date (e.g. in 30 days). This provides protection against accidental deletion by users, accidental account deletions/deactivations, paid accounts terminated due to lack of payment, automated software mistakes (such as this), and so on. The last company I worked at, 80% of our database was inactive, useless, or redundant data, all kept to protect against all kinds of issues (such as reused usernames, subpoenas, annual reporting, etc). We could always zero out PII from an account, but we never removed a record we didn't have to. It led to a lot more RAID migrations than I would have liked to have done, but it certainly made the application easier to manage. Plus, we never worried about fragmentation or holes in our data files, cascading deletes, etc.

ibeckermayer 6y ago

What’s the difference between replication and backups? Is the distinction that backups must be stored on separate infrastructure, whereas replication might still be 1 or 2 points of failure?

lukevp 6y ago
Replication is about having data level redundancy to protect from drive failure. Backups are about having point in time snapshots of the system state, and about having them tiered from a location perspective. The 3-2-1[1] principle says to have 3 total copies, 2 of which are local but on different devices, and 1 of which is offsite. This gives you tiers of recoverability. It’s important from a backup perspective that it’s point in time as well, otherwise as soon as you get ransomware that encrypts your file you now have replicated those changes everywhere. [1] https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
- mumblemumble 6y ago
  I would add to that, that the offsite one should also be offline.
  - ashtonkem 6y ago
    Or at least in an append only data store.
    - mumblemumble 6y ago
      Though, tread carefully there. If the thing that makes it append only is that you're just not using its destructive update features, then the store isn't really append only. Just because everyone agreed to limit the ways they interact with the data store doesn't mean you can trust buggy code, sloppy programming, and attackers to honor that agreement.
  - skoskie 6y ago
    That’s a lot harder to pull off. What methods do you use to accomplish this?
    - randoramax 6y ago
      S3 bucket with object lock is as offline and convenient as it gets
      - reaperducer 6y ago
        I don't know much about S3, but isn't S3, by definition, online, not offline?
        zxcvbn4038 6y ago
        I’ve always wondered if Amazon backs up S3. I don’t think they explicitly say but I get the impression that it is the user’s responsibility to replicate to a second region to guard against data loss so I am guessing not. Object Lock wouldn’t protect against an S3 failure.
        count 6y ago
        They say it's designed to provide significant durability : https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurabili... (99.999999999% durable over a year per object, and able to sustain data loss in 2 facilities). Given the peeks that amazon has provided into the scale of S3, I don't know if you CAN 'back it up'.
deleted 6y ago
[deleted]
geerlingguy 6y ago
Classically, backups are taken then stored offline (e.g. on archive tape, on optical media, or on drives that are disconnected and put in storage between backups). Otherwise, if they are not 'cold' backups, they are still disconnected so if the main storage blows up, it won't impact the data in the backups. Replication usually involves storage that is powered up and connected somehow to the same systems as the main storage; and any data that's corrupted on the main storage would propagate to the replicated storage.
kgermino 6y ago
I think the biggest factor is timing (although backups should also be on a second system). In short, if anything that happens to the files is immediately copied to the "backup" then you don't actually have a chance to recover from any software problems. Whereas, if you make a copy of the data every night and keep the last 30 copies you can find an issue like this and go back in time to retrieve the files from before it started.
ResidentSleeper 6y ago
Backups protect against a wider class of problems. For example, a software vulnerability/bug or a human error could result in the deletion of an object in all replicas (because deletions are replicated too), but you'd usually need another incident to occur for you to lose the backup as well.
throwaway9d0291 6y ago
Replication is intended to create a 1:1 copy of the data. If data is removed from the primary system, replication will ensure it's removed from the replica. A backup can be a 1:1 copy but it should be set up such that something going wrong with the primary (e.g. cryptolocker malware), can't affect it (since "something went wrong with the primary" is what the backup is intended to resolve). To do that you could take the backup offline or use features like filesystem snapshots to ensure that changes can be rolled back.
SAI_Peregrinus 6y ago
If your drive fails (or a single computer fails) replication saves your data, and keeps things running seamlessly. If you `sudo rm -rf / --no-preserve-root` your drive, replication deletes everything, while backups let you restore to the last time you took a backup.
gameswithgo 6y ago
A backup would be resistant to someone accidentally deleting all the files, for instance. Replication would not.
jrochkind1 6y ago
The problem is that most replication setups will replicate deletes (and other modifications) as well. So if misbehaving software deletes (or corrupts) things, those deletes (or corruptions) will be replicated, and the replication does not give you a way to actually recover from a mistaken delete.
njharman 6y ago
Replication includes replicating deletes (and overwrites, and subtilier changes). Replication is READ / WRITE. Backups are WRITE ONCE. They persist if the original is deleted or modified. Separate infrastructure and geographical distribution are orthogonal.
dec0dedab0de 6y ago
I see a bunch of explanations, but I don't think any of them really drove home the reason for why you usually need both. Imagine if you had a document that stored useful information. You had this document automatically replicated to another system in a different time zone every time there was a change. You think you're doing great, if there is an outage in the one system you just get your important document from the other system. Then one day someone accidentally copies and pastes the wrong data into the file. Now what do you do? If you goto your replicated copy it also has the bad data. The answer is you should have had backups too. so you could go back an hour,a day or a week or even much longer. Depending on how much data you have, how often it changes, how important it is, and how quickly someone would notice bad data.

chrismorgan 6y ago

The original story spoke of the rumours of malware involvement, but in the update which forms the first half of the linked article, Canon says there was no malware involved. (“Replication isn’t backup” still applies, of course.)

benatkin 6y ago
accidental malware (I get that the name implies malicious and deliberate but that never stops definitions from expanding)

t_minus_3 6y ago

replication is not backup and backups need to be replicated too.. and it goes on and on..

crazygringo 6y ago

Do any cloud providers create backups on top of replication though? Backing up databases (terabytes) is feasible, they're not that big. But an entire cloud storage for photo and video for millions, we're talking maybe exabytes. The notion of making "separate" backups seems cost-prohibitive. I am curious, though -- for services like Dropbox or Google Drive, how many replicas are there of your files? I know there must be redundancy in case a disk fails, but do they keep 2 instances of your data or more? And are the instances spread out geographically, some or all?

returningfory2 6y ago
> Do any cloud providers create backups on top of replication though? Yes - I work at one of the FAANGs for a team that's doing precisely this. We develops an internal disaster recovery tool that creates backups of data files that can't be touched by the creating application, and that can be read back in a disaster event to recover the data.
- xyzzy_plugh 6y ago
  Are those backups for you or for your customers? The parent is referring to cloud providers, e.g. AWS S3.
  - returningfory2 6y ago
    both
sandworm101 6y ago
>> we're talking maybe exabytes I don't see the size as the big problem. If one cloud can handle the size, the backup storage system can too. For me, the real issue is timing. How do you backup a cloud full of constantly changing data? Do you draw a line in the sand, an image of the cloud state at a particular moment? Is that even possible? You have to do backups of smaller chunks, individual accounts, but eventually that just looks like another software-managed internal array structure rather than a true duplicate. Your backup system is just as susceptible to deletion error as the cloud it lives within. Mandatory xkcd: https://xkcd.com/1737/ https://xkcd.com/1737/

bscphil 6y ago

> It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually the one thing they should have backed up because all of the smaller files can be regenerated from the raw ones. Ironically my experience has been exactly the opposite. It's the demosaic-ed, fully developed copies of my photos that are larger and harder to preserve than the original RAWs. And these files can't be trivially regenerated from the RAWs, either. Let me explain. The most important issue is that the process of taking a raw image and turning it into an edited RGB-pixel image is not obvious, at all. Tons of steps have to happen in this process, and there's currently no way to describe this process in a way that's compatible with a single open standard or even multiple pieces of software. At all. The steps can be broken down roughly into a series of "instructions", but what those instructions mean (e.g. to Lightroom) is entirely opaque and even secret in the case of closed source programs. Even open source programs, like RawTherapee and Darktable, use entirely different and incompatible approaches, algorithms, and instructions. What this means on a practical level is that you can preserve your raw files and the instructions for replicating the edits as carefully as you like, but without the exact same piece of software (often even the same version of the program with the same settings and defaults set), your edits are as good as gone forever. As a result I've had to take fairly drastic steps to make sure my photos are safe. For my older photos, I keep an entire VirtualBox image with Windows and Lightroom backed up along with my raw images, so that I can be sure of restoring exactly the same output files if necessary. (This more-or-less has to be a hacked version of Lightroom because you can't take chances with licensing problems preventing the program from running now that Lightroom is subscription based.) And actually for newer photos, I've moved away from Lightroom to RawTherapee. Even though I feel it's usually inferior, I feel safer since I know the pipeline from raw to burned edit is essentially public. I keep a backed up copy of the RawTherapee source code, but even if that somehow failed someone could make a RawTherapee compatible raw converter from scratch. So that's why edited images are actually harder to preserve: whereas with raw you can just save them in 2-3 different physical locations and storage devices, to keep the edits you have to take several additional steps and there are more points of failure. Why not just keep the static edited images too? Well, I do that for my most essential photos. But that gets into the other issue, that the edited images are actually larger than the original raws. If the point is preserving my edits, not just having a copy that's good enough for Facebook, the images have to be lossless. This pretty much means PNG or TIFF, and in fact the latter seems to be necessary since PNG doesn't handle metadata anywhere nearly as well. Unfortunately, while the compression used on raw files tends to be pretty good (which is further aided by the images being mosaiced), the compression algorithms compatible with TIFF are pretty terrible. Add that to the fact that you almost certainly want 16 bit images, in order to preserve as much raw detail as possible (in case you want high quality prints or need to do further editing), and you end up with whopping huge TIFFs. I regularly see my TIFF output files 4-5 times larger than the corresponding raws. In short, managing backups for a high quality photography workflow is actually a good bit more difficult than it seems at first sight - and how it looks at first is not that easy either!

_nato_ 6y ago

There was an event when a startup I was at asked Basho (they were the company behind Riak db) about backing up our data. Backing was a little side-feature that was possible to rig up, but I recall they looked at this inquiry as if I had two heads -- as if to say, it's replicated, why jump the shark? There was a bug with one of the Riak releases, and all the data was lost. (When we scaled up with this buggy Riak release, the empty node assumed master roll, and all the child nodes went, ah... the new state has no data, let's all delete records 0..k. Fun times.)

pravda 6y ago
BTsync did that to me! One computer had a hard drive failure, BTSync deleted all the files from the computer that didn't. Doubleplus ungood.
- nix23 6y ago
  Thats why syncronisation/replication is NOT a Backup.

danudey 6y ago

Not to mention no delayed deletion. Ideally, a "delete" should mark images as unavailable and queue them for deletion at a later date (e.g. in 30 days). This provides protection against accidental deletion by users, accidental account deletions/deactivations, paid accounts terminated due to lack of payment, automated software mistakes (such as this), and so on. The last company I worked at, 80% of our database was inactive, useless, or redundant data, all kept to protect against all kinds of issues (such as reused usernames, subpoenas, annual reporting, etc). We could always zero out PII from an account, but we never removed a record we didn't have to. It led to a lot more RAID migrations than I would have liked to have done, but it certainly made the application easier to manage. Plus, we never worried about fragmentation or holes in our data files, cascading deletes, etc.