4 ms·

This is one reason why I tell people to have both offsite backups (in case of fire, theft, etc.) and offline backups. There are a lot of us that are just one S

by jewel 13y ago

This is one reason why I tell people to have both offsite backups (in case of fire, theft, etc.) and offline backups. There are a lot of us that are just one SSH worm away from having all of our files destroyed. I've been thinking about ways to create an offline-equivalent backup, so that it can be automated. One way would be to have a computer that is only connected via serial cable, which only accepts new files to be backed up. (No ability to delete via the serial cable.)

nodata 13y ago
You could use rsync.net. If you get rooted, and your files get deleted, you can ask for them to revert to a snapshot.
gatehouse 13y ago
You create a backup system that has credentials into your target system, but the target has no credentials for the backup. The backup system can maintain the point-in-time history. Of course if you backup system gets hit then the whole thing falls apart, but you can isolate this one system and it doesn't need to accept any incoming connections.
- einhverfr 13y ago
  > Of course if you backup system gets hit then the whole thing falls apart, but you can isolate this one system and it doesn't need to accept any incoming connections. Unless you need to remote administer it.... But I suppose that's what virtualization helps with.
- dspillett 13y ago
  Or a step further: don't let either the live and backup sites connect to each other directly, instead use an intermediary site that both connect to (which can not connect back to them).
mjn 13y ago
A pretty low-tech system I saw in use in the '90s, without having to go all the way to serial cables, was just FTP with delete/overwrite disabled, to a machine with no other network services running. You dump a snapshot and FTP it to the backup host over your local ethernet. The FTP server accepts new files, but doesn't allow deletions or modifications of previously uploaded files (lots of FTP servers have this option, in order to be able to create a public "anonymous upload" folder without people overwriting each others' uploads). If you ever need to retrieve the backups, you physically log in on the backup machine and do something from there. This does rely on the FTP implementation being solid (vsftpd seems like a reasonable choice).
- eli 13y ago
  Sounds like a typical tarsnap set up. You can have a key that only appends backups, but can't delete them.
- jewel 13y ago
  That is a good idea. I have considered having a network service like that, but I'm a little too paranoid. I want something that will survive heretofore unseen events, like a 0-day remote root exploit in the Linux kernel networking stack. If such a remote root exploit exists, and the person that discovers it decides to write something like the Code Red worm from 2001, and also decides to have the worm erase all block devices, I'd currently lose every file I have and all of my backups, even though they are geographically dispersed. It's possible that even services like S3 and Dropbox would be wiped in such an event. Once again, I realize I'm unusually paranoid in this regard, but I really don't want to lose my stuff.
  - gaadd33 13y ago
    You could just store backups on tape and ship them to a secure storage facility run by Iron Mountain.
  - tachion 13y ago
    Not everything is running on Linux you know, and therefore not everything is prone to the 0 day remote Linux kernel exploit. Get yourself other backup layers on other operatying systems, like Tarsnap, that runs FreeBSD, and you should get another level of security.
ZoFreX 13y ago
> offline backups If it's online, it isn't a backup, it's redundancy.
- marcosdumay 13y ago
  If it keeps history, it's not just redundancy. And if it can not in any way overwrite history, it's as good as an offline backup. But now we are talking about custom kernels and maybe hardware.
aguki 13y ago
Why not just rsync to a sticky bit enabled directory?
einhverfr 13y ago
> I've been thinking about ways to create an offline-equivalent backup, so that it can be automated. One of the approaches we take with Efficito is to ensure our offsite backups are pulled by the backup server, not pushed there by the backed up server (exception is made for archiving Pg wal segments). A backed-up server has no access to its own offsite, online backups for db base backups, file backups, and the like and only write-once-read-only access to its own WAL segment archives. It's not quite an offline backup (and for our purposes there are real limitations with fully offline backups), but it is about as close as you can get.
dspillett 13y ago
> a lot of us that are just one SSH worm away from having all of our files destroyed. This has happened quite a lot in some high profile cases over recent years, where [Company Name Here] was hacked and as well as taking down their live service the attackers got from there to the backups and killed them too. > I've been thinking about ways to create an offline-equivalent backup, so that it can be automated. The trick I use to improve the safety of my online+offsite backups is to stop any one service being able to connect and authenticate with all the others, disconnecting the online backups from their source as much as possible. My "live" stores connect to an intermediary and push updates, and my online backup stores connect to the intermediary and pull updates (with restores and automated tests being the same thing in reverse). The live stores and the backups can not connect+authenticate to each other directly, and the intermediary can't connect+authenticate to either the live store or the backups. This way if a malicious entity (be it a human or an automated exploit) gains access to a privileged account on any of the three locations it does not automatically gain access to all the others, but the process of taking and testing backups can still be automated. Of course this isn't perfect: it is a little more hassle to setup, needs an extra resource (the intermediate location) which could potentially be a point of failure (though your automated backup tests will hopefully detect problems there before they become serious) and it depends on good credentials management (I sill need to be able to connect to all three for remote admin and if you use the same credentials for them all, and/or store said credentials in the same insecure place, the extra protection is lost) but I find it works well in practise. Also as I'm using the same OS+libs+tools in all three places I'm still susceptable to attacks using "zero day" exploits on some part of that stack, so for that reason (amongst others) this does not remove the need for truly offline backups.
ama729 13y ago
> I've been thinking about ways to create an offline-equivalent backup, so that it can be automated. One way would be to have a computer that is only connected via serial cable, which only accepts new files to be backed up. (No ability to delete via the serial cable.) A lower tech alternative is to use a Linear Tape-Open drive/cartridge. Not only are they archival safe (20 years+), but either you can use a new one every time or I think there is an option somewhere to only write once. https://en.wikipedia.org/wiki/Linear_Tape-Open https://en.wikipedia.org/wiki/Linear_Tape-Open Edit: Indeed, available since LTO3 https://en.wikipedia.org/wiki/Write_Once_Read_Many https://en.wikipedia.org/wiki/Write_Once_Read_Many
- kijin 13y ago
  If the size of your data is less than 4GB, a box of DVD+R's is also a cheap, secure and convenient way to make read-only backups. Just burn a new disc at the end of each workday, or whatever other interval your schedule and budget allows. Of course this isn't really automated, but if you have one of those disc changers that people often use with audio CDs, you might be able to turn that into an automated system with a bit of scripting and hardware hacking.
  - harywilke 13y ago
    DVD's and pretty much all optical media are terrible for backup. they degrade just siting on the shelf. and one scratch and 'poof' goes your data. I'm surprised as a medium that they have lasted this long. [1]"Manufacturers claim that CD-R and DVD-R discs have a shelf life of 5 to 10 years before recording," note that is "before recording" [1] http://www.clir.org/pubs/reports/pub121/sec4.html http://www.clir.org/pubs/reports/pub121/sec4.html
    - kijin 13y ago
      Sure, but shelf life won't be a problem when you burn a new disc every day and you intend to keep at most a couple of months' worth of daily snapshots. As terrible as optical discs are for long-term archiving, I'm sure their shelf life usually exceeds that of a carton of eggs. That should be enough to protect against ransomware and most other kinds of disasters that can strike a personal computer.
cperciva 13y ago
I've been thinking about ways to create an offline-equivalent backup, so that it can be automated. You can do this with Tarsnap. The tarsnap-keymgmt utility allows you to create key file which can only create new archives, not delete or overwrite existing ones.

by jewel 13y ago

nodata 13y ago

You could use rsync.net. If you get rooted, and your files get deleted, you can ask for them to revert to a snapshot.

gatehouse 13y ago

You create a backup system that has credentials into your target system, but the target has no credentials for the backup. The backup system can maintain the point-in-time history. Of course if you backup system gets hit then the whole thing falls apart, but you can isolate this one system and it doesn't need to accept any incoming connections.

einhverfr 13y ago
> Of course if you backup system gets hit then the whole thing falls apart, but you can isolate this one system and it doesn't need to accept any incoming connections. Unless you need to remote administer it.... But I suppose that's what virtualization helps with.
dspillett 13y ago
Or a step further: don't let either the live and backup sites connect to each other directly, instead use an intermediary site that both connect to (which can not connect back to them).

mjn 13y ago

A pretty low-tech system I saw in use in the '90s, without having to go all the way to serial cables, was just FTP with delete/overwrite disabled, to a machine with no other network services running. You dump a snapshot and FTP it to the backup host over your local ethernet. The FTP server accepts new files, but doesn't allow deletions or modifications of previously uploaded files (lots of FTP servers have this option, in order to be able to create a public "anonymous upload" folder without people overwriting each others' uploads). If you ever need to retrieve the backups, you physically log in on the backup machine and do something from there. This does rely on the FTP implementation being solid (vsftpd seems like a reasonable choice).

eli 13y ago
Sounds like a typical tarsnap set up. You can have a key that only appends backups, but can't delete them.
jewel 13y ago
That is a good idea. I have considered having a network service like that, but I'm a little too paranoid. I want something that will survive heretofore unseen events, like a 0-day remote root exploit in the Linux kernel networking stack. If such a remote root exploit exists, and the person that discovers it decides to write something like the Code Red worm from 2001, and also decides to have the worm erase all block devices, I'd currently lose every file I have and all of my backups, even though they are geographically dispersed. It's possible that even services like S3 and Dropbox would be wiped in such an event. Once again, I realize I'm unusually paranoid in this regard, but I really don't want to lose my stuff.
- gaadd33 13y ago
  You could just store backups on tape and ship them to a secure storage facility run by Iron Mountain.
- tachion 13y ago
  Not everything is running on Linux you know, and therefore not everything is prone to the 0 day remote Linux kernel exploit. Get yourself other backup layers on other operatying systems, like Tarsnap, that runs FreeBSD, and you should get another level of security.

ZoFreX 13y ago

> offline backups If it's online, it isn't a backup, it's redundancy.

marcosdumay 13y ago
If it keeps history, it's not just redundancy. And if it can not in any way overwrite history, it's as good as an offline backup. But now we are talking about custom kernels and maybe hardware.

aguki 13y ago

Why not just rsync to a sticky bit enabled directory?

einhverfr 13y ago

> I've been thinking about ways to create an offline-equivalent backup, so that it can be automated. One of the approaches we take with Efficito is to ensure our offsite backups are pulled by the backup server, not pushed there by the backed up server (exception is made for archiving Pg wal segments). A backed-up server has no access to its own offsite, online backups for db base backups, file backups, and the like and only write-once-read-only access to its own WAL segment archives. It's not quite an offline backup (and for our purposes there are real limitations with fully offline backups), but it is about as close as you can get.

dspillett 13y ago

> a lot of us that are just one SSH worm away from having all of our files destroyed. This has happened quite a lot in some high profile cases over recent years, where [Company Name Here] was hacked and as well as taking down their live service the attackers got from there to the backups and killed them too. > I've been thinking about ways to create an offline-equivalent backup, so that it can be automated. The trick I use to improve the safety of my online+offsite backups is to stop any one service being able to connect and authenticate with all the others, disconnecting the online backups from their source as much as possible. My "live" stores connect to an intermediary and push updates, and my online backup stores connect to the intermediary and pull updates (with restores and automated tests being the same thing in reverse). The live stores and the backups can not connect+authenticate to each other directly, and the intermediary can't connect+authenticate to either the live store or the backups. This way if a malicious entity (be it a human or an automated exploit) gains access to a privileged account on any of the three locations it does not automatically gain access to all the others, but the process of taking and testing backups can still be automated. Of course this isn't perfect: it is a little more hassle to setup, needs an extra resource (the intermediate location) which could potentially be a point of failure (though your automated backup tests will hopefully detect problems there before they become serious) and it depends on good credentials management (I sill need to be able to connect to all three for remote admin and if you use the same credentials for them all, and/or store said credentials in the same insecure place, the extra protection is lost) but I find it works well in practise. Also as I'm using the same OS+libs+tools in all three places I'm still susceptable to attacks using "zero day" exploits on some part of that stack, so for that reason (amongst others) this does not remove the need for truly offline backups.

ama729 13y ago

> I've been thinking about ways to create an offline-equivalent backup, so that it can be automated. One way would be to have a computer that is only connected via serial cable, which only accepts new files to be backed up. (No ability to delete via the serial cable.) A lower tech alternative is to use a Linear Tape-Open drive/cartridge. Not only are they archival safe (20 years+), but either you can use a new one every time or I think there is an option somewhere to only write once. https://en.wikipedia.org/wiki/Linear_Tape-Open https://en.wikipedia.org/wiki/Linear_Tape-Open Edit: Indeed, available since LTO3 https://en.wikipedia.org/wiki/Write_Once_Read_Many https://en.wikipedia.org/wiki/Write_Once_Read_Many

kijin 13y ago
If the size of your data is less than 4GB, a box of DVD+R's is also a cheap, secure and convenient way to make read-only backups. Just burn a new disc at the end of each workday, or whatever other interval your schedule and budget allows. Of course this isn't really automated, but if you have one of those disc changers that people often use with audio CDs, you might be able to turn that into an automated system with a bit of scripting and hardware hacking.
- harywilke 13y ago
  DVD's and pretty much all optical media are terrible for backup. they degrade just siting on the shelf. and one scratch and 'poof' goes your data. I'm surprised as a medium that they have lasted this long. [1]"Manufacturers claim that CD-R and DVD-R discs have a shelf life of 5 to 10 years before recording," note that is "before recording" [1] http://www.clir.org/pubs/reports/pub121/sec4.html http://www.clir.org/pubs/reports/pub121/sec4.html
  - kijin 13y ago
    Sure, but shelf life won't be a problem when you burn a new disc every day and you intend to keep at most a couple of months' worth of daily snapshots. As terrible as optical discs are for long-term archiving, I'm sure their shelf life usually exceeds that of a carton of eggs. That should be enough to protect against ransomware and most other kinds of disasters that can strike a personal computer.

cperciva 13y ago

I've been thinking about ways to create an offline-equivalent backup, so that it can be automated. You can do this with Tarsnap. The tarsnap-keymgmt utility allows you to create key file which can only create new archives, not delete or overwrite existing ones.