9 ms·

>2048 bit RSA keys Github recommends 4096 now, for what it's worth. [1] >Pushing database backups offsite This is a really bad idea and a good way to get own

>2048 bit RSA keys Github recommends 4096 now, for what it's worth. [1] >Pushing database backups offsite This is a really bad idea and a good way to get owned. Database backups must be PULLED from the server, not pushed from it. Separately, you also need to test that you can restore from your backups periodically. There were a couple other things I disagree with, but they're in the realm of personal preference. It's also interesting that this guide uses Linode which is a hosting company known to have had quite egregious security issues in the past. [2] [1] https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/ https://help.github.com/articles/generating-a-new-ssh-key-an... [2] https://www.google.com/#q=linode+hack https://www.google.com/#q=linode+hack

nnutter 10y ago
Why must they be PULLED?
- deleted 10y ago
  [deleted]
- mastorrent 10y ago
  If a server can push backups, it must necessarily have credentials for and write access to that backup repository. If the server is compromised, that write access could be used to attack the backup repository as well. Pulling backups from a server removes the need for credentials or access to the backup repository.
  - pfg 10y ago
    That is one valid way of solving this problem. Another option is to have a permission system that allows you specify that the server can only append new data, but not delete or overwrite existing files. This is useful if you don't want the added complexity of having a separate "backup management" service, which could easily become a bottleneck if you are backing up a lot of data from multiple servers. One way to do this is with S3, for example, is to use an IAM role with only the "PutObject" permission, and enable object versioning for the bucket to prevent a compromised server from being able to delete data by overwriting existing files.
    - foodstances 10y ago
      It's also easy to do this with Tarsnap. You can split out your host keys with tarsnap-keymgmt into separate read, write, and delete keys. Just take your delete key off of the host and store it somewhere secure.
      - cperciva 10y ago
        Or better yet, create the key somewhere else and only upload the keys you want to have on the server. Otherwise there's a chance that the original tarsnap key is left in an unallocated block.
    - taneq 10y ago
      Not an expert here, but this seems to me to be stripping away one layer of security. Kind of like saying that you can leave a key to your house lying around as long as the pantry door is locked - if your database server gets owned, they now have some kind of login to your backup server as opposed to no kind of login. Maybe the extra layer doesn't add much security, but if it's a simple config change and it does add something, wouldn't it be worth doing?
      - pfg 10y ago
        Not a bad point - it's a trade-off. You could make a similar point about the centralized backup management server - it needs to have "some kind of login" to all your production database server, so of that host is compromised (which might only store encrypted copies of your backup), so are all your database servers if those privileges can be escalated. You could argue that one backup host is easier to secure than a complex system such as AWS, but then I would argue that it would probably be hard to beat the track record of S3/IAM. ;-) Both approaches have their place. If you're dealing with a large number of hosts and a lot of data, the backup host will quickly become a bottleneck and you're probably better off with the append-only approach. If that's not a concern for you (i.e. you're not running into any bandwidth limits on the backup hosts) and want to deal with operating yet another service to avoid the risk of e.g. IAM privilege escalation, the other solution might be a better fit.
  - walrus01 10y ago
    Doesn't entirely fix the problem, but using public/private key auth and a private key created just for the backup cron job, the server can push a backup via scp/ssh to its own user account which on the destination end is chrooted into a particular backup destination directory.
  - zeeZ 10y ago
    Plus, if you're super paranoid, pulling doesn't require leaving any ports open and software listening on the repository.
    - tamana 10y ago
      But it requires leaving a port open on the DB server where an attacker can get a whole copy of your database, or destroy it.
      - amdavidson 10y ago
        Many database servers are not serving databases solely to themselves and have that port open as part of the normal course of operation.
hueving 10y ago
>This is a really bad idea and a good way to get owned. Database backups must be PULLED Just make sure your database server doesn't have permission to delete backups (e.g. Have it POST a backup via HTTPS). There is nothing wrong with db server initiated backups.
- brobinson 10y ago
  This is still open to abuse in the form of your legitimate backup script being disabled and replaced by one uploading false backups. You wouldn't know anything has happened until you go to restore from one. Of course, this is an edge case, but I believe setting up a pull-based backup system is still going to be less work than a write-only push system.
  - arielb1 10y ago
    A hacked server can have the backup server pull false backups as well. Of course, your servers shouldn't be SSHing to your backup servers, but that goes both ways.
  - tamana 10y ago
    If the backup machine is compromised the attacker can corrupt your backups. Either way, if machine is hacked you risk loss.
  - hueving 10y ago
    That's ridiculous. The same tricks can be played on a pull based system once you have compromised the DB server.
brokenwren 10y ago
I agree that pulling does appear to be more secure, but if someone breaches any of your servers undetected, you're pretty much hosed regardless. Plus, the Inversoft guide specifically states that backups must be encrypted. I could put my backup ZIPs on a public Github repository and no one would be able to access the user data stored inside it. Therefore, it really doesn't matted if they are pushed or pulled.
- jacquesm 10y ago
  That may be true but with a pulled db backup and a good recovery strategy you'll be back up relatively quickly, without you'll be DOA.
- brobinson 10y ago
  If a script or cronjob or whatever on the server has write access to the repo, so does the hypothetical attacker. I remember seeing a "hacking" website get wiped along with all of its backups about 15 years ago and it left a very strong impression on me regarding this issue.
  - jjnoakes 10y ago
    If it is write access but not delete access (nor overwrite access) then there is no issue. It is sometimes more secure to push because it requires no inbound connections or authorizations the live machine.
lebca 10y ago
Could you recommend another hosting service with better security practices than Linode?
- brobinson 10y ago
  I don't currently use them, but I've had good luck with Vultr and they haven't had any incidents like Linode AFAIK. There's also AWS, Azure, whatever Google's thing is called, and all the other big ones. I have no opinion on DigitalOcean even though I currently have a VPS with them since I'm not running anything remotely mission critical there.

nnutter 10y ago

Why must they be PULLED?

deleted 10y ago
[deleted]
mastorrent 10y ago
If a server can push backups, it must necessarily have credentials for and write access to that backup repository. If the server is compromised, that write access could be used to attack the backup repository as well. Pulling backups from a server removes the need for credentials or access to the backup repository.
- pfg 10y ago
  That is one valid way of solving this problem. Another option is to have a permission system that allows you specify that the server can only append new data, but not delete or overwrite existing files. This is useful if you don't want the added complexity of having a separate "backup management" service, which could easily become a bottleneck if you are backing up a lot of data from multiple servers. One way to do this is with S3, for example, is to use an IAM role with only the "PutObject" permission, and enable object versioning for the bucket to prevent a compromised server from being able to delete data by overwriting existing files.
  - foodstances 10y ago
    It's also easy to do this with Tarsnap. You can split out your host keys with tarsnap-keymgmt into separate read, write, and delete keys. Just take your delete key off of the host and store it somewhere secure.
    - cperciva 10y ago
      Or better yet, create the key somewhere else and only upload the keys you want to have on the server. Otherwise there's a chance that the original tarsnap key is left in an unallocated block.
  - taneq 10y ago
    Not an expert here, but this seems to me to be stripping away one layer of security. Kind of like saying that you can leave a key to your house lying around as long as the pantry door is locked - if your database server gets owned, they now have some kind of login to your backup server as opposed to no kind of login. Maybe the extra layer doesn't add much security, but if it's a simple config change and it does add something, wouldn't it be worth doing?
    - pfg 10y ago
      Not a bad point - it's a trade-off. You could make a similar point about the centralized backup management server - it needs to have "some kind of login" to all your production database server, so of that host is compromised (which might only store encrypted copies of your backup), so are all your database servers if those privileges can be escalated. You could argue that one backup host is easier to secure than a complex system such as AWS, but then I would argue that it would probably be hard to beat the track record of S3/IAM. ;-) Both approaches have their place. If you're dealing with a large number of hosts and a lot of data, the backup host will quickly become a bottleneck and you're probably better off with the append-only approach. If that's not a concern for you (i.e. you're not running into any bandwidth limits on the backup hosts) and want to deal with operating yet another service to avoid the risk of e.g. IAM privilege escalation, the other solution might be a better fit.
- walrus01 10y ago
  Doesn't entirely fix the problem, but using public/private key auth and a private key created just for the backup cron job, the server can push a backup via scp/ssh to its own user account which on the destination end is chrooted into a particular backup destination directory.
- zeeZ 10y ago
  Plus, if you're super paranoid, pulling doesn't require leaving any ports open and software listening on the repository.
  - tamana 10y ago
    But it requires leaving a port open on the DB server where an attacker can get a whole copy of your database, or destroy it.
    - amdavidson 10y ago
      Many database servers are not serving databases solely to themselves and have that port open as part of the normal course of operation.

hueving 10y ago

>This is a really bad idea and a good way to get owned. Database backups must be PULLED Just make sure your database server doesn't have permission to delete backups (e.g. Have it POST a backup via HTTPS). There is nothing wrong with db server initiated backups.

brobinson 10y ago
This is still open to abuse in the form of your legitimate backup script being disabled and replaced by one uploading false backups. You wouldn't know anything has happened until you go to restore from one. Of course, this is an edge case, but I believe setting up a pull-based backup system is still going to be less work than a write-only push system.
- arielb1 10y ago
  A hacked server can have the backup server pull false backups as well. Of course, your servers shouldn't be SSHing to your backup servers, but that goes both ways.
- tamana 10y ago
  If the backup machine is compromised the attacker can corrupt your backups. Either way, if machine is hacked you risk loss.
- hueving 10y ago
  That's ridiculous. The same tricks can be played on a pull based system once you have compromised the DB server.

brokenwren 10y ago

I agree that pulling does appear to be more secure, but if someone breaches any of your servers undetected, you're pretty much hosed regardless. Plus, the Inversoft guide specifically states that backups must be encrypted. I could put my backup ZIPs on a public Github repository and no one would be able to access the user data stored inside it. Therefore, it really doesn't matted if they are pushed or pulled.

jacquesm 10y ago
That may be true but with a pulled db backup and a good recovery strategy you'll be back up relatively quickly, without you'll be DOA.
brobinson 10y ago
If a script or cronjob or whatever on the server has write access to the repo, so does the hypothetical attacker. I remember seeing a "hacking" website get wiped along with all of its backups about 15 years ago and it left a very strong impression on me regarding this issue.
- jjnoakes 10y ago
  If it is write access but not delete access (nor overwrite access) then there is no issue. It is sometimes more secure to push because it requires no inbound connections or authorizations the live machine.

lebca 10y ago

Could you recommend another hosting service with better security practices than Linode?

brobinson 10y ago
I don't currently use them, but I've had good luck with Vultr and they haven't had any incidents like Linode AFAIK. There's also AWS, Azure, whatever Google's thing is called, and all the other big ones. I have no opinion on DigitalOcean even though I currently have a VPS with them since I'm not running anything remotely mission critical there.