7 ms·

Wow that really sucks. I'm sorry this happened to you guys. Hopefully you've got a backup on a USB somewhere. If not, all the best with working on a brand-ne

• jszymborski 5y ago
  A mitigation for this sort of thing is for your production server (A) to only have very limited permissions to your backup site (B). It's not a panacea, because hackers will be able to push arbitrary things, but you're greatly reducing your surface.
  • newnamenewface 5y ago
    Seems like all* you need is to have production be able to push a new backup with a cool off period of a non-negligible, not-too-long period. *Please point out how wrong this is.
    • koolba 5y ago
      Production can push as many backups as it wants, it shouldn’t be able to overwrite the previous backups. Deletion and expiry has to happen separately. There’s no reason for it to even be able to list the backups and that restriction combined with a randomized backup ID (in addition to any date/time tagging) limits the ability to corrupt the backups.
    • blueblob 5y ago
      Seems to me it would be better to have the backup pull from production so the scripts cannot be reverse engineered. Maybe it's just 6 in one and a half dozen in the other though.
  • nwilkens 5y ago
    Immutable S3 storage is another method that we deploy when backing up servers like this.. https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident-response-guide/use-immutable-storage.html https://docs.aws.amazon.com/whitepapers/latest/aws-security-... Further, we also use Veeam to backup Linux servers.. a high level read on how it works with immutable storage: https://www.veeam.com/blog/v11-immutable-backup-storage.html https://www.veeam.com/blog/v11-immutable-backup-storage.html
  • ev1 5y ago
    This is a pokemon fansite that's old enough to drink that was just running on xenforo and wordpress. There probably is no "production vs staging" or immutability or ZFS snapshots, or large amounts of engineering going on.
    • jszymborski 5y ago
      They had offsite backups, so I'm just saying however permissions are managed, they should have had them reduced. Idk if they were just pushing to object storage on AWS/Azure/GCS or had another server, but my comment stands. Now, do I blame them? No, like you said it was a fansite. I'm just offering advice on how to avoid this sort of thing in the future for them and sites like them.
      • ev1 5y ago
        I'm kind of curious though - what can you do as a non-technical user to prevent this? Let's say you're on shared hosting - you don't know what object storage is, and you upload your PHP themes via FTP (or at best, FTP+implicit SSL, but the default is plaintext). Your backup is probably downloading something over FTP, or creating a backup in cPanel/DirectAdmin/whatever shared hosting panel. Your host doesn't allow SSH. This is the hypermajority of shared hosting. (My apologies, I didn't mean that directed toward you, it was more of a general "this is HN, not everybody will self host GPG backups on immutable redundant cloud storage")
        • deleted 5y ago
          [deleted]
        • sova 5y ago
          "One Way Synchronization" with DropBox might do the trick [https://www.multcloud.com/tutorials/dropbox-one-way-sync-1234.html https://www.multcloud.com/tutorials/dropbox-one-way-sync-123...]
        • jszymborski 5y ago
          Hmm, in this case I might have a local script running on a cronjob or Scheduled Task to FTP(S) into the server and copy the files on the server locally. You can run this on a desktop or a raspi if one knows how to use it. It sounds from the blog like they already had automated off-sites, which makes me feel like they do have (perhaps limited) SSH access? Maybe it's just a cPanel deal though. No troubles, didn't take it personally and hope I didn't come off that way :) I remember as a teenager subscribing to dirt cheap hosting that only allowed uploads via a portal (:S) and trying to upload phpbb file by file, so I get it ahaha... passion often outstrips means/ability :P