5 ms·

While I get that there are use cases for physical media, as both a data hoarder and data paranoiac (good things in my line of work), I've moved on. It's the dat

While I get that there are use cases for physical media, as both a data hoarder and data paranoiac (good things in my line of work), I've moved on. It's the data that matters, not the media. To that end, I have automated two separate, but complementary, processes for my personal data: - Sync to multiple machines, at least one of which is offsite, in near realtime. This provides rapid access to data in the event of a disk/system/building/etc failure. Use whatever software you want (Dropbox, Resilio, rsync scripts, etc), but in the event of failure, this solves 99% of my issues - I have another device that has fast access to my most recent data, current to within seconds. This is especially important when bringing up a new, upgraded system - just sync over the LAN. (Currently this is 4 devices, 2 offsite, but it flexes up/down over time occassionally). - Backup to multiple cloud providers on a regular cadence (I do hourly encrypted incrementals). This protects me against data loss, corruption, malware attacks, my stupidity deleting something, etc. This solves the remaining 1% of my issues, enabling point-in-time recovery for any bit of data in my past, stretching back many years. I've outsourced the "media" issue to the cloud providers in this case, so they handle whatever is failing, and the cost is getting pretty absurdly cheap for consumers, and will continue to do so. My favorite software is Arq Backup, but there are lots of options. (Currently this is 4 discrete cloud providers, and since this is non-realtime typically, utilizes coldest storage options available). Between these two complimentary, fully automated approaches, I no longer have to worry about the mess of media failure, RAID failures, system failures, human error, getting a new system online, cloud provider being evil, etc etc.

creer 2y ago
You seem well placed to answer this one: how is cost for this resilience? Compare to the cost of the storage itself? Including the cost of migrating from solutions that are withdrawn from the market?
- onethumb 2y ago
  The cost (I assume you're talking about "my time" cost?) is unbelievably low, mostly in part due to good software. It "just works". Specifically, Arq Backup, for example, lets you simply add/remove providers at will. It's happened multiple times, Amazon Drive changed (or went away? I forget...), Google Drive changed their Enterprise policies, etc... No big deal, I just deleted the provider and added another one. I still had plenty of working provider backups, so I wasn't worried while it took a day or two to fill the next provider. (Good argument for having 2+, I'd argue 3+ providers...) Using notifications from your sync apps/systems/scripts/whatever is essential, of course, in case something fails... but all the good software has that built-in (including email and other notifications, not just OS, which helps for remote systems). At this point, it's nearly idiot proof. (Good for idiots like me ;)
  - creer 2y ago
    I meant more monetary cost. Nominally cloud storage for one unit of storage and one unit of time is perfectly "fine". Except that it adds up. More storage, indefinitely held, multiple hosts. Data which needs to be copied from one to the other which incurs costs from both. Add to this routine retrieval costs - if you "live this way". And routine test retrieval costs if you know what's good for you. So last time I looked, unit costs were low - sure. But all-included costs were high.
    - onethumb 2y ago
      Certainly some of this simply comes down to "how valuable is my data?". Currently, given the extremely low (and dropping YoY) cost of storing cold data at rest, the essentially free cost of ingest, and the high cost of retrieving cold data which I almost never have to do, the ROI is wildly positive. For me. And since all of these things (how many providers, which providers, which storage classes, how long to retain the data, etc) are all fine-tunable, you can basically do your own ROI math, then pick the parameters which work for you.
EvanAnderson 2y ago
I get some peace of mind (in both professional and business settings) from having backup include a physically separable and 100% offline component. I like knowing an attacker would need to resort to kinetic means to completely destroy all copies of the data. The physically separable component often lags behind the rest of the copies. It may only be independently verified on an air-gapped machine periodically. It's not the best copy, for sure. I still take comfort in knowing attackers generally won't launch a kinetic attack.
- dboreham 2y ago
  Mr Metorite can launch such an attack but as long as you have two physically gapped backups at a distance greater than the likely blast radius you'll be fine.
- onethumb 2y ago
  That's fair. Nothing like piece of mind. :) Conceptually, though, I think my separation of "sync vs backup" and separating of discrete providers (both software and supplier) accomplishes this same goal. Conceptually, it's not very different, or possibly just a level up, from "online media vs archive media". At least, it seems that way to me.
tatersolid 2y ago
> This protects me against…malware attacks Are you sure about that? Many ransomware attackers do recon for some time to find the backup systems and then render those unusable during the attack. In your case your cloud credentials (with delete permissions?) must be present on your live sou ce systems, rendering the cloud backups vulnerable to your overwrite or deletion. There are immutable options in the bigger cloud storage services but in my experience they are often unused, used incorrectly, or incompatible with tools that update backup metadata in-place. I’ve encountered several tools/scripts mark a file file as immutable for 90 days the first time it is backed up, but not extend that date correctly on the next incremental, leaving older but still critical data vulnerable to ransomware.
- onethumb 2y ago
  There are no delete credentials, and the WORM option is enabled when a provider supports it. I can ~always get back to point-in-time.
  - onethumb 2y ago
    No delete credentials present a cost issue when moving from a provider... I've accidentally left data behind after I thought I'd deleted it. Worth the risk, and learned my lesson.
    - renewiltord 2y ago
      You can set a lifecycle rule. You don't need credentials to delete.
      - onethumb 2y ago
        Only if you allow permissions to set a lifecycle rule...
        renewiltord 2y ago
        You don't set the lifecycle rule at runtime. You set it at environment setup time. The credentials that put your object don't have to have the power to set lifecycle permissions. You obviously don't put your environment setup user in your app. That would be utterly retarded.
rbanffy 2y ago
Backups and archival are different things, with similar requirements, but different priorities. A backup doesn't care about data you think you'll never need again.
deleted 2y ago
[deleted]
mr_mitm 2y ago
Are your backups encrypted and if so, how do you manage and backup the keys?

creer 2y ago

You seem well placed to answer this one: how is cost for this resilience? Compare to the cost of the storage itself? Including the cost of migrating from solutions that are withdrawn from the market?

onethumb 2y ago
The cost (I assume you're talking about "my time" cost?) is unbelievably low, mostly in part due to good software. It "just works". Specifically, Arq Backup, for example, lets you simply add/remove providers at will. It's happened multiple times, Amazon Drive changed (or went away? I forget...), Google Drive changed their Enterprise policies, etc... No big deal, I just deleted the provider and added another one. I still had plenty of working provider backups, so I wasn't worried while it took a day or two to fill the next provider. (Good argument for having 2+, I'd argue 3+ providers...) Using notifications from your sync apps/systems/scripts/whatever is essential, of course, in case something fails... but all the good software has that built-in (including email and other notifications, not just OS, which helps for remote systems). At this point, it's nearly idiot proof. (Good for idiots like me ;)
- creer 2y ago
  I meant more monetary cost. Nominally cloud storage for one unit of storage and one unit of time is perfectly "fine". Except that it adds up. More storage, indefinitely held, multiple hosts. Data which needs to be copied from one to the other which incurs costs from both. Add to this routine retrieval costs - if you "live this way". And routine test retrieval costs if you know what's good for you. So last time I looked, unit costs were low - sure. But all-included costs were high.
  - onethumb 2y ago
    Certainly some of this simply comes down to "how valuable is my data?". Currently, given the extremely low (and dropping YoY) cost of storing cold data at rest, the essentially free cost of ingest, and the high cost of retrieving cold data which I almost never have to do, the ROI is wildly positive. For me. And since all of these things (how many providers, which providers, which storage classes, how long to retain the data, etc) are all fine-tunable, you can basically do your own ROI math, then pick the parameters which work for you.

EvanAnderson 2y ago

I get some peace of mind (in both professional and business settings) from having backup include a physically separable and 100% offline component. I like knowing an attacker would need to resort to kinetic means to completely destroy all copies of the data. The physically separable component often lags behind the rest of the copies. It may only be independently verified on an air-gapped machine periodically. It's not the best copy, for sure. I still take comfort in knowing attackers generally won't launch a kinetic attack.

dboreham 2y ago
Mr Metorite can launch such an attack but as long as you have two physically gapped backups at a distance greater than the likely blast radius you'll be fine.
onethumb 2y ago
That's fair. Nothing like piece of mind. :) Conceptually, though, I think my separation of "sync vs backup" and separating of discrete providers (both software and supplier) accomplishes this same goal. Conceptually, it's not very different, or possibly just a level up, from "online media vs archive media". At least, it seems that way to me.

tatersolid 2y ago

> This protects me against…malware attacks Are you sure about that? Many ransomware attackers do recon for some time to find the backup systems and then render those unusable during the attack. In your case your cloud credentials (with delete permissions?) must be present on your live sou ce systems, rendering the cloud backups vulnerable to your overwrite or deletion. There are immutable options in the bigger cloud storage services but in my experience they are often unused, used incorrectly, or incompatible with tools that update backup metadata in-place. I’ve encountered several tools/scripts mark a file file as immutable for 90 days the first time it is backed up, but not extend that date correctly on the next incremental, leaving older but still critical data vulnerable to ransomware.

onethumb 2y ago
There are no delete credentials, and the WORM option is enabled when a provider supports it. I can ~always get back to point-in-time.
- onethumb 2y ago
  No delete credentials present a cost issue when moving from a provider... I've accidentally left data behind after I thought I'd deleted it. Worth the risk, and learned my lesson.
  - renewiltord 2y ago
    You can set a lifecycle rule. You don't need credentials to delete.
    - onethumb 2y ago
      Only if you allow permissions to set a lifecycle rule...
      - renewiltord 2y ago
        You don't set the lifecycle rule at runtime. You set it at environment setup time. The credentials that put your object don't have to have the power to set lifecycle permissions. You obviously don't put your environment setup user in your app. That would be utterly retarded.

rbanffy 2y ago

Backups and archival are different things, with similar requirements, but different priorities. A backup doesn't care about data you think you'll never need again.

deleted 2y ago

[deleted]

mr_mitm 2y ago

Are your backups encrypted and if so, how do you manage and backup the keys?