4 ms·

Storing encrypted blobs in S3 is my new strategy for bulk media storage. You'll never beat the QoS and resilience of the cloud storage product with something at

Storing encrypted blobs in S3 is my new strategy for bulk media storage. You'll never beat the QoS and resilience of the cloud storage product with something at home. I have completely lost patience with maintaining local hardware like this. If no one has a clue what is inside your blobs, they might as well not exist from their perspective. This feels like smuggling cargo on a federation starship, which is way cooler to me than filling up a bunch of local disks. I don't need 100% of my bytes to be instantly available to me on my network. The most important stuff is already available. I can wait a day for arbitrary media to thaw out for use. Local caching and pre-loading of read-only blobs is an extremely obvious path for smoothing over remote storage. Other advantages should be obvious. There are no limits to the scale of storage and unless you are a top 1% hoarder, the cost will almost certainly be more than amortized by the capex you would have otherwise spent on all that hardware.

disambiguation 2y ago
How much does S3 cost these days? I've been burned by their hidden surge pricing before and hesitate to rely on them for personal storage when self hosting is fairly cheap.
- dmoy 2y ago
  Disk hosting cost isn't much Bandwidth to get all of that back down to your system is much pricier, depending on how much you use that data.
- mystifyingpoi 2y ago
  Storage cost is nice, data transfer costs are often prohibitively high.
3np 2y ago
3-2-1 says you want both. Can be convenient to centralize backups on a local NAS and then publish to the cloud from there.
cm2187 2y ago
For large datasets, the cost of that is like rebuying your NAS every 9-12 months.
akho 2y ago
> You'll never beat the QoS and resilience of the cloud storage product with something at home. You can’t be serious.
frontlodjkgi 2y ago
The only thing making S3 a no-go for me is their outbound traffic costs at $90/TB, which at 100TB restore would make out $9K just to transfer the data back once
viraptor 2y ago
> If no one has a clue what is inside your blobs, they might as well not exist from their perspective. This is not the perspective of actors working on longer timescale. For a number is agencies, preserving some encrypted data is beneficial, because it will be possible to recover in N years, whether any classic improvements, bugs found in key generators, or advances in quantum. Very few people here will be that interesting, but... worth keeping in mind.
- bob1029 2y ago
  The point of encryption in this context is to defeat content fingerprinting techniques, not the focused resources of a nation state.
deleted 2y ago
[deleted]
xyzzy123 2y ago
S3 or glacier? Glacier is cost competitive with local disk but not very practical for the sorts of things people usually need lots of local disk for (media & disk images). Interested in how you use this! 20TB which u can keep in a 2-bay cute little nas will cost you $4k USD / year on S3 infrequent access tier in APAC (where I am). So "payback time" of local hardware is just 6 months vs S3 IA. That's before you pay for any data transfers.
- fodkodrasz 2y ago
  Did you factor in the resilience and redundancy S3 gives you and you cannot opt out from? I have my NAS, and it is cheaper than S3 if I ignore these, but having to run 2 offsite backups would make it much less compelling.
  - xyzzy123 2y ago
    Agree, they are not the same thing. Yes, S3 provides much better durability. I just can't afford it. For my use-case I'm OK with un-hedged risk and dollars staying in my pocket.
    - ornornor 2y ago
      I backup my nas to rsync.net, it’s very cost effective using borg backup.
      - remram 2y ago
        $0.01/GB/mo, that does not seem better than Glacier, is it?
        ornornor 2y ago
        Except it’s not glacier speeds, there are no bandwidth costs, support is on a completely different level than aws (you can actually reach an actual knowledgeable human), and you can use anything that speaks ssh. They also have an expert price at 0.008$/gb/mo here https://www.rsync.net/products/borg.html https://www.rsync.net/products/borg.html

disambiguation 2y ago

How much does S3 cost these days? I've been burned by their hidden surge pricing before and hesitate to rely on them for personal storage when self hosting is fairly cheap.

dmoy 2y ago
Disk hosting cost isn't much Bandwidth to get all of that back down to your system is much pricier, depending on how much you use that data.
mystifyingpoi 2y ago
Storage cost is nice, data transfer costs are often prohibitively high.

3np 2y ago

3-2-1 says you want both. Can be convenient to centralize backups on a local NAS and then publish to the cloud from there.

cm2187 2y ago

For large datasets, the cost of that is like rebuying your NAS every 9-12 months.

akho 2y ago

> You'll never beat the QoS and resilience of the cloud storage product with something at home. You can’t be serious.

frontlodjkgi 2y ago

The only thing making S3 a no-go for me is their outbound traffic costs at $90/TB, which at 100TB restore would make out $9K just to transfer the data back once

viraptor 2y ago

> If no one has a clue what is inside your blobs, they might as well not exist from their perspective. This is not the perspective of actors working on longer timescale. For a number is agencies, preserving some encrypted data is beneficial, because it will be possible to recover in N years, whether any classic improvements, bugs found in key generators, or advances in quantum. Very few people here will be that interesting, but... worth keeping in mind.

bob1029 2y ago
The point of encryption in this context is to defeat content fingerprinting techniques, not the focused resources of a nation state.

deleted 2y ago

[deleted]

xyzzy123 2y ago

S3 or glacier? Glacier is cost competitive with local disk but not very practical for the sorts of things people usually need lots of local disk for (media & disk images). Interested in how you use this! 20TB which u can keep in a 2-bay cute little nas will cost you $4k USD / year on S3 infrequent access tier in APAC (where I am). So "payback time" of local hardware is just 6 months vs S3 IA. That's before you pay for any data transfers.

fodkodrasz 2y ago
Did you factor in the resilience and redundancy S3 gives you and you cannot opt out from? I have my NAS, and it is cheaper than S3 if I ignore these, but having to run 2 offsite backups would make it much less compelling.
- xyzzy123 2y ago
  Agree, they are not the same thing. Yes, S3 provides much better durability. I just can't afford it. For my use-case I'm OK with un-hedged risk and dollars staying in my pocket.
  - ornornor 2y ago
    I backup my nas to rsync.net, it’s very cost effective using borg backup.
    - remram 2y ago
      $0.01/GB/mo, that does not seem better than Glacier, is it?
      - ornornor 2y ago
        Except it’s not glacier speeds, there are no bandwidth costs, support is on a completely different level than aws (you can actually reach an actual knowledgeable human), and you can use anything that speaks ssh. They also have an expert price at 0.008$/gb/mo here https://www.rsync.net/products/borg.html https://www.rsync.net/products/borg.html