5 ms·

I am always scared to put too much trust in new Object Storage services. I love Hetzner and similar but until this new service has been around for a while, I'd

I am always scared to put too much trust in new Object Storage services. I love Hetzner and similar but until this new service has been around for a while, I'd only use them for stuff I can afford to lose. From the outside as a consumer of these services you do not know how robust they actually are on the inside. I remember the data loss from OVH where they put backups in the same building as the primaries and people only found out about this once a fire took out the whole building: https://blocksandfiles.com/2023/03/23/ovh-cloud-must-pay-damages-for-lost-backup-data/ https://blocksandfiles.com/2023/03/23/ovh-cloud-must-pay-dam...

BoredPositron 2y ago
You know that if US East suffers a FULL data loss that the recovery would take weeks with the question if it would even be possible. That's what happened to ovh... it wasn't just one building.
- clarkdave 2y ago
  > It's not like there is a mirror datacenter just two blocks away Isn't that exactly what Availability Zones are for? They're physically separate[0] datacenters and each one contains a copy of each S3 object (unless using the explicit single-zone options) It's also straightforward (although not necessarily that cheap) to replicate S3 objects to another region [0] https://aws.amazon.com/ec2/faqs/?nc1=h_ls#Platform https://aws.amazon.com/ec2/faqs/?nc1=h_ls#Platform
- ahofmann 2y ago
  It literally was one building in france, that burned to the ground: https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire-france-data-center/ https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire...
- abadpoli 2y ago
  If us-east-1 ever suffered a “FULL” data loss, it would be a company-ending event for so many companies that it would practically end society as we know it. OVH’s failure was a single building. That’s the problem with a lot of server hosters - even Google has their availability zones all co-located in the same building, so a physical event like a fire could take down an entire region. AWS has AZs in physically separate locations, each with 1+ separate DCs.
  - t0mas88 2y ago
    Is the any GCP documentation on that? Sounds like a far bigger risk going with GCP than AWS if all zones in a region are in the same building.
numbsafari 2y ago
"Cloud 3-2-1" to the rescue... I take the typical formulation (e.g., [1]), and translate it into: - Keep 3 copies of your data: production + 2. - Keep 2 snapshots, stored separately. - Keep 1 snapshot on-hand (literally in your possession), or with a different provider. It's great to see more options for "different provider". If I were an early adopter, I would either target this as my on-Hetzner snapshot of my Hetzner-hosted data, and replicate it elsewhere; or I would consider trialing it as a backup to non-Hetzner data. To your point, though, I'd probably wait on the latter until it has gone through some growing pains. [1] - https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
vidarh 2y ago
S3 durability is pretty much the only thing I consider worth using EC2 for. S3 egress, on the other hand, is so expensive you can often justify putting a writethrough cache for your entire working set at Hetzner...
- ptman 2y ago
  IIRC Cloudflare R2 is designed as a cache for S3 and has free egress edit: apparently not writethrough: https://developers.cloudflare.com/r2/data-migration/sippy/ https://developers.cloudflare.com/r2/data-migration/sippy/
  - jgrahamc 2y ago
    No, R2 is not an "S3 cache". It's a replacement for S3 with no egress fees.
  - viraptor 2y ago
    Yeah, don't use Sippy. It just stopped working after 2 weeks on multiple buckets and support came back with "have you tried turning it off and on again". It's not production ready. Also R2 is extremely basic compared to S3. For example only trivial permissions are supported and auth tokens are bound to an actual account.
smartbit 2y ago
Couldn't agree with you more. Last spring talked to a senior Hetzner engineer who wasn't too sure about the Object Storage design/implementation and was happy he wasn't involved.
- sangnoir 2y ago
  As everyone knows, this is in sharp contrast with serious engineering operations like Amazon or Google, where there are no disagreements on technical designs. Ever.
timenova 2y ago
I should mention Tigris[0] here. They're also a new Object Storage service, but they have this two-way replication facility with another S3-compatible service. The primary purpose they built it for is to mirror files from your existing S3 to Tigris as files are requested. However they also have an option to copy files that are added to Tigris, to S3 automatically [1] (`--shadow-write-through`). I asked their founder if it's okay to use it as an extra redundancy continuously instead of a one-time migration, and they said they have no issues with it. [0] https://www.tigrisdata.com https://www.tigrisdata.com [1] https://www.tigrisdata.com/docs/migration/#starting-the-migration https://www.tigrisdata.com/docs/migration/#starting-the-migr...
- rmbyrro 2y ago
  Backblaze is 3x less expensive and offers more generous free allowances.
- ChocolateGod 2y ago
  > Access small objects at close to Redis speed This is a massive claim that unless they're storing everything in memory I don't know how you can get close to Redis speed. Even the flash-based Redis alternatives can't get close (but are good enough)
chasd00 2y ago
I toured a data center in Tornado alley back when leasing cages was pretty common. I asked them about disaster planning regarding getting completely wiped off the map and they sorta scoffed at me. Literally two weeks later a tornado missed them by about a 1/4 mile. Would have loved to be a fly on the wall after that. another story, a small company I worked for contracted with a small data center. They did things right and ran a tight ship. However something happened like a lightning strike or so other electrical fault and a critical component welded itself into a position they could t get out of. I wish I could remember the details better but they were down multiple days.
- latchkey 2y ago
  Switch builds their DCs with two roofs... https://www.switch.com/switch-shield/ https://www.switch.com/switch-shield/ The Switch one I'm in (Grand Rapids Pyramid), the servers are about 3/4's underground.
bilekas 2y ago
Years ago there was a fire in OVH who I always trusted as reliable and stable, never needed to worry.. Lost a lot that day. Probably I could have had offsite backup managed myself though. I see no difference here, you can manage your own disaster recovery. From what I see, this actually might be a great backup/recovery solution for S3 in my case at least.
- deleted 2y ago
  [deleted]
- voytec 2y ago
  Maybe if OVH had installed an automatic fire extinguishing system or an electric mains cut-off switch, the firefighters wouldn't have struggled with meter-long electrical arcs and could put out the fire quicker.
  - liotier 2y ago
    Yes, that hosting center was unsafe as hell. All of OVH's early year featured a lot of duct-tape and daring corner-cutting - that is how they were incredibly cheap. But nowadays they are just as mainstream industrial as their competitors and the 2021 fire marked the end of an era... It is sad that it happened as they were already turning the company around to being an established player. A great writeup of the incident and its context, three years afterwards: https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire-france-data-center/ https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire...
    - ChocolateGod 2y ago
      > But nowadays they are just as mainstream industrial as their competitors Shame their control panel is massively behind their competitors, it's the worst cloud control panel I've had the experience of using, even worse than Azure and Google. It's slow slow slow slow slow and horribly buggy, did I mention it's slow?
- lopkeny12ko 2y ago
  People seem to have this misconception that "cloud computing" is some kind of magic bullet that guarantees 100% durability. News flash, your data is still being stored on physical hard drives, in some data center. If the building burns down in a fire, of course your data will be gone, the hard drives are hosed, it's not magic. You really have no one to be mad at other than yourself...
guerby 2y ago
It just caught those who rely on one provider, which is never a good practice. rclone to somewhere else at least. And 3-2-1 rule for backups if you're serious.
fabian2k 2y ago
I'd probably look at using S3 as a backup and only have the working copy with Hetzner. Not sure how much more expensive that would be, the pricing for the non-standard S3 and glacier does seem to require a bit more work to understand properly.
luuurker 2y ago
One of my VPS was lost in that fire, but since my backups where somewhere else, I didn't lose much. The trick is not to put all eggs in the same basket and it's easier to do that when you're paying much less than you'd pay on providers like AWS.

BoredPositron 2y ago

You know that if US East suffers a FULL data loss that the recovery would take weeks with the question if it would even be possible. That's what happened to ovh... it wasn't just one building.

clarkdave 2y ago
> It's not like there is a mirror datacenter just two blocks away Isn't that exactly what Availability Zones are for? They're physically separate[0] datacenters and each one contains a copy of each S3 object (unless using the explicit single-zone options) It's also straightforward (although not necessarily that cheap) to replicate S3 objects to another region [0] https://aws.amazon.com/ec2/faqs/?nc1=h_ls#Platform https://aws.amazon.com/ec2/faqs/?nc1=h_ls#Platform
ahofmann 2y ago
It literally was one building in france, that burned to the ground: https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire-france-data-center/ https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire...
abadpoli 2y ago
If us-east-1 ever suffered a “FULL” data loss, it would be a company-ending event for so many companies that it would practically end society as we know it. OVH’s failure was a single building. That’s the problem with a lot of server hosters - even Google has their availability zones all co-located in the same building, so a physical event like a fire could take down an entire region. AWS has AZs in physically separate locations, each with 1+ separate DCs.
- t0mas88 2y ago
  Is the any GCP documentation on that? Sounds like a far bigger risk going with GCP than AWS if all zones in a region are in the same building.

numbsafari 2y ago

"Cloud 3-2-1" to the rescue... I take the typical formulation (e.g., [1]), and translate it into: - Keep 3 copies of your data: production + 2. - Keep 2 snapshots, stored separately. - Keep 1 snapshot on-hand (literally in your possession), or with a different provider. It's great to see more options for "different provider". If I were an early adopter, I would either target this as my on-Hetzner snapshot of my Hetzner-hosted data, and replicate it elsewhere; or I would consider trialing it as a backup to non-Hetzner data. To your point, though, I'd probably wait on the latter until it has gone through some growing pains. [1] - https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/

vidarh 2y ago

S3 durability is pretty much the only thing I consider worth using EC2 for. S3 egress, on the other hand, is so expensive you can often justify putting a writethrough cache for your entire working set at Hetzner...

ptman 2y ago
IIRC Cloudflare R2 is designed as a cache for S3 and has free egress edit: apparently not writethrough: https://developers.cloudflare.com/r2/data-migration/sippy/ https://developers.cloudflare.com/r2/data-migration/sippy/
- jgrahamc 2y ago
  No, R2 is not an "S3 cache". It's a replacement for S3 with no egress fees.
- viraptor 2y ago
  Yeah, don't use Sippy. It just stopped working after 2 weeks on multiple buckets and support came back with "have you tried turning it off and on again". It's not production ready. Also R2 is extremely basic compared to S3. For example only trivial permissions are supported and auth tokens are bound to an actual account.

smartbit 2y ago

Couldn't agree with you more. Last spring talked to a senior Hetzner engineer who wasn't too sure about the Object Storage design/implementation and was happy he wasn't involved.

sangnoir 2y ago
As everyone knows, this is in sharp contrast with serious engineering operations like Amazon or Google, where there are no disagreements on technical designs. Ever.

timenova 2y ago

I should mention Tigris[0] here. They're also a new Object Storage service, but they have this two-way replication facility with another S3-compatible service. The primary purpose they built it for is to mirror files from your existing S3 to Tigris as files are requested. However they also have an option to copy files that are added to Tigris, to S3 automatically [1] (`--shadow-write-through`). I asked their founder if it's okay to use it as an extra redundancy continuously instead of a one-time migration, and they said they have no issues with it. [0] https://www.tigrisdata.com https://www.tigrisdata.com [1] https://www.tigrisdata.com/docs/migration/#starting-the-migration https://www.tigrisdata.com/docs/migration/#starting-the-migr...

rmbyrro 2y ago
Backblaze is 3x less expensive and offers more generous free allowances.
ChocolateGod 2y ago
> Access small objects at close to Redis speed This is a massive claim that unless they're storing everything in memory I don't know how you can get close to Redis speed. Even the flash-based Redis alternatives can't get close (but are good enough)

chasd00 2y ago

I toured a data center in Tornado alley back when leasing cages was pretty common. I asked them about disaster planning regarding getting completely wiped off the map and they sorta scoffed at me. Literally two weeks later a tornado missed them by about a 1/4 mile. Would have loved to be a fly on the wall after that. another story, a small company I worked for contracted with a small data center. They did things right and ran a tight ship. However something happened like a lightning strike or so other electrical fault and a critical component welded itself into a position they could t get out of. I wish I could remember the details better but they were down multiple days.

latchkey 2y ago
Switch builds their DCs with two roofs... https://www.switch.com/switch-shield/ https://www.switch.com/switch-shield/ The Switch one I'm in (Grand Rapids Pyramid), the servers are about 3/4's underground.

bilekas 2y ago

Years ago there was a fire in OVH who I always trusted as reliable and stable, never needed to worry.. Lost a lot that day. Probably I could have had offsite backup managed myself though. I see no difference here, you can manage your own disaster recovery. From what I see, this actually might be a great backup/recovery solution for S3 in my case at least.

deleted 2y ago
[deleted]
voytec 2y ago
Maybe if OVH had installed an automatic fire extinguishing system or an electric mains cut-off switch, the firefighters wouldn't have struggled with meter-long electrical arcs and could put out the fire quicker.
- liotier 2y ago
  Yes, that hosting center was unsafe as hell. All of OVH's early year featured a lot of duct-tape and daring corner-cutting - that is how they were incredibly cheap. But nowadays they are just as mainstream industrial as their competitors and the 2021 fire marked the end of an era... It is sad that it happened as they were already turning the company around to being an established player. A great writeup of the incident and its context, three years afterwards: https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire-france-data-center/ https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire...
  - ChocolateGod 2y ago
    > But nowadays they are just as mainstream industrial as their competitors Shame their control panel is massively behind their competitors, it's the worst cloud control panel I've had the experience of using, even worse than Azure and Google. It's slow slow slow slow slow and horribly buggy, did I mention it's slow?
lopkeny12ko 2y ago
People seem to have this misconception that "cloud computing" is some kind of magic bullet that guarantees 100% durability. News flash, your data is still being stored on physical hard drives, in some data center. If the building burns down in a fire, of course your data will be gone, the hard drives are hosed, it's not magic. You really have no one to be mad at other than yourself...

guerby 2y ago

It just caught those who rely on one provider, which is never a good practice. rclone to somewhere else at least. And 3-2-1 rule for backups if you're serious.

fabian2k 2y ago

I'd probably look at using S3 as a backup and only have the working copy with Hetzner. Not sure how much more expensive that would be, the pricing for the non-standard S3 and glacier does seem to require a bit more work to understand properly.

luuurker 2y ago

One of my VPS was lost in that fire, but since my backups where somewhere else, I didn't lose much. The trick is not to put all eggs in the same basket and it's easier to do that when you're paying much less than you'd pay on providers like AWS.