50 ms·

What’s your experience like at other storage outfits? I only ask because your post is a bit like singing praises for Cinnabon that they make their own dough.

What’s your experience like at other storage outfits? I only ask because your post is a bit like singing praises for Cinnabon that they make their own dough. The things that you mentioned are standard storage company activities. Checksum-all-the-things is a basic feature of a lot of file systems. If you can already set up your home computer to detect bitrot and alert you, you can bet big storage vendors do it. Keeping track of hard drive failure rates by vendor is normal. Storage companies publicly publish their own reports. The tiny 6-person IT operation I was in had a spreadsheet. Hell, I toured a friend’s friend’s major data center last year and he managed to find time to talk hard drive vendors. Now you. I get it — y’all make spreadsheets. There are a lot of smart people working on storage outside AWS and long before AWS existed.

pclmulqdq 3y ago
When I worked at Google in storage, we had our own figures of merit that showed that we were the best and Amazon's durability was trash in comparison to us. As far as I can tell, every cloud provider's object store is too durable to actually measure ("14 9's"), and it's not a problem.
- breckognize 3y ago
  9's are overblown. When cloud providers report that, they're really saying "Assuming random hard drive failure at the rates we've historically measured and how we quickly we detect and fix those failures, what's the mean time to data loss". But that's burying the lede. By far the greatest risks to a file's durability are: 1. Bugs (which aren't captured by a durability model). This is mitigated by deploying slowly and having good isolation between regions. 2. An act of God that wipes out a facility. The point of my comment was that it's not just about checksums. That's table stakes. The main driver of data loss for storage organizations with competent software is safety culture and physical infrastructure. My experience was that S3's safety culture is outstanding. In terms of physical separation and how "solid" the AZs are, AWS is overbuilt compared to the other players.
  - pclmulqdq 3y ago
    That was not how we treated the 9's at Google. Those had been tested through natural experiments (disasters). I was not at Google for the Clichy fire, but it wasn't the first datacenter fire Google experienced. I think your information about Google's data placement may be incorrect, or you may be mapping AWS concepts onto Google internal infrastructure in the wrong way.
    - fsociety 3y ago
      I would not lose sleep over storing data on GCS, but have heard from several Google Cloud folks that their concept of zones is a mirage at best.
      - pclmulqdq 3y ago
        Yeah, that's definitely true. Google sort of mapped an AWS concept onto its own cluster splits. However, there are enough regional-scale outages at all the major clouds that I don't personally place much stock in the idea of zones to begin with. The only way to get close to true 24/7 five-9's uptime with clouds is to be multi-region (and preferably multi-cloud).
        callalex 3y ago
        I have experienced many outages that were contained to a specific availability zone in AWS, from power failures to flooding to cable cuts. You are correct that 5 9’s still requires multi-region though.
      - kevincox 3y ago
        I think also Google as a whole has pretty good diversity. But Cloud customers demanded regions in big population centers and smaller countries where Google traditionally avoided due to cost reasons. This lead to less redundant sites that were often owned and/or operated by third parties. So in the US and Europe you can probably trust GCP zones quite literally. But other regions (I have heard lots of rumours in the APAC) they may not be quite as diverse as they appear.
- wubrr 3y ago
  > As far as I can tell, every cloud provider's object store is too durable to actually measure ("14 9's"), and it's not a problem. How can you tell that if it's not measurable? As far as I can tell the '11/14 9s' durability numbers are more or less completely made up. That's why AWS doesn't offer any actual durability SLA for S3, only a 99.9% availability SLA[0]. [0] https://aws.amazon.com/s3/sla/ https://aws.amazon.com/s3/sla/
  - flybarrel 3y ago
    buddy you are mixing availability and durability
- urda 3y ago
  Sorry I am not buying the personal anecdote as the public numbers from both orgs tell a different story. When reliability and long term support come up in conversation, Google is not a name to reach for.
  - pclmulqdq 3y ago
    Note that I said "durability," not any other reliability metric. GCP is pretty well-known for its outages and abysmal support. It's a reputation they want to change, but they did earn it. However, Google is very good at not losing data.
    - urda 3y ago
      Yup still not buying it. People who want their production stable, secure, and durable do not choose Google.
- deleted 3y ago
  [deleted]
fierro 3y ago
it's well known and not debatable that Cinnabon is fire
FooBarWidget 3y ago
"Checksum-all-the-things is a basic feature of a lot of file systems" "A lot"? Does anything but ZFS and maybe btrfs do this? Ext4 anf XFS — two very common filesystems — still don't have data checksums.
- Filligree 3y ago
  Bcachefs, and LVM also has a way to do it. Unfortunately I’m not aware of any filesystem that does it while maintaining the full bandwidth of a modern NVMe. Not even with the extra reads factored in; on ZFS I get 800 MB/s max.
  - EraYaN 3y ago
    ZFS should absolutely be able to go faster even with lz4 compression I get writes above 5 GB/s on a older 32-core EPYC CPU. And that is with mostly random and already compressed data. And that write speed is a limitation of the RAIDz2 on top of not the fastest drives (6 PCIe 3.0 intel U.2 ones from 2 years ago).
    - Filligree 3y ago
      Maybe I've configured something wrong, then. I'll do some tests.
4death4 3y ago
This was a few years ago, but blob storage on GCP had a global outage due to an outage in a single zone. That, among numerous other issues with GCP, lost my confidence entirely. Maybe it’s better now.
mauvehaus 3y ago
As a point of order: Not all Cinnabon locations make their own dough. The one I worked at the summer of 2001 made their own dough and frosting that summer, but switched to premade rolls partially that holiday season, and fully to premade rolls and frosting by the 2002 holiday season. Also: you had to be eighteen or older to operate the mixer. It was something like a 60 quart machine, and all the recipes were pre-programmed for time and power with pauses to change from the paddle to the whip if needed.
SlightlyLeftPad 3y ago
The beauty of proprietary systems is that the only information we can ever expect to get about how they’re built is the biased information we get from the builders of those systems.

pclmulqdq 3y ago

When I worked at Google in storage, we had our own figures of merit that showed that we were the best and Amazon's durability was trash in comparison to us. As far as I can tell, every cloud provider's object store is too durable to actually measure ("14 9's"), and it's not a problem.

breckognize 3y ago
9's are overblown. When cloud providers report that, they're really saying "Assuming random hard drive failure at the rates we've historically measured and how we quickly we detect and fix those failures, what's the mean time to data loss". But that's burying the lede. By far the greatest risks to a file's durability are: 1. Bugs (which aren't captured by a durability model). This is mitigated by deploying slowly and having good isolation between regions. 2. An act of God that wipes out a facility. The point of my comment was that it's not just about checksums. That's table stakes. The main driver of data loss for storage organizations with competent software is safety culture and physical infrastructure. My experience was that S3's safety culture is outstanding. In terms of physical separation and how "solid" the AZs are, AWS is overbuilt compared to the other players.
- pclmulqdq 3y ago
  That was not how we treated the 9's at Google. Those had been tested through natural experiments (disasters). I was not at Google for the Clichy fire, but it wasn't the first datacenter fire Google experienced. I think your information about Google's data placement may be incorrect, or you may be mapping AWS concepts onto Google internal infrastructure in the wrong way.
  - fsociety 3y ago
    I would not lose sleep over storing data on GCS, but have heard from several Google Cloud folks that their concept of zones is a mirage at best.
    - pclmulqdq 3y ago
      Yeah, that's definitely true. Google sort of mapped an AWS concept onto its own cluster splits. However, there are enough regional-scale outages at all the major clouds that I don't personally place much stock in the idea of zones to begin with. The only way to get close to true 24/7 five-9's uptime with clouds is to be multi-region (and preferably multi-cloud).
      - callalex 3y ago
        I have experienced many outages that were contained to a specific availability zone in AWS, from power failures to flooding to cable cuts. You are correct that 5 9’s still requires multi-region though.
    - kevincox 3y ago
      I think also Google as a whole has pretty good diversity. But Cloud customers demanded regions in big population centers and smaller countries where Google traditionally avoided due to cost reasons. This lead to less redundant sites that were often owned and/or operated by third parties. So in the US and Europe you can probably trust GCP zones quite literally. But other regions (I have heard lots of rumours in the APAC) they may not be quite as diverse as they appear.
wubrr 3y ago
> As far as I can tell, every cloud provider's object store is too durable to actually measure ("14 9's"), and it's not a problem. How can you tell that if it's not measurable? As far as I can tell the '11/14 9s' durability numbers are more or less completely made up. That's why AWS doesn't offer any actual durability SLA for S3, only a 99.9% availability SLA[0]. [0] https://aws.amazon.com/s3/sla/ https://aws.amazon.com/s3/sla/
- flybarrel 3y ago
  buddy you are mixing availability and durability
urda 3y ago
Sorry I am not buying the personal anecdote as the public numbers from both orgs tell a different story. When reliability and long term support come up in conversation, Google is not a name to reach for.
- pclmulqdq 3y ago
  Note that I said "durability," not any other reliability metric. GCP is pretty well-known for its outages and abysmal support. It's a reputation they want to change, but they did earn it. However, Google is very good at not losing data.
  - urda 3y ago
    Yup still not buying it. People who want their production stable, secure, and durable do not choose Google.
deleted 3y ago
[deleted]

fierro 3y ago

it's well known and not debatable that Cinnabon is fire

FooBarWidget 3y ago

"Checksum-all-the-things is a basic feature of a lot of file systems" "A lot"? Does anything but ZFS and maybe btrfs do this? Ext4 anf XFS — two very common filesystems — still don't have data checksums.

Filligree 3y ago
Bcachefs, and LVM also has a way to do it. Unfortunately I’m not aware of any filesystem that does it while maintaining the full bandwidth of a modern NVMe. Not even with the extra reads factored in; on ZFS I get 800 MB/s max.
- EraYaN 3y ago
  ZFS should absolutely be able to go faster even with lz4 compression I get writes above 5 GB/s on a older 32-core EPYC CPU. And that is with mostly random and already compressed data. And that write speed is a limitation of the RAIDz2 on top of not the fastest drives (6 PCIe 3.0 intel U.2 ones from 2 years ago).
  - Filligree 3y ago
    Maybe I've configured something wrong, then. I'll do some tests.

4death4 3y ago

This was a few years ago, but blob storage on GCP had a global outage due to an outage in a single zone. That, among numerous other issues with GCP, lost my confidence entirely. Maybe it’s better now.

mauvehaus 3y ago

As a point of order: Not all Cinnabon locations make their own dough. The one I worked at the summer of 2001 made their own dough and frosting that summer, but switched to premade rolls partially that holiday season, and fully to premade rolls and frosting by the 2002 holiday season. Also: you had to be eighteen or older to operate the mixer. It was something like a 60 quart machine, and all the recipes were pre-programmed for time and power with pauses to change from the paddle to the whip if needed.

SlightlyLeftPad 3y ago

The beauty of proprietary systems is that the only information we can ever expect to get about how they’re built is the biased information we get from the builders of those systems.