5 ms·

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!" https://www.cbsnews.com/news/cloud-computing-loudoun-county-virginia/ https://www.cbsnews.com/news/cloud-computing-loudoun-county-...

Betelbuddy 9d ago
Clearly since the MBAs took over AWS standards are not anymore what they used to be. That marketing guy should not be talking to the press, as he does not have the skills, and if somebody happens to say...our data center we wont lose any data if we have an issue, without qualifying it will depend on what quality of service, and usage of our services you setup ...is the type of technical answer that should make a hiring interview stop at the moment. He is also violating an enormous amount of compliance requirements, by disclosing the location of the data center, and having strange people inside making a tour. Did he vet the crew and their accompanying party? Did one of them accidentally left some kind of device within the insider perimeter? There at least one or two ISO certifications he is violating there. As customer I would be asking questions... AWS always made very clear they wont copy your data to another region as only you know what your compliance and data residency requirements are. But at the same time they always said, its up to you to come your with your disaster recovery strategy based on your project requirements. And it has always been the case copying your critical data to another region is one of the first things on your check list. And their Well Architected Framework and other docs make this plenty clear: "It is a good practice to always make backups of your data, and copy these to another site (such as another AWS Region)." Also... "All DR strategies require that data sources are backed up within the AWS Region, and then those backups are copied to the recovery Region." And also for single-Region / Multi-AZ architectures: "Where possible, you should also copy data backups to another AWS Region as an additional layer of protection." "AWS Architecture Blog — Disaster Recovery Architecture on AWS, Part II" has a whole section named "Backup to another AWS Region": "By copying your data to another Region, you can handle the largest scope of disasters." https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-ii-backup-and-restore-with-rapid-recovery/ https://aws.amazon.com/blogs/architecture/disaster-recovery-... Or "Creating backup copies across AWS Regions" - https://docs.aws.amazon.com/aws-backup/latest/devguide/cross-region-backup.html https://docs.aws.amazon.com/aws-backup/latest/devguide/cross... This whole thread of people literally saying , "on no I trusted them...I did not know they could lose my data", with no technical context...is the the kind of incompetence I would expect from a generation raised on vibe coding and llm prompt driven miseducation...
- raverbashing 9d ago
  > That marketing guy should not be talking to the press, as he does not have the skills They have way more skills than the technical folk in doing that
sparkling 9d ago
He forgot to add "if it's Multi-AZ" ;)
- bdangubic 9d ago
  multi-AZ doesn’t help against multi-AZ drones :)
  - dzonga 9d ago
    black swan events. this is one of one of those things - were in the current era either a cloud provider should provide automatic backups in another geographic zone. if you're in us-east, then your back-ups should ideally be in eu-west + africa for redundancy.
    - boelboel 9d ago
      It's not really a black swan event just an unlikely one. If it were one it would not have been something people talked about before.
    - Twirrim 9d ago
      That's absolutely something you can configure your S3 bucket to do if you want (I have one of mine replicating elsewhere). The amount of data S3 stores "automatically replicating" to other geographical locations would make things prohibitively expensive, especially when you consider the daily delta, and how much of that is ephemeral or frequently mutated data that is stored. The bandwidth costs alone would be eye-watering, let alone the storage costs. S3 cannot make any automated decision about whether data is, or isn't important, and if they did they'd only open themselves up to lawsuits if they guessed wrong. That's why it's made an option for the end user to enable replication if they want to, or choose to replicate their own data.
      - donavanm 9d ago
        > The amount of data S3 stores "automatically replicating" to other geographical locations would make things prohibitively expensive It did work like this! And it was! My recollection is that the first S3 was out of SEA and had no user concept of region. Then “VDC” was added in virginia. That provided API endpoints in what became us-east-1. A bucket could be accessed from either location, and the original intent was for object store to replicate between them. By the time dub/eu-west-1 came along that was obviously not tenable; itd take 10s of gbs to replicate. So S3 became regional. But the original sea/vdc deployments still had shared APIs and data in both regions. Your object would be stored in the region of the API you geolocated to via DNS, but read from either. _eventually_ all the data migrated to IAD, but those API endpoints were transparently proxying across the continent until 2013 or so. And of course glacier had much more interesting takes on this with cross dc/az/region erasure encoding. But i dont think any of the wacky multi dimensional cross region stuff ever materialised in practice. PS: hi!
krick 9d ago
That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.
- houssc 9d ago
  The caveat is always "if you're using the service correctly" which is not necessarily free. Meaning taking advantage of multiple geo zones, building in redundancy to your stack, etc. Like everything he said is possible if your technology stack living in AWS was designed to survive it. Everyone who has ever had the "we lost your data" email from AWS knows at the end of the day the cloud is just someone else's data center with neat provisioning tools and services.
  - jacquesm 9d ago
    The problem is that lots of people seem to be under the impression that they are doing it right because they are using AWS. They don't realize that AWS is a toolbox, not a 'ready made solution for redundancy against all catastrophes you are possibly exposed to'. They use that to their advantage by pricing such solutions at a level that people will either pay through the nose or will be left without recourse when AWS loses their data. It's stupid, but at the same time these beliefs are surprisingly wide spread.
    - jaggederest 9d ago
      If I were a bit more bloody-minded I would launch a service for vibe-coded apps that, under the hood, did everything "the right way" and just charged a flat fee + percent on the underlying. I feel like if this was done correctly it would eat a bunch of the market, but I question how many people are actually willing to pay for "the right way". The last time I had that experience it was with Heroku which was quite a leaky abstraction.
      - 9d ago
echoangle 9d ago
Isn’t the problem that multiple datacenters in one zone were blown up?
- nobodyandproud 9d ago
  They had nine years and more money than god to build redundancy in an unstable region.
  - nomel 9d ago
    If it's an unstable region, then it might not result in a good return, especially since a blown up data center is 100% loss.
    - lostlogin 9d ago
      This is focusing in too close. Bezos has been working with Trump. The instability has been hugely exacerbated by Trump. The issue lies at home.
      - jojobas 9d ago
        Yeah, Iran blowing up half the Middle East? Totally Bezos' fault.
  - wmf 9d ago
    They did build redundancy but most of it was bombed.
    - recursivegirth 9d ago
      [flagged]
      - anon48293 9d ago
        [flagged]
jasonfarnon 9d ago
that quote is definitely making its way into a lawsuit
- eastbound 9d ago
  Yes. It's super hard to answer interviews or customer demos. Every sentence you say should include all preconditions that were said in previous answers, within the same context, because you may be quoted. You should think in live about all possible contexts in which your app might be used, and your speech should be as detailed as a contract. Obviously here, he should have mentionned that they can recover a hit on a single data center, provided the customer chose multi-AZ hosting. That's probably why companies run their ads on "This watch is a legacy for your children" rather than any material claim.
steveBK123 9d ago
There's something reassuring in this for me. There's a lot of magic & handwaving from hyperscalers like AWS about redundancy. I always wondered about some of the engineering to make this absolutely (and literally) bullet proof. At the end of the day most of their answers when you push hard enough involved paying 2-3x to run everything across multiple zones/regions, and lots of awareness in your application to handle this. In any case, I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.
- antonvs 9d ago
  > I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc. Doesn’t really apply, because the only reason data was lost is because customers chose not to replicate it to other regions, either because of legal data residency requirements, cost, or just not bothering. If Skynet wants to make sure it’s backed up, none of that prevents it from doing so. Although it would be amusing if Skynet was stopped by a billing alert when it tries to copy itself to another region.
  - htrp 9d ago
    aws egress fees: the real hero we didn't know we needed
  - mitxela 9d ago
    AWS promises to keep data safe without requiring cross region redundancy. If you choose not to trust them that's valid but then why use them at all?
    - Betelbuddy 9d ago
      >> AWS promises to keep data safe without requiring cross region redundancy. A perfect example of a claim they never made.
      - mitxela 9d ago
deleted 9d ago
[deleted]
Dban1 9d ago
But AI chatbot advised him that was the case!
ExoticPearTree 9d ago
Unless you have a synchronous like setup where you don’t acknowledge data writes unless the remote has aconowledged them first, you will lose data in case your datacenter is hit by a warhead. Now, there are a few things to consider: - AWS best practices recommend multiple AZs for workloads and cross-region backups for things like databases and other “stateful” data - You have to read the fine-print on what AWS offers in terms of recovery: do they reffer to their own infrastructure when they say “you won”t notice” or your data When Google’s Paris colocation facility was flooded and all AZs there went dark, they sent an email saying “restore from backup in another region and if we can restore your data, we will make it availbale to you”. They did not even issue credits for the downtime.
- Betelbuddy 9d ago
  >> and all AZs there went dark, An extraordinary statement itself that shows the difference between a proper cloud where they AZs are at least 60 to 100 miles apart...and Google or Microsoft... pretend clouds...where those AZs are just firewalls across the same data center...
- paulddraper 9d ago
  All the AZs flooded simultaneously?! How big was this flood?
  - ExoticPearTree 9d ago
    Well, this is how the world found out that Google AZs are just different rooms in the same building on the same floor.
    - antonvs 8d ago
      That’s nothing, I have three AZs in my laptop.
  - everfrustrated 9d ago
    Azure and GCP definitions of AZ allow a single datacenter to have multiple AZs. AWS has by far the strictest definition.
jasonkhanlar 9d ago
articleAuthorId: 30dacdbc-6a8a-11e2-9d12-0018fe8a00b0 articleAuthorName: cbsnews (hidden byline) articleSecondaryAuthors: n/a articleEditors: n/a Is the author a human or machine? Google shows 1 result for "30dacdbc-6a8a-11e2-9d12-0018fe8a00b0" and Brave Search shows 5 results.
cowsandmilk 9d ago
1. This wasn’t a strike on a single data center, it was strikes on many data centers. 2. Since that interview, AWS has started selling versions of their storage that isn’t redundant. It is not surprising that when AWS sells non-redundant storage that some data is not recoverable.
- varispeed 9d ago
  > This wasn’t a strike on a single data center, it was strikes on many data centers. He didn't say "but if you hit many data centres then there is a problem". The premise was if you hit data centre, user won't notice, without caveat that there is a limit.
  - caymanjim 9d ago
    That's an absurd interpretation.
  - JulianWasTaken 9d ago
    > Amazon's Matt Wood isn't worried: "If something does happen or we have a power event or there's a flood in one specific location, that data is held redundantly in other locations as well." > Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Seems pretty silly to argue, but he certainly did say "one specific data center", and I don't think anyone even non-technical will conclude "it's safe if they all go down at once" from this statement.
    - varispeed 9d ago
      The Iranians hit one specific data centre at a time, no?
      - JulianWasTaken 8d ago
        Correct. Again, I don't think the implication you're making (that someone reading the original comment will think that is safe from a data recovery standpoint) is one any reasonable reader will make.
amelius 9d ago
The whole "flawless victory" thing isn't aging well either.
sporkland 9d ago
Since S3 supports regionally and zonally locked buckets it seems like any promise around this would have to except those buckets where that was configured.

Betelbuddy 9d ago

Clearly since the MBAs took over AWS standards are not anymore what they used to be. That marketing guy should not be talking to the press, as he does not have the skills, and if somebody happens to say...our data center we wont lose any data if we have an issue, without qualifying it will depend on what quality of service, and usage of our services you setup ...is the type of technical answer that should make a hiring interview stop at the moment. He is also violating an enormous amount of compliance requirements, by disclosing the location of the data center, and having strange people inside making a tour. Did he vet the crew and their accompanying party? Did one of them accidentally left some kind of device within the insider perimeter? There at least one or two ISO certifications he is violating there. As customer I would be asking questions... AWS always made very clear they wont copy your data to another region as only you know what your compliance and data residency requirements are. But at the same time they always said, its up to you to come your with your disaster recovery strategy based on your project requirements. And it has always been the case copying your critical data to another region is one of the first things on your check list. And their Well Architected Framework and other docs make this plenty clear: "It is a good practice to always make backups of your data, and copy these to another site (such as another AWS Region)." Also... "All DR strategies require that data sources are backed up within the AWS Region, and then those backups are copied to the recovery Region." And also for single-Region / Multi-AZ architectures: "Where possible, you should also copy data backups to another AWS Region as an additional layer of protection." "AWS Architecture Blog — Disaster Recovery Architecture on AWS, Part II" has a whole section named "Backup to another AWS Region": "By copying your data to another Region, you can handle the largest scope of disasters." https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-ii-backup-and-restore-with-rapid-recovery/ https://aws.amazon.com/blogs/architecture/disaster-recovery-... Or "Creating backup copies across AWS Regions" - https://docs.aws.amazon.com/aws-backup/latest/devguide/cross-region-backup.html https://docs.aws.amazon.com/aws-backup/latest/devguide/cross... This whole thread of people literally saying , "on no I trusted them...I did not know they could lose my data", with no technical context...is the the kind of incompetence I would expect from a generation raised on vibe coding and llm prompt driven miseducation...

raverbashing 9d ago
> That marketing guy should not be talking to the press, as he does not have the skills They have way more skills than the technical folk in doing that

sparkling 9d ago

He forgot to add "if it's Multi-AZ" ;)

bdangubic 9d ago
multi-AZ doesn’t help against multi-AZ drones :)
- dzonga 9d ago
  black swan events. this is one of one of those things - were in the current era either a cloud provider should provide automatic backups in another geographic zone. if you're in us-east, then your back-ups should ideally be in eu-west + africa for redundancy.
  - boelboel 9d ago
    It's not really a black swan event just an unlikely one. If it were one it would not have been something people talked about before.
  - Twirrim 9d ago
    That's absolutely something you can configure your S3 bucket to do if you want (I have one of mine replicating elsewhere). The amount of data S3 stores "automatically replicating" to other geographical locations would make things prohibitively expensive, especially when you consider the daily delta, and how much of that is ephemeral or frequently mutated data that is stored. The bandwidth costs alone would be eye-watering, let alone the storage costs. S3 cannot make any automated decision about whether data is, or isn't important, and if they did they'd only open themselves up to lawsuits if they guessed wrong. That's why it's made an option for the end user to enable replication if they want to, or choose to replicate their own data.
    - donavanm 9d ago
      > The amount of data S3 stores "automatically replicating" to other geographical locations would make things prohibitively expensive It did work like this! And it was! My recollection is that the first S3 was out of SEA and had no user concept of region. Then “VDC” was added in virginia. That provided API endpoints in what became us-east-1. A bucket could be accessed from either location, and the original intent was for object store to replicate between them. By the time dub/eu-west-1 came along that was obviously not tenable; itd take 10s of gbs to replicate. So S3 became regional. But the original sea/vdc deployments still had shared APIs and data in both regions. Your object would be stored in the region of the API you geolocated to via DNS, but read from either. _eventually_ all the data migrated to IAD, but those API endpoints were transparently proxying across the continent until 2013 or so. And of course glacier had much more interesting takes on this with cross dc/az/region erasure encoding. But i dont think any of the wacky multi dimensional cross region stuff ever materialised in practice. PS: hi!

krick 9d ago

That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.

houssc 9d ago
The caveat is always "if you're using the service correctly" which is not necessarily free. Meaning taking advantage of multiple geo zones, building in redundancy to your stack, etc. Like everything he said is possible if your technology stack living in AWS was designed to survive it. Everyone who has ever had the "we lost your data" email from AWS knows at the end of the day the cloud is just someone else's data center with neat provisioning tools and services.
- jacquesm 9d ago
  The problem is that lots of people seem to be under the impression that they are doing it right because they are using AWS. They don't realize that AWS is a toolbox, not a 'ready made solution for redundancy against all catastrophes you are possibly exposed to'. They use that to their advantage by pricing such solutions at a level that people will either pay through the nose or will be left without recourse when AWS loses their data. It's stupid, but at the same time these beliefs are surprisingly wide spread.
  - jaggederest 9d ago
    If I were a bit more bloody-minded I would launch a service for vibe-coded apps that, under the hood, did everything "the right way" and just charged a flat fee + percent on the underlying. I feel like if this was done correctly it would eat a bunch of the market, but I question how many people are actually willing to pay for "the right way". The last time I had that experience it was with Heroku which was quite a leaky abstraction.
    - 9d ago

echoangle 9d ago

Isn’t the problem that multiple datacenters in one zone were blown up?

nobodyandproud 9d ago
They had nine years and more money than god to build redundancy in an unstable region.
- nomel 9d ago
  If it's an unstable region, then it might not result in a good return, especially since a blown up data center is 100% loss.
  - lostlogin 9d ago
    This is focusing in too close. Bezos has been working with Trump. The instability has been hugely exacerbated by Trump. The issue lies at home.
    - jojobas 9d ago
      Yeah, Iran blowing up half the Middle East? Totally Bezos' fault.
- wmf 9d ago
  They did build redundancy but most of it was bombed.
  - recursivegirth 9d ago
    [flagged]
    - anon48293 9d ago
      [flagged]

jasonfarnon 9d ago

that quote is definitely making its way into a lawsuit

eastbound 9d ago
Yes. It's super hard to answer interviews or customer demos. Every sentence you say should include all preconditions that were said in previous answers, within the same context, because you may be quoted. You should think in live about all possible contexts in which your app might be used, and your speech should be as detailed as a contract. Obviously here, he should have mentionned that they can recover a hit on a single data center, provided the customer chose multi-AZ hosting. That's probably why companies run their ads on "This watch is a legacy for your children" rather than any material claim.

steveBK123 9d ago

There's something reassuring in this for me. There's a lot of magic & handwaving from hyperscalers like AWS about redundancy. I always wondered about some of the engineering to make this absolutely (and literally) bullet proof. At the end of the day most of their answers when you push hard enough involved paying 2-3x to run everything across multiple zones/regions, and lots of awareness in your application to handle this. In any case, I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.

antonvs 9d ago
> I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc. Doesn’t really apply, because the only reason data was lost is because customers chose not to replicate it to other regions, either because of legal data residency requirements, cost, or just not bothering. If Skynet wants to make sure it’s backed up, none of that prevents it from doing so. Although it would be amusing if Skynet was stopped by a billing alert when it tries to copy itself to another region.
- htrp 9d ago
  aws egress fees: the real hero we didn't know we needed
- mitxela 9d ago
  AWS promises to keep data safe without requiring cross region redundancy. If you choose not to trust them that's valid but then why use them at all?
  - Betelbuddy 9d ago
    >> AWS promises to keep data safe without requiring cross region redundancy. A perfect example of a claim they never made.
    - mitxela 9d ago

deleted 9d ago

[deleted]

Dban1 9d ago

But AI chatbot advised him that was the case!

ExoticPearTree 9d ago

Unless you have a synchronous like setup where you don’t acknowledge data writes unless the remote has aconowledged them first, you will lose data in case your datacenter is hit by a warhead. Now, there are a few things to consider: - AWS best practices recommend multiple AZs for workloads and cross-region backups for things like databases and other “stateful” data - You have to read the fine-print on what AWS offers in terms of recovery: do they reffer to their own infrastructure when they say “you won”t notice” or your data When Google’s Paris colocation facility was flooded and all AZs there went dark, they sent an email saying “restore from backup in another region and if we can restore your data, we will make it availbale to you”. They did not even issue credits for the downtime.

Betelbuddy 9d ago
>> and all AZs there went dark, An extraordinary statement itself that shows the difference between a proper cloud where they AZs are at least 60 to 100 miles apart...and Google or Microsoft... pretend clouds...where those AZs are just firewalls across the same data center...
paulddraper 9d ago
All the AZs flooded simultaneously?! How big was this flood?
- ExoticPearTree 9d ago
  Well, this is how the world found out that Google AZs are just different rooms in the same building on the same floor.
  - antonvs 8d ago
    That’s nothing, I have three AZs in my laptop.
- everfrustrated 9d ago
  Azure and GCP definitions of AZ allow a single datacenter to have multiple AZs. AWS has by far the strictest definition.

jasonkhanlar 9d ago

articleAuthorId: 30dacdbc-6a8a-11e2-9d12-0018fe8a00b0 articleAuthorName: cbsnews (hidden byline) articleSecondaryAuthors: n/a articleEditors: n/a Is the author a human or machine? Google shows 1 result for "30dacdbc-6a8a-11e2-9d12-0018fe8a00b0" and Brave Search shows 5 results.

cowsandmilk 9d ago

1. This wasn’t a strike on a single data center, it was strikes on many data centers. 2. Since that interview, AWS has started selling versions of their storage that isn’t redundant. It is not surprising that when AWS sells non-redundant storage that some data is not recoverable.

varispeed 9d ago
> This wasn’t a strike on a single data center, it was strikes on many data centers. He didn't say "but if you hit many data centres then there is a problem". The premise was if you hit data centre, user won't notice, without caveat that there is a limit.
- caymanjim 9d ago
  That's an absurd interpretation.
- JulianWasTaken 9d ago
  > Amazon's Matt Wood isn't worried: "If something does happen or we have a power event or there's a flood in one specific location, that data is held redundantly in other locations as well." > Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Seems pretty silly to argue, but he certainly did say "one specific data center", and I don't think anyone even non-technical will conclude "it's safe if they all go down at once" from this statement.
  - varispeed 9d ago
    The Iranians hit one specific data centre at a time, no?
    - JulianWasTaken 8d ago
      Correct. Again, I don't think the implication you're making (that someone reading the original comment will think that is safe from a data recovery standpoint) is one any reasonable reader will make.

amelius 9d ago

The whole "flawless victory" thing isn't aging well either.

sporkland 9d ago

Since S3 supports regionally and zonally locked buckets it seems like any promise around this would have to except those buckets where that was configured.