7 ms·

Call me crazy, because this is, perhaps it's their "Room 641a". The purpose of a system is what it does, no point arguing 'should' against reality, etc. They'v

Call me crazy, because this is, perhaps it's their "Room 641a". The purpose of a system is what it does, no point arguing 'should' against reality, etc. They've been charging a premium for, and marketing, "Availability" for decades at this point. I worked for a competitor and made a better product: it could endure any of the zones failing.

voxadam 1y ago
> perhaps it's their "Room 641a". For the uninitiated: https://en.wikipedia.org/wiki/Room_641A https://en.wikipedia.org/wiki/Room_641A
jf 1y ago
Interesting. Langley isn’t that far away
- deleted 1y ago
  [deleted]
nevir 1y ago
It's really not that nefarious. IAD datacenters have forever been the place where Amazon software developers implement services first (well before AWS was a thing). Multi-AZ support often comes second (more than you think; Amazon is a pragmatic company), and not every service is easy to make TRULY multi-AZ. And then other services depend on those services, and may also fall into the same trap. ...and so much of the tech/architectural debt gets concentrated into a single region.
- bravetraveler 1y ago
  Right, like I said: crazy. Anything production with certain other clouds must be multi-AZ. Both reinforced by culture and technical constraints. Sometimes BCDR/contract audits [zones chosen by a third party at random].
  - nevir 1y ago
    It sure is a blast when they decide to cut off (or simulate the loss of) a whole DC just to see what breaks, I bet :)
    - bravetraveler 1y ago
      The disconnect case was simple: breakage was as expected. The island was lost until we drew it on the map again. Things got really interesting when it was a full power-down and back on. Were the docs/tooling up to date? Tough bet. Much easier to fix BGP or whatever.
Anon1096 1y ago
It's possible that you really could endure any zone failure. But I take these claims people make all the time with a grain of salt, unless you're working on AWS scale (basically just 3 companies) and have actually run for years and seen every kind of failure mode claiming to be higher availability is not something that's able to be accurately evaluated. (I'm assuming by zone you mean the equivalent of an AWS region, with multiple connected datacenters)
- bravetraveler 1y ago
  Yes, equivalent. Did endure, repeatedly. Demonstrated to auditors to maintain compliance. They would pick the zone to cut off. We couldn't bias the test. Literal clockwork. I'll let people guess for the sport of it, here's the hint: there were at least 30 of them comprised of Real Datacenters. Thanks for the doubt, though. Implied or otherwise.
  - pinkmuffinere 1y ago
    Just letting you know how this response looks to other people -- Anon1096 raises legitimate objections, and their post seems very measured in their concerns, not even directly criticizing you. But your response here is very defensive, and a bit snarky. Really I don't think you even respond directly to their concerns, they say they'd want to see scale equivalent to AWS because that's the best way to see the wide variety of failure modes, but you mostly emphasize the auditors, which is good but not a replacement for the massive real load and issues that come along with it. It feels miscalibrated to Anon's comment. As a result, I actually trust you less. If you can respond to Anon's comment without being quite as sassy, I think you'd convince more people.
    - bravetraveler 1y ago
      I appreciate the feedback, truly. Defensive and snarky are both fair, though I'm not trying to convince. The business and practices exist, today. At risk of more snark [well-intentioned]: Clouds aren't the Death Star, they don't have to have an exhaust port. It's fair the first one does... for a while.
stronglikedan 1y ago
Was that competitor priced competitively with AWS? I think of the project management triangle here - good, fast, or cheap - pick two. AWS would be fast and cheap.
- bravetraveler 1y ago
  Yes, good point. Pricing is a bit higher. As another reply pointed out: there's ~three that work on the same scale. This was one, another hint I guess: it's mostly B2B. Normal people don't typically go there.
  - franktankbank 1y ago
    I'm guessing Azure which may technically have greater resilience but has dogshit support and UX.
    - seniorThrowaway 1y ago
      Azure, from my experience with it has stuff go down a lot and degrades even more. Seems to either not admit the degradation happened or rely on 1000 pages of fine print SLA docs to prove you don't get any credits for it. I suppose that isn't the same as "lose a region resiliency" so it could still be them given the poster said it is B2B focused and Azure is subject to a lot of exercises like this from it's huge enterprise customers. FWIW I worked as a IaC / devops engineer with the largest tenant in one of the non-public Azure clouds.
- immibis 1y ago
  AWS is not cheap. AWS is one to two orders of magnitude more expensive than DIY.
  - ranger_danger 1y ago
    My $3/mo AWS instance is far cheaper than any DIY solution I could come up with, especially when I have to buy the hardware and supply the power/network/storage/physical space. Not to mention it's not worth my time to DIY something like that in the first place. There can be other valid usecases than your own.

voxadam 1y ago

> perhaps it's their "Room 641a". For the uninitiated: https://en.wikipedia.org/wiki/Room_641A https://en.wikipedia.org/wiki/Room_641A

jf 1y ago

Interesting. Langley isn’t that far away

deleted 1y ago
[deleted]

nevir 1y ago

It's really not that nefarious. IAD datacenters have forever been the place where Amazon software developers implement services first (well before AWS was a thing). Multi-AZ support often comes second (more than you think; Amazon is a pragmatic company), and not every service is easy to make TRULY multi-AZ. And then other services depend on those services, and may also fall into the same trap. ...and so much of the tech/architectural debt gets concentrated into a single region.

bravetraveler 1y ago
Right, like I said: crazy. Anything production with certain other clouds must be multi-AZ. Both reinforced by culture and technical constraints. Sometimes BCDR/contract audits [zones chosen by a third party at random].
- nevir 1y ago
  It sure is a blast when they decide to cut off (or simulate the loss of) a whole DC just to see what breaks, I bet :)
  - bravetraveler 1y ago
    The disconnect case was simple: breakage was as expected. The island was lost until we drew it on the map again. Things got really interesting when it was a full power-down and back on. Were the docs/tooling up to date? Tough bet. Much easier to fix BGP or whatever.

Anon1096 1y ago

It's possible that you really could endure any zone failure. But I take these claims people make all the time with a grain of salt, unless you're working on AWS scale (basically just 3 companies) and have actually run for years and seen every kind of failure mode claiming to be higher availability is not something that's able to be accurately evaluated. (I'm assuming by zone you mean the equivalent of an AWS region, with multiple connected datacenters)

bravetraveler 1y ago
Yes, equivalent. Did endure, repeatedly. Demonstrated to auditors to maintain compliance. They would pick the zone to cut off. We couldn't bias the test. Literal clockwork. I'll let people guess for the sport of it, here's the hint: there were at least 30 of them comprised of Real Datacenters. Thanks for the doubt, though. Implied or otherwise.
- pinkmuffinere 1y ago
  Just letting you know how this response looks to other people -- Anon1096 raises legitimate objections, and their post seems very measured in their concerns, not even directly criticizing you. But your response here is very defensive, and a bit snarky. Really I don't think you even respond directly to their concerns, they say they'd want to see scale equivalent to AWS because that's the best way to see the wide variety of failure modes, but you mostly emphasize the auditors, which is good but not a replacement for the massive real load and issues that come along with it. It feels miscalibrated to Anon's comment. As a result, I actually trust you less. If you can respond to Anon's comment without being quite as sassy, I think you'd convince more people.
  - bravetraveler 1y ago
    I appreciate the feedback, truly. Defensive and snarky are both fair, though I'm not trying to convince. The business and practices exist, today. At risk of more snark [well-intentioned]: Clouds aren't the Death Star, they don't have to have an exhaust port. It's fair the first one does... for a while.

stronglikedan 1y ago

Was that competitor priced competitively with AWS? I think of the project management triangle here - good, fast, or cheap - pick two. AWS would be fast and cheap.

bravetraveler 1y ago
Yes, good point. Pricing is a bit higher. As another reply pointed out: there's ~three that work on the same scale. This was one, another hint I guess: it's mostly B2B. Normal people don't typically go there.
- franktankbank 1y ago
  I'm guessing Azure which may technically have greater resilience but has dogshit support and UX.
  - seniorThrowaway 1y ago
    Azure, from my experience with it has stuff go down a lot and degrades even more. Seems to either not admit the degradation happened or rely on 1000 pages of fine print SLA docs to prove you don't get any credits for it. I suppose that isn't the same as "lose a region resiliency" so it could still be them given the poster said it is B2B focused and Azure is subject to a lot of exercises like this from it's huge enterprise customers. FWIW I worked as a IaC / devops engineer with the largest tenant in one of the non-public Azure clouds.
immibis 1y ago
AWS is not cheap. AWS is one to two orders of magnitude more expensive than DIY.
- ranger_danger 1y ago
  My $3/mo AWS instance is far cheaper than any DIY solution I could come up with, especially when I have to buy the hardware and supply the power/network/storage/physical space. Not to mention it's not worth my time to DIY something like that in the first place. There can be other valid usecases than your own.