6 ms·

How when AWS was down, we were not

203 points by mooreds 1y ago

tptacek 1y ago
This is a rare case where the original bait-y title is probably better than the de-bait-ified title, because the actual article is much less of a brag and much more of an actual case study.
- dang 1y ago
  Re-how'd, plus I've resisted the temptation to insert a comma that feels missing to me.
  - tptacek 1y ago
    "How?! When AWS was down: we were not!"
  - wparad 1y ago
    I spent a long time, trying to figure out, what the title of the article, should be. I'm terrible at SEO and generating click-bait titles, it is unfortunately, what, it, is.
    - dang 1y ago
      You did fine! The title is clear. I was just being playful.
pinkmuffinere 1y ago
> During this time, us-east-1 was offline, and while we only run a limited amount of infrastructure in the region, we have to run it there because we have customers who want it there > [Our service can only go down] five minutes and 15 seconds per year. I don't have much experience in this area, so please correct me if I'm mistaken: Don't these two quotes together imply that they have failed to deliver on their SLA for the subset of their customers that want their service in us-east-1? I understand the customers won't be mad at them in this case, since us-east-1 itself is down, but I feel like their title is incorrect. Some subset of their service is running on top of AWS. When AWS goes down, that subset of their service is down. When AWS was down, it seems like they were also down for some customers.
- PaulRobinson 1y ago
  The bulk of the article discusses their failover strategy, where they detect failures in a region and how they route requests to a backup region, and how to deal with data consistency and cost issues arising from that.
- loloquwowndueo 1y ago
  Depends on what the SLA phrasing is - us-east-1 affinity is a requirement put forth by some customers so I would totally expect the SLA to specifically state it’s subject to us-east-1 availability. Essentially these customers are opting out of Authress’s fault-tolerant infrastructure and the SLA should be clear about that.
  - dylan604 1y ago
    As TFA states, we have to offer services in that region because that's where some users are as well. However, the core of services are not in that region. I have also suggested when the time comes for offering SLAs, that there is explicit wording exempting us-east-1.
- wparad 1y ago
  It's a good point. We don't actually commit to running infrastructure in one specific AWS region. Customers can't request that the infra runs exactly in us-east-1, but they can request that it runs in "Eastern United States". The problem is that with scenarios that might require VPC peering or low latency connections, we can't just run the infrastructure in us-east-2 and commit to never having a problem. For the same reason, what happens if us-east-2 were to have an incident. We have to assume that our customers need it in a relatively close region, and that at the same time need to plan for the contingency that region can be down. Then there are the customer's users to think of as well. In some cases, those users might be globally dispersed, even if the customer infrastructure is only one major location. So while it would be nice to claim "well you were also down at that moment", in practices customer's users will notice, and realistically, we want to make sure we aren't impeding remediation on their side. That is, even if a customer says "use us-east-1", and then us-east-1 is down, it can't look that way to the customer. This gets a lot more complicated, when the services that we are providing may be impacted differently. Consider us-east-1 dynamoDB down, but everything else was still working. Partial failure modes are much harder to deal with.
deleted 1y ago
[deleted]
sharklasers123 1y ago
Is there not an inherent risk using an AWS service (Route 53) to do the health check? Wouldn’t it make more sense to use a different cloud provider for redundancy?
- indigodaddy 1y ago
  Had the same thought, eg if things are really down can it even do the check etc
  - hnuser123456 1y ago
    Ask some friends and family if you can install an RPi on their home network that monitors your service.
- wparad 1y ago
  If the check can't be done, then everything stays stable, so I'm guessing the question is, "What happens if Route 53 does the check and incorrectly reports the result?" In that case, no matter what we are using there is going to be a critical issue. I think the best I could suggest at that point would be to have records in your zone that round robin different cloud providers, but that comes with its own challenges. I believe there are some articles sitting around regarding how AWS plans for failure and the fallback mechanism actually reduces load on the system rather than makes it worse. I think it would require in-depth investigation on the expected failover mode to have a good answer there. For instance, just to make it more concrete, what sort of failure mode are you expecting to happen with the Route 53 health check? Depending on that there could be different recommendations.
  - indigodaddy 1y ago
    Have you considered the scenario of "everything is so dead in aws", that the check doesn't happen, plus the backends are dead too (this is assuming the backend services live in aws as well) ? But I'd guess in that case you'd know quickly enough from supplementary alerting (you guys don't seem the type to not have some sort of awesome monitoring in place) and you have a different/worse DR problem on your hands. As far as the OP's point though, I'm going to probably assume that the health checks need to stay within/from AWS because 3rd party health checks could taint/dilute the point of the in-house AWS HC service to begin with.
oldpersonintx2 1y ago
[dead]
indigodaddy 1y ago
Back in the day (10-12 years ago) at a telecom/cable we accomplished this with F5 Big IP GSLB DNS (and later migrated to A10's GSLB equivalent devices) as the auth DNS server for services/zones that required or were suitable for HA. (I can't totally remember but I'm guessing we must have had a pretty low TTL for this). Had no idea that Route 53 had this sort of functionality
- wparad 1y ago
  Maybe I should have titled the article "AWS Route53 HealthChecks are amazing" :)
- indigodaddy 1y ago
  Speaking of F5 Big IP DNS devices, does anyone know of any auth DNS software solution for GSLB/health checking for DNS (I guess excluding Route 53 or other cloud/SaaS). Last I looked all I could find was the polaris-gslb addon for PowerDNS, but the GitHub for that has no activity in 8 years.
  - loadbalancer 1y ago
    We've been using Polaris for the Loadbalancer.org GSLB for a few years now, and we've found it fast and stable. Its so simple its never needed any updates: https://www.loadbalancer.org/blog/gslb-why-global-server-load-balancers-dont-always-suck-polaris-gslb/ https://www.loadbalancer.org/blog/gslb-why-global-server-loa... Although we did patch a dynamic health check a while back, which will be open source of course. But I'll get someone to check if we actually gave it back to the community or not...
iso1631 1y ago
I'm interested in how they measure that downtime. If you're down for 200 milliseconds, does that accumulate. How do you even measure that you're down for 200ms. (For what it's worth, for some of my services, 200ms is certainly an impact, not as bad as 2 seconds out outage but still noticable and reportable)
- wparad 1y ago
  Good catch. The truth is, while we track downtime for incident reporting, it's much more correct to actually be tracking the number of requests that result in a failure. Our SLAs are based on request volume, and not specifically time. Most customers don't have perfect sustained usage. Being down when they aren't running is irrelevant to everyone. This is where the grey failures can come into play. It's really hard to tell, often impossible to know what the impact of an incident is to a customer, even if you know you are having an incident, without them telling you. In order to know that you are "down", our edge of the HTTP request would need to be able to track requests. For us that is CloudFront, but if there is an issue before that, at DNS, at network level, etc... we just can't know what the actual impact is. As far as measuring how you are down. We can pretty accurately know the list of failures that are happening, (when we can know), and what the results are. That's because most components are behind cloudfront in any case. And if cloudfront isn't having a problem, we'll have telemetry that tells us what the HTTP request/response status codes and connection completions look like. Then it's a matter of measuring from our first detection to the actual remediation being deployed (assuming there is one). Another thing that helps here is that we have multiple other products that also use Authress, and we can run technology in other regions that can report this information, for those accounts (obviously can't be for all customers), which can help us identify with additional accuracy, but is often unnecessary.
- smadge 1y ago
  I think a lot of web services talk about reliability in terms of uptime (e.g. down for less than 5 minutes a year) but in reality operate on failure ratios (less than 0.001% of request to our service fail).
  - wparad 1y ago
wparad 1y ago
Hey, I wrote that article! I'll try to add comments and answer questions where I can. - Warren
- ckozlowski 1y ago
  Hi Warren! I'm Chris, and I'm with AWS, where among other things, I work on the Well-Architected Framework. Would you be willing to talk with us? You can reach me at kozlowck@amazon.com. Thanks! Edit: This is a fantastic write-up by the way!
  - wparad 1y ago
    Thank you!
rdoherty 1y ago
This is probably one of the best summarizations of the past 10 years of my career in SRE. Once your systems get complex enough, something is always broken and you have to prepare for that. Detection & response become just as critical as pre-deploy testing. I do worry about all the automation being another failure point, along with the IaC stuff. That is all software too! How do you update that safely? It's turtles all the way down!
- evanmoran 1y ago
  Iac is definitely a failure point, but the manual alternative is much worse! I’ve had a lot of benefit from using pulumi, simply because the code can be more compact than the terraform hcl was. For example, for the fall over regions (from the article) you could make a pulumi function that parameterizes only the n things that are different per fall over env and guarantee / verify the scripts are nearly identical. Of course, many people use modules / terragrunt for similar reasons, but it ends up being quite powerful.
  - spyspy 1y ago
    If you do use terraform, for the love of god do NOT use Terraform Cloud. Up there with Github in the list of least reliable cloud vendors. I always have a "break glass" method of deploying from my work machine for that very reason.
  - xyzzy123 1y ago
    I actually like terraform for its LACK of power (tho yeah these days when I have a choice I use a lot of small states and orchestrate with tg). Pulumi or CDK are for sure more powerful (and great tools) but when I need to reach for them I also worry that the infra might be getting too complex.
    - wparad 1y ago
      Agreed, it is much too easy to fall into bad habits. The whole goal of OpenTofu is declarative infrastructure. With CDK and pulumi, it's very easy to end up in a place where you lose that. But if you need to do something in a particular way, the tools should never be an obstacle.
hartator 1y ago
Interesting how engineers like to nerd out about SLAs, but never claim or issue credits when something does occur.
- wparad 1y ago
  In the last decade, there has been at least one time where we did issue credits to our customers when there was a problem. Issues credits back to our customers is a small compensation for any issue we're responsible for, and doing so is part of our Terms of Service.
scottlamb 1y ago
I'm surprised the section about retries doesn't mention correlations. They say: > P_{total}(Success) = 1 - P_{3rdParty}(Failure)^{RetryCount} By treating P_{3rdParty}(Failure) as fixed, they're assuming a model in which each each try is completely independent: all the failures are due to background noise. But that's totally wrong, as shown by the existence of big outages like the one they're describing, and not consistent with the way they describe outages in terms of time they are down (rather than purely fraction of requests). In reality, additional retries don't improve reliability as much as that formula says. Given that request 1 failed, request 2 (sent immediately afterward with the same body) probably will too. And there's another important effect: overload. During a major outage, retries often decrease reliability in aggregate—maybe retrying one request makes it more likely to go through, but retrying all the requests causes significant overload, often decreasing the total number of successes. I think this correlation is a much bigger factor than "the reliability of that retry handler" that they go into instead. Not sure what they mean there anyway—if the retry handler is just a loop within the calling code, calling out its reliability separately from the rest of the calling code seems strange to me. Maybe they're talking about an external queue (SQS and the like) for deferred retries, but that brings in a whole different assumption that they're talking about something that can be processed asynchronously. I don't see that mentioned, and it seems inconsistent with the description of these requests as on the critical path for their customers. Or maybe they're talking about hitting a "circuit breaker" that prevents excessive retries—which is a good practice due to the correlation I mentioned above, but if so it seems strange to describe it so obliquely, and again strange to describe its reliability as an inherent/independent thing, rather than a property of the service being called. Additionally, a big pet peeve of mine is talking about reliability without involving latency. In practice, there's only so long your client is willing to wait for the request to succeed. If say that's 1 second, and you're waiting 500 ms for an outbound request before timing out and retrying, you can't even quite make it to 2 full (sequential) tries. You can hedge (wait a bit then send a second request in parallel) for many types of requests, but that also worsens the math on overload and correlated failures. The rest of the article might be much clearer, but I have a fever and didn't make it through.
- lorrin 1y ago
  Agreed, I think the introduction is wrong and detracts from the rest of the article.
deleted 1y ago
[deleted]
markdown 1y ago
BTW clicking on your website logo takes one to https://authress.io/knowledge-base/ https://authress.io/knowledge-base/ instead of https://authress.io https://authress.io
- wparad 1y ago
  Ha, thanks, we'll fix that.
thisnullptr 1y ago
It’s fascinating to me people think their services are so important they can’t survive any downtime. Can we all admit that, while annoying, nothing really bad happened even when us-east-1 was down for almost half a working day?
- JSR_FDED 1y ago
  If you’re providing auth services to many companies then a failure will increase the likelihood of something bad to an unacceptable degree.
- shoo 1y ago
  In many contexts you are correct & further, as someone in that earlier thread about the AWS us-east-1 outage mentioned, customers can be more forgiving of outages if you as the vendor can point to a widespread AWS us-east-1 outage and note that us-east-1 is down for everyone. But, as JSR_FDED's sibling comment notes & as is spelled out in the article, authress' business model offering an auth service means that their outage may entirely brick their clients customer facing auth / machine to machine auth. I've worked in megacorp environments where an outage of certain internal services responsible for auth or issuing JWTs would break tens or hundreds of internal services and break various customer-facing flows. In many business contexts a big messy customer facing outage for a day or so doesn't actually matter but in some contexts it really can. In terms of blast radius, unavailability of a key auth service depended on by hundreds of things is up there with, i dunno, breaking the network.
  - wparad 1y ago
    Absolutely, part of the problem is that a whole region being down is often less of a problem, then just one critical service. And as you point out the blast radius of a critical dependency is huge.
- bostik 1y ago
  As other posters have commented, an external auth service is a very special thing indeed. In modern and/or zero-trust systems if auth doesn't work, then effectively nothing works. My rule of thumb from the past experiences is that if you demand a 99.9% uptime for your own systems and you have an in-house auth, then that auth system must have 99.99% reliability. If you are serving auth for OTHERS, then you have a system that can absolutely never be down, and at that point five nines becomes a baseline requirement. Auth is a critical path component. If your service is in the critical path in both reliability and latency[ß] for third parties, then every one of your failures is magnified by the number of customers getting hit by it. ß: The current top-voted comment thread includes a mention that latency and response time should also be part of an SLA concern. I agree. For any hot-path system you must be always tracking the latency distribution, both from the service's own viewpoint AND from the point of view of the outside world. The typically useful metrics for that are p95, p99, p999 and max. Yes, max is essential to include: you want to always know what was the worst experience someone/something had during any given time window.
JSR_FDED 1y ago
> We test before deployment. There is no better time to test. Love the deadpan delivery.
- wparad 1y ago
  :)
- kailden 1y ago
  On the other hand, saying “Untested code is never released” is a pretty bold statement, even if I understand the good intent
  - wparad 1y ago
    The point here is that, it's important to clarify what you mean by "untested code". Some companies release untested code all the time to production by hide their usage behind feature flags. They admit that the code has not been tested. Code that we release behind feature flags has been tested, the only reason something goes out on a flag is when we don't want to release it to everyone yet, for product reasons, not technical ones. Again going back to "untested", when code gets merged, no one ever says "well taht was untested". But what does tested mean? Of course it means that everyone who is accountable for the code believes it was tested. It's subjective, so there can be no other answer.
0xbadcafebee 1y ago
It's a very rare day that a professional explanation of real operations best practices lands on HN. Good job, Authress!
- wparad 1y ago
  Thank you!
DeathArrow 1y ago
TLDR: they use dynamic DNS routing and have fail over regions.
- netdevphoenix 1y ago
  Can't believe they didn't just mention that in the beginning. There was a lot of show off imo. None of those would have save them on 20th oct if they didn't have the dynamic dns routing
  - wparad 1y ago
    Actually this isn't the only thing that exists. As I pointed out that only exists for resources that are duplicated between regions. There's also the critical fallbacks that exist at the service level to decide which resources to consume. Our usage there is possible both with CloudFront Origin Groups as well as replicated data in our database to multiple regions.
sam-cop-vimes 1y ago
What a well written article! Nothing complex is built overnight, so it is interesting to see how their defenses have evolved to their current state. Requires an engineering team which actually cares about all this and consistency of approach across what seems like 6 years? Impressive.
- wparad 1y ago
  Thank you.

203 points by mooreds 1y ago

This is a rare case where the original bait-y title is probably better than the de-bait-ified title, because the actual article is much less of a brag and much more of an actual case study.

dang 1y ago
Re-how'd, plus I've resisted the temptation to insert a comma that feels missing to me.
- tptacek 1y ago
  "How?! When AWS was down: we were not!"
- wparad 1y ago
  I spent a long time, trying to figure out, what the title of the article, should be. I'm terrible at SEO and generating click-bait titles, it is unfortunately, what, it, is.
  - dang 1y ago
    You did fine! The title is clear. I was just being playful.

pinkmuffinere 1y ago

> During this time, us-east-1 was offline, and while we only run a limited amount of infrastructure in the region, we have to run it there because we have customers who want it there > [Our service can only go down] five minutes and 15 seconds per year. I don't have much experience in this area, so please correct me if I'm mistaken: Don't these two quotes together imply that they have failed to deliver on their SLA for the subset of their customers that want their service in us-east-1? I understand the customers won't be mad at them in this case, since us-east-1 itself is down, but I feel like their title is incorrect. Some subset of their service is running on top of AWS. When AWS goes down, that subset of their service is down. When AWS was down, it seems like they were also down for some customers.

PaulRobinson 1y ago
The bulk of the article discusses their failover strategy, where they detect failures in a region and how they route requests to a backup region, and how to deal with data consistency and cost issues arising from that.
loloquwowndueo 1y ago
Depends on what the SLA phrasing is - us-east-1 affinity is a requirement put forth by some customers so I would totally expect the SLA to specifically state it’s subject to us-east-1 availability. Essentially these customers are opting out of Authress’s fault-tolerant infrastructure and the SLA should be clear about that.
- dylan604 1y ago
  As TFA states, we have to offer services in that region because that's where some users are as well. However, the core of services are not in that region. I have also suggested when the time comes for offering SLAs, that there is explicit wording exempting us-east-1.
wparad 1y ago
It's a good point. We don't actually commit to running infrastructure in one specific AWS region. Customers can't request that the infra runs exactly in us-east-1, but they can request that it runs in "Eastern United States". The problem is that with scenarios that might require VPC peering or low latency connections, we can't just run the infrastructure in us-east-2 and commit to never having a problem. For the same reason, what happens if us-east-2 were to have an incident. We have to assume that our customers need it in a relatively close region, and that at the same time need to plan for the contingency that region can be down. Then there are the customer's users to think of as well. In some cases, those users might be globally dispersed, even if the customer infrastructure is only one major location. So while it would be nice to claim "well you were also down at that moment", in practices customer's users will notice, and realistically, we want to make sure we aren't impeding remediation on their side. That is, even if a customer says "use us-east-1", and then us-east-1 is down, it can't look that way to the customer. This gets a lot more complicated, when the services that we are providing may be impacted differently. Consider us-east-1 dynamoDB down, but everything else was still working. Partial failure modes are much harder to deal with.

deleted 1y ago

[deleted]

sharklasers123 1y ago

Is there not an inherent risk using an AWS service (Route 53) to do the health check? Wouldn’t it make more sense to use a different cloud provider for redundancy?

indigodaddy 1y ago
Had the same thought, eg if things are really down can it even do the check etc
- hnuser123456 1y ago
  Ask some friends and family if you can install an RPi on their home network that monitors your service.
wparad 1y ago
If the check can't be done, then everything stays stable, so I'm guessing the question is, "What happens if Route 53 does the check and incorrectly reports the result?" In that case, no matter what we are using there is going to be a critical issue. I think the best I could suggest at that point would be to have records in your zone that round robin different cloud providers, but that comes with its own challenges. I believe there are some articles sitting around regarding how AWS plans for failure and the fallback mechanism actually reduces load on the system rather than makes it worse. I think it would require in-depth investigation on the expected failover mode to have a good answer there. For instance, just to make it more concrete, what sort of failure mode are you expecting to happen with the Route 53 health check? Depending on that there could be different recommendations.
- indigodaddy 1y ago
  Have you considered the scenario of "everything is so dead in aws", that the check doesn't happen, plus the backends are dead too (this is assuming the backend services live in aws as well) ? But I'd guess in that case you'd know quickly enough from supplementary alerting (you guys don't seem the type to not have some sort of awesome monitoring in place) and you have a different/worse DR problem on your hands. As far as the OP's point though, I'm going to probably assume that the health checks need to stay within/from AWS because 3rd party health checks could taint/dilute the point of the in-house AWS HC service to begin with.

oldpersonintx2 1y ago

[dead]

indigodaddy 1y ago

Back in the day (10-12 years ago) at a telecom/cable we accomplished this with F5 Big IP GSLB DNS (and later migrated to A10's GSLB equivalent devices) as the auth DNS server for services/zones that required or were suitable for HA. (I can't totally remember but I'm guessing we must have had a pretty low TTL for this). Had no idea that Route 53 had this sort of functionality

wparad 1y ago
Maybe I should have titled the article "AWS Route53 HealthChecks are amazing" :)
indigodaddy 1y ago
Speaking of F5 Big IP DNS devices, does anyone know of any auth DNS software solution for GSLB/health checking for DNS (I guess excluding Route 53 or other cloud/SaaS). Last I looked all I could find was the polaris-gslb addon for PowerDNS, but the GitHub for that has no activity in 8 years.
- loadbalancer 1y ago
  We've been using Polaris for the Loadbalancer.org GSLB for a few years now, and we've found it fast and stable. Its so simple its never needed any updates: https://www.loadbalancer.org/blog/gslb-why-global-server-load-balancers-dont-always-suck-polaris-gslb/ https://www.loadbalancer.org/blog/gslb-why-global-server-loa... Although we did patch a dynamic health check a while back, which will be open source of course. But I'll get someone to check if we actually gave it back to the community or not...

iso1631 1y ago

I'm interested in how they measure that downtime. If you're down for 200 milliseconds, does that accumulate. How do you even measure that you're down for 200ms. (For what it's worth, for some of my services, 200ms is certainly an impact, not as bad as 2 seconds out outage but still noticable and reportable)

wparad 1y ago
Good catch. The truth is, while we track downtime for incident reporting, it's much more correct to actually be tracking the number of requests that result in a failure. Our SLAs are based on request volume, and not specifically time. Most customers don't have perfect sustained usage. Being down when they aren't running is irrelevant to everyone. This is where the grey failures can come into play. It's really hard to tell, often impossible to know what the impact of an incident is to a customer, even if you know you are having an incident, without them telling you. In order to know that you are "down", our edge of the HTTP request would need to be able to track requests. For us that is CloudFront, but if there is an issue before that, at DNS, at network level, etc... we just can't know what the actual impact is. As far as measuring how you are down. We can pretty accurately know the list of failures that are happening, (when we can know), and what the results are. That's because most components are behind cloudfront in any case. And if cloudfront isn't having a problem, we'll have telemetry that tells us what the HTTP request/response status codes and connection completions look like. Then it's a matter of measuring from our first detection to the actual remediation being deployed (assuming there is one). Another thing that helps here is that we have multiple other products that also use Authress, and we can run technology in other regions that can report this information, for those accounts (obviously can't be for all customers), which can help us identify with additional accuracy, but is often unnecessary.
smadge 1y ago
I think a lot of web services talk about reliability in terms of uptime (e.g. down for less than 5 minutes a year) but in reality operate on failure ratios (less than 0.001% of request to our service fail).
- wparad 1y ago

wparad 1y ago

Hey, I wrote that article! I'll try to add comments and answer questions where I can. - Warren

ckozlowski 1y ago
Hi Warren! I'm Chris, and I'm with AWS, where among other things, I work on the Well-Architected Framework. Would you be willing to talk with us? You can reach me at kozlowck@amazon.com. Thanks! Edit: This is a fantastic write-up by the way!
- wparad 1y ago
  Thank you!

rdoherty 1y ago

This is probably one of the best summarizations of the past 10 years of my career in SRE. Once your systems get complex enough, something is always broken and you have to prepare for that. Detection & response become just as critical as pre-deploy testing. I do worry about all the automation being another failure point, along with the IaC stuff. That is all software too! How do you update that safely? It's turtles all the way down!

evanmoran 1y ago
Iac is definitely a failure point, but the manual alternative is much worse! I’ve had a lot of benefit from using pulumi, simply because the code can be more compact than the terraform hcl was. For example, for the fall over regions (from the article) you could make a pulumi function that parameterizes only the n things that are different per fall over env and guarantee / verify the scripts are nearly identical. Of course, many people use modules / terragrunt for similar reasons, but it ends up being quite powerful.
- spyspy 1y ago
  If you do use terraform, for the love of god do NOT use Terraform Cloud. Up there with Github in the list of least reliable cloud vendors. I always have a "break glass" method of deploying from my work machine for that very reason.
- xyzzy123 1y ago
  I actually like terraform for its LACK of power (tho yeah these days when I have a choice I use a lot of small states and orchestrate with tg). Pulumi or CDK are for sure more powerful (and great tools) but when I need to reach for them I also worry that the infra might be getting too complex.
  - wparad 1y ago
    Agreed, it is much too easy to fall into bad habits. The whole goal of OpenTofu is declarative infrastructure. With CDK and pulumi, it's very easy to end up in a place where you lose that. But if you need to do something in a particular way, the tools should never be an obstacle.

hartator 1y ago

Interesting how engineers like to nerd out about SLAs, but never claim or issue credits when something does occur.

wparad 1y ago
In the last decade, there has been at least one time where we did issue credits to our customers when there was a problem. Issues credits back to our customers is a small compensation for any issue we're responsible for, and doing so is part of our Terms of Service.

scottlamb 1y ago

I'm surprised the section about retries doesn't mention correlations. They say: > P_{total}(Success) = 1 - P_{3rdParty}(Failure)^{RetryCount} By treating P_{3rdParty}(Failure) as fixed, they're assuming a model in which each each try is completely independent: all the failures are due to background noise. But that's totally wrong, as shown by the existence of big outages like the one they're describing, and not consistent with the way they describe outages in terms of time they are down (rather than purely fraction of requests). In reality, additional retries don't improve reliability as much as that formula says. Given that request 1 failed, request 2 (sent immediately afterward with the same body) probably will too. And there's another important effect: overload. During a major outage, retries often decrease reliability in aggregate—maybe retrying one request makes it more likely to go through, but retrying all the requests causes significant overload, often decreasing the total number of successes. I think this correlation is a much bigger factor than "the reliability of that retry handler" that they go into instead. Not sure what they mean there anyway—if the retry handler is just a loop within the calling code, calling out its reliability separately from the rest of the calling code seems strange to me. Maybe they're talking about an external queue (SQS and the like) for deferred retries, but that brings in a whole different assumption that they're talking about something that can be processed asynchronously. I don't see that mentioned, and it seems inconsistent with the description of these requests as on the critical path for their customers. Or maybe they're talking about hitting a "circuit breaker" that prevents excessive retries—which is a good practice due to the correlation I mentioned above, but if so it seems strange to describe it so obliquely, and again strange to describe its reliability as an inherent/independent thing, rather than a property of the service being called. Additionally, a big pet peeve of mine is talking about reliability without involving latency. In practice, there's only so long your client is willing to wait for the request to succeed. If say that's 1 second, and you're waiting 500 ms for an outbound request before timing out and retrying, you can't even quite make it to 2 full (sequential) tries. You can hedge (wait a bit then send a second request in parallel) for many types of requests, but that also worsens the math on overload and correlated failures. The rest of the article might be much clearer, but I have a fever and didn't make it through.

lorrin 1y ago
Agreed, I think the introduction is wrong and detracts from the rest of the article.

deleted 1y ago

[deleted]

markdown 1y ago

BTW clicking on your website logo takes one to https://authress.io/knowledge-base/ https://authress.io/knowledge-base/ instead of https://authress.io https://authress.io

wparad 1y ago
Ha, thanks, we'll fix that.

thisnullptr 1y ago

It’s fascinating to me people think their services are so important they can’t survive any downtime. Can we all admit that, while annoying, nothing really bad happened even when us-east-1 was down for almost half a working day?

JSR_FDED 1y ago
If you’re providing auth services to many companies then a failure will increase the likelihood of something bad to an unacceptable degree.
shoo 1y ago
In many contexts you are correct & further, as someone in that earlier thread about the AWS us-east-1 outage mentioned, customers can be more forgiving of outages if you as the vendor can point to a widespread AWS us-east-1 outage and note that us-east-1 is down for everyone. But, as JSR_FDED's sibling comment notes & as is spelled out in the article, authress' business model offering an auth service means that their outage may entirely brick their clients customer facing auth / machine to machine auth. I've worked in megacorp environments where an outage of certain internal services responsible for auth or issuing JWTs would break tens or hundreds of internal services and break various customer-facing flows. In many business contexts a big messy customer facing outage for a day or so doesn't actually matter but in some contexts it really can. In terms of blast radius, unavailability of a key auth service depended on by hundreds of things is up there with, i dunno, breaking the network.
- wparad 1y ago
  Absolutely, part of the problem is that a whole region being down is often less of a problem, then just one critical service. And as you point out the blast radius of a critical dependency is huge.
bostik 1y ago
As other posters have commented, an external auth service is a very special thing indeed. In modern and/or zero-trust systems if auth doesn't work, then effectively nothing works. My rule of thumb from the past experiences is that if you demand a 99.9% uptime for your own systems and you have an in-house auth, then that auth system must have 99.99% reliability. If you are serving auth for OTHERS, then you have a system that can absolutely never be down, and at that point five nines becomes a baseline requirement. Auth is a critical path component. If your service is in the critical path in both reliability and latency[ß] for third parties, then every one of your failures is magnified by the number of customers getting hit by it. ß: The current top-voted comment thread includes a mention that latency and response time should also be part of an SLA concern. I agree. For any hot-path system you must be always tracking the latency distribution, both from the service's own viewpoint AND from the point of view of the outside world. The typically useful metrics for that are p95, p99, p999 and max. Yes, max is essential to include: you want to always know what was the worst experience someone/something had during any given time window.

JSR_FDED 1y ago

> We test before deployment. There is no better time to test. Love the deadpan delivery.

wparad 1y ago
:)
kailden 1y ago
On the other hand, saying “Untested code is never released” is a pretty bold statement, even if I understand the good intent
- wparad 1y ago
  The point here is that, it's important to clarify what you mean by "untested code". Some companies release untested code all the time to production by hide their usage behind feature flags. They admit that the code has not been tested. Code that we release behind feature flags has been tested, the only reason something goes out on a flag is when we don't want to release it to everyone yet, for product reasons, not technical ones. Again going back to "untested", when code gets merged, no one ever says "well taht was untested". But what does tested mean? Of course it means that everyone who is accountable for the code believes it was tested. It's subjective, so there can be no other answer.

0xbadcafebee 1y ago

It's a very rare day that a professional explanation of real operations best practices lands on HN. Good job, Authress!

wparad 1y ago
Thank you!

DeathArrow 1y ago

TLDR: they use dynamic DNS routing and have fail over regions.

netdevphoenix 1y ago
Can't believe they didn't just mention that in the beginning. There was a lot of show off imo. None of those would have save them on 20th oct if they didn't have the dynamic dns routing
- wparad 1y ago
  Actually this isn't the only thing that exists. As I pointed out that only exists for resources that are duplicated between regions. There's also the critical fallbacks that exist at the service level to decide which resources to consume. Our usage there is possible both with CloudFront Origin Groups as well as replicated data in our database to multiple regions.

sam-cop-vimes 1y ago

What a well written article! Nothing complex is built overnight, so it is interesting to see how their defenses have evolved to their current state. Requires an engineering team which actually cares about all this and consistency of approach across what seems like 6 years? Impressive.

wparad 1y ago
Thank you.