5 ms·

Cloudwatch just stopped triggering our Lambdas

47 points by artembugara 6y ago

jonathannorris 6y ago
This seems like a larger system outage, we are seeing issues with Kinesis and Cloudwatch.
- athorax 6y ago
  Cloudwatch, cloudwatch events, and kinesis events in us-east-1 have been down for about 2 hours now
dageshi 6y ago
Had Service Unavailable for about 10 mins on Cloudwatch events. Just corrected itself for me at least. us-east-1
tootie 6y ago
AWS status board is reporting a major outage in Kinesis for us-east-1. Causing big problems for us right now.
edoceo 6y ago
For folks who go in on managed services because self-hosted has the danger of an outage - how does this factor? I self-host lots of my company services and frequently catch heck from my peers for it: if your "x" goes down it costs more than the managed. (It's more than just cost tho) Seems like folk factor managed as if it was zero-downtime game and that self-hosted is an huge burden. Maybe these monitors aren't critical but what about when it is?
- dageshi 6y ago
  In some sense I suppose if AWS goes down in some fashion it's kind of like an act of nature. There's nothing individually you could do to prevent it. On the other hand if you self host and there's an outage, the buck stops with you, it's your problem to diagnose and fix.
  - abraxas 6y ago
    In other words we're in the "Nobody gets fired for choosing managed AWS" paradigm.
    - dageshi 6y ago
      Basically. If you go down you're probably in the company of half the fortune 500. And of course AWS is a legitimately capable and useful platform that does remove a lot "ops" although it probably frontloads a lot of "config" in return for that.
    - derefr 6y ago
      It's not just diffusion of responsibility. It's the fact that — assuming you're a B2B SaaS — when you're down, the customers depending on your service are probably down too, since they're very likely to also be dependent on the same service that knocked you offline. It's like a city-wide power outage, or a flood in a city that's in a flood-basin. The city's entire economy will "pause." If you're a B2B business affected by the outage/flood, you don't have to worry about losing sales/customers, as your customers won't even be trying to make any requests of you, since their capacity to serve requests to their customers is knocked out, and without that, they won't be making any requests of you. In fact, as long as they're down, they may not even notice that you're down!
    - wdb 6y ago
      Still if I cause P2/P1 issue I still need to explain to the board of top 50 fortune 500 company. Had to do it once its no fun at all.
  - briffle 6y ago
    There is also (and especially if your a small shop) the thing where if your self hosted stuff goes down, you have a crappy time fixing it, over weekend, holiday, etc, until its done. If AWS goes down, thousands of other people are working on it frantically. And most likely, many of them are smarter/better than you.
    - tlarkworthy 6y ago
- staticassertion 6y ago
  a) Don't deploy prod to us-east-1 b) Don't stick to a single region/ AZ. Cross-region/AZ failures are quite rare c) If AWS goes down a huge portion of the internet is fucked. Customer expectations are totally different. When 90% of the internet isn't loading images no one's coming to you to complain. d) The fire is elsewhere. Someone else is on it. At worst we spend some time trying to work around it, at best we work on other things while we wait.
- markbnj 6y ago
  >> For folks who go in on managed services because self-hosted has the danger of an outage Seems like an overly simplistic and narrow view of the tradeoffs. The mainstream cloud services providers operate at very high levels of availability, but they nonetheless still have the "danger of an outage." There are many reasons why managed infrastructure may make sense for an organization, from ease of scaling to the completeness of the available solution space and cost efficiencies, but "absolutely avoiding any chance of downtime" isn't one of them.
- PHGamer 6y ago
  if big companies go down. the ops guy can blame it on the hosting provider esepcially if its amazon. no place can guarantee 100% uptime but if you pick some joe crabs hosting you will be blamed but if you pick amazon or azure the company will be like oh well shit happens.
- mandelbrotwurst 6y ago
  If I'm using a managed service, it's more that I am expecting to get better reliability and less work on our part than it is expecting perfect reliability and no work on our part, both of which are a pipe dream no matter which route you go.
Zanfa 6y ago
From their own status page: "We are currently blue on Kinesis, Cognito, IoT Core, EventBridge and CloudWatch given an increase in errors for Kinesis in the US-EAST-1 Region. It's not posted on SHD as the issue has impacted our ability to post there. We will update this banner if there continue to be issues with the SHD."
- hyperdimension 6y ago
  > "It's not posted on SHD as the issue has impacted our ability to post there." At the risk of stating the obvious...why host your status page of a service on something that depends on that service?!
deleted 6y ago
[deleted]
smt88 6y ago
It has been disappointing to see how many people on HN have business-critical production deployments in a single AZ. I thought we learned not to trust any one AWS location years ago when Netflix learned for us.
- risaacs99 6y ago
  This is a regional outage, not a single AZ.
  - smt88 6y ago
    You're right, I misspoke. I meant across regions. All the abstractions AWS offers (like RDS) make it easy to replicate across regions, but of course not everyone uses those.
- tootie 6y ago
  I recently joined an org that has all their eggs in the us-east-1 basket. I was warned as much by the devops lead that it was at least a partially conscious decision that the cost of multi-region would outweigh the risk. I think I'm going to start questioning that point of view.
- jlmorton 6y ago
  This is affecting all of us-east-1, not just a single availability zone. Over the past few years, it's become easier to host an application in multiple regions, but it's still a heavy lift, requiring a lot of thought not just from site reliability engineers, but application architecture.
Nexeo 6y ago
Is anyone else getting "Capacity unavailable" when trying to add tasks in Fargate?

47 points by artembugara 6y ago

jonathannorris 6y ago

This seems like a larger system outage, we are seeing issues with Kinesis and Cloudwatch.

athorax 6y ago
Cloudwatch, cloudwatch events, and kinesis events in us-east-1 have been down for about 2 hours now

dageshi 6y ago

Had Service Unavailable for about 10 mins on Cloudwatch events. Just corrected itself for me at least. us-east-1

tootie 6y ago

AWS status board is reporting a major outage in Kinesis for us-east-1. Causing big problems for us right now.

edoceo 6y ago

For folks who go in on managed services because self-hosted has the danger of an outage - how does this factor? I self-host lots of my company services and frequently catch heck from my peers for it: if your "x" goes down it costs more than the managed. (It's more than just cost tho) Seems like folk factor managed as if it was zero-downtime game and that self-hosted is an huge burden. Maybe these monitors aren't critical but what about when it is?

dageshi 6y ago
In some sense I suppose if AWS goes down in some fashion it's kind of like an act of nature. There's nothing individually you could do to prevent it. On the other hand if you self host and there's an outage, the buck stops with you, it's your problem to diagnose and fix.
- abraxas 6y ago
  In other words we're in the "Nobody gets fired for choosing managed AWS" paradigm.
  - dageshi 6y ago
    Basically. If you go down you're probably in the company of half the fortune 500. And of course AWS is a legitimately capable and useful platform that does remove a lot "ops" although it probably frontloads a lot of "config" in return for that.
  - derefr 6y ago
    It's not just diffusion of responsibility. It's the fact that — assuming you're a B2B SaaS — when you're down, the customers depending on your service are probably down too, since they're very likely to also be dependent on the same service that knocked you offline. It's like a city-wide power outage, or a flood in a city that's in a flood-basin. The city's entire economy will "pause." If you're a B2B business affected by the outage/flood, you don't have to worry about losing sales/customers, as your customers won't even be trying to make any requests of you, since their capacity to serve requests to their customers is knocked out, and without that, they won't be making any requests of you. In fact, as long as they're down, they may not even notice that you're down!
  - wdb 6y ago
    Still if I cause P2/P1 issue I still need to explain to the board of top 50 fortune 500 company. Had to do it once its no fun at all.
- briffle 6y ago
  There is also (and especially if your a small shop) the thing where if your self hosted stuff goes down, you have a crappy time fixing it, over weekend, holiday, etc, until its done. If AWS goes down, thousands of other people are working on it frantically. And most likely, many of them are smarter/better than you.
  - tlarkworthy 6y ago
staticassertion 6y ago
a) Don't deploy prod to us-east-1 b) Don't stick to a single region/ AZ. Cross-region/AZ failures are quite rare c) If AWS goes down a huge portion of the internet is fucked. Customer expectations are totally different. When 90% of the internet isn't loading images no one's coming to you to complain. d) The fire is elsewhere. Someone else is on it. At worst we spend some time trying to work around it, at best we work on other things while we wait.
markbnj 6y ago
>> For folks who go in on managed services because self-hosted has the danger of an outage Seems like an overly simplistic and narrow view of the tradeoffs. The mainstream cloud services providers operate at very high levels of availability, but they nonetheless still have the "danger of an outage." There are many reasons why managed infrastructure may make sense for an organization, from ease of scaling to the completeness of the available solution space and cost efficiencies, but "absolutely avoiding any chance of downtime" isn't one of them.
PHGamer 6y ago
if big companies go down. the ops guy can blame it on the hosting provider esepcially if its amazon. no place can guarantee 100% uptime but if you pick some joe crabs hosting you will be blamed but if you pick amazon or azure the company will be like oh well shit happens.
mandelbrotwurst 6y ago
If I'm using a managed service, it's more that I am expecting to get better reliability and less work on our part than it is expecting perfect reliability and no work on our part, both of which are a pipe dream no matter which route you go.

Zanfa 6y ago

From their own status page: "We are currently blue on Kinesis, Cognito, IoT Core, EventBridge and CloudWatch given an increase in errors for Kinesis in the US-EAST-1 Region. It's not posted on SHD as the issue has impacted our ability to post there. We will update this banner if there continue to be issues with the SHD."

hyperdimension 6y ago
> "It's not posted on SHD as the issue has impacted our ability to post there." At the risk of stating the obvious...why host your status page of a service on something that depends on that service?!

deleted 6y ago

[deleted]

smt88 6y ago

It has been disappointing to see how many people on HN have business-critical production deployments in a single AZ. I thought we learned not to trust any one AWS location years ago when Netflix learned for us.

risaacs99 6y ago
This is a regional outage, not a single AZ.
- smt88 6y ago
  You're right, I misspoke. I meant across regions. All the abstractions AWS offers (like RDS) make it easy to replicate across regions, but of course not everyone uses those.
tootie 6y ago
I recently joined an org that has all their eggs in the us-east-1 basket. I was warned as much by the devops lead that it was at least a partially conscious decision that the cost of multi-region would outweigh the risk. I think I'm going to start questioning that point of view.
jlmorton 6y ago
This is affecting all of us-east-1, not just a single availability zone. Over the past few years, it's become easier to host an application in multiple regions, but it's still a heavy lift, requiring a lot of thought not just from site reliability engineers, but application architecture.

Nexeo 6y ago

Is anyone else getting "Capacity unavailable" when trying to add tasks in Fargate?