4 ms·
And I'm sure Lambda will never go down. Right? Right?? (It has. Completely and silently stopped processing against Kinesis queues for a few hours recently. Gue
by falcolas 10y ago
And I'm sure Lambda will never go down. Right? Right??
(It has. Completely and silently stopped processing against Kinesis queues for a few hours recently. Guess what AWS Step is built on?)
- jlgaddis 10y agoWell, sure, of course it will, but I don't think Nick is advocating replacing a complete, full featured monitoring system with this. It could be very useful to, for example, keep an eye on your monitoring system. At $work, we have a pretty extensive monitoring system that we've built out. We use an external service to watch over the monitoring system, though, to alert us of any issues with it that we haven't otherwise caught. Besides, like he said, it's "fun" and kinda neat.
- cddotdotslash 10y agoOf course it can go down, and you can have CloudWatch alerts to alert you about that. But so can your Nagios server sending pings go down or the fancy SaaS you signed up for.
- robinson-wall 10y agoDid you just suggest using a third AWS service to let you know if the second AWS service monitoring your first AWS service goes down?
- cddotdotslash 10y agoYes, because they're different services running on different architecture and distributed differently. I challenge you to find one time in the past five years where CloudWatch was down at the same time as other services. Even if you can, I'm sure your custom built Nagios server in your datacenter has gone down as many or more times too. But my bigger point here is that you're essentially asking "well how do you monitor your monitor?" At which point up the chain do you have enough? Also, I think the original post was simply a demo of what is possible. Yet whenever someone posts something, people go in the comments to belittle it. "Yeah, you built a monitoring solution... Well what happens if that goes down?" Which is a legitimate question. But obviously if your production service is that critical to your business, you won't be monitoring it with a service that costs $0.0000002 per execution.
- falcolas 10y ago> they're different services running on different architecture and distributed differently I think you underestimate the interdependency of services in AWS. Historically, if there were problems with S3 or EBS in us-east-1, you could expect the entire API to be flaky, and things like autoscaling to fail. These have been better distributed, but failures still cascade. > I think the original post was simply a demo of what is possible No, it wasn't a demo, it was an actual production issue. No alarms, no error logs, no way to tell it wasn't working other than someone noticing the queues were getting larger and contacting AWS. > people go in the comments to belittle it Only because the original project projects AWS Lambda as "the solution" for such problems, not realizing that it is just as fallible a solution as everything else. > Well what happens if that goes down? The solution to this is well known - two monitoring systems in physically separate locations that monitor each other as well as mission critical systems. Nagios, Icinga, and a dozen other well-tested solutions work remarkably well for these roles, yet people keep writing "new" solutions over and over and over. > But obviously if your production service is that critical to your business, you won't be monitoring it with [this] service Then what's it's value, other than as an intellectual exercise?
- cddotdotslash 10y agoLaunch two Lambda functions, heck, 8 Lambda functions, one in each AWS region that supports it. They all monitor one another, plus run your checks. Next, are you going to say all 8 regions will go down at once? The whole setup will still cost $0/month. > The solution to this is well known - two monitoring systems in physically separate locations that monitor each other as well as mission critical systems. Nagios, Icinga, and a dozen other well-tested solutions work remarkably well for these roles, yet people keep writing "new" solutions over and over and over. Because not everyone needs heavy solutions to do something simple. Side projects, small sites, etc. And some people enjoy implementing old use cases using new technology. When Go was rising in popularity, half the posts on the front page were re-implementing fairly common features in Go. Even if you're not going to implement this yourself, there can still be some value for other readers.
- 10y ago
- thisone 10y agodon't worry, the status was green the entire time...
- falcolas 10y agoYes, it was indeed green the entire time. Of course, AWS is almost always green, so long as something is up...