4 ms·

I'm not sure what qualifies as "stateful", but > Fewer dependencies to fail and take down the service. No logging? No metrics? No monitoring? (& yes, you'd th

I'm not sure what qualifies as "stateful", but > Fewer dependencies to fail and take down the service. No logging? No metrics? No monitoring? (& yes, you'd think those shouldn't take down the stack if they went offline. And I'd agree. And yet, I've witnessed that failure mode multiple times. In one, a call to Sentry was synchronous & a hard-fail, so when Sentry went down, that service 500'd. In another, syslog couldn't push logs out to the logging service, as that was very down, having been inadvertently deleted by someone who ran "terraform apply", didn't read the plan, & then said "make it so"; syslog then responded to the logging service being down by logging that error to a local file. Repeatedly. As fast as it possibly could. Fill the disk. Disk is full. Service fails.) I've also seen our alerting provider have an outage during an outage we're having & thus not sending pages for our outage, causing me to ponder and wonder how I'd just rolled a 1 on the SRE d20 and what god did I anger? Also who watches the watchmen? > A common pitfall is to introduce something like ElasticSearch, only to realize a few months later that no one knows how to run it. Yeah I've seen that exact pit fallen into. No DNS? Global Cloudflare outage == fun. No certificates? I've seen certs fail so many different way. Of course not getting renewed, that's your table stakes "welcome to certs!" failure mode. Certs get renewed but an allegedly Semver compatible upgrade changed the defaults, and required extensions don't get included leading to the client rejecting the cert. I've seen a service which watches certs to make sure they don't expire (see the outage earlier in this paragraph!) have an outage (which, b/c it's monitoring, wasn't customer visible) because a tool issued a malformed cert (…by… default…) that the monitor failed to parse (as it was malformed). Oh, and then the LE cross-signing expiration took out an Azure service that wasn't ready for it, a service from a third-party of ours that wasn't ready for it, and our CI system b/c several tools were out of date including an up to date system on Debian that was theoretically "supported"… but still shipped an ancient crypto library riddled with bugs in its path validation. > Okay fine, S3 too, but that’s a different animal. Is it? I've seen that have outages too, & bring down a service with it. (There really wasn't a choice there; S3 was the service's backing store, & without it, the service was truly screwed.) But of course, all this is to say I violently agree with the article's core point: think carefully about each dependency, as they have a very real production cost. (I've recently been considering changing my title to SRE because I have done very little in the way of SWE recently…)

Too 5y ago
Metrics feels like another area where you should allow exception. Prometheus is like a tank, I’ve never seen it misbehave and if you set retention short it’s very close to a fire and forget deployment with very little configuration, especially if you only use it to monitor two other static services. These metrics will be invaluable to tell you how close to the limit you are running your single dependency, to avoid adding premature cache in front or whatnot. The integration of Prometheus with other tools also make it worth the extra component, rather than reinventing your own time series in PG. Logging is a whole different beast. Especially if you even think about Elastic.
- deathanatos 5y ago
  > Logging is a whole different beast. Especially if you even think about Elastic. I think you kinda have to have something like Elastic. (Not necessarily Elastic, but honestly, I've yet to see anything better. Which … isn't great.) As soon as you're dealing with multiple services or multiple instances of a service, you'll want some way to aggregate logs. Even for a single service, on a single instance, some way to search logs (no, grep does not count), and at some point, alert.
  - Too 5y ago
    Take a look at Grafana Loki. It’s a log aggregator designed for ease of operation. With some compromises on query capabilities instead, well worth it as it’s still incredibly powerful.

Too 5y ago

Metrics feels like another area where you should allow exception. Prometheus is like a tank, I’ve never seen it misbehave and if you set retention short it’s very close to a fire and forget deployment with very little configuration, especially if you only use it to monitor two other static services. These metrics will be invaluable to tell you how close to the limit you are running your single dependency, to avoid adding premature cache in front or whatnot. The integration of Prometheus with other tools also make it worth the extra component, rather than reinventing your own time series in PG. Logging is a whole different beast. Especially if you even think about Elastic.

deathanatos 5y ago
> Logging is a whole different beast. Especially if you even think about Elastic. I think you kinda have to have something like Elastic. (Not necessarily Elastic, but honestly, I've yet to see anything better. Which … isn't great.) As soon as you're dealing with multiple services or multiple instances of a service, you'll want some way to aggregate logs. Even for a single service, on a single instance, some way to search logs (no, grep does not count), and at some point, alert.
- Too 5y ago
  Take a look at Grafana Loki. It’s a log aggregator designed for ease of operation. With some compromises on query capabilities instead, well worth it as it’s still incredibly powerful.