3 ms·
Sort of tangential to the article, but... Setting aggressive HTTP timeouts can be quite informative in finding "interesting" HTTP endpoints. Set them, enable t
by jrockway 7y ago
Sort of tangential to the article, but... Setting aggressive HTTP timeouts can be quite informative in finding "interesting" HTTP endpoints. Set them, enable tracing, and search for timeouts every morning. At my last job, I found an interesting "charge_cards" endpoint and learned that someone logs in every day, clicks "charge cards" and all billing is handled in the frontend web process instead of the event queue. It was idempotent so it wasn't a problem (just click reload until it returns), but was an interesting bit of visibility into a large system. Knowing that that's how it works makes it less surprising when someone wonders "I noticed cards didn't get charged". Is the person who normally does that on vacation? Aha, that's why. This never caused us any problems so we didn't rewrite it, but it's always nice to have weirdness exposed front and center so you are less surprised when it starts causing an issue.
It was also interesting to read about a wide deployment of circuit breaking. That's a relatively new concept to me (never ran into it at Google, for whatever reason), but it's another way to prevent weird outages. Last week I upgraded something in my k8s cluster and the API server stopped responding. With the API server unresponsive, I couldn't kill the workload (or even determine if that's what was causing it). I had to kill all the nodes before I could get back in. If the job were communicating to the API server through a circuit breaker, it would have popped after requests started taking too long, and I would have been able to just kill the job instead of having downtime. But, I guess it's still a relatively esoteric concept.
I know this article is about not doing dumb things in production. I just assume those are a given. I'm more interested in the nuts and bolts of day-to-day operation ;)