4 ms·

> List the good outcomes you want, and the bad outcomes to be avoided I feel like sometimes you don't know the bad outcomes, until they happen. E.g., years ago

> List the good outcomes you want, and the bad outcomes to be avoided I feel like sometimes you don't know the bad outcomes, until they happen. E.g., years ago, my team had a fairly big outage/issue caused by S3 reads taking multiple of 60s. (That is, our latency graph had very large spikes right over 60s, 120s, 180s¹. These were reads in the typically in the 10 KiB to 100 KiB range, sometimes as large as single-digit megs — i.e., they should take milliseconds, maybe a second, not 3 minutes. It took a significant amount of back-and-forth² as it either only effected our bucket, or just nobody else noticed. But it slowed our ability to process down that we built up an incredibly backlogs. (Processing a file had previously taken <1s, and was now taking over three minutes in some cases, a 200x slowdown!) This is still not covered by the S3 SLA. Also had a different cloud provider where the "Create VM" API returned 200 OK. The VM never finished booting. The SLA was over the 200 OK, not the actual completion of the task. Basically, exactly the example in the article, with a real world "we're paying for this?" provider. ¹I'm simplifying. It was weirdly actually 70s, 130s, and 190s. I can only presume that's a 10s timeout and n 60s timeouts, somewhere. ²Woe unto the person who doesn't come to support with request IDs. My impression the whole time is "these guys can't see their own response latency?"

jsty 7y ago
> Woe unto the person who doesn't come to support with request IDs. My impression the whole time is "these guys can't see their own response latency?" They can probably see their own latencies at a service level, but if this was only affecting a few tenants, their internal tooling might have security-related restrictions on "dig into the request characteristics of this tenant". Some providers deal with this by making CSRs log a reason for each access that you need to be prepared to justify when audited, others just lock it down completely outside of particular request patterns (e.g. "provide a request ID"). Pros and cons on both sides really.
perlgeek 7y ago
> ²Woe unto the person who doesn't come to support with request IDs. My impression the whole time is "these guys can't see their own response latency?" Response latency is often only sampled, so if you observe rare outliers, support might not be any visibility by default.
jkoudys 7y ago
I did service work at a big corp for years, and we would've loved to take a problem like yours. I'd always want the tickets that came in with some interesting, visible pattern and a client who's already done some analysis. Much more fun than taking 6 hours telling a room full of panicky executives how important their problem is to me, before getting to the 10 minutes it actually takes to solve it.
vkou 7y ago
> ²Woe unto the person who doesn't come to support with request IDs. My impression the whole time is "these guys can't see their own response latency?" They may see that there's a spike of high-latency requests for the system, as a whole, on their chart, but need a customer's explicit permission & particular request IDs to run a "Select from request_log where ..." query. Generally speaking, just because you see a spike on a graph doesn't necessarily give you leave to start digging through random customer queries.
breischl 7y ago
I think it's hilarious that everyone is responding to >²Woe unto the person who doesn't come to support with request IDs. My impression the whole time is "these guys can't see their own response latency?" ... so I'm going to do the same. :) It's also possible that while that latency is weird for your particular usage, it's totally expected for someone else's. That can make it very hard to find your problematic requests amidst the noise of everyone else's average day at the bitbucket.
8note 7y ago
I get those timeouts from Dynamo. the solution according to support is to set your own aws client config that kills connections if the timeout is larger than expected. the 60s is defined in the default client config

jsty 7y ago

> Woe unto the person who doesn't come to support with request IDs. My impression the whole time is "these guys can't see their own response latency?" They can probably see their own latencies at a service level, but if this was only affecting a few tenants, their internal tooling might have security-related restrictions on "dig into the request characteristics of this tenant". Some providers deal with this by making CSRs log a reason for each access that you need to be prepared to justify when audited, others just lock it down completely outside of particular request patterns (e.g. "provide a request ID"). Pros and cons on both sides really.

perlgeek 7y ago

> ²Woe unto the person who doesn't come to support with request IDs. My impression the whole time is "these guys can't see their own response latency?" Response latency is often only sampled, so if you observe rare outliers, support might not be any visibility by default.

jkoudys 7y ago

I did service work at a big corp for years, and we would've loved to take a problem like yours. I'd always want the tickets that came in with some interesting, visible pattern and a client who's already done some analysis. Much more fun than taking 6 hours telling a room full of panicky executives how important their problem is to me, before getting to the 10 minutes it actually takes to solve it.

vkou 7y ago

> ²Woe unto the person who doesn't come to support with request IDs. My impression the whole time is "these guys can't see their own response latency?" They may see that there's a spike of high-latency requests for the system, as a whole, on their chart, but need a customer's explicit permission & particular request IDs to run a "Select from request_log where ..." query. Generally speaking, just because you see a spike on a graph doesn't necessarily give you leave to start digging through random customer queries.

breischl 7y ago

I think it's hilarious that everyone is responding to >²Woe unto the person who doesn't come to support with request IDs. My impression the whole time is "these guys can't see their own response latency?" ... so I'm going to do the same. :) It's also possible that while that latency is weird for your particular usage, it's totally expected for someone else's. That can make it very hard to find your problematic requests amidst the noise of everyone else's average day at the bitbucket.

8note 7y ago

I get those timeouts from Dynamo. the solution according to support is to set your own aws client config that kills connections if the timeout is larger than expected. the 60s is defined in the default client config