3 ms·
The client - "Gitlab Pages" in this article - should be retrying 502s on a different socket anyway.
by jeffbee 5y ago
The client - "Gitlab Pages" in this article - should be retrying 502s on a different socket anyway.
- morelisp 5y agoNice in theory, but in practice the client might be a public web browser, or even something internal making a non-idempotent request.
- nrmitchi 5y agoIt could be, but in this instance it was not. It was an internal application that Gitlab had control over making an internal idempotent request. The underlying root cause should be identified so that this same issue can be corrected when retries aren't a solution, but if this was causing Gitlab Pages to burn through it's SLO (which is stated in the article), the Pages team could have (and should have) stopped the leak with a retry which the root cause was being determined.
- morelisp 5y agoBut an immediate retry would've ended up going over the same socket and getting the same response. Yes, "do some backoff", etc., but then we're in much murkier waters where Pages probably also has a latency SLI and if you're talking to a connection that might not let you know it's going to reset for up to 30 seconds, that's probably not going to fly either.
- jeffbee 5y agoWhy would a retry go over the same channel?
- morelisp 5y agoWhy would you close a TCP connection just because you got an HTTP error?
- jeffbee 5y agoI wouldn't, but I wouldn't use it again to serve the 2nd attempt of the same request.
- morelisp 5y agoThis would be at best unidiomatic and at worse, frustratingly complicated in code where you're sharing a client across many goroutines. You can do it (and I have done it, to deal with a very weird server) but reusing an http.Client is rarely seen as a source of problems - nor should it be. The client has no clue how many layers of proxy and lb might be between its contact and the real server; if it wants me gone, it should close it or respond with `Connection: close`. If you're in some other environments, e.g. a browser, this might not even be possible. And again: If the request is non-idempotent I'm SOL no matter what. (I don't see anything in the post that says it was or wasn't.)
- jeffbee 5y agoHaving a structured way to select the best among several backends is completely canonical and expected in systems like these. Reliable systems are built at the client, not at the backend. You get those extra 9s from having a smarter client, not from having a more reliable backend. Trying to make your lowest layers flawless is a fool's errand.
- nrmitchi 5y ago> an immediate retry would've ended up going over the same socket and getting the same response This is kind of a big assumption you're making. If you're going to do retries when a backend is unavailable, explicitly doing it against the same backend is obviously a decision that defeats the entire purpose. > probably also has a latency SLI Then take this into account w.r.t timeouts on internal requests/retries. My point is that this was a problem that was (at least partially) addressable by the team with the impacted SLO, and did not have to wait for someone to fix the root cause in order to improve their situation.