6 ms·

It could be, but in this instance it was not. It was an internal application that Gitlab had control over making an internal idempotent request. The underlying

It could be, but in this instance it was not. It was an internal application that Gitlab had control over making an internal idempotent request. The underlying root cause should be identified so that this same issue can be corrected when retries aren't a solution, but if this was causing Gitlab Pages to burn through it's SLO (which is stated in the article), the Pages team could have (and should have) stopped the leak with a retry which the root cause was being determined.

morelisp 4y ago
But an immediate retry would've ended up going over the same socket and getting the same response. Yes, "do some backoff", etc., but then we're in much murkier waters where Pages probably also has a latency SLI and if you're talking to a connection that might not let you know it's going to reset for up to 30 seconds, that's probably not going to fly either.
- jeffbee 4y ago
  Why would a retry go over the same channel?
  - morelisp 4y ago
    Why would you close a TCP connection just because you got an HTTP error?
    - jeffbee 4y ago
      I wouldn't, but I wouldn't use it again to serve the 2nd attempt of the same request.
      - morelisp 4y ago
        This would be at best unidiomatic and at worse, frustratingly complicated in code where you're sharing a client across many goroutines. You can do it (and I have done it, to deal with a very weird server) but reusing an http.Client is rarely seen as a source of problems - nor should it be. The client has no clue how many layers of proxy and lb might be between its contact and the real server; if it wants me gone, it should close it or respond with `Connection: close`. If you're in some other environments, e.g. a browser, this might not even be possible. And again: If the request is non-idempotent I'm SOL no matter what. (I don't see anything in the post that says it was or wasn't.)
        jeffbee 4y ago
        Having a structured way to select the best among several backends is completely canonical and expected in systems like these. Reliable systems are built at the client, not at the backend. You get those extra 9s from having a smarter client, not from having a more reliable backend. Trying to make your lowest layers flawless is a fool's errand.
        morelisp 4y ago
        If public, I'd love to see a real example of Go code you believe does this correctly.
        jeffbee 4y ago
        https://github.com/grpc/grpc-go/tree/master/balancer https://github.com/grpc/grpc-go/tree/master/balancer
        morelisp 4y ago
        I mean yeah if HTTP is your layer 4 and not your layer 7 you can do this transparently, but you also just kicked the problem up a level. gRPC connections will have the same issue if you return some gRPC error code. In gRPC terms the connection state was always Ready, there’s no reason the conn would’ve been kicked out of the lb pool.
        jeffbee 4y ago
        It doesn't necessarily have to get kicked out of the pool. You can use error-averse balancers that will never send the same request to a channel that already gave it an error, or you can have N channels and just roll the dice on a (N-1)/N probability of getting a different channel the second time.
- nrmitchi 4y ago
  > an immediate retry would've ended up going over the same socket and getting the same response This is kind of a big assumption you're making. If you're going to do retries when a backend is unavailable, explicitly doing it against the same backend is obviously a decision that defeats the entire purpose. > probably also has a latency SLI Then take this into account w.r.t timeouts on internal requests/retries. My point is that this was a problem that was (at least partially) addressable by the team with the impacted SLO, and did not have to wait for someone to fix the root cause in order to improve their situation.

morelisp 4y ago

But an immediate retry would've ended up going over the same socket and getting the same response. Yes, "do some backoff", etc., but then we're in much murkier waters where Pages probably also has a latency SLI and if you're talking to a connection that might not let you know it's going to reset for up to 30 seconds, that's probably not going to fly either.

jeffbee 4y ago
Why would a retry go over the same channel?
- morelisp 4y ago
  Why would you close a TCP connection just because you got an HTTP error?
  - jeffbee 4y ago
    I wouldn't, but I wouldn't use it again to serve the 2nd attempt of the same request.
    - morelisp 4y ago
      This would be at best unidiomatic and at worse, frustratingly complicated in code where you're sharing a client across many goroutines. You can do it (and I have done it, to deal with a very weird server) but reusing an http.Client is rarely seen as a source of problems - nor should it be. The client has no clue how many layers of proxy and lb might be between its contact and the real server; if it wants me gone, it should close it or respond with `Connection: close`. If you're in some other environments, e.g. a browser, this might not even be possible. And again: If the request is non-idempotent I'm SOL no matter what. (I don't see anything in the post that says it was or wasn't.)
      - jeffbee 4y ago
        Having a structured way to select the best among several backends is completely canonical and expected in systems like these. Reliable systems are built at the client, not at the backend. You get those extra 9s from having a smarter client, not from having a more reliable backend. Trying to make your lowest layers flawless is a fool's errand.
        morelisp 4y ago
        If public, I'd love to see a real example of Go code you believe does this correctly.
        jeffbee 4y ago
        https://github.com/grpc/grpc-go/tree/master/balancer https://github.com/grpc/grpc-go/tree/master/balancer
        morelisp 4y ago
        I mean yeah if HTTP is your layer 4 and not your layer 7 you can do this transparently, but you also just kicked the problem up a level. gRPC connections will have the same issue if you return some gRPC error code. In gRPC terms the connection state was always Ready, there’s no reason the conn would’ve been kicked out of the lb pool.
        jeffbee 4y ago
        It doesn't necessarily have to get kicked out of the pool. You can use error-averse balancers that will never send the same request to a channel that already gave it an error, or you can have N channels and just roll the dice on a (N-1)/N probability of getting a different channel the second time.
nrmitchi 4y ago
> an immediate retry would've ended up going over the same socket and getting the same response This is kind of a big assumption you're making. If you're going to do retries when a backend is unavailable, explicitly doing it against the same backend is obviously a decision that defeats the entire purpose. > probably also has a latency SLI Then take this into account w.r.t timeouts on internal requests/retries. My point is that this was a problem that was (at least partially) addressable by the team with the impacted SLO, and did not have to wait for someone to fix the root cause in order to improve their situation.