5 ms·

The world has moved on in the sense that "good enough" and cloud eats into their balance sheets I'm sure, but there's loads and loads of banks and legacy enterp

The world has moved on in the sense that "good enough" and cloud eats into their balance sheets I'm sure, but there's loads and loads of banks and legacy enterprises that maintain their ivory tower data centers and there's nothing to replace these with AFAIK. Google has Maglev, AWS perhaps something similar, MS no idea, everyone else just buys F5 or doesn't need it.

JackSlateur 3y ago
Lots of people are using haproxy
- eddieroger 3y ago
  My org moved off nginx for haproxy after we learned that (at the time, maybe it changed) reloading an nginx config, even if done gracefully through kernel signals, would drop existing connections, where haproxy could handle it gracefully. That was a fun week of diving in to some C code looking for why it was behaving that way.
  - nullify88 3y ago
    How did you come to that conclusion? I always believed a reload spawned new workers and let the old one drain off.
    - CogitoCogito 3y ago
      Yes I reload nginx all the time and it doesn’t drop connections. I just use the debian nginx package. Not sure what the gp is talking about.
      - nvarsj 3y ago
        Nginx abruptly drops http/1.1 persistent connections on reloads. This has been an issue forever and Maxim refused to ever fix it, saying it was to spec (yes it was, but there are better ways to deal with it). It’s a reason why many large, modern infra deployments have moved away from nginx.
        xorcist 3y ago
        It doesn't drop it, it's just not persistent on reload, isn't that what you mean? Actually dropping a connection mid-request is something I haven't seen nginx (or indeed Apache) do for many years despite doing some weird things with it. I can see where you're coming from, but it's not unreasonable behaviour, is it? Connections needs to migrated over to the new worker and that's how all major servers do it. If that's a problem then maybe something designed as proxy only instead of a real server is the way to go?
        nullify88 3y ago
        Isn't the typical behaviour of an application to re-establish the persistent connection on demand? I wonder what the requirement is to have these persistent with no timeout.
        Too 3y ago
        Yep. Persistent connections are bound to fail sooner or later anyway, so a robust application should have its own recovery.
        nvarsj 3y ago
        It doesn't drop mid request. But it closes the TCP socket abruptly after any in flight requests are completed. Clients have no idea the connection is closed, and try to reuse it and get back a RST. In heavily dynamic environments where nginx reloads happen frequently, it leads to large amounts of RSTs/broken connections and high error rates (you can't necessarily auto-retry a POST, a RST could mean anything). The sane approach is connection draining - you send a `connection: close` response header on the old worker, then finally remove any remaining idle connections at the end of the drain. In http/2 it's not an issue as it has a way for the server to explicitly say the connection is closed.
        eddieroger 3y ago
        I'm late to return to the thread, but this was the exact scenario we hit. We had it behind a CDN as well as behind an L4 load balancer for some very high volume internal consumers, and when it would just blast back RST packets, the consumers would freak out and break connection, returning errors on their end that weren't matched in our logs, unless we were lucky and got a 499 (now Maxim can talk about standards). As a general purpose reverse proxy for many clients on the Internet, I'm sure that's fine, but in our use case this made nginx unpredictable and no longer desirable.
  - jamespwilliams 3y ago
    nginx supports graceful reloading and I’m pretty sure it has for a very long time - there are references to it in the changelog from 2005 https://nginx.org/en/docs/control.html https://nginx.org/en/docs/control.html
  - zetsurin 3y ago
    we went in the opposite direction, not because haproxy was bad, just because nginx had a simpler config, and i think we were paying for haproxy but don't pay for nginx. all that said, neither drops existing connections on reload
  - yakshaving_jgt 3y ago
    Another issue with nginx IIRC is that it allows HTTP request smuggling, which is a critical security vulnerability.
    - callahad 3y ago
      That's been fixed for years. The CVE I can find was resolved in 1.17.7 (Dec 2019), and further hardening was applied in 1.21.1 (Jul 2021).
      - yakshaving_jgt 3y ago
        Can nginx send requests upstream over HTTP/2? I see this question has remained unanswered for a couple of years. https://security.stackexchange.com/questions/257823/what-are-nginx-reverseproxy-users-doing-to-prevent-http-request-smuggling https://security.stackexchange.com/questions/257823/what-are...
        callahad 3y ago
        It cannot. There's more detailed reasoning at https://trac.nginx.org/nginx/ticket/923 https://trac.nginx.org/nginx/ticket/923, but the tl;dr is: > nginx is already good at mitigating HTTP desync / request smuggling attacks, even without using HTTP/2 to backends. In particular because it normalizes Content-Length and Transfer-Encoding while routing requests (and also does not reuse connections to backend servers unless explicitly configured to do so)
jvolkman 3y ago
Amazon used to run entirely behind Citrix NetScaler hardware; no F5 at all. This was back in the early 2010s so I assume things have changed by now.
- doormatt 3y ago
  Yup - there was a massive internal push to move off of SSL terminating LBs back in ~2018
  - martinohansen 3y ago
    How come?
    - doormatt 3y ago
      Cost. Now, SSL termination is done at the host level, using a distributed SSL termination proxy developed by S3 called "JBLRelay"

JackSlateur 3y ago

Lots of people are using haproxy

eddieroger 3y ago
My org moved off nginx for haproxy after we learned that (at the time, maybe it changed) reloading an nginx config, even if done gracefully through kernel signals, would drop existing connections, where haproxy could handle it gracefully. That was a fun week of diving in to some C code looking for why it was behaving that way.
- nullify88 3y ago
  How did you come to that conclusion? I always believed a reload spawned new workers and let the old one drain off.
  - CogitoCogito 3y ago
    Yes I reload nginx all the time and it doesn’t drop connections. I just use the debian nginx package. Not sure what the gp is talking about.
    - nvarsj 3y ago
      Nginx abruptly drops http/1.1 persistent connections on reloads. This has been an issue forever and Maxim refused to ever fix it, saying it was to spec (yes it was, but there are better ways to deal with it). It’s a reason why many large, modern infra deployments have moved away from nginx.
      - xorcist 3y ago
        It doesn't drop it, it's just not persistent on reload, isn't that what you mean? Actually dropping a connection mid-request is something I haven't seen nginx (or indeed Apache) do for many years despite doing some weird things with it. I can see where you're coming from, but it's not unreasonable behaviour, is it? Connections needs to migrated over to the new worker and that's how all major servers do it. If that's a problem then maybe something designed as proxy only instead of a real server is the way to go?
        nullify88 3y ago
        Isn't the typical behaviour of an application to re-establish the persistent connection on demand? I wonder what the requirement is to have these persistent with no timeout.
        Too 3y ago
        Yep. Persistent connections are bound to fail sooner or later anyway, so a robust application should have its own recovery.
        nvarsj 3y ago
        It doesn't drop mid request. But it closes the TCP socket abruptly after any in flight requests are completed. Clients have no idea the connection is closed, and try to reuse it and get back a RST. In heavily dynamic environments where nginx reloads happen frequently, it leads to large amounts of RSTs/broken connections and high error rates (you can't necessarily auto-retry a POST, a RST could mean anything). The sane approach is connection draining - you send a `connection: close` response header on the old worker, then finally remove any remaining idle connections at the end of the drain. In http/2 it's not an issue as it has a way for the server to explicitly say the connection is closed.
        eddieroger 3y ago
        I'm late to return to the thread, but this was the exact scenario we hit. We had it behind a CDN as well as behind an L4 load balancer for some very high volume internal consumers, and when it would just blast back RST packets, the consumers would freak out and break connection, returning errors on their end that weren't matched in our logs, unless we were lucky and got a 499 (now Maxim can talk about standards). As a general purpose reverse proxy for many clients on the Internet, I'm sure that's fine, but in our use case this made nginx unpredictable and no longer desirable.
- jamespwilliams 3y ago
  nginx supports graceful reloading and I’m pretty sure it has for a very long time - there are references to it in the changelog from 2005 https://nginx.org/en/docs/control.html https://nginx.org/en/docs/control.html
- zetsurin 3y ago
  we went in the opposite direction, not because haproxy was bad, just because nginx had a simpler config, and i think we were paying for haproxy but don't pay for nginx. all that said, neither drops existing connections on reload
- yakshaving_jgt 3y ago
  Another issue with nginx IIRC is that it allows HTTP request smuggling, which is a critical security vulnerability.
  - callahad 3y ago
    That's been fixed for years. The CVE I can find was resolved in 1.17.7 (Dec 2019), and further hardening was applied in 1.21.1 (Jul 2021).
    - yakshaving_jgt 3y ago
      Can nginx send requests upstream over HTTP/2? I see this question has remained unanswered for a couple of years. https://security.stackexchange.com/questions/257823/what-are-nginx-reverseproxy-users-doing-to-prevent-http-request-smuggling https://security.stackexchange.com/questions/257823/what-are...
      - callahad 3y ago
        It cannot. There's more detailed reasoning at https://trac.nginx.org/nginx/ticket/923 https://trac.nginx.org/nginx/ticket/923, but the tl;dr is: > nginx is already good at mitigating HTTP desync / request smuggling attacks, even without using HTTP/2 to backends. In particular because it normalizes Content-Length and Transfer-Encoding while routing requests (and also does not reuse connections to backend servers unless explicitly configured to do so)

jvolkman 3y ago

Amazon used to run entirely behind Citrix NetScaler hardware; no F5 at all. This was back in the early 2010s so I assume things have changed by now.

doormatt 3y ago
Yup - there was a massive internal push to move off of SSL terminating LBs back in ~2018
- martinohansen 3y ago
  How come?
  - doormatt 3y ago
    Cost. Now, SSL termination is done at the host level, using a distributed SSL termination proxy developed by S3 called "JBLRelay"