5 ms·

Meta: NamePros tech admin here. This hammered our servers so hard that we actually uncovered a fairly obscure bug in Nginx's FastCGI caching. It's gone unnot

Meta: NamePros tech admin here. This hammered our servers so hard that we actually uncovered a fairly obscure bug in Nginx's FastCGI caching. It's gone unnoticed for years, including during rigorous load testing.

hathawsh 8y ago
More details please. :-) This sounds like it deserves a full article.
- zenexer 8y ago
  It probably does. It'll take me a while to dig through the source to find the cause, but preliminary evidence suggests that fastcgi_cache_key doesn't handle $cookie_* variables correctly. If the cookie doesn't exist, it seems to result in a unique key every time. (Edit: The workaround is to assign the $cookie_* variable to another variable and then use that in fastcgi_cache_key.) If anyone is curious what sort of increase in traffic you should expect from a link to your site appearing on HN, we're getting about 230 additional backend requests per second--that doesn't include anything cached by Cloudflare. You can monitor it in realtime here: https://www.nameprosstatus.com/ https://www.nameprosstatus.com/
  - nimbosa 8y ago
    nice live stats monitor, can you link a short short postmortem analysis under that, shortly? Thanks!
    - zenexer 8y ago
      Yup, it'll be on there once I finish debugging. Edit: I'm having trouble reproducing it in a controlled environment, so a proper post mortem is probably going to have to wait for another day.
      - nimbosa 8y ago
        true that, some corner cases only appear in the wild and very hard to reproduce as we don't really know what caused them, anyways, keep us updated.. ;)
jedberg 8y ago
Nginx has had some strange bugs under heavy load. Way back in the day, reddit used Nginx as its frontend webserver/load balancer. All of a sudden one day I noticed that it was sending 99% of the traffic to the app server listed first in the config, then 99% of the remaining traffic to the next one, and so on down the line, such that the last app server in the list (only about 10 at the time) was essentially getting no traffic. If I changed the config, the traffic pattern was still based on the config order, so it wasn't a bug in the hashing or the numerical values of the IPs of the app servers or anything like that. Never did figure out what caused that problem. We just switched out Nginx for Haproxy and never had a problem again. :)
- brynjolf 8y ago
  How does it feel knowing you spend everyday enabling people who send death threats and are influenced by Russian bots? I mean all your efforts just make the internet overall a worse place. Must be hard.
  - dang 8y ago
    Uncivil comments will get you banned here. Please review https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and follow the rules when posting to HN. That means commenting civilly and substantively, or not at all.
- Calib3r 8y ago
  ConnectionLeaseTimeout?

hathawsh 8y ago

More details please. :-) This sounds like it deserves a full article.

zenexer 8y ago
It probably does. It'll take me a while to dig through the source to find the cause, but preliminary evidence suggests that fastcgi_cache_key doesn't handle $cookie_* variables correctly. If the cookie doesn't exist, it seems to result in a unique key every time. (Edit: The workaround is to assign the $cookie_* variable to another variable and then use that in fastcgi_cache_key.) If anyone is curious what sort of increase in traffic you should expect from a link to your site appearing on HN, we're getting about 230 additional backend requests per second--that doesn't include anything cached by Cloudflare. You can monitor it in realtime here: https://www.nameprosstatus.com/ https://www.nameprosstatus.com/
- nimbosa 8y ago
  nice live stats monitor, can you link a short short postmortem analysis under that, shortly? Thanks!
  - zenexer 8y ago
    Yup, it'll be on there once I finish debugging. Edit: I'm having trouble reproducing it in a controlled environment, so a proper post mortem is probably going to have to wait for another day.
    - nimbosa 8y ago
      true that, some corner cases only appear in the wild and very hard to reproduce as we don't really know what caused them, anyways, keep us updated.. ;)

jedberg 8y ago

Nginx has had some strange bugs under heavy load. Way back in the day, reddit used Nginx as its frontend webserver/load balancer. All of a sudden one day I noticed that it was sending 99% of the traffic to the app server listed first in the config, then 99% of the remaining traffic to the next one, and so on down the line, such that the last app server in the list (only about 10 at the time) was essentially getting no traffic. If I changed the config, the traffic pattern was still based on the config order, so it wasn't a bug in the hashing or the numerical values of the IPs of the app servers or anything like that. Never did figure out what caused that problem. We just switched out Nginx for Haproxy and never had a problem again. :)

brynjolf 8y ago
How does it feel knowing you spend everyday enabling people who send death threats and are influenced by Russian bots? I mean all your efforts just make the internet overall a worse place. Must be hard.
- dang 8y ago
  Uncivil comments will get you banned here. Please review https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and follow the rules when posting to HN. That means commenting civilly and substantively, or not at all.
Calib3r 8y ago
ConnectionLeaseTimeout?