5 ms·

Can you share more details about what the 4 HAProxy servers are doing? I run a SaaS too, but on a single beefy machine handling web and DB stuff. I'm curious wh

Can you share more details about what the 4 HAProxy servers are doing? I run a SaaS too, but on a single beefy machine handling web and DB stuff. I'm curious what I'm missing out on not using HAProxy in front.

russellendicott 4y ago
In the post he says they're for SSL termination and for rolling upgrades of the servers behind them.
cuu508 4y ago
To answer the question of why four of them: * the traffic from the monitored systems comes with spikes. Looking at netdata graphs, currently the baseline is 600 requests/s, but there is a 2000 requests/s spike every minute, and 4000 requests/s spike every 10 minutes. * want to maintain redundancy and capacity even when a load balancer is removed from DNS rotation (due to network problems, or for upgrade) There are spare resources on the servers, especially RAM, and I could pack more things on fewer hosts. But, with Hetzner prices, why bother? :-)
- tothrowaway 4y ago
  I didn't realize you were getting that many requests per second. That's quite remarkable when you only have 18 app cores available. In my experience, you're lucky if you can get Django to service a request in 10ms. So I'd expect closer to 40 cores to handle that level of traffic. I suppose since it is spiky, and not a particularly latency sensitive application, you can get by with fewer cores.
  - cuu508 4y ago
    I've got a small program written in Golang for handling the HTTP "ping" requests. There's also rate limiting, not every request makes it past NGINX.
  - deleted 4y ago
    [deleted]
- dubbel 4y ago
  Damn, I never thought about it, but I guess the next cron/scheduling system should really be configurable down to the second, and if it is not defined set a random value between 0-59 seconds as a default. That might make the entire internet traffic and global compute use less spiky.
- boxed 4y ago
  Why uWSGI? Every time I've talked to someone who uses it and got them to try gunicorn they've seen quite big performance improvement for no code change and switched.
- mdasen 4y ago
  I understand having multiple proxies, but it seems a bit odd to have more proxy servers than web servers, no? Do they also do caching so that of the up-to 4,000 requests/second, a lot of them never reach the web servers? Presumably the Python code is more resource intensive than HAProxy.

russellendicott 4y ago

In the post he says they're for SSL termination and for rolling upgrades of the servers behind them.

cuu508 4y ago

To answer the question of why four of them: * the traffic from the monitored systems comes with spikes. Looking at netdata graphs, currently the baseline is 600 requests/s, but there is a 2000 requests/s spike every minute, and 4000 requests/s spike every 10 minutes. * want to maintain redundancy and capacity even when a load balancer is removed from DNS rotation (due to network problems, or for upgrade) There are spare resources on the servers, especially RAM, and I could pack more things on fewer hosts. But, with Hetzner prices, why bother? :-)

tothrowaway 4y ago
I didn't realize you were getting that many requests per second. That's quite remarkable when you only have 18 app cores available. In my experience, you're lucky if you can get Django to service a request in 10ms. So I'd expect closer to 40 cores to handle that level of traffic. I suppose since it is spiky, and not a particularly latency sensitive application, you can get by with fewer cores.
- cuu508 4y ago
  I've got a small program written in Golang for handling the HTTP "ping" requests. There's also rate limiting, not every request makes it past NGINX.
- deleted 4y ago
  [deleted]
dubbel 4y ago
Damn, I never thought about it, but I guess the next cron/scheduling system should really be configurable down to the second, and if it is not defined set a random value between 0-59 seconds as a default. That might make the entire internet traffic and global compute use less spiky.
boxed 4y ago
Why uWSGI? Every time I've talked to someone who uses it and got them to try gunicorn they've seen quite big performance improvement for no code change and switched.
mdasen 4y ago
I understand having multiple proxies, but it seems a bit odd to have more proxy servers than web servers, no? Do they also do caching so that of the up-to 4,000 requests/second, a lot of them never reach the web servers? Presumably the Python code is more resource intensive than HAProxy.