4 ms·

> being hugged to death Whenever I read about this I can't help to think that people don't make a good use of a load balancer like HAProxy in front of their we

> being hugged to death Whenever I read about this I can't help to think that people don't make a good use of a load balancer like HAProxy in front of their web-servers. Even if it's a single server behind the load balancer. There's this sweet spot where your server is handling requests as fast as it can but if you cross above it everything halts. To make it simple, this usually happens when one or more hardware resources (CPU, RAM, disk) reaches 90% utilization. So just do some basic benchmarks and find that sweet spot and put a limit in HAProxy. Say 100 concurrent connections for static assets and 50 concurrent connections for everything else. All requests above the limit will go in a HAProxy queue but they should not stay there for long because the web-server is able to work at full speed without being overloaded. PS: I have quite a dream. To write some fabulous blog post that reaches the HN front page and serve it from a raspberry pi device. Too bad I don't write at all.

notheguyouthink 9y ago
This is informative, thank you. Any HAProxy-like software you recommend? Or is HAProxy the king?
xigency 9y ago
You might be able to climb the ladder with a meta-post. Just do a write-up on how to host a front page post on a Raspberry Pi and tune the headline for the right audience. "This Site Can Handle 1,000,000 Connections on a Raspberry Pi" If you could collect data from the experiment and do a postmortem, that'd be great, too.
- smallbigfish 9y ago
  > postmortem Hehe, why so negative?
jaggederest 9y ago
I've actually dealt with this precise problem, HAProxy doesn't really solve the problem of an excess of traffic. At some point you'll start dropping connections or people will give up waiting. What you really want is something more like Varnish, a pass through caching proxy that handles known requests from memory, and any novel requests go through.
- smallbigfish 9y ago
  I agree that Varnish might be a good addition but in the end it depends on the exact setup you're running and which hardware resource is used the most. My opinion is that the CPU is usually the first to fall and you will not help it quite that much with an in-memory cache when the actual problem is the https overhead or the PHP/DB processing.
peterwwillis 9y ago
What you're describing is queueing, but the solution you're really looking for is back pressure. Any queue will eventually fill and when you fill it, things fall out. So rather than just dropping requests or letting the requests swamp your box, what you want to do is intelligently reject requests. On the web, this means sending 503 responses. In a more intelligent client-server model, this might be the client doing an exponential back-off retry with jitter. A plain-old retry can swamp the server and just kill everything. An exponential back-off retry will allow the backend to achieve a level of service that isn't fast, but also isn't unavailable. Hopefully in this case the haproxy is used intelligently and simply returns cached responses to GET requests without query strings. You don't want to hold onto connections or pass them off to the dynamic layer behind the proxy if you're overloaded. You can also respond to the client with a page with an auto-refresh, implementing an exponential back-off retry.
- smallbigfish 9y ago
  The setup I described will not get you from 10 requests/second to 1000. It will let you to make the most of your hardware though.

notheguyouthink 9y ago

This is informative, thank you. Any HAProxy-like software you recommend? Or is HAProxy the king?

xigency 9y ago

You might be able to climb the ladder with a meta-post. Just do a write-up on how to host a front page post on a Raspberry Pi and tune the headline for the right audience. "This Site Can Handle 1,000,000 Connections on a Raspberry Pi" If you could collect data from the experiment and do a postmortem, that'd be great, too.

smallbigfish 9y ago
> postmortem Hehe, why so negative?

jaggederest 9y ago

I've actually dealt with this precise problem, HAProxy doesn't really solve the problem of an excess of traffic. At some point you'll start dropping connections or people will give up waiting. What you really want is something more like Varnish, a pass through caching proxy that handles known requests from memory, and any novel requests go through.

smallbigfish 9y ago
I agree that Varnish might be a good addition but in the end it depends on the exact setup you're running and which hardware resource is used the most. My opinion is that the CPU is usually the first to fall and you will not help it quite that much with an in-memory cache when the actual problem is the https overhead or the PHP/DB processing.

peterwwillis 9y ago

What you're describing is queueing, but the solution you're really looking for is back pressure. Any queue will eventually fill and when you fill it, things fall out. So rather than just dropping requests or letting the requests swamp your box, what you want to do is intelligently reject requests. On the web, this means sending 503 responses. In a more intelligent client-server model, this might be the client doing an exponential back-off retry with jitter. A plain-old retry can swamp the server and just kill everything. An exponential back-off retry will allow the backend to achieve a level of service that isn't fast, but also isn't unavailable. Hopefully in this case the haproxy is used intelligently and simply returns cached responses to GET requests without query strings. You don't want to hold onto connections or pass them off to the dynamic layer behind the proxy if you're overloaded. You can also respond to the client with a page with an auto-refresh, implementing an exponential back-off retry.

smallbigfish 9y ago
The setup I described will not get you from 10 requests/second to 1000. It will let you to make the most of your hardware though.