3 ms·

> If you're running HAProxy on AWS, be sure your AWS firewall rules are stateless, otherwise you may well run into the unpublished connection tracking limits.

> If you're running HAProxy on AWS, be sure your AWS firewall rules are stateless, otherwise you may well run into the unpublished connection tracking limits. This. I got bit by this the other day, a bunch of servers were having very low CPU usage but new connections to them were failing. Turns out it was because they were hitting the conntrack limit. Changing the security group rules to allow all from all "fixed" it. Although after a quick test, I think there's another limit, also unpublished, as I still can't establish as many connections as I want on a smaller instance – same number of requests on fewer connections works, and also a bigger instance works with many connections. In every case, the CPU was largely unused. AWS doc regarding the issue: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/security-group-connection-tracking.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/security...

toast0 5y ago
> Although after a quick test, I think there's another limit, also unpublished, as I still can't establish as many connections as I want on a smaller instance. Any chance this is an OS limit in your instance? Either a connection tracking firewall there, or FD/socket count limits? I don't have experience beyond the AWS connection tracking limits because I ran into them while testing something, but only found out how to avoid them much later (the test didn't go well, because of the limits, so we didn't go forward, and our assigned rep didn't tell us about the workaround either; I only found out by complaining about the limit to enough people that someone knew how to fix it)
- vladvasiliu 5y ago
  I doubt that, because the same exact VM, after changing to a beefier instance type (c5.4xl, instead of the original t3.micro and later c5.large) accepted the connections without any problem. If it's OS related, it may be some dynamic limit based on RAM size. But the limit was something around 100-200 connections, so nothing particularly crazy. The way I found out about the limit was that downstream servers couldn't establish connections, and a manual test showed significant packet loss, while the servers were under 10% CPU and accepted connections locally. I opened a support ticket and the guy apparently saw that they were hitting the conntrack limit. Once I replaced the security group with a blanket allow all as he suggested, connections could be established again. But two days later, when the number of connections increased further (while the security group was still allow all), I started seeing again connections not being established, although the servers were still not particularly loaded. As these are UDP (DNS) servers, I can't usefully quantify the number of different clients. I set up a quick lab with two isolated EC2 instances and figured there must be another limit to the number of connections, although it seems somewhat higher – or maybe the conntrack limit as shown by ethtool doesn't update immediately. This happened yesterday, Saturday, while I was on call, so I didn't push the tests far enough to reach any clear-cut conclusion. I just doubled the servers and everything went back to normal. However, the limit does seem to be related to the instance size and per interface. Meaning that adding another interface doubled the connections I could establish. Again, I didn't push the tests any further, though I intend to this sometime next week.

toast0 5y ago

> Although after a quick test, I think there's another limit, also unpublished, as I still can't establish as many connections as I want on a smaller instance. Any chance this is an OS limit in your instance? Either a connection tracking firewall there, or FD/socket count limits? I don't have experience beyond the AWS connection tracking limits because I ran into them while testing something, but only found out how to avoid them much later (the test didn't go well, because of the limits, so we didn't go forward, and our assigned rep didn't tell us about the workaround either; I only found out by complaining about the limit to enough people that someone knew how to fix it)

vladvasiliu 5y ago
I doubt that, because the same exact VM, after changing to a beefier instance type (c5.4xl, instead of the original t3.micro and later c5.large) accepted the connections without any problem. If it's OS related, it may be some dynamic limit based on RAM size. But the limit was something around 100-200 connections, so nothing particularly crazy. The way I found out about the limit was that downstream servers couldn't establish connections, and a manual test showed significant packet loss, while the servers were under 10% CPU and accepted connections locally. I opened a support ticket and the guy apparently saw that they were hitting the conntrack limit. Once I replaced the security group with a blanket allow all as he suggested, connections could be established again. But two days later, when the number of connections increased further (while the security group was still allow all), I started seeing again connections not being established, although the servers were still not particularly loaded. As these are UDP (DNS) servers, I can't usefully quantify the number of different clients. I set up a quick lab with two isolated EC2 instances and figured there must be another limit to the number of connections, although it seems somewhat higher – or maybe the conntrack limit as shown by ethtool doesn't update immediately. This happened yesterday, Saturday, while I was on call, so I didn't push the tests far enough to reach any clear-cut conclusion. I just doubled the servers and everything went back to normal. However, the limit does seem to be related to the instance size and per interface. Meaning that adding another interface doubled the connections I could establish. Again, I didn't push the tests any further, though I intend to this sometime next week.