4 ms·
I doubt that, because the same exact VM, after changing to a beefier instance type (c5.4xl, instead of the original t3.micro and later c5.large) accepted the co
by vladvasiliu 5y ago
I doubt that, because the same exact VM, after changing to a beefier instance type (c5.4xl, instead of the original t3.micro and later c5.large) accepted the connections without any problem.
If it's OS related, it may be some dynamic limit based on RAM size. But the limit was something around 100-200 connections, so nothing particularly crazy.
The way I found out about the limit was that downstream servers couldn't establish connections, and a manual test showed significant packet loss, while the servers were under 10% CPU and accepted connections locally. I opened a support ticket and the guy apparently saw that they were hitting the conntrack limit.
Once I replaced the security group with a blanket allow all as he suggested, connections could be established again. But two days later, when the number of connections increased further (while the security group was still allow all), I started seeing again connections not being established, although the servers were still not particularly loaded. As these are UDP (DNS) servers, I can't usefully quantify the number of different clients.
I set up a quick lab with two isolated EC2 instances and figured there must be another limit to the number of connections, although it seems somewhat higher – or maybe the conntrack limit as shown by ethtool doesn't update immediately. This happened yesterday, Saturday, while I was on call, so I didn't push the tests far enough to reach any clear-cut conclusion. I just doubled the servers and everything went back to normal.
However, the limit does seem to be related to the instance size and per interface. Meaning that adding another interface doubled the connections I could establish. Again, I didn't push the tests any further, though I intend to this sometime next week.