3 ms·
First, you should test on an unloaded domU, not anything in EC2; - EC2 will dynamically adjust your CPU share as you try and use it, so you will _not_ get cons
by abofh 15y ago
First, you should test on an unloaded domU, not anything in EC2;
- EC2 will dynamically adjust your CPU share as you try and use it, so you will _not_ get consistent results over any short period.
- EC2 is subject to other people's loads, which may be IO, CPU or network bound.
Going from there:
- Xen is slower any time you need dom0/domU coordination - it wouldn't surprise me to learn that there's some sort of coordination happening in accept() to tag the session through the upper dom0 firewalls.
- You don't describe what your backlog is on the listening socket, but you should make sure you're accepting as many as you can during your CPU share on EC2 -- your slice _will_ be interrupted at inopportune times.
Finally, EC2 is _lousy_ performance-wise, especially w/r/t disk IO - it doesn't sound like it, but if you're logging to disk after accept(), this could be the killer.
Tangentially -- you _might_ get better accept() performance if you turn ON syncookies, as then the handshake occurs basically at the kernel, and the accept() is only relevant _after_ the handshake is done. It's a bit hacky, but with _large_ numbers of connections, it can improve your performance a bit.
- justincormack 15y agoHe is not using EC2...
- guan 15y agoThe author of the original article is, at least for some of his tests: “(on an 8-core EC2 instance, c1.xlarge running Xen)”
- justincormack 15y agoYeah, true, rereading it he is not clear about whether he has run the software on xen on the same box he ran raw on, although it is implied. It is a very different question on EC2.
- cgbystrom 15y agoI've tried it on variety of different servers/hardware using both my server+apachebench and netperf as benchmarking tools. * On EC2 I've tried the c1.xlarge and the giant cc1.4xlarge. With cc1.4xlarge, I saw maybe a ~10% increase accept() rate. * Two separate, virtualized servers at the office. * A private, virtualized server on Rackspace was briefly tested as well. A compilation of netperf results are available at https://gist.github.com/985475 https://gist.github.com/985475
- bigiain 15y agoI doubt it'll reveal anything super useful, but running your tests on a single-tenant EC2 instance could add an extra datapoint. I suspect it'll only reveal that the problem _isn't_ due to EC2/Xen swapping your virtualized machine out. (assuming you haven't tried it already...)
- cgbystrom 15y ago(I wrote the Serverfault question) Thanks for the suggestions, here's some clarifications: * I clarified what hardware I've tested in a comment (see below). * I've run tests with my server ranging from 10 sec up to 10 minutes. They're consistently bad unfortunately. * Interesting what you say about dom0/domU. I'm no Xen guru, but the culprit is probably something like that. I've been using a backlog of 1024 for the server tests (set both in Java land and sysctl.conf). The netperf are all defaults, both in terms of run time and backlog. Was actually trying to monitor the backlog somehow, but I'm not sure that's even possible in Linux? * The server isn't doing anything disk IO-bound so this shouldn't be the case. * syncookies seems like a good idea, I will definitely try that along with a lot bigger backlog and see if it makes any difference. I'll also try and see if netperf can be tweaked as well to provide a better, isolated test case. Writing this off as Xen overhead would be such a shame, virtualization should not cause this much overhead. I'll continue investigating!
- ay 15y agoA nit re. syncoookies: TCP three-way handshake always occurs in the kernel. You can grab the kernel source, the relevant stuff is in net/ipv4/tcp_input.c; What syncookie may somewhat help you with is if you have super-large number of half-open (SYNRCVD state) connections - and even then, the data structures for those are supposed to be efficient enough for this not to be a problem. OTOH, what you will be trading off with syncookies, is that they subtly violate the TCP standard, and especially this will be a problem with the "server talks first" connections (like [E]SMTP, SSH): if your third ACK of the three way handshake gets lost on the way from client to the server, the canonical TCP implementation would have retransmitted the SYNACK from the server side. Except that the whole point of the syncookies is not to keep the state on the server side, i.e. there is nothing that can at all retransmit the SYNACK. The HTTP, being by nature "client talks first" protocol, hides this problem (the ACK for the SYNACK will be effectively retransmitted because it will be part of the data segment), but I thought it might be useful to remind that turning syncookies on by default is not the standard modus operandi.