3 ms·
I had a similar issue with Windows kernels "recently" (2016~?)... I don't have the memory or patience to write a long and inspiring blog post, but it comes dow
by dijit 6y ago
I had a similar issue with Windows kernels "recently" (2016~?)...
I don't have the memory or patience to write a long and inspiring blog post, but it comes down to:
Even with IOCP/multiple threads: network traffic is single threaded in the kernel, even worse, there's a mutex there. Putting the effective limit on PPS for windows to something like 1.1M for 3.0GHz.
The task of this machine was /basically/ a connection multiplexer with some TLS offloading; so listen on a socket, get an encrypted connection, check your connection pool and forward where appropriate.
Our machine basically sat waiting (in kernel space) for this lock 99.7% of the time, 0.3% was spent on SSL handshaking..
We solved our "issue" by spreading such load over many more machines and gave them low-core-count high-clock-speed Xeons instead of the normal complement of 20vCPU Xeons.
AFAIK that issue persists, I'd be interested to know if someone else managed to coerce windows to do the right thing here.
- kevingadd 6y agoDo you know whether it's a single thread for all network devices, or just per device? It would be interesting if this ended up being a driver level constraint or something that can be fixed by having multiple NICs in the machine.
- dijit 6y agoit was LACP'd... buuuuuuut; it was only one PCIe card. Otherwise you can't easily do LACP because it can't be offloaded to the card. We tried without LACP, but again, only one PCIe card.
- jandrese 6y agoDid it have to be Windows? This is the sort of thing Linux or *BSD boxes are better suited for. I wouldn't even consider a Windows machine for the task unless there's some sort of licensed software you need to run on it to get the job done.
- dijit 6y ago> This is the sort of thing Linux or *BSD boxes are better suited for. Definitely, though enabling conntrack on Linux has similar characteristics (forces single thread with some kind of internal mutex) though it can do 5x the b/w.. We tried having stateful firewalls in front of our windows boxen, that's how I know. Seems like Cloudflare has an older blog post detailing this too: https://blog.cloudflare.com/conntrack-tales-one-thousand-and-one-flows/ https://blog.cloudflare.com/conntrack-tales-one-thousand-and... Anyway, to answer your question: AAA GameDev (and their backends, if highly tailored) are Windows.
- toast0 6y agoI did some work optimizing a similar problem, but simpler and on another OS[1]. The basic concept that worked was Receive Side Scaling (RSS), which was developed by Microsoft, for Windows Server. Did you come accross that? It needs support in the NIC and the driver, but intel gigE cards do it, so you don't need the really fancy cards. I don't know what the interface is like for Windows, but inbound RSS for FreeBSD is pretty easy, and skimming Windows docs, it seemed like you could do more advanced things there. The harder part was aligning the outgoing connections; for max performance, you want all of the related connections pinned to the same CPU, so that there's no inter CPU messaging; for me that meant a frontend connection needs to hash to the same NIC queue as the backend connection; for you, that needs to be all of the demultiplexed connections on the same queue as the multiplexed connection. Windows may have an API to make connections that will hash properly, FreeBSD didn't (doesn't?), so my code had to manage the local source ip and port when connecting to remote servers so that the connection would hash as needed. Assuming a lot of connections, you end up needing to self-manage source ip and port anyway, and at least HAProxy has code for that already, but running the rss hash to qualify ports was new development, and a bit tricky because bulk calculating it gets costly. Once I got everything setup well with respect to CPUs, things got a lot better; still had some kernel bottlenecks though, I wouldn't know how to resolve that for Windows, but there were some easy wins for FreeBSD. Low core count is the right way to go though; I think the NICs I used could only do 16 way RSS hashing, so my dual 14 core xeon (2690v4) weren't a great fit; 12 cores were 100% idle all the time; something power of two would be best. Email in profile if you want to continue the discussion off HN (or after it fizzles out here). [1] Load balancing/proxying, but no TLS and no multiplexing, on FreeBSD.
- drewg123 6y agoDo you actually use RSS via options RSS / options PCBGROUP? I've tried it several times, and its just so hard to get right & have matching cores / rx rings, etc. I've made it work with a local patch to nginx, but it was so fragile that I abandoned it. I had been thinking that RSS/PCBGROUP was totally abandoned and could potentially be removed.
- toast0 6y ago
- betaporter 6y agoSounds like you didn't have receive side scaling enabled; by default flows are queued to core 0 to prevent reordering. If you enable RSS, your flows will be hashed to core-specific queues. It's inaccurate to describe traffic processing as single-threaded in the kernel.
- dboreham 6y ago> network traffic is single threaded in the kernel Hmm. It definitely was not years ago. Perhaps something to do with a specific NIC driver? Or perhaps Cutler retired?
- slowstart 6y agoYou should look into the RSS configuration. https://docs.microsoft.com/en-us/windows-hardware/drivers/network/introduction-to-receive-side-scaling https://docs.microsoft.com/en-us/windows-hardware/drivers/ne... and https://docs.microsoft.com/en-us/windows-server/networking/technologies/network-subsystem/net-sub-performance-tuning-nics https://docs.microsoft.com/en-us/windows-server/networking/t...