3 ms·
The Linux network stack has been the bane of my existence trying to squeeze more performance out of KeyDB. I really hope it gets this kind of love in the future
by jdsully 7y ago
The Linux network stack has been the bane of my existence trying to squeeze more performance out of KeyDB. I really hope it gets this kind of love in the future.
On Linux there’s a spinlock in do soft_irq that blocks even in non-blocking IO.
- easytiger 7y agoHave you tried optimising and benchmarking on a 2.6 kernel? I found the more recent kennels in rhel7 that the amount of kernel work being done very hard to control. The first core on any numa node is especially problematic and induce horrific jitter
- shaklee3 7y agoWhy not use DPDK? All of these problems go away, and people have reported hitting 1Tbps on a single node.
- jdsully 7y agoWe have! https://docs.keydb.dev/blog/2019/06/17/blog-post/ https://docs.keydb.dev/blog/2019/06/17/blog-post/ Still though I hate seeing needless waste. There’s no reason the active thread needs to block on a soft_irq when there are unused cores to process them.
- morning_gelato 7y agoHave you tried reaching out to the linux netdev mailing list about this? They may at least be able to explain their design decision.
- shaklee3 7y agoCan you move all softirqs into different cores? That's usually one of the dpdk tuning steps.
- jdsully 7y agoMy reading of the source code is there is no way to prevent this. The spinlock is guarding the check which determines if there is work to do. Its been a few months and I had intended to go back and try using a try_lock(). But I’m not normally a kernel dev.
- throw0101a 7y ago> I really hope it gets this kind of love in the future. Have you considered using FreeBSD? ;)
- jdsully 7y agoUnfortunately the vast majority of KeyDB users run Linux.