4 ms·
> Networks are not reliable, despite what you hear, so latency is used to mask re-tries and delays. Of course, but even the 50%ile case is strangely slow, and
by scottlamb 3y ago
> Networks are not reliable, despite what you hear, so latency is used to mask re-tries and delays.
Of course, but even the 50%ile case is strangely slow, and if that involves retries something is deeply wrong.
- KaiserPro 3y agoYou're right, but TCP doesn't like packets being dropped halfway through a stream. If you have a highly QoS'd link then you'll see latency spikes.
- scottlamb 3y agoAgain, I'm not talking about spikes (though better tail latency is always desirable) but poor latency in the 50%ile case. And for high-QoS applications, not batch stuff. The snap paper linked elsewhere in the thread shows 10 µs latencies; they've put in some optimization to achieve that, but I don't really understand why we don't expect close to that with standard kernel networking and TCP.
- vitus 3y ago> The snap paper linked elsewhere in the thread shows 10 µs latencies; they've put in some optimization to achieve that, but I don't really understand why we don't expect close to that with standard kernel networking and TCP. You can get similar results by looking at comparisons between DPDK and kernel networking. Most of the usual gap comes from not needing to context-switch for kernel interrupt handling, zero-copy abstractions, and busy polling (wherein you trade CPU for lower latency instead of sleeping between iterations if there's no work to be done). https://talawah.io/blog/linux-kernel-vs-dpdk-http-performance-showdown/ https://talawah.io/blog/linux-kernel-vs-dpdk-http-performanc... goes into some amount of detail comparing request throughput of an unoptimized kernel networking stack, optimized kernel networking stack, and DPDK. I'm not aware of any benchmarks (public or private) comparing Snap vs DPDK vs Linux, so that's probably as close as you'll get.
- CoolCold 3y agoGreat reading, thanks for the link on vanilla vs dpdk
- scottlamb 3y agoThanks for the link. How does this compare to the analogous situation for SSD access? I know there are also userspace IO stacks for similar reasons, but it seems like SSD-via-kernel is way ahead of network-via-kernel in the sense that it adds less latency per operation over the best userspace stack.
- vitus 3y agoI can't say offhand, since I don't have much experience with storage. I'd expect that most of the work from a SSD read is offloaded to the disk controller, which presumably uses DMA, and you don't have nearly as many round trips (a sequential read can be done with a single SCSI command). I'm inclined to agree with the explanation given by other commenters that the limiting factor for SSD r/w speeds in the cloud is due to throttling in the hypervisor to provide users with predictable performance as well as isolation in a multitenant environment.