2 ms·
Blog says 100Gbit NICs and 200MB/s achieved. Big gap! Since both endpoints are controlled by you, you should be able to tune the tcp buffers. In either case RT
by klabb3 2y ago
Blog says 100Gbit NICs and 200MB/s achieved. Big gap!
Since both endpoints are controlled by you, you should be able to tune the tcp buffers. In either case RTT and iperf3 with dozens concurrent tcp conns would be the first step to determine a baseline for what could be expected.
Author if you are here that data would be very interesting to know.
- levkk 2y agoThat remains a mystery for now. We measured upload speeds to S3 to be upwards of 700MB/second (still not 12,800MB/second you'd hope for, but S3 has to write to disk(s)). Download speeds from S3 were low - very low in fact, somewhere around 3-4MB/second when multiple files are downloaded concurrently, and a maximum of again ~30MB/second. I think you're right and high latency (~40ms) between endpoints triggered some bad TCP behavior. Next time we do something like this, I'll definitely look up how to tune those settings.
- deleted 2y ago[deleted]
- klabb3 2y ago> I think you're right and high latency (~40ms) between endpoints triggered some bad TCP behavior. 40 ms, 200MB/s. The BDP limit for those numbers is 7.6 MB (if I’m holding it right) which is very close to tcp_wmem/tcp_rmem max on Debian like distros, so that sounds about right. Linux can be quite stingy with buffers by default. Easy to increase though!
- attentive 2y agozfs|zstd|wireguard
- kccqzy 2y ago100Gbit NIC means nothing. My cheap desktop has a 2.5Gbit NIC; doesn't mean I can actually connect to the internet at 2.5Gbps.