3 ms·
We use Mellanox ConnectX-4 NICs (cx5 and cx6 would be fine too), as well as Chelsio T6 in mostly Supermicro boards, making sure to have a full Gen3 x16 (or Gen4
by drewg123 7y ago
We use Mellanox ConnectX-4 NICs (cx5 and cx6 would be fine too), as well as Chelsio T6 in mostly Supermicro boards, making sure to have a full Gen3 x16 (or Gen4 x8).
The Mellanox (and Chelsio) NICs are helpful for us because they support RSS assisted TCP LRO. If you're doing just plain packet forwarding, that would not matter.
- pdimitar 7y agoThanks a lot! I am not sure about the plain packet forwarding or if I am gonna do more, but a few hundred (or thousand) bucks more sound pretty good for future-proofing a neighbourhood level of networking. Pretty valuable info for a hobbyist looking to get better (and trying to get into community work). Thank you.
- gonzo 7y agoWell, the RSS does, but LRO does not.
- drewg123 7y agoRSS assisted LRO is a FreeBSD feature where LRO holds batches of hundreds or thousands of packets, then sorts packets by arrival time and LRO hash result. That puts packets from the same connection adjacent to each other, and lets LRO combine them. Delivering packets in batches of hundreds requires some work from the driver (and requires the driver use a new LRO API). At least on our (Netflix) workload with tens of thousands of active connections, RSS assisted LRO increases our aggregation rate from almost nothing to about 2:1, and saves about 10% CPU. Its been on my TODO list to add RSS-assisted LRO support to iflib, which would give the intel and broadcom 100g drivers access to it, along with assorted Intel and other 1g, 10g, and 40g devices. But it still hasn't happened yet.
- cthalupa 7y agoI think he was saying that RSS would still matter for plain packet forwarding.