5 ms·
As there was mention of SolarFlare cards, i'd just like to say that I had much success in a previous life with their products for reducing latency in trading sy
by cesaref 6y ago
As there was mention of SolarFlare cards, i'd just like to say that I had much success in a previous life with their products for reducing latency in trading systems - the products are really robust, and OpenOnload works really well.
At the most basic level you can run an existing binary under Onload (which basically uses an LD_LIBRARY_PATH to direct the socket system calls to their own stack) and get a significant performance improvement, especially if you are careful about how the card is configured, and to ensure your app is running on cores on the same socket that the network card is connected to.
Taking it further, you can use a zero copy API to get inbound packets whilst they are still in the network card buffer, and do very cool things with both hardware timestamping and pre-caching of packets on the network card memory (sort of like a message template) which can reduce the latency to sending responses if you have a set of stock responses you want to supply.
Anyhow, do check them out if you are into that sort of thing.
- hawk_ 6y agoi am into that kind of thing :-) if i wanted to try this kind of stuff at home, i.e. buying any necessary hardware myself, how much do you reckon the whole setup would cost? any suggestions on used NICs/switches to try this with?
- ncmncm 6y agoUsed Solarflare cards can be had for ~$300. Warning, the fancier features are disabled on cheap cards; you pay Solarflare to upgrade the firmware to get access. The kernel-bypass software and kernel driver (for setup) are usually free to download. A new ExaNIC might be $2000? last I checked. Netronome, maybe $1200? You can get experience with ordinary Intel hardware and AF_XDP or DPDK.
- scott00 6y agoI'll add that you should look for the PLUS skus, ie SFN-8522-PLUS. They come with all the fun stuff. Might also want to look at Packet. They offer bare metal cloud servers by the hour, some of which have Mellanox NICs in them[0]. I'm less familiar with Mellanox than SolarFlare, but I think they have a similar feature set. [0] https://www.packet.com/cloud/servers/c3-small/ https://www.packet.com/cloud/servers/c3-small/
- ncmncm 6y agoThank you, I was not aware of Packet. My employer's customers don't use Mellanox anymore, so I haven't. I don't know why they don't, but would not be surprised if price had something to do with it.
- kd913 6y agoJust FYI, I think it may be a good idea just waiting a little bit as there are currently plans being put in place for a slightly different direction for Onload. I don't think I am allowed to reveal much but it will soon be a lot easier and cheaper/test/try out.
- secondcoming 6y agoI assume it's LD_PRELOAD instead of LD_LIBRARY_PATH ?
- cesaref 6y agoDoh, yes of course, thanks for pointing that out.