3 ms·
Seconded, where is the Solar Flare card bypassing the OS for network ops? (Does that even work in windows? Doubtful)
by brianwawok 11y ago
Seconded, where is the Solar Flare card bypassing the OS for network ops? (Does that even work in windows? Doubtful)
- benaadams 11y agoUses the Registered Input/Output (RIO) API Extensions in Windows for user mode TCP/IP kernel bypass. PacketDirect is the next step for Server 2016 which moves closer to the NIC https://www.youtube.com/watch?v=KaXfDjIhn0U https://www.youtube.com/watch?v=KaXfDjIhn0U
- SEJeff 11y agoA followup post on even more detail of this would be incredible if you're able to / have the time. I'm a Linux jockey who simply had no idea modern windows was capable of using hardware this well. Nicely done.
- terry526 11y agoTo be fair, it seems that even MS are a bit surprised about this throughput: https://www.youtube.com/watch?v=CJeWIWkhVow&feature=youtu.be&t=29m5s https://www.youtube.com/watch?v=CJeWIWkhVow&feature=youtu.be...
- accountatwork 11y agoSolarflare isn't the only option for OS bypass. In fact, no major cloud vendor uses Solarflare NICs even for applications where OS bypass is critical because Solarflare is way too expensive for what you get (the retail price is substantially higher than Mellanox, Annapurna back when you could buy those publicly, etc., without discounts, and the volume discount you get with other vendors is more than an order of magnitude larger than the discount Solarflare will give you). The advantage of Solarflare is that it's easy to use, but at the scale large cloud vendors operate at (millions of machines), it's much cheaper to just have a networking staff that can properly operate another vendor's NICs.
- brianwawok 11y agoIt was kind of a rhetorical question... because if you actually care about latency, you do not use windows nor use the public cloud. I saw a few hundred Solarflare cards purchased, the prices were not that absurd. Only 10-20% of the total cost of the server, for a pretty amazing perf gain. Mellanox etc. tested similar but had less helpful sales engineers IIRC.
- accountatwork 11y agoSure, you won't get much of a discount from anyone for quantities of a few hundred. You might be looking at a 2x price difference, which is pocket change for a few hundred machines. In quantities of hundreds of thousands or millions, the price difference is well over an order of magnitude because Solarflare won't play ball the same way other vendors will. That adds up to a significant number, even if it's "only" 10%-20% per server, and you're probably overestimating the cost of the rest of the server since large companies are able to get steep discounts on almost everything in the quantities they buy in. > Mellanox etc. tested similar but had less helpful sales engineers We've found Mellanox engineers to be more helpful than Solarflare folks. I don't think that's because one company has inherently more helpful engineers. We're just not in a market that Solarflare cares about and you're not in a market that Mellanox cares about. > if you actually care about latency, you do not use windows nor use the public cloud You might be surprised by how many HPC shops have moved from fancy in-house infiniband networks and custom fabric to cloud hosted HPC clusters. HFT isn't moving to the public cloud anytime soon, but a number of large customers who care about sub-microsecond network latency have found real value in moving to the public cloud.
- brianwawok 11y ago> In quantities of hundreds of thousands or millions Sorry, I do not know anyone who buys millions of high performance NIC cards. Maybe 1 or 2 supercomputer labs in the world? And with that in mind, I am not sure the price break a company will provide to someone buying a million $500 network cards is super important to me? > That adds up to a significant number, even if it's "only" 10%-20% per server, and you're probably overestimating the cost of the rest of the server since large companies are able to get steep discounts on almost everything in the quantities they buy in. Nope, I definitely knew what the whole server cost.