4 ms·
> For quite a long time, I've thought it would be nifty if there was a NIC that kept received data in its own RAM and then allowed it to be efficiently DMAed to
by JoshTriplett 2y ago
> For quite a long time, I've thought it would be nifty if there was a NIC that kept received data in its own RAM and then allowed it to be efficiently DMAed to application memory when the application was ready for it.
I wonder if we could do a more advanced version of receive-packet steering that sufficiently identifies packets as definitely for a given process and DMAs them directly to that process's pre-provided buffers for later notification? In particular, can we offload enough information to a smart NIC that it can identify where something should be DMAed to?
- amluto 2y agoI don’t think the result would be compatible with the socket or io_uring API, but maybe io_uring could be extended a bit. Basically the kernel would opportunistically program a “flow director” or similar rule to send packets to special rx queue, and that queue would point to (pinned) application memory. Getting this to be compatible with iptables/nftables would be a mess or maybe entirely impossible. I’ve never seen the accelerated steering stuff work well in practice, sadly. The code is messy, the diagnostics are basically nonexistent, and it’s not clear to me that many drivers support it well.
- mgaunard 2y agoMost advanced NICs support flow steering, which makes the NIC write to different buffers depending on the target port. In practice though, you only have a limited amount of these buffers, and it causes complications if multiple processes need to consume the same multicast.
- eptcyka 2y agoMulticast may well be shitcanned to an expensive slow path, given that multicast is rarely used for high bandwidth scenarios, especially when multiple processes need to receive the same packet.
- lokar 2y agoThe main real use of multicast I’ve seen is pretty high packet rate. High frequency traders get multicast feeds of tick data from the exchange.
- mgaunard 2y agomulticast is precisely used for low-latency high-throughput message buses.
- eptcyka 2y agoWith multiple processes listening for the data? I think that's a market niche. In terms of billions of devices, multicast is mostly used for zero-config service discovery. I am not saying there isn't a market for high-bandwidth multicast, I am stating that for the vast majority of software deployments, multi-cast performance is not an issue. For whatever deployments it is an issue, they can specialize. And, as in the sibling comment mentions, people who need breakneck speeds have already proven that they can create a market for themselves.
- mgaunard 2y agoThat's not a market niche, that's the normal mode of operation of a message bus. The cloud doesn't implement multicast, but that doesn't mean it doesn't get used by people that build non-Internet networks and applications.