3 ms·
> networking folks like something called DMA (direct memory access). As I understand it, it allows one machine to access “directly” the memory of another machin
by abainbridge 8y ago
> networking folks like something called DMA (direct memory access). As I understand it, it allows one machine to access “directly” the memory of another machine, without impacting the remote CPU. Thus it should allow you to take several distinct nodes and make them “appear” like a multi-CPU machine. Building software on top of that requires some tooling and it is not clear to me how good it is right now.
In the old days CPUs couldn't saturate DRAM bandwidths but DMA could, so DMA was important. Today a single core of a CPU can read or write about 100 GByte/sec, about 4x more than the bandwidth of a single DRAM module (https://zsmith.co/images/XeonE5-2630v4.png https://zsmith.co/images/XeonE5-2630v4.png and https://en.wikipedia.org/wiki/DDR4_SDRAM#Modules https://en.wikipedia.org/wiki/DDR4_SDRAM#Modules). Also PCs tend to have more CPU cores than DRAM channels.
If you use something like DPDK to get fast network performance, then I'm not sure what more DMA brings.
If you are already using all your CPU power and not saturating your DRAM channels, then DMA is a win. But instead of buying the fancy NIC with the DMA support, you could have bought more CPU cores.
I can see that the latency and power consumption of DMA might be lower than if the CPU was involved because when a NIC supports DMA, the data path looks like this:
NIC <-> PCIe <-> Northbridge <-> DRAM
If the CPU is involved, it looks like this:
NIC <-> PCIe <-> Northbridge <-> CPU <-> Northbridge <-> DRAM
And you have to jump through hoops to prevent the DPDK NIC polling thread from being descheduled occasionally, particularly if you are running in a cloud VM.
Another interesting thing is that modern machines allow the NIC to write traffic into CPU cache directly, bypassing DRAM (https://www.intel.co.uk/content/www/uk/en/io/data-direct-i-o-technology.html https://www.intel.co.uk/content/www/uk/en/io/data-direct-i-o...). If the data you are trying to read (via DMA) is in CPU cache, then I'm guessing it'll be slower than if you asked the CPU for it.
And another thing: the memory controller in the Northbridge maintains a table of outstanding memory requests that CPU cores or PCIe peripherals have demanded. I don't know what the contention policy is. That could alter things drastically.
- gaius 8y agoBuilding software on top of that requires some tooling and it is not clear to me how good it is right now. DMA is a pretty mature technology these days, it was being used in trading systems a decade ago. Author of this piece is pretty far behind the state of the industry, let alone state-of-the-art. RDMA over Infiniband, yadda yadda.
- abainbridge 8y agoI think it would be nice if you answered the author's question rather than attack him. I, for one, would like to know more about this area if you've got knowledge to share. Can I write portable code that works on all the different vendors' RDMA hardware? Do the major cloud hosts support it without imposing other restrictions? Do I have to re-write my software when I run it on a machine that doesn't support RDMA? I don't know. I think if you know the answers, you are at the bleeding edge :-)
- gaius 8y agoCan I write portable code that works on all the different vendors' RDMA hardware? Noone using this stuff cares about that :-) You would be totally replacing the hardware every 2-3 years anyway (at most) and rewriting everything to take advantage of new hardware features, new CPU instructions, everything. It’s the same in games consoles, you write for the hardware and noone will sacrifice performance for portability that they know they’re never going to see the benefits of. Or in supercomputing. I think it would be nice if you answered the author's question rather than attack him Sorry for the snark but this does claim to be a piece on “modern hardware” and it just... isn’t.
- abainbridge 8y ago> Noone using this stuff cares about that I do. I expect the Mellanox ConnectX 3s in the Azure machines I develop on will be obsolete by the time my software is deployed. I don't even know what they will be replaced with, let alone have access to such exotica. I think my choices today are POSIX sockets or DPDK. I hope for a day when I can try out this new fangled RDMA thing :-) I do accept your point that these things were possible a long time ago but I think it takes time for them to mature enough for mainstream adoption (ie for use in software you might expect to keep working for decades on a variety of hardware). I think Lemire's question of, "Building software on top of that requires some tooling and it is not clear to me how good it is right now", remains open.
- 8y ago
- bogomipz 8y agoThere hasn't been a Northbridge for quite some time. This was replaced by the QPI/HyperTransport and an integrated memory controller.
- abainbridge 8y agoOK, perhaps I'm showing my age by clinging to these archaic terms. By Northbridge I just meant the functional block that contains the memory controller.
- bogomipz 8y agoI totally understand, it was around forever :) I thought you made good observations, I was trying to add to it. Cheers.
- nkurz 8y agoToday a single core of a CPU can read or write about 100 GByte/sec, about 4x more than the bandwidth of a single DRAM module At least for x64 systems, I'm almost sure this is wrong. I think your linked image on zsmith.co must be for multiple cores, and probably for multiple sockets. On all other tests I've seen and run, single threaded memory bandwidth is much less than full system bandwidth. Here's a fully labeled chart showing results more like I'd expect: https://www.anandtech.com/show/11544/intel-skylake-ep-vs-amd-epyc-7000-cpu-battle-of-the-decade/12 https://www.anandtech.com/show/11544/intel-skylake-ep-vs-amd... In this chart, a single core of an Intel "Broadwell-EP" Xeon E5-2699v4 (bigger sibling to the core you link) only gets 18GB/s, while 8 cores on the same socket max out at 60GB/s. Only for the whole machine (with two sockets and thus two memory controllers) are throughputs higher than 100GB/s seen. In practice, the limiting factor is the number of outstanding read requests supported per core. In accordance with Little's Law, you need about 100 requests in flight to overcome the latency of a read from RAM. On Intel, each core can only keep track of 10 outstanding requests. Prefetch can effectively double this, but you still need 4-6 cores to saturate a modern memory bus. See here for more details: http://sites.utexas.edu/jdm4372/2016/11/22/sc16-invited-talk-memory-bandwidth-and-system-balance-in-hpc-systems/ http://sites.utexas.edu/jdm4372/2016/11/22/sc16-invited-talk... If instead it's my understanding that's wrong here, I would love to be corrected!