5 ms·

There are some missing I/O things involving DMA. In the old days, DMA from (say) a PCI device would go directly to and from DRAM. This incurs a high latency i

There are some missing I/O things involving DMA. In the old days, DMA from (say) a PCI device would go directly to and from DRAM. This incurs a high latency if the CPU needs to access this data. Network processors found a simple solution: DMA goes to the cache, not the DRAM. This reduces the I/O latency to the processor and simplifies I/O coherency. I know Cavium's NPs rely on this. Intel picked this up for server and desktop processors once both the memory controller and PCIe were integrated on the same die. They called it DDIO: https://www.intel.com/content/www/us/en/io/data-direct-i-o-technology.html https://www.intel.com/content/www/us/en/io/data-direct-i-o-t... You can support 100 G Ethernet with Intel Xeon processors these days due to this. Another story is how DMA in the x86 world is cache coherent (no need to use uncached memory or flush before starting an I/O operation- which I have to do it in ARM). This is awesome from a device driver writer's point of view and is the result of having to support old operating systems from the pre-cache days. I think the future will involve better control of how cache is shared. For example, if you know a program is going to access a lot of memory, but does not need to keep it around for a long time, it will as a side effect, eject useful data from the cache. Better would be to declare that a thread should only be able to use some fraction of the cache so that it does not interfere with other threads so much.

signa11 8y ago
indeed. dpdk uses this (and other techniques) to achieve 10gbps line-rate (at least) packet forwarding on minimally sized packets per (x86) core.
rayiner 8y ago
Small nit: Intel desktop and workstation processors (Core i5/i7, Xeon E3, Xeon W) do not have DDIO.
dragontamer 8y ago
> Another story is how DMA in the x86 world is cache coherent (no need to use uncached memory or flush before starting an I/O operation- which I have to do it in ARM). This is awesome from a device driver writer's point of view and is the result of having to support old operating systems from the pre-cache days. Nitpick: You mean "Sequentially consistent". ARM is cache coherent, but NOT sequentially consistent. x86 is almost sequentially consistent (only a few obscure instructions here and there violate it).
- gpderetta 8y ago
  x86 is not sequentially consistent. Its consistency model is Total Store Order (same as most SPARCs). The store buffer is architecturally visible and newer loads can be reordered above older stores. More formally, all CPUs agree on the order of remote stores but might see their own stores in a different order. For example Dekker algorithm fails on x86 without explicit fences or explicitly sequentially consistent stores (all atomic RMW operations are sequentially consistent on x86). edit: I think the OP really meant cache coherency; while all ARM CPUs in a system are in the same coherency domain, the IO space might be outside of it.
  - dragontamer 8y ago
    > edit: I think the OP really meant cache coherency; while all ARM CPUs in a system are in the same coherency domain, the IO space might be outside of it. If that's the case, then I stand corrected. My understanding was that ARM was fully cache coherent, but it makes sense that I/O would be a different case all together.
    - gpderetta 8y ago
      to be clear: I do not know whether IO on ARM is cache coherent or not, I'm just pointing out that just because all CPUs are cache coherent it doesn't imply that peripherals on external buses must be as well.
- jhallenworld 8y ago
  I read coherent i/o begins with cortex-A9, interesting.
  - dragontamer 8y ago
    I wasn't thinking of the I/O case when I typed up my post earlier. I was thinking CPU-to-CPU coherency. In which case, you are probably right on this front.
gpderetta 8y ago
> I think the future will involve better control of how cache is shared. More fine grained cache control is definitely in the near future. x86 CPUs already allow partitioning L3 cache regions to different cores as desired. AMD CPUs provide the CLZERO instruction to quickly acquire a cacheline in exclusive mode and drop its content without waiting for any remotely modified data, which is great to implement message queues.

signa11 8y ago

indeed. dpdk uses this (and other techniques) to achieve 10gbps line-rate (at least) packet forwarding on minimally sized packets per (x86) core.

rayiner 8y ago

Small nit: Intel desktop and workstation processors (Core i5/i7, Xeon E3, Xeon W) do not have DDIO.

dragontamer 8y ago

> Another story is how DMA in the x86 world is cache coherent (no need to use uncached memory or flush before starting an I/O operation- which I have to do it in ARM). This is awesome from a device driver writer's point of view and is the result of having to support old operating systems from the pre-cache days. Nitpick: You mean "Sequentially consistent". ARM is cache coherent, but NOT sequentially consistent. x86 is almost sequentially consistent (only a few obscure instructions here and there violate it).

gpderetta 8y ago
x86 is not sequentially consistent. Its consistency model is Total Store Order (same as most SPARCs). The store buffer is architecturally visible and newer loads can be reordered above older stores. More formally, all CPUs agree on the order of remote stores but might see their own stores in a different order. For example Dekker algorithm fails on x86 without explicit fences or explicitly sequentially consistent stores (all atomic RMW operations are sequentially consistent on x86). edit: I think the OP really meant cache coherency; while all ARM CPUs in a system are in the same coherency domain, the IO space might be outside of it.
- dragontamer 8y ago
  > edit: I think the OP really meant cache coherency; while all ARM CPUs in a system are in the same coherency domain, the IO space might be outside of it. If that's the case, then I stand corrected. My understanding was that ARM was fully cache coherent, but it makes sense that I/O would be a different case all together.
  - gpderetta 8y ago
    to be clear: I do not know whether IO on ARM is cache coherent or not, I'm just pointing out that just because all CPUs are cache coherent it doesn't imply that peripherals on external buses must be as well.
jhallenworld 8y ago
I read coherent i/o begins with cortex-A9, interesting.
- dragontamer 8y ago
  I wasn't thinking of the I/O case when I typed up my post earlier. I was thinking CPU-to-CPU coherency. In which case, you are probably right on this front.

gpderetta 8y ago

> I think the future will involve better control of how cache is shared. More fine grained cache control is definitely in the near future. x86 CPUs already allow partitioning L3 cache regions to different cores as desired. AMD CPUs provide the CLZERO instruction to quickly acquire a cacheline in exclusive mode and drop its content without waiting for any remotely modified data, which is great to implement message queues.