4 ms·

An overview of direct memory access (2014)

64 points by panic 7y ago

leggomylibro 7y ago
Cool! I hadn't realized that PCI and PCIe were successors to DMA, and it's nice to see an overview of how application processors shuttle data around quickly. If you want to play with DMA concepts on a less complex level, most modern microcontrollers implement it in some form or another. If you write a program that needs to move a lot of data quickly, like drawing to a display or using an SD card, the difference in speed and reduced overhead is dramatic. I guess it's a basic concept that most people here will find pedestrian and unremarkable, but I really appreciate how flexible and powerful it can be. You just have to configure a DMA channel pointing at two memory addresses, and then the data moves in the background while the CPU goes off to do other things. You can send data from memory to an external device, or vice-versa, or within the chip's internal memory. You can usually wire two peripherals together to implement 'dumb pipes' which require almost no intervention from the CPU to keep working. Most chips will also let you trigger DMA transfers from arbitrary signals such as hardware timers, so you can also 'set and forget' things like sending audio data to a DAC at a specific bitrate. I can see how it would be inadequate for the enormous bandwidth required to host multiple devices like GPUs and Ethernet ports, but DMA is still fun. A more elegant protocol for a more civilized age :)
- cperciva 7y ago
  DMA originates from a time when the CPU accessed memory via the North Bridge chipset; in effect, the CPU was just one of many peers in accessing memory, and data could be transferred between memory and devices without ever touching the CPU. These days, RAM hangs off the CPU -- and even "DMA" involves data passing through the CPU.
  - leggomylibro 7y ago
    >These days, RAM hangs off the CPU -- and even "DMA" involves data passing through the CPU. Not for the microcontrollers I was talking about, it doesn't. They have built-in bus arbiters to manage access to the internal RAM across the CPU and each DMA channel. Not every "CPU" is an application processor. But thank you for the history lesson! I guess the term "DMA" has become sort of generic these days.
continuations 7y ago
Is RDMA the same as kernel bypass?
- dooglius 7y ago
  Generally RDMA implies kernel bypass as the point is to write into a process' address space efficiently, but you could conceivably implement RDMA in a way that goes through the kernel, e.g. by using a conventional NIC and doing a copy into the corresponding address space manually. Note that there are many other things using kernel bypass that are not RDMA.
bogomipz 7y ago
The author states: >"Virtual memory is continuous as seen from a process’ point-of-view thanks to page tables and the memory management unit (MMU). However, it’s non-continuous as seen from the device point-of-view, because there is no MMU between the PCIe bus and the memory controller (well, some CPUs have an IO-MMU but let’s keep things simple). Hence, in a single DMA transfer only one page could be copied at a time. To overcome this limitation OS usually provide a scatter/gather API. Such an API chains together multiple page-sized memory transfers by creating a list of addresses of pages to be transferred." So a DMA controller accesses physical memory pages and not virtual memory pages? Is there a reason why it doesn't go through the MMU like everything else? Is this something historical? If a IO-MMU is present would there be an reason to use a scatter/gather API?
- dooglius 7y ago
  It would be expensive as the DMA's MMU would now have to maintain a large TLB to be performant, and there would be a startup overhead as new page information is loaded.
hootbootscoot 7y ago
This author needs to spend some time looking at ARM, AARCH64, RISC-V etc, as the entire article has a heavy Intel/AMD/PC bias.
- hootbootscoot 7y ago
  because, yes, "computers" still contain DMA controllers. we use them all the time.
  - hootbootscoot 7y ago
    https://www.st.com/content/ccc/resource/technical/document/application_note/47/41/32/e8/6f/42/43/bd/CD00160362.pdf/files/CD00160362.pdf/jcr:content/translations/en.CD00160362.pdf https://www.st.com/content/ccc/resource/technical/document/a... https://www.sifive.com/soc-ip/dma-controller https://www.sifive.com/soc-ip/dma-controller These are far from "legacy" applications, I can assure you. Very little to do with the ISA bus, I can assure you. The author neglects to mention how DMA works, nor discuss Stride size, etc. Additionally, the topic of how the MMU integrates with DMA and the issues raised are merely glossed over. https://patchwork.kernel.org/patch/10671331/ https://patchwork.kernel.org/patch/10671331/ and yes, "address translation"... as DMA needs physical addresses. to be sure, stuff like the R-Pi confuses people, as the CPU is a "device" to the broadcom GPU that drives the boot process/show, and hence "bus addressing" internal to the CPU...sigh.
    - vvanders 7y ago
      Rather than casting scorn why not step up to the plate and do what the author has done here but for the architectures that you know? I spend a fair bit of time in the Arm space and didn't find the article incorrect, just PC focused. I don't think the author claimed to be the authority on everything DMA.
      - hootbootscoot 7y ago
        Because I don't know enough to properly inform you of what is what and rewrite this article, but rather, know just enough to complain on Hacker News :D (aka, know just enough to note that this is not a very good "DMA in a nutshell" for those seeking to actually use it, and without contextualizing the limited space in which this even IS correct info...) I'm not claiming DMA expertise, hence not writing/posting articles about it. See my posts as a warning rather than scornful.

64 points by panic 7y ago

Cool! I hadn't realized that PCI and PCIe were successors to DMA, and it's nice to see an overview of how application processors shuttle data around quickly. If you want to play with DMA concepts on a less complex level, most modern microcontrollers implement it in some form or another. If you write a program that needs to move a lot of data quickly, like drawing to a display or using an SD card, the difference in speed and reduced overhead is dramatic. I guess it's a basic concept that most people here will find pedestrian and unremarkable, but I really appreciate how flexible and powerful it can be. You just have to configure a DMA channel pointing at two memory addresses, and then the data moves in the background while the CPU goes off to do other things. You can send data from memory to an external device, or vice-versa, or within the chip's internal memory. You can usually wire two peripherals together to implement 'dumb pipes' which require almost no intervention from the CPU to keep working. Most chips will also let you trigger DMA transfers from arbitrary signals such as hardware timers, so you can also 'set and forget' things like sending audio data to a DAC at a specific bitrate. I can see how it would be inadequate for the enormous bandwidth required to host multiple devices like GPUs and Ethernet ports, but DMA is still fun. A more elegant protocol for a more civilized age :)

cperciva 7y ago
DMA originates from a time when the CPU accessed memory via the North Bridge chipset; in effect, the CPU was just one of many peers in accessing memory, and data could be transferred between memory and devices without ever touching the CPU. These days, RAM hangs off the CPU -- and even "DMA" involves data passing through the CPU.
- leggomylibro 7y ago
  >These days, RAM hangs off the CPU -- and even "DMA" involves data passing through the CPU. Not for the microcontrollers I was talking about, it doesn't. They have built-in bus arbiters to manage access to the internal RAM across the CPU and each DMA channel. Not every "CPU" is an application processor. But thank you for the history lesson! I guess the term "DMA" has become sort of generic these days.

continuations 7y ago

Is RDMA the same as kernel bypass?

dooglius 7y ago
Generally RDMA implies kernel bypass as the point is to write into a process' address space efficiently, but you could conceivably implement RDMA in a way that goes through the kernel, e.g. by using a conventional NIC and doing a copy into the corresponding address space manually. Note that there are many other things using kernel bypass that are not RDMA.

bogomipz 7y ago

The author states: >"Virtual memory is continuous as seen from a process’ point-of-view thanks to page tables and the memory management unit (MMU). However, it’s non-continuous as seen from the device point-of-view, because there is no MMU between the PCIe bus and the memory controller (well, some CPUs have an IO-MMU but let’s keep things simple). Hence, in a single DMA transfer only one page could be copied at a time. To overcome this limitation OS usually provide a scatter/gather API. Such an API chains together multiple page-sized memory transfers by creating a list of addresses of pages to be transferred." So a DMA controller accesses physical memory pages and not virtual memory pages? Is there a reason why it doesn't go through the MMU like everything else? Is this something historical? If a IO-MMU is present would there be an reason to use a scatter/gather API?

dooglius 7y ago
It would be expensive as the DMA's MMU would now have to maintain a large TLB to be performant, and there would be a startup overhead as new page information is loaded.

hootbootscoot 7y ago

This author needs to spend some time looking at ARM, AARCH64, RISC-V etc, as the entire article has a heavy Intel/AMD/PC bias.

hootbootscoot 7y ago
because, yes, "computers" still contain DMA controllers. we use them all the time.
- hootbootscoot 7y ago
  https://www.st.com/content/ccc/resource/technical/document/application_note/47/41/32/e8/6f/42/43/bd/CD00160362.pdf/files/CD00160362.pdf/jcr:content/translations/en.CD00160362.pdf https://www.st.com/content/ccc/resource/technical/document/a... https://www.sifive.com/soc-ip/dma-controller https://www.sifive.com/soc-ip/dma-controller These are far from "legacy" applications, I can assure you. Very little to do with the ISA bus, I can assure you. The author neglects to mention how DMA works, nor discuss Stride size, etc. Additionally, the topic of how the MMU integrates with DMA and the issues raised are merely glossed over. https://patchwork.kernel.org/patch/10671331/ https://patchwork.kernel.org/patch/10671331/ and yes, "address translation"... as DMA needs physical addresses. to be sure, stuff like the R-Pi confuses people, as the CPU is a "device" to the broadcom GPU that drives the boot process/show, and hence "bus addressing" internal to the CPU...sigh.
  - vvanders 7y ago
    Rather than casting scorn why not step up to the plate and do what the author has done here but for the architectures that you know? I spend a fair bit of time in the Arm space and didn't find the article incorrect, just PC focused. I don't think the author claimed to be the authority on everything DMA.
    - hootbootscoot 7y ago
      Because I don't know enough to properly inform you of what is what and rewrite this article, but rather, know just enough to complain on Hacker News :D (aka, know just enough to note that this is not a very good "DMA in a nutshell" for those seeking to actually use it, and without contextualizing the limited space in which this even IS correct info...) I'm not claiming DMA expertise, hence not writing/posting articles about it. See my posts as a warning rather than scornful.