5 ms·

So one thing I still wonder about is why PCIe peripherals often present themselves as a region of memory to the main processor. If the chipset/CPU is buffering

So one thing I still wonder about is why PCIe peripherals often present themselves as a region of memory to the main processor. If the chipset/CPU is buffering writes for fast transmission, does it guess when you are done writing? Is any regular of the normal RAM caches used? If so what happens when write only a partial cache line? It seems so backward to me.

avianlyric 5y ago
This enables DMA (direct memory access) which is a much faster way of moving data around. It allows the CPU (or any other device in the PCI bus) to write/read data to/from the device when it’s convenient for the sender, without having to interrupt whatever the receiving device is currently doing. You still need to coordinate with the device to tell it that you’ve written to its memory, or read from it. But that’s a pretty cheap operation. The alternative is the both CPU and GPU would have to stop what their doing and manage the data copy while doing nothing else. So it’s basically the difference between sending someone an email vs giving them a call. With DMA your emailing a big document, the later calling to make sure they received it. Without DMA it would be like calling them and reading the document out over the phone. One is clearly better for everyone’s productivity.
- superjan 5y ago
  Thanks! I could still read this two ways however: one where the memory is on the peripheral, and one where the memory is main memory, where the peripheral is copying to/from using DMA. Which one is it?
  - __padding 5y ago
    Typically with devices like network cards (that also operate over PCI-E) You send the device a circular list of descriptors (pointers) to a region of main memory. In order to send data to the device, you write your network packet to the memory region associated with the pointer of the current ‘head’ of the descriptor list. So far, you have a ring of pointers, one of those pointers points to a location you just wrote to in ram. You then tell the device that the head of the list has changed (as you just wrote some data to the region that the head of the list is pointing to - so it can consume that pointer), the device then goes ahead and copies the data from ram into an internal buffer on the card. Once the data is consumed, the tail pointer of the ring buffer is updated to indicate that the card is finished with that memory region.
  - umanghere 5y ago
    __padding replied to you, but unfortunately their comment is dead because of their account being new, so I’ve reposted it as I cannot vouch yet. > __padding 45 minutes ago [dead] [–] > Typically with devices like network cards (that also operate over PCI-E) You send the device a circular list of descriptors (pointers) to a region of main memory. In order to send data to the device, you write your network packet to the memory region associated with the pointer of the current ‘head’ of the descriptor list. So far, you have a ring of pointers, one of those pointers points to a location you just wrote to in ram. You then tell the device that the head of the list has changed (as you just wrote some data to the region that the head of the list is pointing to - so it can consume that pointer), the device then goes ahead and copies the data from ram into an internal buffer on the card. Once the data is consumed, the tail pointer of the ring buffer is updated to indicate that the card is finished with that memory region.
  - avianlyric 5y ago
    DMA can work in both directions. You could put information in a specific location in main memory, then ask a peripheral to read it. But equally a peripheral can expose its own memory and ask the host to write into it. Cheap devices tend to do the former because it avoids the need to have expensive memory built in. They can just “borrow” system memory. More expensive, performance optimised, devices tend to do the latter. It’s also worth mentioning that DMA tends to work between every device attached to the PCIe bus. So Microsoft’s DirectStorage API seems to be using this feature, by having the GPU directly read data from an SSD, without the data ever touching the CPU or main memory.
tomerv 5y ago
Such regions of memory are usually marked as uncachable (UC) or write-through (WT) or sometimes write-combine (WC). These specify what type of caching is allowed, and if you somehow configure an incorrect caching type you're going to have a bad time.
jeffreygoesto 5y ago
There are a number of mechanisms at work for such use cases. You can flush caches manually, or have the memory uncached, also you need to make sure, some accesses don't reorder. If you are interested, you might search i.e. for ARM device memory. https://www.embedded.com/dealing-with-memory-access-ordering-in-complex-embedded-designs/ https://www.embedded.com/dealing-with-memory-access-ordering...
LeifCarrotson 5y ago
How do you thinking they should appear? It's interesting to compare to embedded processors without a memory management unit, like this STM32 reference manual, see p.68 and following: https://www.st.com/resource/en/reference_manual/dm00124865-stm32f75xxx-and-stm32f74xxx-advanced-armbased-32bit-mcus-stmicroelectronics.pdf https://www.st.com/resource/en/reference_manual/dm00124865-s... Everything looks like a memory address. Note that it's not actually memory, the processor just diverts requests for that address to the peripheral instead of memory. But on that little ARM processor, if you want to write to actual RAM, that's memory addresses 0x2001 0000 to 0x2001 BFFF. Data in the onboard Flash memory is at 0x0020 0000 - 0x002F FFFF. If you want to talk to something on a serial port, write to registers from 0x4001 1000 - 0x4001 13FF. If you want to show something on an attached LCD, or pull a buffer from the USB or Ethernet peripherals, or work with GPIO, or do anything at all, really, it's at some memory offset. This chip has some DMA, you can set it up to automatically push from one peripheral memory space to actual RAM or vice versa. But everything happens at a region of memory.
- superjan 5y ago
  I imagine that you have regions of normal RAM reserved for Peripheral IO, and that the peripheral directly copies it’s data to or from that region using DMA. For the CPU and the driver developer, this means there is no special handling required of these ranges of memory. If the PCIe bus is slow, the CPU is not slowed down because the transfer can be async. This is perhaps the DMA scenario you describe in the last two lines, my thinking is that it would make sense to do this all the time, at least when transfers are large.
  - toast0 5y ago
    You need more than just a shared area to put the data, which main memory, with DMA for devices to access works great for, you also need a way to coordinate on what data is ready to send or ready to receive. That could be in main memory and polled, but it usually makes sense to signal the device directly that something is ready to be sent so the device isn't constantly issuing reads to the memory to poll. For the device to signal the host, you'd generally raise an interrupt somehow, and then the host would ask the device what was ready. Usually you want that host to device communication to be direct, and memory mapping the i/o is a straightforward way to do that. (Using DMA to store/retreive the data also means the device doesn't need a whole lot of buffer space: enough to store a packet to be sent and maybe two packets to receive assuming you'll have time to DMA the first before the second is done, if the bus may be more congested, you'll need more a bigger receive buffer.
toast0 5y ago
I/O to the peripherals can generally be done with either memory mapped I/O or a separate I/O address space (port mapped I/O) accessed with different instructions than memory accesses. x86 can do either, but some architectures don't have a separate I/O space. PCI allows decices to require I/O access to be used, so on systems without it, it's got to be emulated by the controller somehow. For x86, the I/O space is only 16-bit, vs at least 32-bit for MMIO (but many devices allow a 64-bit MMIO address). For devices like GPUs with large amounts of RAM, you can feasibly map the whole memory with MMIO and access it directly, whereas with I/O ports, you'd probably need to write a destination address to one port and then read/write from another port. If you wanted concurrency, you'd need multiple pairs of access ports or locking, rather than letting the memory subsystems arbitrate access. As others have said, you'd normally configure the MMIO space for uncached access, or you'd need to be careful to force the memory ordering you need. The device specific interfacing requirements would be the guide there. Devices can indicate if their MMIO ranges are prefetchable or not, which should indicate if stray reads would cause side effects or not. One bonus of MMIO is DMA could interface with other devices, whereas I don't think devices are allowed to drive the I/O bus like that.

avianlyric 5y ago

This enables DMA (direct memory access) which is a much faster way of moving data around. It allows the CPU (or any other device in the PCI bus) to write/read data to/from the device when it’s convenient for the sender, without having to interrupt whatever the receiving device is currently doing. You still need to coordinate with the device to tell it that you’ve written to its memory, or read from it. But that’s a pretty cheap operation. The alternative is the both CPU and GPU would have to stop what their doing and manage the data copy while doing nothing else. So it’s basically the difference between sending someone an email vs giving them a call. With DMA your emailing a big document, the later calling to make sure they received it. Without DMA it would be like calling them and reading the document out over the phone. One is clearly better for everyone’s productivity.

superjan 5y ago
Thanks! I could still read this two ways however: one where the memory is on the peripheral, and one where the memory is main memory, where the peripheral is copying to/from using DMA. Which one is it?
- __padding 5y ago
  Typically with devices like network cards (that also operate over PCI-E) You send the device a circular list of descriptors (pointers) to a region of main memory. In order to send data to the device, you write your network packet to the memory region associated with the pointer of the current ‘head’ of the descriptor list. So far, you have a ring of pointers, one of those pointers points to a location you just wrote to in ram. You then tell the device that the head of the list has changed (as you just wrote some data to the region that the head of the list is pointing to - so it can consume that pointer), the device then goes ahead and copies the data from ram into an internal buffer on the card. Once the data is consumed, the tail pointer of the ring buffer is updated to indicate that the card is finished with that memory region.
- umanghere 5y ago
  __padding replied to you, but unfortunately their comment is dead because of their account being new, so I’ve reposted it as I cannot vouch yet. > __padding 45 minutes ago [dead] [–] > Typically with devices like network cards (that also operate over PCI-E) You send the device a circular list of descriptors (pointers) to a region of main memory. In order to send data to the device, you write your network packet to the memory region associated with the pointer of the current ‘head’ of the descriptor list. So far, you have a ring of pointers, one of those pointers points to a location you just wrote to in ram. You then tell the device that the head of the list has changed (as you just wrote some data to the region that the head of the list is pointing to - so it can consume that pointer), the device then goes ahead and copies the data from ram into an internal buffer on the card. Once the data is consumed, the tail pointer of the ring buffer is updated to indicate that the card is finished with that memory region.
- avianlyric 5y ago
  DMA can work in both directions. You could put information in a specific location in main memory, then ask a peripheral to read it. But equally a peripheral can expose its own memory and ask the host to write into it. Cheap devices tend to do the former because it avoids the need to have expensive memory built in. They can just “borrow” system memory. More expensive, performance optimised, devices tend to do the latter. It’s also worth mentioning that DMA tends to work between every device attached to the PCIe bus. So Microsoft’s DirectStorage API seems to be using this feature, by having the GPU directly read data from an SSD, without the data ever touching the CPU or main memory.

tomerv 5y ago

Such regions of memory are usually marked as uncachable (UC) or write-through (WT) or sometimes write-combine (WC). These specify what type of caching is allowed, and if you somehow configure an incorrect caching type you're going to have a bad time.

jeffreygoesto 5y ago

There are a number of mechanisms at work for such use cases. You can flush caches manually, or have the memory uncached, also you need to make sure, some accesses don't reorder. If you are interested, you might search i.e. for ARM device memory. https://www.embedded.com/dealing-with-memory-access-ordering-in-complex-embedded-designs/ https://www.embedded.com/dealing-with-memory-access-ordering...

LeifCarrotson 5y ago

How do you thinking they should appear? It's interesting to compare to embedded processors without a memory management unit, like this STM32 reference manual, see p.68 and following: https://www.st.com/resource/en/reference_manual/dm00124865-stm32f75xxx-and-stm32f74xxx-advanced-armbased-32bit-mcus-stmicroelectronics.pdf https://www.st.com/resource/en/reference_manual/dm00124865-s... Everything looks like a memory address. Note that it's not actually memory, the processor just diverts requests for that address to the peripheral instead of memory. But on that little ARM processor, if you want to write to actual RAM, that's memory addresses 0x2001 0000 to 0x2001 BFFF. Data in the onboard Flash memory is at 0x0020 0000 - 0x002F FFFF. If you want to talk to something on a serial port, write to registers from 0x4001 1000 - 0x4001 13FF. If you want to show something on an attached LCD, or pull a buffer from the USB or Ethernet peripherals, or work with GPIO, or do anything at all, really, it's at some memory offset. This chip has some DMA, you can set it up to automatically push from one peripheral memory space to actual RAM or vice versa. But everything happens at a region of memory.

superjan 5y ago
I imagine that you have regions of normal RAM reserved for Peripheral IO, and that the peripheral directly copies it’s data to or from that region using DMA. For the CPU and the driver developer, this means there is no special handling required of these ranges of memory. If the PCIe bus is slow, the CPU is not slowed down because the transfer can be async. This is perhaps the DMA scenario you describe in the last two lines, my thinking is that it would make sense to do this all the time, at least when transfers are large.
- toast0 5y ago
  You need more than just a shared area to put the data, which main memory, with DMA for devices to access works great for, you also need a way to coordinate on what data is ready to send or ready to receive. That could be in main memory and polled, but it usually makes sense to signal the device directly that something is ready to be sent so the device isn't constantly issuing reads to the memory to poll. For the device to signal the host, you'd generally raise an interrupt somehow, and then the host would ask the device what was ready. Usually you want that host to device communication to be direct, and memory mapping the i/o is a straightforward way to do that. (Using DMA to store/retreive the data also means the device doesn't need a whole lot of buffer space: enough to store a packet to be sent and maybe two packets to receive assuming you'll have time to DMA the first before the second is done, if the bus may be more congested, you'll need more a bigger receive buffer.

toast0 5y ago

I/O to the peripherals can generally be done with either memory mapped I/O or a separate I/O address space (port mapped I/O) accessed with different instructions than memory accesses. x86 can do either, but some architectures don't have a separate I/O space. PCI allows decices to require I/O access to be used, so on systems without it, it's got to be emulated by the controller somehow. For x86, the I/O space is only 16-bit, vs at least 32-bit for MMIO (but many devices allow a 64-bit MMIO address). For devices like GPUs with large amounts of RAM, you can feasibly map the whole memory with MMIO and access it directly, whereas with I/O ports, you'd probably need to write a destination address to one port and then read/write from another port. If you wanted concurrency, you'd need multiple pairs of access ports or locking, rather than letting the memory subsystems arbitrate access. As others have said, you'd normally configure the MMIO space for uncached access, or you'd need to be careful to force the memory ordering you need. The device specific interfacing requirements would be the guide there. Devices can indicate if their MMIO ranges are prefetchable or not, which should indicate if stray reads would cause side effects or not. One bonus of MMIO is DMA could interface with other devices, whereas I don't think devices are allowed to drive the I/O bus like that.