4 ms·

The trouble with 64-bit DMA in Linux

47 points by chmaynard 4y ago

jasoneckert 4y ago
"One of the core rules of kernel development is that good things rarely result from breaking Linus Torvalds's machine..." sums up this post beautifully.
- don-code 4y ago
  I was personally surprised that maintainers would "be brave" at all with a codebase as widely used as the Linux kernel - just as much as to hear Linus suggest that this sort of thing happens time and time again.
  - bombcar 4y ago
    The implication of “we can probably find out where it breaks and the distros will catch it” is kinda fly-by-the-pants. Honestly surprised by it.
    - toast0 4y ago
      It is kind of fly-by-the-pants... But when you know a change could break at runtime on some hardware, and there's no testing organization with a full or even representative sample of supported hardware, and there's no real way to try it and then fallback at runtime, the options are push it out through the (emergent) testing process and see what happens or push it out disabled where almost nobody will use it. If only a few issues were noticed, those device dricers could likely be fixed with small code changes and most people would be happy. When Linus's machine breaks, there's a heuristic that many machines will break, so that's a recipie for a release that will cause lots of upset.
wrs 4y ago
This is a tangent, but I was reminded of early Macintosh days, where the 68000 processor in the original Mac hardware only paid attention to the lower 24 bits of addresses, and only the lower 22 were actually connected to memory. The heap manager used the upper bits of pointers as a convenient place to store flags, and didn’t bother zeroing them on dereference because they were ignored anyway. The release of the 68020 and Macs with more RAM resulted in the need for a whole transition to “32-bit clean” system and apps that took a while. I also seem to recall something about early versions of Microsoft Mac apps being limited to the lower megabyte of RAM for a similar reason.
- bombcar 4y ago
  As we get to 64 bits and more the temptation to do “useful” things with the unused bits will grow …
  - cesarb 4y ago
    There's even official hardware support for doing that. ARM calls it Top Byte Ignore, it's supported on all 64-bit ARM CPUs, and it's been enabled on Linux for a long time (see https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/arm64/tagged-pointers.rst?h=v5.19 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... and https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/arm64/tagged-address-abi.rst?h=v5.19 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...). Intel calls it Linear Address Masking and AMD calls it Upper Address Ignore, however these will be available only on very recent hardware, and the Linux patches for them are still under development (see https://lwn.net/Articles/902094/ https://lwn.net/Articles/902094/ and https://lwn.net/Articles/888914/ https://lwn.net/Articles/888914/).
  - cyber_kinetist 4y ago
    People are already doing all sorts of useful things with it! Only the 48 bits of the address space are actually being used by virtual memory (no current 64-bit OS is anticipating people to use more than 256TB of address space), so you can use the remaining 16 bits for... lots of things. (Note that there's also the lower 3 bits that aren't being used since memory addresses are aligned to a multiple of 8.) The primary example is Lua (before 5.4, and also LuaJIT): they use NaN-boxing to make their object take only 8 bytes. Although a double (Lua's only number type) already takes up 8 bytes of memory, there are some bits residing on the high 16-bits that are free to use when the number is NaN. With that you can store the type tag in there, which enables you to use the remaining bits for things other than double (like for example, storing memory addresses...) The Crafting Interpreters book has a much better explanation of this: https://craftinginterpreters.com/optimization.html#nan-boxing https://craftinginterpreters.com/optimization.html#nan-boxin... You can look up at https://en.wikipedia.org/wiki/Tagged_pointer https://en.wikipedia.org/wiki/Tagged_pointer to see some more examples. Apple seems to use this a lot for their Objective-C runtime on iOS where the leftover bits are used for things like refcounts or various boolean flags for their objects. Personally I had experience exploiting this in my game engine: for storing the generational index for a memory arena (to catch any use-after-free errors).
synergy20 4y ago
For super large systems, you do something like DPDK or XDP instead for large memory and super high throughput, the kernel's network stack is not designed for that anyways. HPC has been moving huge data cross PCIe running Linux for a long while, there must be multiple ways to overcome IOVA-4GB-allocation limit. Last, they can be dealt with at hardware level, let the (smart) hardware(e.g. an offload engine, or accelerator) to manage the memory on its own for DMAs.
chmaynard 4y ago
See https://lwn.net/SubscriberLink/904210/46bbced99963b245/ https://lwn.net/SubscriberLink/904210/46bbced99963b245/
- tuckerpo 4y ago
  Thanks.
- sophacles 4y ago
  @dang - Can you edit the top link to this subscriber link so folks with out an account at lwn don't have to search the comments please? Also for everyone else - just a reminder, LWN is a great resource and is funded through subscriptions. Personally, I've found the information on this site to be extremely helpful to understanding how the kernel works - to the point that multiple products I've built were only really doable because of LWN articles explaining the subsystem I'm working with (namespaces, io_uring, etc are all are the topic of fantastic articles there). A personal subscription is extremely affordable and very much worth it. They have corporate site license subscriptions too.
  - altairprime 4y ago
    The string “@dang” has no special effects at HN. If you need a response, you’ll need to use the Contact link in the footer to email the site mods with link change requests.
- ltbarcly3 4y ago
  I don't think it's really appropriate to help folks reach subscriber only content without a subscription, especially on such a niche, subscriber supported site.
  - yjftsjthsd-h 4y ago
    It's an officially supported feature; if lwn doesn't mind, why should we?
  - wtallis 4y ago

47 points by chmaynard 4y ago

"One of the core rules of kernel development is that good things rarely result from breaking Linus Torvalds's machine..." sums up this post beautifully.

don-code 4y ago
I was personally surprised that maintainers would "be brave" at all with a codebase as widely used as the Linux kernel - just as much as to hear Linus suggest that this sort of thing happens time and time again.
- bombcar 4y ago
  The implication of “we can probably find out where it breaks and the distros will catch it” is kinda fly-by-the-pants. Honestly surprised by it.
  - toast0 4y ago
    It is kind of fly-by-the-pants... But when you know a change could break at runtime on some hardware, and there's no testing organization with a full or even representative sample of supported hardware, and there's no real way to try it and then fallback at runtime, the options are push it out through the (emergent) testing process and see what happens or push it out disabled where almost nobody will use it. If only a few issues were noticed, those device dricers could likely be fixed with small code changes and most people would be happy. When Linus's machine breaks, there's a heuristic that many machines will break, so that's a recipie for a release that will cause lots of upset.

wrs 4y ago

This is a tangent, but I was reminded of early Macintosh days, where the 68000 processor in the original Mac hardware only paid attention to the lower 24 bits of addresses, and only the lower 22 were actually connected to memory. The heap manager used the upper bits of pointers as a convenient place to store flags, and didn’t bother zeroing them on dereference because they were ignored anyway. The release of the 68020 and Macs with more RAM resulted in the need for a whole transition to “32-bit clean” system and apps that took a while. I also seem to recall something about early versions of Microsoft Mac apps being limited to the lower megabyte of RAM for a similar reason.

bombcar 4y ago
As we get to 64 bits and more the temptation to do “useful” things with the unused bits will grow …
- cesarb 4y ago
  There's even official hardware support for doing that. ARM calls it Top Byte Ignore, it's supported on all 64-bit ARM CPUs, and it's been enabled on Linux for a long time (see https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/arm64/tagged-pointers.rst?h=v5.19 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... and https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/arm64/tagged-address-abi.rst?h=v5.19 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...). Intel calls it Linear Address Masking and AMD calls it Upper Address Ignore, however these will be available only on very recent hardware, and the Linux patches for them are still under development (see https://lwn.net/Articles/902094/ https://lwn.net/Articles/902094/ and https://lwn.net/Articles/888914/ https://lwn.net/Articles/888914/).
- cyber_kinetist 4y ago
  People are already doing all sorts of useful things with it! Only the 48 bits of the address space are actually being used by virtual memory (no current 64-bit OS is anticipating people to use more than 256TB of address space), so you can use the remaining 16 bits for... lots of things. (Note that there's also the lower 3 bits that aren't being used since memory addresses are aligned to a multiple of 8.) The primary example is Lua (before 5.4, and also LuaJIT): they use NaN-boxing to make their object take only 8 bytes. Although a double (Lua's only number type) already takes up 8 bytes of memory, there are some bits residing on the high 16-bits that are free to use when the number is NaN. With that you can store the type tag in there, which enables you to use the remaining bits for things other than double (like for example, storing memory addresses...) The Crafting Interpreters book has a much better explanation of this: https://craftinginterpreters.com/optimization.html#nan-boxing https://craftinginterpreters.com/optimization.html#nan-boxin... You can look up at https://en.wikipedia.org/wiki/Tagged_pointer https://en.wikipedia.org/wiki/Tagged_pointer to see some more examples. Apple seems to use this a lot for their Objective-C runtime on iOS where the leftover bits are used for things like refcounts or various boolean flags for their objects. Personally I had experience exploiting this in my game engine: for storing the generational index for a memory arena (to catch any use-after-free errors).

synergy20 4y ago

For super large systems, you do something like DPDK or XDP instead for large memory and super high throughput, the kernel's network stack is not designed for that anyways. HPC has been moving huge data cross PCIe running Linux for a long while, there must be multiple ways to overcome IOVA-4GB-allocation limit. Last, they can be dealt with at hardware level, let the (smart) hardware(e.g. an offload engine, or accelerator) to manage the memory on its own for DMAs.

chmaynard 4y ago

See https://lwn.net/SubscriberLink/904210/46bbced99963b245/ https://lwn.net/SubscriberLink/904210/46bbced99963b245/

tuckerpo 4y ago
Thanks.
sophacles 4y ago
@dang - Can you edit the top link to this subscriber link so folks with out an account at lwn don't have to search the comments please? Also for everyone else - just a reminder, LWN is a great resource and is funded through subscriptions. Personally, I've found the information on this site to be extremely helpful to understanding how the kernel works - to the point that multiple products I've built were only really doable because of LWN articles explaining the subsystem I'm working with (namespaces, io_uring, etc are all are the topic of fantastic articles there). A personal subscription is extremely affordable and very much worth it. They have corporate site license subscriptions too.
- altairprime 4y ago
  The string “@dang” has no special effects at HN. If you need a response, you’ll need to use the Contact link in the footer to email the site mods with link change requests.
ltbarcly3 4y ago
I don't think it's really appropriate to help folks reach subscriber only content without a subscription, especially on such a niche, subscriber supported site.
- yjftsjthsd-h 4y ago
  It's an officially supported feature; if lwn doesn't mind, why should we?
- wtallis 4y ago