6 ms·

I think the X32 feature is vastly underused. Ideally every application that is unlikely to use more than 2GB of memory should use the X32 ABI. Cache latency i

by sprash 8y ago

I think the X32 feature is vastly underused. Ideally every application that is unlikely to use more than 2GB of memory should use the X32 ABI. Cache latency is the most important bottleneck of today's general purpose CPUs. X32 Helps a lot with that and not just in "synthetic" benchmarks.

geocar 8y ago
100% agreed, but I'm not so convinced it needs (much) kernel support (as per my other comment). I don't usually use much libc, so an -m32 that still made elf-x86_64 would probably be enough for me. It is absolutely idiotic to have 64-bit pointers when I compile a program that uses less than 4 gigabytes of RAM. When such pointer values appear inside a struct, they not only waste half the memory, they effectively throw away half of the cache. -Knuth
- 0x0 8y ago
  I dunno, having a large address space could be useful if you work with large or lots of memory mapped files?
  - AstralStorm 8y ago
    Or you could use large file support and not mmap everything. It is probably faster too and puts less pressure on overcommit mechanism.
    - Hello71 8y ago
      file-backed mmap is by definition not related to overcommit, because the memory is committed by the file.
  - geocar 8y ago
    Sure. Parts of my (multi-process) application can use 64-bit pointers and parts can use 32-bit.
pedrocr 8y ago
> Ideally every application that is unlikely to use more than 2GB of memory should use the X32 ABI. It's often useful to be able to address in 64bits even if you are never going to use 2GB of memory. And it's probably more and more true the more you care about performance (e.g., mmap() and friends). So I don't think the set of useful workloads that get a performance boost from smaller pointers and don't get hindered by a small address space is as large as you are assuming.
- geocar 8y ago
  > So I don't think the set of useful workloads that get a performance boost... https://link.springer.com/chapter/10.1007%2F11688839_14?LI=true https://link.springer.com/chapter/10.1007%2F11688839_14?LI=t... found speedups of 13.4% for integer workloads. That's significant for my space, hence my interest. Using a single 1G page per process means switching is faster, and might have other benefits in a post-spectre world...
  - pedrocr 8y ago
    A 13% speedup in the critical sections seems much too low to pay for the loss in caching and other optimizations from the 64bit address space. But I'm considering general purpose applications where the critical sections are not run 100%. If all you do is simulation work in clusters for example, I could see how this would be significant. I was replying to saying that most apps should use x32 if they don't use much memory. That sounds like a nightmare in a general purpose distribution, with having every library duplicated and interoperability issues.
    - geocar 8y ago
      What do you mean by "loss in caching and other optimizations"? I get the part about duplicated libraries, but I'm not certain it's that big of a deal: Most people don't have more than 2G of code so it should be possible for the 64-bit libraries to be used (and just mapped low) with an m32-built application, if the things that create new pointers can be limited to 32-bits.
chrisseaton 8y ago
Has anyone ever run a native application with compressed pointers, rather than just 32 bit pointers?
- AstralStorm 8y ago
  Do you mean like compressed code? The hit is way bigger than gains.
  - jmalicki 8y ago
    The value in the pointer is an index into an array of 64bit ints rather than bytes. This lets you address 32GB with a 32bit pointer rather than 4GB.
  - AnIdiotOnTheNet 8y ago
    One wonders if the time might come when it won't be. Cache is so important to performance that one can imagine some hardware level compression might actually be faster than uncompressed code and data simply because more stuff can be in the cache.
deleted 8y ago
[deleted]
jabl 8y ago
Well, most code isn't performance critical. And most which are, aren't bottlenecked by CPU performance but by network/disk. And of those performance critical programs which are CPU/memory bound, certainly a sizeable bunch might require more than 4 GB memory, depending on the input data. Or are you going to keep a X32 version for "normal" use and a separate x86-64 version for large inputs? Or might want to mmap() large files, etc. So all the hassle of maintaining a separate ABI (toolchains, kernel, libraries etc.), for a modest performance increase for some subset of applications? Starts to smell pretty niche to me.
- sprash 8y ago
  > Well, most code isn't performance critical. In principle yes, but it is hard to predict which code will be at some point performance critical and which code will not. In the worst case you chain a lot of alleged "non performance critical" tasks together and in the sum (actually product, performance percentage is multiplicative) your problem becomes a performance problem. > Or might want to mmap() large files This is a fringe application that is not covered by my definition of "application that is unlikely to use more than 2GB of memory". Most programs read linear from files. > So all the hassle of maintaining a separate ABI (toolchains, kernel, libraries etc.), for a modest performance increase for some subset of applications? Starts to smell pretty niche to me. As far as I see it there is not enough awareness of this feature. So instead of declaring it "niche" that causes "hassles" maybe advertise it more since the hard work (toolchains, kernel, libraries etc.) has already been done. (My personal use case is derivatives market simulations and performance the gain is in the double digits percentage. I'm pretty sure other use cases exists, they just need to found.) In the end, who defines "hassle", how big is it really, and who defines "niche"? Potentially accelerating 90% + x of code running on your CPU is hardly a niche.
  - mg794613 8y ago
    I hope that "potentially accelerating 90%" was an intentional joke? There is a reason why they consider removing support. We've tried to use it and if there was any improvement anywhere at all, it was minimal. There is this misunderstanding that this magically makes code more efficient. It does not work that way. It absolutely negligible compared to a simple optimization in code itself.
gwbas1c 8y ago
(Joke) Because 64-bit is the future!!! Seriously, most people I know just don't think that critically about 64-bit, and don't understand the performance gain of pairing 64-bit instructions with 32-bit pointers.
ajross 8y ago
Yes, but the issue at hand isn't whether or not apps can or should be compiled to x32. That capability isn't going away (it happens entirely in userspace and is basically just a recompiled set of libraries). This is a discussion about whether the kernel itself wants to run using x32 mode, which is arguably much less useful. Such a system would be limited to 4G of physical memory, etc... There aren't that many such systems left in the x86 world (that run Linux, anyway -- I'm literally working on an x86_64 port for Zephyr now, and it uses x32).
- pantalaimon 8y ago
  No this is about running userspace x32 programs that will make x32 syscalls with 32bit pointers. Such kernel will run x86 binaries (in compat mode), x86_64 binaries and x86_32 ones.
- jabl 8y ago
  > Yes, but the issue at hand isn't whether or not apps can or should be compiled to x32. That capability isn't going away (it happens entirely in userspace and is basically just a recompiled set of libraries). If the kernel removes X32 support, I guess it won't take long for toolchains to remove it either. Why should they pay the maintenance burden of it if nobody uses it? > This is a discussion about whether the kernel itself wants to run using x32 mode, No, the kernel has never supported running in X32 mode itself. This is about the compatibility syscalls allowing a X32 userspace with a x86-64 kernel.
andreyv 8y ago
One counterargument is that for each ABI you need a different set of libraries loaded in memory. So, if some programs in the system use the 64-bit ABI and some use x32, then memory usage may substantially increase. And, of course, this also affects CPU caches.

by sprash 8y ago

geocar 8y ago

100% agreed, but I'm not so convinced it needs (much) kernel support (as per my other comment). I don't usually use much libc, so an -m32 that still made elf-x86_64 would probably be enough for me. It is absolutely idiotic to have 64-bit pointers when I compile a program that uses less than 4 gigabytes of RAM. When such pointer values appear inside a struct, they not only waste half the memory, they effectively throw away half of the cache. -Knuth

0x0 8y ago
I dunno, having a large address space could be useful if you work with large or lots of memory mapped files?
- AstralStorm 8y ago
  Or you could use large file support and not mmap everything. It is probably faster too and puts less pressure on overcommit mechanism.
  - Hello71 8y ago
    file-backed mmap is by definition not related to overcommit, because the memory is committed by the file.
- geocar 8y ago
  Sure. Parts of my (multi-process) application can use 64-bit pointers and parts can use 32-bit.

pedrocr 8y ago

> Ideally every application that is unlikely to use more than 2GB of memory should use the X32 ABI. It's often useful to be able to address in 64bits even if you are never going to use 2GB of memory. And it's probably more and more true the more you care about performance (e.g., mmap() and friends). So I don't think the set of useful workloads that get a performance boost from smaller pointers and don't get hindered by a small address space is as large as you are assuming.

geocar 8y ago
> So I don't think the set of useful workloads that get a performance boost... https://link.springer.com/chapter/10.1007%2F11688839_14?LI=true https://link.springer.com/chapter/10.1007%2F11688839_14?LI=t... found speedups of 13.4% for integer workloads. That's significant for my space, hence my interest. Using a single 1G page per process means switching is faster, and might have other benefits in a post-spectre world...
- pedrocr 8y ago
  A 13% speedup in the critical sections seems much too low to pay for the loss in caching and other optimizations from the 64bit address space. But I'm considering general purpose applications where the critical sections are not run 100%. If all you do is simulation work in clusters for example, I could see how this would be significant. I was replying to saying that most apps should use x32 if they don't use much memory. That sounds like a nightmare in a general purpose distribution, with having every library duplicated and interoperability issues.
  - geocar 8y ago
    What do you mean by "loss in caching and other optimizations"? I get the part about duplicated libraries, but I'm not certain it's that big of a deal: Most people don't have more than 2G of code so it should be possible for the 64-bit libraries to be used (and just mapped low) with an m32-built application, if the things that create new pointers can be limited to 32-bits.

chrisseaton 8y ago

Has anyone ever run a native application with compressed pointers, rather than just 32 bit pointers?

AstralStorm 8y ago
Do you mean like compressed code? The hit is way bigger than gains.
- jmalicki 8y ago
  The value in the pointer is an index into an array of 64bit ints rather than bytes. This lets you address 32GB with a 32bit pointer rather than 4GB.
- AnIdiotOnTheNet 8y ago
  One wonders if the time might come when it won't be. Cache is so important to performance that one can imagine some hardware level compression might actually be faster than uncompressed code and data simply because more stuff can be in the cache.

deleted 8y ago

[deleted]

jabl 8y ago

Well, most code isn't performance critical. And most which are, aren't bottlenecked by CPU performance but by network/disk. And of those performance critical programs which are CPU/memory bound, certainly a sizeable bunch might require more than 4 GB memory, depending on the input data. Or are you going to keep a X32 version for "normal" use and a separate x86-64 version for large inputs? Or might want to mmap() large files, etc. So all the hassle of maintaining a separate ABI (toolchains, kernel, libraries etc.), for a modest performance increase for some subset of applications? Starts to smell pretty niche to me.

sprash 8y ago
> Well, most code isn't performance critical. In principle yes, but it is hard to predict which code will be at some point performance critical and which code will not. In the worst case you chain a lot of alleged "non performance critical" tasks together and in the sum (actually product, performance percentage is multiplicative) your problem becomes a performance problem. > Or might want to mmap() large files This is a fringe application that is not covered by my definition of "application that is unlikely to use more than 2GB of memory". Most programs read linear from files. > So all the hassle of maintaining a separate ABI (toolchains, kernel, libraries etc.), for a modest performance increase for some subset of applications? Starts to smell pretty niche to me. As far as I see it there is not enough awareness of this feature. So instead of declaring it "niche" that causes "hassles" maybe advertise it more since the hard work (toolchains, kernel, libraries etc.) has already been done. (My personal use case is derivatives market simulations and performance the gain is in the double digits percentage. I'm pretty sure other use cases exists, they just need to found.) In the end, who defines "hassle", how big is it really, and who defines "niche"? Potentially accelerating 90% + x of code running on your CPU is hardly a niche.
- mg794613 8y ago
  I hope that "potentially accelerating 90%" was an intentional joke? There is a reason why they consider removing support. We've tried to use it and if there was any improvement anywhere at all, it was minimal. There is this misunderstanding that this magically makes code more efficient. It does not work that way. It absolutely negligible compared to a simple optimization in code itself.

gwbas1c 8y ago

(Joke) Because 64-bit is the future!!! Seriously, most people I know just don't think that critically about 64-bit, and don't understand the performance gain of pairing 64-bit instructions with 32-bit pointers.

ajross 8y ago

Yes, but the issue at hand isn't whether or not apps can or should be compiled to x32. That capability isn't going away (it happens entirely in userspace and is basically just a recompiled set of libraries). This is a discussion about whether the kernel itself wants to run using x32 mode, which is arguably much less useful. Such a system would be limited to 4G of physical memory, etc... There aren't that many such systems left in the x86 world (that run Linux, anyway -- I'm literally working on an x86_64 port for Zephyr now, and it uses x32).

pantalaimon 8y ago
No this is about running userspace x32 programs that will make x32 syscalls with 32bit pointers. Such kernel will run x86 binaries (in compat mode), x86_64 binaries and x86_32 ones.
jabl 8y ago
> Yes, but the issue at hand isn't whether or not apps can or should be compiled to x32. That capability isn't going away (it happens entirely in userspace and is basically just a recompiled set of libraries). If the kernel removes X32 support, I guess it won't take long for toolchains to remove it either. Why should they pay the maintenance burden of it if nobody uses it? > This is a discussion about whether the kernel itself wants to run using x32 mode, No, the kernel has never supported running in X32 mode itself. This is about the compatibility syscalls allowing a X32 userspace with a x86-64 kernel.

andreyv 8y ago

One counterargument is that for each ABI you need a different set of libraries loaded in memory. So, if some programs in the system use the 64-bit ABI and some use x32, then memory usage may substantially increase. And, of course, this also affects CPU caches.