4 ms·

A 13% speedup in the critical sections seems much too low to pay for the loss in caching and other optimizations from the 64bit address space. But I'm consideri

A 13% speedup in the critical sections seems much too low to pay for the loss in caching and other optimizations from the 64bit address space. But I'm considering general purpose applications where the critical sections are not run 100%. If all you do is simulation work in clusters for example, I could see how this would be significant. I was replying to saying that most apps should use x32 if they don't use much memory. That sounds like a nightmare in a general purpose distribution, with having every library duplicated and interoperability issues.

geocar 8y ago
What do you mean by "loss in caching and other optimizations"? I get the part about duplicated libraries, but I'm not certain it's that big of a deal: Most people don't have more than 2G of code so it should be possible for the 64-bit libraries to be used (and just mapped low) with an m32-built application, if the things that create new pointers can be limited to 32-bits.
- noselasd 8y ago
  You'll need 2 or 3 sets of all libraries on a system. One for x86_64, one for x32. And if you want to support traditional 32 bit application you'll need the i686 variant as well. None of these variants are ABI compatible with eachother. If nothing else, that's a burden on logistics.
- pedrocr 8y ago
  > What do you mean by "loss in caching and other optimizations"? The most common use case is to just mmap() a bunch of cache files and just fully outsource the cache hierarchy of those to the Linux VM subsystem which will do a much better job than almost any app even can (because it does global optimization across all processes). > I get the part about duplicated libraries, but I'm not certain it's that big of a deal: Most people don't have more than 2G of code so it should be possible for the 64-bit libraries to be used (and just mapped low) with an m32-built application, if the things that create new pointers can be limited to 32-bits. This is way too much complexity. "Things that create new pointers" are any functions that return new memory and there are too many of those. A general purpose distro will never take on that much complexity for a 13% speedup. If they wanted even a fraction of that complexity they'd compile x86_64 at several levels of supported instructions for bigger gains from just tweaking a few compiler flags.
  - geocar 8y ago
    > The most common use case is to just mmap() a bunch of cache files and just fully outsource the cache hierarchy of those to the Linux VM subsystem which will do a much better job than you ever will. I don't understand what you're saying here. Why do you think having the pointers be 64-bit instead of 32-bit help if the program isn't using more than 2G? Can you give me a specific example? > This is way too much complexity. "Things that create new pointers" are any functions that return new memory and there are too many of those. mmap() and brk() are the two big ones, and almost everything else derives from them. The tricky part is to get MAP_32BIT into every call to mmap(). That means hacking up the dynamic linker, or having a personality flag that just pins it kernel-side. I don't think either is onerous though (probably a few hours work).
    - pedrocr 8y ago
      > Why do you think having the pointers be 64-bit instead of 32-bit help if the program isn't using more than 2G? The program isn't using more than 2G of RAM it just mmap()s more than 2G of files and so needs enough address space to fit those mappings. I've come across that when doing caching for an image processing program. The simple way, that works very well, is to just have the thumbnails on disk and mmap() them in. With even medium sized collections it's easy to lazy load more than 2GB of files but that's fine because 64bit address space is cheap and the Linux VM will take care of throwing away things that haven't been used. Makes for a very simple memory+disk cache hierarchy that the OS manages for you entirely. > mmap() and brk() are the two big ones, and almost everything else derives from them. Sure but that's at the base of the call chain. Each library will then bubble up those 64bit pointers to it's callers. You can't have only 32bit versions because at least some part of the apps will need more memory so you're stuck having two versions of everything.
      - geocar 8y ago
        > The program isn't using more than 2G of RAM it just mmap()s more than 2G of files and so needs enough address space to fit those mappings. If you need the address space, then use 64-bit. I don't need the address space. If that was confusing, my apologies. > Sure but that's at the base of the call chain. Each library will then bubble up those 64bit pointers to it's callers. Which will have clear upper 32-bits, which means using %eax instead of %rax in main() will work fine.
- Iwan-Zotow 8y ago
  "loss in caching" means you have pieces of x64 glibc in your $I as well as pieces of x32 glibc in your $I, and I$ space is rather limited.
AstralStorm 8y ago
If your application really wants to random access over 2 GB of files you might have a point. Most applications do not, except database servers. Cache works with all your memory and is handled with 64 bit offsets in kernel, the limitation is in mmap, but you can mmap parts of a big file no problem with large file support in libc.
- pedrocr 8y ago
  The use case I'm thinking of is not random access over large files. It's using lots of very small files mmap()'ed so that you get the Linux VM helping you run an efficient cache. I've run across this with thumbnail caches in image processing software. It's easy for these kinds of things to be present in generally used libraries that your standard app will be using just because it has a GTK file selector for example. > you can mmap parts of a big file no problem with large file support in libc. For the speedups discussed here I definitely don't want to spend any time writing special code just to be able to read a 3GB file. That ship has long sailed and good riddance to all those subtle issues where things seem to work fine until they don't and it's not clear why.
- CJefferson 8y ago
  Actually, you can get a lot less than that once libraries are loaded and such like. I've found on some 32 bit linuxes you can't mmap a single file bigger than around 500mb, which means I couldn't check out a git repository containing 650mb cd images on a 32 bit Linux.
  - geocar 8y ago
    You might check resource limits and/or disable address randomisation (or other "security" features) since they can really eat up your (limited) address space. Don't forget to not put the bugs in, though!
pantalaimon 8y ago
You already have every lib duplicated if you want to run 32bit x86 applications like steam. Distributions handle it just fine. And as the kernel is responsible for caching (or what kind of caching do you mean?) You will not lose that.

geocar 8y ago

What do you mean by "loss in caching and other optimizations"? I get the part about duplicated libraries, but I'm not certain it's that big of a deal: Most people don't have more than 2G of code so it should be possible for the 64-bit libraries to be used (and just mapped low) with an m32-built application, if the things that create new pointers can be limited to 32-bits.

noselasd 8y ago
You'll need 2 or 3 sets of all libraries on a system. One for x86_64, one for x32. And if you want to support traditional 32 bit application you'll need the i686 variant as well. None of these variants are ABI compatible with eachother. If nothing else, that's a burden on logistics.
pedrocr 8y ago
> What do you mean by "loss in caching and other optimizations"? The most common use case is to just mmap() a bunch of cache files and just fully outsource the cache hierarchy of those to the Linux VM subsystem which will do a much better job than almost any app even can (because it does global optimization across all processes). > I get the part about duplicated libraries, but I'm not certain it's that big of a deal: Most people don't have more than 2G of code so it should be possible for the 64-bit libraries to be used (and just mapped low) with an m32-built application, if the things that create new pointers can be limited to 32-bits. This is way too much complexity. "Things that create new pointers" are any functions that return new memory and there are too many of those. A general purpose distro will never take on that much complexity for a 13% speedup. If they wanted even a fraction of that complexity they'd compile x86_64 at several levels of supported instructions for bigger gains from just tweaking a few compiler flags.
- geocar 8y ago
  > The most common use case is to just mmap() a bunch of cache files and just fully outsource the cache hierarchy of those to the Linux VM subsystem which will do a much better job than you ever will. I don't understand what you're saying here. Why do you think having the pointers be 64-bit instead of 32-bit help if the program isn't using more than 2G? Can you give me a specific example? > This is way too much complexity. "Things that create new pointers" are any functions that return new memory and there are too many of those. mmap() and brk() are the two big ones, and almost everything else derives from them. The tricky part is to get MAP_32BIT into every call to mmap(). That means hacking up the dynamic linker, or having a personality flag that just pins it kernel-side. I don't think either is onerous though (probably a few hours work).
  - pedrocr 8y ago
    > Why do you think having the pointers be 64-bit instead of 32-bit help if the program isn't using more than 2G? The program isn't using more than 2G of RAM it just mmap()s more than 2G of files and so needs enough address space to fit those mappings. I've come across that when doing caching for an image processing program. The simple way, that works very well, is to just have the thumbnails on disk and mmap() them in. With even medium sized collections it's easy to lazy load more than 2GB of files but that's fine because 64bit address space is cheap and the Linux VM will take care of throwing away things that haven't been used. Makes for a very simple memory+disk cache hierarchy that the OS manages for you entirely. > mmap() and brk() are the two big ones, and almost everything else derives from them. Sure but that's at the base of the call chain. Each library will then bubble up those 64bit pointers to it's callers. You can't have only 32bit versions because at least some part of the apps will need more memory so you're stuck having two versions of everything.
    - geocar 8y ago
      > The program isn't using more than 2G of RAM it just mmap()s more than 2G of files and so needs enough address space to fit those mappings. If you need the address space, then use 64-bit. I don't need the address space. If that was confusing, my apologies. > Sure but that's at the base of the call chain. Each library will then bubble up those 64bit pointers to it's callers. Which will have clear upper 32-bits, which means using %eax instead of %rax in main() will work fine.
Iwan-Zotow 8y ago
"loss in caching" means you have pieces of x64 glibc in your $I as well as pieces of x32 glibc in your $I, and I$ space is rather limited.

AstralStorm 8y ago

If your application really wants to random access over 2 GB of files you might have a point. Most applications do not, except database servers. Cache works with all your memory and is handled with 64 bit offsets in kernel, the limitation is in mmap, but you can mmap parts of a big file no problem with large file support in libc.

pedrocr 8y ago
The use case I'm thinking of is not random access over large files. It's using lots of very small files mmap()'ed so that you get the Linux VM helping you run an efficient cache. I've run across this with thumbnail caches in image processing software. It's easy for these kinds of things to be present in generally used libraries that your standard app will be using just because it has a GTK file selector for example. > you can mmap parts of a big file no problem with large file support in libc. For the speedups discussed here I definitely don't want to spend any time writing special code just to be able to read a 3GB file. That ship has long sailed and good riddance to all those subtle issues where things seem to work fine until they don't and it's not clear why.
CJefferson 8y ago
Actually, you can get a lot less than that once libraries are loaded and such like. I've found on some 32 bit linuxes you can't mmap a single file bigger than around 500mb, which means I couldn't check out a git repository containing 650mb cd images on a 32 bit Linux.
- geocar 8y ago
  You might check resource limits and/or disable address randomisation (or other "security" features) since they can really eat up your (limited) address space. Don't forget to not put the bugs in, though!

pantalaimon 8y ago

You already have every lib duplicated if you want to run 32bit x86 applications like steam. Distributions handle it just fine. And as the kernel is responsible for caching (or what kind of caching do you mean?) You will not lose that.