15 ms·
Linux Kernel Developers Discuss Dropping x32 Support
- deleted 8y ago[deleted]
- geocar 8y agoI read this as just dropping building an x32 kernel: It seems that mmap(MAP_32BIT) should continue to allow an x32 userspace (which is all I ever want anyway). Does anyone have more details on this?
- nwmcsween 8y agoWhat? How would MAP_32BIT reimplement an entire ABI? It has more in common with x86-64 than i386.
- rocqua 8y agoFrom a very cursory look, it seems like it doesn't. Based on the man page for mmap [1], it seems like all MAP_32BIT does is ensure that memory is allocated in the lower 2GB of addresses. Notably, this means the actual pointers to that memory are still 64 bit. Obviously, when you change pointers to be 32 bit, that requires an ABI change because the binary layout of your data structures is different. http://man7.org/linux/man-pages/man2/mmap.2.html http://man7.org/linux/man-pages/man2/mmap.2.html
- syn0byte 8y agoYou can easily truncate or pad to convert between 64-32bit as long as the 64bit end stays in the lower 2GB. You could do everything else in userspace if you really wanted. At the cost of safety, type handling and optimization from the compiler as all it would see is 64bit pointers being typecast and generic 32bit numeric data. Whatever you were looking to gain would be lost in extra work, and you can forget about debugging. But theoretically...
- geocar 8y agox32 isn't i386. It's x86_64 with truncated pointers. There aren't many syscalls that return pointers; mmap() is one of them, and using MAP_32BIT means that the pointer will be 32-bit.
- AstralStorm 8y agoBut the offset still is 64bit right? (As per large file support)
- walkingolof 8y agoDepends, the offset is a size_t, i.e. and unsigned integer, so if you compile the program as a 64 bit binary, its 64 bit, 32 bit otherwise.
- jabl 8y agoX32 has it's own (partial) set of syscalls. Similar to how a x86-64 kernel ran run traditional x86-32 code. 32-bit code tells the kernel to allocate in the lower 32-bit address space (via MAP_32BIT) precisely so that the 64-bit pointers the kernel works with can be safely truncated to 32-bit user space pointers.
- alexlarsson 8y agoThere was never a x32 kernel, it always ran on x86_64 kernels. x32 is just a userspace ABI.
- gnode 8y agoIt's an ABI for x86_64 rather than a kernel / architecture. The Linux ABI includes the memory map, syscalling convention and data structures passed to / from the kernel. For a start, the kernel sets up virtual memory with stack at the high addresses, which would be out of reach for 32-bit pointers (without translation). Secondly, pointers passed to the kernel would have to be converted to 64-bit and vice versa. Conceivably, you could remap your virtual address space to have a stack and libraries within the 32-bit space, and then jump into x32-like code which translated pointers to and from 64-bit for syscalls and kernel data structures. This would retain most of the benefits of x32 for CPU-bound processes.
- geocar 8y ago> Conceivably, you could remap your virtual address space to have a stack and libraries within the 32-bit space, and then jump into x32-like code which translated pointers to and from 64-bit for syscalls and kernel data structures. This would retain most of the benefits of x32 for CPU-bound processes. That's what I'm thinking about. You'd need a little cooperation from libc and the dynamic linker in order to have a good "user" experience, and I think you can even share 64-bit libs (provided they don't call mmap() themselves and return the pointer to your application), but these seem almost manageable.
- sprash 8y agoI think the X32 feature is vastly underused. Ideally every application that is unlikely to use more than 2GB of memory should use the X32 ABI. Cache latency is the most important bottleneck of today's general purpose CPUs. X32 Helps a lot with that and not just in "synthetic" benchmarks.
- geocar 8y ago100% agreed, but I'm not so convinced it needs (much) kernel support (as per my other comment). I don't usually use much libc, so an -m32 that still made elf-x86_64 would probably be enough for me. It is absolutely idiotic to have 64-bit pointers when I compile a program that uses less than 4 gigabytes of RAM. When such pointer values appear inside a struct, they not only waste half the memory, they effectively throw away half of the cache. -Knuth
- 0x0 8y agoI dunno, having a large address space could be useful if you work with large or lots of memory mapped files?
- AstralStorm 8y agoOr you could use large file support and not mmap everything. It is probably faster too and puts less pressure on overcommit mechanism.
- Hello71 8y agofile-backed mmap is by definition not related to overcommit, because the memory is committed by the file.
- geocar 8y agoSure. Parts of my (multi-process) application can use 64-bit pointers and parts can use 32-bit.
- pedrocr 8y ago
- hyperman1 8y agoJava has an interesting pointer model between 32 bit and 64 bit pointers: Compressed OOps. A pointer to a data structure is shifted right by 3 a.k.a divided by 8. You now have 35 bits and can addres 16GB. Data structures should be aligned on an 8 byte boundary. At runtime, the pointer is expanded to 64 bit in a register, multiplied by 8, and the correct offset for a struct field is added. So you can't point at individual bytes, and you have to shift, but you gain 4 bytes per pointer. It is said this hits the current sweet spot for Java.
- bogomipz 8y agoI believe this is why you sometimes see recommendations to no to set the JVM heap larger than 32 Gigs. Is this still the case though? Does JVM performance fall off a cliff without compressed pointers? Or is the issue more GC beyond 32 Gigs.
- lmm 8y agoThe advice I've heard is that there's no point setting a heap size between 32G and 48G: between those limits the bigger object references will cost you more space than you gain. Long pauses for full GC are a distinct issue that large heaps have. I believe the newer garbage collectors are designed to address this, but don't have much experience with them in practice.
- joemag 8y agoPerformance hit is barely noticeable for most apps. The biggest surprise is usually heap usage increase. You can increase your heap from 28 to 34 GB, and end up with less free memory.
- user5994461 8y agoA common misconception unfortunately. The heap is limited to roughly 30 GB, plus a bit of overhead. You will never run in 32bits mode if you set a 32GB heap, because overhead. From experience, a 30GB heap is similar to a 40GB in usable space. 32 vs 64 bits pointers respectively.
- bogomipz 8y agoThe author states: >"The x32 ABI allows for making use of the additional registers and other features of x86_64 but with just 32-bit pointers in order to provide faster performance when 64-bit pointers are unnecessary." Can someone say why would 32 bit pointers provide faster performance then 64 bit on modern hardware? Is this performance difference really non-negligible?
- jabl 8y ago> Can someone say why would 32 bit pointers provide faster performance then 64 bit on modern hardware? Is this performance difference really non-negligible? With narrower pointers your memory & caches becomes effectively bigger, as well as memory and cache bandwidth.
- danieldk 8y agoMaybe fewer cache misses in pointer-heavy data structures?
- geocar 8y ago"Negligible" means different things to different people. There's a benchmark floating around that saw 13% boost for some operations. That could mean saving an hour or more of compute time for some of my applications. The reason why it's faster is because your program is smaller, and your data is smaller, and that means a better (more efficient) use of the on-CPU cache.
- Slartie 8y ago64 bit pointers require double the space in memory, hence your caches inside the processor can only hold half the number of pointers. This means more cache misses and thus more waiting for abysmally slow (when compared with L1/L2 caches) RAM. The impact of this is AFAIK generally in the single-digit percentage area, but if you have an algorithm that just fits everything into the caches with 32 bit pointers, the change to 64 bit pointers might push it over the edge, which can then result in significant (>= double-digit percentage) performance decrease.
- jgowdy 8y agoIt’s simple. Pointer sizes being twice as big consume twice as much cache, thus you’re effectively cutting your cache size in half (worst case, if the cache line were stuffed end to end with pointers). I imagine the idea that cutting the cache size by up to 50% causes a severe performance impact doesn’t require much explanation.
- gnode 8y agoIf the problem is that a significant amount of cache is routinely wasted by storing many pointers of the form: 0xffffffffxxxxxxxx and 0x00000000xxxxxxxx, then I wonder why CPU designers haven't added an optimisation to their caches to store these numbers in 32-bit (or slightly larger) cells.
- pedrocr 8y agoThe CPU caches are caching normal memory pages. From the point of view of the CPU it doesn't know if what it's caching is a pointer or just any other random 64bit number until the code actually does anything with the cache. What you are suggesting might work generally if it was somehow worthwhile to try and do some lossless compression of page contents before storing them in the CPU cache. If that's not done yet I'm sure it's because the energy and extra transistors needed for it are not worth it, and not because CPU designers haven't thought about it.
- gok 8y agoMemory compression is actually pretty widely used. Linux has zram, which Android and ChromeOS have on by default. macOS/iOS has something similar.
- gnode 8y agoI wasn't suggesting determining whether the values were pointers or not. The number itself suggests whether it can be stored with less bits. If the occurrence of such numbers were relatively high, then this becomes a sensible tradeoff. In set associative caches, one or more of the "ways" in each index could be optimised for these relatively common small numbers, with fewer bits given to it, capable of only storing numbers of the small form.
- Wowfunhappy 8y agoI'm a little confused: is X32 support different from 32-bit x86 support, a la 32 bit mode in Windows?
- gsnedders 8y agoYes. It's essentially x86_64 but with all addresses being 32-bits, so you get all the things like extra registers that x86_64 added without the added memory bloat of longer addresses for everything.
- mg794613 8y agoWhat "bloat"? It's curious to see that when a technique comes out and is being misunderstood by the masses and praised as the best thing since sliced bread. Suddenly robustness, accountability and predictability turn into "bloat". Do you have a real world example?
- tomlu 8y agoIf you double the size of pointers (64 bit, up from 32 bit) then your program will take up more heap. This is especially significant for languages where everything is a heap allocation.
- zakk 8y agoIf you code does intresive computation with structures containing a lot of pointers, you’ll get much more cache miss when using 64 but pointers. The performance hit can be quite heavy! Cache is a limited resource, and you want as many pointers as possible to fit there.
- chaosite 8y agoIt's bloat used properly, in the most literal sense. Your pointers are 64bit, but you always use less than 32bits. So, all your pointers are twice is big. This wasted space from bits you do not need is "bloat".
- jcranmer 8y ago
- dragontamer 8y agoIf you want 32-bit pointers for compression, it seems more like the job of a "32-bit malloc()" like interface. The malloc_32bit() would return a pointer in a 32-bit 4GB space, and you just add a 64-bit pointer to get the true address. An add takes 1-clock cycle (latency). An L1 cache lookup takes 4-clock cycles (latency). So an add + L1 lookup would be only 5-cycles of latency. Skylake can do two memory lookups per clock and many adds per clock (I think 3 adds per clock?), so it wouldn't take many resources at all. The actual memory lookup takes ~100+ Cycles if its outside of the cache. It seems like this problem can be 100% solved in userland without needing kernel support. For something like following a linked-list chain, I'd expect almost no change in speed due to out-of-order executions and memory latency hiding techniques of the CPU. ---------- 64-bit user space should remain the default, if only for ASLR / security reasons. Its actually trivial for CPUs to traverse an entire 32-bit space (4-billion isn't a big number: most computers today are ~3GHz, or 3-billion operations per second...) So if you want your address space randomization to actually have any security what-so-ever, you better use the 64-bit (erm... really 48-bit) address space.
- kllrnohj 8y agoThis is what, for example, Android's ART does. On 64-bit platforms all the JVM references remained 32-bit. They know the heap base address so they can trivially expand the references out on deref, saving the memory overhead. But in this case you can't do it entirely in userspace because things like mmap exist. You get pointers from syscalls all the time, and those can't really be compressed down since there's not a single fixed base address to add onto them. You'd have to use a different type to represent malloc'd memory that is conforming to 32-bit compressed references, but that gets real messy real fast.
- TrainedMonkey 8y agoSure you can make it work, but how much existing userland code would need to be rewritten?
- Vogtinator 8y agoShould be trivial to implement using ptmalloc/dlmalloc/whatevermalloc and constraining it to 4GiB. Main issue I see is that this requires a new ABI for passing pointers around.
- qwerty456127 8y agoI just wonder how many consumer applications actually need more than 3 GiBs of RAM per process (and I fear to imagine such an app and the idea these are going to become common). As for now only professional (data science, graphics/multimedia editing, CAD etc) and server-side apps may need that much AFAIK, 64-bit pointers seem a waste on an average PC.
- kbirkeland 8y ago64 bit pointers are pretty important for security. When using ASLR, a certain number of the bits cannot be randomized. This leaves you with a randomization space of about 12 bits with 32-bit addresses, but over 40 bits of randomization with 64 bit addresses.
- hyperman1 8y agoOur company bought a nasty piece of proxy/security software that injects a bunch of DLLs right at the center of the address space. 2GB is small if you lose 256MB on this kind of contraption. We had regular problems when ASLR made unfortunate random choices at startup, and not enough continuous adress space was available for end user software.The only option was a reboot.
- ben0x539 8y agoVideo games will probably get there easily.
- kllrnohj 8y ago3 GiBs of virtual space can get exhausted pretty quick, particularly when working with large files or a lot of mmaps fragmenting the virtual space. Just because a process doesn't need more than 3 GiBs of physical RAM doesn't mean it's happy with only having 3 GiBs of virtual address space.
- perlgeek 8y agoA browser with several tabs open, some of them being more web applications than pages, can easily reach that limit. Like it or not, more and more applications are being replaced with web applications, and many of use simply have too many communication channels, and have web apps open for a calendar, slack, email, WhatsApp etc.
- deleted 8y ago[deleted]
- loeg 8y agoBlogspam website. Original source: https://lkml.org/lkml/2018/12/10/1145 https://lkml.org/lkml/2018/12/10/1145