3 ms·

This assumes that executable code pages can be shared between processes. I'm skeptical that this is still a notable optimization on modern systems because dynam

by gmueckl 7mo ago

This assumes that executable code pages can be shared between processes. I'm skeptical that this is still a notable optimization on modern systems because dynamic linking writes to executable memory to perform relocations in the loaded code. So this would counteract copy on write. And at least with ASLR, the result should be different for each process anyway.

cataphract 7mo ago
ld writes to the GOT. The executable segment where .text lives is not written to (it's position independent code in dynamic libraries). ASLR is not an obstacle -- the same exact code can be mapped into different base addresses in different processes, so they can be backed by the same actual memory.
- manwe150 7mo ago
  That’s true on most systems (modern or not), but actually never been true on Windows due to PE/COFF format limitations. But also, that system doesn’t/can’t do effective ASLR because of the binary slide being part of the object file spec.
- gmueckl 7mo ago
  I can't reconcile this with the code that GCC generates for accessing global variables. There is no additional indirection there, just a constant 0 address that needs to be replaced later.
  - wat10000 7mo ago
    Are you looking at the code before or after the static linker runs?
  - cataphract 7mo ago
    Assuming the symbol is defined in the library, when the static linker runs (ld -- we're not talking ld.so), it will decide whether the global variable is preemptable or not, that is, if it can be resolved to a symbol outside the dso. Generally, by default it is, though this depends on many things -- visibility attributes, linker scripts, -Bsymbolic, etc. If it is, ld will have the final code reach into the GOT. If not, it can just use instruction (PC) relative offsets.
    - gmueckl 7mo ago
      I've never observed a (non-LTO) linker exchange instructions. I want to see an example before I can believe this.
      - cataphract 7mo ago
        I'm not sure if you're just trolling, but I'll give the same example I gave before (you can get even wilder simplifications -- called relaxations -- with TLS, since there are 4 levels of generality there). I'm not sure what you meant by "changing isntructions", but in the first case the linker did the fixup indicated by the relocation and in the second reduced the generality of the reference (one less level of indirection by changing mov to lea) because it knew the symbol could not be preempted (more exactly, the R_X86_64_REX_GOTPCRELX relocation allows the linker to do the relaxation if it can determine that it's safe to) root@1f0775a74fd7:/tmp# cat a.c int glob; int main() { return glob; } root@1f0775a74fd7:/tmp# gcc -c a.c -fPIC -o a.o root@1f0775a74fd7:/tmp# objdump --disassemble=main a.o a.o: file format elf64-x86-64 Disassembly of section .text: 0000000000000000 <main>: 0: f3 0f 1e fa endbr64 4: 55 push %rbp 5: 48 89 e5 mov %rsp,%rbp 8: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # f <main+0xf> f: 8b 00 mov (%rax),%eax 11: 5d pop %rbp 12: c3 ret root@1f0775a74fd7:/tmp# readelf -rW a.o | grep glob 000000000000000b 000000030000002a R_X86_64_REX_GOTPCRELX 0000000000000000 glob - 4 root@1f0775a74fd7:/tmp# gcc -shared -o a.so a.o root@1f0775a74fd7:/tmp# objdump --disassemble=main a.so (...) 00000000000010f9 <main>: 10f9: f3 0f 1e fa endbr64 10fd: 55 push %rbp 10fe: 48 89 e5 mov %rsp,%rbp 1101: 48 8b 05 b8 2e 00 00 mov 0x2eb8(%rip),%rax # 3fc0 <glob-0x4c> 1108: 8b 00 mov (%rax),%eax 110a: 5d pop %rbp 110b: c3 ret (...) root@1f0775a74fd7:/tmp# readelf -r a.so | grep glob 000000003fc0 000600000006 R_X86_64_GLOB_DAT 000000000000400c glob + 0 root@1f0775a74fd7:/tmp# gcc -shared -Wl,-Bsymbolic -o a.symb.so a.o root@1f0775a74fd7:/tmp# readelf -r a.symb.so | grep glob root@1f0775a74fd7:/tmp# objdump --disassemble=main a.symb.so (...) Disassembly of section .text: 00000000000010f9 <main>: 10f9: f3 0f 1e fa endbr64 10fd: 55 push %rbp 10fe: 48 89 e5 mov %rsp,%rbp 1101: 48 8d 05 04 2f 00 00 lea 0x2f04(%rip),%rax # 400c <glob> 1108: 8b 00 mov (%rax),%eax 110a: 5d pop %rbp 110b: c3 ret (...)
  - gmueckl 7mo ago
    OK, I spent a few additional minutes digging into this. It's been too long since I looked at those mechanisms. Turns out my brain was stuck in pre-PIE world. Global variables in PIC shared libraries are really weird: the shared library's variable is placed into the main program image data segment and the relocation is happening in the shared library, which means that there is an indirection generated in the library's machine code.
wat10000 7mo ago
Dynamic linking doesn't have to write to code. I'm not familiar with other platforms, but on macOS, relocations are all in data, and any code that needs a relocation will indirect through non-code pages. I assume it's similar on other OSes. This optimization is essential. A typical process maps in hundreds of megabytes of code from the OS. There are hundreds of processes running at any given time. Eyeballing the numbers on an older Mac I have here (a newer one would surely be worse) I'd need maybe 50GB of RAM just to hold the code of all the running processes if the pages couldn't be shared.

by gmueckl 7mo ago

cataphract 7mo ago

ld writes to the GOT. The executable segment where .text lives is not written to (it's position independent code in dynamic libraries). ASLR is not an obstacle -- the same exact code can be mapped into different base addresses in different processes, so they can be backed by the same actual memory.

manwe150 7mo ago
That’s true on most systems (modern or not), but actually never been true on Windows due to PE/COFF format limitations. But also, that system doesn’t/can’t do effective ASLR because of the binary slide being part of the object file spec.
gmueckl 7mo ago
I can't reconcile this with the code that GCC generates for accessing global variables. There is no additional indirection there, just a constant 0 address that needs to be replaced later.
- wat10000 7mo ago
  Are you looking at the code before or after the static linker runs?
- cataphract 7mo ago
  Assuming the symbol is defined in the library, when the static linker runs (ld -- we're not talking ld.so), it will decide whether the global variable is preemptable or not, that is, if it can be resolved to a symbol outside the dso. Generally, by default it is, though this depends on many things -- visibility attributes, linker scripts, -Bsymbolic, etc. If it is, ld will have the final code reach into the GOT. If not, it can just use instruction (PC) relative offsets.
  - gmueckl 7mo ago
    I've never observed a (non-LTO) linker exchange instructions. I want to see an example before I can believe this.
    - cataphract 7mo ago
      I'm not sure if you're just trolling, but I'll give the same example I gave before (you can get even wilder simplifications -- called relaxations -- with TLS, since there are 4 levels of generality there). I'm not sure what you meant by "changing isntructions", but in the first case the linker did the fixup indicated by the relocation and in the second reduced the generality of the reference (one less level of indirection by changing mov to lea) because it knew the symbol could not be preempted (more exactly, the R_X86_64_REX_GOTPCRELX relocation allows the linker to do the relaxation if it can determine that it's safe to) root@1f0775a74fd7:/tmp# cat a.c int glob; int main() { return glob; } root@1f0775a74fd7:/tmp# gcc -c a.c -fPIC -o a.o root@1f0775a74fd7:/tmp# objdump --disassemble=main a.o a.o: file format elf64-x86-64 Disassembly of section .text: 0000000000000000 <main>: 0: f3 0f 1e fa endbr64 4: 55 push %rbp 5: 48 89 e5 mov %rsp,%rbp 8: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # f <main+0xf> f: 8b 00 mov (%rax),%eax 11: 5d pop %rbp 12: c3 ret root@1f0775a74fd7:/tmp# readelf -rW a.o | grep glob 000000000000000b 000000030000002a R_X86_64_REX_GOTPCRELX 0000000000000000 glob - 4 root@1f0775a74fd7:/tmp# gcc -shared -o a.so a.o root@1f0775a74fd7:/tmp# objdump --disassemble=main a.so (...) 00000000000010f9 <main>: 10f9: f3 0f 1e fa endbr64 10fd: 55 push %rbp 10fe: 48 89 e5 mov %rsp,%rbp 1101: 48 8b 05 b8 2e 00 00 mov 0x2eb8(%rip),%rax # 3fc0 <glob-0x4c> 1108: 8b 00 mov (%rax),%eax 110a: 5d pop %rbp 110b: c3 ret (...) root@1f0775a74fd7:/tmp# readelf -r a.so | grep glob 000000003fc0 000600000006 R_X86_64_GLOB_DAT 000000000000400c glob + 0 root@1f0775a74fd7:/tmp# gcc -shared -Wl,-Bsymbolic -o a.symb.so a.o root@1f0775a74fd7:/tmp# readelf -r a.symb.so | grep glob root@1f0775a74fd7:/tmp# objdump --disassemble=main a.symb.so (...) Disassembly of section .text: 00000000000010f9 <main>: 10f9: f3 0f 1e fa endbr64 10fd: 55 push %rbp 10fe: 48 89 e5 mov %rsp,%rbp 1101: 48 8d 05 04 2f 00 00 lea 0x2f04(%rip),%rax # 400c <glob> 1108: 8b 00 mov (%rax),%eax 110a: 5d pop %rbp 110b: c3 ret (...)
- gmueckl 7mo ago
  OK, I spent a few additional minutes digging into this. It's been too long since I looked at those mechanisms. Turns out my brain was stuck in pre-PIE world. Global variables in PIC shared libraries are really weird: the shared library's variable is placed into the main program image data segment and the relocation is happening in the shared library, which means that there is an indirection generated in the library's machine code.

wat10000 7mo ago

Dynamic linking doesn't have to write to code. I'm not familiar with other platforms, but on macOS, relocations are all in data, and any code that needs a relocation will indirect through non-code pages. I assume it's similar on other OSes. This optimization is essential. A typical process maps in hundreds of megabytes of code from the OS. There are hundreds of processes running at any given time. Eyeballing the numbers on an older Mac I have here (a newer one would surely be worse) I'd need maybe 50GB of RAM just to hold the code of all the running processes if the pages couldn't be shared.