5 ms·

The PLT is mostly a necessary evil of two requirements: lazy-loaded symbols and W^X memory. The former is necessary to reduce binary load times, and the latter

The PLT is mostly a necessary evil of two requirements: lazy-loaded symbols and W^X memory. The former is necessary to reduce binary load times, and the latter is necessary for security (RWX segments in memory are ripe for abuse in the event of a vulnerability). A JIT trades off very differently: slow startup ("warm up" period) and often creates RWX segments, but gains speed advantages as seen here. Of course, the GOT referred to by the PLT is itself is a bit of a security vulnerability, because it's usually a writable chunk of memory with a ton of frequently-called function pointers; corrupting it gives you relatively easy control over program flow. The new mitigation is "full RelRO", which resolves all GOT entries during binary load, then marks the GOT segment read-only. This gains in security, but trades off binary load time, and it is still going to be slower at runtime than the JIT. Full RelRO is usually off by default because of this added overhead. Modifying the code segment to include direct function references (either during binary load or later, during runtime) is also not a great approach because it means being unable to share the code segment between processes, which would increase memory pressure. We could do what Windows does, which is to have static randomization (randomization of binary addresses is performed once per boot, not once per process), enabling code to be "statically relocated" lazily per boot, but this carries its own complexities, and opens you up to different security vulnerabilities. There's no magic bullet here, unfortunately. There are just tradeoffs everywhere in many directions - between runtime speed, startup time, security and memory usage.

ioquatix 8y ago
Static linking avoids this overhead right?
- richardwhiuk 8y ago
  Increases memory usage, disk space, patch requirements and load time due to lack of shared library caching
- saagarjha 8y ago
  Yes, at the cost of a bloated binary size.
  - gaius 8y ago
    But what if you link in just the functions you need, vs having the enormity of Library X on your disk that the package manager put there because it can’t know what you actually use/need?
    - saagarjha 8y ago
      Sure, but generally Library X is already on your system because someone else needs it. So why not use it if it's there?
      - gaius 8y ago
        Maybe you want it in a container with minimum dependency
- nneonneo 8y ago
  Yes, a statically-linked binary will have all internal references resolved by the linker. If PIE is disabled, the addresses will all be fixed, which makes exploiting such a binary much easier. Glibc has only just added support for static PIE executables (in Feb of this year), and I'm not sure what the implementation is - whether it relocates all the references in the binary (increased memory usage), or uses a PLT/GOT approach to support relocations (increased runtime overhead).
  - brandmeyer 8y ago
    Depending on the linkage, function calls internal to the library can use ordinary PC-relative function calls instead of fully indirect function calls. So you get the low per-call cost, but pay a higher per-process cost in memory consumption.
    - BeeOnRope 8y ago
      Exactly. The cost of indirect jumps isn't implied by position independent (PIE/PIC) code alone, but by PIE/PIC combined with jumps across separately loadable segments. For jumps _within_ a segment you can just use a direct relative call which is the fastest type of call and doesn't require any load-time fixup ever. Of course then there is also the consideration of symbol interposition: if you want to support that even _within_ a statically linked thing (usually something other than the main executable), then you need a GOT-like thing, but not necessarily a PLT-like thing.
  - tannhaeuser 8y ago
    Does it necessarily use more memory though? Eg. code is usually read-only hence can be shared by multiple processes (for apps that create many invocations of the same executable).
    - saagarjha 8y ago
      You'd have to check every page to see if you already have a read-only copy in memory though, right?
- mehrdadn 8y ago
  Not quite, necessarily. It means another program can't share the same DLL as that one. So you might well end up using more memory that way.
  - jcelerier 8y ago
    But you can use LTO with more libs which may reduce your overall binary size.
rurban 8y ago
I tested this overhead in one of my latest jits. With full PIC and PIE and ASLR (indirect calls via PLT via GOT), I got a 10x overhead over static no PIC, no PIE, all direct near calls. This was mostly observable on windows, as this was the only secure by default platform then. Linux was 10x faster. I.e. I got no observable win by replacing the calls with jitted near calls. Nowadays with PIE, PIC and ASLR maybe the jit solution is better again. One big factor of direct calls vs indirect calls is the CPU i-cache, which preloads the direct call, but not the indirect call.
- tlb 8y ago
  I believe the branch target buffer on modern x86 hardware predicts and preloads indirect calls too. This analysis claims that Core 2 and later chips predict indirect branches quite well: http://www.agner.org/optimize/microarchitecture.pdf http://www.agner.org/optimize/microarchitecture.pdf
  - brandmeyer 8y ago
    That's certainly true. But if you are consuming fewer BTB resources by making more direct calls, then the CPU can spend its BTBs accelerating things like predictable virtual function calls and the like.
    - rurban 8y ago
      Yes, this. And the BTB is always slower than a direct call. The first call always takes a big hit, only then it will be cached and subsequent calls are about the same speed is direct calls, with 1-2 cycs overhead. The address rarely changes, so it will always be in the first level BTB. But still.
Diggsey 8y ago
You could have an initial code segment which generates the dynamic mappings and then performs a system call. The system call would map in a new non-shared page which the kernel has dynamically generated from the supplied mappings. The user-land never sees the page as writable, only as executable, the kernel generates only jump instructions, and limits this behaviour to specific pages determined when the executable is loaded.
- saagarjha 8y ago
  That's what full Relocation Read-Only essentially does, right? You pay for this by having to bind all addresses at startup time rather than lazily as they're needed.
dataflow 8y ago
> Modifying the code segment to include direct function references is also not a great approach because it means being unable to share the code segment between processes Realistically how big of a deal is this? I feel like it's overblown, no? Literally the only huge program I have that sneezes out a gazillion instances of itself is Chrome. On everything else there's usually only one and most a few instances.
- MereInterest 8y ago
  If you work on a non-Windows platform, then many executables will rely on libraries provided by the package manager. Your code might not be using that library from within a different process, but that doesn't mean that somebody else isn't.
  - mehrdadn 8y ago
    Yeah sorry I wasn't clear, I was talking about Windows since this is the approach Windows takes. And somehow (perhaps mistakenly) I assumed "executable" meant EXE, not DLL (since I refer to the two as "binaries"). Otherwise yeah I agree [1]. [1] https://news.ycombinator.com/item?id=17175361 https://news.ycombinator.com/item?id=17175361
  - pjmlp 8y ago
    Not all non-Windows platforms have package managers, or are UNIX clones.
- bjourne 8y ago
  All JIT:ted VM:s modify their code segments. So not a big deal at all. Code modification is done to implement dynamic dispatch efficiently. For example, suppose the compiler generates code for "a + b" If it can't infer the types of a and b, it generates a call to a stub function. When that stub function is called, it looks at the types of a and b and generates efficient dispatch code. For example, if a and b were ints it would generate code like: "if (type(a)==int && type(b)==int) { return fast_add(a, b); } else { slow_but_generic_add(a, b); }" A sophisticated JIT could even keep track on how frequently the fast track is taken and generate a new stub based on that. It could even inline the stub into the calling function and recompile that one.

ioquatix 8y ago

Static linking avoids this overhead right?

richardwhiuk 8y ago
Increases memory usage, disk space, patch requirements and load time due to lack of shared library caching
saagarjha 8y ago
Yes, at the cost of a bloated binary size.
- gaius 8y ago
  But what if you link in just the functions you need, vs having the enormity of Library X on your disk that the package manager put there because it can’t know what you actually use/need?
  - saagarjha 8y ago
    Sure, but generally Library X is already on your system because someone else needs it. So why not use it if it's there?
    - gaius 8y ago
      Maybe you want it in a container with minimum dependency
nneonneo 8y ago
Yes, a statically-linked binary will have all internal references resolved by the linker. If PIE is disabled, the addresses will all be fixed, which makes exploiting such a binary much easier. Glibc has only just added support for static PIE executables (in Feb of this year), and I'm not sure what the implementation is - whether it relocates all the references in the binary (increased memory usage), or uses a PLT/GOT approach to support relocations (increased runtime overhead).
- brandmeyer 8y ago
  Depending on the linkage, function calls internal to the library can use ordinary PC-relative function calls instead of fully indirect function calls. So you get the low per-call cost, but pay a higher per-process cost in memory consumption.
  - BeeOnRope 8y ago
    Exactly. The cost of indirect jumps isn't implied by position independent (PIE/PIC) code alone, but by PIE/PIC combined with jumps across separately loadable segments. For jumps _within_ a segment you can just use a direct relative call which is the fastest type of call and doesn't require any load-time fixup ever. Of course then there is also the consideration of symbol interposition: if you want to support that even _within_ a statically linked thing (usually something other than the main executable), then you need a GOT-like thing, but not necessarily a PLT-like thing.
- tannhaeuser 8y ago
  Does it necessarily use more memory though? Eg. code is usually read-only hence can be shared by multiple processes (for apps that create many invocations of the same executable).
  - saagarjha 8y ago
    You'd have to check every page to see if you already have a read-only copy in memory though, right?
mehrdadn 8y ago
Not quite, necessarily. It means another program can't share the same DLL as that one. So you might well end up using more memory that way.
- jcelerier 8y ago
  But you can use LTO with more libs which may reduce your overall binary size.

rurban 8y ago

I tested this overhead in one of my latest jits. With full PIC and PIE and ASLR (indirect calls via PLT via GOT), I got a 10x overhead over static no PIC, no PIE, all direct near calls. This was mostly observable on windows, as this was the only secure by default platform then. Linux was 10x faster. I.e. I got no observable win by replacing the calls with jitted near calls. Nowadays with PIE, PIC and ASLR maybe the jit solution is better again. One big factor of direct calls vs indirect calls is the CPU i-cache, which preloads the direct call, but not the indirect call.

tlb 8y ago
I believe the branch target buffer on modern x86 hardware predicts and preloads indirect calls too. This analysis claims that Core 2 and later chips predict indirect branches quite well: http://www.agner.org/optimize/microarchitecture.pdf http://www.agner.org/optimize/microarchitecture.pdf
- brandmeyer 8y ago
  That's certainly true. But if you are consuming fewer BTB resources by making more direct calls, then the CPU can spend its BTBs accelerating things like predictable virtual function calls and the like.
  - rurban 8y ago
    Yes, this. And the BTB is always slower than a direct call. The first call always takes a big hit, only then it will be cached and subsequent calls are about the same speed is direct calls, with 1-2 cycs overhead. The address rarely changes, so it will always be in the first level BTB. But still.

Diggsey 8y ago

You could have an initial code segment which generates the dynamic mappings and then performs a system call. The system call would map in a new non-shared page which the kernel has dynamically generated from the supplied mappings. The user-land never sees the page as writable, only as executable, the kernel generates only jump instructions, and limits this behaviour to specific pages determined when the executable is loaded.

saagarjha 8y ago
That's what full Relocation Read-Only essentially does, right? You pay for this by having to bind all addresses at startup time rather than lazily as they're needed.

dataflow 8y ago

> Modifying the code segment to include direct function references is also not a great approach because it means being unable to share the code segment between processes Realistically how big of a deal is this? I feel like it's overblown, no? Literally the only huge program I have that sneezes out a gazillion instances of itself is Chrome. On everything else there's usually only one and most a few instances.

MereInterest 8y ago
If you work on a non-Windows platform, then many executables will rely on libraries provided by the package manager. Your code might not be using that library from within a different process, but that doesn't mean that somebody else isn't.
- mehrdadn 8y ago
  Yeah sorry I wasn't clear, I was talking about Windows since this is the approach Windows takes. And somehow (perhaps mistakenly) I assumed "executable" meant EXE, not DLL (since I refer to the two as "binaries"). Otherwise yeah I agree [1]. [1] https://news.ycombinator.com/item?id=17175361 https://news.ycombinator.com/item?id=17175361
- pjmlp 8y ago
  Not all non-Windows platforms have package managers, or are UNIX clones.
bjourne 8y ago
All JIT:ted VM:s modify their code segments. So not a big deal at all. Code modification is done to implement dynamic dispatch efficiently. For example, suppose the compiler generates code for "a + b" If it can't infer the types of a and b, it generates a call to a stub function. When that stub function is called, it looks at the types of a and b and generates efficient dispatch code. For example, if a and b were ints it would generate code like: "if (type(a)==int && type(b)==int) { return fast_add(a, b); } else { slow_but_generic_add(a, b); }" A sophisticated JIT could even keep track on how frequently the fast track is taken and generate a new stub based on that. It could even inline the stub into the calling function and recompile that one.