4 ms·

So if I'm understanding correctly, a useful hardening step would be to ensure that each dynamically linked library had it's own GOT & that the tables are marked

So if I'm understanding correctly, a useful hardening step would be to ensure that each dynamically linked library had it's own GOT & that the tables are marked as read-only once dynamic linking completes (i.e. you can't reach across dynamic boundaries to patch someone else's ifunc entries). That would improve the supply chain security of code getting linked in somewhere but never executed. EDIT: Or even better, perhaps ifunc should be implemented in a declarative fashion so that you can't just cause arbitrary code execution from each library you link against. That might be harder to implement at this point considering back compat, but probably is something that could be layered in over a longer time period (i.e. if you build any library with "declaratively linked ifunc" feature bit set, then the dynamic linker will force all linked libraries to have that feature flag or fail to launch).

jerjerjer 3y ago
From another angle: build systems. Currently building most libraries usually involves executing a series of highly complex byzantine scripts, requiring turing-complete environment. This gives attacker an endless attack surface, and when the build process is hijacked - well, the opportunities are there. Moving to a declarative build process with only a limited state machine as an executor would help. Requiring all source blobs being reproducible might also be something to think about.
- deleted 3y ago
  [deleted]
- kimixa 3y ago
  Having a "simple" declarative build definition won't help if the thing that interprets and executes that definition is shipped with the package, as said interpreter is likely orders of magnitude more complex and harder to personally review. As is what happened with the xz example - the "attack" was hidden in code that is autogenerated by autotools, not the build or configuration definitions. People put trust in distros and packagers to having something of a review chain - there's 0% chance you personally be an expert in everything executing on your workstation right now (outside of maybe toy systems). I'm not an expert in m4 or bash, but I hope enough experts are in the chain to get to my distro's package library are that such things are less likely. But that is all bypassed here. I think this particular approach is a one-off, as I know of no other build environment where it's expected to have the generated executable of the build system "helpfully" packaged in the tarball as a packaging step. If it is in some I'm not aware of, I hope that decision is being re-examined now.
  - AtlasBarfed 3y ago
    It needs money and people. Government funding for defense of the economy and computing in the "free world". Certainly the defense department has a billion to spare, and so does the EU
  - MichaelZuo 3y ago
    This doesn't work for anything actually complex. For example if you've ever taken a look at the bluetooth specs you would not trust a single person in the world to implement it correctly and you probably wouldn't even trust an arbitrarily large team to implement it correctly. Unless they had a long demonstrated and credible track record of shipping perfectly functional products and an effectively unlimited budget, i. e. Apple and maybe 1 or 2 other groups, at most.
    - jancsika 3y ago
      > For example if you've ever taken a look at the bluetooth specs you would not trust a single person in the world to implement it correctly and you probably wouldn't even trust an arbitrarily large team to implement it correctly. I messed around a tiny bit with Bluetooth on Linux recently. Going to rankly speculate that Bluetooth is such a special case of hell such that it makes a distracting example here. I mean, as a joke suppose we wanted to design a 3.5 mm patch cord that pitch shifts down a 1/4 step for randomly chosen stretches. It turns out to be easy-- just remove the wire from the casing and replace it with cheap bluetooth chips at either end. You'll get that behavior for free! :) Compare to, say, USB, where your point above would apply just as well. I wouldn't be distracted by that example because even cheapo, decades-old USB drives to this day let me read/write without interleaving zeros in my data. Shit, now I'm distracted again. Does Bluetooth audio even have the notion of a buffer size that I can set from the sender? And I guess technically the receiver isn't interleaving the signal with zeros-- it's adjusting the rate at which it sends blocks of the received data to the audio subsystem. Was Bluetooth audio basically designed just for the human voice under the assumption we're constantly pitch shifting? Oops, I almost forgot-- the sample-rate shift is preceded by a dropout, so I do get interleaved zeros in the audio data! I actually get a smoother ramp in my Prius changing from battery to ICE than I do in my wireless audio system!!! Anyhow, what were we talking about again?
- nonrandomstring 3y ago
  Yes. Other than the vulnerability of developers/maintainers the other big takeaway I get from this incident is that build systems have become dangerously unwieldy. There's just too many moving parts and too many places to hide bad stuff.
  - tsimionescu 3y ago
    Build systems have always been like this. It is in fact more recent build systems that are limiting the amount of crazyness you can do compared to the older ones.
- duped 3y ago
  This is how we got autoconf in the first place - an under specified and under powered build configuration language (make) which led to people that did need arbitrary logic to generate the build files at configuration time. Don't limit the build system language. It makes builds more complex, not less.
saagarjha 3y ago
Yes and no, but mostly no. This would prevent simple use of ifuncs in this way, but it's important to understand that the author of this could inject arbitrary code into the library that ends up in the address space of a sensitive process. At that point, all bets are off: it could remap the GOT as writable if it so chose, or (this is mostly here for the EDR people will certainly bring it up after reading this) if trying to do that is flagged as "suspicious" or the OS gains the ability to block such a transition, the injected code can subvert control flow in hundreds of other ways. It has arbitrary read/write, code execution, everything: there is no security mitigation that can stop it. If it so wishes it can leak private keys and send them to the attacker directly. It can spawn a shell. Trying to design protections at this stage of compromise is a fool's errand.
- tux3 3y ago
  >it's important to understand that the author of this could inject arbitrary code into the library that ends up in the address space of a sensitive process. At that point, all bets are off: it could remap the GOT as writable if it so chose, or (this is mostly here for the EDR people will certainly bring it up after reading this) if trying to do that is flagged as "suspicious" or the OS gains the ability to block such a transition, the injected code can subvert control flow in hundreds of other ways. I think it's important to push back on a very specific point here. It's true in general that if the attacker has added a backdoor in your library, and you're going to call that library's code, you've pretty much lost. Game over. Go home. But this was a very different attack, in that the attackers couldn't directly target sshd. They couldn't manage to target any library that sshd calls directly either. The library with the backdoor is code that was never actually called at runtime. This is important because it means it *doesn't* have a hundred ways to reach code execution. It only had a few select ways, mainly constructors and indirect function resolvers. The amount of weirdness in glibc's runtime loader is not unbounded. There aren't actually a hundred places where it allows random libs to run code before main. And we should take a good look at those couple place that are clearly juicy high-value gadgets to attacker. When glibc's runtime loader first loads a binary, first reaches a relocation for a STT_GNU_IFUNC, everything is still in a pristine state and no arbitrary code can run without being explicitly called. Attackers don't have magical powers that allow them to run code before you hand them the control flow. At this point in the runtime loader, an ifunc resolver cannot do anything without being caught. It cannot "just open /proc/self/mem" or "just call mprotect". It cannot "just disassemble the caller" or "just overwrite the GOT". I really want to hammer home how different that is from letting the attacker run code after main. There's nothing you can do if you directly call a backdoored library. But we shouldn't let attackers get away with spending 400ms parsing and disassembling ELFs in memory in an ifunc resolver of all things. Overwriting the GOT like nobody's watching. The backdoor isn't magic. For all the beautiful sophistication that went into it, it made many mistakes along the way that could have led to a detection. From valgrind errors to unacceptable amount of noise (400ms!) before main.
ajross 3y ago
> the tables are marked as read-only once dynamic linking completes Alas this won't work. Dynamic linking is lazy, there's no moment when it's "complete". The correct function pointers get loaded and inserted into the table (in place of stubs) when called the first time, which can be arbitrarily far into the future. In fact in most large library ecosystems (gtk apps, etc...) most of the linked functions are never called at all.
- CUViper 3y ago
  Best practice is to resolve them early now: https://www.redhat.com/en/blog/hardening-elf-binaries-using-relocation-read-only-relro https://www.redhat.com/en/blog/hardening-elf-binaries-using-...
- lyu07282 3y ago
  in this case it takes advantage of the fact that sshd is compiled with early binding not lazy. The ifunc resolver function is called early, right after the dynamic libraries are loaded. It's eager binding (that's what LD_BIND_NOW=1 / -W,-z,now are doing) is a security feature, GOT table will be readonly early. Didn't help with security in this case lol
  - lyu07282 3y ago
    just to add a detail: with lazy binding the function resolver might not even be called at all, the link tree is like sshd.elf -> systemd.so -> lzma.so. If systemd uses a ifunc symbol in lzma but sshd is not using that symbol: if lazy binding the resolver will never run, if eager binding the resolver will run. Something the backdoor also took advantage of
BobbyTables2 3y ago
And yet such is all trivially defeated by writing read-only memory using /proc/self/mem. Even a Rust program with no unsafe code can do this. Patching readonly libc code from Python without external modules is also trivial, resulting in arbitrary execution!
afh1 3y ago
If you build for the host architecture you can completely disable ifunc without losing anything. In Gentoo it is common to build with -march=native, and disabling ifunc is as simple as setting -multiarch in glibc's USE flags. I've seen no negative impact from it.
- markhahn 3y ago
  goodie! more, and less-controlled build environments to compromise, what could possibly go wrong?
candiddevmike 3y ago
Are there any programming languages that can sandbox library imports?
- pcwalton 3y ago
  Not really, except for a few researchy object capability based systems. (This is kind of the promise of ocap systems: functions you call can only use what they're given and nothing more.) If you don't trust a library, you can run it in a separate sandboxed process (or app domain, or wasm container, or whatever your language provides).
- jjav 3y ago
  Yes/no. The Java security manager can, but it was deprecated ;-( I know it was a pain to use correctly. But still.

jerjerjer 3y ago

From another angle: build systems. Currently building most libraries usually involves executing a series of highly complex byzantine scripts, requiring turing-complete environment. This gives attacker an endless attack surface, and when the build process is hijacked - well, the opportunities are there. Moving to a declarative build process with only a limited state machine as an executor would help. Requiring all source blobs being reproducible might also be something to think about.

deleted 3y ago
[deleted]
kimixa 3y ago
Having a "simple" declarative build definition won't help if the thing that interprets and executes that definition is shipped with the package, as said interpreter is likely orders of magnitude more complex and harder to personally review. As is what happened with the xz example - the "attack" was hidden in code that is autogenerated by autotools, not the build or configuration definitions. People put trust in distros and packagers to having something of a review chain - there's 0% chance you personally be an expert in everything executing on your workstation right now (outside of maybe toy systems). I'm not an expert in m4 or bash, but I hope enough experts are in the chain to get to my distro's package library are that such things are less likely. But that is all bypassed here. I think this particular approach is a one-off, as I know of no other build environment where it's expected to have the generated executable of the build system "helpfully" packaged in the tarball as a packaging step. If it is in some I'm not aware of, I hope that decision is being re-examined now.
- AtlasBarfed 3y ago
  It needs money and people. Government funding for defense of the economy and computing in the "free world". Certainly the defense department has a billion to spare, and so does the EU
- MichaelZuo 3y ago
  This doesn't work for anything actually complex. For example if you've ever taken a look at the bluetooth specs you would not trust a single person in the world to implement it correctly and you probably wouldn't even trust an arbitrarily large team to implement it correctly. Unless they had a long demonstrated and credible track record of shipping perfectly functional products and an effectively unlimited budget, i. e. Apple and maybe 1 or 2 other groups, at most.
  - jancsika 3y ago
    > For example if you've ever taken a look at the bluetooth specs you would not trust a single person in the world to implement it correctly and you probably wouldn't even trust an arbitrarily large team to implement it correctly. I messed around a tiny bit with Bluetooth on Linux recently. Going to rankly speculate that Bluetooth is such a special case of hell such that it makes a distracting example here. I mean, as a joke suppose we wanted to design a 3.5 mm patch cord that pitch shifts down a 1/4 step for randomly chosen stretches. It turns out to be easy-- just remove the wire from the casing and replace it with cheap bluetooth chips at either end. You'll get that behavior for free! :) Compare to, say, USB, where your point above would apply just as well. I wouldn't be distracted by that example because even cheapo, decades-old USB drives to this day let me read/write without interleaving zeros in my data. Shit, now I'm distracted again. Does Bluetooth audio even have the notion of a buffer size that I can set from the sender? And I guess technically the receiver isn't interleaving the signal with zeros-- it's adjusting the rate at which it sends blocks of the received data to the audio subsystem. Was Bluetooth audio basically designed just for the human voice under the assumption we're constantly pitch shifting? Oops, I almost forgot-- the sample-rate shift is preceded by a dropout, so I do get interleaved zeros in the audio data! I actually get a smoother ramp in my Prius changing from battery to ICE than I do in my wireless audio system!!! Anyhow, what were we talking about again?
nonrandomstring 3y ago
Yes. Other than the vulnerability of developers/maintainers the other big takeaway I get from this incident is that build systems have become dangerously unwieldy. There's just too many moving parts and too many places to hide bad stuff.
- tsimionescu 3y ago
  Build systems have always been like this. It is in fact more recent build systems that are limiting the amount of crazyness you can do compared to the older ones.
duped 3y ago
This is how we got autoconf in the first place - an under specified and under powered build configuration language (make) which led to people that did need arbitrary logic to generate the build files at configuration time. Don't limit the build system language. It makes builds more complex, not less.

saagarjha 3y ago

Yes and no, but mostly no. This would prevent simple use of ifuncs in this way, but it's important to understand that the author of this could inject arbitrary code into the library that ends up in the address space of a sensitive process. At that point, all bets are off: it could remap the GOT as writable if it so chose, or (this is mostly here for the EDR people will certainly bring it up after reading this) if trying to do that is flagged as "suspicious" or the OS gains the ability to block such a transition, the injected code can subvert control flow in hundreds of other ways. It has arbitrary read/write, code execution, everything: there is no security mitigation that can stop it. If it so wishes it can leak private keys and send them to the attacker directly. It can spawn a shell. Trying to design protections at this stage of compromise is a fool's errand.

tux3 3y ago
>it's important to understand that the author of this could inject arbitrary code into the library that ends up in the address space of a sensitive process. At that point, all bets are off: it could remap the GOT as writable if it so chose, or (this is mostly here for the EDR people will certainly bring it up after reading this) if trying to do that is flagged as "suspicious" or the OS gains the ability to block such a transition, the injected code can subvert control flow in hundreds of other ways. I think it's important to push back on a very specific point here. It's true in general that if the attacker has added a backdoor in your library, and you're going to call that library's code, you've pretty much lost. Game over. Go home. But this was a very different attack, in that the attackers couldn't directly target sshd. They couldn't manage to target any library that sshd calls directly either. The library with the backdoor is code that was never actually called at runtime. This is important because it means it *doesn't* have a hundred ways to reach code execution. It only had a few select ways, mainly constructors and indirect function resolvers. The amount of weirdness in glibc's runtime loader is not unbounded. There aren't actually a hundred places where it allows random libs to run code before main. And we should take a good look at those couple place that are clearly juicy high-value gadgets to attacker. When glibc's runtime loader first loads a binary, first reaches a relocation for a STT_GNU_IFUNC, everything is still in a pristine state and no arbitrary code can run without being explicitly called. Attackers don't have magical powers that allow them to run code before you hand them the control flow. At this point in the runtime loader, an ifunc resolver cannot do anything without being caught. It cannot "just open /proc/self/mem" or "just call mprotect". It cannot "just disassemble the caller" or "just overwrite the GOT". I really want to hammer home how different that is from letting the attacker run code after main. There's nothing you can do if you directly call a backdoored library. But we shouldn't let attackers get away with spending 400ms parsing and disassembling ELFs in memory in an ifunc resolver of all things. Overwriting the GOT like nobody's watching. The backdoor isn't magic. For all the beautiful sophistication that went into it, it made many mistakes along the way that could have led to a detection. From valgrind errors to unacceptable amount of noise (400ms!) before main.

ajross 3y ago

> the tables are marked as read-only once dynamic linking completes Alas this won't work. Dynamic linking is lazy, there's no moment when it's "complete". The correct function pointers get loaded and inserted into the table (in place of stubs) when called the first time, which can be arbitrarily far into the future. In fact in most large library ecosystems (gtk apps, etc...) most of the linked functions are never called at all.

CUViper 3y ago
Best practice is to resolve them early now: https://www.redhat.com/en/blog/hardening-elf-binaries-using-relocation-read-only-relro https://www.redhat.com/en/blog/hardening-elf-binaries-using-...
lyu07282 3y ago
in this case it takes advantage of the fact that sshd is compiled with early binding not lazy. The ifunc resolver function is called early, right after the dynamic libraries are loaded. It's eager binding (that's what LD_BIND_NOW=1 / -W,-z,now are doing) is a security feature, GOT table will be readonly early. Didn't help with security in this case lol
- lyu07282 3y ago
  just to add a detail: with lazy binding the function resolver might not even be called at all, the link tree is like sshd.elf -> systemd.so -> lzma.so. If systemd uses a ifunc symbol in lzma but sshd is not using that symbol: if lazy binding the resolver will never run, if eager binding the resolver will run. Something the backdoor also took advantage of

BobbyTables2 3y ago

And yet such is all trivially defeated by writing read-only memory using /proc/self/mem. Even a Rust program with no unsafe code can do this. Patching readonly libc code from Python without external modules is also trivial, resulting in arbitrary execution!

afh1 3y ago

If you build for the host architecture you can completely disable ifunc without losing anything. In Gentoo it is common to build with -march=native, and disabling ifunc is as simple as setting -multiarch in glibc's USE flags. I've seen no negative impact from it.

markhahn 3y ago
goodie! more, and less-controlled build environments to compromise, what could possibly go wrong?

candiddevmike 3y ago

Are there any programming languages that can sandbox library imports?

pcwalton 3y ago
Not really, except for a few researchy object capability based systems. (This is kind of the promise of ocap systems: functions you call can only use what they're given and nothing more.) If you don't trust a library, you can run it in a separate sandboxed process (or app domain, or wasm container, or whatever your language provides).
jjav 3y ago
Yes/no. The Java security manager can, but it was deprecated ;-( I know it was a pain to use correctly. But still.