8 ms·

SSHD: Random boot time relinking, OpenBSD

101 points by codesniperjoe 4y ago

codesniperjoe 4y ago
Finally! At least, someone finally understands that static, fully predictable, reproduce-able-builds are only an convenience feature for the attacker side.
- Someone 4y ago
  It’s not that finally. OpenBSD got kernel boot time relinking in 2017 (https://marc.info/?l=openbsd-tech&m=149887978201230&w=2 https://marc.info/?l=openbsd-tech&m=149887978201230&w=2). This extends it to an outward-facing executable. I guess the holy grail would be to combine this with hot patching (https://en.wikipedia.org/wiki/Patch_(computing)#HOT-PATCHING https://en.wikipedia.org/wiki/Patch_(computing)#HOT-PATCHING), and relink the kernel every now and then while it is running (currently, a system under attack would have to be rebooted every now and then, and that’s undesirable). That would face ‘a few’ technical hurdles, though.
  - camgunz 4y ago
    Yeah I was just thinking this; I've got like years of uptime on my OpenBSD server--don't know how much boot time relinking is helping me. But for like, desktops and laptops, it's fine and a great feature IMO (you probably wade through a lot more muck on a personal machine)
    - somat 4y ago
      If you have years of uptime on an openbsd machine you are not keeping it up to date. I have to admit I am guilty of this as well, but any mantained openbsd setup should have an uptime of no more than 6 months and a well maintained openbsd setup will be shorter than that as security patches are applied. Having said that one of the things I like about openbsd is that if you want to go dark and have an ultra stable system(no updates ever) all the pieces are there for you, (you will want to have the source, I would also make sure I have the ports tree for that release and a copy of the ports dist files.)
      - camgunz 4y ago
        This is true; my VPS has some kind of problem updating a FDE machine and I've procrastinated doing something about it for years. The answer is probably putting everything on tarsnap and reinstalling.
- saagarjha 4y ago
  I mean they're also great for being able to symbolicate crash logs and such.
- patrakov 4y ago
  Please don't throw out the baby with the bathwater. Fully reproducible builds provide great assurance against the supply chain attacks. But 100% reproducibility is in some cases a bit too much. What matters is whether the artifact can be easily proven to be functionally identical to the canonical one. So I am 100% for a fully predictable sshd random-relink kit, producing unpredictable sshd binaries, but only as long as there is an instruction how to check that the sshd binary that allegedly came from it indeed could have come from it, and was not quietly replaced by some malicious entity.
  - rollcat 4y ago
    > So I am 100% for a fully predictable sshd random-relink kit, producing unpredictable sshd binaries, but only as long as there is an instruction how to check that the sshd binary that allegedly came from it indeed could have come from it, and was not quietly replaced by some malicious entity. You can easily verify the integrity of the object files that are used in the random relinking - they are included in the binary distribution, and are necessary to perform the relinking. The debate of static vs dynamic linking is still going on, and a very strong argument against static linking has always been that upgrading vulnerable libraries is made difficult. But think of it: package managers already hold the meta-data of what links to what; object files can be distributed just as easily as shared objects; the last necessary step is to move the actual linking step from the kernel to the package manager.
    - wahern 4y ago
      On OpenBSD all static system binaries are compiled as static PIE, so they already benefit from ASLR. The issue, IIUC, is that ASLR only randomizes entire ELF sections relative to each other. In any executable or library, whether static or dynamic, the code is placed into one giant .text section, so the relative offsets remain static. In a dynamic executable all library dependencies are loaded separately, so at least each section of each library gets a unique base address. A leak that exposes the address of a function only leaks the address of other functions in that library, not every function in the process. But in a static executable all those libraries are also placed into the same .text section as the main program code, so a leak of any function address leaks all function addresses. In theory all functions, or more realistically groups of functions spanning page-size increments, could be dynamically located. The obvious way to achieve that would be to have multiple .text sections within a main executable or library. But off-hand I don't know if that's actually supported by ELF, or if so whether the standard tool chains and environments could easily support it.
- snvzz 4y ago
  It is possible to get your reproduceable build, 100% identical, and then random-relink it.
viraptor 4y ago
It's there a good link for the details? I'm guessing this does more than ASLR?
- codesniperjoe 4y ago
  Plain, simple and effective as always! The highly complex 'black magic' is 'sort --random' and (re-)link it all again. :) Makefile.relink: cc -o sshd `echo ${OBJS} | tr ' ' '\n' | sort -R` ${LDADD} ./sshd -V && install -o root -g wheel -m ${BINMODE} sshd /usr/sbin/sshd https://github.com/openbsd/src/commit/898412097f87ba70d4012f7a2e827f59719c659d#diff-cd8659908f01fc079dcdbc809d4b57f813b634420b0f7d8a3533258e23f122b6 https://github.com/openbsd/src/commit/898412097f87ba70d4012f...
  - viraptor 4y ago
    Ah, so that will have some features of ASLR missing. Specifically, you can't do this on a read only root and it didn't randomise the stack location as far as I can tell?
    - kryptiskt 4y ago
      Wouldn't the stack location be set at runtime and given by the stack register at the entry point?
      - viraptor 4y ago
        I think I've got a better idea now. So openbsd has ASLR which affects data, code/library, and stack positions. Then this solution works on top of it by reordering symbols within the code. One thing I'm still not sure about is whether the kernel could theoretically do the same reordering at load time using relocatable symbols.
        actionfromafar 4y ago
        Sounds like Amiga HUNK format with the HUNK_OVERLAY hunk. For instance various functions could be loaded anywhere in memory.
rfoo 4y ago
Does anyone know an actually-happened example case where a fine-grained ASLR (like the OpenBSD relink one) successfully mitigates or significantly hinders an exploit, and the usual ASLR doesn't? I'm curious because years ago the academic strongly pushes the FG ASLR story, then OpenBSD did kernel relinking, but I haven't heard any industry story on how effective this is.
- nine_k 4y ago
  How would you measure that? How would you notice a failed attempt at finding a hole, of stack smashing, etc, if it did not even result in a crash, or any other dysfunction? Not a rhetorical question.
  - codetrotter 4y ago
    I suppose if someone found a technique in the wild that defeats regular ASLR (even if only sometimes), they could then test that same technique against fine-grained ASLR and evaluate if the FG ASLR was more effective at preventing exploitation.
    - TheCondor 4y ago
      Any classic buffer overflow/stackssmash can defeat ASLR, it just might take a long time to get lucky guessing addresses. Couldn’t we Monte Carlo this? Maybe take a known vulnerable exec, create a fuzzing attacker and run it both ways seeing how long it takes to get lucky a few times. The more secure version should take longer.
  - insanitybit 4y ago
    There are whitehats constantly looking at this sort of thing. It's why we can say that KASLR is a huge waste of time - because it's both theoretically bogus and we have actual exploit devs saying that they don't care about it.
  - rfoo 4y ago
    It's not about failed attempt at finding a hole, it's about failed attempt at actually making use of a hole (i.e. write an exploit) given a hole. (FG)ASLR is more of a "targeting at exploit instead of vulnerability" style mitigation.
nine_k 4y ago
I remember how back in MS DOS days polymorphic viruses first appeared, in an attempt to avoid detection by antivirus software (useful and essential back then). Now the tables have turned, and legitimate software has to become somehow polymorphic to thwart attacks by malware.
- codesniperjoe 4y ago
  Yes, the base idea is not that new. I store since years every GO based application I use as small (few kb) source code tree checkout only, no binary at all. At runtime the wrapper compiles a randomized individual one-time-temporary-uniq binary via garble [0]. [0] https://github.com/burrowers/garble https://github.com/burrowers/garble
  - okl 4y ago
    How do you ensure that your compiler and libs are clean though?
    - codesniperjoe 4y ago
      The compiler (go) is part of a static read-only (compressed/in-memory) RootFS. Build on a air-gap build server, touching only signed/verified/reviewed code from git-offline mirror snaps. Go has no libs, all static. The resulting runtime only binaries are totally uniq/randomized and dependency free, straight from (signed) source code.
rwmj 4y ago
Takes some dedication to still be using CVS. Do they use another version control system to feed into CVS, or is CVS the tool they use directly?
- WhyNotHugo 4y ago
  They're working on GoT (game of trees), an OpenBSD-born client that interoperates with git repositories. AFAIK, they still interface with CVS directly, but I assume the expectation is to eventually transition to got. https://www.gameoftrees.org/ https://www.gameoftrees.org/
- lolpython 4y ago
  IIRC they pass around patches on email before committing to CVS. That workflow affords a bit more flexibility.
- krylon 4y ago
  They are working on a git replacement, got (Game Of Trees): http://gameoftrees.org/ http://gameoftrees.org/ So it looks like they are going to move on from CVS eventually.
- SoftTalker 4y ago
  There is a git mirror of the repo, and some devs use that for their own work. The project itself uses CVS for formal change commits.
theamk 4y ago
In order to enable relinking, they had to keep around all original .o and .a files, as well as a Makefile. Not a problem for OpenBSD but pretty unusual for binary Linux distros. I wonder if it is possible to make a relinker which only requires binary output -- so it could be easily incorporated into existing systems. One way I can think of is to keep relocation/original object information in the debug sections, so that one can reconstruct original object files and re-link them. But I am guessing this will not work with LTO though... Or maybe we can just make a bunch of debug sections and store input object/library files verbatim -- this will at least double the binary size, but will allow for easier relinking.
- wolf550e 4y ago
  I think Facebook BOLT relinks binaries: https://research.facebook.com/publications/bolt-a-practical-binary-optimizer-for-data-centers-and-beyond/ https://research.facebook.com/publications/bolt-a-practical-... https://groups.google.com/g/llvm-dev/c/ef3mKzAdJ7U/m/1shV64BYBAAJ?pli=1 https://groups.google.com/g/llvm-dev/c/ef3mKzAdJ7U/m/1shV64B...
planede 4y ago
If a squint hard then this is a custom dynamic loader for .o files with rudimentary ASLR (where all your entropy comes from the permutation of the .o files), that happens to cache to disk.
anonymousiam 4y ago
Dynamic re-linking is cool, but it can result in less-than-optimal executables. Sometimes it can be beneficial to optimize the link so most of the main thread stays in cache. Obviously this only really matters for CPU-intensive programs.
LinuxBender 4y ago
What impact will this have on anti-tampering software that looks for changes in executable checksums? Tripwire and OSSEC come to mind and both can report their findings to a centralized server. Do package manager integrity tests still work? I assume anyone here using BSD in a PCI environment have already figured something out. Some people also feed checksums into Splunk.
- antx 4y ago
  Very good point. As per Job snijder's own words [0], "the sshd binary becomes unique on every openbsd machine". So checksum-checking systems will indeed trip everytime. [0]: https://marc.info/?l=openbsd-tech&m=167388832715992&w=2 https://marc.info/?l=openbsd-tech&m=167388832715992&w=2

101 points by codesniperjoe 4y ago

Finally! At least, someone finally understands that static, fully predictable, reproduce-able-builds are only an convenience feature for the attacker side.

Someone 4y ago
It’s not that finally. OpenBSD got kernel boot time relinking in 2017 (https://marc.info/?l=openbsd-tech&m=149887978201230&w=2 https://marc.info/?l=openbsd-tech&m=149887978201230&w=2). This extends it to an outward-facing executable. I guess the holy grail would be to combine this with hot patching (https://en.wikipedia.org/wiki/Patch_(computing)#HOT-PATCHING https://en.wikipedia.org/wiki/Patch_(computing)#HOT-PATCHING), and relink the kernel every now and then while it is running (currently, a system under attack would have to be rebooted every now and then, and that’s undesirable). That would face ‘a few’ technical hurdles, though.
- camgunz 4y ago
  Yeah I was just thinking this; I've got like years of uptime on my OpenBSD server--don't know how much boot time relinking is helping me. But for like, desktops and laptops, it's fine and a great feature IMO (you probably wade through a lot more muck on a personal machine)
  - somat 4y ago
    If you have years of uptime on an openbsd machine you are not keeping it up to date. I have to admit I am guilty of this as well, but any mantained openbsd setup should have an uptime of no more than 6 months and a well maintained openbsd setup will be shorter than that as security patches are applied. Having said that one of the things I like about openbsd is that if you want to go dark and have an ultra stable system(no updates ever) all the pieces are there for you, (you will want to have the source, I would also make sure I have the ports tree for that release and a copy of the ports dist files.)
    - camgunz 4y ago
      This is true; my VPS has some kind of problem updating a FDE machine and I've procrastinated doing something about it for years. The answer is probably putting everything on tarsnap and reinstalling.
saagarjha 4y ago
I mean they're also great for being able to symbolicate crash logs and such.
patrakov 4y ago
Please don't throw out the baby with the bathwater. Fully reproducible builds provide great assurance against the supply chain attacks. But 100% reproducibility is in some cases a bit too much. What matters is whether the artifact can be easily proven to be functionally identical to the canonical one. So I am 100% for a fully predictable sshd random-relink kit, producing unpredictable sshd binaries, but only as long as there is an instruction how to check that the sshd binary that allegedly came from it indeed could have come from it, and was not quietly replaced by some malicious entity.
- rollcat 4y ago
  > So I am 100% for a fully predictable sshd random-relink kit, producing unpredictable sshd binaries, but only as long as there is an instruction how to check that the sshd binary that allegedly came from it indeed could have come from it, and was not quietly replaced by some malicious entity. You can easily verify the integrity of the object files that are used in the random relinking - they are included in the binary distribution, and are necessary to perform the relinking. The debate of static vs dynamic linking is still going on, and a very strong argument against static linking has always been that upgrading vulnerable libraries is made difficult. But think of it: package managers already hold the meta-data of what links to what; object files can be distributed just as easily as shared objects; the last necessary step is to move the actual linking step from the kernel to the package manager.
  - wahern 4y ago
    On OpenBSD all static system binaries are compiled as static PIE, so they already benefit from ASLR. The issue, IIUC, is that ASLR only randomizes entire ELF sections relative to each other. In any executable or library, whether static or dynamic, the code is placed into one giant .text section, so the relative offsets remain static. In a dynamic executable all library dependencies are loaded separately, so at least each section of each library gets a unique base address. A leak that exposes the address of a function only leaks the address of other functions in that library, not every function in the process. But in a static executable all those libraries are also placed into the same .text section as the main program code, so a leak of any function address leaks all function addresses. In theory all functions, or more realistically groups of functions spanning page-size increments, could be dynamically located. The obvious way to achieve that would be to have multiple .text sections within a main executable or library. But off-hand I don't know if that's actually supported by ELF, or if so whether the standard tool chains and environments could easily support it.
snvzz 4y ago
It is possible to get your reproduceable build, 100% identical, and then random-relink it.

viraptor 4y ago

It's there a good link for the details? I'm guessing this does more than ASLR?

codesniperjoe 4y ago
Plain, simple and effective as always! The highly complex 'black magic' is 'sort --random' and (re-)link it all again. :) Makefile.relink: cc -o sshd `echo ${OBJS} | tr ' ' '\n' | sort -R` ${LDADD} ./sshd -V && install -o root -g wheel -m ${BINMODE} sshd /usr/sbin/sshd https://github.com/openbsd/src/commit/898412097f87ba70d4012f7a2e827f59719c659d#diff-cd8659908f01fc079dcdbc809d4b57f813b634420b0f7d8a3533258e23f122b6 https://github.com/openbsd/src/commit/898412097f87ba70d4012f...
- viraptor 4y ago
  Ah, so that will have some features of ASLR missing. Specifically, you can't do this on a read only root and it didn't randomise the stack location as far as I can tell?
  - kryptiskt 4y ago
    Wouldn't the stack location be set at runtime and given by the stack register at the entry point?
    - viraptor 4y ago
      I think I've got a better idea now. So openbsd has ASLR which affects data, code/library, and stack positions. Then this solution works on top of it by reordering symbols within the code. One thing I'm still not sure about is whether the kernel could theoretically do the same reordering at load time using relocatable symbols.
      - actionfromafar 4y ago
        Sounds like Amiga HUNK format with the HUNK_OVERLAY hunk. For instance various functions could be loaded anywhere in memory.

rfoo 4y ago

Does anyone know an actually-happened example case where a fine-grained ASLR (like the OpenBSD relink one) successfully mitigates or significantly hinders an exploit, and the usual ASLR doesn't? I'm curious because years ago the academic strongly pushes the FG ASLR story, then OpenBSD did kernel relinking, but I haven't heard any industry story on how effective this is.

nine_k 4y ago
How would you measure that? How would you notice a failed attempt at finding a hole, of stack smashing, etc, if it did not even result in a crash, or any other dysfunction? Not a rhetorical question.
- codetrotter 4y ago
  I suppose if someone found a technique in the wild that defeats regular ASLR (even if only sometimes), they could then test that same technique against fine-grained ASLR and evaluate if the FG ASLR was more effective at preventing exploitation.
  - TheCondor 4y ago
    Any classic buffer overflow/stackssmash can defeat ASLR, it just might take a long time to get lucky guessing addresses. Couldn’t we Monte Carlo this? Maybe take a known vulnerable exec, create a fuzzing attacker and run it both ways seeing how long it takes to get lucky a few times. The more secure version should take longer.
- insanitybit 4y ago
  There are whitehats constantly looking at this sort of thing. It's why we can say that KASLR is a huge waste of time - because it's both theoretically bogus and we have actual exploit devs saying that they don't care about it.
- rfoo 4y ago
  It's not about failed attempt at finding a hole, it's about failed attempt at actually making use of a hole (i.e. write an exploit) given a hole. (FG)ASLR is more of a "targeting at exploit instead of vulnerability" style mitigation.

nine_k 4y ago

I remember how back in MS DOS days polymorphic viruses first appeared, in an attempt to avoid detection by antivirus software (useful and essential back then). Now the tables have turned, and legitimate software has to become somehow polymorphic to thwart attacks by malware.

codesniperjoe 4y ago
Yes, the base idea is not that new. I store since years every GO based application I use as small (few kb) source code tree checkout only, no binary at all. At runtime the wrapper compiles a randomized individual one-time-temporary-uniq binary via garble [0]. [0] https://github.com/burrowers/garble https://github.com/burrowers/garble
- okl 4y ago
  How do you ensure that your compiler and libs are clean though?
  - codesniperjoe 4y ago
    The compiler (go) is part of a static read-only (compressed/in-memory) RootFS. Build on a air-gap build server, touching only signed/verified/reviewed code from git-offline mirror snaps. Go has no libs, all static. The resulting runtime only binaries are totally uniq/randomized and dependency free, straight from (signed) source code.

rwmj 4y ago

Takes some dedication to still be using CVS. Do they use another version control system to feed into CVS, or is CVS the tool they use directly?

WhyNotHugo 4y ago
They're working on GoT (game of trees), an OpenBSD-born client that interoperates with git repositories. AFAIK, they still interface with CVS directly, but I assume the expectation is to eventually transition to got. https://www.gameoftrees.org/ https://www.gameoftrees.org/
lolpython 4y ago
IIRC they pass around patches on email before committing to CVS. That workflow affords a bit more flexibility.
krylon 4y ago
They are working on a git replacement, got (Game Of Trees): http://gameoftrees.org/ http://gameoftrees.org/ So it looks like they are going to move on from CVS eventually.
SoftTalker 4y ago
There is a git mirror of the repo, and some devs use that for their own work. The project itself uses CVS for formal change commits.

theamk 4y ago

In order to enable relinking, they had to keep around all original .o and .a files, as well as a Makefile. Not a problem for OpenBSD but pretty unusual for binary Linux distros. I wonder if it is possible to make a relinker which only requires binary output -- so it could be easily incorporated into existing systems. One way I can think of is to keep relocation/original object information in the debug sections, so that one can reconstruct original object files and re-link them. But I am guessing this will not work with LTO though... Or maybe we can just make a bunch of debug sections and store input object/library files verbatim -- this will at least double the binary size, but will allow for easier relinking.

wolf550e 4y ago
I think Facebook BOLT relinks binaries: https://research.facebook.com/publications/bolt-a-practical-binary-optimizer-for-data-centers-and-beyond/ https://research.facebook.com/publications/bolt-a-practical-... https://groups.google.com/g/llvm-dev/c/ef3mKzAdJ7U/m/1shV64BYBAAJ?pli=1 https://groups.google.com/g/llvm-dev/c/ef3mKzAdJ7U/m/1shV64B...

planede 4y ago

If a squint hard then this is a custom dynamic loader for .o files with rudimentary ASLR (where all your entropy comes from the permutation of the .o files), that happens to cache to disk.

anonymousiam 4y ago

Dynamic re-linking is cool, but it can result in less-than-optimal executables. Sometimes it can be beneficial to optimize the link so most of the main thread stays in cache. Obviously this only really matters for CPU-intensive programs.

LinuxBender 4y ago

What impact will this have on anti-tampering software that looks for changes in executable checksums? Tripwire and OSSEC come to mind and both can report their findings to a centralized server. Do package manager integrity tests still work? I assume anyone here using BSD in a PCI environment have already figured something out. Some people also feed checksums into Splunk.

antx 4y ago
Very good point. As per Job snijder's own words [0], "the sshd binary becomes unique on every openbsd machine". So checksum-checking systems will indeed trip everytime. [0]: https://marc.info/?l=openbsd-tech&m=167388832715992&w=2 https://marc.info/?l=openbsd-tech&m=167388832715992&w=2