9 ms·

Yet another proof for the following: 1. It's reasonable to claim that amd64 (x86_64) is more secure than x86. x86_64 has larger address space, thus higher ASLR

Yet another proof for the following: 1. It's reasonable to claim that amd64 (x86_64) is more secure than x86. x86_64 has larger address space, thus higher ASLR entropy. The exploit needs 10 minutes to crack ASLR on x86, but 70 minutes on amd64. If some alert systems have been deploy on the server (attacks need to keep crashing systemd-journald in this process), it buys time. In other cases, it makes exploitation infeasible. 2. CFLAGS hardening works, in addition to ASLR, it's the last line of defense for all C programs. As long as there are still C programs running, patching all memory corruption bugs is impossible. Using mitigation techniques and sandbox-based isolation are the only two ways to limit the damage. All hardening flags should be turned on by all distributions, unless there is a special reason. Fedora turned "-fstack-clash-protection" on since Fedora 28 (https://fedoraproject.org/wiki/Changes/HardeningFlags28 https://fedoraproject.org/wiki/Changes/HardeningFlags28). If you are releasing a C program on Linux, please consider the following, -D_FORTIFY_SOURCE=2 glibc hardening -Wp,-D_GLIBCXX_ASSERTIONS glibc++ hardening -fstack-protector-strong stack smash protection -fstack-clash-protection stack clash protection -fPIE -pie better ASLR protection -Wl,-z,noexecstack don't allow code on stack -Wl,-z,relro ELF hardening -Wl,-z,now ELF hardening Major Linux distributions, including Fedora, Debian, Arch Linux, openSUSE are already doing it. Similarly, Firefox and Chromium are using many of these flags too. Unfortunately, Debian did not use `-fstack-clash-protection` and got hit by the exploit, because it was only added since GCC 8. For a more comprehensive review, check * Recommended compiler and linker flags for GCC: https://developers.redhat.com/blog/2018/03/21/compiler-and-linker-flags-gcc/ https://developers.redhat.com/blog/2018/03/21/compiler-and-l... * Debian Hardening https://wiki.debian.org/Hardening https://wiki.debian.org/Hardening

devereaux 8y ago
What is the performance overhead for using these flags?
- segfaultbuserr 8y ago
  It's very hard to find solid numbers, because these flags are introduced and popularized at different times. But I'll try to give you my best summary. Long shory short, these flags are considered essentially and most web browsers and distributions have already enabled them without problems. Let's see... -D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS These two are considered very cheap. It enables some sanity checks for buffer overflow in glibc and glibc++, respectively. See https://access.redhat.com/blogs/766093/posts/3606481 https://access.redhat.com/blogs/766093/posts/3606481 and -fstack-protector-strong Also known as stack canary. The earliest version is simply called "-fstack-protector", but the coverage was bad (only "protects" <2% functions), there was also "-fstack-protector-full", adding checks to ALL functions, but the performance overhead is unacceptable. The "-strong" variant is an upgraded version, it uses a heuristic when determining if a function needs stack protectors, provides much better code coverage and a tradeoff between security and performance. It's relatively expensive, but is considered essential to prevent attackers from smashing the stack, and is enabled in most web browser engines. See https://lwn.net/Articles/584225/ https://lwn.net/Articles/584225/ -fstack-clash-protection stack clash protection Introduced in GCC 8.0 after the Stack Clash attack was discovered, see https://securingsoftware.blogspot.com/2017/12/stack-clash-protection-in-gcc.html https://securingsoftware.blogspot.com/2017/12/stack-clash-pr... -fPIE -pie better ASLR protection PIE is Position Independent-Executable, Address Space Layout Randomization needs it to achieve its full level of protection. It has been around for a long time, but only saw a mass adoption recently, possibly due to higher performance impact on 32-bit x86, but favorable on 64-bit systems. Red Hat has a performance analysis. * https://access.redhat.com/blogs/766093/posts/1975803 https://access.redhat.com/blogs/766093/posts/1975803 Arch Linux has also benchmarked its performance impact before deciding to enable it. * https://github.com/pid1/test-sec-flags/wiki https://github.com/pid1/test-sec-flags/wiki -Wl,-z,noexecstack don't allow code on stack It just disables the executable stack, no overhead. And normally the stack is NOT executable, unless under some special conditions like linking assembly code, due to bad default settings in GNU toolchains for historical reasons. This explicitly tells the linker: don't allow an executable stack, no matter what. -Wl,-z,relro ELF hardening -Wl,-z,now ELF hardening It marks the GOT table as readonly, and force the system to resolve all function references to shared libraries at startup time, instead of runtime. It has some impact on startup time, but mostly negligible. See https://blog.osiris.cyber.nyu.edu/exploitation%20mitigation%20techniques/exploitation%20techniques/2011/06/02/relro-relocation-read-only/ https://blog.osiris.cyber.nyu.edu/exploitation%20mitigation%....
wahern 8y ago
FWIW, the stack vulnerabilities here aren't just a C problem. Most languages, including every language relying on LLVM and GCC until the most recent versions, failed to perform stack probing. I hesitate to call stack probing "hardening". IMO it's better understood as a failure by compilers to emit proper code in the first place, and it's been a glaringly obvious deficiency for years if not decades.
- ajross 8y ago
  These are alloa overflows. In the first one, the code tries to put the entire (attacker-controlled) command line on its stack. Compilers can't detect or prevent that, though with ABI support they may be able to detect it before failure
  - wahern 8y ago
    alloca isn't part of C, it's a compiler intrinsic with analogs in other languages. In LLVM it even has it's own instruction. C99 and Fortran also provide dynamically-sized array constructs which historically used alloca internally, and which in any event have similar issues.[1] An alloca overflow is from a language perspective little different than a recursive function blowing the stack. The proper job of the compiler is to make sure that the generated code doesn't blow the stack in a way that overwrites random memory, whether because of alloca, a large stack-allocated object (of static or dynamic size), or recursion. It's true that blowing the stack in C is undefined whereas in a language like Rust it's supposed to terminate the program.[2] But that's beside the point because Rust didn't actually implement stack probing either and therefore was just as susceptible to these vulnerabilities. The only sane, acceptable behavior for the compiler is to generate stack probes for any stack allocations that may exceed the page size; not only for alloca, but even for regular, non-array objects which happen to be large. Both programmers and compilers for complex languages like C++, Rust, and Swift aggressively attempt to stack allocate as much as possible, which makes the issue even more acute for those languages. As others have hinted, both alloca and dynamic arrays have been frowned upon in C for a long time (C99 added dynamic arrays principally for the Fortran crowd, who typically consume trusted data, anyhow). The fact that most of the stack smash and stack clash exploits you see are for C is a consequence of most popular software libraries being written in C, and researchers being most familiar with developing PoCs for C-based codebases.[3] [1] Crucially, C99 dynamic arrays have block-scoped lifetimes whereas alloca allocations have function-scope lifetimes. Meaning calling alloca in a loop is doubly crazy. Early C99 implementations that simply reused the pre-existing alloca intrinsic were buggy. C99 compound literals were also buggy for several years for somewhat similar reasons. [2] In truth it's de facto undefined in most languages, because most language designers and compiler authors have historically been content to ignore the issue. [3] To be clear, I'm not saying that C only seems error prone because its popular. I'm only speaking to the particular issue of stack allocation overflow.
lmm 8y ago
"Proof" suggests a level of absolute confidence that this example certainly does not give. > The exploit needs 10 minutes to crack ASLR on x86, but 70 minutes on amd64. Is there any realistic threat model under which the difference between 10 minutes and 70 minutes is the difference between "insecure" and "secure"? > Using mitigation techniques and sandbox-based isolation are the only two ways to limit the damage. I'm not at all convinced that mitigation techniques represent a real improvement in security, because by definition a mitigation technique is not backed by a solid model. If you're letting an attacker control the modification of memory that your security model assumes isn't modifiable, how confident can you be that ad-hoc mitigations for all the ways you could think of to exploit that cover all the possible ways to exploit that? E.g. I can remember a time when ASLR was touted as a solution to C's endemic security vulnerabilities; now cracking ASLR as part of vulnerability exploitation is routine, as seen here. Mitigations appear to give a security improvement because an app with mitigations is no longer the low-hanging fruit, but I suspect this is a case of "you don't have to outrun the bear": as long as there are C programs without mitigations, attackers will go after those first. That's different from saying that mitigations provide substantial protection.
- FakeComments 8y ago
  The hands-on-keyboard SLA for a lot of on-calls is 30 minutes. So in an “attack was detected, break all the glass” scenario, the difference between 10 and 70 minutes is sufficient to allow human operators to render the attack moot by offlining its target, while the attackers are still trying to break through API servers. At both big corps I’ve been at, the incident response plan for an exfiltration attack on customer data was invalidate DB creds and take the system down ourselves. Better to be out of service than lose custody of customer data.
- dTal 8y ago
  >Is there any realistic threat model under which the difference between 10 minutes and 70 minutes is the difference between "insecure" and "secure"? How about an intrusion detection system that flags up a human response? 10 minutes is hardly any time at all to respond, an hour gives you a chance to roll out of bed.
vectorEQ 8y ago
you can patch binaries to fix memory corruption issues :D compiler flags won't help on already compiled softwares, and if they are yet to be compiled, the sources could be improved... that being said i'm not against these compiler things, just don't lean on those to fix your broken code... which is what a lot of developers seem to imply when they say this is the way to fix things.
- segfaultbuserr 8y ago
  Sure, you need both. I said, > CFLAGS hardening works, in addition to ASLR, it's the last line of defense for all C programs. [...] Using mitigation techniques and sandbox-based isolation are the only two ways to limit the damage. Mitigations are here to limit the damage, not to eliminate vulnerabilities and exploits. Looking for bugs and improving the code for security for as hard as you can, and in case you missed one (you certainly will), CFLAGS and ASLR is your last line of defense, and you can only hope for the best that it is able to buy you some time to patch it, before a better exploit appears...
fao_ 8y ago
Hey, you will want to add: -fcf-protection=full ROP protection to that list :) (see: https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.h...)
loeg 8y ago
I would add: alloca() should be considered harmful and warned out of existence, like gets(). If your malloc() implementation is slow, fix that! tcmalloc and jemalloc have been around for a long time.
- dane-pgp 8y ago
  I'm reminded that the custom memory allocation system in OpenSSL was cited as being potentially implicated in the Heartbleed bug: https://www.darkreading.com/vulnerabilities---threats/did-a-faulty-memory-feature-lead-to-heartbleed/d/d-id/1204505 https://www.darkreading.com/vulnerabilities---threats/did-a-... or it at least made it harder for the code to be analysed by automated tools: https://dwheeler.com/essays/heartbleed.html#basic-fuzzers https://dwheeler.com/essays/heartbleed.html#basic-fuzzers
  - loeg 8y ago
    I'm not sure if you're trying to compare OpenSSL's FIFO malloc cache to tcmalloc or jemalloc, or not. They have nothing in common. I was not proposing adding a crappy linked list cache in front of malloc(3), but instead linking a better malloc(3) implementation to your program (and gave specific, high-quality examples).

devereaux 8y ago

What is the performance overhead for using these flags?

segfaultbuserr 8y ago
It's very hard to find solid numbers, because these flags are introduced and popularized at different times. But I'll try to give you my best summary. Long shory short, these flags are considered essentially and most web browsers and distributions have already enabled them without problems. Let's see... -D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS These two are considered very cheap. It enables some sanity checks for buffer overflow in glibc and glibc++, respectively. See https://access.redhat.com/blogs/766093/posts/3606481 https://access.redhat.com/blogs/766093/posts/3606481 and -fstack-protector-strong Also known as stack canary. The earliest version is simply called "-fstack-protector", but the coverage was bad (only "protects" <2% functions), there was also "-fstack-protector-full", adding checks to ALL functions, but the performance overhead is unacceptable. The "-strong" variant is an upgraded version, it uses a heuristic when determining if a function needs stack protectors, provides much better code coverage and a tradeoff between security and performance. It's relatively expensive, but is considered essential to prevent attackers from smashing the stack, and is enabled in most web browser engines. See https://lwn.net/Articles/584225/ https://lwn.net/Articles/584225/ -fstack-clash-protection stack clash protection Introduced in GCC 8.0 after the Stack Clash attack was discovered, see https://securingsoftware.blogspot.com/2017/12/stack-clash-protection-in-gcc.html https://securingsoftware.blogspot.com/2017/12/stack-clash-pr... -fPIE -pie better ASLR protection PIE is Position Independent-Executable, Address Space Layout Randomization needs it to achieve its full level of protection. It has been around for a long time, but only saw a mass adoption recently, possibly due to higher performance impact on 32-bit x86, but favorable on 64-bit systems. Red Hat has a performance analysis. * https://access.redhat.com/blogs/766093/posts/1975803 https://access.redhat.com/blogs/766093/posts/1975803 Arch Linux has also benchmarked its performance impact before deciding to enable it. * https://github.com/pid1/test-sec-flags/wiki https://github.com/pid1/test-sec-flags/wiki -Wl,-z,noexecstack don't allow code on stack It just disables the executable stack, no overhead. And normally the stack is NOT executable, unless under some special conditions like linking assembly code, due to bad default settings in GNU toolchains for historical reasons. This explicitly tells the linker: don't allow an executable stack, no matter what. -Wl,-z,relro ELF hardening -Wl,-z,now ELF hardening It marks the GOT table as readonly, and force the system to resolve all function references to shared libraries at startup time, instead of runtime. It has some impact on startup time, but mostly negligible. See https://blog.osiris.cyber.nyu.edu/exploitation%20mitigation%20techniques/exploitation%20techniques/2011/06/02/relro-relocation-read-only/ https://blog.osiris.cyber.nyu.edu/exploitation%20mitigation%....

wahern 8y ago

FWIW, the stack vulnerabilities here aren't just a C problem. Most languages, including every language relying on LLVM and GCC until the most recent versions, failed to perform stack probing. I hesitate to call stack probing "hardening". IMO it's better understood as a failure by compilers to emit proper code in the first place, and it's been a glaringly obvious deficiency for years if not decades.

ajross 8y ago
These are alloa overflows. In the first one, the code tries to put the entire (attacker-controlled) command line on its stack. Compilers can't detect or prevent that, though with ABI support they may be able to detect it before failure
- wahern 8y ago
  alloca isn't part of C, it's a compiler intrinsic with analogs in other languages. In LLVM it even has it's own instruction. C99 and Fortran also provide dynamically-sized array constructs which historically used alloca internally, and which in any event have similar issues.[1] An alloca overflow is from a language perspective little different than a recursive function blowing the stack. The proper job of the compiler is to make sure that the generated code doesn't blow the stack in a way that overwrites random memory, whether because of alloca, a large stack-allocated object (of static or dynamic size), or recursion. It's true that blowing the stack in C is undefined whereas in a language like Rust it's supposed to terminate the program.[2] But that's beside the point because Rust didn't actually implement stack probing either and therefore was just as susceptible to these vulnerabilities. The only sane, acceptable behavior for the compiler is to generate stack probes for any stack allocations that may exceed the page size; not only for alloca, but even for regular, non-array objects which happen to be large. Both programmers and compilers for complex languages like C++, Rust, and Swift aggressively attempt to stack allocate as much as possible, which makes the issue even more acute for those languages. As others have hinted, both alloca and dynamic arrays have been frowned upon in C for a long time (C99 added dynamic arrays principally for the Fortran crowd, who typically consume trusted data, anyhow). The fact that most of the stack smash and stack clash exploits you see are for C is a consequence of most popular software libraries being written in C, and researchers being most familiar with developing PoCs for C-based codebases.[3] [1] Crucially, C99 dynamic arrays have block-scoped lifetimes whereas alloca allocations have function-scope lifetimes. Meaning calling alloca in a loop is doubly crazy. Early C99 implementations that simply reused the pre-existing alloca intrinsic were buggy. C99 compound literals were also buggy for several years for somewhat similar reasons. [2] In truth it's de facto undefined in most languages, because most language designers and compiler authors have historically been content to ignore the issue. [3] To be clear, I'm not saying that C only seems error prone because its popular. I'm only speaking to the particular issue of stack allocation overflow.

lmm 8y ago

"Proof" suggests a level of absolute confidence that this example certainly does not give. > The exploit needs 10 minutes to crack ASLR on x86, but 70 minutes on amd64. Is there any realistic threat model under which the difference between 10 minutes and 70 minutes is the difference between "insecure" and "secure"? > Using mitigation techniques and sandbox-based isolation are the only two ways to limit the damage. I'm not at all convinced that mitigation techniques represent a real improvement in security, because by definition a mitigation technique is not backed by a solid model. If you're letting an attacker control the modification of memory that your security model assumes isn't modifiable, how confident can you be that ad-hoc mitigations for all the ways you could think of to exploit that cover all the possible ways to exploit that? E.g. I can remember a time when ASLR was touted as a solution to C's endemic security vulnerabilities; now cracking ASLR as part of vulnerability exploitation is routine, as seen here. Mitigations appear to give a security improvement because an app with mitigations is no longer the low-hanging fruit, but I suspect this is a case of "you don't have to outrun the bear": as long as there are C programs without mitigations, attackers will go after those first. That's different from saying that mitigations provide substantial protection.

FakeComments 8y ago
The hands-on-keyboard SLA for a lot of on-calls is 30 minutes. So in an “attack was detected, break all the glass” scenario, the difference between 10 and 70 minutes is sufficient to allow human operators to render the attack moot by offlining its target, while the attackers are still trying to break through API servers. At both big corps I’ve been at, the incident response plan for an exfiltration attack on customer data was invalidate DB creds and take the system down ourselves. Better to be out of service than lose custody of customer data.
dTal 8y ago
>Is there any realistic threat model under which the difference between 10 minutes and 70 minutes is the difference between "insecure" and "secure"? How about an intrusion detection system that flags up a human response? 10 minutes is hardly any time at all to respond, an hour gives you a chance to roll out of bed.

vectorEQ 8y ago

you can patch binaries to fix memory corruption issues :D compiler flags won't help on already compiled softwares, and if they are yet to be compiled, the sources could be improved... that being said i'm not against these compiler things, just don't lean on those to fix your broken code... which is what a lot of developers seem to imply when they say this is the way to fix things.

segfaultbuserr 8y ago
Sure, you need both. I said, > CFLAGS hardening works, in addition to ASLR, it's the last line of defense for all C programs. [...] Using mitigation techniques and sandbox-based isolation are the only two ways to limit the damage. Mitigations are here to limit the damage, not to eliminate vulnerabilities and exploits. Looking for bugs and improving the code for security for as hard as you can, and in case you missed one (you certainly will), CFLAGS and ASLR is your last line of defense, and you can only hope for the best that it is able to buy you some time to patch it, before a better exploit appears...

fao_ 8y ago

Hey, you will want to add: -fcf-protection=full ROP protection to that list :) (see: https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.h...)

loeg 8y ago

I would add: alloca() should be considered harmful and warned out of existence, like gets(). If your malloc() implementation is slow, fix that! tcmalloc and jemalloc have been around for a long time.

dane-pgp 8y ago
I'm reminded that the custom memory allocation system in OpenSSL was cited as being potentially implicated in the Heartbleed bug: https://www.darkreading.com/vulnerabilities---threats/did-a-faulty-memory-feature-lead-to-heartbleed/d/d-id/1204505 https://www.darkreading.com/vulnerabilities---threats/did-a-... or it at least made it harder for the code to be analysed by automated tools: https://dwheeler.com/essays/heartbleed.html#basic-fuzzers https://dwheeler.com/essays/heartbleed.html#basic-fuzzers
- loeg 8y ago
  I'm not sure if you're trying to compare OpenSSL's FIFO malloc cache to tcmalloc or jemalloc, or not. They have nothing in common. I was not proposing adding a crappy linked list cache in front of malloc(3), but instead linking a better malloc(3) implementation to your program (and gave specific, high-quality examples).