5 ms·

So we mark our super secret entropy memory area WIPEONFORK. Now we fork and our super secret entropy memory area feeding all RNG is all zeroes. We've implemente

So we mark our super secret entropy memory area WIPEONFORK. Now we fork and our super secret entropy memory area feeding all RNG is all zeroes. We've implemented a Sony RNG! So we still need to detect forks. Not sure this is entirely the silver bullet this makes it out to be. Maybe fork() is just a bad idea, for any number of reasons.

kevan 9y ago
From colmmacc's link: The option can be used to robustly reset a guard variable, and hence re-initialize an RNG, DRBG, PRF, or other generator where duplicate state across processes would be a security issue.
- cperciva 9y ago
  Isn't there a race condition here? What if the fork happens between when you check the flag and when you access the rng state?
  - infogulch 9y ago
    I had this exact same question. I guess you can make sure the code that checks and reads from the rng state can't fork in the middle of it (or call out to user code). But that only helps in the single-threaded case, what happens if you have another thread that is accessing it at the same time as the fork? Even if you check before and after, the initialized bool could be set again by the time you check after even though you just read a bunch of zeros. A rng-per-thread combined with initialization check before and after might cover all the cases I think...
    - revelation 9y ago
      From what I can tell in their s2n code, the canary was/is already TLS.
      - deleted 9y ago
        [deleted]
  - colmmacc 9y ago
    I'm still experimenting with the best approach. On Linux fork() clones a single thread's context, and since we know there's no call to fork() between the line of code checking the guard and the lines of code generating data, we don't need to worry about that so much (the RNGs are thread local). But re-entrancy might be a problem ... e.g. if some crazy application writer has called fork() from a signal handler, and that happened right between those lines. For a case like this, I think moving the guard to post-generation (but pre-use) and using an atomic CAS option to have the guard effect the return value of the generator, can do the trick at the cost of having to very occasionally throw generated data away. But I haven't tried it on all platforms.
    - jwilk 9y ago
      Crazy is the correct word here. glibc's fork() implementation is not async-signal-safe (even though POSIX currently says it should be), so you're going to have a bad time if you fork from a signal handler. https://sourceware.org/bugzilla/show_bug.cgi?id=4737 https://sourceware.org/bugzilla/show_bug.cgi?id=4737
    - cperciva 9y ago
      Sorry, I missed this comment when I brought the issue up below. (I was on my phone, which is sort of an excuse...) With a thread-local RNG (and a thread-local page which gets zeroed on fork) it sounds like you're probably best off extracting data from the RNG and then checking the flag to see if you need to go back and try again. The only cost of doing it this way is a very marginal extra cost whenever you need to regenerate entropy -- but you can think of that as just being a slight increase in the cost of the fork. The only other question which comes to mind is whether you need to worry about breakage on systems which have weaker-than-x86 cache coherency. I'm 99% certain that you're fine, but I'd prefer to be 100% certain. I assume you'd like to see this functionality added to other platforms as well?
  - deleted 9y ago
    [deleted]
  - tedunangst 9y ago
    How can that happen? The racing thread doesn't exist in the child.
    - cperciva 9y ago
      How about if the fork happens from a signal handler?
      - colmmacc 9y ago
        See my reply here ... https://news.ycombinator.com/item?id=15761490 https://news.ycombinator.com/item?id=15761490
      - tedunangst 9y ago
        Why does that matter? The parent will eventually return from the signal handler and resume execution. What does fork have to do with anything? arc4random (or whatever you want to call it) isn't marked signal safe.
        cperciva 9y ago
        The child will also eventually return from the signal handler and resume execution. In the middle of the PRNG, thus returning the same random numbers in both processes. I'm not sure what you mean by "arc4random isn't marked signal safe" -- you mean it can't be used in any process which receives signals?
bmm6o 9y ago
Wiping the page sets all of the data back to 0, just as it was on initial process launch. The keys are wiped, but so is your BOOL IsInitialized, so you would seed from hardware entropy the same way the parent process did when it saw that the generator wasn't initialized. [at least, that's my understanding]
- baby 9y ago
  Every time I read this kind of article I wish there was some snippet example.
agwa 9y ago
WIPEONFORK makes it possible to reliably detect forks. Without WIPEONFORK, it's impossible to reliably detect forks 100% of the time.
- revelation 9y ago
  Sounds like bad news for 99.9% of the world not running 4.14 and the messy backwards compatibility path they have to maintain forever. That part of the code is already screaming "goto fail": https://github.com/awslabs/s2n/blob/master/utils/s2n_random.c#L227 https://github.com/awslabs/s2n/blob/master/utils/s2n_random.... (And to be clear, I mean "goto fail" as in "I'm sure it's correct, but it's also very very difficult to tell")
  - cbhl 9y ago
    "Forever" is only ten or twenty years, right?
    - wbl 9y ago
      The IRS master tax database has been around since the 1960s on the same machine.
      - asveikau 9y ago
        Was trying to see how a 1960s machine is relevant to a discussion of brand new linux syscall behavior... Took me a while to see that the systems of today will look like the 60s do to us in ... The late 2060s. So if the 2067 tax database picks up an older kernel than this new madvise flag they are hosed, for some definition thereof (more like they need to be more careful when forking RNGs). Somehow that seems pretty fringe.
  - colmmacc 9y ago
    We can't change history, only improve the future. There are three levels of defense, precisely because until now there's been no way to robustly detect a fork(), but it will be nice to nail that for 4.14 and beyond and fleets can move surprisingly quickly to newer kernel versions and get the benefit of that. I think we have a high bar for a code readability and that the code isn't screaming goto fail. Though we do hate ifdef's and goto's and that file is one of the few places we have any, again precisely because of the ickyness of the challenge prior to 4.14. But if it helps reassure you, we also have a test case to make sure that the fork detection works: https://github.com/awslabs/s2n/blob/master/tests/unit/s2n_random_test.c https://github.com/awslabs/s2n/blob/master/tests/unit/s2n_ra... The test checks that a child and a parent really do produce different random streams.
colmmacc 9y ago
bmm6o's reply captures the approach; you WIPEONFORK a guard variable, and use the state of that variable to re-initialize. When we're writing a crypto library, like s2n, or OpenSSL, we don't get a choice to avoid fork() ... the application can fork() any time it wants and we just have to be able to deal with it. There have been a few approaches: * Always feed new entropy on every call (but this isn't always available) * Use getpid() to detect a fork() ... but this can break because some versions of libc cache the value of getpid(). Another problem is the grand-child problem, where an application can fork(), have have the child start a new process group, then the parent exits, then the child fork()s again ... in that situation there's some chance that the grandchild ends up with the same PID as the original process that started everything (and may have used the RNG). * To fix that, one clever approach that BoringSSL used for a while was to check both getpid() and getppid(), which signaficantly reduced the likelihood of occurrence but was still probabilistic. * All sorts of weird clone() flags, that things like language runtimes and virtual machines use, can bypass pthread_atfork and PID changes.
- revelation 9y ago
  What's wrong with setting it DONTFORK and handling the page fault in the process? Someone can still prevent you from correctly handling the fault in the new process, but at least that keeps the RNG in an obvious broken state. Right now, the <4.14 fallback seems strictly worse. (Disregarding that this is mostly academic, and that whoever is calling fork() while using your library is unlikely to have spent the same attention to detail when it comes to stuff like ephemeral keys..)
  - colmmacc 9y ago
    I couldn't make this work reliably ... the kinds of applications that call clone() directly and bypass the pthreads stuff are language runtimes and VMs that want to emulate their own lightweight processes. They also have their own page fault handling, and documentation says that behavior is undefined when these things start interacting recursively. It's very difficult for a library to exert control. At a same time these environments often use native crypto that's implemented in C, for performance, and so it all comes together. Of course the other mitigations present still work, and we're talking about a very very obscure and unlikely set of circumstances, but why leave even a tiny door open.
deleted 9y ago
[deleted]

kevan 9y ago

From colmmacc's link: The option can be used to robustly reset a guard variable, and hence re-initialize an RNG, DRBG, PRF, or other generator where duplicate state across processes would be a security issue.

cperciva 9y ago
Isn't there a race condition here? What if the fork happens between when you check the flag and when you access the rng state?
- infogulch 9y ago
  I had this exact same question. I guess you can make sure the code that checks and reads from the rng state can't fork in the middle of it (or call out to user code). But that only helps in the single-threaded case, what happens if you have another thread that is accessing it at the same time as the fork? Even if you check before and after, the initialized bool could be set again by the time you check after even though you just read a bunch of zeros. A rng-per-thread combined with initialization check before and after might cover all the cases I think...
  - revelation 9y ago
    From what I can tell in their s2n code, the canary was/is already TLS.
    - deleted 9y ago
      [deleted]
- colmmacc 9y ago
  I'm still experimenting with the best approach. On Linux fork() clones a single thread's context, and since we know there's no call to fork() between the line of code checking the guard and the lines of code generating data, we don't need to worry about that so much (the RNGs are thread local). But re-entrancy might be a problem ... e.g. if some crazy application writer has called fork() from a signal handler, and that happened right between those lines. For a case like this, I think moving the guard to post-generation (but pre-use) and using an atomic CAS option to have the guard effect the return value of the generator, can do the trick at the cost of having to very occasionally throw generated data away. But I haven't tried it on all platforms.
  - jwilk 9y ago
    Crazy is the correct word here. glibc's fork() implementation is not async-signal-safe (even though POSIX currently says it should be), so you're going to have a bad time if you fork from a signal handler. https://sourceware.org/bugzilla/show_bug.cgi?id=4737 https://sourceware.org/bugzilla/show_bug.cgi?id=4737
  - cperciva 9y ago
    Sorry, I missed this comment when I brought the issue up below. (I was on my phone, which is sort of an excuse...) With a thread-local RNG (and a thread-local page which gets zeroed on fork) it sounds like you're probably best off extracting data from the RNG and then checking the flag to see if you need to go back and try again. The only cost of doing it this way is a very marginal extra cost whenever you need to regenerate entropy -- but you can think of that as just being a slight increase in the cost of the fork. The only other question which comes to mind is whether you need to worry about breakage on systems which have weaker-than-x86 cache coherency. I'm 99% certain that you're fine, but I'd prefer to be 100% certain. I assume you'd like to see this functionality added to other platforms as well?
- deleted 9y ago
  [deleted]
- tedunangst 9y ago
  How can that happen? The racing thread doesn't exist in the child.
  - cperciva 9y ago
    How about if the fork happens from a signal handler?
    - colmmacc 9y ago
      See my reply here ... https://news.ycombinator.com/item?id=15761490 https://news.ycombinator.com/item?id=15761490
    - tedunangst 9y ago
      Why does that matter? The parent will eventually return from the signal handler and resume execution. What does fork have to do with anything? arc4random (or whatever you want to call it) isn't marked signal safe.
      - cperciva 9y ago
        The child will also eventually return from the signal handler and resume execution. In the middle of the PRNG, thus returning the same random numbers in both processes. I'm not sure what you mean by "arc4random isn't marked signal safe" -- you mean it can't be used in any process which receives signals?

bmm6o 9y ago

Wiping the page sets all of the data back to 0, just as it was on initial process launch. The keys are wiped, but so is your BOOL IsInitialized, so you would seed from hardware entropy the same way the parent process did when it saw that the generator wasn't initialized. [at least, that's my understanding]

baby 9y ago
Every time I read this kind of article I wish there was some snippet example.

agwa 9y ago

WIPEONFORK makes it possible to reliably detect forks. Without WIPEONFORK, it's impossible to reliably detect forks 100% of the time.

revelation 9y ago
Sounds like bad news for 99.9% of the world not running 4.14 and the messy backwards compatibility path they have to maintain forever. That part of the code is already screaming "goto fail": https://github.com/awslabs/s2n/blob/master/utils/s2n_random.c#L227 https://github.com/awslabs/s2n/blob/master/utils/s2n_random.... (And to be clear, I mean "goto fail" as in "I'm sure it's correct, but it's also very very difficult to tell")
- cbhl 9y ago
  "Forever" is only ten or twenty years, right?
  - wbl 9y ago
    The IRS master tax database has been around since the 1960s on the same machine.
    - asveikau 9y ago
      Was trying to see how a 1960s machine is relevant to a discussion of brand new linux syscall behavior... Took me a while to see that the systems of today will look like the 60s do to us in ... The late 2060s. So if the 2067 tax database picks up an older kernel than this new madvise flag they are hosed, for some definition thereof (more like they need to be more careful when forking RNGs). Somehow that seems pretty fringe.
- colmmacc 9y ago
  We can't change history, only improve the future. There are three levels of defense, precisely because until now there's been no way to robustly detect a fork(), but it will be nice to nail that for 4.14 and beyond and fleets can move surprisingly quickly to newer kernel versions and get the benefit of that. I think we have a high bar for a code readability and that the code isn't screaming goto fail. Though we do hate ifdef's and goto's and that file is one of the few places we have any, again precisely because of the ickyness of the challenge prior to 4.14. But if it helps reassure you, we also have a test case to make sure that the fork detection works: https://github.com/awslabs/s2n/blob/master/tests/unit/s2n_random_test.c https://github.com/awslabs/s2n/blob/master/tests/unit/s2n_ra... The test checks that a child and a parent really do produce different random streams.

colmmacc 9y ago

bmm6o's reply captures the approach; you WIPEONFORK a guard variable, and use the state of that variable to re-initialize. When we're writing a crypto library, like s2n, or OpenSSL, we don't get a choice to avoid fork() ... the application can fork() any time it wants and we just have to be able to deal with it. There have been a few approaches: * Always feed new entropy on every call (but this isn't always available) * Use getpid() to detect a fork() ... but this can break because some versions of libc cache the value of getpid(). Another problem is the grand-child problem, where an application can fork(), have have the child start a new process group, then the parent exits, then the child fork()s again ... in that situation there's some chance that the grandchild ends up with the same PID as the original process that started everything (and may have used the RNG). * To fix that, one clever approach that BoringSSL used for a while was to check both getpid() and getppid(), which signaficantly reduced the likelihood of occurrence but was still probabilistic. * All sorts of weird clone() flags, that things like language runtimes and virtual machines use, can bypass pthread_atfork and PID changes.

revelation 9y ago
What's wrong with setting it DONTFORK and handling the page fault in the process? Someone can still prevent you from correctly handling the fault in the new process, but at least that keeps the RNG in an obvious broken state. Right now, the <4.14 fallback seems strictly worse. (Disregarding that this is mostly academic, and that whoever is calling fork() while using your library is unlikely to have spent the same attention to detail when it comes to stuff like ephemeral keys..)
- colmmacc 9y ago
  I couldn't make this work reliably ... the kinds of applications that call clone() directly and bypass the pthreads stuff are language runtimes and VMs that want to emulate their own lightweight processes. They also have their own page fault handling, and documentation says that behavior is undefined when these things start interacting recursively. It's very difficult for a library to exert control. At a same time these environments often use native crypto that's implemented in C, for performance, and so it all comes together. Of course the other mitigations present still work, and we're talking about a very very obscure and unlikely set of circumstances, but why leave even a tiny door open.

deleted 9y ago

[deleted]