8 ms·

Better Random Number Generation for OpenSSL, Libc, and Linux Mainline

265 points by gbrown_ 9y ago

colmmacc 9y ago
If you work on crypto, or any application where sensitive data is handled, then this short post on the WIPEONFORK option may be interesting: http://www.metzdowd.com/pipermail/cryptography/2017-November/033133.html http://www.metzdowd.com/pipermail/cryptography/2017-November...
- throwaway613834 9y ago
  How does this work with heap allocation? When you free memory and then allocate it again and get back the same address range, wouldn't the flag still apply to the new block?
  - colmmacc 9y ago
    madvise() works at the page level and if you're using madvise(), then you're expected to be using mmap() to allocate pages explicitly. It would be unusual and a bad idea to start madvise()ing memory that's being handled by an allocator (like malloc). Of course you could also write a security focused heap allocator that uses these madvise() options under the hood.
VeejayRampay 9y ago
Glad to see some open source work benefiting everyone, especially coming from the likes of Amazon.
revelation 9y ago
So we mark our super secret entropy memory area WIPEONFORK. Now we fork and our super secret entropy memory area feeding all RNG is all zeroes. We've implemented a Sony RNG! So we still need to detect forks. Not sure this is entirely the silver bullet this makes it out to be. Maybe fork() is just a bad idea, for any number of reasons.
- kevan 9y ago
  From colmmacc's link: The option can be used to robustly reset a guard variable, and hence re-initialize an RNG, DRBG, PRF, or other generator where duplicate state across processes would be a security issue.
  - cperciva 9y ago
    Isn't there a race condition here? What if the fork happens between when you check the flag and when you access the rng state?
    - infogulch 9y ago
      I had this exact same question. I guess you can make sure the code that checks and reads from the rng state can't fork in the middle of it (or call out to user code). But that only helps in the single-threaded case, what happens if you have another thread that is accessing it at the same time as the fork? Even if you check before and after, the initialized bool could be set again by the time you check after even though you just read a bunch of zeros. A rng-per-thread combined with initialization check before and after might cover all the cases I think...
      - revelation 9y ago
        From what I can tell in their s2n code, the canary was/is already TLS.
        deleted 9y ago
        [deleted]
    - colmmacc 9y ago
      I'm still experimenting with the best approach. On Linux fork() clones a single thread's context, and since we know there's no call to fork() between the line of code checking the guard and the lines of code generating data, we don't need to worry about that so much (the RNGs are thread local). But re-entrancy might be a problem ... e.g. if some crazy application writer has called fork() from a signal handler, and that happened right between those lines. For a case like this, I think moving the guard to post-generation (but pre-use) and using an atomic CAS option to have the guard effect the return value of the generator, can do the trick at the cost of having to very occasionally throw generated data away. But I haven't tried it on all platforms.
deleted 9y ago
[deleted]
jedberg 9y ago
Before I clicked I was expecting this to be a "random numbers as a service" announcement, based on lava lamps or something. And yes I know CloudFlare did the lava lamps thing, like a bunch of other people did it before them. But it would be totally inline with Amazon to have a random numbers as a service service to compete with random.org.
- colmmacc 9y ago
  http://docs.aws.amazon.com/cli/latest/reference/kms/generate-random.html http://docs.aws.amazon.com/cli/latest/reference/kms/generate... :)
  - jedberg 9y ago
    Right, but that doesn't use lava lamps or background atmospheric disturbances. :) It said "better random numbers" so I was hoping for something better than KMS random numbers.
    - tankenmate 9y ago
      I think the real value of this software is the fact that it has been proven to perform as designed, both theoretically and in implementation.
- bjoli 9y ago
  I have always wondered why they aren't using plasma globes instead of lava lamps. Not only would it give a higher bandwidth, but it would also be a lot more energy efficient.
bogomipz 9y ago
I had a question about this passage: >" Firstly, as mentioned, s2n uses the so-called “Prediction resistant” mode, which means that s2n feeds the random number generator with more entropy (from hardware) on every call, as long as the right hardware is available" Would this be the RDRAND instruction or something else?
- colmmacc 9y ago
  We use RDRAND if it's available, and we also have our own hardware entropy generators in some systems. In either case, we use this entropy as additional input that provides prediction resistance, and it's fed into the AES_CTR generator. Here's a picture from the blog post: https://d2908q01vomqb2.cloudfront.net/ca3512f4dfa95a03169c5a670a4c91a19b3077b4/2017/11/22/s2ndrbg-1024x507.png https://d2908q01vomqb2.cloudfront.net/ca3512f4dfa95a03169c5a... Basically when the DRBG is initialized we start with /dev/urandom and the personalization string. That sets up the initial state, and then hardware entropy is "mixed in" every time the generator is used. Because the mixing in happens before AES is applied, that means that even if either stream of entropy is biased, AES will randomize it anyway (or else AES is a broken cipher!). That means that even if you controlled both inputs, it would still require at least as much computational work as it takes to compute AES to be able to generate a predictable output. That's a nice property of the DRBG construction.
  - bogomipz 9y ago
    Thanks for the clarification and detailed response. Cheers!
Nicholas7900 9y ago
So many young Hackers have been asking for the easiest way to hack an email, Do you know how many email services there are, how many kinds of mail servers, how many forms of custom security? In my time, I've hacked numerous email servers and services, usually incidentally to compromising the rest of a company or entity. It's always been different. Sometimes it´s as simple as replacing a username in a URL. Other times I've had to compromise the entire support system for an email service. It varies. It takes imagination. There's no formula. There's no ¨teaching to hack email¨. There's just understanding how systems work and learning to extrapolate. I will reluctantly say that a public favorite seems to be guessing ¨secret answers¨ to ¨secret questions¨ needed to reset passwords on some systems (For Guccifer, hacking email is easy — prison is hard). I never personally used this technique, since it felt like cheating. It's hard to explain why. But certain passwords, once discovered, made sense for certain people (Ih8K1ds for an extremely unpleasant lady at [redacted],SPQR for a CIA officer [on a system with very low password standards]).So many people always think i only work for big companies, that i dont work for private individuals, For urgent requests for professional Hack Advice Or Hack Services, You can contact me directly via compositehacks@gmail.com. I'm also trying out a new beta ticketing system, which can be accessed at "Composite Hacks & Co" So many people always think i only work for big companies, that i dont work for private individuals, here is my mail """"""compositehacks@gmail.com,""""" feel free to contact me for any hacking services you need, i'll do your job professionally. So many people always ask me if i'm a Hacker or a Cracker, as a matter of fact, I don't really care. You can consider me whatever you like - it's entirely subjective. I've been called a white hat, gray hat, and black hat, depending on the attitude and perspective of the writer, and it's not my place to insist that one or the other is correct. I encourage readers to reach conclusions from their own mind and feelings, not arbitrary segregation. I do what I do, and leave it to others to document it - or not - in the way that suits them best. Distinctions like these work for role playing games, and it's kind of like asking "Is Adrian Lamo True Neutral?" - an interesting intellectual exercise, but with no particular relevance to reality. So many people always think i only work for big companies, that i dont work for private individuals, For urgent requests for professional Hack Advice Or Hack Services, You can contact me directly via """""compositehacks@gmail.com"""". I'm also trying out a new beta ticketing system, which can be accessed at "Composite Hacks & Co"
jabl 9y ago
> Recently, both the OpenSSL and glibc projects have been looking to replace their random number generators. They, too, are going with AES_CTR_DRBG, based in some part on the work in s2n and the availability of formal verifications that can be applied to code. That’s pretty sweet. I tried to search for this csprng in glibc, but I wasn't able to find anything. Can anyone provide more info?
NelsonMinar 9y ago
This bit is an aside but it's interesting to me: "Earlier this year, with Galois, we completed a proof of our implementation of AES_CTR_DRBG, and formally verified that our code in s2n is equivalent to the written specification." That's amazing! I think it may be the first bit of provably correct code that I'm likely to actually use in practice.

265 points by gbrown_ 9y ago

If you work on crypto, or any application where sensitive data is handled, then this short post on the WIPEONFORK option may be interesting: http://www.metzdowd.com/pipermail/cryptography/2017-November/033133.html http://www.metzdowd.com/pipermail/cryptography/2017-November...

throwaway613834 9y ago
How does this work with heap allocation? When you free memory and then allocate it again and get back the same address range, wouldn't the flag still apply to the new block?
- colmmacc 9y ago
  madvise() works at the page level and if you're using madvise(), then you're expected to be using mmap() to allocate pages explicitly. It would be unusual and a bad idea to start madvise()ing memory that's being handled by an allocator (like malloc). Of course you could also write a security focused heap allocator that uses these madvise() options under the hood.

VeejayRampay 9y ago

Glad to see some open source work benefiting everyone, especially coming from the likes of Amazon.

revelation 9y ago

So we mark our super secret entropy memory area WIPEONFORK. Now we fork and our super secret entropy memory area feeding all RNG is all zeroes. We've implemented a Sony RNG! So we still need to detect forks. Not sure this is entirely the silver bullet this makes it out to be. Maybe fork() is just a bad idea, for any number of reasons.

kevan 9y ago
From colmmacc's link: The option can be used to robustly reset a guard variable, and hence re-initialize an RNG, DRBG, PRF, or other generator where duplicate state across processes would be a security issue.
- cperciva 9y ago
  Isn't there a race condition here? What if the fork happens between when you check the flag and when you access the rng state?
  - infogulch 9y ago
    I had this exact same question. I guess you can make sure the code that checks and reads from the rng state can't fork in the middle of it (or call out to user code). But that only helps in the single-threaded case, what happens if you have another thread that is accessing it at the same time as the fork? Even if you check before and after, the initialized bool could be set again by the time you check after even though you just read a bunch of zeros. A rng-per-thread combined with initialization check before and after might cover all the cases I think...
    - revelation 9y ago
      From what I can tell in their s2n code, the canary was/is already TLS.
      - deleted 9y ago
        [deleted]
  - colmmacc 9y ago
    I'm still experimenting with the best approach. On Linux fork() clones a single thread's context, and since we know there's no call to fork() between the line of code checking the guard and the lines of code generating data, we don't need to worry about that so much (the RNGs are thread local). But re-entrancy might be a problem ... e.g. if some crazy application writer has called fork() from a signal handler, and that happened right between those lines. For a case like this, I think moving the guard to post-generation (but pre-use) and using an atomic CAS option to have the guard effect the return value of the generator, can do the trick at the cost of having to very occasionally throw generated data away. But I haven't tried it on all platforms.

deleted 9y ago

[deleted]

jedberg 9y ago

Before I clicked I was expecting this to be a "random numbers as a service" announcement, based on lava lamps or something. And yes I know CloudFlare did the lava lamps thing, like a bunch of other people did it before them. But it would be totally inline with Amazon to have a random numbers as a service service to compete with random.org.

colmmacc 9y ago
http://docs.aws.amazon.com/cli/latest/reference/kms/generate-random.html http://docs.aws.amazon.com/cli/latest/reference/kms/generate... :)
- jedberg 9y ago
  Right, but that doesn't use lava lamps or background atmospheric disturbances. :) It said "better random numbers" so I was hoping for something better than KMS random numbers.
  - tankenmate 9y ago
    I think the real value of this software is the fact that it has been proven to perform as designed, both theoretically and in implementation.
bjoli 9y ago
I have always wondered why they aren't using plasma globes instead of lava lamps. Not only would it give a higher bandwidth, but it would also be a lot more energy efficient.

bogomipz 9y ago

I had a question about this passage: >" Firstly, as mentioned, s2n uses the so-called “Prediction resistant” mode, which means that s2n feeds the random number generator with more entropy (from hardware) on every call, as long as the right hardware is available" Would this be the RDRAND instruction or something else?

colmmacc 9y ago
We use RDRAND if it's available, and we also have our own hardware entropy generators in some systems. In either case, we use this entropy as additional input that provides prediction resistance, and it's fed into the AES_CTR generator. Here's a picture from the blog post: https://d2908q01vomqb2.cloudfront.net/ca3512f4dfa95a03169c5a670a4c91a19b3077b4/2017/11/22/s2ndrbg-1024x507.png https://d2908q01vomqb2.cloudfront.net/ca3512f4dfa95a03169c5a... Basically when the DRBG is initialized we start with /dev/urandom and the personalization string. That sets up the initial state, and then hardware entropy is "mixed in" every time the generator is used. Because the mixing in happens before AES is applied, that means that even if either stream of entropy is biased, AES will randomize it anyway (or else AES is a broken cipher!). That means that even if you controlled both inputs, it would still require at least as much computational work as it takes to compute AES to be able to generate a predictable output. That's a nice property of the DRBG construction.
- bogomipz 9y ago
  Thanks for the clarification and detailed response. Cheers!

Nicholas7900 9y ago

So many young Hackers have been asking for the easiest way to hack an email, Do you know how many email services there are, how many kinds of mail servers, how many forms of custom security? In my time, I've hacked numerous email servers and services, usually incidentally to compromising the rest of a company or entity. It's always been different. Sometimes it´s as simple as replacing a username in a URL. Other times I've had to compromise the entire support system for an email service. It varies. It takes imagination. There's no formula. There's no ¨teaching to hack email¨. There's just understanding how systems work and learning to extrapolate. I will reluctantly say that a public favorite seems to be guessing ¨secret answers¨ to ¨secret questions¨ needed to reset passwords on some systems (For Guccifer, hacking email is easy — prison is hard). I never personally used this technique, since it felt like cheating. It's hard to explain why. But certain passwords, once discovered, made sense for certain people (Ih8K1ds for an extremely unpleasant lady at [redacted],SPQR for a CIA officer [on a system with very low password standards]).So many people always think i only work for big companies, that i dont work for private individuals, For urgent requests for professional Hack Advice Or Hack Services, You can contact me directly via compositehacks@gmail.com. I'm also trying out a new beta ticketing system, which can be accessed at "Composite Hacks & Co" So many people always think i only work for big companies, that i dont work for private individuals, here is my mail """"""compositehacks@gmail.com,""""" feel free to contact me for any hacking services you need, i'll do your job professionally. So many people always ask me if i'm a Hacker or a Cracker, as a matter of fact, I don't really care. You can consider me whatever you like - it's entirely subjective. I've been called a white hat, gray hat, and black hat, depending on the attitude and perspective of the writer, and it's not my place to insist that one or the other is correct. I encourage readers to reach conclusions from their own mind and feelings, not arbitrary segregation. I do what I do, and leave it to others to document it - or not - in the way that suits them best. Distinctions like these work for role playing games, and it's kind of like asking "Is Adrian Lamo True Neutral?" - an interesting intellectual exercise, but with no particular relevance to reality. So many people always think i only work for big companies, that i dont work for private individuals, For urgent requests for professional Hack Advice Or Hack Services, You can contact me directly via """""compositehacks@gmail.com"""". I'm also trying out a new beta ticketing system, which can be accessed at "Composite Hacks & Co"

jabl 9y ago

> Recently, both the OpenSSL and glibc projects have been looking to replace their random number generators. They, too, are going with AES_CTR_DRBG, based in some part on the work in s2n and the availability of formal verifications that can be applied to code. That’s pretty sweet. I tried to search for this csprng in glibc, but I wasn't able to find anything. Can anyone provide more info?

NelsonMinar 9y ago

This bit is an aside but it's interesting to me: "Earlier this year, with Galois, we completed a proof of our implementation of AES_CTR_DRBG, and formally verified that our code in s2n is equivalent to the written specification." That's amazing! I think it may be the first bit of provably correct code that I'm likely to actually use in practice.