5 ms·

The article you cite cites a Peter Gutmann article [1, Section 3.6] that details what programmers will do when things don't work as expected. Additionally, wha

by pbsd 13y ago

The article you cite cites a Peter Gutmann article [1, Section 3.6] that details what programmers will do when things don't work as expected. Additionally, what the article recommends (Section 6.2) is that OS generators behave like FreeBSD: block only until enough entropy has been gathered, and then block no more. That is achievable on Linux using tptacek's suggestion, but of course a native solution would be better. [1] https://www.cs.auckland.ac.nz/~pgut001/pubs/usenix02.pdf https://www.cs.auckland.ac.nz/~pgut001/pubs/usenix02.pdf

saurik 13y ago
I did not claim that no one would do something so stupid as to hack the code to remove randomness: I am claiming that those people are so "beyond help" that trading them off against the people who simply fail to realize that decisions made during boot affect their system security (which sounds like a problem even a well-meaning and careful developer could fall into) should take precedence, if that is seriously the only substantial argument. (Also, yes: it would be great if Linux were fixed; if and when it is fixed in a way developers can rely on, then of course that would be a great argument against everything I've said: as far as I know, it hasn't. Hence the authors of the article I referenced recommending the importance of libraries use more secure mechanisms by default, including /dev/random.)
- tptacek 13y ago
  I'm confused. /dev/random is not in fact more secure than urandom. Can you restate your argument here? Developers use bogus userland libraries because the man pages tell them (wrongly) that urandom is unsafe for cryptography, and when they use /dev/random, their application randomly hangs. Exactly why would you ever recommend that a developer attempt to use /dev/random?
  - saurik 13y ago
    During boot of the system, /dev/random will block until there is sufficient entropy while /dev/urandom will not: it will just return, as documented in detail by the paper I referenced, a deterministic stream of "random" data. The researchers document that during 1000 trial boots, the only entropy in the returned data was the position in the stream. This is just asking for system integration errors, such as the wide-spread issue with headless routers generating keys at boot. Hell: even the desktop systems tested by the researchers were only just barely managing to run the key generation late enough that the keys weren't running into the "deterministic data returned from /dev/urandom" problem. Yes: this is fixable by making certain the system has enough entropy sources and doesn't attempt to do anything with /dev/urandom until enough entropy has hit the system to make the data returned actually-random. However, in practice, that isn't happening, it isn't happening often, and it is trivially defendable against by using /dev/random. As it stands, the only reason anyone has come up with for using /dev/urandom instead of /dev/random is "occasionally /dev/random blocks". The question is then whether the "sometimes the system blocks under load" is more or less dangerous of a default than "sometimes keys are generated that are actually the same as every other device". Now, maybe you believe that the former issue is more important than the latter: that people building systems under load are more likely to break them by replacing /dev/random with a file full of constant "random" data (rather than replacing it with, for example, /dev/urandom) than people generating keys at boot without realizing their device is nothing more than a CPU with 16kB of flash and 32MB of ROM attached to a dead network. If you do feel this way, (and it isn't at all clear to me, as you still seem to be beating the "/dev/random doesn't generate better data than /dev/urandom iff the random entropy pool has been correctly seeded" horse, which I maintain was dead long before I arrived with my argument) I encourage you to read the other comment I've written elsewhere in this thread showing why this seems like a poor choice of default behavior. https://news.ycombinator.com/item?id=7362239 https://news.ycombinator.com/item?id=7362239
    - tptacek 13y ago
      This is 6 grafs of concern about the Linux first-boot process that can be addressed by 1 line in an rc script. Seed urandom at boot from /dev/random, and then forget about /dev/random.
      - saurik 13y ago
        For the billionth time: is the user actually doing this, and does the user or system integrator realize they should do this? Apparently, the answer is no, based on real-world systems that don't do this. In some situations (such as an Android phone) it might not even be possible to do this (locked bootloader). Why risk this when the solution is so simple? I maintain that you are encouraging a situation where this kind of mistake happens, when there seems to be a negligible cost to working around it as a developer, protecting your users from their own stupidity and protecting your users from the stupidity of their system integrator.
        acqq 13y ago
        No common library is supposed to protect the user from something that should independently be done by the system, once. It's more than non trivial. Consider modern boot systems which do initialisations in parallel.
      - ScottBurson 13y ago
        It's a real problem, Thomas, particularly in headless VMs cloned from a freshly-installed image. On these machines, reading /dev/random at boot time can hang indefinitely. Your "1 line in an rc script" will make the OS appear to be broken. It's possible the problem can be ameliorated by a combination of approaches. Educating sysadmins is clearly important. I can imagine a userspace process monitoring /dev/random at boot time and screaming if someone tries to read from it and it blocks. (Though on a headless VM, "screaming" in such a way as to reliably be heard may be nontrivial.) The point is, work needs to be done here, by a variety of people, including the distro publishers. Getting that to happen is going to take some advocacy.
- pbsd 13y ago
  Here's my understanding of your argument: - Using /dev/random occasionally blocks applications, but that's acceptable because it's only a small high-throughtput subset of them, and those users can deal with it themselves. - Using /dev/urandom never blocks, but a small headless subset of devices will in some situations generate predictable streams. This is unacceptable. I fear the assessment of the damage done by /dev/random blocking is understated. When it comes to generating long-term keys, yeah, sure, use /dev/random if you want; it doesn't matter how long that takes. But in online applications, these blocks can make an application unusable. Also remember than CSPRNGs are not only used for short key generation: generating CBC IVs is a nice example of something relatively high-throughput that really should not block. You claim that only idiots beyond help will resort to working around /dev/random to prevent blocking. I disagree. I claim that any sufficiently frustrated developer will eventually go down that route if no other obvious choice is available. As an aside, these arguments also put some amount of faith in the entropy estimator's quality, i.e., that /dev/random blocks only when it should. This may not be the case, as shown by [1] in (admittedly) somewhat contrived conditions. [1] https://eprint.iacr.org/2013/338 https://eprint.iacr.org/2013/338
  - saurik 13y ago
    > You claim that only idiots beyond help will resort to working around /dev/random to prevent blocking. I disagree. I claim that any sufficiently frustrated developer will eventually go down that route if no other obvious choice is available. The good workaround for /dev/random is trivial: you use /dev/urandom instead; if you are sufficiently frustrated that you hardcode /dev/random to a file of static data, you probably have made numerous other horrible mistakes. Yes: I maintain that this developer is an idiot who is "beyond help"; it is harder to make a static file for this than to use /dev/urandom. This can also be a configuration file option: "potentially blocks, but always works" vs. "will never block, but you'd better have setup your system correctly". Even better, the developer could go to steps to verify /dev/urandom is ready (such as reading from /dev/random and writing it to /dev/urandom) before using it receptively in ways that could block. > I fear the assessment of the damage done by /dev/random blocking is understated. When /dev/random blocks you know it happened, and a FAQ entry can guide you to a reasonable solution. (And again: given that you can apparently seed /dev/urandom from /dev/random, a solution that the advocates of /dev/urandom are still unhappy with for some reason, it seems like there's no excuse to end up with /dev/random blocking; instead, what everyone should be encouraging is to seed /dev/urandom from /dev/random and then use /dev/urandom.) Alternatively, the people who end up building systems that do not have enough entropy at boot have no way of knowing this happened, and are sufficiently divorced from what is being executed that they are unlikely to even realize that someone is relying on something they shouldn't. This is where "sane defaults" come in: on Linux, it seems like software should not rely on /dev/urandom until the software proves somehow to itself that it is "ready".

by pbsd 13y ago

saurik 13y ago

I did not claim that no one would do something so stupid as to hack the code to remove randomness: I am claiming that those people are so "beyond help" that trading them off against the people who simply fail to realize that decisions made during boot affect their system security (which sounds like a problem even a well-meaning and careful developer could fall into) should take precedence, if that is seriously the only substantial argument. (Also, yes: it would be great if Linux were fixed; if and when it is fixed in a way developers can rely on, then of course that would be a great argument against everything I've said: as far as I know, it hasn't. Hence the authors of the article I referenced recommending the importance of libraries use more secure mechanisms by default, including /dev/random.)

tptacek 13y ago
I'm confused. /dev/random is not in fact more secure than urandom. Can you restate your argument here? Developers use bogus userland libraries because the man pages tell them (wrongly) that urandom is unsafe for cryptography, and when they use /dev/random, their application randomly hangs. Exactly why would you ever recommend that a developer attempt to use /dev/random?
- saurik 13y ago
  During boot of the system, /dev/random will block until there is sufficient entropy while /dev/urandom will not: it will just return, as documented in detail by the paper I referenced, a deterministic stream of "random" data. The researchers document that during 1000 trial boots, the only entropy in the returned data was the position in the stream. This is just asking for system integration errors, such as the wide-spread issue with headless routers generating keys at boot. Hell: even the desktop systems tested by the researchers were only just barely managing to run the key generation late enough that the keys weren't running into the "deterministic data returned from /dev/urandom" problem. Yes: this is fixable by making certain the system has enough entropy sources and doesn't attempt to do anything with /dev/urandom until enough entropy has hit the system to make the data returned actually-random. However, in practice, that isn't happening, it isn't happening often, and it is trivially defendable against by using /dev/random. As it stands, the only reason anyone has come up with for using /dev/urandom instead of /dev/random is "occasionally /dev/random blocks". The question is then whether the "sometimes the system blocks under load" is more or less dangerous of a default than "sometimes keys are generated that are actually the same as every other device". Now, maybe you believe that the former issue is more important than the latter: that people building systems under load are more likely to break them by replacing /dev/random with a file full of constant "random" data (rather than replacing it with, for example, /dev/urandom) than people generating keys at boot without realizing their device is nothing more than a CPU with 16kB of flash and 32MB of ROM attached to a dead network. If you do feel this way, (and it isn't at all clear to me, as you still seem to be beating the "/dev/random doesn't generate better data than /dev/urandom iff the random entropy pool has been correctly seeded" horse, which I maintain was dead long before I arrived with my argument) I encourage you to read the other comment I've written elsewhere in this thread showing why this seems like a poor choice of default behavior. https://news.ycombinator.com/item?id=7362239 https://news.ycombinator.com/item?id=7362239
  - tptacek 13y ago
    This is 6 grafs of concern about the Linux first-boot process that can be addressed by 1 line in an rc script. Seed urandom at boot from /dev/random, and then forget about /dev/random.
    - saurik 13y ago
      For the billionth time: is the user actually doing this, and does the user or system integrator realize they should do this? Apparently, the answer is no, based on real-world systems that don't do this. In some situations (such as an Android phone) it might not even be possible to do this (locked bootloader). Why risk this when the solution is so simple? I maintain that you are encouraging a situation where this kind of mistake happens, when there seems to be a negligible cost to working around it as a developer, protecting your users from their own stupidity and protecting your users from the stupidity of their system integrator.
      - acqq 13y ago
        No common library is supposed to protect the user from something that should independently be done by the system, once. It's more than non trivial. Consider modern boot systems which do initialisations in parallel.
    - ScottBurson 13y ago
      It's a real problem, Thomas, particularly in headless VMs cloned from a freshly-installed image. On these machines, reading /dev/random at boot time can hang indefinitely. Your "1 line in an rc script" will make the OS appear to be broken. It's possible the problem can be ameliorated by a combination of approaches. Educating sysadmins is clearly important. I can imagine a userspace process monitoring /dev/random at boot time and screaming if someone tries to read from it and it blocks. (Though on a headless VM, "screaming" in such a way as to reliably be heard may be nontrivial.) The point is, work needs to be done here, by a variety of people, including the distro publishers. Getting that to happen is going to take some advocacy.
pbsd 13y ago
Here's my understanding of your argument: - Using /dev/random occasionally blocks applications, but that's acceptable because it's only a small high-throughtput subset of them, and those users can deal with it themselves. - Using /dev/urandom never blocks, but a small headless subset of devices will in some situations generate predictable streams. This is unacceptable. I fear the assessment of the damage done by /dev/random blocking is understated. When it comes to generating long-term keys, yeah, sure, use /dev/random if you want; it doesn't matter how long that takes. But in online applications, these blocks can make an application unusable. Also remember than CSPRNGs are not only used for short key generation: generating CBC IVs is a nice example of something relatively high-throughput that really should not block. You claim that only idiots beyond help will resort to working around /dev/random to prevent blocking. I disagree. I claim that any sufficiently frustrated developer will eventually go down that route if no other obvious choice is available. As an aside, these arguments also put some amount of faith in the entropy estimator's quality, i.e., that /dev/random blocks only when it should. This may not be the case, as shown by [1] in (admittedly) somewhat contrived conditions. [1] https://eprint.iacr.org/2013/338 https://eprint.iacr.org/2013/338
- saurik 13y ago
  > You claim that only idiots beyond help will resort to working around /dev/random to prevent blocking. I disagree. I claim that any sufficiently frustrated developer will eventually go down that route if no other obvious choice is available. The good workaround for /dev/random is trivial: you use /dev/urandom instead; if you are sufficiently frustrated that you hardcode /dev/random to a file of static data, you probably have made numerous other horrible mistakes. Yes: I maintain that this developer is an idiot who is "beyond help"; it is harder to make a static file for this than to use /dev/urandom. This can also be a configuration file option: "potentially blocks, but always works" vs. "will never block, but you'd better have setup your system correctly". Even better, the developer could go to steps to verify /dev/urandom is ready (such as reading from /dev/random and writing it to /dev/urandom) before using it receptively in ways that could block. > I fear the assessment of the damage done by /dev/random blocking is understated. When /dev/random blocks you know it happened, and a FAQ entry can guide you to a reasonable solution. (And again: given that you can apparently seed /dev/urandom from /dev/random, a solution that the advocates of /dev/urandom are still unhappy with for some reason, it seems like there's no excuse to end up with /dev/random blocking; instead, what everyone should be encouraging is to seed /dev/urandom from /dev/random and then use /dev/urandom.) Alternatively, the people who end up building systems that do not have enough entropy at boot have no way of knowing this happened, and are sufficiently divorced from what is being executed that they are unlikely to even realize that someone is relying on something they shouldn't. This is where "sane defaults" come in: on Linux, it seems like software should not rely on /dev/urandom until the software proves somehow to itself that it is "ready".