14 ms·
Ruby Bug: SecureRandom should try /dev/urandom first
- astrodust 10y agoIs there a good random library that's not the giant ball of death that's OpenSSL?
- viraptor 10y agoDepends on why you want to use a random number generating library. What are your requirements and specifically what are your requirements that aren't satisfied with the system's csprng. Once you have that defined, you can start looking at what algorithms you want and from there what libraries implement it. Edit: or to put another way, first define what is it that linux's getrandom() and /dev/urandom doesn't provide you.
- astrodust 10y agoThis talk about C++ and rand being harmful is quite educational: https://channel9.msdn.com/Events/GoingNative/2013/rand-Considered-Harmful https://channel9.msdn.com/Events/GoingNative/2013/rand-Consi... What it points out is that even a good random number generator can be used incorrectly, and without the right tools your efforts to produce truly random numbers are doomed from the start. C++ has an embarrassing wealth of random number generators. The Ruby core has almost nothing that can measure up to that, yet it seems like a huge oversight. I like that C++ has a generator for many different use cases, they all have their reason for being there, but Ruby has a singular one with unknown properties. Porting over what C++ has and making a proper Random library for Ruby would make a lot of sense here.
- tptacek 10y agoNone of those C++ RNGs are suitable for cryptography. The answer for C++ CSPRNGs is, like in every other language, to use /dev/urandom. Multiple CSPRNGs just mean multiple single points of failure.
- astrodust 10y agoNot every random number generator has to be cryptographically secure. Sometimes they just need to be properly random.
- yarou 10y agoI've used Mersenne Twister[0] in the past. [0] http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html If you're talking about an RNG library for cryptographic purposes, then it depends on your use case.
- viraptor 10y agoThat one needs manual seeding too, which may lead to more issues.
- tptacek 10y agoIt does not depend on your use case. The answer, in all cases, is just to use /dev/urandom.
- sibrahim 10y agoI had a rather specialized case where it was the pragmatic choice (note, not technically required): running a lottery with potentially litigious losers. If you used a CSPRNG with a seed space smaller than the set of possible lottery outcomes, losers could argue (misleadingly, since we still couldn't feasibly bias the result) that not all outcomes were equally probable and try to get the results thrown out. That is, the fact that there are widespread misconceptions about /dev/random can very rarely be a reason to use it :P However, I agree that the rule is that you should just use /dev/urandom.
- netheril96 10y agoThen what CSPRNG do you use? Any that has seeds larger than 256-bit?
- sibrahim 10y agoAnything that reseeds during operation can qualify. In fact, if the CSPRNG's internal state isn't large enough, you need to periodically reseed or face the same objection. But a CSPRNG which you need to explicitly seed with random bits as big as your output isn't providing much value (simply whitening) since generating the seed is the same problem you had before adding the CSPRNG. So you end up looking at a TRNG.
- ryuuchin 10y agoAs others have said it depends on your needs[6] and whether or not it has to be a CSPRNG (cryptographically secure). Since you mentioned OpenSSL I'll assume that in this context we are talking about a CSPRNG. The short answer is to just use the OS provided one if available. Linux has /dev/urandom[1] and the getrandom[2] syscall and Windows has RtlGenRandom[3][4] (there's also CryptGenRandom on Windows which is the "official" version but is more awkward to use since it requires a CSP context to use). On Windows this uses AES-256 in CTR mode as specified in NIST 800-90. Outside of those (if for whatever reason[6] you're not satisfied with just using what the OS provides you) you can also take a look at how BoringSSL does things[5] (and LibreSSL although I'm less familiar with it). It uses a ChaCha20 instance to filter rdrand output (if supported, otherwise it just uses the system CSPRNG directly). The system CSPRNG is used to key the ChaCha20 instance and then for every call to RAND_bytes it uses rdrand output equal to however many bytes you requested and that gets essentially filtered through the ChaCha20 instance. It's fast and pretty simple. I found the code to be pretty easy to understand[5]. I believe LibreSSL does something similar with ChaCha20 although I'm not sure it uses rdrand. I think there should be a pretty good reason if you're choosing not to use what the OS provides for you. tl;dr Just use /dev/urandom (or the getrandom syscall) on Linux and RtlGenRandom on Windows. [1] http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/ http://sockpuppet.org/blog/2014/02/25/safely-generate-random... [2] http://man7.org/linux/man-pages/man2/getrandom.2.html http://man7.org/linux/man-pages/man2/getrandom.2.html [3] https://msdn.microsoft.com/en-us/library/windows/desktop/aa387694(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... [4] https://boringssl.googlesource.com/boringssl/+/master/crypto/rand/windows.c#26 https://boringssl.googlesource.com/boringssl/+/master/crypto... (RtlGenRandom requires slightly special definitions to work). [5] https://boringssl.googlesource.com/boringssl/+/master/crypto/rand/rand.c https://boringssl.googlesource.com/boringssl/+/master/crypto... [6] There really isn't a "depending on your use case" decision to make. /dev/urandom (or RtlGenRandom) is the correct choice for all cases. With that being said there may be two small caviats to that. Whether or not it has to be a CSPRNG and whether or not /dev/urandom (or RtlGenRandom) can provide the throughput which you require. In certain cases you may have to expand /dev/urandom with something like ChaCha20 if /dev/urandom is too slow (the BoringSSL devs mentioned that for AES-CBC IV generation on servers this can sometimes be the case, see [5] for the BoringSSL implementation).
- forgotpwtomain 10y agoISAAC (http://burtleburtle.net/bob/rand/isaacafa.html http://burtleburtle.net/bob/rand/isaacafa.html) for a CSPRNG. But what is your use-case for something non-standard? Just use /dev/urandom
- derefr 10y agoHow about, backing the equivalent of /dev/urandom for a unikernel OS?
- dchest 10y agoNo, no, no, don't use ISAAC.
- forgotpwtomain 10y agoDo you have a citation you can provide? I don't know of any research that has shown a significant weakness in ISAAC.
- dchest 10y agoWhy would you use a PRNG with unknown cryptographic properties, not designed by a cryptographer, as opposed to one of the NIST's DRBG or a good stream cipher, such as ChaCha? Weakness: https://eprint.iacr.org/2006/438 https://eprint.iacr.org/2006/438 — "huge subsets of internal states which induce a strongly non-uniform distribution in the 8192 first bits produced" Finally, why is deterministic PRNG suggested as a replacement for OpenSSL's random number generator? In general, the advice to write your own userspace PRNG replacement for OpenSSL is not a good advice, because many people are not competent enough to do it.
- forgotpwtomain 10y ago> as opposed to one of the NIST's DRBG I certainly wouldn't go anywhere near another NIST DRBG.. > https://eprint.iacr.org/2006/438 https://eprint.iacr.org/2006/438 From my brief understanding Aumasson's paper uses a different seeding routine from the example provided in the c implementation which allows the weaker states to be produced - indeed it's mentioned on the author's website. > Finally, why is deterministic PRNG suggested as a replacement for OpenSSL's random number generator? In general, the advice to write your own userspace PRNG replacement for OpenSSL is not a good advice, because many people are not competent enough to do it. If you read my above post I clearly do not suggest this.
- viraptor 10y agoSo once again, the man page for urandom creates more problems than solutions. (https://bugzilla.kernel.org/show_bug.cgi?id=71211 https://bugzilla.kernel.org/show_bug.cgi?id=71211)
- nitrogen 10y agoWhy is there such a discrepancy between the prevailing sentiment on HN and the actions of whoever controls the manual? What is preventing one side from convincing the other, apart from stubbornness? Also, why are the Ruby devs so dead set on the manual page?
- tptacek 10y agoIt's not the prevailing sentiment "on HN". It's the prevailing sentiment among virtually all experts everywhere. I can make a list of those experts, but I'll leave it to someone else, so as not to bogart an lay-up high-quality upvote magnet comment. The reason this is a problem is simply that the Linux maintainer is obstinately wrong. Theodore Ts'o was on HN a year or so ago, and in defending the current design (and man pages), made some incoherent arguments, such as that urandom might be OK for nonces, but not key generation (a head-scratcher for anyone familiar with cryptography engineering).
- nitrogen 10y agoI didn't mean to dismiss the word of the experts. I only said "on HN" because that's where I'm familiar with your arguments, which I already put into practice by reading /dev/urandom when I need pseudorandom bytes.
- matt_wulfeck 10y agoCryptographers really are part of the problem. Do you ever Read the discussions in the metzdowd/crypto list? Nobody agrees on anything.
- tptacek 10y ago
- CiPHPerCoder 10y agoYou know your language is bad when PHP -- possibly the most hated web scripting language on the planet -- does it better. Come on Ruby, get your act together. You can do better, right? Also, https://github.com/openssl/openssl/issues/898 https://github.com/openssl/openssl/issues/898
- tacos 10y ago"securerandom.rb will consume too much entoropy if /dev/urandom is used directry." This is a quote for the ages.
- decentrality 10y agoRegardless, whether it be OpenSSL first or /dev/urandom first, "haveged" entropy generation is still needed for virtual machines, which is the most common use case. In my experience both ways are a no-go without special measures like haveged being put in place to super-charge the available entropy. I've had processes hang up for MINUTES or just under TWO HOURS waiting for more entropy to be available.
- tptacek 10y agoNo, it's not. Don't use haveged. Virtual machines should simply get their initial entropy from the already-seeded urandom pool of their hypervisor host. Processes don't "hang waiting for more entropy"; they hang because /dev/random inexplicably goes on strike. The answer to that problem is "never use /dev/random".
- matt_wulfeck 10y agotrue but even if we don't want to use /dev/random, there's still software using it all over the place that we don't necessarily want to patch. I end up installing haveged just because I don't want the system mysteriously locking up because some random daemon wants to create a 4096 bit key on first startup.
- viraptor 10y agoReplacing random with urandom for one app is just one LD_PRELOAD away. Similar to https://rafalcieslak.wordpress.com/2013/04/02/dynamic-linker-tricks-using-ld_preload-to-cheat-inject-features-and-investigate-programs/ https://rafalcieslak.wordpress.com/2013/04/02/dynamic-linker... you can replace open("/dev/random") with open("/dev/urandom")
- throwaway2048 10y agoIts much easier and less error prone to just replace the device node.
- Freaky 10y agoJRuby uses Java's java.security.SecureRandom, which (by default on OpenJDK 8) uses /dev/random, mixed with its own SHA1PRNG (this so setSeed() can be guaranteed to actually do something). Rubinius copies MRI circa 2014 via rubysl - https://github.com/rubysl/rubysl-securerandom https://github.com/rubysl/rubysl-securerandom
- viraptor 10y agoDoes it really? From https://bugs.openjdk.java.net/browse/JDK-4705093 https://bugs.openjdk.java.net/browse/JDK-4705093 : > If you call: new SecureRandom() on Linux and the default values are used, it will read from /dev/urandom and not block. (By default on Solaris, the PKCS11 SecureRandom is used, and also calls into /dev/urandom.)
- jcranmer 10y agoI ran strace on both OpenJDK 7 and 8. It appears to open both /dev/random and /dev/urandom, but it only read from /dev/urandom. From reading the source code, it appears that /dev/random is used if you call SecureRandom::generateSeed, but /dev/urandom is used for SecureRandom::next. Of course, you can change a whole heck of a lot of stuff by setting properties on the command line or changing $JAVA_HOME/lib/security/java.security.
- Freaky 10y agoYep, shouldn't have second-guessed myself. Teach me to comment at 4am. http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/solaris/classes/sun/security/provider/NativePRNG.java http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/s...
- meshko 10y agoWell, I dunno what the man pages say and what some dudes on the internet say, but I know that on CentOS release 6.7 (Final) I need to run with -Djava.security.egd=file:///dev/urandom or else I can't connect to Oracle half the time because Java runs out of entropy. So there must still be some difference between /dev/random and /dev/urandom.
- ascorbic 10y agoNobody's claiming there's no difference between /dev/random and /dev/urandom. They're just saying you should always use /dev/urandom.
- meshko 10y agoNot sure what up with all the downvotes. I'm just saying that Java defaults to /dev/random and Java is kind of a big and important and all that, so not sure why everyone is giving crap to poor little Ruby about this.
- Perceptes 10y agoThis is sad and embarrassing for every involved party. The people leaving rude, entitled, hyperbolic comments in the thread, the Ruby developers who refuse to look into the best practice suggested by experts in the field, and the man page maintainer who refuses to update the man page in accordance with similar information.
- educar 10y agoAgreed that tensions are running high. But I think the ruby devs are in the right here. They are just following the man page. What else is the authoritarian source? How is one supposed to know who wrote those blog posts? Just because it is linuxexpert.com does not mean they are linux experts. It's funny, if people changed it randomly following blogs then people will claim this is some NSA conspiracy :-) How does one verify the person behind the blogs? node has similar https://github.com/nodejs/node/issues/5798 https://github.com/nodejs/node/issues/5798
- lifthrasiir 10y ago> They are just following the man page. I think Ruby devs' position is more than this. By keeping the faulty man page, Linux maintainers are implicitly communicating that they intended `/dev/urandom` to be a limited and less recommended way of doing things. The intention is important: even though `/dev/urandom` is actually better in the current kernel it may not in the future. It is not the only answer, as other languages did another choice, but it is perfectly reasonable to be conservative like this. We badly need to change that intention, really.
- tptacek 10y agoExactly what, do you imagine, Linux could do to make urandom less secure? I'm sorry, but the word for that concern is "nonsensical". Linus Torvalds has world-shattering conniption fits when developers make changes that hurt performance. Can you imagine breaking every one of the many security applications --- for instance, every Go program ever written --- that depend on urandom? No, that is not a legitimate concern.
- jasonjei 10y ago[deleted/retracted]
- jrochkind1 10y agoYes, it's just you. The core ruby team is mostly Japanese-speaking, and Japanese developers are less likely to speak English well even than people in a lot of other countries where English is not a common first language. I think that does serve as a barrier to communication and transparency (with/towards English-speakers!). I think that's all there is, nothing "about Asian culture", or "decisions are made as a group rather than on an individual basis," or cliquishness. We English speakers have the luxury of seeing almost all open source development happen in our first language. Most of the world doesn't. Ruby is an exception, with a mostly Japanese core team. Just how it is.
- jasonjei 10y agoThanks for the enlightenment, I didn't know this aspect of the Ruby core team.
- jrochkind1 10y agoYeah, I too find it hard to figure out exactly who the ruby committers are, or what goes on amongst them. I believe this is simply because of that language barrier. From what I do know, the way MRI is developed is not at all atypical or unusual compared to most other large open source (primarily English-language) projects. Here's a useful (to those curious to learn more about ruby committers) interview from 2013 with 15 ruby committers, note the questions and answers have been translated between English and Japanese. http://www.sitepoint.com/meet-fifteen-ruby-core-committers/ http://www.sitepoint.com/meet-fifteen-ruby-core-committers/ My impression is that a lot of discussion on MRI happens in Japanese on Japanese listservs. Of course, it doesn't take a language barrier to make things seem not so transparent -- for comparison a few years ago, Rails core team started taking the majority of their discussion to private core-team-only listservs. (I gather because they were finding dealing with the peanut gallery made it too hard to get things done/decided). At least there's still lots of open (and English-language) communication on the github issues or open listserv, but there are often times when I'm not really sure who or how or on what basis architectural decisions are made for Rails too, a lot is done in private, is my impression. But I suspect to curious Japanese-speakers, MRI development may even be _more_ transparent than Rails is to English-speakers! I think (but not entirely sure) all of the listservs MRI devs use to collaborate are actually open -- you just have to read/write Japanese to understand what's there or engage effectively. :)
- jrochkind1 10y agoWhy haven't the urandom man pages been fixed yet? Is there, like, no process for fixing man pages at all, they just wind up set in stone forever?
- zurn 10y agoIf you look at the bug filed against the man page, it doesn't exactly make a strong case. It just suggests some weasel words, about qualifying "large amount of data" etc. (https://bugzilla.kernel.org/show_bug.cgi?id=71211 https://bugzilla.kernel.org/show_bug.cgi?id=71211) So bug report understates the issue, nobody has gotten around to writing a good quality patch, everyone is just loudly complaining elsewhere.
- jrochkind1 10y agoActually, I'd say that issue/bug filed two years ago _does_ make a strong case that the current man page is insufficient, that it's actually the current man page that's full of ambiguous language and "weasel words". But you may make a good point that perhaps nobody has submitted a good patch yet -- you're right that issue isn't a good patch, just an invitation to enter into a discussion toward one (an invitation that doesn't seem to have been taken up, at least on the issue tracker). One would hope that the actual maintainers of the kernel subsystems would feel responsibility to improve a clearly insufficient man page. Which makes me suspect there are some underlying politics or personality conflicts going on. (Or just burn-out?) But if all it needs is someone to write some good text -- is it really the case that none of the cryptographers who have been writing extensive blog posts about this for years care to submit a doc patch? If so, I wonder why? Perhaps, just guessing, another potential issue is that nobody really wants to _take responsibility_ for such text, in case they make a mistake. So the existing clearly insufficient text remains. Tragedy of the open source commons?
- zurn 10y ago> it's actually the current man page that's full of ambiguous language and "weasel words". The report suggests "clarifying" what the man page means by the (completely incorrect) statement that "Users should be very economical in the amount of seed material that they read from /dev/urandom" Probably the reporter was just being polite, but in the absence of other comments in the bug or any kernel developers weighing in, it just sounds like an editorial suggestion coming from a single Linux user. Remember that the man-pages project is separate from kernel development.
- Orangeair 10y agoI really don't like the sentiment of "We shouldn't change it unless the man page says we can". That's exactly the kind of senseless bureaucracy that the open source community should be avoiding.
- educar 10y agoHuh? So you want them to make code changes which is against the man page recommendation? Why have man pages at all then? Do you expect ruby developers to dig into kernel source code and more importantly actually understand what is in there? If you think they should take the word of 'experts', how does one qualify as an expert? How do you verify that the expert is the one who wrote the blog post or mail etc etc. The process is already there. Get the change for man page in kernel and user space will follow suit.
- mordocai 10y ago> Do you expect ruby developers to dig into kernel source code and more importantly actually understand what is in there? Yes. Anyone maintaining and distributing the security related code of a language runtime should understand as much as humanly possible about what is actually going on from the application level all the way down to the hardware.
- derefr 10y agoThe manpages for syscalls and device nodes are basically their API specifications. They tell you what behaviours can be guaranteed (i.e. what properties are a part of the interface contract) and what behaviours cannot (i.e. what properties are implementation details of the current implementation, and could change at any time.) Just because the implementation of /dev/urandom has good properties that make usable as a system-wide CSPRNG, doesn't mean that it's supposed to have those good properties. Only the docs can make that assertion. Without such an assertion, the kernel is free to make /dev/urandom have less-sound properties in a future release—because there's no spec saying they shouldn't. (They try not to break userland code, yes, but userland code only relies on things the docs say it can, so...)
- wrs 10y agoTo make this discussion really useful: what's the easiest way for a Ruby application to make SecureRandom use /dev/urandom, without waiting for Ruby-core to be convinced?
- technion 10y agoThis whole situation has annoyed me enough to start writing this gem: https://github.com/technion/use_urandom https://github.com/technion/use_urandom Note I said "start", so I'm aware it needs work. Feedback is appreciated however.
- matthewrudy 10y agoI'd suggest you just use the `rescue` definition from SecureRandom. Then you don't have to do any work yourself. def SecureRandom.gen_random(n) ret = Random.raw_seed(n) unless ret raise NotImplementedError, "No random device" end unless ret.length == n raise NotImplementedError, "Unexpected partial read from random device: only #{ret.length} for #{n} bytes" end ret end
- deleted 10y ago[deleted]
- paloaltokid 10y agoI was recently told about a method to generate a random number using something called "quadratic residue". There was too much math for me to understand it. Is anyone familiar with this?