10 ms·
Myths about /dev/urandom (2014)
- jMyles 8y agoThis document is very informative for sure, but I don't really think it's dispositive for many of today's use cases for randomness. In fact, I think this document is pretty dangerous. I referenced this document here in this stackexchange question, which I think has a good and lively discussion about the real stakes of /dev/urandom today: https://security.stackexchange.com/questions/186086/is-always-use-dev-urandom-still-good-advice-in-an-age-of-containers-and-isola https://security.stackexchange.com/questions/186086/is-alway...
- tptacek 8y agoI don't think you're going to find too many cryptography engineers recommending that you use /dev/random, or, for that matter, the blocking variant of getrandom(2). This might be a live debate on message boards, or on the wrong stack overflow boards --- though Pornin's answer[1] on this question is one of the all-time good Stack Overflow answers --- but among practitioners, this is long settled. [1]: https://security.stackexchange.com/questions/3936/is-a-rand-from-dev-urandom-secure-for-a-login-key/3939 https://security.stackexchange.com/questions/3936/is-a-rand-...
- jMyles 8y agoWell, perhaps you have a better answer to my question than forest's answer, to which I awarded the bounty. For my money, I think that's one of the finest and most balanced pieces of writing on this topic available on the internet today. > I don't think you're going to find too many cryptography engineers recommending that you use...the blocking variant of getrandom(2). It depends on what you mean by "the blocking variant". To quote Cory Benfield: > Can we please try to be clear about what kind of blocking we mean? getrandom(flags=0) absolutely can block: that's what the original issue was all about. To ensure it never blocks you need to call getrandom(GRND_NONBLOCK): that's why the flag exists. It may (or may not) surprise you to learn that PEP 524, which uses this (sometimes blocking) variant of getrandom, has been accepted. Here's a discussion of the matter featuring several reputable, kickass security-minded python people discussing the matter and indeed endorsing the sometimes-blocking (flags=0) variant of getrandom: https://bugs.python.org/issue27266 https://bugs.python.org/issue27266 For full transparency: I read your post on this topic before I posted my question and found that, as far as I was able to read, it added nothing new to the public discourse on this topic. It does not seem to overcome the very real and well-sourced concerns raised by forest in his answer to my question. Anyway, I'm not sure what you mean by "the wrong stack overflow boards" - can you tell me which are right and which are wrong? The link which you've provided, which of course I had already read thoroughly, as I pointed out in my question, is not from Stack Overflow, but from Security StackExchange, just like my question. I haven't yet marked an answer correct, so if you actually have something substantive to add that you think is better than forest's answer, it's possible that you can grab the check.
- tptacek 8y agoRegarding that Stack Exchange answer: avoiding the Linux KRNG in favor of alternate RNG's like OpenSSL's is literally the opposite of what cryptographic software security engineers will tell you. For the past decade and a half, virtually all notable cryptographic failures owing to bad randomness came from systems that weren't using the kernel RNG.
- jMyles 8y agoYeah, as you can see, I pushed back on that. But, as far as I can tell - and I do indeed write with this in mind - the movement to create applications for very fresh, naive installations such as containers or IoT devices, does indeed present a problem for purely-blocking random. My sense is that you and others in the community encouraging blind trust of /dev/urandom need to adjust your language when consulting on this matter in the future. Since I mostly write python, I'm happy that this has been worked out and that python now uses blocking random (via getrandom(flags=0)) in these very rare situations.
- tptacek 8y agoOnce again: there is exactly one time in the lifecycle of an entire system when you want a blocking read from the Linux KRNG: when the system is first booting, to ensure it's seeded (FreeBSD is smarter about this and handles that automatically). Apart from that one time, every other read on your system should be from urandom. No, this doesn't need to updated for IoT or "containers" (Linux containers share the KRNG).
- jMyles 8y ago> there is exactly one time in the lifecycle of an entire system when you want a blocking read from the Linux KRNG Right. I'm glad to hear you phrasing it this way. And listen: I don't mean to be flamy here, and it's possible that my sense of these things is tainted by our many... colorful discussions here on HN, but the unradom orthodoxy, to which getrandom and python's os.urandom are in part a response, is something for which you are arguably a tiny bit responsible, having actually written "Use urandom. Use urandom. Use urandom. Use urandom. Use urandom. Use urandom" on your blog. That kind of punditry has no place in security engineering. There is a need for a degree of sobriety with these sorts of security considerations. Sometimes edge cases move toward the mainstream, and that's exactly what has happened with this odd little phenomenon of programs being executed much, much earlier in the OS lifecycle in the case of containerization and IoT devices. Yes, some containers are deployed in settings where the host KRNG can be and is used. But not all. And many IoT devices suffer from a sufficiently weak set of entropy sources as to need to block. These are real concerns, as identified by the study I linked from the University of Michigan. The researchers there were able to replicate over 1% of the SSH keys that they found in the wild because the devices that created them had insufficient entropy at the time of the key generation. All I'm saying is: don't treat these sorts of security issues as subject to orthodoxy. Clear thinking and solid engineering are the orders of the day.
- adrianmonk 8y ago> I just claim that it's bad to block when the entropy estimate is low. As always when it comes to engineering, good and bad can only be defined when you are specific enough about what it is you're trying to accomplish. Is it a single-user client system with an intelligent user who won't try to subvert security (maybe because you've defaulted to /dev/urandom but given them a choice to override and use /dev/random), and you're trying to do something of crucial importance like generate a long-lived crypto key? Then blocking might be fine or even the better way. Is it a multi-user and/or server system where randomness is a scarce resource whose exhaustion could bring down the system like any other scarce resource (disk, network, RAM, CPU, etc.)? Then blocking is probably a terrible idea and could even open you up to denial of service attacks, thus causing you to achieve less security.
- tptacek 8y agoFirst problem with this response: programmers have a terrible understanding of what "crucial importance" is. In fact, most things in cryptography that want random values need unpredictable values. The old Linux man page creates the impression that you want "good" random for keys, and "mediocre" random for everything else, which is false. Second problem is, there is no such thing as "good" and "poor" random. There are two states: unpredictable (seeded) and predictable (not yet seeded). Once you read seeded state, you're done; blocking serves literally no purpose.
- adrianmonk 8y ago> programmers have a terrible understanding of what "crucial importance" is. If you cannot understand requirements, you cannot produce satisfactory software. Anyway, it was intentional that I gave an example where an educated end user is able to make the choice rather than a programmer having to guess. > The old Linux man page creates the impression that you want "good" random for keys, and "mediocre" random for everything else, which is false. It certainly is false, and it is a myth that many people believe, but it isn't what I was saying. > Once you read seeded state, you're done; blocking serves literally no purpose. As the article explains under the last section ("Not everything is perfect"), /dev/urandom does not guarantee that you are reading seeded state. Also, those numbers' unpredictability rests on the crypto being really good. Which I'm sure it is, but crypto weaknesses have been discovered in the past. It would be defensible to seek additional protection against that possibility for peace of mind if peace of mind is more important to you than running time and/or availability.
- Taniwha 8y ago(disclaimer I sell a HW random number generator) I think both the diagrams here are simplistic - in reality pre 4.8 there were 3 pools (an input pool and 2 output pools, one of which blocked) and post 4.8 where there are 2 pools, an input pool and a blocking pool (urandom now pulls from the input pool thru a CSPRNG). One big downside of the new (post 4.8) architecture is that urandom_min_reseed_secs is ignored - pre-4.8 you could ask the kernel to periodically pull a little bit of real entropy from the input to refresh the urandom output (non-blocking) pool, by default that was once every 60 seconds, but in our case with lots of entropy available we'd set it to happen once a second - now post 4.8 it looks like urandom_min_reseed_secs is ignored and the input pool (where /dev/urandom gets its data from) is not regularly updated unless data is also pulled from /dev/random - which may make sense if entropy is rare, but is not so useful when back pressure from /dev/random is left to control the rate at which external entropy flows into the system - I suspect it's time to present a patch that makes urandom_min_reseed_secs work again Nevertheless the conclusion "use /dev/urandom" is still valid
- Taniwha 8y agoBTW - a tiny bit of python to keep /dev/urandom fed post 4.8 - useful if you have a hardware RNG (either an external one or the one in your CPU): #! /usr/bin/python # # OneRNG /dev/urandom tickler # (C) Copyright Paul Campbell Moonbase Otago 2018 # Released under a GPL 3.0 license # import time with open("/dev/random", 'rb') as r: while True: with open("/proc/sys/kernel/random/entropy_avail", 'r') as e: iv = e.read() print(iv) if iv > 1500: x= r.read(1000) e.close() time.sleep(1)
- canhascodez 8y agoRuby version: #!/usr/bin/env ruby source = File.open('/dev/random', 'rb') loop do source.read(1000) unless File.read('/proc/sys/kernel/random/entropy_avail').to_i < 1500 sleep 1 end
- cuspycode 8y ago
- loeg 8y agoThis seems to just muddy the waters. https://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/ https://sockpuppet.org/blog/2014/02/25/safely-generate-rando... is shorter, more succinct, and to the point.
- Johnny555 8y agoOn a VM that someone else runs (i.e. Google or AWS), is either /dev/random or /dev/urandom cryptographically secure? If no, are there any "safe" choices? (other than an external source like an HSM or hardware random number generator)
- Buge 8y agoWhat do you mean cryptographically secure? Do you mean secure against Google or AWS attacking you by poisoning or recording the randomness? That type of security doesn't sound possible to me. There is the field of homomorphic encryption that has the goal of protecting you against attacks by your cloud provider, but from what I've heard it's extremely slow and requires to you rearchitect all your jobs.
- TheDong 8y agoMost vms / cloud providers can provide a virtualized source of entropy to be mixed into the pool (google "virtio-rng"). Good OSs (like linux with the right virtio modules) will use that to have a high amount of entropy very early into boot. If that's what you're asking, then the answer is "yes, clouds have good urandom available, usually earlier than on real machines". If instead you're asking "but can the cloud compromise me?" then the answer is "you're wrecked". If you do not trust the hardware your vm is running on (be it real or virtual hardware), then you have no hope. Any source of random numbers you read from will read those numbers into memory. The hypervisor or sufficiently-malicious-hardware will then be able to read your memory and get the data, even if it came from an hsm. The "safe" choice is to trust that your hardware/hypervisor is not stealing such data. If you can't trust that, you may have to start fabricating your own processors and running your own data-centres. Be sure to hire better physical security than AWS has or else it's all for naught when someone comes in and yanks your memory to read it.
- deleted 8y ago[deleted]
- matbess 8y agoWell written post, I enjoyed it, it makes sense. The part tearing into the man page was especially excellent! Following your point a little further.. the thought of updating the man pages with warnings like the following made me laugh: “AES, SHA-3, RSA, Diffie-Hellman, Elliptic curves, and all those crypto packages you're using, OpenSSL, GnuTLS, Keyczar, your operating system's crypto API, are all theoretically vulnerable to a cryptographic attack. Knowledge of how to do this is not available in the current unclassified literature, but it is theoretically possible that such an attack may exist. If this is a concern in your application, do not use them.”
- matbess 8y agoThis article naturally leads to a question that I didn’t see addressed, perhaps you can answer it here: If it’s unnecessary to ever prefer /dev/random to urandom, then why does /dev/random’s current blocking implementation still exist? Surely between all of the kernel developers, between all of the Unix-ish OS’s, over the last couple of decades would have realized that /dev/random should simply act like /dev/urandom?
- Scaevolus 8y agoBecause Theodore Ts'o, a major Linux contributor and maintainer, created and maintains /dev/random and blocks attempts to make it nonblocking like every other cryptographically secure random source. It's a political problem, not a technical one. See https://news.ycombinator.com/item?id=6550256 https://news.ycombinator.com/item?id=6550256 for a much longer argument about it.
- matbess 8y agoI suspected that “politics” was going to end up being the answer to my question. Simply because this seems to be a very contentious issue, even if it doesn’t need to be. Edit: Thanks for the link!
- kragormonkey 8y agoI don't see how this is political. In that link he presents actual problems that have arisen from non-blocking versions. I'd call that a technical justification. I don't necessarily agree with his justification, but that's also a difference in technical opinion. The only way I'd fault him is if I can come up with a an approach that addresses his concerns yet he still refuses to accept them. Full disclosure: I've actually interacted with him on a few occasions. IMO calling him risk/change-averse would be an understatement, which partly explains this stance. While at times frustrating for us, this is exactly the quality you want in someone maintaining such a critical part of the kernel.
- John_KZ 8y agoWhy should it be made non-blocking? If you want a large string with little entropy just use /dev/urandom. The political/dogmatic argument is that is should be made non-blocking. The process of acquiring entropy in typical computers is definitely blocking.
- dang 8y ago2014: https://news.ycombinator.com/item?id=7359992 https://news.ycombinator.com/item?id=7359992 2015: https://news.ycombinator.com/item?id=10149019 https://news.ycombinator.com/item?id=10149019 2017: https://news.ycombinator.com/item?id=13332741 https://news.ycombinator.com/item?id=13332741
- matbess 8y agoNice! Thanks