7 ms·
Given that rdrand is so flakey across systems, is there a reason why systemd even bothers using it? The comments suggest it's only used to generate uuids, which
by gruez 6y ago
Given that rdrand is so flakey across systems, is there a reason why systemd even bothers using it? The comments suggest it's only used to generate uuids, which makes me think it's not performance sensitive code. If that's the case why increase code complexity and expose yourself to hardware flakiness just to save a few miliseconds?
- okl 6y agoPossibly because it helps to bootstrap the entropy pool, see for example these ramblings on the topic: https://daniel-lange.com/archives/152-Openssh-taking-minutes-to-become-available,-booting-takes-half-an-hour-...-because-your-server-waits-for-a-few-bytes-of-randomness.html https://daniel-lange.com/archives/152-Openssh-taking-minutes...
- espadrine 6y agoIsn’t that an even bigger problem, though, if Linux’ entropy pool gets polluted by an inentropic source that still increments its entropy counter? I recall discussions to maybe rely on something like jitter entropy when the pool is empty and getentropy() is called.
- garaetjjte 6y ago>on something like jitter entropy when the pool is empty and getentropy() is called This ended up being implemented in 5.4 kernel: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=50ee7529ec45 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- altfredd 6y agoLinux entropy pool is designed to accept and mix different low-quality entropy sources and has explicit workarounds for problems like these. Systemd is literally the only software, that has this problem. I am not aware of any other software, that uses rdrand and expects high-quality cryptography-grade randomness. Precisely, because is does not work. Intel CPUs used to have very similar issues with rdrand and so did AMD. Furthermore, CPU implementation of rdrand is a very attractive targets for state backdoors, so most sensible developers either follow the "GNUPG way" (ask for randomness from user) or simply read from /dev/random. The rdrand instruction is great for games, because it allows to make white noise without complex algorithms and system call overhead. It is also handy for few situations, like interrupt handlers, when you needs to create some semi-random value without using stack space or calling into outside code. Unfortunately, when it was introduced, rdrand was documented to generate "cryptographically strong random numbers" (did Intel developers ever knew, what that means?) Of course, most actual cryptography experts didn't buy into that. As a consequence, and because multi-platform software needed to have it's own RNG anyway, the instruction remained largely unused for actual cryptography. Unused = untested and sometimes broken. If I were in systemd developer's place, I would not hinge bootability of my systems on something like that.
- ficklepickle 6y agoIt's an early boot problem. If they tried to use the kernel facilities it would block, making boot take much longer. Fixed in recent kernel versions. I know it's in vogue to rip on systemd, but let's at least try to be fair.
- freeone3000 6y agoSomehow sysv init never ran into this issue. Maybe it's okay to let the kernel seed its entropy pool before initializing your hash table.
- Foxboron 6y agoI don't think sysvinit made use of hashtables, and if they did they would probably hit the same snag.
- aidenn0 6y agosysvinit (or rather RC systems often built on top of it) tended to use trees (most often the filesystem). Unlike hash-tables, trees don't need randomness to guarantee asymptotic behavior.
- Dylan16807 6y ago> The rdrand instruction is great for games, because it allows to make white noise without complex algorithms and system call overhead. Eh. You're looking at hundreds of cycles per use. That makes it slower than a secure software RNG, let alone an insecure one.
- jcelerier 6y agoRight ! For games if it's for e.g. texture noise or things like that you shouldn't need more than a couple xors and shifts
- tadfisher 6y ago
- CJefferson 6y agoWhen it comes to random seeding, I've always thought the best option would be to use every source available, and hope that at least one is high quality. If we combine: * RDRAND * Seed from previous boot * Jitter * Result of every sensor (for example temperature) attached to the machine * Seed from hypervisor (if we are in a VM) * Time & Date * Any information about the physical computer we are on (attached hardware)
- recursive 6y agoWho's spending multiple milliseconds generating uuids? I'm sure it happens somewhere, somehow, but that does not seem like a common occurrence.
- stefan_ 6y agoIt's really ridiculous of AMD to keep fucking this up, even if the backdoor thing never stuck for RDRAND, the fact that one CPU vendor can't manage to make the thing not return identical values many many times over sure might. Similar story for systemd of course, nobody would expect them to learn their lesson. How nice of an init system to keep your system dead because it apparently needs CPRNG generated job ids of all things.
- temac 6y agoGiven this is broken over and over again in successives AMD chips, this is actually a little bit likely to be a backdoor. People will e.g. generate keys with broken rdrand, and AMD will later fix it with plausible deniability, just saying "oops, sorry, it was just our silly usual mistake, and we don't even promise we won't do it again, lol!". In the meantime the NSA will enjoy the resulting compromises. And well, even if it was just the same silly mistake made over and over again, the NSA is likely to still enjoy the results? So it does not really matter if it was intentional or not: regardless of the root cause of this bug, it should now be considered that rdrand should never be trusted (unless you truly don't care that your systems risk to randomly break or you risk to randomly generate compromised keys) and its mere usage (except maybe indirectly in mitigated ways designed by experts and audited for years) considered a very likely security vulnerability.
- altfredd 6y agoIt seems, that many RedHat developers have rather unique world views. When PulseAudio introduces random crackles and scratches in sound output, they claim, that I have a misbehaving sound card. It does not "misbehave", when I use the same applications with ALSA, but whatever. When rdrand fails to produce high-quality entropy — again and again, over and over — Poettering complains about buggy CPUs. I hope, that this particular group never ends up working on network hardware.
- Avamander 6y ago> It does not "misbehave", when I use the same applications with ALSA, but whatever. I mean RDRAND is a great example here, if you never use it, you never encounter these bugs. It's not impossible that PulseAudio tries to play higher quality audio or simply uses the device differently. > I hope, that this particular group never ends up working on network hardware. I'd kinda love it, probably quite a few fun hardware bugs will be found.
- Dylan16807 6y agoI'm reminded of this story about hidden network card bugs. Working in a bunch of tests doesn't mean something is correct, especially when it's complex. https://devblogs.microsoft.com/oldnewthing/20050512-48/?p=35653 https://devblogs.microsoft.com/oldnewthing/20050512-48/?p=35...
- vetinari 6y ago> It's not impossible that PulseAudio tries to play higher quality audio or simply uses the device differently. Pulseaudio mixes sound from several applications, unlike Alsa. Alsa just configures your soundcard for that one stream, Pulseaudio has no such luxury. In order to mix relatively small buffers (because latency), it needs to do timing. If your soundcard has problem with timing (and many do!), you will get crackling. The good news is, that the problem with timing can be only at specific frequencies; in the past, I had a computer with Nvidia Ion and it did this with 44,1kHz audio. With 48kHz, everything was OK. So fixing the output at 48kHz by forcing Pulse to mix at this frequency fixed the crackling.
- tadfisher 6y agoIt's not systemd's job to fix hardware bugs, it's the kernel's (or AMD's). The fix for the last time this happened was to mask the offending CPU IDs in the kernel's RNG from using RDRAND. systemd did have a temporary hack in place for their direct RDRAND implementation, but this current bug isn't the same (not just emitting 0 like last time, but eventually emitting duplicate values in pairs).
- klmadfejno 6y agoThat doesn't seem to answer the question of why use it
- tadfisher 6y agoIf they don't use it, the kernel does in order to initialize the entropy pool, which will also result in broken boots. To work around historically flaky kernel implementations, systemd calls RDRAND directly if it detects it needs to with a heuristic. This is why the fix for the current issue is to set both 'nordrand' and 'SYSTEMD_RDRAND=0'.
- tedunangst 6y agoWhat happened to it's not systemd's job to fix kernel bugs?
- tadfisher 6y agoI didn't type that, so I'm not sure what you're referring to. systemd is fundamentally tied to Linux and thus needs to implement workarounds from time-to-time when kernel facilities are lacking. This is complicated by the requirement that systemd continue working on older Linux versions.
- numpad0 6y agoIt generally makes more sense to trust CPU specifications and utilize it fully, than to assume everything is tampered with and only ever use `mov`
- AshamedCaptain 6y agoAnd why do you even need globally UUIDs here? What's wrong with a plain old sequence counter or the like?
- stragies 6y agoAmong others is that predictability of behavior of systems makes analysis and/or attack easier. Also "plain old sequence counters" in clusters of (heterogeneous) hardware makes distributing (floating) tasks more challenging. There are a bunch of other reasons.
- garaetjjte 6y agoAs much as I don't like systemd this probably was reasonable decision back at the time. systemd needs entropy for generating some random identifiers during early boot. Trying to read it from kernel random facilities might block infinitely until kernel 5.4. Before that pulling entropy from RDRAND might not be that unreasonable decision, nowadays it probably should just call getrandom().
- tremon 6y agoBut why does systemd need random identifiers during early boot? I can understand the need for seeding a hash table (but even there, they already admit that the code can re-seed at any time), but I don't see why a boot program should rely on random id generation in the first place.
- LukeShu 6y agoThe source code has a lengthy comment explaining why. https://github.com/systemd/systemd/blob/bcac754d66374782a85a8de39fd543c1eaf9fa82/src/basic/random-util.c#L40-L98 https://github.com/systemd/systemd/blob/bcac754d66374782a85a...
- robocat 6y agoint rdrand(unsigned long *ret) { /* So, you are a "security researcher", and you wonder why we bother with using raw RDRAND here, * instead of sticking to /dev/urandom or getrandom()? * * Here's why: early boot. On Linux, during early boot the random pool that backs /dev/urandom and * getrandom() is generally not initialized yet. It is very common that initialization of the random * pool takes a longer time (up to many minutes), in particular on embedded devices that have no * explicit hardware random generator, as well as in virtualized environments such as major cloud * installations that do not provide virtio-rng or a similar mechanism. * * In such an environment using getrandom() synchronously means we'd block the entire system boot-up * until the pool is initialized, i.e. *very* long. Using getrandom() asynchronously (GRND_NONBLOCK) * would mean acquiring randomness during early boot would simply fail. Using /dev/urandom would mean * generating many kmsg log messages about our use of it before the random pool is properly * initialized. Neither of these outcomes is desirable. * * Thus, for very specific purposes we use RDRAND instead of either of these three options. RDRAND * provides us quickly and relatively reliably with random values, without having to delay boot, * without triggering warning messages in kmsg. * * Note that we use RDRAND only under very specific circumstances, when the requirements on the * quality of the returned entropy permit it. Specifically, here are some cases where we *do* use * RDRAND: * * • UUID generation: UUIDs are supposed to be universally unique but are not cryptographic * key material. The quality and trust level of RDRAND should hence be OK: UUIDs should be * generated in a way that is reliably unique, but they do not require ultimate trust into * the entropy generator. systemd generates a number of UUIDs during early boot, including * 'invocation IDs' for every unit spawned that identify the specific invocation of the * service globally, and a number of others. Other alternatives for generating these UUIDs * have been considered, but don't really work: for example, hashing uuids from a local * system identifier combined with a counter falls flat because during early boot disk * storage is not yet available (think: initrd) and thus a system-specific ID cannot be * stored or retrieved yet. * * • Hash table seed generation: systemd uses many hash tables internally. Hash tables are * generally assumed to have O(1) access complexity, but can deteriorate to prohibitive * O(n) access complexity if an attacker manages to trigger a large number of hash * collisions. Thus, systemd (as any software employing hash tables should) uses seeded * hash functions for its hash tables, with a seed generated randomly. The hash tables * systemd employs watch the fill level closely and reseed if necessary. This allows use of * a low quality RNG initially, as long as it improves should a hash table be under attack: * the attacker after all needs to trigger many collisions to exploit it for the purpose * of DoS, but if doing so improves the seed the attack surface is reduced as the attack * takes place. * * Some cases where we do NOT use RDRAND are: * * • Generation of cryptographic key material * * • Generation of cryptographic salt values * * This function returns: * * -EOPNOTSUPP → RDRAND is not available on this system * -EAGAIN → The operation failed this time, but is likely to work if you try again a few * times * -EUCLEAN → We got some random value, but it looked strange, so we refused using it. * This failure might or might not be temporary. */
- zuzun 6y ago> just to save a few miliseconds? RDRAND is rather slow, especially with the SRBDS microcode update. One instruction takes 1,200 ns on my system, that's 6.6 MB/s. A software implementation of a CSPRNG will do hundreds of MB/s, the general purpose PRNGs do several GB/s.