13 ms·

RDRAND on AMD Ryzen 9 5900X is flakey

119 points by Avamander 6y ago

gruez 6y ago
Given that rdrand is so flakey across systems, is there a reason why systemd even bothers using it? The comments suggest it's only used to generate uuids, which makes me think it's not performance sensitive code. If that's the case why increase code complexity and expose yourself to hardware flakiness just to save a few miliseconds?
- okl 6y ago
  Possibly because it helps to bootstrap the entropy pool, see for example these ramblings on the topic: https://daniel-lange.com/archives/152-Openssh-taking-minutes-to-become-available,-booting-takes-half-an-hour-...-because-your-server-waits-for-a-few-bytes-of-randomness.html https://daniel-lange.com/archives/152-Openssh-taking-minutes...
  - espadrine 6y ago
    Isn’t that an even bigger problem, though, if Linux’ entropy pool gets polluted by an inentropic source that still increments its entropy counter? I recall discussions to maybe rely on something like jitter entropy when the pool is empty and getentropy() is called.
    - garaetjjte 6y ago
      >on something like jitter entropy when the pool is empty and getentropy() is called This ended up being implemented in 5.4 kernel: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=50ee7529ec45 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
    - altfredd 6y ago
      Linux entropy pool is designed to accept and mix different low-quality entropy sources and has explicit workarounds for problems like these. Systemd is literally the only software, that has this problem. I am not aware of any other software, that uses rdrand and expects high-quality cryptography-grade randomness. Precisely, because is does not work. Intel CPUs used to have very similar issues with rdrand and so did AMD. Furthermore, CPU implementation of rdrand is a very attractive targets for state backdoors, so most sensible developers either follow the "GNUPG way" (ask for randomness from user) or simply read from /dev/random. The rdrand instruction is great for games, because it allows to make white noise without complex algorithms and system call overhead. It is also handy for few situations, like interrupt handlers, when you needs to create some semi-random value without using stack space or calling into outside code. Unfortunately, when it was introduced, rdrand was documented to generate "cryptographically strong random numbers" (did Intel developers ever knew, what that means?) Of course, most actual cryptography experts didn't buy into that. As a consequence, and because multi-platform software needed to have it's own RNG anyway, the instruction remained largely unused for actual cryptography. Unused = untested and sometimes broken. If I were in systemd developer's place, I would not hinge bootability of my systems on something like that.
      - ficklepickle 6y ago
        It's an early boot problem. If they tried to use the kernel facilities it would block, making boot take much longer. Fixed in recent kernel versions. I know it's in vogue to rip on systemd, but let's at least try to be fair.
        freeone3000 6y ago
        Somehow sysv init never ran into this issue. Maybe it's okay to let the kernel seed its entropy pool before initializing your hash table.
        Foxboron 6y ago
        I don't think sysvinit made use of hashtables, and if they did they would probably hit the same snag.
        aidenn0 6y ago
        sysvinit (or rather RC systems often built on top of it) tended to use trees (most often the filesystem). Unlike hash-tables, trees don't need randomness to guarantee asymptotic behavior.
      - Dylan16807 6y ago
        > The rdrand instruction is great for games, because it allows to make white noise without complex algorithms and system call overhead. Eh. You're looking at hundreds of cycles per use. That makes it slower than a secure software RNG, let alone an insecure one.
        jcelerier 6y ago
        Right ! For games if it's for e.g. texture noise or things like that you shouldn't need more than a couple xors and shifts
      - tadfisher 6y ago
    - CJefferson 6y ago
      When it comes to random seeding, I've always thought the best option would be to use every source available, and hope that at least one is high quality. If we combine: * RDRAND * Seed from previous boot * Jitter * Result of every sensor (for example temperature) attached to the machine * Seed from hypervisor (if we are in a VM) * Time & Date * Any information about the physical computer we are on (attached hardware)
- recursive 6y ago
  Who's spending multiple milliseconds generating uuids? I'm sure it happens somewhere, somehow, but that does not seem like a common occurrence.
- stefan_ 6y ago
  It's really ridiculous of AMD to keep fucking this up, even if the backdoor thing never stuck for RDRAND, the fact that one CPU vendor can't manage to make the thing not return identical values many many times over sure might. Similar story for systemd of course, nobody would expect them to learn their lesson. How nice of an init system to keep your system dead because it apparently needs CPRNG generated job ids of all things.
  - temac 6y ago
    Given this is broken over and over again in successives AMD chips, this is actually a little bit likely to be a backdoor. People will e.g. generate keys with broken rdrand, and AMD will later fix it with plausible deniability, just saying "oops, sorry, it was just our silly usual mistake, and we don't even promise we won't do it again, lol!". In the meantime the NSA will enjoy the resulting compromises. And well, even if it was just the same silly mistake made over and over again, the NSA is likely to still enjoy the results? So it does not really matter if it was intentional or not: regardless of the root cause of this bug, it should now be considered that rdrand should never be trusted (unless you truly don't care that your systems risk to randomly break or you risk to randomly generate compromised keys) and its mere usage (except maybe indirectly in mitigated ways designed by experts and audited for years) considered a very likely security vulnerability.
- altfredd 6y ago
  It seems, that many RedHat developers have rather unique world views. When PulseAudio introduces random crackles and scratches in sound output, they claim, that I have a misbehaving sound card. It does not "misbehave", when I use the same applications with ALSA, but whatever. When rdrand fails to produce high-quality entropy — again and again, over and over — Poettering complains about buggy CPUs. I hope, that this particular group never ends up working on network hardware.
  - Avamander 6y ago
    > It does not "misbehave", when I use the same applications with ALSA, but whatever. I mean RDRAND is a great example here, if you never use it, you never encounter these bugs. It's not impossible that PulseAudio tries to play higher quality audio or simply uses the device differently. > I hope, that this particular group never ends up working on network hardware. I'd kinda love it, probably quite a few fun hardware bugs will be found.
    - Dylan16807 6y ago
      I'm reminded of this story about hidden network card bugs. Working in a bunch of tests doesn't mean something is correct, especially when it's complex. https://devblogs.microsoft.com/oldnewthing/20050512-48/?p=35653 https://devblogs.microsoft.com/oldnewthing/20050512-48/?p=35...
    - vetinari 6y ago
      > It's not impossible that PulseAudio tries to play higher quality audio or simply uses the device differently. Pulseaudio mixes sound from several applications, unlike Alsa. Alsa just configures your soundcard for that one stream, Pulseaudio has no such luxury. In order to mix relatively small buffers (because latency), it needs to do timing. If your soundcard has problem with timing (and many do!), you will get crackling. The good news is, that the problem with timing can be only at specific frequencies; in the past, I had a computer with Nvidia Ion and it did this with 44,1kHz audio. With 48kHz, everything was OK. So fixing the output at 48kHz by forcing Pulse to mix at this frequency fixed the crackling.
- tadfisher 6y ago
  It's not systemd's job to fix hardware bugs, it's the kernel's (or AMD's). The fix for the last time this happened was to mask the offending CPU IDs in the kernel's RNG from using RDRAND. systemd did have a temporary hack in place for their direct RDRAND implementation, but this current bug isn't the same (not just emitting 0 like last time, but eventually emitting duplicate values in pairs).
  - klmadfejno 6y ago
    That doesn't seem to answer the question of why use it
    - tadfisher 6y ago
      If they don't use it, the kernel does in order to initialize the entropy pool, which will also result in broken boots. To work around historically flaky kernel implementations, systemd calls RDRAND directly if it detects it needs to with a heuristic. This is why the fix for the current issue is to set both 'nordrand' and 'SYSTEMD_RDRAND=0'.
      - tedunangst 6y ago
        What happened to it's not systemd's job to fix kernel bugs?
        tadfisher 6y ago
        I didn't type that, so I'm not sure what you're referring to. systemd is fundamentally tied to Linux and thus needs to implement workarounds from time-to-time when kernel facilities are lacking. This is complicated by the requirement that systemd continue working on older Linux versions.
    - numpad0 6y ago
      It generally makes more sense to trust CPU specifications and utilize it fully, than to assume everything is tampered with and only ever use `mov`
- AshamedCaptain 6y ago
  And why do you even need globally UUIDs here? What's wrong with a plain old sequence counter or the like?
  - stragies 6y ago
    Among others is that predictability of behavior of systems makes analysis and/or attack easier. Also "plain old sequence counters" in clusters of (heterogeneous) hardware makes distributing (floating) tasks more challenging. There are a bunch of other reasons.
    - effie 6y ago
      How is using UUIDs making software more predictable than using sequence numbers?
      - stragies 6y ago
        the opposite: UUIDs make things less predictable. And in many situations this is very good.
- garaetjjte 6y ago
  As much as I don't like systemd this probably was reasonable decision back at the time. systemd needs entropy for generating some random identifiers during early boot. Trying to read it from kernel random facilities might block infinitely until kernel 5.4. Before that pulling entropy from RDRAND might not be that unreasonable decision, nowadays it probably should just call getrandom().
  - tremon 6y ago
    But why does systemd need random identifiers during early boot? I can understand the need for seeding a hash table (but even there, they already admit that the code can re-seed at any time), but I don't see why a boot program should rely on random id generation in the first place.
- LukeShu 6y ago
  The source code has a lengthy comment explaining why. https://github.com/systemd/systemd/blob/bcac754d66374782a85a8de39fd543c1eaf9fa82/src/basic/random-util.c#L40-L98 https://github.com/systemd/systemd/blob/bcac754d66374782a85a...
  - robocat 6y ago
    int rdrand(unsigned long *ret) { /* So, you are a "security researcher", and you wonder why we bother with using raw RDRAND here, * instead of sticking to /dev/urandom or getrandom()? * * Here's why: early boot. On Linux, during early boot the random pool that backs /dev/urandom and * getrandom() is generally not initialized yet. It is very common that initialization of the random * pool takes a longer time (up to many minutes), in particular on embedded devices that have no * explicit hardware random generator, as well as in virtualized environments such as major cloud * installations that do not provide virtio-rng or a similar mechanism. * * In such an environment using getrandom() synchronously means we'd block the entire system boot-up * until the pool is initialized, i.e. *very* long. Using getrandom() asynchronously (GRND_NONBLOCK) * would mean acquiring randomness during early boot would simply fail. Using /dev/urandom would mean * generating many kmsg log messages about our use of it before the random pool is properly * initialized. Neither of these outcomes is desirable. * * Thus, for very specific purposes we use RDRAND instead of either of these three options. RDRAND * provides us quickly and relatively reliably with random values, without having to delay boot, * without triggering warning messages in kmsg. * * Note that we use RDRAND only under very specific circumstances, when the requirements on the * quality of the returned entropy permit it. Specifically, here are some cases where we *do* use * RDRAND: * * • UUID generation: UUIDs are supposed to be universally unique but are not cryptographic * key material. The quality and trust level of RDRAND should hence be OK: UUIDs should be * generated in a way that is reliably unique, but they do not require ultimate trust into * the entropy generator. systemd generates a number of UUIDs during early boot, including * 'invocation IDs' for every unit spawned that identify the specific invocation of the * service globally, and a number of others. Other alternatives for generating these UUIDs * have been considered, but don't really work: for example, hashing uuids from a local * system identifier combined with a counter falls flat because during early boot disk * storage is not yet available (think: initrd) and thus a system-specific ID cannot be * stored or retrieved yet. * * • Hash table seed generation: systemd uses many hash tables internally. Hash tables are * generally assumed to have O(1) access complexity, but can deteriorate to prohibitive * O(n) access complexity if an attacker manages to trigger a large number of hash * collisions. Thus, systemd (as any software employing hash tables should) uses seeded * hash functions for its hash tables, with a seed generated randomly. The hash tables * systemd employs watch the fill level closely and reseed if necessary. This allows use of * a low quality RNG initially, as long as it improves should a hash table be under attack: * the attacker after all needs to trigger many collisions to exploit it for the purpose * of DoS, but if doing so improves the seed the attack surface is reduced as the attack * takes place. * * Some cases where we do NOT use RDRAND are: * * • Generation of cryptographic key material * * • Generation of cryptographic salt values * * This function returns: * * -EOPNOTSUPP → RDRAND is not available on this system * -EAGAIN → The operation failed this time, but is likely to work if you try again a few * times * -EUCLEAN → We got some random value, but it looked strange, so we refused using it. * This failure might or might not be temporary. */
- zuzun 6y ago
  > just to save a few miliseconds? RDRAND is rather slow, especially with the SRBDS microcode update. One instruction takes 1,200 ns on my system, that's 6.6 MB/s. A software implementation of a CSPRNG will do hundreds of MB/s, the general purpose PRNGs do several GB/s.
HarryHirsch 6y ago
Make that: "systemd and RDRAND on Ryzen 9 is flakey"
- Dylan16807 6y ago
  I'm sure a lot of other things are being affected by this defect, just less visibly. It's not like the software is breaking x86 instructions somehow.
  - HarryHirsch 6y ago
    Someone please explain why someone would use the hardware random generator directly. Everyone else uses /dev/{u,}random because somehow HRNGs always manage to be buggy, and then there's the concern about NSA involvement. Using OS-provided interface mitigates either concern.
    - mjg59 6y ago
      https://github.com/systemd/systemd/blob/bcac754d66374782a85a8de39fd543c1eaf9fa82/src/basic/random-util.c#L40-L98 https://github.com/systemd/systemd/blob/bcac754d66374782a85a...
      - HarryHirsch 6y ago
        Dude is stubborn in the face of reality. It's not as if there hadn't been RDRAND hardware bugs before.
      - zamadatix 6y ago
        Not sure I understand the /dev/urandom concern in that comment. If this doesn't need to be cryptographically good and you don't want logspam every time a new number is needed why not seed this once with /dev/urandom. Surely a single log message is better than trying to reinvent the wheel, especially since this isn't the first time I've seen this particular code linked on HN due to the assumptions not holding true.
xeeeeeeeeeeenu 6y ago
Why does this keep happening? RDRAND was broken in Ryzen 3xxx[1] and Jaguar/Puma[2] too. [1] - https://arstechnica.com/gadgets/2019/10/how-a-months-old-amd-microcode-bug-destroyed-my-weekend/ https://arstechnica.com/gadgets/2019/10/how-a-months-old-amd... [2] - https://linuxreviews.org/RDRAND_stops_returning_random_values_on_older_AMD_CPUs_after_suspend https://linuxreviews.org/RDRAND_stops_returning_random_value...
- stabbles 6y ago
  I remember I ran into this Ryzen problem too. Spent a long time trying to figure out a bug in my code (random numbers were deterministic between runs), which ultimately was fixed by upgrading the bios.
- Avamander 6y ago
  > Why does this keep happening? Windows doesn't use RDRAND.
dschuetz 6y ago
systemd: we'll fork our own rdrand
- uncledave 6y ago
  Headline in two years “Pwnie award for systemd-rdrand”
- LukeShu 6y ago
  systemd: It's not our job to hack around hardware bugs, it's the kernel's. We will not implement our own thing. HN: How dare you.[1] Also HN: Criticizes systemd for implementing its own thing Systemd just can't win. [1] https://news.ycombinator.com/item?id=10999335 https://news.ycombinator.com/item?id=10999335
  - dschuetz 6y ago
    They can. Just fork Linux and do their own thing, with blackjack and hookers.
  - na85 6y ago
    apt: Package Gnome3-Shell depends on systemd-rdrand which is not installed. Package systemd-rdrand conflicts gnome3-shell. Uninstall 1230982 packages to resolve? [y/n]
gbrown_ 6y ago
Not to be snarky but OpenBSD's random system has always seemed very elegant to me and seems like such a system could avoid this failure mode in systemd, video[1], slides[2]. That said it's good that the faulty RDRAND was discovered. As pointed out this isn't the first time processors (AMD particular) have had such issues. Do we need to just be skeptical of this and run RDRAND tests every time a new processor comes out? Or perhaps even each microcode update? [1] https://www.youtube.com/watch?v=aWmLWx8ut20 https://www.youtube.com/watch?v=aWmLWx8ut20 [2] https://www.openbsd.org/papers/hackfest2014-arc4random/index.html https://www.openbsd.org/papers/hackfest2014-arc4random/index...
- segfaultbuserr 6y ago
  > seems like such a system could avoid this failure mode in systemd, video[1], slides[2] Yes and no. Yes, OpenBSD is not vulnerable to this failure mode. But no, it's only because OpenBSD made assumptions that systemd couldn't. The problem is not how well the random system performs once it started, but simply what to do when it's not initialized yet during early boot. From Page 20: Kernel initialization from boot, OpenBSD's initialization sequence is: 1. If available, use rdrand to generate random seeds. 2. Read random seed file from the disk, saved from the previous boots or installation, very early by the bootloader. 3. Read the Stack Protector cookie from the kernel binary, by the bootloader. 4. Mix them together. The pool is initialized at this point. 5. Keep collecting entropy from various sources of randomness, such as interrupts. Nice, it ensures that you always get a initialized entropy pool, and hides all complexity away from the userspace. But do note that (2) and (3) provides no security during the first early boot if the same OS image is replicated across many machines. --- Now what about Linux? 1. If available... - use rdrand to generate random seeds. It's optional and can be disabled by the user. - use hardware random number generators and TPMs, but only if they're compiled into the kernel (or loaded early enough) and trusted by the user. - use the in-kernel jitter entropy collector, "a modern out-of-order CPU, even quite simple loops show a fair amount of hard-to-predict timing variability." But only available since Linux 5.3 released in 2019. 2. Read random seed file from the disk, saved from the previous boots. But it's the responsibility of the userspace to do that, this is problematic for systemd. - systemd needs a source of randomness at an earlier time, before the entropy pool is seeded by the file. How did OpenBSD solve it? Instead of loading it after boot in userspace, OpenBSD loads the file early, really early, via the bootloader. - systemd doesn't trust the file. The file is read, but by default, its entropy is not credited, because systemd doesn't want to take the responsibility if someone accidentally replicated the random file across millions of machines via a system image (OpenBSD seems to be okay with the lack of protection of image replication, but systemd is more cautious, probably because it has zero control over the rest of the system). As a result, the system may block at step 4 (entropy collection) during boot for a long time. 3. If Latent Entropy GCC plugin is used, Linux kernel can use entropy embedded in the kernel binary. It's a creative innovation by PaX/grsec, it uses a random seed inserted at build time, but also inserts local variables in every marked function, so that different runtime code paths and control flows create different entropy seeds. But it's only an optional feature and almost nobody uses it - PaX/grsec ideas are too radical to most people ;-) - Note that it's security properties can still be seriously weakened by a replicated kernel binary. Also, it's entropy is seeded to the pool, but it's also not trusted and credited, since it's considered a workaround, not a solution. 4. Keep collecting entropy from various sources of randomness, such as interrupts. As you can see, it's actually really similar to OpenBSD (I meant the concepts, not the implementation), the only difference is that almost everything is optional and nothing is guaranteed to work. --- So as a tradeoff, systemd uses the following logic instead. 1. If available, bypass the kernel, use rdrand directly for non-crypto randomness. 2. For everything else (no rdrand machines & crypto), use the system's entropy pool, it may block for a long time before the pool is initialized, even with a random seed file or Latent Entropy, because their entropy is uncredited. I think this is a reasonable tradeoff, but if rdrand is broken, everything breaks down. I think the differences between OpenBSD and Linux is basically due to how much control you have over the operating system. OpenBSD has the advantage you have when you're building an entire operating system, not just a kernel. --- Update: As pointed out by the comment, systemd supports bootloader entropy too, but only uses it via systemd-boot on EFI systems since it's not vulnerable to the image replication problem. With UEFI, it can combine the seed file with a machine-specific EFI variable ("system token") in UEFI's NVRAM generated during system installation. Systemd also criticizes NetBSD's bootloader entropy for being vulnerable to the image replication problem, see https://systemd.io/RANDOM_SEEDS/ https://systemd.io/RANDOM_SEEDS/ > This is boring: NetBSD had boot loader entropy seed support since ages! > Yes, NetBSD has that, and the above is inspired by that (note though: this article is about a lot more than that). NetBSD’s support is not really safe, since it neither updates the random seed before using it, nor has any safeguards against replicating the same disk image with its random seed on multiple machines (which the ‘system token’ mentioned above is supposed to address). This means reuse of the same random seed by the boot loader is much more likely. Update 2: Rewrite the comment entirely.
nitrobeast 6y ago
Functions that return random results do not fit into the most common testing paradigms. I have seen people try to verify the distribution of random results, and even that could have concerns for theoretical flakiness. I wonder if the nature of the function contributed to the apparent lack of testing coverage here.
dragontamer 6y ago
Hmmm... Well, first problem is using RDRAND in this case. RDRAND isn't meant for seeds, its for a random number. If you're seeding something, you're supposed to use RDSEED, which has more guarantees about seeding / entropy behavior. I don't think RDRAND was ever intended to be used as an entropy source. (Think of it this way: RDSEED is kinda like /dev/random, while RDRAND is kinda-like /dev/urandom) ------ RDRAND is an older instruction than RDSEED. So that then brings up the question: what is the proper startup behavior of systems without RDSEED but with RDRAND? Well, since RDRAND never had entropy guarantees, I guess its up to other sources of entropy (see clock-entropy, such as RDTSC). RDTSC isn't a source of entropy, its just a clock. But as a GHz-level precise timer, it can be used to capture entropy from some other source that does have time-based entropy. (IE: hard drive response rates, or something). Will the hard drive respond in 10,000,000 clock ticks? Or 10,000,020 clock ticks? That's a source of entropy, a bad one, but maybe you have enough hard drive I/O going on that you can get enough entropy to boot up Systemd. EDIT: Hmm... hard drives may not be on all systems. In that case, DRAM-latency could possibly be the answer. DRAM refreshes on a regular basis, locking the CPU out temporarily. As such, the amount of time it takes to access data from DRAM is actually somewhat random (in the scope of a GHz clock anyway). Maybe that could be used as an entropy-source of last resort? (as long as RDTSC exists anyway).
- mzs 6y ago
  please don't over complicate this: >you're right, after I've generated many randoms, I was able to get collisions. RDRAND() = 0x0081da17 RDRAND() = 0x0081da17 RDRAND() = 0x0178d2ea RDRAND() = 0x0178d2ea RDRAND() = 0x02a91db5 RDRAND() = 0x02a91db5 RDRAND() = 0x06c4385b RDRAND() = 0x06c4385b RDRAND() = 0x095d1bf8 RDRAND() = 0x095d1bf8 RDRAND() = 0x0990b335 RDRAND() = 0x0990b335 RDRAND() = 0x0ab033e4 RDRAND() = 0x0ab033e4 RDRAND() = 0x0ac21fae RDRAND() = 0x0ac21fae RDRAND() = 0x0d39390b RDRAND() = 0x0d39390b RDRAND() = 0x0df2f5ce RDRAND() = 0x0df2f5ce RDRAND() = 0x109e5c8a RDRAND() = 0x109e5c8a https://github.com/systemd/systemd/issues/18184#issuecomment-758155962 https://github.com/systemd/systemd/issues/18184#issuecomment...
  - dragontamer 6y ago
    >> https://github.com/systemd/systemd/issues/18184#issuecomment-758213876 https://github.com/systemd/systemd/issues/18184#issuecomment... >> If you can, please try it without using the C intrinsic. I have seen multiple compiler bug reports--https://patchwork.ozlabs.org/project/gcc/patch/20120520170426.GA7774@intel.com/ https://patchwork.ozlabs.org/project/gcc/patch/2012052017042... is the easiest to find--that indicate that the intrinsics don't always return the right value. Hypothetically, the sequence discussed there could be a compiler-bug, unrolling the loop and failing to treat the results of RDRAND as volatile. So... yeah. Lets not overcomplicate things, but lets not undercomplicate things either. Without an actual disassembly of the code posted, there's no proof yet.
Aardwolf 6y ago
"flakey" sounds perfect for a random number generator, but I guess that's not what they mean here
effie 6y ago
RDRAND, Linux and systemd is so weird grotesque neverending story. Standard Linux distributions still fail to boot on mainstream hardware several years after it was discovered that userspace requiring super random numbers from kernel to boot is a dumb idea. Torvalds seems to understand this, but now there is only so much he can do. Poettering's stance seems so oblivious (get a better hardware or fix from manufacturer), one wonders whether he should leave maintaining/programming to people who actually care about users. It has been accepted standard for decades that software should work around bugs in hardware and it makes sense - it is much easier to change software than to fix hardware. So glad my CPU does not have RDRAND and operating systems I use do not require super random numbers to boot.
- segfaultbuserr 6y ago
  > Poettering's stance seems so oblivious (get a better hardware or fix from manufacturer) or just disable rdrand.
  - bcrl 6y ago
    Or just use a piece of software for which fragility isn't the modus operandi of the author. Oh wait, we don't have that choice in any of the mainstream distros.
    - segfaultbuserr 6y ago
      As I explained in this comment, RDRAND was a reasonable decision back then. https://news.ycombinator.com/item?id=25737894 https://news.ycombinator.com/item?id=25737894
      - bcrl 6y ago
        It's the fragility that I'm opposed to. As a kernel developer, your code must be robust against all failure modes. systemd is not robust against all failure modes. I don't see why he's getting a pass for writing fragile code.
chmod775 6y ago
Can HN just collectively hold their horses please? That goes for any tech-journalists who may be reading this too. That github issue contains a bunch of people who aren't even sure whether their methodology makes any sense, yet HN is already taking the title as a foregone conclusion. In fact the current consensus on that github issue (for probably the next 10 minutes) is "it seems RDRAND is fine after all". Let the poor people conclude their research before you go and make a big deal out of it. You will cause them great embarrassment if it turns out the cause was something less spectacular. Thank you.
- Avamander 6y ago
  I took it from Poettering's own tweet, sorry if that's misleading. I guess we'll see what the end result is and it'll get fixed thanks to all the attention.
- lostmsu 6y ago
  EDIT: From this point people explain that there's no issue with RDRAND (but may be with systemd), and users don't understand birthday paradox: https://github.com/systemd/systemd/issues/18184#issuecomment-758243271 https://github.com/systemd/systemd/issues/18184#issuecomment...
  - holtalanm 6y ago
    i mean, it really isn't hard to scroll down, but hey, here ya go: https://github.com/systemd/systemd/issues/18184#issuecomment-758258973 https://github.com/systemd/systemd/issues/18184#issuecomment... https://github.com/systemd/systemd/issues/18184#issuecomment-758260911 https://github.com/systemd/systemd/issues/18184#issuecomment... https://github.com/systemd/systemd/issues/18184#issuecomment-758271053 https://github.com/systemd/systemd/issues/18184#issuecomment... edit: this was in reply to the _original_ parent comment, before it was edited, where they asked for a link to the comments stating RDRAND looked fine.
    - chmod775 6y ago
      Their (holtalanm) parent comment just got edited from a question to a statement, and now this looks really out of context.
lrossi 6y ago
Some questions that come to mind: 1. Why does systemd implement its own random number generation code? https://github.com/systemd/systemd/blob/27231dc14c925935b6698b689da76980eeabf6cf/src/basic/random-util.c#L160 https://github.com/systemd/systemd/blob/27231dc14c925935b669... 2. systemd uses RDRAND to generate UUIDs only. Why isn’t the algorithm robust to collisions? https://github.com/systemd/systemd/issues/18184#issuecomment-758119633 https://github.com/systemd/systemd/issues/18184#issuecomment... 3. Why do you have to pass systemd args on the kernel command line? https://github.com/systemd/systemd/issues/18184#issuecomment-758089410 https://github.com/systemd/systemd/issues/18184#issuecomment... 4. This is not the first time this happens. Why weren’t they able to provide a solution in 2 years? https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1835809 https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/18358...
- chmod775 6y ago
  1. That's not random number generation code, that's code that uses random number generators. 2. What algorithm? It's just RDRAND. 3. Where else would you pass them? 4. What do you mean 'they'? CPU manufacturers? You might also be interested in this comment: https://github.com/systemd/systemd/blob/bcac754d66374782a85a8de39fd543c1eaf9fa82/src/basic/random-util.c#L40-L98 https://github.com/systemd/systemd/blob/bcac754d66374782a85a...
  - lrossi 6y ago
    Not sure if you followed the links I posted, but as Poettering said, the problem is that there are collisions in UUIDs. UUID generation should be robust to random number collisions.
    - chmod775 6y ago
      So you actually want them to wrap RDRAND in their own random number generator? > UUID generation should be robust to random number collisions. The UUID standards say that nowhere at all, it is only acknowledged that collisions may happen even if they are unlikely.
captaincrowbar 6y ago
tl,dr: (1) There's no evidence of anything wrong with AMD's RNG. (2) Lots of people on Github and HN don't understand the birthday paradox.
- rurban 6y ago
  Even AMD admits their rdrand is broken and offers patches. Test a bad AMD system by yourself. I have such a system, unpatched, and it's atrocious. Slow and Bad. But more atrocious is that systemd keeps on using this possible backdoor. It's bad and slow, most other LCG's are miles better. https://rurban.github.io/dieharder/QUALITY.html https://rurban.github.io/dieharder/QUALITY.html https://github.com/rurban/dieharder/blob/master/libdieharder/rng_rdrand.c https://github.com/rurban/dieharder/blob/master/libdieharder... using the intrinsic, not inline asm
lostmsu 6y ago
@dang the title should be changed to "Systemd fails to boot on some Ryzen machines". As discussed in the issue itself, there's no apparent problem with RDRAND on Ryzen.
jeroenhd 6y ago
Why does systemd need super random, unique numbers? Why can't it fall back on something like a PRNG until /dev/(u)random gains enough entropy? The original crash report generates an error for a file name already existing. A simple PRNG algorithm, perhaps seeded with a "random" RDRAND number, should fit that purpose easily, shouldn't it? A 32 bit random number, as used to generate the examples, isn't good enough to guarantee uniqueness anyway. It seems like the original reporter's specific CPU silicon or their loaded microcode does have a problem (because the boot fails on that particular chip so often) but the general dependency on RDRAND continues to confuse me.

119 points by Avamander 6y ago

Given that rdrand is so flakey across systems, is there a reason why systemd even bothers using it? The comments suggest it's only used to generate uuids, which makes me think it's not performance sensitive code. If that's the case why increase code complexity and expose yourself to hardware flakiness just to save a few miliseconds?

okl 6y ago
Possibly because it helps to bootstrap the entropy pool, see for example these ramblings on the topic: https://daniel-lange.com/archives/152-Openssh-taking-minutes-to-become-available,-booting-takes-half-an-hour-...-because-your-server-waits-for-a-few-bytes-of-randomness.html https://daniel-lange.com/archives/152-Openssh-taking-minutes...
- espadrine 6y ago
  Isn’t that an even bigger problem, though, if Linux’ entropy pool gets polluted by an inentropic source that still increments its entropy counter? I recall discussions to maybe rely on something like jitter entropy when the pool is empty and getentropy() is called.
  - garaetjjte 6y ago
    >on something like jitter entropy when the pool is empty and getentropy() is called This ended up being implemented in 5.4 kernel: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=50ee7529ec45 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
  - altfredd 6y ago
    Linux entropy pool is designed to accept and mix different low-quality entropy sources and has explicit workarounds for problems like these. Systemd is literally the only software, that has this problem. I am not aware of any other software, that uses rdrand and expects high-quality cryptography-grade randomness. Precisely, because is does not work. Intel CPUs used to have very similar issues with rdrand and so did AMD. Furthermore, CPU implementation of rdrand is a very attractive targets for state backdoors, so most sensible developers either follow the "GNUPG way" (ask for randomness from user) or simply read from /dev/random. The rdrand instruction is great for games, because it allows to make white noise without complex algorithms and system call overhead. It is also handy for few situations, like interrupt handlers, when you needs to create some semi-random value without using stack space or calling into outside code. Unfortunately, when it was introduced, rdrand was documented to generate "cryptographically strong random numbers" (did Intel developers ever knew, what that means?) Of course, most actual cryptography experts didn't buy into that. As a consequence, and because multi-platform software needed to have it's own RNG anyway, the instruction remained largely unused for actual cryptography. Unused = untested and sometimes broken. If I were in systemd developer's place, I would not hinge bootability of my systems on something like that.
    - ficklepickle 6y ago
      It's an early boot problem. If they tried to use the kernel facilities it would block, making boot take much longer. Fixed in recent kernel versions. I know it's in vogue to rip on systemd, but let's at least try to be fair.
      - freeone3000 6y ago
        Somehow sysv init never ran into this issue. Maybe it's okay to let the kernel seed its entropy pool before initializing your hash table.
        Foxboron 6y ago
        I don't think sysvinit made use of hashtables, and if they did they would probably hit the same snag.
        aidenn0 6y ago
        sysvinit (or rather RC systems often built on top of it) tended to use trees (most often the filesystem). Unlike hash-tables, trees don't need randomness to guarantee asymptotic behavior.
    - Dylan16807 6y ago
      > The rdrand instruction is great for games, because it allows to make white noise without complex algorithms and system call overhead. Eh. You're looking at hundreds of cycles per use. That makes it slower than a secure software RNG, let alone an insecure one.
      - jcelerier 6y ago
        Right ! For games if it's for e.g. texture noise or things like that you shouldn't need more than a couple xors and shifts
    - tadfisher 6y ago
  - CJefferson 6y ago
    When it comes to random seeding, I've always thought the best option would be to use every source available, and hope that at least one is high quality. If we combine: * RDRAND * Seed from previous boot * Jitter * Result of every sensor (for example temperature) attached to the machine * Seed from hypervisor (if we are in a VM) * Time & Date * Any information about the physical computer we are on (attached hardware)
recursive 6y ago
Who's spending multiple milliseconds generating uuids? I'm sure it happens somewhere, somehow, but that does not seem like a common occurrence.
stefan_ 6y ago
It's really ridiculous of AMD to keep fucking this up, even if the backdoor thing never stuck for RDRAND, the fact that one CPU vendor can't manage to make the thing not return identical values many many times over sure might. Similar story for systemd of course, nobody would expect them to learn their lesson. How nice of an init system to keep your system dead because it apparently needs CPRNG generated job ids of all things.
- temac 6y ago
  Given this is broken over and over again in successives AMD chips, this is actually a little bit likely to be a backdoor. People will e.g. generate keys with broken rdrand, and AMD will later fix it with plausible deniability, just saying "oops, sorry, it was just our silly usual mistake, and we don't even promise we won't do it again, lol!". In the meantime the NSA will enjoy the resulting compromises. And well, even if it was just the same silly mistake made over and over again, the NSA is likely to still enjoy the results? So it does not really matter if it was intentional or not: regardless of the root cause of this bug, it should now be considered that rdrand should never be trusted (unless you truly don't care that your systems risk to randomly break or you risk to randomly generate compromised keys) and its mere usage (except maybe indirectly in mitigated ways designed by experts and audited for years) considered a very likely security vulnerability.
altfredd 6y ago
It seems, that many RedHat developers have rather unique world views. When PulseAudio introduces random crackles and scratches in sound output, they claim, that I have a misbehaving sound card. It does not "misbehave", when I use the same applications with ALSA, but whatever. When rdrand fails to produce high-quality entropy — again and again, over and over — Poettering complains about buggy CPUs. I hope, that this particular group never ends up working on network hardware.
- Avamander 6y ago
  > It does not "misbehave", when I use the same applications with ALSA, but whatever. I mean RDRAND is a great example here, if you never use it, you never encounter these bugs. It's not impossible that PulseAudio tries to play higher quality audio or simply uses the device differently. > I hope, that this particular group never ends up working on network hardware. I'd kinda love it, probably quite a few fun hardware bugs will be found.
  - Dylan16807 6y ago
    I'm reminded of this story about hidden network card bugs. Working in a bunch of tests doesn't mean something is correct, especially when it's complex. https://devblogs.microsoft.com/oldnewthing/20050512-48/?p=35653 https://devblogs.microsoft.com/oldnewthing/20050512-48/?p=35...
  - vetinari 6y ago
    > It's not impossible that PulseAudio tries to play higher quality audio or simply uses the device differently. Pulseaudio mixes sound from several applications, unlike Alsa. Alsa just configures your soundcard for that one stream, Pulseaudio has no such luxury. In order to mix relatively small buffers (because latency), it needs to do timing. If your soundcard has problem with timing (and many do!), you will get crackling. The good news is, that the problem with timing can be only at specific frequencies; in the past, I had a computer with Nvidia Ion and it did this with 44,1kHz audio. With 48kHz, everything was OK. So fixing the output at 48kHz by forcing Pulse to mix at this frequency fixed the crackling.
tadfisher 6y ago
It's not systemd's job to fix hardware bugs, it's the kernel's (or AMD's). The fix for the last time this happened was to mask the offending CPU IDs in the kernel's RNG from using RDRAND. systemd did have a temporary hack in place for their direct RDRAND implementation, but this current bug isn't the same (not just emitting 0 like last time, but eventually emitting duplicate values in pairs).
- klmadfejno 6y ago
  That doesn't seem to answer the question of why use it
  - tadfisher 6y ago
    If they don't use it, the kernel does in order to initialize the entropy pool, which will also result in broken boots. To work around historically flaky kernel implementations, systemd calls RDRAND directly if it detects it needs to with a heuristic. This is why the fix for the current issue is to set both 'nordrand' and 'SYSTEMD_RDRAND=0'.
    - tedunangst 6y ago
      What happened to it's not systemd's job to fix kernel bugs?
      - tadfisher 6y ago
        I didn't type that, so I'm not sure what you're referring to. systemd is fundamentally tied to Linux and thus needs to implement workarounds from time-to-time when kernel facilities are lacking. This is complicated by the requirement that systemd continue working on older Linux versions.
  - numpad0 6y ago
    It generally makes more sense to trust CPU specifications and utilize it fully, than to assume everything is tampered with and only ever use `mov`
AshamedCaptain 6y ago
And why do you even need globally UUIDs here? What's wrong with a plain old sequence counter or the like?
- stragies 6y ago
  Among others is that predictability of behavior of systems makes analysis and/or attack easier. Also "plain old sequence counters" in clusters of (heterogeneous) hardware makes distributing (floating) tasks more challenging. There are a bunch of other reasons.
  - effie 6y ago
    How is using UUIDs making software more predictable than using sequence numbers?
    - stragies 6y ago
      the opposite: UUIDs make things less predictable. And in many situations this is very good.
garaetjjte 6y ago
As much as I don't like systemd this probably was reasonable decision back at the time. systemd needs entropy for generating some random identifiers during early boot. Trying to read it from kernel random facilities might block infinitely until kernel 5.4. Before that pulling entropy from RDRAND might not be that unreasonable decision, nowadays it probably should just call getrandom().
- tremon 6y ago
  But why does systemd need random identifiers during early boot? I can understand the need for seeding a hash table (but even there, they already admit that the code can re-seed at any time), but I don't see why a boot program should rely on random id generation in the first place.
LukeShu 6y ago
The source code has a lengthy comment explaining why. https://github.com/systemd/systemd/blob/bcac754d66374782a85a8de39fd543c1eaf9fa82/src/basic/random-util.c#L40-L98 https://github.com/systemd/systemd/blob/bcac754d66374782a85a...
- robocat 6y ago
  int rdrand(unsigned long *ret) { /* So, you are a "security researcher", and you wonder why we bother with using raw RDRAND here, * instead of sticking to /dev/urandom or getrandom()? * * Here's why: early boot. On Linux, during early boot the random pool that backs /dev/urandom and * getrandom() is generally not initialized yet. It is very common that initialization of the random * pool takes a longer time (up to many minutes), in particular on embedded devices that have no * explicit hardware random generator, as well as in virtualized environments such as major cloud * installations that do not provide virtio-rng or a similar mechanism. * * In such an environment using getrandom() synchronously means we'd block the entire system boot-up * until the pool is initialized, i.e. *very* long. Using getrandom() asynchronously (GRND_NONBLOCK) * would mean acquiring randomness during early boot would simply fail. Using /dev/urandom would mean * generating many kmsg log messages about our use of it before the random pool is properly * initialized. Neither of these outcomes is desirable. * * Thus, for very specific purposes we use RDRAND instead of either of these three options. RDRAND * provides us quickly and relatively reliably with random values, without having to delay boot, * without triggering warning messages in kmsg. * * Note that we use RDRAND only under very specific circumstances, when the requirements on the * quality of the returned entropy permit it. Specifically, here are some cases where we *do* use * RDRAND: * * • UUID generation: UUIDs are supposed to be universally unique but are not cryptographic * key material. The quality and trust level of RDRAND should hence be OK: UUIDs should be * generated in a way that is reliably unique, but they do not require ultimate trust into * the entropy generator. systemd generates a number of UUIDs during early boot, including * 'invocation IDs' for every unit spawned that identify the specific invocation of the * service globally, and a number of others. Other alternatives for generating these UUIDs * have been considered, but don't really work: for example, hashing uuids from a local * system identifier combined with a counter falls flat because during early boot disk * storage is not yet available (think: initrd) and thus a system-specific ID cannot be * stored or retrieved yet. * * • Hash table seed generation: systemd uses many hash tables internally. Hash tables are * generally assumed to have O(1) access complexity, but can deteriorate to prohibitive * O(n) access complexity if an attacker manages to trigger a large number of hash * collisions. Thus, systemd (as any software employing hash tables should) uses seeded * hash functions for its hash tables, with a seed generated randomly. The hash tables * systemd employs watch the fill level closely and reseed if necessary. This allows use of * a low quality RNG initially, as long as it improves should a hash table be under attack: * the attacker after all needs to trigger many collisions to exploit it for the purpose * of DoS, but if doing so improves the seed the attack surface is reduced as the attack * takes place. * * Some cases where we do NOT use RDRAND are: * * • Generation of cryptographic key material * * • Generation of cryptographic salt values * * This function returns: * * -EOPNOTSUPP → RDRAND is not available on this system * -EAGAIN → The operation failed this time, but is likely to work if you try again a few * times * -EUCLEAN → We got some random value, but it looked strange, so we refused using it. * This failure might or might not be temporary. */
zuzun 6y ago
> just to save a few miliseconds? RDRAND is rather slow, especially with the SRBDS microcode update. One instruction takes 1,200 ns on my system, that's 6.6 MB/s. A software implementation of a CSPRNG will do hundreds of MB/s, the general purpose PRNGs do several GB/s.

HarryHirsch 6y ago

Make that: "systemd and RDRAND on Ryzen 9 is flakey"

Dylan16807 6y ago
I'm sure a lot of other things are being affected by this defect, just less visibly. It's not like the software is breaking x86 instructions somehow.
- HarryHirsch 6y ago
  Someone please explain why someone would use the hardware random generator directly. Everyone else uses /dev/{u,}random because somehow HRNGs always manage to be buggy, and then there's the concern about NSA involvement. Using OS-provided interface mitigates either concern.
  - mjg59 6y ago
    https://github.com/systemd/systemd/blob/bcac754d66374782a85a8de39fd543c1eaf9fa82/src/basic/random-util.c#L40-L98 https://github.com/systemd/systemd/blob/bcac754d66374782a85a...
    - HarryHirsch 6y ago
      Dude is stubborn in the face of reality. It's not as if there hadn't been RDRAND hardware bugs before.
    - zamadatix 6y ago
      Not sure I understand the /dev/urandom concern in that comment. If this doesn't need to be cryptographically good and you don't want logspam every time a new number is needed why not seed this once with /dev/urandom. Surely a single log message is better than trying to reinvent the wheel, especially since this isn't the first time I've seen this particular code linked on HN due to the assumptions not holding true.

xeeeeeeeeeeenu 6y ago

Why does this keep happening? RDRAND was broken in Ryzen 3xxx[1] and Jaguar/Puma[2] too. [1] - https://arstechnica.com/gadgets/2019/10/how-a-months-old-amd-microcode-bug-destroyed-my-weekend/ https://arstechnica.com/gadgets/2019/10/how-a-months-old-amd... [2] - https://linuxreviews.org/RDRAND_stops_returning_random_values_on_older_AMD_CPUs_after_suspend https://linuxreviews.org/RDRAND_stops_returning_random_value...

stabbles 6y ago
I remember I ran into this Ryzen problem too. Spent a long time trying to figure out a bug in my code (random numbers were deterministic between runs), which ultimately was fixed by upgrading the bios.
Avamander 6y ago
> Why does this keep happening? Windows doesn't use RDRAND.

dschuetz 6y ago

systemd: we'll fork our own rdrand

uncledave 6y ago
Headline in two years “Pwnie award for systemd-rdrand”
LukeShu 6y ago
systemd: It's not our job to hack around hardware bugs, it's the kernel's. We will not implement our own thing. HN: How dare you.[1] Also HN: Criticizes systemd for implementing its own thing Systemd just can't win. [1] https://news.ycombinator.com/item?id=10999335 https://news.ycombinator.com/item?id=10999335
- dschuetz 6y ago
  They can. Just fork Linux and do their own thing, with blackjack and hookers.
- na85 6y ago
  apt: Package Gnome3-Shell depends on systemd-rdrand which is not installed. Package systemd-rdrand conflicts gnome3-shell. Uninstall 1230982 packages to resolve? [y/n]

gbrown_ 6y ago

Not to be snarky but OpenBSD's random system has always seemed very elegant to me and seems like such a system could avoid this failure mode in systemd, video[1], slides[2]. That said it's good that the faulty RDRAND was discovered. As pointed out this isn't the first time processors (AMD particular) have had such issues. Do we need to just be skeptical of this and run RDRAND tests every time a new processor comes out? Or perhaps even each microcode update? [1] https://www.youtube.com/watch?v=aWmLWx8ut20 https://www.youtube.com/watch?v=aWmLWx8ut20 [2] https://www.openbsd.org/papers/hackfest2014-arc4random/index.html https://www.openbsd.org/papers/hackfest2014-arc4random/index...

segfaultbuserr 6y ago
> seems like such a system could avoid this failure mode in systemd, video[1], slides[2] Yes and no. Yes, OpenBSD is not vulnerable to this failure mode. But no, it's only because OpenBSD made assumptions that systemd couldn't. The problem is not how well the random system performs once it started, but simply what to do when it's not initialized yet during early boot. From Page 20: Kernel initialization from boot, OpenBSD's initialization sequence is: 1. If available, use rdrand to generate random seeds. 2. Read random seed file from the disk, saved from the previous boots or installation, very early by the bootloader. 3. Read the Stack Protector cookie from the kernel binary, by the bootloader. 4. Mix them together. The pool is initialized at this point. 5. Keep collecting entropy from various sources of randomness, such as interrupts. Nice, it ensures that you always get a initialized entropy pool, and hides all complexity away from the userspace. But do note that (2) and (3) provides no security during the first early boot if the same OS image is replicated across many machines. --- Now what about Linux? 1. If available... - use rdrand to generate random seeds. It's optional and can be disabled by the user. - use hardware random number generators and TPMs, but only if they're compiled into the kernel (or loaded early enough) and trusted by the user. - use the in-kernel jitter entropy collector, "a modern out-of-order CPU, even quite simple loops show a fair amount of hard-to-predict timing variability." But only available since Linux 5.3 released in 2019. 2. Read random seed file from the disk, saved from the previous boots. But it's the responsibility of the userspace to do that, this is problematic for systemd. - systemd needs a source of randomness at an earlier time, before the entropy pool is seeded by the file. How did OpenBSD solve it? Instead of loading it after boot in userspace, OpenBSD loads the file early, really early, via the bootloader. - systemd doesn't trust the file. The file is read, but by default, its entropy is not credited, because systemd doesn't want to take the responsibility if someone accidentally replicated the random file across millions of machines via a system image (OpenBSD seems to be okay with the lack of protection of image replication, but systemd is more cautious, probably because it has zero control over the rest of the system). As a result, the system may block at step 4 (entropy collection) during boot for a long time. 3. If Latent Entropy GCC plugin is used, Linux kernel can use entropy embedded in the kernel binary. It's a creative innovation by PaX/grsec, it uses a random seed inserted at build time, but also inserts local variables in every marked function, so that different runtime code paths and control flows create different entropy seeds. But it's only an optional feature and almost nobody uses it - PaX/grsec ideas are too radical to most people ;-) - Note that it's security properties can still be seriously weakened by a replicated kernel binary. Also, it's entropy is seeded to the pool, but it's also not trusted and credited, since it's considered a workaround, not a solution. 4. Keep collecting entropy from various sources of randomness, such as interrupts. As you can see, it's actually really similar to OpenBSD (I meant the concepts, not the implementation), the only difference is that almost everything is optional and nothing is guaranteed to work. --- So as a tradeoff, systemd uses the following logic instead. 1. If available, bypass the kernel, use rdrand directly for non-crypto randomness. 2. For everything else (no rdrand machines & crypto), use the system's entropy pool, it may block for a long time before the pool is initialized, even with a random seed file or Latent Entropy, because their entropy is uncredited. I think this is a reasonable tradeoff, but if rdrand is broken, everything breaks down. I think the differences between OpenBSD and Linux is basically due to how much control you have over the operating system. OpenBSD has the advantage you have when you're building an entire operating system, not just a kernel. --- Update: As pointed out by the comment, systemd supports bootloader entropy too, but only uses it via systemd-boot on EFI systems since it's not vulnerable to the image replication problem. With UEFI, it can combine the seed file with a machine-specific EFI variable ("system token") in UEFI's NVRAM generated during system installation. Systemd also criticizes NetBSD's bootloader entropy for being vulnerable to the image replication problem, see https://systemd.io/RANDOM_SEEDS/ https://systemd.io/RANDOM_SEEDS/ > This is boring: NetBSD had boot loader entropy seed support since ages! > Yes, NetBSD has that, and the above is inspired by that (note though: this article is about a lot more than that). NetBSD’s support is not really safe, since it neither updates the random seed before using it, nor has any safeguards against replicating the same disk image with its random seed on multiple machines (which the ‘system token’ mentioned above is supposed to address). This means reuse of the same random seed by the boot loader is much more likely. Update 2: Rewrite the comment entirely.

nitrobeast 6y ago

Functions that return random results do not fit into the most common testing paradigms. I have seen people try to verify the distribution of random results, and even that could have concerns for theoretical flakiness. I wonder if the nature of the function contributed to the apparent lack of testing coverage here.

dragontamer 6y ago

Hmmm... Well, first problem is using RDRAND in this case. RDRAND isn't meant for seeds, its for a random number. If you're seeding something, you're supposed to use RDSEED, which has more guarantees about seeding / entropy behavior. I don't think RDRAND was ever intended to be used as an entropy source. (Think of it this way: RDSEED is kinda like /dev/random, while RDRAND is kinda-like /dev/urandom) ------ RDRAND is an older instruction than RDSEED. So that then brings up the question: what is the proper startup behavior of systems without RDSEED but with RDRAND? Well, since RDRAND never had entropy guarantees, I guess its up to other sources of entropy (see clock-entropy, such as RDTSC). RDTSC isn't a source of entropy, its just a clock. But as a GHz-level precise timer, it can be used to capture entropy from some other source that does have time-based entropy. (IE: hard drive response rates, or something). Will the hard drive respond in 10,000,000 clock ticks? Or 10,000,020 clock ticks? That's a source of entropy, a bad one, but maybe you have enough hard drive I/O going on that you can get enough entropy to boot up Systemd. EDIT: Hmm... hard drives may not be on all systems. In that case, DRAM-latency could possibly be the answer. DRAM refreshes on a regular basis, locking the CPU out temporarily. As such, the amount of time it takes to access data from DRAM is actually somewhat random (in the scope of a GHz clock anyway). Maybe that could be used as an entropy-source of last resort? (as long as RDTSC exists anyway).

mzs 6y ago
please don't over complicate this: >you're right, after I've generated many randoms, I was able to get collisions. RDRAND() = 0x0081da17 RDRAND() = 0x0081da17 RDRAND() = 0x0178d2ea RDRAND() = 0x0178d2ea RDRAND() = 0x02a91db5 RDRAND() = 0x02a91db5 RDRAND() = 0x06c4385b RDRAND() = 0x06c4385b RDRAND() = 0x095d1bf8 RDRAND() = 0x095d1bf8 RDRAND() = 0x0990b335 RDRAND() = 0x0990b335 RDRAND() = 0x0ab033e4 RDRAND() = 0x0ab033e4 RDRAND() = 0x0ac21fae RDRAND() = 0x0ac21fae RDRAND() = 0x0d39390b RDRAND() = 0x0d39390b RDRAND() = 0x0df2f5ce RDRAND() = 0x0df2f5ce RDRAND() = 0x109e5c8a RDRAND() = 0x109e5c8a https://github.com/systemd/systemd/issues/18184#issuecomment-758155962 https://github.com/systemd/systemd/issues/18184#issuecomment...
- dragontamer 6y ago
  >> https://github.com/systemd/systemd/issues/18184#issuecomment-758213876 https://github.com/systemd/systemd/issues/18184#issuecomment... >> If you can, please try it without using the C intrinsic. I have seen multiple compiler bug reports--https://patchwork.ozlabs.org/project/gcc/patch/20120520170426.GA7774@intel.com/ https://patchwork.ozlabs.org/project/gcc/patch/2012052017042... is the easiest to find--that indicate that the intrinsics don't always return the right value. Hypothetically, the sequence discussed there could be a compiler-bug, unrolling the loop and failing to treat the results of RDRAND as volatile. So... yeah. Lets not overcomplicate things, but lets not undercomplicate things either. Without an actual disassembly of the code posted, there's no proof yet.

Aardwolf 6y ago

"flakey" sounds perfect for a random number generator, but I guess that's not what they mean here

effie 6y ago

RDRAND, Linux and systemd is so weird grotesque neverending story. Standard Linux distributions still fail to boot on mainstream hardware several years after it was discovered that userspace requiring super random numbers from kernel to boot is a dumb idea. Torvalds seems to understand this, but now there is only so much he can do. Poettering's stance seems so oblivious (get a better hardware or fix from manufacturer), one wonders whether he should leave maintaining/programming to people who actually care about users. It has been accepted standard for decades that software should work around bugs in hardware and it makes sense - it is much easier to change software than to fix hardware. So glad my CPU does not have RDRAND and operating systems I use do not require super random numbers to boot.

segfaultbuserr 6y ago
> Poettering's stance seems so oblivious (get a better hardware or fix from manufacturer) or just disable rdrand.
- bcrl 6y ago
  Or just use a piece of software for which fragility isn't the modus operandi of the author. Oh wait, we don't have that choice in any of the mainstream distros.
  - segfaultbuserr 6y ago
    As I explained in this comment, RDRAND was a reasonable decision back then. https://news.ycombinator.com/item?id=25737894 https://news.ycombinator.com/item?id=25737894
    - bcrl 6y ago
      It's the fragility that I'm opposed to. As a kernel developer, your code must be robust against all failure modes. systemd is not robust against all failure modes. I don't see why he's getting a pass for writing fragile code.

chmod775 6y ago

Can HN just collectively hold their horses please? That goes for any tech-journalists who may be reading this too. That github issue contains a bunch of people who aren't even sure whether their methodology makes any sense, yet HN is already taking the title as a foregone conclusion. In fact the current consensus on that github issue (for probably the next 10 minutes) is "it seems RDRAND is fine after all". Let the poor people conclude their research before you go and make a big deal out of it. You will cause them great embarrassment if it turns out the cause was something less spectacular. Thank you.

Avamander 6y ago
I took it from Poettering's own tweet, sorry if that's misleading. I guess we'll see what the end result is and it'll get fixed thanks to all the attention.
lostmsu 6y ago
EDIT: From this point people explain that there's no issue with RDRAND (but may be with systemd), and users don't understand birthday paradox: https://github.com/systemd/systemd/issues/18184#issuecomment-758243271 https://github.com/systemd/systemd/issues/18184#issuecomment...
- holtalanm 6y ago
  i mean, it really isn't hard to scroll down, but hey, here ya go: https://github.com/systemd/systemd/issues/18184#issuecomment-758258973 https://github.com/systemd/systemd/issues/18184#issuecomment... https://github.com/systemd/systemd/issues/18184#issuecomment-758260911 https://github.com/systemd/systemd/issues/18184#issuecomment... https://github.com/systemd/systemd/issues/18184#issuecomment-758271053 https://github.com/systemd/systemd/issues/18184#issuecomment... edit: this was in reply to the _original_ parent comment, before it was edited, where they asked for a link to the comments stating RDRAND looked fine.
  - chmod775 6y ago
    Their (holtalanm) parent comment just got edited from a question to a statement, and now this looks really out of context.

lrossi 6y ago

Some questions that come to mind: 1. Why does systemd implement its own random number generation code? https://github.com/systemd/systemd/blob/27231dc14c925935b6698b689da76980eeabf6cf/src/basic/random-util.c#L160 https://github.com/systemd/systemd/blob/27231dc14c925935b669... 2. systemd uses RDRAND to generate UUIDs only. Why isn’t the algorithm robust to collisions? https://github.com/systemd/systemd/issues/18184#issuecomment-758119633 https://github.com/systemd/systemd/issues/18184#issuecomment... 3. Why do you have to pass systemd args on the kernel command line? https://github.com/systemd/systemd/issues/18184#issuecomment-758089410 https://github.com/systemd/systemd/issues/18184#issuecomment... 4. This is not the first time this happens. Why weren’t they able to provide a solution in 2 years? https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1835809 https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/18358...

chmod775 6y ago
1. That's not random number generation code, that's code that uses random number generators. 2. What algorithm? It's just RDRAND. 3. Where else would you pass them? 4. What do you mean 'they'? CPU manufacturers? You might also be interested in this comment: https://github.com/systemd/systemd/blob/bcac754d66374782a85a8de39fd543c1eaf9fa82/src/basic/random-util.c#L40-L98 https://github.com/systemd/systemd/blob/bcac754d66374782a85a...
- lrossi 6y ago
  Not sure if you followed the links I posted, but as Poettering said, the problem is that there are collisions in UUIDs. UUID generation should be robust to random number collisions.
  - chmod775 6y ago
    So you actually want them to wrap RDRAND in their own random number generator? > UUID generation should be robust to random number collisions. The UUID standards say that nowhere at all, it is only acknowledged that collisions may happen even if they are unlikely.

captaincrowbar 6y ago

tl,dr: (1) There's no evidence of anything wrong with AMD's RNG. (2) Lots of people on Github and HN don't understand the birthday paradox.

rurban 6y ago
Even AMD admits their rdrand is broken and offers patches. Test a bad AMD system by yourself. I have such a system, unpatched, and it's atrocious. Slow and Bad. But more atrocious is that systemd keeps on using this possible backdoor. It's bad and slow, most other LCG's are miles better. https://rurban.github.io/dieharder/QUALITY.html https://rurban.github.io/dieharder/QUALITY.html https://github.com/rurban/dieharder/blob/master/libdieharder/rng_rdrand.c https://github.com/rurban/dieharder/blob/master/libdieharder... using the intrinsic, not inline asm

lostmsu 6y ago

@dang the title should be changed to "Systemd fails to boot on some Ryzen machines". As discussed in the issue itself, there's no apparent problem with RDRAND on Ryzen.

jeroenhd 6y ago

Why does systemd need super random, unique numbers? Why can't it fall back on something like a PRNG until /dev/(u)random gains enough entropy? The original crash report generates an error for a file name already existing. A simple PRNG algorithm, perhaps seeded with a "random" RDRAND number, should fit that purpose easily, shouldn't it? A 32 bit random number, as used to generate the examples, isn't good enough to guarantee uniqueness anyway. It seems like the original reporter's specific CPU silicon or their loaded microcode does have a problem (because the boot fails on that particular chip so often) but the general dependency on RDRAND continues to confuse me.