6 ms·

Ubuntu 25.10's Rust Coreutils Is Causing Major Breakage for Some Executables

97 points by dabinat 1y ago

fn-mote 1y ago
Can someone post details about why md5sum from the Rust Coreutils is producing different results from GNU Coreutils. The post does not claim this is a bug. (Surprisingly.)
- treesknees 1y ago
  The end of the post links to the bug tracker pointing out it’s already being tracked. But it actually appears to be an issue with dd, not md5sum - https://github.com/VirtualBox/virtualbox/issues/226#issuecomment-3335037186 https://github.com/VirtualBox/virtualbox/issues/226#issuecom...
  - zahlman 1y ago
    > Out of the box, script will pass bs= option to dd for it to be aware of how much to skip from the beginning of input data (and on later while loop iterations). This seem to have handled by dd either improperly or at least in a different way than it was in the past (with GNU core utils). However, once bs= is replaced with ibs=, all seems to go back to normal. The bs/ibs/obs options don't "skip" anything, but determine how much data to buffer in memory at a time while transferring. Regardless, it's hard to fathom how something this simple got messed up, especially considering that the suite supposedly has good test coverage and has been getting close to a full green bar.
    - hulitu 1y ago
      > However, once bs= is replaced with ibs=, all seems to go back to normal. so it is a bug. bs is one thing, ibs is another.
      - zahlman 1y ago
        > bs is one thing, ibs is another. bs=BYTES read and write up to BYTES bytes at a time (default: 512); over‐ rides ibs and obs As described, the script should have worked as is, and the problem is in the handling of the dd options. (But I didn't verify the accuracy of the description.)
        lesuorac 1y ago
        Wonder if `\00` is handled different between them. Not sure how to run the rust version but my md5sum seems to care how many null bytes there are. echo -e "\00" | md5sum 8f7cbbbe0e8f898a6fa93056b3de9c9c - echo -e "\00\00" | md5sum a4dd23550d4586aee3b15d27b5cec433 -
        zahlman 1y ago
        > Wonder if `\00` is handled different between them. `dd` is for copying all the bytes of a source (unless you explicitly set a limit with the `count` option), regardless of whether they're zero. It's fundamentally not for null-terminated strings but arbitrary binary I/O. In fact, "copying from" /dev/zero is a common use case. It seems frankly implausible that the `dd` implementation is just stopping at a null byte; that would break a lot of tests and demonstrate a complete, fundamental misunderstanding of what's supposed to be implemented. > Not sure how to run the rust version but my md5sum seems to care how many null bytes there are. Yes, the md5 algorithm also fundamentally operates on and produces binary data; `md5sum` just converts the bytes to a hex dump at the end. The result you get is expected (edit: modulo hiccuphippo's correct observation), because the correct md5 sum changes with every byte of input, even if that byte has a zero value.
        hiccuphippo 1y ago
        Kind of off-topic, but those commands also add a newline character to the md5sums, giving unexpected results. I was trying it in a php interpreter and getting different values. Add -n to echo to avoid the new line.
  - sionisrecur 1y ago
    And dd is also part of coreutils. So this is still a rust-coreutils issue, or an issue in gnu-coreutils that scripts rely on.
    - kps 1y ago
      Which has inadequate tests and gets it wrong? (‘All of them’ is an option.)
b_e_n_t_o_n 1y ago
Hmm, this plus the performance regressions makes me wonder if it's too soon to move to the rust version of Coreutils. And makes me wonder if this is gonna cause more pushback regarding the rust in the kernel movement.
- mustache_kimono 1y ago
  > this is gonna cause more pushback regarding the rust in the kernel movement. Only among those that don't understand that, if this is a problem, then it is Canonical problem, not a Rust problem. To give another example, Canonical includes ZFS in Ubuntu too. And, for a while, Canonical shipped a broken snapshot mechanism called zsys with Ubuntu too. Canonical ultimately ripped zsys out because it didn't work very well. zsys would choke on more than 4000 snapshots, etc. zsys was developed in Go, while other snapshot systems developed in Perl and Python did a little less and worked better. Now, is zsys a Go problem? Of course not. It wasn't ready because Canonical sometimes ships broken stuff.
  - zahlman 1y ago
    > Only among those that don't understand that, if this is a problem, then it is Canonical problem, not a Rust problem. (This is hard to express in a careful way where I'm confident of not offending anyone. Please take me at my word that I'm not trying to take sides in this at all.) The dominant narrative behind this pushback, as far as I can tell, is nothing to do with the Rust language itself (aside perhaps from a few fringe people who see the adoption of Rust as some kind of signal of non-programming-related politics, and who are counter-signaling). Rather, the opposition is to re-implementing "working" software (including in the sense that nobody seems to have noticed any memory-handling faults all this time) for the sake of seemingly nebulous benefits (like compiler-checked memory safety). The Rust code will probably also be more maintainable by Rust developers than the C code currently is by C developers given the advantages of Rust's language design. (Unless it turns out that the C developers are just intrinsically better at programming and/or software engineering; I'm pretty skeptical of that.) But most long-time C users are probably not going to want to abandon their C expertise and learn Rust. And they seem to outnumber the new Rust developers by quite a bit, at least for now.
deleted 1y ago
[deleted]
sionisrecur 1y ago
As expected! I wonder how many tools depend on bugs or edgecases in GNU Coreutils.
- kpcyrd 1y ago
  Almost as if programming against losely defined command-line interfaces is maybe not a very robust way of programming~
  - yjftsjthsd-h 1y ago
    No, the same thing can and does happen on any API. As an obvious example, there are an annoying number of programs that depend on GNU's libc in particular, and which therefore break when someone tries to compile against ex. musl.
    - kpcyrd 1y ago
      This isn't true, libc is magnitudes better defined than binary names in shell scripting, but it's still yet another case of approximation-API with multiple competing implementations. There are plenty of ecosystems where programs declare a specific library implementation they expect to call into (Rust, Python, Npm, Ruby, Perl, ...) often even constrained by versions. But also if you depend on libcurl you are only going to have to deal with multiple verions of the same implementation (that you can still constrain in e.g. pkg-config). In shell scripting you have to deal with stuff like "in nc, EOF on stdin is undefined behavior and implementation specific".
      - yjftsjthsd-h 1y ago
        I don't see the difference? There's a POSIX spec for coreutils (and some other shell-usable executables), there's a POSIX spec for libc (and some other C libs), and both are a somewhat painful subset of what most programs/scripts actually use. And yes, in both cases often the solution is to explicitly depend on a particular implementation or even version thereof; systemd explicitly only supports glibc, shell scripts targeting macOS may require that you install a newer version of bash than the OS ships, and yes, if you need nc to behave a particular way then you need to depend on the right one (I've actually personally hit that; IIRC I installed netcat-openbsd but it didn't include a feature I needed). In all cases, there may be a portable standard, but it doesn't cover everything, so if you're not inside what's covered by the standard then you should specify your actual dependencies. It still doesn't matter whether the API you're touching is mediated by the link phase of a compiler chain or *sh calling particular binaries.
mongol 1y ago
Someone need to be the first to take the hit, and apparently Ubuntu volunteered
- zahlman 1y ago
  Indeed. They knew there was risk associated with this, which is why they didn't just plop it into the LTS release. If it isn't working acceptably by the 26.04 release window, it'll just get reverted.
  - subsistence234 1y ago
    non-LTS release means beta now?
    - dfc 1y ago
      No, but sort of yes. This is from Ubuntu: "Interim releases will introduce new capabilities from Canonical and upstream open source projects, they serve as a proving ground for these new capabilities." https://ubuntu.com/about/release-cycle https://ubuntu.com/about/release-cycle
- packetlost 1y ago
  They're doing it precisely so they can identify shortcomings and bugs. This is expected and good.
  - 3836293648 1y ago
    It's expected. Good is an entirely different issue. More permissively licenced core components is 100% a bad thing
    - Volundr 1y ago
      > More permissively licenced core components is 100% a bad thing I don't follow, can you explain why?
flykespice 1y ago
[dead]
dmitrygr 1y ago
Hyrum's Law With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.
- Avamander 1y ago
  One more reason to actually reimplement coreutils. Figuring out these undocumented promises also makes it easier to ensure correctness in general.
  - steveklabnik 1y ago
    You’re downvoted but you’re correct. The uutils folks have been submitting bugs and test cases upstream. It actively helps both projects.
- lioeters 1y ago
  Every change breaks someone's workflow. https://xkcd.com/1172/ https://xkcd.com/1172/
samdoesnothing 1y ago
If they would have used a modern, safe programming language like rust those errors would be impossible to... oh wait... Never mind.
- arp242 1y ago
  No one has ever claimed Rust prevents all bugs. This is such a tired strawman trope.
  - tssva 1y ago
    "No one has ever" regarding human actions is quite a bold claim to make in relation to anything.
    - arp242 1y ago
      Prove me wrong. Find someone saying that Rust will prevent all bugs.
      - tssva 1y ago
        I believe the way it generally works is the person that makes a claim has the burden to provide evidence it is true.
  - spacechild1 1y ago
    There a people who claim that "once it compiles, it works". I've seen this quite a few times here on HN. Just a random example: https://news.ycombinator.com/item?id=45046182 https://news.ycombinator.com/item?id=45046182
    - acdha 1y ago
      That’s a far more nuanced comment than you’re portraying it as, especially as it’s appearing for exactly this scenario: the new dd is working as designed, it’s not segfaulting or corrupting data, but its design isn’t identical to the GNU version and that logic error is the kind of thing Rust can’t prevent short of AGI.
arp242 1y ago
As an aside on the GNU *sum tools, I found they're quite slow. A few months ago I wrote a simple replacement in Go for UX reasons and somewhat to my surprise, the Go implementation of most hash algorithms seems about 2 to 4 times as fast when using a simple naïve "obvious" single-threaded implementation. It can be sped up even more by using more than one core. Go has assembly implementations for most hash functions. I didn't really look at the coreutils implementation but I'm guessing it's "just" in C. At any rate, small teething issues aside, long-term things should be better and faster.
- collinfunk 1y ago
  What distribution do you use? GNU Coreutils uses the OpenSSL implementation of hashes by default, but some distributions have disabled it using './configure --with-openssl=no'. Debian used to do this, but now links to OpenSSL.
  - arp242 1y ago
    This is on Void. It doesn't have --with-openssl configure args in the package, although the binary also doesn't link to lib{ssl,crypto}. It probably gets auto-detected to "no"(?)
    - collinfunk 1y ago
      Yep, from a Void Linux container it appears that they do not link to libcrypto: $ ldd /usr/bin/cksum linux-vdso.so.1 (0x00007fb354763000) libc.so.6 => /usr/lib64/libc.so.6 (0x00007fb354549000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb354765000) For context, I am a committer to GNU Coreutils. We have used OpenSSL by default for a few years now [1]. Previously it was disabled by default because the OpenSSL license is not compatible with the GPL license [2]. This was resolved when they switched to the Apache License, Version 2.0 in OpenSSL 3.0.0. If the Void people wanted to enable OpenSSL, they would probably just need to add openssl (or whatever they package it as) to the package dependencies. [1] https://github.com/coreutils/coreutils/commit/0d77e1b7ea2840dbc10994f61894f1a3f64b46f9 https://github.com/coreutils/coreutils/commit/0d77e1b7ea2840... [2] https://www.gnu.org/licenses/license-list.html#OpenSSL https://www.gnu.org/licenses/license-list.html#OpenSSL
lousken 1y ago
it's not LTS release so it's mostly fine, but they still should postpone the release until such bugs are fixed
kpcyrd 1y ago
hopefully they got reported as bugs, I spent some time in June making sure a basic Arch Linux system can compile itself using uutils, and things mostly worked, the only build failures could be argued were shell scripts depending on undefined behavior, like "calling install(1) with multiple -m arguments": https://github.com/uutils/coreutils/issues/8033 https://github.com/uutils/coreutils/issues/8033 The other one was an edge-case with dangling symlinks: https://github.com/uutils/coreutils/issues/8044 https://github.com/uutils/coreutils/issues/8044 Both got fixed promptly, after taking the time to report them properly. :)

97 points by dabinat 1y ago

Can someone post details about why md5sum from the Rust Coreutils is producing different results from GNU Coreutils. The post does not claim this is a bug. (Surprisingly.)

treesknees 1y ago
The end of the post links to the bug tracker pointing out it’s already being tracked. But it actually appears to be an issue with dd, not md5sum - https://github.com/VirtualBox/virtualbox/issues/226#issuecomment-3335037186 https://github.com/VirtualBox/virtualbox/issues/226#issuecom...
- zahlman 1y ago
  > Out of the box, script will pass bs= option to dd for it to be aware of how much to skip from the beginning of input data (and on later while loop iterations). This seem to have handled by dd either improperly or at least in a different way than it was in the past (with GNU core utils). However, once bs= is replaced with ibs=, all seems to go back to normal. The bs/ibs/obs options don't "skip" anything, but determine how much data to buffer in memory at a time while transferring. Regardless, it's hard to fathom how something this simple got messed up, especially considering that the suite supposedly has good test coverage and has been getting close to a full green bar.
  - hulitu 1y ago
    > However, once bs= is replaced with ibs=, all seems to go back to normal. so it is a bug. bs is one thing, ibs is another.
    - zahlman 1y ago
      > bs is one thing, ibs is another. bs=BYTES read and write up to BYTES bytes at a time (default: 512); over‐ rides ibs and obs As described, the script should have worked as is, and the problem is in the handling of the dd options. (But I didn't verify the accuracy of the description.)
      - lesuorac 1y ago
        Wonder if `\00` is handled different between them. Not sure how to run the rust version but my md5sum seems to care how many null bytes there are. echo -e "\00" | md5sum 8f7cbbbe0e8f898a6fa93056b3de9c9c - echo -e "\00\00" | md5sum a4dd23550d4586aee3b15d27b5cec433 -
        zahlman 1y ago
        > Wonder if `\00` is handled different between them. `dd` is for copying all the bytes of a source (unless you explicitly set a limit with the `count` option), regardless of whether they're zero. It's fundamentally not for null-terminated strings but arbitrary binary I/O. In fact, "copying from" /dev/zero is a common use case. It seems frankly implausible that the `dd` implementation is just stopping at a null byte; that would break a lot of tests and demonstrate a complete, fundamental misunderstanding of what's supposed to be implemented. > Not sure how to run the rust version but my md5sum seems to care how many null bytes there are. Yes, the md5 algorithm also fundamentally operates on and produces binary data; `md5sum` just converts the bytes to a hex dump at the end. The result you get is expected (edit: modulo hiccuphippo's correct observation), because the correct md5 sum changes with every byte of input, even if that byte has a zero value.
        hiccuphippo 1y ago
        Kind of off-topic, but those commands also add a newline character to the md5sums, giving unexpected results. I was trying it in a php interpreter and getting different values. Add -n to echo to avoid the new line.
- sionisrecur 1y ago
  And dd is also part of coreutils. So this is still a rust-coreutils issue, or an issue in gnu-coreutils that scripts rely on.
  - kps 1y ago
    Which has inadequate tests and gets it wrong? (‘All of them’ is an option.)

b_e_n_t_o_n 1y ago

Hmm, this plus the performance regressions makes me wonder if it's too soon to move to the rust version of Coreutils. And makes me wonder if this is gonna cause more pushback regarding the rust in the kernel movement.

mustache_kimono 1y ago
> this is gonna cause more pushback regarding the rust in the kernel movement. Only among those that don't understand that, if this is a problem, then it is Canonical problem, not a Rust problem. To give another example, Canonical includes ZFS in Ubuntu too. And, for a while, Canonical shipped a broken snapshot mechanism called zsys with Ubuntu too. Canonical ultimately ripped zsys out because it didn't work very well. zsys would choke on more than 4000 snapshots, etc. zsys was developed in Go, while other snapshot systems developed in Perl and Python did a little less and worked better. Now, is zsys a Go problem? Of course not. It wasn't ready because Canonical sometimes ships broken stuff.
- zahlman 1y ago
  > Only among those that don't understand that, if this is a problem, then it is Canonical problem, not a Rust problem. (This is hard to express in a careful way where I'm confident of not offending anyone. Please take me at my word that I'm not trying to take sides in this at all.) The dominant narrative behind this pushback, as far as I can tell, is nothing to do with the Rust language itself (aside perhaps from a few fringe people who see the adoption of Rust as some kind of signal of non-programming-related politics, and who are counter-signaling). Rather, the opposition is to re-implementing "working" software (including in the sense that nobody seems to have noticed any memory-handling faults all this time) for the sake of seemingly nebulous benefits (like compiler-checked memory safety). The Rust code will probably also be more maintainable by Rust developers than the C code currently is by C developers given the advantages of Rust's language design. (Unless it turns out that the C developers are just intrinsically better at programming and/or software engineering; I'm pretty skeptical of that.) But most long-time C users are probably not going to want to abandon their C expertise and learn Rust. And they seem to outnumber the new Rust developers by quite a bit, at least for now.

deleted 1y ago

[deleted]

sionisrecur 1y ago

As expected! I wonder how many tools depend on bugs or edgecases in GNU Coreutils.

kpcyrd 1y ago
Almost as if programming against losely defined command-line interfaces is maybe not a very robust way of programming~
- yjftsjthsd-h 1y ago
  No, the same thing can and does happen on any API. As an obvious example, there are an annoying number of programs that depend on GNU's libc in particular, and which therefore break when someone tries to compile against ex. musl.
  - kpcyrd 1y ago
    This isn't true, libc is magnitudes better defined than binary names in shell scripting, but it's still yet another case of approximation-API with multiple competing implementations. There are plenty of ecosystems where programs declare a specific library implementation they expect to call into (Rust, Python, Npm, Ruby, Perl, ...) often even constrained by versions. But also if you depend on libcurl you are only going to have to deal with multiple verions of the same implementation (that you can still constrain in e.g. pkg-config). In shell scripting you have to deal with stuff like "in nc, EOF on stdin is undefined behavior and implementation specific".
    - yjftsjthsd-h 1y ago
      I don't see the difference? There's a POSIX spec for coreutils (and some other shell-usable executables), there's a POSIX spec for libc (and some other C libs), and both are a somewhat painful subset of what most programs/scripts actually use. And yes, in both cases often the solution is to explicitly depend on a particular implementation or even version thereof; systemd explicitly only supports glibc, shell scripts targeting macOS may require that you install a newer version of bash than the OS ships, and yes, if you need nc to behave a particular way then you need to depend on the right one (I've actually personally hit that; IIRC I installed netcat-openbsd but it didn't include a feature I needed). In all cases, there may be a portable standard, but it doesn't cover everything, so if you're not inside what's covered by the standard then you should specify your actual dependencies. It still doesn't matter whether the API you're touching is mediated by the link phase of a compiler chain or *sh calling particular binaries.

mongol 1y ago

Someone need to be the first to take the hit, and apparently Ubuntu volunteered

zahlman 1y ago
Indeed. They knew there was risk associated with this, which is why they didn't just plop it into the LTS release. If it isn't working acceptably by the 26.04 release window, it'll just get reverted.
- subsistence234 1y ago
  non-LTS release means beta now?
  - dfc 1y ago
    No, but sort of yes. This is from Ubuntu: "Interim releases will introduce new capabilities from Canonical and upstream open source projects, they serve as a proving ground for these new capabilities." https://ubuntu.com/about/release-cycle https://ubuntu.com/about/release-cycle
packetlost 1y ago
They're doing it precisely so they can identify shortcomings and bugs. This is expected and good.
- 3836293648 1y ago
  It's expected. Good is an entirely different issue. More permissively licenced core components is 100% a bad thing
  - Volundr 1y ago
    > More permissively licenced core components is 100% a bad thing I don't follow, can you explain why?

flykespice 1y ago

[dead]

dmitrygr 1y ago

Hyrum's Law With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

Avamander 1y ago
One more reason to actually reimplement coreutils. Figuring out these undocumented promises also makes it easier to ensure correctness in general.
- steveklabnik 1y ago
  You’re downvoted but you’re correct. The uutils folks have been submitting bugs and test cases upstream. It actively helps both projects.
lioeters 1y ago
Every change breaks someone's workflow. https://xkcd.com/1172/ https://xkcd.com/1172/

samdoesnothing 1y ago

If they would have used a modern, safe programming language like rust those errors would be impossible to... oh wait... Never mind.

arp242 1y ago
No one has ever claimed Rust prevents all bugs. This is such a tired strawman trope.
- tssva 1y ago
  "No one has ever" regarding human actions is quite a bold claim to make in relation to anything.
  - arp242 1y ago
    Prove me wrong. Find someone saying that Rust will prevent all bugs.
    - tssva 1y ago
      I believe the way it generally works is the person that makes a claim has the burden to provide evidence it is true.
- spacechild1 1y ago
  There a people who claim that "once it compiles, it works". I've seen this quite a few times here on HN. Just a random example: https://news.ycombinator.com/item?id=45046182 https://news.ycombinator.com/item?id=45046182
  - acdha 1y ago
    That’s a far more nuanced comment than you’re portraying it as, especially as it’s appearing for exactly this scenario: the new dd is working as designed, it’s not segfaulting or corrupting data, but its design isn’t identical to the GNU version and that logic error is the kind of thing Rust can’t prevent short of AGI.

arp242 1y ago

As an aside on the GNU *sum tools, I found they're quite slow. A few months ago I wrote a simple replacement in Go for UX reasons and somewhat to my surprise, the Go implementation of most hash algorithms seems about 2 to 4 times as fast when using a simple naïve "obvious" single-threaded implementation. It can be sped up even more by using more than one core. Go has assembly implementations for most hash functions. I didn't really look at the coreutils implementation but I'm guessing it's "just" in C. At any rate, small teething issues aside, long-term things should be better and faster.

collinfunk 1y ago
What distribution do you use? GNU Coreutils uses the OpenSSL implementation of hashes by default, but some distributions have disabled it using './configure --with-openssl=no'. Debian used to do this, but now links to OpenSSL.
- arp242 1y ago
  This is on Void. It doesn't have --with-openssl configure args in the package, although the binary also doesn't link to lib{ssl,crypto}. It probably gets auto-detected to "no"(?)
  - collinfunk 1y ago
    Yep, from a Void Linux container it appears that they do not link to libcrypto: $ ldd /usr/bin/cksum linux-vdso.so.1 (0x00007fb354763000) libc.so.6 => /usr/lib64/libc.so.6 (0x00007fb354549000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb354765000) For context, I am a committer to GNU Coreutils. We have used OpenSSL by default for a few years now [1]. Previously it was disabled by default because the OpenSSL license is not compatible with the GPL license [2]. This was resolved when they switched to the Apache License, Version 2.0 in OpenSSL 3.0.0. If the Void people wanted to enable OpenSSL, they would probably just need to add openssl (or whatever they package it as) to the package dependencies. [1] https://github.com/coreutils/coreutils/commit/0d77e1b7ea2840dbc10994f61894f1a3f64b46f9 https://github.com/coreutils/coreutils/commit/0d77e1b7ea2840... [2] https://www.gnu.org/licenses/license-list.html#OpenSSL https://www.gnu.org/licenses/license-list.html#OpenSSL

lousken 1y ago

it's not LTS release so it's mostly fine, but they still should postpone the release until such bugs are fixed

kpcyrd 1y ago

hopefully they got reported as bugs, I spent some time in June making sure a basic Arch Linux system can compile itself using uutils, and things mostly worked, the only build failures could be argued were shell scripts depending on undefined behavior, like "calling install(1) with multiple -m arguments": https://github.com/uutils/coreutils/issues/8033 https://github.com/uutils/coreutils/issues/8033 The other one was an edge-case with dangling symlinks: https://github.com/uutils/coreutils/issues/8044 https://github.com/uutils/coreutils/issues/8044 Both got fixed promptly, after taking the time to report them properly. :)