8 ms·

> What’s notable is that all of these bugs landed in a production Rust codebase, written by people who knew what they were doing They knew how to write Rust, b

by wahern 6mo ago

> What’s notable is that all of these bugs landed in a production Rust codebase, written by people who knew what they were doing They knew how to write Rust, but clearly weren't sufficiently experienced with Unix APIs, semantics, and pitfalls. Most of those mistakes are exceedingly amateur from the perspective of long-time GNU coreutils (or BSD or Solaris base) developers, issues that were identified and largely hashed out decades ago, notwithstanding the continued long tail of fixes--mostly just a trickle these days--to the old codebases.

AlotOfReading 6mo ago
Someone once coined a related term, "disassembler rage". It's the idea that every mistake looks amateur when examined closely enough. Comes from people sitting in a disassembler and raging the high level programmers who had the gall to e.g. use conditionals instead of a switch statement inside a function call a hundred frames deep. We're looking solely at the few things they got wrong, and not the thousands of correct lines around them.
- irishcoffee 6mo ago
  When I read the article I came away with the impression that shipping bugs this severe in a rewrite of utils used by hundreds of millions of people daily (hourly?) isn’t ok. I don’t think brushing the bad parts off with “most of the code was really good!” is a fair way to look at this. Cloudflare crashed a chunk of the internet with a rust app a month or so ago, deploying a bad config file iirc. Rust isn’t a panacea, it’s a programming language. It’s ok that it’s flawed, all languages are.
  - gmueckl 6mo ago
    I think that legitimate real world issues in rust code should be talked about more often. Right now the language enjoys a reputation that is essentiaöly misleading marketing. It isn't possible to create a programing language that doesn't allow bugs to happen (even with formal verification you can still prove correctness based on a wrong set of assumptions). This weird, kind of religious belief that rust leads to magically completely bug free programs needs to be countered and brought in touch with reality IMO.
    - testdelacc1 6mo ago
      Is it possible you’ve misunderstood what Rust promises? > It isn't possible to create a programing language that doesn't allow bugs to happen Yes, that’s true. No one doubts this. Except you seem to think that Rust promises no bugs at all? I don’t know where you got this impression from, but it is incorrect. Rust promises that certain kinds of bugs like use-after-free are much, much less likely. It eliminates some kinds of bugs, not all bugs altogether. It’s possible that you’ve read the claim on kinds of bugs, and misinterpreted it as all bugs. I’ve had this conversation before, and it usually ends like https://www.smbc-comics.com/comic/aaaah https://www.smbc-comics.com/comic/aaaah
      - adrian_b 6mo ago
        "Rust" obviously does not promise that. On the other hand, there are too many less-experienced Rust fans who do claim that "Rust" promises this and that any project that does not use Rust is doomed and that any of the existing decades-old software projects should be rewritten in Rust to decrease the chances that they may have bugs. What is described in TFA is not surprising at all, because it is exactly what has been predicted about this and other similar projects. Anyone who desires to rewrite in Rust any old project, should certainly do it. It will be at least a good learning experience and whenever an ancient project is rewritten from scratch, the current knowledge should enable the creation of something better than the original. Nonetheless, the rewriters should never claim that what they have just produced has currently less bugs than the original, because neither they nor Rust can guarantee this, but only a long experience with using the rewritten application. Such rewritten software packages should remain for years as optional alternatives to the originals. Any aggressive push to substitute the originals immediately is just stupid (and yes, I have seen people trying to promote this). Moreover, someone who proposes the substitution of something as basic as coreutils, must first present to the world the results of a huge set of correctness tests and performance benchmarks comparing the old package with the new package, before the substitution idea is even put forward.
        testdelacc1 6mo ago
        Where are these rust fans? Are they in the room with us right now? You’ve constructed a strawman with no basis in reality. You know what actual Rust fans sound like? They sound like Matthias Endler, who wrote the article we’re discussing. Matthias hosts a popular podcast Rust in Production where talks with people about sharp edges and difficulties they experienced using Rust. A true Rust advocate like him writes articles titled “Bugs Rust Won’t Catch”. > Such rewritten software packages should remain for years as optional alternatives to the originals. This project was started a decade ago. (https://news.ycombinator.com/item?id=7882211 https://news.ycombinator.com/item?id=7882211) > must first present to the world the results of a huge set of correctness tests and performance benchmarks Yeah, you can see those in https://github.com/uutils/coreutils https://github.com/uutils/coreutils. This project has also worked with GNU coreutils maintainers to add more tests over time. Check out the graph where the total number of tests increases over time. > before the substitution idea is even put forward I partly agree. But notice that these CVEs come from a thorough security audit paid for by Canonical. Canonical is paying for it because they have a plan to substitute in the immediate future. Without a plan to substitute it’s hard to advocate for funding. Without funding it’s hard to find and fix these issues. With these issues unfixed it’s hard to plan to substitute. Chicken and egg problem. > less bugs Fewer.
  - lelanthran 6mo ago
    I find it hilarious that this comment is being downvoted. Exactly what is the controversial take here? > I don’t think brushing the bad parts off with “most of the code was really good!” is a fair way to look at this. Nope. this is fine. > Cloudflare crashed a chunk of the internet with a rust app a month or so ago, deploying a bad config file iirc. Maybe this? > Rust isn’t a panacea, it’s a programming language. It’s ok that it’s flawed, all languages are. Nope, this is fine too.
    - dbdr 6mo ago
      I didn't downvote, but I feel the last two points show a lack of nuance. It's saying "Rust doesn't prevent 100% of the bugs, like all other programming languages", while failing to acknowledge that if a programming language prevents entire classes of bugs, it's a very significant improvement.
      - adrian_b 6mo ago
        Nobody disputes that Rust is one of the programming languages that prevent several classes of frequent bugs, which is a valuable feature when compared with C/C++, even if that is a very low bar. What many do not accept among the claims of the Rust fans is that rewriting a mature and very big codebase from another language into Rust is likely to reduce the number of bugs of that codebase. For some buggier codebases, a rewrite in Rust or any other safer language may indeed help, but I agree with the opinion expressed by many other people that in most cases a rewrite from scratch is much more likely to have bugs, regardless in what programming language it is written. If someone has the time to do it, a rewrite is useful in most cases, but it should be expected that it will take a lot of time after the completion of the project until it will have as few bugs as mature projects.
        kibwen 6mo ago
        As other people have mentioned, the goal of uutils was not "let's reduce bugs in coreutils by rewriting it in Rust", it was "it's 2013 and here's a pre-1.0 language that looks neat and claims to be a credible replacement for C, let's test that hypothesis by porting coreutils, giving us an excuse to learn and play with a new language in the process". It seems worth emphasizing that its creation was neither ideologically motivated nor part of some nefarious GPL-erasure scheme, it was just some people hacking on a codebase for fun. Whether or not it was wise for Canonical to attempt to then take that codebase and uplift it into Ubuntu is a different story altogether, but one that has no bearing on the motivations of the people behind the original port itself. You can see an alternative approach with the authors of sudo-rs. Rather than porting all of userspace to Rust for fun, they identified a single component of a particularly security-critical nature (sudo), and then further justified their rewrite by removing legacy features, thereby producing an overall simpler tool with less surface area to attack in the first place. It was not "we're going to rewrite sudo in Rust so it has fewer bugs", it was "we're going to rewrite sudo with the goal of having fewer bugs, and as one subcomponent of that, we're going to use Rust". And of course sudo-rs has had fresh bugs of its own, as any rewrite will. But the mere existence of bugs does not invalidate their hypothesis, which is that a conscientious rewrite of a tool can result in fewer bugs overall.
  - fluffybucktsnek 6mo ago
    If I'm not mistaken, in the Cloudflare case, both the Rust rewrite and the C++ original version crashed. The primary cause being the bad config file.
    - adrian_b 6mo ago
      Yes, but the point was that rewriting something in Rust is not sufficient per se to prevent such bugs. The goal claimed by all these rewrites is the elimination of bugs.
      - saghm 6mo ago
        The "elimination of bugs" is not synonymous with "the elimination of all bugs". The way you're presenting it, any single bug in a rewrite would be grounds to consider the the entire endeavor a failure, which is a ridiculous standard. There are plenty of strong arguments to be made against rewriting something in Rust, but this is a pretty weak one.
- Cthulhu_ 6mo ago
  Thing is, these tools are so critical that even one error may cause systems to be compromised; rewriting them should never be taken lightly. (Actually ideally there's formal verification tools that can accurately test for all of the issues found in this review / audit, like the very timing specific path changes, but that's a codebase on its own)
  - bluGill 6mo ago
    Is formal verification able to find most of these issues? I'm no expert on formal analysis, but I suspect most systems are not able to handle many of these errors. It seems more likely that the system will assume the file doesn't change between two syscalls - which seems to be the majority of issues. Modeling that possibility at least makes the formal system much harder to make.
nine_k 6mo ago
More than that: it seems that Rust stdlib nudges the developer towards using neat APIs at an incorrect level of abstraction, like path-based instead of handle-based file operations. I hope I'm wrong.
- NobodyNada 6mo ago
  Nearly every available filesystem API in Rust's stdlib maps one-to-one with a Unix syscall (see Rust's std::fs module [0] for reference -- for example, the `File` struct is just a wrapper around a file descriptor, and its associated methods are essentially just the syscalls you can perform on file descriptors). The only exceptions are a few helper functions like `read_to_string` or `create_dir_all` that perform slightly higher-level operations. And, yeah, the Unix syscalls are very prone to mistakes like this. For example, Unix's `rename` syscall takes two paths as arguments; you can't rename a file by handle; and so Rust has a `rename` function that takes two paths rather than an associated function on a `File`. Rust exposes path-based APIs where Unix exposes path-based APIs, and file-handle-based APIs where Unix exposes file-handle-based APIs. So I agree that Rust's stdilb is somewhat mistake prone; not so much because it's being opinionated and "nudg[ing] the developer towards using neat APIs", but because it's so low-level that it's not offering much "safety" in filesystem access over raw syscalls beyond ensuring that you didn't write a buffer overflow. [0]: https://doc.rust-lang.org/std/fs/index.html https://doc.rust-lang.org/std/fs/index.html
  - masklinn 6mo ago
    > For example, Unix's `rename` syscall takes two paths as arguments; you can't rename a file by handle And then there’s renameat(2) which takes two dirfd… and two paths from there, which mostly has all the same issues rename(2) does (and does not even take flags so even O_NOFOLLOW is not available). I’m not sure what you’d need to make a safe renameat(), maybe a triplet of (dirfd, filefd, name[1]) from the source, (dirfd, name) from the target, and some sort of flag to indicate whether it is allowed to create, overwrite, or both. As the recent https://blog.sebastianwick.net/posts/how-hard-is-it-to-open-a-file/ https://blog.sebastianwick.net/posts/how-hard-is-it-to-open-... talks about (just for file but it applies to everything) secure file system interaction is absolutely heinous. [1]: not path
slopinthebag 6mo ago
Seems pretty impressive they rewrote the coreutils in a new language, with so little Unix experience, and managed to do such a good job with very little bugs or vulns. I would have expected an order of magnitude more at least. Shows how good Rust is, that even inexperienced Unix devs can write stuff like this and make almost no mistakes.
- nine_k 6mo ago
  Yes, it's the lack of Unix experience that's terrifying. So many of mistakes listed are rookie mistakes, like not propagating the most severe errors, or the `kill -1` thing. Why were people who apparently did not have much experience using coreutils assigned to rewrite coreutils?
  - aw1621107 6mo ago
    > Why were people who apparently did not have much experience using coreutils assigned to rewrite coreutils? From what I understand, "assigned" probably isn't the best way to put it. uutils started off back in 2013 as a way to learn Rust [0] way before the present kerfuffle. [0]: https://github.com/uutils/coreutils/tree/9653ed81a2fbf393f420d9a8007c574d922018d1 https://github.com/uutils/coreutils/tree/9653ed81a2fbf393f42...
    - nineteen999 6mo ago
      Yeah perhaps learning UNIX API's and Rust at the same time doesn't lead to a drop in replacement ready to be shipped in major distributions. Who whould have thunk it.
      - aw1621107 6mo ago
        Strictly speaking it doesn't preclude eventually producing a production-ready drop-in replacement either, though evidently that needs a fresh set of eyes.
    - bpbp-mango 6mo ago
pando85 6mo ago
Memory safety catches buffer overflows. CI catches logic bugs. Neither catches the Unix API gotchas nobody documented.
- cubefox 6mo ago
  LLM account
- bjourne 6mo ago
  CI catches all kinds of bugs.
- vhantz 6mo ago
  How does CI catch logic bugs?
  - bluGill 6mo ago
    That depends on what tests you are running. In any significant projects you need a test suite so large that you wouldn't run all the tests before pushing to CI - instead you are the targeted tests that test the area of code you changed, but there are more "integration tests" that go through you code and thus could break, but you don't actually run. You can also run some static analysis that is too long to run locally every time, but once in a while it will point out "this code pattern is legal buy is almost always a bug" It is also possible to do some formal analysis of code on CI that you wouldn't always run locally - I'm not an expert on these.
    - vhantz 5mo ago
      That's true in general. In this case where the logic bugs are from not understanding the API being implemented (and in any similar case), tests wouldn't catch the bugs either (even integration tests) because good tests require understanding the contract of the unit being tested.
- Arch-TK 6mo ago
  They're not API gotchas in most cases. And writing comprehensive tests for this behaviour is very difficult regardless of which language you are using. I am all for rust rewrites of things. But in this case, these are mistakes which were encouraged by the lazy design of `std::fs` and the developers' lack of relevant experience. And to clarify, I don't blame the developers for lacking the relevant experience. Working on such a project is precisely the right place to learn stuff like this. I think it's an absurdly dumb move by Canonical to take this project and beta-test it on normal users' machines though…
concinds 6mo ago
Reading that Canonical thread was jaw-dropping. Paraphrased: "Rust is more secure, security is our priority, therefore deploying this full-rewrite of core utils is an emergency. If things break that's fine, we'll fix it :)". I would not want to run any code on my machines made by people who think like this. And I'm pro-Rust. Rust is only "more secure" all else being equal. But all else is not equal. A rewrite necessarily has orders of magnitude more bugs and vulnerabilities than a decades-old well-maintained codebase, so the security argument was only valid for a long-term transition, not a rushed one. And the people downplaying user impact post-rollout, arguing that "this is how we'll surface bugs", and "the old coreutils didn't have proper test cases anyway" are so irresponsible. Users are not lab rats. Maintainers have a moral responsibility to not harm users' systems' reliability (I know that's a minority opinion these days). Their reasoning was flawed, and their values were wrong.
- zx8080 6mo ago
  Agree with the point. Asking sincerely, how to filter out installing any rust-rewrite packages on my machines? Does anyone know the way?
  - kibwen 6mo ago
    If you don't want Canonical's packages, you should probably just be using Debian rather than Ubuntu. It's not 2008 anymore, stock Debian is quite user-friendly.
    - tambre 6mo ago
      Worth noting is that in Debian experimental coreutils defaults to coreutils-from-uutils [0]. This came as a big surprise and as far as I can tell there's been no discussion. A Canonical developer seems to have unilaterally overwritten the coreutils package without discussing with the maintainer. All the package renames that are in Ubuntu aren't in Debian so you can't switch to GNU utils either without deep trickery in a separate recovery environment. I'm used to running experimental software but I wasn't ready for my computer to not boot one day because of uutils. The `-Z` flag for `cp` wasn't implemented in the 9 month old version shipped in Debian at that time so initramfs creation failed... [0] https://packages.debian.org/experimental/coreutils https://packages.debian.org/experimental/coreutils
onlyrealcuzzo 6mo ago
> They knew how to write Rust, but clearly weren't sufficiently experienced with Unix APIs, semantics, and pitfalls. The point of Rust is that you shouldn't have to worry about the biggest, easiest to fall in pitfalls. I think the author's point of this article, is that a proper file system API should do the same.
empath75 6mo ago
Having panics in these are pretty amateur hour even just on a Rust level. I could see if they were like alloc errors which you can't handle, but expect and unwraps are inexcusable unless you are very carefully guarding them with invariants that prevent that code path from ever running.

by wahern 6mo ago

AlotOfReading 6mo ago

Someone once coined a related term, "disassembler rage". It's the idea that every mistake looks amateur when examined closely enough. Comes from people sitting in a disassembler and raging the high level programmers who had the gall to e.g. use conditionals instead of a switch statement inside a function call a hundred frames deep. We're looking solely at the few things they got wrong, and not the thousands of correct lines around them.

irishcoffee 6mo ago
When I read the article I came away with the impression that shipping bugs this severe in a rewrite of utils used by hundreds of millions of people daily (hourly?) isn’t ok. I don’t think brushing the bad parts off with “most of the code was really good!” is a fair way to look at this. Cloudflare crashed a chunk of the internet with a rust app a month or so ago, deploying a bad config file iirc. Rust isn’t a panacea, it’s a programming language. It’s ok that it’s flawed, all languages are.
- gmueckl 6mo ago
  I think that legitimate real world issues in rust code should be talked about more often. Right now the language enjoys a reputation that is essentiaöly misleading marketing. It isn't possible to create a programing language that doesn't allow bugs to happen (even with formal verification you can still prove correctness based on a wrong set of assumptions). This weird, kind of religious belief that rust leads to magically completely bug free programs needs to be countered and brought in touch with reality IMO.
  - testdelacc1 6mo ago
    Is it possible you’ve misunderstood what Rust promises? > It isn't possible to create a programing language that doesn't allow bugs to happen Yes, that’s true. No one doubts this. Except you seem to think that Rust promises no bugs at all? I don’t know where you got this impression from, but it is incorrect. Rust promises that certain kinds of bugs like use-after-free are much, much less likely. It eliminates some kinds of bugs, not all bugs altogether. It’s possible that you’ve read the claim on kinds of bugs, and misinterpreted it as all bugs. I’ve had this conversation before, and it usually ends like https://www.smbc-comics.com/comic/aaaah https://www.smbc-comics.com/comic/aaaah
    - adrian_b 6mo ago
      "Rust" obviously does not promise that. On the other hand, there are too many less-experienced Rust fans who do claim that "Rust" promises this and that any project that does not use Rust is doomed and that any of the existing decades-old software projects should be rewritten in Rust to decrease the chances that they may have bugs. What is described in TFA is not surprising at all, because it is exactly what has been predicted about this and other similar projects. Anyone who desires to rewrite in Rust any old project, should certainly do it. It will be at least a good learning experience and whenever an ancient project is rewritten from scratch, the current knowledge should enable the creation of something better than the original. Nonetheless, the rewriters should never claim that what they have just produced has currently less bugs than the original, because neither they nor Rust can guarantee this, but only a long experience with using the rewritten application. Such rewritten software packages should remain for years as optional alternatives to the originals. Any aggressive push to substitute the originals immediately is just stupid (and yes, I have seen people trying to promote this). Moreover, someone who proposes the substitution of something as basic as coreutils, must first present to the world the results of a huge set of correctness tests and performance benchmarks comparing the old package with the new package, before the substitution idea is even put forward.
      - testdelacc1 6mo ago
        Where are these rust fans? Are they in the room with us right now? You’ve constructed a strawman with no basis in reality. You know what actual Rust fans sound like? They sound like Matthias Endler, who wrote the article we’re discussing. Matthias hosts a popular podcast Rust in Production where talks with people about sharp edges and difficulties they experienced using Rust. A true Rust advocate like him writes articles titled “Bugs Rust Won’t Catch”. > Such rewritten software packages should remain for years as optional alternatives to the originals. This project was started a decade ago. (https://news.ycombinator.com/item?id=7882211 https://news.ycombinator.com/item?id=7882211) > must first present to the world the results of a huge set of correctness tests and performance benchmarks Yeah, you can see those in https://github.com/uutils/coreutils https://github.com/uutils/coreutils. This project has also worked with GNU coreutils maintainers to add more tests over time. Check out the graph where the total number of tests increases over time. > before the substitution idea is even put forward I partly agree. But notice that these CVEs come from a thorough security audit paid for by Canonical. Canonical is paying for it because they have a plan to substitute in the immediate future. Without a plan to substitute it’s hard to advocate for funding. Without funding it’s hard to find and fix these issues. With these issues unfixed it’s hard to plan to substitute. Chicken and egg problem. > less bugs Fewer.
- lelanthran 6mo ago
  I find it hilarious that this comment is being downvoted. Exactly what is the controversial take here? > I don’t think brushing the bad parts off with “most of the code was really good!” is a fair way to look at this. Nope. this is fine. > Cloudflare crashed a chunk of the internet with a rust app a month or so ago, deploying a bad config file iirc. Maybe this? > Rust isn’t a panacea, it’s a programming language. It’s ok that it’s flawed, all languages are. Nope, this is fine too.
  - dbdr 6mo ago
    I didn't downvote, but I feel the last two points show a lack of nuance. It's saying "Rust doesn't prevent 100% of the bugs, like all other programming languages", while failing to acknowledge that if a programming language prevents entire classes of bugs, it's a very significant improvement.
    - adrian_b 6mo ago
      Nobody disputes that Rust is one of the programming languages that prevent several classes of frequent bugs, which is a valuable feature when compared with C/C++, even if that is a very low bar. What many do not accept among the claims of the Rust fans is that rewriting a mature and very big codebase from another language into Rust is likely to reduce the number of bugs of that codebase. For some buggier codebases, a rewrite in Rust or any other safer language may indeed help, but I agree with the opinion expressed by many other people that in most cases a rewrite from scratch is much more likely to have bugs, regardless in what programming language it is written. If someone has the time to do it, a rewrite is useful in most cases, but it should be expected that it will take a lot of time after the completion of the project until it will have as few bugs as mature projects.
      - kibwen 6mo ago
        As other people have mentioned, the goal of uutils was not "let's reduce bugs in coreutils by rewriting it in Rust", it was "it's 2013 and here's a pre-1.0 language that looks neat and claims to be a credible replacement for C, let's test that hypothesis by porting coreutils, giving us an excuse to learn and play with a new language in the process". It seems worth emphasizing that its creation was neither ideologically motivated nor part of some nefarious GPL-erasure scheme, it was just some people hacking on a codebase for fun. Whether or not it was wise for Canonical to attempt to then take that codebase and uplift it into Ubuntu is a different story altogether, but one that has no bearing on the motivations of the people behind the original port itself. You can see an alternative approach with the authors of sudo-rs. Rather than porting all of userspace to Rust for fun, they identified a single component of a particularly security-critical nature (sudo), and then further justified their rewrite by removing legacy features, thereby producing an overall simpler tool with less surface area to attack in the first place. It was not "we're going to rewrite sudo in Rust so it has fewer bugs", it was "we're going to rewrite sudo with the goal of having fewer bugs, and as one subcomponent of that, we're going to use Rust". And of course sudo-rs has had fresh bugs of its own, as any rewrite will. But the mere existence of bugs does not invalidate their hypothesis, which is that a conscientious rewrite of a tool can result in fewer bugs overall.
- fluffybucktsnek 6mo ago
  If I'm not mistaken, in the Cloudflare case, both the Rust rewrite and the C++ original version crashed. The primary cause being the bad config file.
  - adrian_b 6mo ago
    Yes, but the point was that rewriting something in Rust is not sufficient per se to prevent such bugs. The goal claimed by all these rewrites is the elimination of bugs.
    - saghm 6mo ago
      The "elimination of bugs" is not synonymous with "the elimination of all bugs". The way you're presenting it, any single bug in a rewrite would be grounds to consider the the entire endeavor a failure, which is a ridiculous standard. There are plenty of strong arguments to be made against rewriting something in Rust, but this is a pretty weak one.
Cthulhu_ 6mo ago
Thing is, these tools are so critical that even one error may cause systems to be compromised; rewriting them should never be taken lightly. (Actually ideally there's formal verification tools that can accurately test for all of the issues found in this review / audit, like the very timing specific path changes, but that's a codebase on its own)
- bluGill 6mo ago
  Is formal verification able to find most of these issues? I'm no expert on formal analysis, but I suspect most systems are not able to handle many of these errors. It seems more likely that the system will assume the file doesn't change between two syscalls - which seems to be the majority of issues. Modeling that possibility at least makes the formal system much harder to make.

nine_k 6mo ago

More than that: it seems that Rust stdlib nudges the developer towards using neat APIs at an incorrect level of abstraction, like path-based instead of handle-based file operations. I hope I'm wrong.

NobodyNada 6mo ago
Nearly every available filesystem API in Rust's stdlib maps one-to-one with a Unix syscall (see Rust's std::fs module [0] for reference -- for example, the `File` struct is just a wrapper around a file descriptor, and its associated methods are essentially just the syscalls you can perform on file descriptors). The only exceptions are a few helper functions like `read_to_string` or `create_dir_all` that perform slightly higher-level operations. And, yeah, the Unix syscalls are very prone to mistakes like this. For example, Unix's `rename` syscall takes two paths as arguments; you can't rename a file by handle; and so Rust has a `rename` function that takes two paths rather than an associated function on a `File`. Rust exposes path-based APIs where Unix exposes path-based APIs, and file-handle-based APIs where Unix exposes file-handle-based APIs. So I agree that Rust's stdilb is somewhat mistake prone; not so much because it's being opinionated and "nudg[ing] the developer towards using neat APIs", but because it's so low-level that it's not offering much "safety" in filesystem access over raw syscalls beyond ensuring that you didn't write a buffer overflow. [0]: https://doc.rust-lang.org/std/fs/index.html https://doc.rust-lang.org/std/fs/index.html
- masklinn 6mo ago
  > For example, Unix's `rename` syscall takes two paths as arguments; you can't rename a file by handle And then there’s renameat(2) which takes two dirfd… and two paths from there, which mostly has all the same issues rename(2) does (and does not even take flags so even O_NOFOLLOW is not available). I’m not sure what you’d need to make a safe renameat(), maybe a triplet of (dirfd, filefd, name[1]) from the source, (dirfd, name) from the target, and some sort of flag to indicate whether it is allowed to create, overwrite, or both. As the recent https://blog.sebastianwick.net/posts/how-hard-is-it-to-open-a-file/ https://blog.sebastianwick.net/posts/how-hard-is-it-to-open-... talks about (just for file but it applies to everything) secure file system interaction is absolutely heinous. [1]: not path

slopinthebag 6mo ago

Seems pretty impressive they rewrote the coreutils in a new language, with so little Unix experience, and managed to do such a good job with very little bugs or vulns. I would have expected an order of magnitude more at least. Shows how good Rust is, that even inexperienced Unix devs can write stuff like this and make almost no mistakes.

nine_k 6mo ago
Yes, it's the lack of Unix experience that's terrifying. So many of mistakes listed are rookie mistakes, like not propagating the most severe errors, or the `kill -1` thing. Why were people who apparently did not have much experience using coreutils assigned to rewrite coreutils?
- aw1621107 6mo ago
  > Why were people who apparently did not have much experience using coreutils assigned to rewrite coreutils? From what I understand, "assigned" probably isn't the best way to put it. uutils started off back in 2013 as a way to learn Rust [0] way before the present kerfuffle. [0]: https://github.com/uutils/coreutils/tree/9653ed81a2fbf393f420d9a8007c574d922018d1 https://github.com/uutils/coreutils/tree/9653ed81a2fbf393f42...
  - nineteen999 6mo ago
    Yeah perhaps learning UNIX API's and Rust at the same time doesn't lead to a drop in replacement ready to be shipped in major distributions. Who whould have thunk it.
    - aw1621107 6mo ago
      Strictly speaking it doesn't preclude eventually producing a production-ready drop-in replacement either, though evidently that needs a fresh set of eyes.
  - bpbp-mango 6mo ago

pando85 6mo ago

Memory safety catches buffer overflows. CI catches logic bugs. Neither catches the Unix API gotchas nobody documented.

cubefox 6mo ago
LLM account
bjourne 6mo ago
CI catches all kinds of bugs.
vhantz 6mo ago
How does CI catch logic bugs?
- bluGill 6mo ago
  That depends on what tests you are running. In any significant projects you need a test suite so large that you wouldn't run all the tests before pushing to CI - instead you are the targeted tests that test the area of code you changed, but there are more "integration tests" that go through you code and thus could break, but you don't actually run. You can also run some static analysis that is too long to run locally every time, but once in a while it will point out "this code pattern is legal buy is almost always a bug" It is also possible to do some formal analysis of code on CI that you wouldn't always run locally - I'm not an expert on these.
  - vhantz 5mo ago
    That's true in general. In this case where the logic bugs are from not understanding the API being implemented (and in any similar case), tests wouldn't catch the bugs either (even integration tests) because good tests require understanding the contract of the unit being tested.
Arch-TK 6mo ago
They're not API gotchas in most cases. And writing comprehensive tests for this behaviour is very difficult regardless of which language you are using. I am all for rust rewrites of things. But in this case, these are mistakes which were encouraged by the lazy design of `std::fs` and the developers' lack of relevant experience. And to clarify, I don't blame the developers for lacking the relevant experience. Working on such a project is precisely the right place to learn stuff like this. I think it's an absurdly dumb move by Canonical to take this project and beta-test it on normal users' machines though…

concinds 6mo ago

Reading that Canonical thread was jaw-dropping. Paraphrased: "Rust is more secure, security is our priority, therefore deploying this full-rewrite of core utils is an emergency. If things break that's fine, we'll fix it :)". I would not want to run any code on my machines made by people who think like this. And I'm pro-Rust. Rust is only "more secure" all else being equal. But all else is not equal. A rewrite necessarily has orders of magnitude more bugs and vulnerabilities than a decades-old well-maintained codebase, so the security argument was only valid for a long-term transition, not a rushed one. And the people downplaying user impact post-rollout, arguing that "this is how we'll surface bugs", and "the old coreutils didn't have proper test cases anyway" are so irresponsible. Users are not lab rats. Maintainers have a moral responsibility to not harm users' systems' reliability (I know that's a minority opinion these days). Their reasoning was flawed, and their values were wrong.

zx8080 6mo ago
Agree with the point. Asking sincerely, how to filter out installing any rust-rewrite packages on my machines? Does anyone know the way?
- kibwen 6mo ago
  If you don't want Canonical's packages, you should probably just be using Debian rather than Ubuntu. It's not 2008 anymore, stock Debian is quite user-friendly.
  - tambre 6mo ago
    Worth noting is that in Debian experimental coreutils defaults to coreutils-from-uutils [0]. This came as a big surprise and as far as I can tell there's been no discussion. A Canonical developer seems to have unilaterally overwritten the coreutils package without discussing with the maintainer. All the package renames that are in Ubuntu aren't in Debian so you can't switch to GNU utils either without deep trickery in a separate recovery environment. I'm used to running experimental software but I wasn't ready for my computer to not boot one day because of uutils. The `-Z` flag for `cp` wasn't implemented in the 9 month old version shipped in Debian at that time so initramfs creation failed... [0] https://packages.debian.org/experimental/coreutils https://packages.debian.org/experimental/coreutils

onlyrealcuzzo 6mo ago

> They knew how to write Rust, but clearly weren't sufficiently experienced with Unix APIs, semantics, and pitfalls. The point of Rust is that you shouldn't have to worry about the biggest, easiest to fall in pitfalls. I think the author's point of this article, is that a proper file system API should do the same.

empath75 6mo ago

Having panics in these are pretty amateur hour even just on a Rust level. I could see if they were like alloc errors which you can't handle, but expect and unwraps are inexcusable unless you are very carefully guarding them with invariants that prevent that code path from ever running.