4 ms·

Hi, I am one of the maintainers of GNU Coreutils. Thanks for the article, it covers some interesting topics. In the little Rust that I have used, I have felt th

Hi, I am one of the maintainers of GNU Coreutils. Thanks for the article, it covers some interesting topics. In the little Rust that I have used, I have felt that it is far too easy to write TOCTOU races using std::fs. I hope the standard library gets an API similar to openat eventually. I just want to mention that I disagree with the section titled "Rule: Resolve Paths Before Comparing Them". Generally, it is better to make calls to fstat and compare the st_dev and st_ino. However, that was mentioned in the article. A side effect that seems less often considered is the performance impact. Here is an example in practice: $ mkdir -p $(yes a/ | head -n $((32 * 1024)) | tr -d '\n') $ while cd $(yes a/ | head -n 1024 | tr -d '\n'); do :; done 2>/dev/null $ echo a > file $ time cp file copy real 0m0.010s user 0m0.002s sys 0m0.003s $ time uu_cp file copy real 0m12.857s user 0m0.064s sys 0m12.702s I know people are very unlikely to do something like that in real life. However, GNU software tends to work very hard to avoid arbitrary limits [1]. Also, the larger point still stands, but the article says "The Rust rewrite has shipped zero of these [memory saftey bugs], over a comparable window of activity." However, this is not true [2]. :) [1] https://www.gnu.org/prep/standards/standards.html#Semantics https://www.gnu.org/prep/standards/standards.html#Semantics [2] https://github.com/advisories/GHSA-w9vv-q986-vj7x https://github.com/advisories/GHSA-w9vv-q986-vj7x

s20n 5mo ago
Sorry, complete noob here. Why didn't you just cd into $(yes a/ | head -n $((32 * 1024)) | tr -d '\n')? Why do you need to use the while loop for cd? EDIT: got it. -bash: cd: a/a/a/....../a/a/: File name too long
- collinfunk 5mo ago
  No need to apologize at all. Doing it in one cd invocation would fail since the file name is longer than PATH_MAX. In that case passing it to a system call would fail with errno set to ENAMETOOLONG. You could probably make the loop more efficient, but it works good enough. Also, some shells don't allow you to enter directories that deep entirely. It doesn't work on mksh, for example.
  - dapperdrake 5mo ago
    Facetious reply: > However, GNU software tends to work very hard to avoid arbitrary limits [1].
    - Joker_vD 5mo ago
      Yes? The quote says "tends to", and you still can cd into that directory, albeit not in a single invocation. Windows has similar limitations [0], it's just that their MAX_PATH is just 260 so it's somewhat more noticeable... and IIRC the hard limit of 32 K for paths in non-negotiable. [0] https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation https://learn.microsoft.com/en-us/windows/win32/fileio/maxim...
      - dapperdrake 5mo ago
        Isn’t "cd" a unix syscall , because it changes the process's working directory? There was something written somewhere that it cannot be a unix utility for this very reason, but has to be a shell built-in. The syscall is a "single operation" from the point of a single-threaded process. What did I get wrong there? Side note: Missing bash$ man 1 cd ; Useful output bash$ help cd ;
        comex 5mo ago
        Yes, it’s a shell builtin that makes the shell execute a chdir() syscall. Therefore it isn’t subject to argument length limits imposed by the kernel when executing processes. But it is still subject to path length limits imposed by the kernel’s implementation of chdir() itself. While the shell may be a GNU project (bash), the kernel generally is not (unless you are running Hurd), so this isn’t GNU’s fault per se. However, the shell could theoretically chunk long cd arguments into multiple calls to chdir(), splitting on slashes. I believe this would be fully semantically correct: you are not losing any atomicity guarantees because the kernel doesn’t provide such guarantees in the first place for lookups involving multiple path components. I’m not surprised that bash doesn’t bother implementing this, and I don’t know if I’d call that an “arbitrary limitation” on bash’s part (as opposed to a lack of workaround for another component’s arbitrary limitation). But it would be possible.
dapperdrake 5mo ago
First of all, thank you for presenting a succinct take on this viewpoint from the other side of the fence from where I am at. So how can I learn from this? (Asking very aggressively, especially for Internet writing, to make the contrast unmistakable. And contrast helps with perceiving differences and mistakes.) (You also don’t owe me any of your time or mental bandwidth, whatsoever.) So here goes: Question 1: How come "speed", "performance", race conditions and st_ino keep getting brought up? Speed (latency), physically writing things out to storage (sequentially, atomically (ACID), all of HDD NVME SSD ODD FDD tape, "haskell monad", event horizons, finite speed of light and information, whatever) as well as race conditions all seem to boil down to the same thing. For reliable systems like accounting the path seems to be ACID or the highway. And "unreliable" systems forget fast enough that computers don’t seem to really make a difference there. Question 2: Does throughput really matter more than latency in everyday application? Question 3 (explanation first, this time): The focus on inode numbers is at least understandable with regards to the history of C and unix-like operating systems and GNU coreutils. What about this basic example? Just make a USB thumb drive "work" for storing files (ignoring nand flash decay and USB). Without getting tripped up in libc IO buffering, fflush, kernel buffering (Hurd if you prefer it over Linux or FreeBSD), more than one application running on a multi-core and/or time-sliced system (to really weed out single-core CPUs running only a single user-land binary with blocking IO).
- dijit 5mo ago
  > Does throughput really matter more than latency in everyday application? In my experience latency and throughput are intrinsically linked unless you have the buffer-space to handle the throughput you want. Which you can't guarantee on all the systems where GNU Coreutils run.
  - dapperdrake 5mo ago
    Higher throughput increases the risk of high latency. Low latency increases the risk of "wasted cycles”, i.e. lowers (machine) throughput. Helps with human discovery throughput, though. The sled.rs people had a well readable take on this in their performance guide.
- 5mo ago
cyberax 5mo ago
To be fair, Vec::set_len bug in Rust was in 2021. And even then it had to be annotated as `unsafe`. It was then deprecated and a linter check was added: https://github.com/rust-lang/rust-clippy/issues/7681 https://github.com/rust-lang/rust-clippy/issues/7681
- Dr_Emann 5mo ago
  To be even fair-er, it wasn't actually memory unsafety, it was "just" unsoundness, there was a type, that IF you gave it an io reader implementation that was weird, that implementation could see uninit data, or expose uninit data elsewhere, but the only readers actually used were well behaved readers.
  - dapperdrake 5mo ago
    > well behaved readers. Around and around we go.
- orlp 5mo ago
  Vec::set_len is by no means deprecated. The lint you linked only covers a very specific unsound pattern using set_len.
  - kibwen 5mo ago
    Indeed, and it doesn't need to be deprecated, because it's an API explicitly designed to give you low-level control where you need it, and because it is appropriately defined as an `unsafe` function with documented safety invariants that must be manually upheld in order for usage to be memory-safe. The documentation also suggests several other (safe) functions that should be used instead when possible, and provides correct usage examples: https://doc.rust-lang.org/std/vec/struct.Vec.html#method.set_len https://doc.rust-lang.org/std/vec/struct.Vec.html#method.set... .
    - bcjdjsndon 5mo ago
      > and because it is appropriately defined as an `unsafe` function with documented safety invariants that must be manually upheld in order for usage to be memory-safe. Didn't we learn from c, and the entire raison detre for rust, is that coders cannot be trusted to follow rules like this? If coders could "(document) safety invariants that must be manually upheld in order for usage to be memory-safe." there's be no need for Rust. This is the tautology underlying rust as I see it
theteapot 5mo ago
Probably a dumb question, but is GNU Core utils interested in / planning on doing its own rust rewrite?
- greatgib 5mo ago
  The rewrite in Rust is mostly vanity and marketing but not based on a real technical need... So I don't see why they would want to do that.
  - kibwen 5mo ago
    Canonical's usage of uutils is likely for marketing. But the codebase itself was developed for fun, as an excuse for people to have a hands-on way to learn Rust back before Rust was even released, with a minor justification as being cross-platform. From the original README in 2013: Why? ---- Many GNU, linux and other utils are pretty awesome, and obviously some effort has been spent in the past to port them to windows. However those projects are either old, abandonned, hosted on CVS, written in platform-specific C, etc. Rust provides a good platform-agnostic way of writing systems utils that are easy to compile anywhere, and this is as good a way as any to try and learn it. https://github.com/uutils/coreutils/blob/9653ed81a2fbf393f420d9a8007c574d922018d1/README.md https://github.com/uutils/coreutils/blob/9653ed81a2fbf393f42...
    - PunchyHamster 5mo ago
      >Canonical's usage of uutils is likely for marketing Currently their usage is actively worsening the security of their distro
      - Avamander 5mo ago
        These things were caught and basically all of them weren't covered by any test suite (not even GNU coreutils'). It's a bit bold to claim that it's actively worsening it when it's not an LTS.
        5mo ago
lyrie 5mo ago
[dead]
gzread 5mo ago
I see even the coreutils maintainers find themselves needing -n (no newlines) and -c (count) options to "yes".
- dapperdrake 5mo ago
  GNU coreutils is known for adding command libe options. One of the big philosophical differences to the BSD's. For a human being, it sucks both ways.
jimmypk 5mo ago
[flagged]
safercplusplus 5mo ago
I don't know if you're aware, but there is a demonstration of wget (a fellow "gnu utility", right?) being auto-translated to a memory-safe subset of C++ [1]. Because the translation essentially does a one-for-one substitution of potentially unsafe C elements with safe C++ counterparts that mirror the behavior, the translation should be much less susceptible to the introduction of new bugs and behaviors in the way a rewrite would be. With a little cleaning-up of the original code, the code translation ends up being fully automatic and so can be used as a build step to produce (slightly slower) memory-safe executables from the original C source. [1] https://duneroadrunner.github.io/scpp_articles/PoC_autotranslation_of_wget https://duneroadrunner.github.io/scpp_articles/PoC_autotrans...
- dapperdrake 5mo ago
  Filesystem access is mostly treated by users as serialized ACID transactions on "files in directories." "Managing this resource centrally" is where unix syscalls came from. An OS kernel can be used like a specialized library for ACID transactions on hardware singletons. People then got fancy with virtual memory, interrupts, signals, time-slicing, re-entrancy, thread-safety, and injectivity. It doesn’t matter, whether you call the "kernel library" from C, C++, Fortan, BASIC, Golang, bash, Rust, etc.
joaohaas 5mo ago
>the article says "The Rust rewrite has shipped zero of these [memory saftey bugs], over a comparable window of activity." However, this is not true That bug got fixed before the Ubuntu release, and is from way before Canonical was even involved with the project.
- rossvor 5mo ago
  In the given list of GNU CVEs in the original article, it included a buffer overrun in tail from 2021. So for a fair comparison 2021 is part of the "window of activity" (the year uu_od CVE was published).
pornel 5mo ago
Indeed, std::fs suffers from being a lowest common denominator. Rust had to have something at 1.0, and unfortunately it stayed like that. Rust uutils would be a good place to design a more foolproof replacement for Rust's std::fs API.
- dapperdrake 5mo ago
  Unix embodies this, as well. When K&R created unix and C there was still the better option of moving changes that were better to have in the "kernel" into the kernel. Now we have "standards" that even cause headaches between Linux and BSD's. Linux back-propagates stuff like mmap, io_uring, etc. to where it belongs. In this way it is like the original unix. And deservedly running on most servers out there.

s20n 5mo ago

Sorry, complete noob here. Why didn't you just cd into $(yes a/ | head -n $((32 * 1024)) | tr -d '\n')? Why do you need to use the while loop for cd? EDIT: got it. -bash: cd: a/a/a/....../a/a/: File name too long

collinfunk 5mo ago
No need to apologize at all. Doing it in one cd invocation would fail since the file name is longer than PATH_MAX. In that case passing it to a system call would fail with errno set to ENAMETOOLONG. You could probably make the loop more efficient, but it works good enough. Also, some shells don't allow you to enter directories that deep entirely. It doesn't work on mksh, for example.
- dapperdrake 5mo ago
  Facetious reply: > However, GNU software tends to work very hard to avoid arbitrary limits [1].
  - Joker_vD 5mo ago
    Yes? The quote says "tends to", and you still can cd into that directory, albeit not in a single invocation. Windows has similar limitations [0], it's just that their MAX_PATH is just 260 so it's somewhat more noticeable... and IIRC the hard limit of 32 K for paths in non-negotiable. [0] https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation https://learn.microsoft.com/en-us/windows/win32/fileio/maxim...
    - dapperdrake 5mo ago
      Isn’t "cd" a unix syscall , because it changes the process's working directory? There was something written somewhere that it cannot be a unix utility for this very reason, but has to be a shell built-in. The syscall is a "single operation" from the point of a single-threaded process. What did I get wrong there? Side note: Missing bash$ man 1 cd ; Useful output bash$ help cd ;
      - comex 5mo ago
        Yes, it’s a shell builtin that makes the shell execute a chdir() syscall. Therefore it isn’t subject to argument length limits imposed by the kernel when executing processes. But it is still subject to path length limits imposed by the kernel’s implementation of chdir() itself. While the shell may be a GNU project (bash), the kernel generally is not (unless you are running Hurd), so this isn’t GNU’s fault per se. However, the shell could theoretically chunk long cd arguments into multiple calls to chdir(), splitting on slashes. I believe this would be fully semantically correct: you are not losing any atomicity guarantees because the kernel doesn’t provide such guarantees in the first place for lookups involving multiple path components. I’m not surprised that bash doesn’t bother implementing this, and I don’t know if I’d call that an “arbitrary limitation” on bash’s part (as opposed to a lack of workaround for another component’s arbitrary limitation). But it would be possible.

dapperdrake 5mo ago

First of all, thank you for presenting a succinct take on this viewpoint from the other side of the fence from where I am at. So how can I learn from this? (Asking very aggressively, especially for Internet writing, to make the contrast unmistakable. And contrast helps with perceiving differences and mistakes.) (You also don’t owe me any of your time or mental bandwidth, whatsoever.) So here goes: Question 1: How come "speed", "performance", race conditions and st_ino keep getting brought up? Speed (latency), physically writing things out to storage (sequentially, atomically (ACID), all of HDD NVME SSD ODD FDD tape, "haskell monad", event horizons, finite speed of light and information, whatever) as well as race conditions all seem to boil down to the same thing. For reliable systems like accounting the path seems to be ACID or the highway. And "unreliable" systems forget fast enough that computers don’t seem to really make a difference there. Question 2: Does throughput really matter more than latency in everyday application? Question 3 (explanation first, this time): The focus on inode numbers is at least understandable with regards to the history of C and unix-like operating systems and GNU coreutils. What about this basic example? Just make a USB thumb drive "work" for storing files (ignoring nand flash decay and USB). Without getting tripped up in libc IO buffering, fflush, kernel buffering (Hurd if you prefer it over Linux or FreeBSD), more than one application running on a multi-core and/or time-sliced system (to really weed out single-core CPUs running only a single user-land binary with blocking IO).

dijit 5mo ago
> Does throughput really matter more than latency in everyday application? In my experience latency and throughput are intrinsically linked unless you have the buffer-space to handle the throughput you want. Which you can't guarantee on all the systems where GNU Coreutils run.
- dapperdrake 5mo ago
  Higher throughput increases the risk of high latency. Low latency increases the risk of "wasted cycles”, i.e. lowers (machine) throughput. Helps with human discovery throughput, though. The sled.rs people had a well readable take on this in their performance guide.
5mo ago

cyberax 5mo ago

To be fair, Vec::set_len bug in Rust was in 2021. And even then it had to be annotated as `unsafe`. It was then deprecated and a linter check was added: https://github.com/rust-lang/rust-clippy/issues/7681 https://github.com/rust-lang/rust-clippy/issues/7681

Dr_Emann 5mo ago
To be even fair-er, it wasn't actually memory unsafety, it was "just" unsoundness, there was a type, that IF you gave it an io reader implementation that was weird, that implementation could see uninit data, or expose uninit data elsewhere, but the only readers actually used were well behaved readers.
- dapperdrake 5mo ago
  > well behaved readers. Around and around we go.
orlp 5mo ago
Vec::set_len is by no means deprecated. The lint you linked only covers a very specific unsound pattern using set_len.
- kibwen 5mo ago
  Indeed, and it doesn't need to be deprecated, because it's an API explicitly designed to give you low-level control where you need it, and because it is appropriately defined as an `unsafe` function with documented safety invariants that must be manually upheld in order for usage to be memory-safe. The documentation also suggests several other (safe) functions that should be used instead when possible, and provides correct usage examples: https://doc.rust-lang.org/std/vec/struct.Vec.html#method.set_len https://doc.rust-lang.org/std/vec/struct.Vec.html#method.set... .
  - bcjdjsndon 5mo ago
    > and because it is appropriately defined as an `unsafe` function with documented safety invariants that must be manually upheld in order for usage to be memory-safe. Didn't we learn from c, and the entire raison detre for rust, is that coders cannot be trusted to follow rules like this? If coders could "(document) safety invariants that must be manually upheld in order for usage to be memory-safe." there's be no need for Rust. This is the tautology underlying rust as I see it

theteapot 5mo ago

Probably a dumb question, but is GNU Core utils interested in / planning on doing its own rust rewrite?

greatgib 5mo ago
The rewrite in Rust is mostly vanity and marketing but not based on a real technical need... So I don't see why they would want to do that.
- kibwen 5mo ago
  Canonical's usage of uutils is likely for marketing. But the codebase itself was developed for fun, as an excuse for people to have a hands-on way to learn Rust back before Rust was even released, with a minor justification as being cross-platform. From the original README in 2013: Why? ---- Many GNU, linux and other utils are pretty awesome, and obviously some effort has been spent in the past to port them to windows. However those projects are either old, abandonned, hosted on CVS, written in platform-specific C, etc. Rust provides a good platform-agnostic way of writing systems utils that are easy to compile anywhere, and this is as good a way as any to try and learn it. https://github.com/uutils/coreutils/blob/9653ed81a2fbf393f420d9a8007c574d922018d1/README.md https://github.com/uutils/coreutils/blob/9653ed81a2fbf393f42...
  - PunchyHamster 5mo ago
    >Canonical's usage of uutils is likely for marketing Currently their usage is actively worsening the security of their distro
    - Avamander 5mo ago
      These things were caught and basically all of them weren't covered by any test suite (not even GNU coreutils'). It's a bit bold to claim that it's actively worsening it when it's not an LTS.
      - 5mo ago

lyrie 5mo ago

[dead]

gzread 5mo ago

I see even the coreutils maintainers find themselves needing -n (no newlines) and -c (count) options to "yes".

dapperdrake 5mo ago
GNU coreutils is known for adding command libe options. One of the big philosophical differences to the BSD's. For a human being, it sucks both ways.

jimmypk 5mo ago

[flagged]

safercplusplus 5mo ago

I don't know if you're aware, but there is a demonstration of wget (a fellow "gnu utility", right?) being auto-translated to a memory-safe subset of C++ [1]. Because the translation essentially does a one-for-one substitution of potentially unsafe C elements with safe C++ counterparts that mirror the behavior, the translation should be much less susceptible to the introduction of new bugs and behaviors in the way a rewrite would be. With a little cleaning-up of the original code, the code translation ends up being fully automatic and so can be used as a build step to produce (slightly slower) memory-safe executables from the original C source. [1] https://duneroadrunner.github.io/scpp_articles/PoC_autotranslation_of_wget https://duneroadrunner.github.io/scpp_articles/PoC_autotrans...

dapperdrake 5mo ago
Filesystem access is mostly treated by users as serialized ACID transactions on "files in directories." "Managing this resource centrally" is where unix syscalls came from. An OS kernel can be used like a specialized library for ACID transactions on hardware singletons. People then got fancy with virtual memory, interrupts, signals, time-slicing, re-entrancy, thread-safety, and injectivity. It doesn’t matter, whether you call the "kernel library" from C, C++, Fortan, BASIC, Golang, bash, Rust, etc.

joaohaas 5mo ago

>the article says "The Rust rewrite has shipped zero of these [memory saftey bugs], over a comparable window of activity." However, this is not true That bug got fixed before the Ubuntu release, and is from way before Canonical was even involved with the project.

rossvor 5mo ago
In the given list of GNU CVEs in the original article, it included a buffer overrun in tail from 2021. So for a fair comparison 2021 is part of the "window of activity" (the year uu_od CVE was published).

pornel 5mo ago

Indeed, std::fs suffers from being a lowest common denominator. Rust had to have something at 1.0, and unfortunately it stayed like that. Rust uutils would be a good place to design a more foolproof replacement for Rust's std::fs API.

dapperdrake 5mo ago
Unix embodies this, as well. When K&R created unix and C there was still the better option of moving changes that were better to have in the "kernel" into the kernel. Now we have "standards" that even cause headaches between Linux and BSD's. Linux back-propagates stuff like mmap, io_uring, etc. to where it belongs. In this way it is like the original unix. And deservedly running on most servers out there.