3 ms·

A nice example of how a simple program can be done in Rust. I think that if some of the setup (mkfs, mount) is left to external tools¹, it show clearly how sim

by berkes 3y ago

A nice example of how a simple program can be done in Rust. I think that if some of the setup (mkfs, mount) is left to external tools¹, it show clearly how simple a program in Rust can be. Sure, in Ruby, python, or even bash, it would be two or three lines, under ten statements and probably under 200 characters while remaining readable. But still it shows us how the idea that "in rust there is a lot of boilerplate" isn't really a practical issue. I write most of my simple tools in rust lately. Needed a million URLs parsed and matched against denylists last week: my Python code took 5 minutes to run that (way too long for trial-error), the bash cut,sed,awk,grep pipe-chain quickly too wieldy to keep playing around with (yet seconds to run, not minutes). In a few minutes I was running a rust version that used all my cores, well readable and easy to iterate with. I guess go would be the perfect fit, but I don't speak go that well (nor have its tooling ready and up-front). ¹something I'd do regardless. If only to follow the part of the Unix philosophy "do one thing".

JackSlateur 3y ago
Well, reading the rust code, this is nothing but a couple of exec and one loop Something like, in bash: test -f output.img || (fallocate -s 1T output.img && mkfs.ext4 output.img) mkdir -p toto mount output.img toto for i in {1..1000000000} do touch toto/$i done umount toto Or am I missing something ?
- chx 3y ago
  Having a billion files in a single directory versus a more typical tree has its own challenges. http://be-n.com/spw/you-can-list-a-million-files-in-a-directory-but-not-with-ls.html http://be-n.com/spw/you-can-list-a-million-files-in-a-direct...
  - jeffbee 3y ago
    Possibly, but at the scale discussed in the article, not so much. % time ls | wc 10000000 10000000 78888897 ls 1.90s user 0.73s system 99% cpu 2.637 total wc 0.28s user 0.02s system 11% cpu 2.597 total ETA: and most of that time is spent sorting the output. % time ls -U | wc 10000000 10000000 78888897 ls -U 0.53s user 0.29s system 99% cpu 0.823 total wc 0.26s user 0.02s system 33% cpu 0.822 total
    - sllabres 3y ago
      To my experience (and if only the filename is needed) the "-1" option helps saving some time too. Be aware that output from ls without options differ, if redirected. time (real user sys) in seconds with 1e6 files in a xfs filesystem. Directory used 27 MB space. ls -l > /dev/null : 49.12 23.90 13.47 ls -lU > /dev/null : 46.62 33.11 8.01 ls > /dev/null : 6.36 1.97 3.63 ls -1 > /dev/null : 2.49 1.83 0.36 ls -1U > /dev/null : 0.31 0.18 0.08 (find) : 0.77 0.49 0.19 and for 1e6 files in an ext4 filesystem. Directory used 23 MB space. (but as i had no ext4 fs with one million free inodes on a loop device) ls -l > /dev/null : 51.39 34.02 11.39 ls -lU > /dev/null : 47.39 30.72 11.09 ls > /dev/null : 4.83 3.56 0.65 ls -1 > /dev/null : 4.46 3.45 0.50 ls -1U > /dev/null : 0.51 0.18 0.26 (find) : 1.29 0.76 0.36
      - jeffbee 3y ago
        Good info. But isn't `-1` implicit when the output is a pipe?
        sllabres 3y ago
        Good reply, you are right. I have even written ("Be aware that output from ls without options differ, if redirected") that it is so but was somewhat fooled by my experiment and the different timings for "ls" and "ls -1" when using XFS. Of course I was not able to reproduce this difference and assume that I did work on the system and it was not idle. I have repeated the the tests, on another mostly idle system. (Therefor the timing is not comparable to the first test). This time both filesystems live in a loop device and the benchmarks were run with hyperfine (3 warmup loops, 3 measurements each). XFS ls -l >/dev/null Time (mean ± σ): 5.890 s ± 0.394 s [User: 1.096 s, System: 3.491 s] ls -lU >/dev/null Time (mean ± σ): 2.254 s ± 0.386 s [User: 0.648 s, System: 1.602 s] ls >/dev/null Time (mean ± σ): 542.7 ms ± 4.4 ms [User: 443.8 ms, System: 98.7 ms] ls -1 >/dev/null Time (mean ± σ): 534.4 ms ± 2.3 ms [User: 450.1 ms, System: 84.1 ms] ls -1U >/dev/null Time (mean ± σ): 115.8 ms ± 1.8 ms [User: 80.9 ms, System: 34.6 ms] ls -U >/dev/null Time (mean ± σ): 121.0 ms ± 14.6 ms [User: 80.9 ms, System: 40.0 ms] find >/dev/null Time (mean ± σ): 225.0 ms ± 16.2 ms [User: 150.1 ms, System: 74.7 ms] EXT4 ls -l >/dev/null Time (mean ± σ): 10.493 s ± 0.392 s [User: 2.495 s, System: 5.032 s] ls -lU >/dev/null Time (mean ± σ): 8.395 s ± 0.157 s [User: 0.735 s, System: 5.049 s] ls >/dev/null Time (mean ± σ): 1.966 s ± 0.010 s [User: 1.760 s, System: 0.202 s] ls -1 >/dev/null Time (mean ± σ): 2.014 s ± 0.032 s [User: 1.798 s, System: 0.216 s] ls -1U >/dev/null Time (mean ± σ): 221.4 ms ± 1.3 ms [User: 83.7 ms, System: 137.6 ms] ls -U >/dev/null Time (mean ± σ): 222.5 ms ± 2.0 ms [User: 78.9 ms, System: 143.6 ms] find >/dev/null Time (mean ± σ): 531.0 ms ± 4.0 ms [User: 377.3 ms, System: 153.7 ms]
  - JackSlateur 3y ago
    That is easily implemented with a double loop, where the first one is a directory (integer hexed with printf '%x') and the second is the files in it However, that was not the point of my reply
tyingq 3y ago
For it to be a good example, I'd make some changes. As one example, there are a few places where stderr is thrown away in an error and substituted for a very generic failure message. And some where the failure is assumed to be something that it may not be. Though, I understand the author wasn't claiming to deliver an example of how to do "scripting" in Rust.

by berkes 3y ago

JackSlateur 3y ago

Well, reading the rust code, this is nothing but a couple of exec and one loop Something like, in bash: test -f output.img || (fallocate -s 1T output.img && mkfs.ext4 output.img) mkdir -p toto mount output.img toto for i in {1..1000000000} do touch toto/$i done umount toto Or am I missing something ?

chx 3y ago
Having a billion files in a single directory versus a more typical tree has its own challenges. http://be-n.com/spw/you-can-list-a-million-files-in-a-directory-but-not-with-ls.html http://be-n.com/spw/you-can-list-a-million-files-in-a-direct...
- jeffbee 3y ago
  Possibly, but at the scale discussed in the article, not so much. % time ls | wc 10000000 10000000 78888897 ls 1.90s user 0.73s system 99% cpu 2.637 total wc 0.28s user 0.02s system 11% cpu 2.597 total ETA: and most of that time is spent sorting the output. % time ls -U | wc 10000000 10000000 78888897 ls -U 0.53s user 0.29s system 99% cpu 0.823 total wc 0.26s user 0.02s system 33% cpu 0.822 total
  - sllabres 3y ago
    To my experience (and if only the filename is needed) the "-1" option helps saving some time too. Be aware that output from ls without options differ, if redirected. time (real user sys) in seconds with 1e6 files in a xfs filesystem. Directory used 27 MB space. ls -l > /dev/null : 49.12 23.90 13.47 ls -lU > /dev/null : 46.62 33.11 8.01 ls > /dev/null : 6.36 1.97 3.63 ls -1 > /dev/null : 2.49 1.83 0.36 ls -1U > /dev/null : 0.31 0.18 0.08 (find) : 0.77 0.49 0.19 and for 1e6 files in an ext4 filesystem. Directory used 23 MB space. (but as i had no ext4 fs with one million free inodes on a loop device) ls -l > /dev/null : 51.39 34.02 11.39 ls -lU > /dev/null : 47.39 30.72 11.09 ls > /dev/null : 4.83 3.56 0.65 ls -1 > /dev/null : 4.46 3.45 0.50 ls -1U > /dev/null : 0.51 0.18 0.26 (find) : 1.29 0.76 0.36
    - jeffbee 3y ago
      Good info. But isn't `-1` implicit when the output is a pipe?
      - sllabres 3y ago
        Good reply, you are right. I have even written ("Be aware that output from ls without options differ, if redirected") that it is so but was somewhat fooled by my experiment and the different timings for "ls" and "ls -1" when using XFS. Of course I was not able to reproduce this difference and assume that I did work on the system and it was not idle. I have repeated the the tests, on another mostly idle system. (Therefor the timing is not comparable to the first test). This time both filesystems live in a loop device and the benchmarks were run with hyperfine (3 warmup loops, 3 measurements each). XFS ls -l >/dev/null Time (mean ± σ): 5.890 s ± 0.394 s [User: 1.096 s, System: 3.491 s] ls -lU >/dev/null Time (mean ± σ): 2.254 s ± 0.386 s [User: 0.648 s, System: 1.602 s] ls >/dev/null Time (mean ± σ): 542.7 ms ± 4.4 ms [User: 443.8 ms, System: 98.7 ms] ls -1 >/dev/null Time (mean ± σ): 534.4 ms ± 2.3 ms [User: 450.1 ms, System: 84.1 ms] ls -1U >/dev/null Time (mean ± σ): 115.8 ms ± 1.8 ms [User: 80.9 ms, System: 34.6 ms] ls -U >/dev/null Time (mean ± σ): 121.0 ms ± 14.6 ms [User: 80.9 ms, System: 40.0 ms] find >/dev/null Time (mean ± σ): 225.0 ms ± 16.2 ms [User: 150.1 ms, System: 74.7 ms] EXT4 ls -l >/dev/null Time (mean ± σ): 10.493 s ± 0.392 s [User: 2.495 s, System: 5.032 s] ls -lU >/dev/null Time (mean ± σ): 8.395 s ± 0.157 s [User: 0.735 s, System: 5.049 s] ls >/dev/null Time (mean ± σ): 1.966 s ± 0.010 s [User: 1.760 s, System: 0.202 s] ls -1 >/dev/null Time (mean ± σ): 2.014 s ± 0.032 s [User: 1.798 s, System: 0.216 s] ls -1U >/dev/null Time (mean ± σ): 221.4 ms ± 1.3 ms [User: 83.7 ms, System: 137.6 ms] ls -U >/dev/null Time (mean ± σ): 222.5 ms ± 2.0 ms [User: 78.9 ms, System: 143.6 ms] find >/dev/null Time (mean ± σ): 531.0 ms ± 4.0 ms [User: 377.3 ms, System: 153.7 ms]
- JackSlateur 3y ago
  That is easily implemented with a double loop, where the first one is a directory (integer hexed with printf '%x') and the second is the files in it However, that was not the point of my reply

tyingq 3y ago

For it to be a good example, I'd make some changes. As one example, there are a few places where stderr is thrown away in an error and substituted for a very generic failure message. And some where the failure is assumed to be something that it may not be. Though, I understand the author wasn't claiming to deliver an example of how to do "scripting" in Rust.