9 ms·

Is anyone else impressed with the quality (and speed!) of some of the tools written in Rust? I'm an avid user of fd and bat, the former being ridiculously fast.

by orf 8y ago

Is anyone else impressed with the quality (and speed!) of some of the tools written in Rust? I'm an avid user of fd and bat, the former being ridiculously fast. Often I find something on github, I'm impressed by the quality of the documentation, features, UI etc, then lo and behold it's written in Rust. Another one potentially for this list is tokei[1] I was trying to count the code in our repos at work and used the venerable 'cloc' utility. It took over 5 minutes. Looking around I found tokei, written in rust. Same-ish results (more accurate actually) took 10 seconds. 1. https://github.com/Aaronepower/tokei https://github.com/Aaronepower/tokei

steveklabnik 8y ago
IIRC, loc (in rust) might be even faster than tokei at times? I always forget, and I’m sure I saw an old benchmark... We have a whole working group this year focused on making the experience of writing CLIs awesome, so hopefully we’ll see even more great tools in the future!
- vram22 8y ago
  >We have a whole working group this year focused on making the experience of writing CLIs awesome Can you explain what you mean by this? Something like the people are going to focus on language and library features that help with writing CLIs? Or something else? Asking because CLIs are one of my interests.
  - isaacaggrey 8y ago
    Here is the Rust CLI working group announcement: https://internals.rust-lang.org/t/announcing-the-cli-working-group/6872 https://internals.rust-lang.org/t/announcing-the-cli-working... Rust has different community working groups that focus on improving the Rust ecosystem in different ways this year: https://internals.rust-lang.org/t/announcing-the-2018-domain-working-groups/6737 https://internals.rust-lang.org/t/announcing-the-2018-domain...
    - vram22 8y ago
      Thank you, will take a look.
  - epage 8y ago
    We've tried to dig into various problematic areas of writing CLIs in Rust [0], worked to create or improve libraries [1], and are working on writing a "book" for Rust-based CLIs [2]. You can reach out to us on gitter [3]. [0]: https://github.com/rust-lang-nursery/cli-wg/issues https://github.com/rust-lang-nursery/cli-wg/issues [1]: https://github.com/rust-clique/ https://github.com/rust-clique/ and https://github.com/assert-rs/ https://github.com/assert-rs/ [2]: https://rust-lang-nursery.github.io/cli-wg/index.html https://rust-lang-nursery.github.io/cli-wg/index.html [3]: https://gitter.im/rust-lang/WG-CLI https://gitter.im/rust-lang/WG-CLI The areas we're trying to finish up for Rust 2018 Edition: - Get clap (arg parsing) to 3.0 [4] - Get assert_cmd [5] / assert_fs [6] to 1.0 - Finish work on man-page generation [7] - Make it easier to package binaries [8] and document the CI for it [9] [4]: https://github.com/rust-lang-nursery/cli-wg/issues/41 https://github.com/rust-lang-nursery/cli-wg/issues/41 [5]: https://github.com/assert-rs/assert_cmd/ https://github.com/assert-rs/assert_cmd/ [6]: https://github.com/assert-rs/assert_fs https://github.com/assert-rs/assert_fs [7]: https://github.com/rust-lang-nursery/cli-wg/issues/38 https://github.com/rust-lang-nursery/cli-wg/issues/38 [8]: https://github.com/crate-ci/cargo-tarball https://github.com/crate-ci/cargo-tarball [9]: https://github.com/crate-ci/crate-ci.github.io https://github.com/crate-ci/crate-ci.github.io
    - vram22 8y ago
      Thanks for the detailed answer. Will check out some of those links.
- cgag 8y ago
  Loc (mine) is usually faster in my tests but doesn't handle comments that start in strings, so if there's something like x="/*" it can be way off, so I usually still point people at tokei. I need to try implementing the string thing. It'd be a lot more fun to try to compete on speed if we were the same on accuracy.
- Aaronepower 8y ago
  Well luckily I recently published a new comparison benchmark[0]. :) The TL;DR is that loc is faster by a few hundred milliseconds depending on repository size, but as cgag mentions doesn't have comment in string detection so can be quite off in its metrics, for example on the Rust repo Tokei says it has 643,754 lines of code where as loc says it's 635,849. [0]: https://github.com/Aaronepower/tokei/blob/master/COMPARISON.md https://github.com/Aaronepower/tokei/blob/master/COMPARISON....
  - steveklabnik 8y ago
    Awesome! :)
dijit 8y ago
I wonder if there's a name for this phenomenon. New and not-popularly adopted languages tend to have more senior people learning and developing software using them; this leads to people (read: recruiters) looking for the people with experience.. People equate the quality of software with something innate with the language, or that the developers are exceptional. Then after being popularly adopted it slumps in perceived elegance and goes on a downward spiral until it ends up like PHP, Ruby or JAVA. (not to undersell the folks who predominantly use those languages, I'm just picking languages that I've seen follow this pattern)
- shimon 8y ago
  The Python Paradox (2004): http://www.paulgraham.com/pypar.html http://www.paulgraham.com/pypar.html The name may not have aged well...
- adwhit 8y ago
  A more generous interpretation might be that Rust has sparked a CLI renaissance, since it lets developers write CLIs that have that satisfying zip that previously was only possible in C. None of PHP, Ruby or Java is responsive enough for a good CLI tool (also, static compilation is a must for wide deployment).
  - jetti 8y ago
    Not to mention you get a language that is enjoyable but also compiles to native code. My first Rust project was a Haml parser and I created a CLI for it too that is similar to the Ruby version. It is nice being able to ship the 8Mb executable and not have to worry about users having the right runtime.
    - pjmlp 8y ago
      Just like any other compiled language. GCC's static linking issues with glibc don't apply to other compilers.
      - 8y ago
the8472 8y ago
I had a case where tar was too slow for my needs. Rust did let me cobble the right syscalls and threads together to make it more HDD-friendly and faster while forcing me to handle all the gritty filesystem error cases from the start. Lo and behold, about 4 times faster on an idle system and orders of magnitude faster on a busy system. It just would not have been possible to do it by combining shell utilities and too finicky in C.
- thethirdone 8y ago
  Do you have the code posted online? I would like to take a look through it.
  - the8472 8y ago
    https://github.com/the8472/fastar https://github.com/the8472/fastar
    - tretiy3 8y ago
      Only 117 lines of code? Impressive.
      - the8472 8y ago
        I have moved the meat of the optimizations to separate libraries, they're linked in the readme.
- carlmr 8y ago
  That's the thing with Rust, it makes you work hard right from the start to make everything right, but when you have something working it's usually fast, too.
topspin 8y ago
tokei is actually the very first Rust program that I found 'in the wild' while searching for a tool. I had run many Rust programs prior to then, but they had always been in the context of experimenting with Rust.
vram22 8y ago
Have you done any comparison (even informal) of similar tools written in Rust and Go?
- vram22 8y ago
  I'm asking because although I know Rust is likely to be faster than Go, I'm interested in the amount of difference.
  - burntsushi 8y ago
    You might be interested in Ben Boyter's journey to write a source code line counter: https://boyter.org/posts/sloc-cloc-code/ https://boyter.org/posts/sloc-cloc-code/ https://boyter.org/posts/why-count-lines-of-code/ https://boyter.org/posts/why-count-lines-of-code/
    - vram22 8y ago
      Thanks, will take a look at those.
- bostonvaulter2 8y ago
  A very informal analysis of bat and ccat is that ccat (written in Go) has choked on large files multiple times (which resulted in panics), while bat hasn't crashed on me yet. Of course I've only tried bat on two files so far ;)
xxpor 8y ago
I'm surprised they said ack || ag over ripgrep. I'm a total convert. It's AMAZINGLY fast.
- KingMob 8y ago
  In the comments, the author admitted he hadn't heard of ripgrep, and would give it a try.
prakashdanish 8y ago
Since we're on the subject, I would ask this question I've been meaning to ask for a while. I love writing command line tools and I've written a few in Python but the performance isn't there. Would you suggest me to learn Rust in order to write CLI programs? I'm also looking at haskell for the same but after this thread, I'm really thinking of going the rust way. Ideas?
- cgag 8y ago
  I personally would strongly recommend going the rust route having spent a decent amount of time learning Haskell. I think youll spend more time working on real problems and learn more about low level programming, in addition to the code being much faster.
- lyqwyd 8y ago
  I would say it depends on your goals. If performance is your primary concern, then go with rust, one of rust’s primary focuses is performance. I don’t think of Haskell as the best language for command line dev, but I’m far from expert in either language.
- mixmastamyk 8y ago
  Also try cython.
akrasuski1 8y ago
This could actually be caused by disk caching effects. Have you tried running cloc second time afterwards?
- rofrol 8y ago
  I tried cloc multiple times. tokei much faster. Try it yourself.
hultner 8y ago
tokei is fast but isn't the smartest tool (at least for python). I compared it to pygount (and looked on individual files). Total lines is correct but it's grossly miscalculating code vs comments resulting in almost 2x total actual SLOC for a semi large project I manage. I manually counted a few files for comparison, I believe tokei doesn't understand python-doc-comments and think it's code.

by orf 8y ago

steveklabnik 8y ago

IIRC, loc (in rust) might be even faster than tokei at times? I always forget, and I’m sure I saw an old benchmark... We have a whole working group this year focused on making the experience of writing CLIs awesome, so hopefully we’ll see even more great tools in the future!

vram22 8y ago
>We have a whole working group this year focused on making the experience of writing CLIs awesome Can you explain what you mean by this? Something like the people are going to focus on language and library features that help with writing CLIs? Or something else? Asking because CLIs are one of my interests.
- isaacaggrey 8y ago
  Here is the Rust CLI working group announcement: https://internals.rust-lang.org/t/announcing-the-cli-working-group/6872 https://internals.rust-lang.org/t/announcing-the-cli-working... Rust has different community working groups that focus on improving the Rust ecosystem in different ways this year: https://internals.rust-lang.org/t/announcing-the-2018-domain-working-groups/6737 https://internals.rust-lang.org/t/announcing-the-2018-domain...
  - vram22 8y ago
    Thank you, will take a look.
- epage 8y ago
  We've tried to dig into various problematic areas of writing CLIs in Rust [0], worked to create or improve libraries [1], and are working on writing a "book" for Rust-based CLIs [2]. You can reach out to us on gitter [3]. [0]: https://github.com/rust-lang-nursery/cli-wg/issues https://github.com/rust-lang-nursery/cli-wg/issues [1]: https://github.com/rust-clique/ https://github.com/rust-clique/ and https://github.com/assert-rs/ https://github.com/assert-rs/ [2]: https://rust-lang-nursery.github.io/cli-wg/index.html https://rust-lang-nursery.github.io/cli-wg/index.html [3]: https://gitter.im/rust-lang/WG-CLI https://gitter.im/rust-lang/WG-CLI The areas we're trying to finish up for Rust 2018 Edition: - Get clap (arg parsing) to 3.0 [4] - Get assert_cmd [5] / assert_fs [6] to 1.0 - Finish work on man-page generation [7] - Make it easier to package binaries [8] and document the CI for it [9] [4]: https://github.com/rust-lang-nursery/cli-wg/issues/41 https://github.com/rust-lang-nursery/cli-wg/issues/41 [5]: https://github.com/assert-rs/assert_cmd/ https://github.com/assert-rs/assert_cmd/ [6]: https://github.com/assert-rs/assert_fs https://github.com/assert-rs/assert_fs [7]: https://github.com/rust-lang-nursery/cli-wg/issues/38 https://github.com/rust-lang-nursery/cli-wg/issues/38 [8]: https://github.com/crate-ci/cargo-tarball https://github.com/crate-ci/cargo-tarball [9]: https://github.com/crate-ci/crate-ci.github.io https://github.com/crate-ci/crate-ci.github.io
  - vram22 8y ago
    Thanks for the detailed answer. Will check out some of those links.
cgag 8y ago
Loc (mine) is usually faster in my tests but doesn't handle comments that start in strings, so if there's something like x="/*" it can be way off, so I usually still point people at tokei. I need to try implementing the string thing. It'd be a lot more fun to try to compete on speed if we were the same on accuracy.
Aaronepower 8y ago
Well luckily I recently published a new comparison benchmark[0]. :) The TL;DR is that loc is faster by a few hundred milliseconds depending on repository size, but as cgag mentions doesn't have comment in string detection so can be quite off in its metrics, for example on the Rust repo Tokei says it has 643,754 lines of code where as loc says it's 635,849. [0]: https://github.com/Aaronepower/tokei/blob/master/COMPARISON.md https://github.com/Aaronepower/tokei/blob/master/COMPARISON....
- steveklabnik 8y ago
  Awesome! :)

dijit 8y ago

I wonder if there's a name for this phenomenon. New and not-popularly adopted languages tend to have more senior people learning and developing software using them; this leads to people (read: recruiters) looking for the people with experience.. People equate the quality of software with something innate with the language, or that the developers are exceptional. Then after being popularly adopted it slumps in perceived elegance and goes on a downward spiral until it ends up like PHP, Ruby or JAVA. (not to undersell the folks who predominantly use those languages, I'm just picking languages that I've seen follow this pattern)

shimon 8y ago
The Python Paradox (2004): http://www.paulgraham.com/pypar.html http://www.paulgraham.com/pypar.html The name may not have aged well...
adwhit 8y ago
A more generous interpretation might be that Rust has sparked a CLI renaissance, since it lets developers write CLIs that have that satisfying zip that previously was only possible in C. None of PHP, Ruby or Java is responsive enough for a good CLI tool (also, static compilation is a must for wide deployment).
- jetti 8y ago
  Not to mention you get a language that is enjoyable but also compiles to native code. My first Rust project was a Haml parser and I created a CLI for it too that is similar to the Ruby version. It is nice being able to ship the 8Mb executable and not have to worry about users having the right runtime.
  - pjmlp 8y ago
    Just like any other compiled language. GCC's static linking issues with glibc don't apply to other compilers.
    - 8y ago

the8472 8y ago

I had a case where tar was too slow for my needs. Rust did let me cobble the right syscalls and threads together to make it more HDD-friendly and faster while forcing me to handle all the gritty filesystem error cases from the start. Lo and behold, about 4 times faster on an idle system and orders of magnitude faster on a busy system. It just would not have been possible to do it by combining shell utilities and too finicky in C.

thethirdone 8y ago
Do you have the code posted online? I would like to take a look through it.
- the8472 8y ago
  https://github.com/the8472/fastar https://github.com/the8472/fastar
  - tretiy3 8y ago
    Only 117 lines of code? Impressive.
    - the8472 8y ago
      I have moved the meat of the optimizations to separate libraries, they're linked in the readme.
carlmr 8y ago
That's the thing with Rust, it makes you work hard right from the start to make everything right, but when you have something working it's usually fast, too.

topspin 8y ago

tokei is actually the very first Rust program that I found 'in the wild' while searching for a tool. I had run many Rust programs prior to then, but they had always been in the context of experimenting with Rust.

vram22 8y ago

Have you done any comparison (even informal) of similar tools written in Rust and Go?

vram22 8y ago
I'm asking because although I know Rust is likely to be faster than Go, I'm interested in the amount of difference.
- burntsushi 8y ago
  You might be interested in Ben Boyter's journey to write a source code line counter: https://boyter.org/posts/sloc-cloc-code/ https://boyter.org/posts/sloc-cloc-code/ https://boyter.org/posts/why-count-lines-of-code/ https://boyter.org/posts/why-count-lines-of-code/
  - vram22 8y ago
    Thanks, will take a look at those.
bostonvaulter2 8y ago
A very informal analysis of bat and ccat is that ccat (written in Go) has choked on large files multiple times (which resulted in panics), while bat hasn't crashed on me yet. Of course I've only tried bat on two files so far ;)

xxpor 8y ago

I'm surprised they said ack || ag over ripgrep. I'm a total convert. It's AMAZINGLY fast.

KingMob 8y ago
In the comments, the author admitted he hadn't heard of ripgrep, and would give it a try.

prakashdanish 8y ago

Since we're on the subject, I would ask this question I've been meaning to ask for a while. I love writing command line tools and I've written a few in Python but the performance isn't there. Would you suggest me to learn Rust in order to write CLI programs? I'm also looking at haskell for the same but after this thread, I'm really thinking of going the rust way. Ideas?

cgag 8y ago
I personally would strongly recommend going the rust route having spent a decent amount of time learning Haskell. I think youll spend more time working on real problems and learn more about low level programming, in addition to the code being much faster.
lyqwyd 8y ago
I would say it depends on your goals. If performance is your primary concern, then go with rust, one of rust’s primary focuses is performance. I don’t think of Haskell as the best language for command line dev, but I’m far from expert in either language.
mixmastamyk 8y ago
Also try cython.

akrasuski1 8y ago

This could actually be caused by disk caching effects. Have you tried running cloc second time afterwards?

rofrol 8y ago
I tried cloc multiple times. tokei much faster. Try it yourself.

hultner 8y ago

tokei is fast but isn't the smartest tool (at least for python). I compared it to pygount (and looked on individual files). Total lines is correct but it's grossly miscalculating code vs comments resulting in almost 2x total actual SLOC for a semi large project I manage. I manually counted a few files for comparison, I believe tokei doesn't understand python-doc-comments and think it's code.