6 ms·

This is an extremely amazing project because it's actually trying to come up with a replacement that can be used as such. Nobody is going to rewrite scripts to

by est31 5y ago

This is an extremely amazing project because it's actually trying to come up with a replacement that can be used as such. Nobody is going to rewrite scripts to use exa instead of ls for example. The coreutils are also somewhat the "basis" of the shell userspace. Outside of the test failures and missing features, the only reason one might not want the coreutils is the size. Right now, cat on my system is 44K large, while the default output size for the cat executable in release mode is 4.4 megabytes in release mode. If you enable a bunch of options in Cargo.toml to reduce the default bloaty settings (lto = true, codegen-units = 1, strip = true, debug = false), you get that down to 876K. Which is still 20 times larger than the native cat. For true, it's a similar story. 40K vs 812K.

sylvestre 5y ago
I added some CI task and replayed the history to graph the evolution of the size: https://github.com/uutils/coreutils-tracking#binary-size-evolution https://github.com/uutils/coreutils-tracking#binary-size-evo... This will help making sure that we aren't regressing (more) ;)
bsdetector 5y ago
It'd also be a great time to establish a better standard for using these programs with a shell or scripting. For example, if ls had a --shell option that would write out file information as quoted eval'able variables, or even JSON, or anything that was easily and reliably parsable it would remove a huge portion of scripting headaches and errors.
andai 5y ago
Is that a typical ratio for software ported to Rust? Where is all the extra size coming from?
- rpeden 5y ago
  As someone else mentioned, shared libraries. If you built ls and statically linked libc and whatever else it needs, ls would be a lot bigger.
- est31 5y ago
  According to cargo bloat, 40.1% of the .text section comes from clap, and 46.7% of the .text section comes from std. clap is quite famous to be extremely bloaty. To give an answer to your other question: a lot of the additional code size is a one time cost. That means that the relative difference is the largest in these cases, for binaries with little logic of their own. The more complex the program gets, the less the relative difference is. However, even if you don't account for these one time costs, Rust creates larger programs than C, due to heavy use of generics compared to C.
  - Ar-Curunir 5y ago
    Maybe clap should store text strings in compressed form, and then decompress as needed. Could greatly reduce the bloat.
    - mlindner 5y ago
      .text is the section of the binary where the code is.
- duckerude 5y ago
  At least 500K comes from clap, the argument parsing library. That's how much true's size drops if you comment it out. It's a lovely library. Generated help text with colors and line wrapping based on your terminal width, informative error messages that suggest corrections for typos, and a declarative model that lets you express subtle relationships between options. You don't get that from getopt. But it's also pretty large. https://github.com/rust-cli/argparse-benchmarks-rs https://github.com/rust-cli/argparse-benchmarks-rs has some numbers. (Disclaimer: I wrote one of the competitors on that page, lexopt. But I also use clap, depending on the project, and I'm happy with it.)
  - camel-cdr 5y ago
    Out of curiosity, why should each application implement line wrapping for terminals. That seems like a feature the terminal should implement, and from what I've read there is no standard way of getting the terminal size anyway.
    - zaarn 5y ago
      The terminal isn't aware that you're, for example, using an indent because the help text of an option should appear indented from the option it's explaining. So the line wrap will go into the next line and ruin the readability.
      - camel-cdr 5y ago
        Ah, that makes sense.
    - ksherlock 5y ago
      There's the TIOCGWINSZ ioctl which seems pretty standard.
  - twsted 5y ago
    500K to parse options is frankly too much...
cogman10 5y ago
I believe what you are seeing there is the power of shared libraries.
ris 5y ago
My guess is people will start "solving" this kind of problem through multi-call binaries a la busybox.
- duckerude 5y ago
  uutils already (optionally) builds that way. The GitHub release has a single 8.4M binary. That doesn't seem so bad, the GNU coreutils Debian package has an Installed-Size of 17.9MB. Though there might be hidden drawbacks I don't know about.
  - est31 5y ago
    > The GitHub release has a single 8.4M binary. That's a definitely acceptable size, thanks for pointing that out. Yeah a bunch of symlinks should solve the issue. If the binary is kept cached in RAM at least, there should be no overhead in starting it. I think it's pretty rare to have environments that are so memory constrained that they can't hold onto 8 MB of RAM. Note that not all targets are that small though. Musl has indeed 8.3 MiB, but in coreutils-0.0.12-x86_64-unknown-linux-gnu.tar.gz, the size of the coreutils executable is 12.6 MiB. On the bright side, on the same target, a local build with the cargo features to reduce bloat enabled gives only 7.1M. All these numbers are acceptable.
    - deleted 5y ago
      [deleted]
  - loeg 5y ago
    Is anything in coreutils setuid? That’s the only possible drawback I have in mind.
  - ris 5y ago
    Yes, the only reason I say "solved" is that this approach will only help for suites of many small tools that ship together.
deleted 5y ago
[deleted]
lr1970 5y ago
Rust executables are statically linked by default. This makes them oftentimes substantially larger than their dynamically linked counterparts. For me it is not a deal breaker.

by est31 5y ago

sylvestre 5y ago

I added some CI task and replayed the history to graph the evolution of the size: https://github.com/uutils/coreutils-tracking#binary-size-evolution https://github.com/uutils/coreutils-tracking#binary-size-evo... This will help making sure that we aren't regressing (more) ;)

bsdetector 5y ago

It'd also be a great time to establish a better standard for using these programs with a shell or scripting. For example, if ls had a --shell option that would write out file information as quoted eval'able variables, or even JSON, or anything that was easily and reliably parsable it would remove a huge portion of scripting headaches and errors.

andai 5y ago

Is that a typical ratio for software ported to Rust? Where is all the extra size coming from?

rpeden 5y ago
As someone else mentioned, shared libraries. If you built ls and statically linked libc and whatever else it needs, ls would be a lot bigger.
est31 5y ago
According to cargo bloat, 40.1% of the .text section comes from clap, and 46.7% of the .text section comes from std. clap is quite famous to be extremely bloaty. To give an answer to your other question: a lot of the additional code size is a one time cost. That means that the relative difference is the largest in these cases, for binaries with little logic of their own. The more complex the program gets, the less the relative difference is. However, even if you don't account for these one time costs, Rust creates larger programs than C, due to heavy use of generics compared to C.
- Ar-Curunir 5y ago
  Maybe clap should store text strings in compressed form, and then decompress as needed. Could greatly reduce the bloat.
  - mlindner 5y ago
    .text is the section of the binary where the code is.
duckerude 5y ago
At least 500K comes from clap, the argument parsing library. That's how much true's size drops if you comment it out. It's a lovely library. Generated help text with colors and line wrapping based on your terminal width, informative error messages that suggest corrections for typos, and a declarative model that lets you express subtle relationships between options. You don't get that from getopt. But it's also pretty large. https://github.com/rust-cli/argparse-benchmarks-rs https://github.com/rust-cli/argparse-benchmarks-rs has some numbers. (Disclaimer: I wrote one of the competitors on that page, lexopt. But I also use clap, depending on the project, and I'm happy with it.)
- camel-cdr 5y ago
  Out of curiosity, why should each application implement line wrapping for terminals. That seems like a feature the terminal should implement, and from what I've read there is no standard way of getting the terminal size anyway.
  - zaarn 5y ago
    The terminal isn't aware that you're, for example, using an indent because the help text of an option should appear indented from the option it's explaining. So the line wrap will go into the next line and ruin the readability.
    - camel-cdr 5y ago
      Ah, that makes sense.
  - ksherlock 5y ago
    There's the TIOCGWINSZ ioctl which seems pretty standard.
- twsted 5y ago
  500K to parse options is frankly too much...

cogman10 5y ago

I believe what you are seeing there is the power of shared libraries.

ris 5y ago

My guess is people will start "solving" this kind of problem through multi-call binaries a la busybox.

duckerude 5y ago
uutils already (optionally) builds that way. The GitHub release has a single 8.4M binary. That doesn't seem so bad, the GNU coreutils Debian package has an Installed-Size of 17.9MB. Though there might be hidden drawbacks I don't know about.
- est31 5y ago
  > The GitHub release has a single 8.4M binary. That's a definitely acceptable size, thanks for pointing that out. Yeah a bunch of symlinks should solve the issue. If the binary is kept cached in RAM at least, there should be no overhead in starting it. I think it's pretty rare to have environments that are so memory constrained that they can't hold onto 8 MB of RAM. Note that not all targets are that small though. Musl has indeed 8.3 MiB, but in coreutils-0.0.12-x86_64-unknown-linux-gnu.tar.gz, the size of the coreutils executable is 12.6 MiB. On the bright side, on the same target, a local build with the cargo features to reduce bloat enabled gives only 7.1M. All these numbers are acceptable.
  - deleted 5y ago
    [deleted]
- loeg 5y ago
  Is anything in coreutils setuid? That’s the only possible drawback I have in mind.
- ris 5y ago
  Yes, the only reason I say "solved" is that this approach will only help for suites of many small tools that ship together.

deleted 5y ago

[deleted]

lr1970 5y ago

Rust executables are statically linked by default. This makes them oftentimes substantially larger than their dynamically linked counterparts. For me it is not a deal breaker.