8 ms·

My complaint about the async rust world isn't so much function coloring, but dependency craziness. For instance, the postgres driver for rust (which is an exce

My complaint about the async rust world isn't so much function coloring, but dependency craziness. For instance, the postgres driver for rust (which is an excellent driver, by the way), is built in on async primitives. That's a good thing, because it means that it can be used effectively from async code. Unfortunately, a "hello world" program that uses the client takes 38s to build and downloads 65 dependencies, many of which are at 0.x.y versions and I have no idea what they do (parking_lot_core v0.8.5? slab v0.4.5?). This is not just an issue of build time, but overall confidence in the quality and security of the code, and knowledge of who the authors even are. Also, it just adds complexity and magic, and sometimes I just want my code to be simple and tight and understandable from beginning to end. One idea to resolve this is if rust has a standard futures runtime API, and you can just choose one in Cargo.toml or something. By default it would be a built-in single-threaded runtime, but you could choose tokio if you want. If you want to call async code from sync code, there would be an easy way to do it (maybe more syntax?) that would use whatever runtime you chose in Cargo.toml.

jkarneges 5y ago
I typically vet dependencies by doing reverse lookups, e.g.: https://crates.io/crates/slab/reverse_dependencies https://crates.io/crates/slab/reverse_dependencies And then looking into the biggest results and understanding community sentiment. Ultimately you need to trust the authors. Do this enough and you start seeing the same popular projects, making it easier to trust things. > sometimes I just want my code to be simple and tight and understandable from beginning to end I hear ya! This is one reason I wrote my own async runtime, and the output of “cargo build” fits on my screen. :) But I don’t recommend this unless you have a LOT of time.
- lijogdfljk 5y ago
  Yea, i don't really get the distrust of the ecosystem tbh. Yea, i agree the `0.x` stuff is annoying/concerning, but the volume of dependencies feels very.. meh. I use NixOS for example. The number of things i download is _insane_ when i build my system. This isn't a Rust or JavaScript "problem", it's just how natural distribution of work occurs. People build off of the work of their peers. And if you reduce the friction of work distribution, like how NPM and Crates do, you naturally get a lot - sometimes to a comical degree. However are we going to expect crate authors to repeatedly reinvent wheels? Would we trust them if they did?
  - user-the-name 5y ago
    I would say that this is why reducing friction this way is not actually a good thing. It has disastrous effects on the entire ecosystem when including a dependency is too easy. It means people will not question whether they should be including a dependency, because it is so easy to do.
    - lijogdfljk 5y ago
      I don't think the issue is "too easy" here, but the lack of downstream verification tools. If you had a clear metric for all the parts that go into a library you wouldn't worry about how many there were. And if the library author likewise cared about their libraries metric, they wouldn't arbitrarily choose low-quality libraries. Friction isn't useful here imo. Trust is just as bad with high friction, just maybe less moving pieces. Not having verified metrics seem to me the real problem. Some sort of safety rating that bubbles up from all the dependencies. Changes to software owners would also have to cause issues in this system too.
  - nonameiguess 5y ago
    This doesn't ring true to me. I get that you need to have some dependencies. But if you go build Linux From Scratch, you get a total of 68 packages, most of which are part of the GNU project and come from common authors. That gets you an entire operating system. On the other hand, when I was writing a book using mdbook and wanted a plugin that does token replacement, it pulled in 238 dependencies. To do token replacement. Up that to the scale of an entire operating system written in Rust, and now what are you looking at? At least thousands. Supply chain auditing eventually becomes impossible if you have a separate vendor per function. Put another way, if you go and build Firefox from source, the number of Rust crates and NPM modules it pulls in automatically during its own build is more than the total number of packages in Linux From Scratch and Beyond Linux From Scratch combined. Not every library needs to be the size of glibc, but Rust and Node have gone too far in the opposite direction. I don't know what you have in your Nix build, but even Linux From Scratch is already bloated because it includes a bunch of build and test tools you don't need to just run an OS. The Arch base system is 27 packages.
- michael_j_ward 5y ago
  It would be nice to subscribe to any change in that dependency relationship. If `proxy-auditor-crate` upgrades, then probably I should to. And probably more importantly, if `proxy-auditor-crate` drops a dependency because they no longer trust it, then I definitely want to be notified.
davidkunz 5y ago
JavaScript had the advantage that is was async first (it started with callback-style APIs, but they can be converted to Promises). In Rust, this is not the case and leads to fragmentation. There are a lot of high-quality libraries which cannot be efficiently used in an async context (e.g. Diesel). And since Rust isn't shipped with a default runtime (tokio, async-std, ...), some crates are incompatible with the rest of your program. I understand the reasoning (there is no "one size fits all") but it makes things more complicated.
- weiznich 5y ago
  At least for diesel that's not true anymore. I've build a async connection implementation that will published alongside the next major release. See https://github.com/weiznich/diesel_async https://github.com/weiznich/diesel_async for details
  - davidkunz 5y ago
    That's great news, thanks for the info and contribution, weiznich!
__s 5y ago
Rust does have a standard futures runtime API (edit: thinking over, maybe you meant there should be a runtime in std) This then opens up alternative runtimes to tokio (async-std, smol, glommio, monoio) Then your suggestion of a single threaded executor exists already: https://github.com/enlightware/simple-async-local-executor https://github.com/enlightware/simple-async-local-executor Granted, in practice many libraries end up building a hard dependency on tokio
PudgePacket 5y ago
Tangential but the slab concept is actually really neat, basically a userland allocator for a specific type, or a vector that gives you the key inserted at depending on how you prefer to think. https://docs.rs/slab/ https://docs.rs/slab/.
- oconnor663 5y ago
  The parking lot concept is also really neat! The premise is that a program with lots of mutexes doesn't actually need to allocate lots of blocking queues, because the number of simultaneous waiters is bounded by the number of threads. I don't think it's a coincidence that both of the examples here are actually quite reasonable as standalone crates. They're important abstractions, but just a little bit too niche to justify a place in the standard library. That said, I agree that learning what all these crates mean is daunting, and that finding ways to make that easier somehow would be valuable.
  - roblabla 5y ago
    > just a little bit too niche to justify a place in the standard library parking_lot has actually been considered for inclusion in libstd[0], as a better alternative to the native mutex implementation (parking_lot mutex are smaller, can be const-initialized, can be moved, and tend to perform better - all while not depending on external, non-rust code). The effort is currently on hold, but it's possible that it will resume in the future. [0]: https://github.com/rust-lang/rust/pull/56410 https://github.com/rust-lang/rust/pull/56410
nicoburns 5y ago
parking_lot is a mutex. slab is a slab allocator. It seems to me that in a C or C++ project you’d have everyone writing their own rather than a shared project that everyone depends on. Personally I have much more confidence in the community de facto standard implementation of something than I do in the half-baked version that someone writes themselves to avoid pulling in a dependency.
- platinumrad 5y ago
  Personally I think it's a good thing when my hello world program is vulnerable to 65 different supply chain attacks.
- nonameiguess 5y ago
  Not sure why a C project would do that from scratch. libc includes that functionality, so you already get it for free in any POSIX system. In Windows, I'm sure the system C library provides the same stuff.
  - nicoburns 5y ago
    Perhaps not those things, but I see lots of C projects with custom hash map implementations, etc.
- gpderetta 5y ago
  I don't think parking_lot is a mutex. It is more similar to a futex, i.e. an ephemeral wait queue. You can use it to implement a mutex, but that's just one use case.
dwattttt 5y ago
There's an interesting inversion in people's expectations between compiled programs and source. If it's compiled, it should do one thing and do it well. If it's a source package, it should implement as much as it can and use external packages as little as possible.
rastignack 5y ago
All of this because the c++ devs who wrote rust do not realize that having a big and well conceived stdlib helps a ton (go is a great example).

jkarneges 5y ago

I typically vet dependencies by doing reverse lookups, e.g.: https://crates.io/crates/slab/reverse_dependencies https://crates.io/crates/slab/reverse_dependencies And then looking into the biggest results and understanding community sentiment. Ultimately you need to trust the authors. Do this enough and you start seeing the same popular projects, making it easier to trust things. > sometimes I just want my code to be simple and tight and understandable from beginning to end I hear ya! This is one reason I wrote my own async runtime, and the output of “cargo build” fits on my screen. :) But I don’t recommend this unless you have a LOT of time.

lijogdfljk 5y ago
Yea, i don't really get the distrust of the ecosystem tbh. Yea, i agree the `0.x` stuff is annoying/concerning, but the volume of dependencies feels very.. meh. I use NixOS for example. The number of things i download is _insane_ when i build my system. This isn't a Rust or JavaScript "problem", it's just how natural distribution of work occurs. People build off of the work of their peers. And if you reduce the friction of work distribution, like how NPM and Crates do, you naturally get a lot - sometimes to a comical degree. However are we going to expect crate authors to repeatedly reinvent wheels? Would we trust them if they did?
- user-the-name 5y ago
  I would say that this is why reducing friction this way is not actually a good thing. It has disastrous effects on the entire ecosystem when including a dependency is too easy. It means people will not question whether they should be including a dependency, because it is so easy to do.
  - lijogdfljk 5y ago
    I don't think the issue is "too easy" here, but the lack of downstream verification tools. If you had a clear metric for all the parts that go into a library you wouldn't worry about how many there were. And if the library author likewise cared about their libraries metric, they wouldn't arbitrarily choose low-quality libraries. Friction isn't useful here imo. Trust is just as bad with high friction, just maybe less moving pieces. Not having verified metrics seem to me the real problem. Some sort of safety rating that bubbles up from all the dependencies. Changes to software owners would also have to cause issues in this system too.
- nonameiguess 5y ago
  This doesn't ring true to me. I get that you need to have some dependencies. But if you go build Linux From Scratch, you get a total of 68 packages, most of which are part of the GNU project and come from common authors. That gets you an entire operating system. On the other hand, when I was writing a book using mdbook and wanted a plugin that does token replacement, it pulled in 238 dependencies. To do token replacement. Up that to the scale of an entire operating system written in Rust, and now what are you looking at? At least thousands. Supply chain auditing eventually becomes impossible if you have a separate vendor per function. Put another way, if you go and build Firefox from source, the number of Rust crates and NPM modules it pulls in automatically during its own build is more than the total number of packages in Linux From Scratch and Beyond Linux From Scratch combined. Not every library needs to be the size of glibc, but Rust and Node have gone too far in the opposite direction. I don't know what you have in your Nix build, but even Linux From Scratch is already bloated because it includes a bunch of build and test tools you don't need to just run an OS. The Arch base system is 27 packages.
michael_j_ward 5y ago
It would be nice to subscribe to any change in that dependency relationship. If `proxy-auditor-crate` upgrades, then probably I should to. And probably more importantly, if `proxy-auditor-crate` drops a dependency because they no longer trust it, then I definitely want to be notified.

davidkunz 5y ago

JavaScript had the advantage that is was async first (it started with callback-style APIs, but they can be converted to Promises). In Rust, this is not the case and leads to fragmentation. There are a lot of high-quality libraries which cannot be efficiently used in an async context (e.g. Diesel). And since Rust isn't shipped with a default runtime (tokio, async-std, ...), some crates are incompatible with the rest of your program. I understand the reasoning (there is no "one size fits all") but it makes things more complicated.

weiznich 5y ago
At least for diesel that's not true anymore. I've build a async connection implementation that will published alongside the next major release. See https://github.com/weiznich/diesel_async https://github.com/weiznich/diesel_async for details
- davidkunz 5y ago
  That's great news, thanks for the info and contribution, weiznich!

__s 5y ago

Rust does have a standard futures runtime API (edit: thinking over, maybe you meant there should be a runtime in std) This then opens up alternative runtimes to tokio (async-std, smol, glommio, monoio) Then your suggestion of a single threaded executor exists already: https://github.com/enlightware/simple-async-local-executor https://github.com/enlightware/simple-async-local-executor Granted, in practice many libraries end up building a hard dependency on tokio

PudgePacket 5y ago

Tangential but the slab concept is actually really neat, basically a userland allocator for a specific type, or a vector that gives you the key inserted at depending on how you prefer to think. https://docs.rs/slab/ https://docs.rs/slab/.

oconnor663 5y ago
The parking lot concept is also really neat! The premise is that a program with lots of mutexes doesn't actually need to allocate lots of blocking queues, because the number of simultaneous waiters is bounded by the number of threads. I don't think it's a coincidence that both of the examples here are actually quite reasonable as standalone crates. They're important abstractions, but just a little bit too niche to justify a place in the standard library. That said, I agree that learning what all these crates mean is daunting, and that finding ways to make that easier somehow would be valuable.
- roblabla 5y ago
  > just a little bit too niche to justify a place in the standard library parking_lot has actually been considered for inclusion in libstd[0], as a better alternative to the native mutex implementation (parking_lot mutex are smaller, can be const-initialized, can be moved, and tend to perform better - all while not depending on external, non-rust code). The effort is currently on hold, but it's possible that it will resume in the future. [0]: https://github.com/rust-lang/rust/pull/56410 https://github.com/rust-lang/rust/pull/56410

nicoburns 5y ago

parking_lot is a mutex. slab is a slab allocator. It seems to me that in a C or C++ project you’d have everyone writing their own rather than a shared project that everyone depends on. Personally I have much more confidence in the community de facto standard implementation of something than I do in the half-baked version that someone writes themselves to avoid pulling in a dependency.

platinumrad 5y ago
Personally I think it's a good thing when my hello world program is vulnerable to 65 different supply chain attacks.
nonameiguess 5y ago
Not sure why a C project would do that from scratch. libc includes that functionality, so you already get it for free in any POSIX system. In Windows, I'm sure the system C library provides the same stuff.
- nicoburns 5y ago
  Perhaps not those things, but I see lots of C projects with custom hash map implementations, etc.
gpderetta 5y ago
I don't think parking_lot is a mutex. It is more similar to a futex, i.e. an ephemeral wait queue. You can use it to implement a mutex, but that's just one use case.

dwattttt 5y ago

There's an interesting inversion in people's expectations between compiled programs and source. If it's compiled, it should do one thing and do it well. If it's a source package, it should implement as much as it can and use external packages as little as possible.

rastignack 5y ago

All of this because the c++ devs who wrote rust do not realize that having a big and well conceived stdlib helps a ton (go is a great example).