4 ms·

In general, there are many reasons (fragmentation, quality, portability, dependency management, compiler support) it's preferable to have concurrency tools stan

In general, there are many reasons (fragmentation, quality, portability, dependency management, compiler support) it's preferable to have concurrency tools standardized. Standardization is only going to get harder in the future. Specifically, it's great that Rust has a capable atomics library. I'm sure we'll see many good concurrent data structures come from it. But that's not quite what I meant by concurrency tools. The thread primitives Rust offers are the same ones POSIX standardized twenty years ago. It would be very valuable to have something like OpenMP (parallel loops, etc.) and something for concurrent IO (either asynchronous or coroutine-based). There's no reason Rust can't have the performance of C++ and the easy concurrency tools of Go or Erlang.

Manishearth 11y ago
>It would be very valuable to have something like OpenMP (parallel loops, etc.) and something for concurrent IO (either asynchronous or coroutine-based). https://crates.io/crates/mio https://crates.io/crates/mio for the async IO. There was another library out there that provided tons of concurrency utilities, but I can't find it now. Parallel loop-like syntax should be easy with scoped_threadpool and a macro though. Rust tries to avoid stuffing everything into the standard library. So the lack of utilities in the stdlib isn't an artifact of some ignorance of concurrency, it's because Rust doesn't want to keep everything in the stdlib. Rust keeps the basic framework for thread safety in the stdlib (though it doesn't need to, not exactly), but the rest is built upon by the community.
- acconsta 11y ago
  >Rust tries to avoid stuffing everything into the standard library That's a valid philosophy, but also one that leads to problems with fragmentation, quality, portability, dependency management, and compiler support. Concurrent IO is surely important enough to standardize.
  - burntsushi 11y ago
    > That's a valid philosophy, but also one that leads to problems with fragmentation, quality, portability, dependency management, and compiler support. No, not necessarily. We would like for the standard library to remain minimal, but one of its important functions is to collect common interfaces to maximize interoperability between crates. This has worked well in practice so far. Similarly, the standard library provides portable facades on top of platform specific APIs, for example, for performing IO. Crates can take advantage of this so that they can be portable themselves. Moreover, crates themselves can also provide portable facades over platform specific APIs, so I'm not convinced that this will be a problem in practice. Dependency management is handled quite well by Cargo. It has been a wonderful tool to have at our disposal and is really the crux of what makes a small standard library possible. Compiler support is a good argument, but one that I hope becomes weaker in time as we stabilize more functionality. To be clear, I agree that a small standard library has its own downsides. In particular, quality is IMO on of the best arguments against a small standard library. Fortunately, we trying to mitigate this by adopting officially blessed libraries into the `rust-lang` organization: https://github.com/rust-lang/rfcs/blob/master/text/1242-rust-lang-crates.md https://github.com/rust-lang/rfcs/blob/master/text/1242-rust... --- This allows us to avoid the problems with a big standard library (too much stuff that is hard to evolve because of stability) while still providing quality with crates that we promise to maintain.
    - 15155 11y ago
      > so I'm not convinced that this will be a problem in practice. Historically, it's been a massive problem. See my other post. Rust is already seeing IO-related fragmentation. C, C++, Ruby, Python, etc. are all massively-fragmented ecosystems regarding IO, concurrency, (safe) parallelism. I don't think we need to soil a great language (Rust) with these same mistakes.
      - burntsushi 11y ago
        I was specifically speaking about the size of Rust's standard library. Python, at least, has a massive standard library, so I'm not sure how that's applicable here. (Or, at the very least, demonstrates that a big standard library isn't sufficient. What matters is what is in the standard library, which is not fundamentally incompatible with a small standard library.) Also, none of those languages started out with a tool like Cargo. I made a few other comments about mitigating this as well that should be considered.
- 15155 11y ago
  What happens when one has 10 different ways of handling async IO? Massive ecosystem fragmentation. Ruby has EventMachine, the standard lib, and Celluloid. None of these are interoperable. Python has the standard lib, Twisted, and a number of other projects. All have the same problems. C++ has Boost, Asio, a few others. This is more along the lines of where Rust is headed. C has /countless/ options. None of them are standard, and it's totally understood in such an old language. --- Go has goroutines: it's a harmonious, unified ecosystem. I greatly dislike go, but this is one thing they do absolutely correctly. Erlang has fully-preemptive multitasking (which the entire ecosystem is built upon). Haskell has incredible parallelization and concurrency primitives built right into the stdlib. Certain things belong in the stdlib. Async IO is definitely one of them. If not an implementation, a defined, common interface of some nature, to help prevent some of the fragmentation. async/await, if I am not mistaken, will require compiler and borrow-checker support. This would be a good start.
steveklabnik 11y ago
We do have libraries for asynchronous io, and co routines, they're just not ready yet. I absolutely agree with you that these things are useful, but the code needs to be written by someone. We're just at the stages where these things are getting good enough to try out. (And the language can't have exactly the same stuff as Go or Erlang without adding a runtime, which is contrary to the goals of the project.)
- acconsta 11y ago
  That's great. I look forward to seeing them evolve and hopefully become standardized. >(And the language can't have exactly the same stuff as Go or Erlang without adding a runtime, which is contrary to the goals of the project.) Here is 90% of what Go gives you as a C library. No heavyweight runtime necessary: http://libmill.org/ http://libmill.org/
  - burntsushi 11y ago
    > Here is 90% of what Go gives you as a C library. No heavyweight runtime necessary Rust has channels in its standard library. Admittedly, they do not have the same functionality as Go's concurrency primitives. The two most significant omissions are probably a multi-producer/multi-consumer channel and a more complete (and stable) `select` construct. There is also my `chan` library, which replicates Go functionality with respect to channels: http://burntsushi.net/rustdoc/chan/ http://burntsushi.net/rustdoc/chan/ --- Notably, you still have to use native threads. Your `libmill` example is a coroutine library with a scheduler and everything. That is definitely too much runtime for standard Rust. However, there are people working on coroutines in Rust, on which something like `libmill` could be built: https://crates.io/search?q=coroutine https://crates.io/search?q=coroutine
    - acconsta 11y ago
      >That is definitely too much runtime Is it incompatible with native threads? (no) Does it affect non-coroutine functions? (no) Does it have any effect whatsoever when not using the library? (no) I encourage you to look over the code before dismissing it as "definitely too much runtime."

Manishearth 11y ago

>It would be very valuable to have something like OpenMP (parallel loops, etc.) and something for concurrent IO (either asynchronous or coroutine-based). https://crates.io/crates/mio https://crates.io/crates/mio for the async IO. There was another library out there that provided tons of concurrency utilities, but I can't find it now. Parallel loop-like syntax should be easy with scoped_threadpool and a macro though. Rust tries to avoid stuffing everything into the standard library. So the lack of utilities in the stdlib isn't an artifact of some ignorance of concurrency, it's because Rust doesn't want to keep everything in the stdlib. Rust keeps the basic framework for thread safety in the stdlib (though it doesn't need to, not exactly), but the rest is built upon by the community.

acconsta 11y ago
>Rust tries to avoid stuffing everything into the standard library That's a valid philosophy, but also one that leads to problems with fragmentation, quality, portability, dependency management, and compiler support. Concurrent IO is surely important enough to standardize.
- burntsushi 11y ago
  > That's a valid philosophy, but also one that leads to problems with fragmentation, quality, portability, dependency management, and compiler support. No, not necessarily. We would like for the standard library to remain minimal, but one of its important functions is to collect common interfaces to maximize interoperability between crates. This has worked well in practice so far. Similarly, the standard library provides portable facades on top of platform specific APIs, for example, for performing IO. Crates can take advantage of this so that they can be portable themselves. Moreover, crates themselves can also provide portable facades over platform specific APIs, so I'm not convinced that this will be a problem in practice. Dependency management is handled quite well by Cargo. It has been a wonderful tool to have at our disposal and is really the crux of what makes a small standard library possible. Compiler support is a good argument, but one that I hope becomes weaker in time as we stabilize more functionality. To be clear, I agree that a small standard library has its own downsides. In particular, quality is IMO on of the best arguments against a small standard library. Fortunately, we trying to mitigate this by adopting officially blessed libraries into the `rust-lang` organization: https://github.com/rust-lang/rfcs/blob/master/text/1242-rust-lang-crates.md https://github.com/rust-lang/rfcs/blob/master/text/1242-rust... --- This allows us to avoid the problems with a big standard library (too much stuff that is hard to evolve because of stability) while still providing quality with crates that we promise to maintain.
  - 15155 11y ago
    > so I'm not convinced that this will be a problem in practice. Historically, it's been a massive problem. See my other post. Rust is already seeing IO-related fragmentation. C, C++, Ruby, Python, etc. are all massively-fragmented ecosystems regarding IO, concurrency, (safe) parallelism. I don't think we need to soil a great language (Rust) with these same mistakes.
    - burntsushi 11y ago
      I was specifically speaking about the size of Rust's standard library. Python, at least, has a massive standard library, so I'm not sure how that's applicable here. (Or, at the very least, demonstrates that a big standard library isn't sufficient. What matters is what is in the standard library, which is not fundamentally incompatible with a small standard library.) Also, none of those languages started out with a tool like Cargo. I made a few other comments about mitigating this as well that should be considered.
15155 11y ago
What happens when one has 10 different ways of handling async IO? Massive ecosystem fragmentation. Ruby has EventMachine, the standard lib, and Celluloid. None of these are interoperable. Python has the standard lib, Twisted, and a number of other projects. All have the same problems. C++ has Boost, Asio, a few others. This is more along the lines of where Rust is headed. C has /countless/ options. None of them are standard, and it's totally understood in such an old language. --- Go has goroutines: it's a harmonious, unified ecosystem. I greatly dislike go, but this is one thing they do absolutely correctly. Erlang has fully-preemptive multitasking (which the entire ecosystem is built upon). Haskell has incredible parallelization and concurrency primitives built right into the stdlib. Certain things belong in the stdlib. Async IO is definitely one of them. If not an implementation, a defined, common interface of some nature, to help prevent some of the fragmentation. async/await, if I am not mistaken, will require compiler and borrow-checker support. This would be a good start.

steveklabnik 11y ago

We do have libraries for asynchronous io, and co routines, they're just not ready yet. I absolutely agree with you that these things are useful, but the code needs to be written by someone. We're just at the stages where these things are getting good enough to try out. (And the language can't have exactly the same stuff as Go or Erlang without adding a runtime, which is contrary to the goals of the project.)

acconsta 11y ago
That's great. I look forward to seeing them evolve and hopefully become standardized. >(And the language can't have exactly the same stuff as Go or Erlang without adding a runtime, which is contrary to the goals of the project.) Here is 90% of what Go gives you as a C library. No heavyweight runtime necessary: http://libmill.org/ http://libmill.org/
- burntsushi 11y ago
  > Here is 90% of what Go gives you as a C library. No heavyweight runtime necessary Rust has channels in its standard library. Admittedly, they do not have the same functionality as Go's concurrency primitives. The two most significant omissions are probably a multi-producer/multi-consumer channel and a more complete (and stable) `select` construct. There is also my `chan` library, which replicates Go functionality with respect to channels: http://burntsushi.net/rustdoc/chan/ http://burntsushi.net/rustdoc/chan/ --- Notably, you still have to use native threads. Your `libmill` example is a coroutine library with a scheduler and everything. That is definitely too much runtime for standard Rust. However, there are people working on coroutines in Rust, on which something like `libmill` could be built: https://crates.io/search?q=coroutine https://crates.io/search?q=coroutine
  - acconsta 11y ago
    >That is definitely too much runtime Is it incompatible with native threads? (no) Does it affect non-coroutine functions? (no) Does it have any effect whatsoever when not using the library? (no) I encourage you to look over the code before dismissing it as "definitely too much runtime."