17 ms·

The Rust Platform

362 points by anp 10y ago

coolsunglasses 10y ago
Haskell Platform is the last thing you should take inspiration from. Many of us have been doing our best to kill it off. Maybe the downsides involved wouldn't affect Rust in the same ways. My suggestion, look at how Stack (the tool) and Stackage (the platform) work: https://docs.haskellstack.org/en/stable/README/ https://docs.haskellstack.org/en/stable/README/ https://github.com/commercialhaskell/stack https://github.com/commercialhaskell/stack https://www.stackage.org/ https://www.stackage.org/
- dikaiosune 10y ago
  For those of us unfamiliar with Haskell, could you expand on the downsides a bit?
  - coolsunglasses 10y ago
    This is the main thing I've written publicly on it: https://mail.haskell.org/pipermail/haskell-community/2015-September/000014.html https://mail.haskell.org/pipermail/haskell-community/2015-Se... My email is what finally moved the committee and GHC devs on including Stack with the Platform and on some other decisions concerning the website. Like I said, not all the downsides may be applicable to how Cargo works or what aturon has in mind, but please don't cite it as an exemplar of anything.
    - dikaiosune 10y ago
      That's quite the email! Without a TLDR, I'm not sure how to assess how susceptible Cargo is to those problems. My personal experience with Rust has been that there are very few instances where versioning or dependency issues cause me problems. The specific instances I have dealt with would be resolved by the proposal in the blog post (trying to use a version of serde which differs from what another core-ish library wants to use, etc.).
      - Ericson2314 10y ago
        IMO the root problem is that the Haskell Platform is both a set of curated packages and a bandaid for the fact that so many things Haskell are hard to build/install for no-good reason. On the first front I guess it's alright (but stackage is better) and on the second front it's totally inadequate—maybe even harmful in that it probably made the problem just less painful enough to spur procrastination. I think most of this lesson doesn't apply to rust, but still good to be aware. It certainly gave me a strong eversion to course-grained dependency management as a bandaid.
        coolsunglasses 10y ago
        >so many things Haskell are hard to build/install for no-good reason. Were. Were hard. It's quite fine now with Stack. I know users of all kinds of languages that all miss Stack when they're working on their non-Haskell projects.
        seagreen 10y ago
        Not sure why you're being downvoted. Haskell tooling definitely started turning around with Stack. I've used a fair amount of package managers (including cutting edge stuff like Nixpkgs) and Stack is by far my favorite. EDIT: To elaborate on why I like Stack: + Fully declarative. I don't run commands to edit my Stack environment, instead I modify the relevant stack.yml file. This means that the current environment can always be easily examined, committed to Git, etc. + Easy NixOS integration which means I can also describe the non-haskell dependencies for a project, and enforce that _absolutely_ nothing else on my system gets used. This is amazing. + I like that it uses Stackage by default, and that I can pin projects to LTS Stackage releases. This means that many users of my projects will probably already have most dependencies installed -- especially cool for small things like tutorials where users won't have the patience for long build times. + It reuses the already existing `.cabal` file format so it's easy to make libraries compatible with cabal-install, the other package manager in the Haskell ecosystem.
        saysjonathan 10y ago
- steveklabnik 10y ago
  I actually cross-posted this to /r/haskell to get explicit feedback. Someone else mentioned stack/stackage. In my understanding, Cargo already does this specific behavior. Can anyone who's more familiar with both confirm this?
  - coolsunglasses 10y ago
    I checked that other thread, I agree completely with jeremyjh's summary. The problem with Haskell Platform came down to a set of intersecting issues: - A culture of setting very strict & narrow version bounds. (Based on known-good rather than based on avoiding known-broken) - Tools (Cabal) that enforce dependency version bounds. If there's mutual/transitive incompatibiltiy in version bounds - the build fails, period. You had to figure out the problem and fix it yourself if there was a truly irreconcilable issue. - Recommended installer on the website (Platform) was unnecessarily installing packages into the global package database, making you "stuck" with those versions for _all_ packages you attempted to build. Cabal was restricted to finding builds that abide by dependency versions narrowed to the ones provided by the global package database. These problems led to beginners being confused by seemingly spurious build failures because Platform would fall out of date with the rest of the ecosystem. Cabal would be unable to find sets of compatible dependencies and say it couldn't build the package. What non-beginners were doing to avoid these problems was: 1. Install the bare compiler, no Platform 2. Use package database sandboxes for each project All of these (UX and technical) problems were solved without compromising dependency conflict enforcement via Stackage and Stack. Speaking hypothetically, if Cargo behaves like Maven or Ivy and pulls in two dependencies who want conflicting versions (1.1 and 1.2, say) of a particular library and just picks a winner, then you'll never see something like this.
    - steveklabnik 10y ago
      Cargo will do its best to unify separate versions, but if it can't, then it will just include both. So it sounds like this would be a non-issue here.
      - coolsunglasses 10y ago
- saysjonathan 10y ago
  Most of your arguments linked in a comment below are unrelated to the application of metapackages to cargo. Cargo already includes a good portion of the behaviors found in Stack and Stackage. The idealogical battle of Stack vs Haskell Platform is irrelevant to this proposal. Haskell Platform is a perfectly adequate example to take high-level inspiration for the core of this idea: use Cargo (which is like Stack/Stackage for rust) to help bootstrap Rust libraries when using rustup to install and upgrade (mostly equivalent to `stack setup`).
- Ericson2314 10y ago
  The one thing stack does do nicely that this proposal doesn't is allow curated comparable versions without forcing course dependencies. In plainer English, stack can pick the versions while you still opt in pack by package. I beleive that this is crucial to not slow down the evolution of the ecosystem. In Cargo jargon, a solutuon would be for metapackages could double as sources: `foo = { metapackage = bar }` to use foo from bar.
  - burntsushi 10y ago
    If I'm understanding your concern correctly, that's totally a part of the proposal: But we can do even better. In practice, while code will continue working with an old metapackage version, people are going to want to upgrade. We can smooth that process by allowing metapackage dependencies to be overridden if they appear explicitly in the Cargo.toml file.
    - Ericson2314 10y ago
      Nah that's different. This is about avoiding unused dependencies, especially when they constrain portability.
  - quodlibetor 10y ago
    To make this more concrete, you're suggesting something like [dependencies] hyper = { metapackage = { rust-platform = "2.7" } } I like the general theme of having the platform be just a set of known-compatible versions, but on the other hand this feels like it loses out on many of the ease-of-use advantages of just specifying a platform version and knowing that you have all the crates inside of it.
    - Ericson2314 10y ago
      Yeah that's a proper example. I don't think we need to pick one way exclusivity. My specific use-case is making PRs for packages to work in kernelspace / in unikernels. The library night initially be packaged the easy way but then I'd use this. I don't want the PR recipient to also worry they might get out of sync with the platform as a side effect.
deleted 10y ago
[deleted]
saysjonathan 10y ago
I love seeing Aaron's work within the Rust community. I had the pleasure of studying under his father and his family's gifts are clear in both their work. Are metapackages going to be available for others to utilize? If so, how will conflicts be resolved if packages require two different versions of the same package?
- brson 10y ago
  Yes, metapackages are intended to be a general cargo feature, available to all. I suspect the design has not progressed far enough to definitively answer your question about conflict resolution, but I'd imagine you have to override that dep explicitly to fix it.
qwertyuiop924 10y ago
Good work! The idea of dropping extern crate is worrying, however. Most of the ways that that would be done would add more irregularity, complexity, and implied environment to the language (Rust Platform is always there, whether or not you want it), all of which are opportunities for bugs to creep into code.
- steveklabnik 10y ago
  What worries you about dropping extern crate? A lot of people feel like it's duplicating what's in your Cargo.toml anyway.
  - dikaiosune 10y ago
    Would it still work correctly when invoking rustc on its own or from another tool? I've seen a couple of people on IRC asking about driving rustc from a non-Cargo tool.
    - steveklabnik 10y ago
      rustc already requires passing an --extern flag for each 'extern crate' in the source, so in some sense, yes. You'd be passing that stuff along. You already have to know where those deps are on disk; the difference is that you wouldn't have a list of which deps in the source code. But if you did, it would work.
      - qwertyuiop924 10y ago
        Ah. I was working under the assumption that extern crate acted as that, but file-embedded. If rust already needs --extern, I have no objections.
      - gpm 10y ago
        > rustc already requires passing an --extern flag for each 'extern crate' in the source Only if it can't find the crate otherwise. Generally it just searches your library path and any directories you specify with -L, e.g. most cargo executables can be recompiled (the first compilation via cargo used to compile the dependencies) via `rustc src/main.rs -L target/debug/deps`
Animats 10y ago
I'm a bit worried when a language develops a "platform" and an "ecosystem". This usually means you need to bring in a large amount of vaguely relevant stuff to do anything. It adds another layer of cruft, and more dependencies. Write standalone tools, but don't create a "platform". Don't make the use of the language dependent on your tools. C++ does not have a "platform". Nor does it need one.
- eridius 10y ago
  I get the feeling that you didn't really read the post. Nothing in the platform is required to use Rust, and you can trivially write Rust packages that don't use the platform. The point of the platform is for convenience. In most cases it will make sense to use it because it provides a convenient set of libraries that are known to work well together, but you could also choose to just ignore the Rust platform entirely and continue to use Rust the same way we've been using it up to now.
  - Animats 10y ago
    "In general, rustup is intended to be the primary mechanism for distribution; it’s expected that it will soon replace the guts of our official installers, becoming the primary way to acquire the Rust Platform and all that comes with it." Then, of course, the other installers will gradually break and be abandoned. The effect is that users must run the "Rust Platform", unless they have the resources to build their own distro. Is there a monetization scheme behind this? Does someone aspire to be the Canonical of the Rust ecosystem?
    - dikaiosune 10y ago
      I don't see anything in this proposal which indicates that rustup would by default install any of this, and it's already the primary target of development efforts, Rust Platform or not. What other installers are you referring to, exactly? The old rustup.sh which couldn't support multiple toolchains installed alongside each other? multirust which didn't work on Windows? rustup is a massive improvement over both, IMO.
    - 10y ago
cocktailpeanuts 10y ago
Can someone summarize what's going on here for lay people? I know this "Rust" thing is pretty popular on HN nowadays but not sure what the difference is between a language and a platform. I was trying to read the article, but I don't have enough knowledge about rust itself to understand what it's talking about...
- hueving 10y ago
  Rust is the language itself. The platform in this context is the standard libraries you use for things like string manipulation, network connections, etc.
  - dikaiosune 10y ago
    Pedantically, the standard library (strings, basic networking, containers/collections, etc.) is already well defined and quite small. The Rust Platform idea seems more aimed at providing access to curated, stabilized versions of community-developed librares for common higher-level tasks (async I/O, serialization/deserialization, etc.). Personally, I think that letting users (especially beginners) opt in to a larger set of starting libraries could be very beneficial for adoption.
    - themihai 10y ago
      Wouldn't be enough to have list of curated libraries documented somewhere in the official rust documentation? It seems to me that a "platform" support would encourage monolitic designs(i.e packages that work only within a specific "platform")
tibbe 10y ago
I wouldn't recommend following the Haskell approach. It hasn't worked well for us. (I took part in creating the Haskell Platform and the process used to add packages to it. I also used to maintain a few of our core libraries, like our containers packages and networking). Small vs large standard library: A small standard library with most functionality in independent, community-maintained packages has given us API friction as types, traits (type classes), etc are hard to coordinate across maintainers and separate package release cycles. We ended up with lots of uncomfortable conversions at API boundaries. Here's a number of examples of problems we currently have: - Conversions between our 5(!) string types are very common. - Standard library I/O modules cannot use new, de-facto standard string types (i.e. `Text` and `ByteString`) defined outside it because of dependency cycle. - Standard library cannot use containers, other than lists, for the same reason. - No standard traits for containers, like maps and sets, as those are defined outside the standard library. Result is that code is written against one concrete implementation. - Newtype wrapping to avoid orphan instances. Having traits defined in packages other than the standard library makes it harder to write non-orphan instances. - It's too difficult to make larger changes as we cannot atomically update all the packages at once. Thus such changes don't happen. Empirically, languages that have large standard libraries (e.g. Java, Python, Go) seem to do better than their competitors.
- saysjonathan 10y ago
  One of the common mantras I've heard among Rust core devs is "std is where code goes to die". Where do you feel the line should be drawn between standard lib and external libraries?
  - skybrian 10y ago
    Maybe a change in attitude? Stability can be a good thing. Go's standard library doesn't change that much, and that's a strength. How about: "std is where code goes when it's done". As is, really done. The API's won't need changing.
    - steveklabnik 10y ago
Ericson2314 10y ago
Combine this with my stdlib deps RFC[1], and I think we could shrink the nunber standard library crates versioned with the language! The standard library metapackage would pull crates from crates.io as needed. [1]: https://github.com/rust-lang/rfcs/pull/1133 https://github.com/rust-lang/rfcs/pull/1133
yoshuaw 10y ago
If a language is to play the long game, they must be conservative on what they add. Even a minimal runtime like Node is still wounded by the addition of a few broken interfaces into the core platform (even emitters, streams, domains to name a few). These cannot be stripped out of the platform because they're "blessed" and now everyone will forever have a bad time. I suggest we don't do that for Rust. For a language to remain relevant in the long term, a system must be capable of evolving. Going out there and blessing potentially conceptually incorrect packages such as Mio is therefore not a good idea. The notion of "platforms" best resides in userland, where collections can compete and evolve. By keeping platforms / package collections in userland we can get organic collections of stuff that make sense to bundle together. Imagine a "server platform", an "event loop platform", a "calculus platform", a "programming language platform". It would be detrimental to creativity to have all these collections live in the shadow of the "rust committee blessed platform". But so yeah, I'm not opposed to the idea of platforms - I just don't think blessing userland stuff from the top down is the right play in the long term. Tl;Dr: package collections sound like a cool idea, but if you care about the long term, imposing them from the top down is bad
- rcthompson 10y ago
  > These cannot be stripped out of the platform because they're "blessed" and now everyone will forever have a bad time. I suggest we don't do that for Rust. It sounds like the proposal in the OP avoids this problem by having versioned platforms that are independent of the Rust version. So if something turns out to be a bad idea, it can be stripped out of later platform versions and replaced with something better without disrupting users of the older platforms.
  - vvanders 10y ago
    Yeah, I really liked how DirectX did versioning in that respect. Lets you improve the API while preserving backwards API and signature. If you can give Microsoft something it's that they do backwards compatibility well. I don't see why that couldn't be worked into this idea.
    - 10y ago
EugeneOZ 10y ago
Good chance to see how Rust team will accept negative reaction to their idea. If they will.
- steveklabnik 10y ago
  Historically, there have been a number of proposals by the Rust team that were received badly. We've always benefited from outside eyeballs on things; usually the final proposals end up much stronger. We've only sought to _increase_ this kind of thing over time.
hsivonen 10y ago
Having used Java and having experienced how you learn to replace the JDK URL parser with something else, the JDK HTTP client with something else, the JDK encoding libraries with something else, etc., I'm worried about entrenching first-published libraries beyond their natural first-mover advantage and making it even harder for better later-comers to be adopted. OTOH, compared to e.g. C++, where even strings aren't standard across multiple real large code bases (e.g. Gecko and Qt) having standard types especially for foundational stuff can be very useful if the foundation is right or right enough. Java having a single (though not optimal) foundational notion for Unicode (UTF-16 even though UTF-8 would be optimal) and a common efficient and spec-wise correct (though not design-wise optimal) XML API (SAX) was a key reason to develop Validator.nu in Java as opposed to e.g. Python (whose notion of Unicode varied between UTF-16 and UTF-32 based on how the interpreter was compiled!). Still, at this early stage, I'd point to crates.io in general as the "batteries" instead of creating a layer of official approval on top. But I'm biased, because lately, I've been writing a second-to-market crate for a topic for which a crate already exists.
hubert123 10y ago
I mean really, why is this necessary? The reasons for adding this much bloat are tiny and irrelevant. You get a big download full of packages you dont all need, you dont know if you need and all for what? We all have Google, if I want an HTTP library for Rust I'll google whatever the best one is and make a judgement call myself.
- sametmax 10y ago
  You can still have a tiny install if you want to. But most people not knowing what to do will be able to have batteries included setup. Basically: people knowing will still have freedom, people not knowing will have an easier life. And in the end, in 2016, you generally don't care if you download a few extra 100Mo. If you do, you will just find spend the time to find to have a smaller setup.
  - hubert123 10y ago
    people who dont know what to do exactly? Downloading a package? If they can't download a package then they have an entirely different problem to solve which they have to solve anyway if they want to ever productively program Rust.
    - agmcleod 10y ago
      Knowing what packages to choose, and what works together wouldn't be a problem for them to worry about.
  - themihai 10y ago
    So if you don't know what to do the solution is to download all the possible packages?Is there a user story/use case solved this way? The documentation for each package is available online so wouldn't make sense to actually read it before to download a package?
    - sametmax 10y ago
      One of the great things with Python is the stdlib with a lot of things provided for you. And one of the most terrible things in JS is the total lack of it. Rust choose a middle ground: provide a plateform by default, yet let you choose to not use it. Win win.
AsyncAwait 10y ago
If the aim of the Rust Platform is to provide a 'blessed set' of 3rd-party libraries, why not have something akin to an official, curated 'Awesome Rust', where depending on what you want to achieve you can get a streamlined list of well-maintained libraries, perhaps with user-submitted usage-examples, comments, alternatives and the like - that way, you're not constraining Rust itself to be in sync with all the 3rd-party libs, (and thus by necessity stagnating to a certain degree), nor you're making authors of libraries that are not in the Platform feel essentially invisible.
- steveklabnik 10y ago
  How is that different than the proposal? The only part that's maybe different is "feel essentially invisible", which is something that might happen, I'll grant you that. But I don't think that people feel invisible when some package is in a standard library, which is roughly equivalent here.
  - AsyncAwait 10y ago
    I just don't think that going beyond a simple website with 'blessed' crates has much benefit, but it certainly has a lot of disadvantages as outlined in this whole thread. I'm simply saying that if the goal is to have a central place to point to when somebody asks, 'how do I do X', then perhaps making a website/improving crates.io in this regard is the better approach for the continued growth of the still very much evolving Rust ecosystem, rather than creating a Rust "distribution" as such. P.S. Thanks for all your work Steve, it's really appreciated.
    - steveklabnik 10y ago
      Thank you both for the compliment, and for elaborating. :)
moogly 10y ago
For contrast, with .NET Core, Microsoft decided to cut up .NET's quite extensive stdlib (BCL + FCL) into packages. On paper, it looks pretty good, but it remains to see how well the versioning will work in the long run.
xfactor973 10y ago
I agree with the others here. I like rust the way it is and I think it's fine to leave it alone :)
jonstewart 10y ago
The relationship between Boost and the C++ standard library might provide a good example (especially in recent years, as there's been sustained focus on expanding stdlib). Boost has a stringent peer review and comment period. Sometimes I think they let in designs that are too clever, but they rarely let in the clunkers that are seen in Python's standard library. People then gain experience with it and sometimes subtle issues emerge. Those issues can be fixed, either in Boost itself or when libraries move from Boost to stdlib (and sometimes the issues inform language features). While C++'s standard library is not "batteries included" like Python's, it's been very gratifying to see it expand slowly and surely over the last few years.
lindstorm 10y ago
Any roadmap for Rust 2.0?
- steveklabnik 10y ago
  There are no current plans for a Rust 2.0.
pc2g4d 10y ago
The TeXLive of the Rust world? Could be helpful. But the small std probably implies a constantly revolving cast of "best practice" libraries that's hard to keep up with. I know in TeXLive there are no stability guarantees regarding the collection as a whole. It's more of a collection than a platform.
pc2g4d 10y ago
Couldn't much of the function of such a "platform" be automated? One of the main constraints that would guide selection of crates for this platform metapackage is that they must have compatible dependencies. So package A depending on Bv1 and package C depending on Bv2 wouldn't work because they Bv1 and Bv2 would both have to be included, leading to a conflict. But this information is (in theory) encoded in semantic versioning. Assuming proper semantic versions for crates, a target set of crates to be included could be specified, and then automatically the various sets of crate versions that do not have conflicting dependencies could be calculated. These compatible crate/version sets could be automatically generated and published as metacrates. Consider the following crate/version dependencies: Av1 -> Bv1 Cv1 -> Bv1 Av2 -> Bv2 Cv2 -> Bv1 Av3 -> Bv2 Cv3 -> Bv2 compatible_sets({'A', 'C'}) returns {{'Av1','Cv1'},{'Av1','Cv2'},{'Av2','Cv3'},{'Av3','Cv3'}} By imposing an ordering on these compatible sets they could be automatically identified. compatible_A_C_0 is {'Av1','Cv1'}, compatible_A_C_1 is {'Av1','Cv2'}, and so on. Obviously the semver could be wrong and unexpected incompatibilities could crop up. But couldn't these just be autogenerated and then voted on? Then the top best compatible sets will filter to the top and, de facto, the Rust Platform has been autogenerated?
- steveklabnik 10y ago
  > So package A depending on Bv1 and package C depending on Bv2 wouldn't > work because they Bv1 and Bv2 would both have to be included, > leading to a conflict. As mentioned in this thread, this already works just fine. Rust can handle both versions. Furthermore, it's more than just a constraint problem; there's also integration issues, testing the whole thing together, etc.

362 points by anp 10y ago

Haskell Platform is the last thing you should take inspiration from. Many of us have been doing our best to kill it off. Maybe the downsides involved wouldn't affect Rust in the same ways. My suggestion, look at how Stack (the tool) and Stackage (the platform) work: https://docs.haskellstack.org/en/stable/README/ https://docs.haskellstack.org/en/stable/README/ https://github.com/commercialhaskell/stack https://github.com/commercialhaskell/stack https://www.stackage.org/ https://www.stackage.org/

dikaiosune 10y ago
For those of us unfamiliar with Haskell, could you expand on the downsides a bit?
- coolsunglasses 10y ago
  This is the main thing I've written publicly on it: https://mail.haskell.org/pipermail/haskell-community/2015-September/000014.html https://mail.haskell.org/pipermail/haskell-community/2015-Se... My email is what finally moved the committee and GHC devs on including Stack with the Platform and on some other decisions concerning the website. Like I said, not all the downsides may be applicable to how Cargo works or what aturon has in mind, but please don't cite it as an exemplar of anything.
  - dikaiosune 10y ago
    That's quite the email! Without a TLDR, I'm not sure how to assess how susceptible Cargo is to those problems. My personal experience with Rust has been that there are very few instances where versioning or dependency issues cause me problems. The specific instances I have dealt with would be resolved by the proposal in the blog post (trying to use a version of serde which differs from what another core-ish library wants to use, etc.).
    - Ericson2314 10y ago
      IMO the root problem is that the Haskell Platform is both a set of curated packages and a bandaid for the fact that so many things Haskell are hard to build/install for no-good reason. On the first front I guess it's alright (but stackage is better) and on the second front it's totally inadequate—maybe even harmful in that it probably made the problem just less painful enough to spur procrastination. I think most of this lesson doesn't apply to rust, but still good to be aware. It certainly gave me a strong eversion to course-grained dependency management as a bandaid.
      - coolsunglasses 10y ago
        >so many things Haskell are hard to build/install for no-good reason. Were. Were hard. It's quite fine now with Stack. I know users of all kinds of languages that all miss Stack when they're working on their non-Haskell projects.
        seagreen 10y ago
        Not sure why you're being downvoted. Haskell tooling definitely started turning around with Stack. I've used a fair amount of package managers (including cutting edge stuff like Nixpkgs) and Stack is by far my favorite. EDIT: To elaborate on why I like Stack: + Fully declarative. I don't run commands to edit my Stack environment, instead I modify the relevant stack.yml file. This means that the current environment can always be easily examined, committed to Git, etc. + Easy NixOS integration which means I can also describe the non-haskell dependencies for a project, and enforce that _absolutely_ nothing else on my system gets used. This is amazing. + I like that it uses Stackage by default, and that I can pin projects to LTS Stackage releases. This means that many users of my projects will probably already have most dependencies installed -- especially cool for small things like tutorials where users won't have the patience for long build times. + It reuses the already existing `.cabal` file format so it's easy to make libraries compatible with cabal-install, the other package manager in the Haskell ecosystem.
        saysjonathan 10y ago
steveklabnik 10y ago
I actually cross-posted this to /r/haskell to get explicit feedback. Someone else mentioned stack/stackage. In my understanding, Cargo already does this specific behavior. Can anyone who's more familiar with both confirm this?
- coolsunglasses 10y ago
  I checked that other thread, I agree completely with jeremyjh's summary. The problem with Haskell Platform came down to a set of intersecting issues: - A culture of setting very strict & narrow version bounds. (Based on known-good rather than based on avoiding known-broken) - Tools (Cabal) that enforce dependency version bounds. If there's mutual/transitive incompatibiltiy in version bounds - the build fails, period. You had to figure out the problem and fix it yourself if there was a truly irreconcilable issue. - Recommended installer on the website (Platform) was unnecessarily installing packages into the global package database, making you "stuck" with those versions for _all_ packages you attempted to build. Cabal was restricted to finding builds that abide by dependency versions narrowed to the ones provided by the global package database. These problems led to beginners being confused by seemingly spurious build failures because Platform would fall out of date with the rest of the ecosystem. Cabal would be unable to find sets of compatible dependencies and say it couldn't build the package. What non-beginners were doing to avoid these problems was: 1. Install the bare compiler, no Platform 2. Use package database sandboxes for each project All of these (UX and technical) problems were solved without compromising dependency conflict enforcement via Stackage and Stack. Speaking hypothetically, if Cargo behaves like Maven or Ivy and pulls in two dependencies who want conflicting versions (1.1 and 1.2, say) of a particular library and just picks a winner, then you'll never see something like this.
  - steveklabnik 10y ago
    Cargo will do its best to unify separate versions, but if it can't, then it will just include both. So it sounds like this would be a non-issue here.
    - coolsunglasses 10y ago
saysjonathan 10y ago
Most of your arguments linked in a comment below are unrelated to the application of metapackages to cargo. Cargo already includes a good portion of the behaviors found in Stack and Stackage. The idealogical battle of Stack vs Haskell Platform is irrelevant to this proposal. Haskell Platform is a perfectly adequate example to take high-level inspiration for the core of this idea: use Cargo (which is like Stack/Stackage for rust) to help bootstrap Rust libraries when using rustup to install and upgrade (mostly equivalent to `stack setup`).
Ericson2314 10y ago
The one thing stack does do nicely that this proposal doesn't is allow curated comparable versions without forcing course dependencies. In plainer English, stack can pick the versions while you still opt in pack by package. I beleive that this is crucial to not slow down the evolution of the ecosystem. In Cargo jargon, a solutuon would be for metapackages could double as sources: `foo = { metapackage = bar }` to use foo from bar.
- burntsushi 10y ago
  If I'm understanding your concern correctly, that's totally a part of the proposal: But we can do even better. In practice, while code will continue working with an old metapackage version, people are going to want to upgrade. We can smooth that process by allowing metapackage dependencies to be overridden if they appear explicitly in the Cargo.toml file.
  - Ericson2314 10y ago
    Nah that's different. This is about avoiding unused dependencies, especially when they constrain portability.
- quodlibetor 10y ago
  To make this more concrete, you're suggesting something like [dependencies] hyper = { metapackage = { rust-platform = "2.7" } } I like the general theme of having the platform be just a set of known-compatible versions, but on the other hand this feels like it loses out on many of the ease-of-use advantages of just specifying a platform version and knowing that you have all the crates inside of it.
  - Ericson2314 10y ago
    Yeah that's a proper example. I don't think we need to pick one way exclusivity. My specific use-case is making PRs for packages to work in kernelspace / in unikernels. The library night initially be packaged the easy way but then I'd use this. I don't want the PR recipient to also worry they might get out of sync with the platform as a side effect.

deleted 10y ago

[deleted]

saysjonathan 10y ago

I love seeing Aaron's work within the Rust community. I had the pleasure of studying under his father and his family's gifts are clear in both their work. Are metapackages going to be available for others to utilize? If so, how will conflicts be resolved if packages require two different versions of the same package?

brson 10y ago
Yes, metapackages are intended to be a general cargo feature, available to all. I suspect the design has not progressed far enough to definitively answer your question about conflict resolution, but I'd imagine you have to override that dep explicitly to fix it.

qwertyuiop924 10y ago

Good work! The idea of dropping extern crate is worrying, however. Most of the ways that that would be done would add more irregularity, complexity, and implied environment to the language (Rust Platform is always there, whether or not you want it), all of which are opportunities for bugs to creep into code.

steveklabnik 10y ago
What worries you about dropping extern crate? A lot of people feel like it's duplicating what's in your Cargo.toml anyway.
- dikaiosune 10y ago
  Would it still work correctly when invoking rustc on its own or from another tool? I've seen a couple of people on IRC asking about driving rustc from a non-Cargo tool.
  - steveklabnik 10y ago
    rustc already requires passing an --extern flag for each 'extern crate' in the source, so in some sense, yes. You'd be passing that stuff along. You already have to know where those deps are on disk; the difference is that you wouldn't have a list of which deps in the source code. But if you did, it would work.
    - qwertyuiop924 10y ago
      Ah. I was working under the assumption that extern crate acted as that, but file-embedded. If rust already needs --extern, I have no objections.
    - gpm 10y ago
      > rustc already requires passing an --extern flag for each 'extern crate' in the source Only if it can't find the crate otherwise. Generally it just searches your library path and any directories you specify with -L, e.g. most cargo executables can be recompiled (the first compilation via cargo used to compile the dependencies) via `rustc src/main.rs -L target/debug/deps`

Animats 10y ago

I'm a bit worried when a language develops a "platform" and an "ecosystem". This usually means you need to bring in a large amount of vaguely relevant stuff to do anything. It adds another layer of cruft, and more dependencies. Write standalone tools, but don't create a "platform". Don't make the use of the language dependent on your tools. C++ does not have a "platform". Nor does it need one.

eridius 10y ago
I get the feeling that you didn't really read the post. Nothing in the platform is required to use Rust, and you can trivially write Rust packages that don't use the platform. The point of the platform is for convenience. In most cases it will make sense to use it because it provides a convenient set of libraries that are known to work well together, but you could also choose to just ignore the Rust platform entirely and continue to use Rust the same way we've been using it up to now.
- Animats 10y ago
  "In general, rustup is intended to be the primary mechanism for distribution; it’s expected that it will soon replace the guts of our official installers, becoming the primary way to acquire the Rust Platform and all that comes with it." Then, of course, the other installers will gradually break and be abandoned. The effect is that users must run the "Rust Platform", unless they have the resources to build their own distro. Is there a monetization scheme behind this? Does someone aspire to be the Canonical of the Rust ecosystem?
  - dikaiosune 10y ago
    I don't see anything in this proposal which indicates that rustup would by default install any of this, and it's already the primary target of development efforts, Rust Platform or not. What other installers are you referring to, exactly? The old rustup.sh which couldn't support multiple toolchains installed alongside each other? multirust which didn't work on Windows? rustup is a massive improvement over both, IMO.
  - 10y ago

cocktailpeanuts 10y ago

Can someone summarize what's going on here for lay people? I know this "Rust" thing is pretty popular on HN nowadays but not sure what the difference is between a language and a platform. I was trying to read the article, but I don't have enough knowledge about rust itself to understand what it's talking about...

hueving 10y ago
Rust is the language itself. The platform in this context is the standard libraries you use for things like string manipulation, network connections, etc.
- dikaiosune 10y ago
  Pedantically, the standard library (strings, basic networking, containers/collections, etc.) is already well defined and quite small. The Rust Platform idea seems more aimed at providing access to curated, stabilized versions of community-developed librares for common higher-level tasks (async I/O, serialization/deserialization, etc.). Personally, I think that letting users (especially beginners) opt in to a larger set of starting libraries could be very beneficial for adoption.
  - themihai 10y ago
    Wouldn't be enough to have list of curated libraries documented somewhere in the official rust documentation? It seems to me that a "platform" support would encourage monolitic designs(i.e packages that work only within a specific "platform")

tibbe 10y ago

I wouldn't recommend following the Haskell approach. It hasn't worked well for us. (I took part in creating the Haskell Platform and the process used to add packages to it. I also used to maintain a few of our core libraries, like our containers packages and networking). Small vs large standard library: A small standard library with most functionality in independent, community-maintained packages has given us API friction as types, traits (type classes), etc are hard to coordinate across maintainers and separate package release cycles. We ended up with lots of uncomfortable conversions at API boundaries. Here's a number of examples of problems we currently have: - Conversions between our 5(!) string types are very common. - Standard library I/O modules cannot use new, de-facto standard string types (i.e. `Text` and `ByteString`) defined outside it because of dependency cycle. - Standard library cannot use containers, other than lists, for the same reason. - No standard traits for containers, like maps and sets, as those are defined outside the standard library. Result is that code is written against one concrete implementation. - Newtype wrapping to avoid orphan instances. Having traits defined in packages other than the standard library makes it harder to write non-orphan instances. - It's too difficult to make larger changes as we cannot atomically update all the packages at once. Thus such changes don't happen. Empirically, languages that have large standard libraries (e.g. Java, Python, Go) seem to do better than their competitors.

saysjonathan 10y ago
One of the common mantras I've heard among Rust core devs is "std is where code goes to die". Where do you feel the line should be drawn between standard lib and external libraries?
- skybrian 10y ago
  Maybe a change in attitude? Stability can be a good thing. Go's standard library doesn't change that much, and that's a strength. How about: "std is where code goes when it's done". As is, really done. The API's won't need changing.
  - steveklabnik 10y ago

Ericson2314 10y ago

Combine this with my stdlib deps RFC[1], and I think we could shrink the nunber standard library crates versioned with the language! The standard library metapackage would pull crates from crates.io as needed. [1]: https://github.com/rust-lang/rfcs/pull/1133 https://github.com/rust-lang/rfcs/pull/1133

yoshuaw 10y ago

If a language is to play the long game, they must be conservative on what they add. Even a minimal runtime like Node is still wounded by the addition of a few broken interfaces into the core platform (even emitters, streams, domains to name a few). These cannot be stripped out of the platform because they're "blessed" and now everyone will forever have a bad time. I suggest we don't do that for Rust. For a language to remain relevant in the long term, a system must be capable of evolving. Going out there and blessing potentially conceptually incorrect packages such as Mio is therefore not a good idea. The notion of "platforms" best resides in userland, where collections can compete and evolve. By keeping platforms / package collections in userland we can get organic collections of stuff that make sense to bundle together. Imagine a "server platform", an "event loop platform", a "calculus platform", a "programming language platform". It would be detrimental to creativity to have all these collections live in the shadow of the "rust committee blessed platform". But so yeah, I'm not opposed to the idea of platforms - I just don't think blessing userland stuff from the top down is the right play in the long term. Tl;Dr: package collections sound like a cool idea, but if you care about the long term, imposing them from the top down is bad

rcthompson 10y ago
> These cannot be stripped out of the platform because they're "blessed" and now everyone will forever have a bad time. I suggest we don't do that for Rust. It sounds like the proposal in the OP avoids this problem by having versioned platforms that are independent of the Rust version. So if something turns out to be a bad idea, it can be stripped out of later platform versions and replaced with something better without disrupting users of the older platforms.
- vvanders 10y ago
  Yeah, I really liked how DirectX did versioning in that respect. Lets you improve the API while preserving backwards API and signature. If you can give Microsoft something it's that they do backwards compatibility well. I don't see why that couldn't be worked into this idea.
  - 10y ago

EugeneOZ 10y ago

Good chance to see how Rust team will accept negative reaction to their idea. If they will.

steveklabnik 10y ago
Historically, there have been a number of proposals by the Rust team that were received badly. We've always benefited from outside eyeballs on things; usually the final proposals end up much stronger. We've only sought to _increase_ this kind of thing over time.

hsivonen 10y ago

Having used Java and having experienced how you learn to replace the JDK URL parser with something else, the JDK HTTP client with something else, the JDK encoding libraries with something else, etc., I'm worried about entrenching first-published libraries beyond their natural first-mover advantage and making it even harder for better later-comers to be adopted. OTOH, compared to e.g. C++, where even strings aren't standard across multiple real large code bases (e.g. Gecko and Qt) having standard types especially for foundational stuff can be very useful if the foundation is right or right enough. Java having a single (though not optimal) foundational notion for Unicode (UTF-16 even though UTF-8 would be optimal) and a common efficient and spec-wise correct (though not design-wise optimal) XML API (SAX) was a key reason to develop Validator.nu in Java as opposed to e.g. Python (whose notion of Unicode varied between UTF-16 and UTF-32 based on how the interpreter was compiled!). Still, at this early stage, I'd point to crates.io in general as the "batteries" instead of creating a layer of official approval on top. But I'm biased, because lately, I've been writing a second-to-market crate for a topic for which a crate already exists.

hubert123 10y ago

I mean really, why is this necessary? The reasons for adding this much bloat are tiny and irrelevant. You get a big download full of packages you dont all need, you dont know if you need and all for what? We all have Google, if I want an HTTP library for Rust I'll google whatever the best one is and make a judgement call myself.

sametmax 10y ago
You can still have a tiny install if you want to. But most people not knowing what to do will be able to have batteries included setup. Basically: people knowing will still have freedom, people not knowing will have an easier life. And in the end, in 2016, you generally don't care if you download a few extra 100Mo. If you do, you will just find spend the time to find to have a smaller setup.
- hubert123 10y ago
  people who dont know what to do exactly? Downloading a package? If they can't download a package then they have an entirely different problem to solve which they have to solve anyway if they want to ever productively program Rust.
  - agmcleod 10y ago
    Knowing what packages to choose, and what works together wouldn't be a problem for them to worry about.
- themihai 10y ago
  So if you don't know what to do the solution is to download all the possible packages?Is there a user story/use case solved this way? The documentation for each package is available online so wouldn't make sense to actually read it before to download a package?
  - sametmax 10y ago
    One of the great things with Python is the stdlib with a lot of things provided for you. And one of the most terrible things in JS is the total lack of it. Rust choose a middle ground: provide a plateform by default, yet let you choose to not use it. Win win.

AsyncAwait 10y ago

If the aim of the Rust Platform is to provide a 'blessed set' of 3rd-party libraries, why not have something akin to an official, curated 'Awesome Rust', where depending on what you want to achieve you can get a streamlined list of well-maintained libraries, perhaps with user-submitted usage-examples, comments, alternatives and the like - that way, you're not constraining Rust itself to be in sync with all the 3rd-party libs, (and thus by necessity stagnating to a certain degree), nor you're making authors of libraries that are not in the Platform feel essentially invisible.

steveklabnik 10y ago
How is that different than the proposal? The only part that's maybe different is "feel essentially invisible", which is something that might happen, I'll grant you that. But I don't think that people feel invisible when some package is in a standard library, which is roughly equivalent here.
- AsyncAwait 10y ago
  I just don't think that going beyond a simple website with 'blessed' crates has much benefit, but it certainly has a lot of disadvantages as outlined in this whole thread. I'm simply saying that if the goal is to have a central place to point to when somebody asks, 'how do I do X', then perhaps making a website/improving crates.io in this regard is the better approach for the continued growth of the still very much evolving Rust ecosystem, rather than creating a Rust "distribution" as such. P.S. Thanks for all your work Steve, it's really appreciated.
  - steveklabnik 10y ago
    Thank you both for the compliment, and for elaborating. :)

moogly 10y ago

For contrast, with .NET Core, Microsoft decided to cut up .NET's quite extensive stdlib (BCL + FCL) into packages. On paper, it looks pretty good, but it remains to see how well the versioning will work in the long run.

xfactor973 10y ago

I agree with the others here. I like rust the way it is and I think it's fine to leave it alone :)

jonstewart 10y ago

The relationship between Boost and the C++ standard library might provide a good example (especially in recent years, as there's been sustained focus on expanding stdlib). Boost has a stringent peer review and comment period. Sometimes I think they let in designs that are too clever, but they rarely let in the clunkers that are seen in Python's standard library. People then gain experience with it and sometimes subtle issues emerge. Those issues can be fixed, either in Boost itself or when libraries move from Boost to stdlib (and sometimes the issues inform language features). While C++'s standard library is not "batteries included" like Python's, it's been very gratifying to see it expand slowly and surely over the last few years.

lindstorm 10y ago

Any roadmap for Rust 2.0?

steveklabnik 10y ago
There are no current plans for a Rust 2.0.

pc2g4d 10y ago

The TeXLive of the Rust world? Could be helpful. But the small std probably implies a constantly revolving cast of "best practice" libraries that's hard to keep up with. I know in TeXLive there are no stability guarantees regarding the collection as a whole. It's more of a collection than a platform.

pc2g4d 10y ago

Couldn't much of the function of such a "platform" be automated? One of the main constraints that would guide selection of crates for this platform metapackage is that they must have compatible dependencies. So package A depending on Bv1 and package C depending on Bv2 wouldn't work because they Bv1 and Bv2 would both have to be included, leading to a conflict. But this information is (in theory) encoded in semantic versioning. Assuming proper semantic versions for crates, a target set of crates to be included could be specified, and then automatically the various sets of crate versions that do not have conflicting dependencies could be calculated. These compatible crate/version sets could be automatically generated and published as metacrates. Consider the following crate/version dependencies: Av1 -> Bv1 Cv1 -> Bv1 Av2 -> Bv2 Cv2 -> Bv1 Av3 -> Bv2 Cv3 -> Bv2 compatible_sets({'A', 'C'}) returns {{'Av1','Cv1'},{'Av1','Cv2'},{'Av2','Cv3'},{'Av3','Cv3'}} By imposing an ordering on these compatible sets they could be automatically identified. compatible_A_C_0 is {'Av1','Cv1'}, compatible_A_C_1 is {'Av1','Cv2'}, and so on. Obviously the semver could be wrong and unexpected incompatibilities could crop up. But couldn't these just be autogenerated and then voted on? Then the top best compatible sets will filter to the top and, de facto, the Rust Platform has been autogenerated?

steveklabnik 10y ago
> So package A depending on Bv1 and package C depending on Bv2 wouldn't > work because they Bv1 and Bv2 would both have to be included, > leading to a conflict. As mentioned in this thread, this already works just fine. Rust can handle both versions. Furthermore, it's more than just a constraint problem; there's also integration issues, testing the whole thing together, etc.