6 ms·

You're nitpicking one example. Putting base64 into std makes the higher level comment go from complaining about 100 dependencies to complaining about 99 depende

You're nitpicking one example. Putting base64 into std makes the higher level comment go from complaining about 100 dependencies to complaining about 99 dependencies. So maybe it's harder to work out than you think.

tptacek 5y ago
I don't have an opinion about the broader dependency problem in Rust. The comment to which I responded was hosting an internal debate about Base64 and UTF-8. I think that debate is easy to resolve: use UTF-8, put Base64 in the stdlib. Do you disagree?
- burntsushi 5y ago
  The comment you responded to was using base64 as an example. That's why your comment is a nitpick in this conversation. And yeah, I doubt I would support including base64 in std unless there were a compelling reason to do so. I can't think of one. And yes, use UTF-8. Not contesting that.
  - tptacek 5y ago
    I guess I'd just say that the aggregation of these "keep base64 out of std" decisions are why every Rust thingy I put together ends up with a zillion dependencies. There might be an upside to that (I have not detected it), but there's an obvious downside. Rust gets a bunch of stuff right that other languages missed. I don't think this one of them. I think Go's strategy of having a pretty-ok version of most of the table-stakes things developers do is probably the right one, just like I think sum types and option/result are better than error types and `nil`. I share your irritation at people who drop into the middle of discussions like these to nitpick random language war arguments, but this is a thread that is actually about Rust dependency hell, and the comment I responded to, I think, sort of diagnoses why Rust has that "problem" (if you think it's a problem --- I do, but whatever). For whatever this is worth, by the way, languages that have Base64 in std appear to include Ruby, Javascript, Python, Clojure, Java, Scala, Go, Elixir, Nim, and Swift. OCaml, C++ and Haskell do not.
    - pixel_fcker 5y ago
      What’s the “obvious” downside?
      - tptacek 5y ago
        Having to find, download, review, and build a bunch of extra dependencies (and surveil all the dependencies you do explicitly want to make sure they're not pulling in fucky code to do table-stakes things that the stdlib could just stdize.) Again: these are downsides, but not dispositions. On balance there could be good reasons to have a terse, svelte, balletic std. I'm just saying, there's clearly a downside to the decision.
    - burntsushi 5y ago
      > I guess I'd just say that the aggregation of these "keep base64 out of std" decisions are why every Rust thingy I put together ends up with a zillion dependencies. Right, I partially agree, and this is a more substantive critique. But it's a much harder position to put into a pithy HN comment that sounds obviously correct. ;-) With that said, it's pretty common in my experience to wind up in exactly the same sort of situation in Go-land. Go's standard library might hold it off for a little bit, but I seem to accrue large dependency trees there too. > There might be an upside to that (I have not detected it) Aw c'mon. The comment you were originally nit-picking even called it out. I'm sure you've heard the quip, "the standard library is where things go to die." And Python's in particular is often brought up as an example. Go's standard library hasn't suffered as badly as Python's has, but there are parts that have died. If you've used Python a lot, then the fact that you have to use 'requests' and not std's HTTP client is probably not a downside. You've already digested that ecosystem knowledge. But as stewards of the standard library, this is something we see as a downside. > For whatever this is worth, by the way, languages that have Base64 in std appear to include Ruby, Javascript, Python, Clojure, Java, Scala, Go, Elixir, Nim, and Swift. OCaml, C++ and Haskell do not. Yes. Our design philosophy around std learns from some of those languages. Python in particular has influence in two respects. First is what I said above: when you have to make an HTTP request in Python, the standard advice is "use the third party requests library" despite the standard library having an HTTP client. This should make it clear that following your advice is merely a necessary criterion; it is not sufficient. Second is that "std should remain backward compatible for approximately forever." That means we have a very high bar for including things. (I am saying "we" here because I am on the Rust team that decides these things.) API evolution is more difficult in std then outside std, because we can't just release a 2.0. So if we get the API wrong, it sucks. And eventually a superior third party crate appears and everyone has to "learn" to use the third party crate instead of std. And of course, adding things to std also means a higher maintenance burden. There have to be people willing to champion and maintain these sorts of huge additions. And many of us (now speaking as the regex crate maintainer) are actively opposed to throwing it into std precisely because it becomes more difficult to evolve. Even if we did all this and put whatever pet thing it is you want to use without pulling in dependencies (I guess that's "HTTP client" here), that doesn't really solve the general problem. Eventually, you're going to pull in a dependency for _something_. And that crate's author may have decided that depending on something else instead of rolling their own thing was the better choice. Or also the opposite: perhaps they want to split apart the internals of their crate so that others can reuse those components. I did that with the regex crate. So on the one hand, I'm contributing to the high dependency count number, but on the other, I'm solving actual problems people have by providing access to internal APIs that are separately versioned. How many other regex libraries do that? Not many. Because dependencies are too annoying in the C ecosystem, which is where most regex engines live. Getting more hand wavy... In the Go world---and I speak as someone who has used and enjoyed Go for over a decade now---performance is not a #1 concern. Not like it is in the Rust ecosystem. This has huge API implications. Culturally, people expect Rust programs to be fast and they expect to be able to achieve congruent performance as they could in C. In this environment, API complexity isn't much of a reason to not doing something. Everything gets sacrificed at the alter of performance. I know, because I've been carrying sacrificial lambs to the alter for years in the Rust ecosystem. But in Go? Nope. API simplicity is valued higher relative to things like performance. Here's the punch line: if you want to write a program that sends an HTTP request and has a dependency count of 0, then Rust isn't the right language for you. It probably never will be. Also, to be clear, I am someone that works both sides of this debate. I can often be found advocating against the use of dependencies and decreasing dependency count. Precisely because I perceive and experience costs associated with them. I've done work to reduce the dependency tree size of core ecosystem libraries. But there are some fundamental trade offs that put floors on how long you can go. And you can also seen me as an advocate against using many of my crates if a "simple" solution works for you. Take aho-corasick for example. My implementation of it is several thousand lines because I sacrificed just about every lamb possible at the alter of performance. But the <10 line naive implementation of multiple substring search might be just fine for your use case. And if it is, just write that and don't bring out the big guns.
- chrismorgan 5y ago
  Without getting into Rust’s standard library inclusion philosophy, I would note that UTF-8 and Base64 are quite different cases, and shouldn’t be combined in this way. String encoding is pervasive, and if the standard library didn’t have strings or didn’t specify the encoding, there would be much grief throughout the ecosystem, because you can’t just patch it in at the library level, it’s a language-level feature. So string encoding is a choice that has to be made. Base64, on the other hand, is just a bunch of functions, nothing special about it that means it has to be in the standard library.
  - tptacek 5y ago
    It does need to be in the standard library, because everyone needs it, almost nobody has special needs for it, and if it's not in the stdlib then you have to find, download, review, and build it and its dependencies, and that is problematic.
    - chrismorgan 5y ago
      > It does need to be in the standard library When I say that something “has to be in the standard library”, I mean that it can’t be implemented outside the standard library. That’s certainly not the case here. You’re using an outright bad definition of “need” here—subjective opinion rather than objective requirement. > because everyone needs it This is factually wildly wrong. I wrote a fair bit more here but decided it wasn’t helpful. Précis: web stuff tends to load it indirectly (though amusingly most of the time actually not use it, so that Base64 code won’t actually end up in your binary), but it’s not terribly common outside of internet stuff to reach for Base64. I’ll leave just one more remark about Base64: once things are in the standard library, breaking changes can no longer be made; the base64 crate is still experiencing breaking changes (<https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md https://github.com/marshallpierce/rust-base64/blob/master/RE...>, 0.12 and 0.13 were last year and 0.20 is not released), largely for performance reasons. Please don’t just call the thin-std approach “problematic” without acknowledging that the alternative is at least as problematic, just with a different set of caveats.
      - 5y ago

tptacek 5y ago

I don't have an opinion about the broader dependency problem in Rust. The comment to which I responded was hosting an internal debate about Base64 and UTF-8. I think that debate is easy to resolve: use UTF-8, put Base64 in the stdlib. Do you disagree?

burntsushi 5y ago
The comment you responded to was using base64 as an example. That's why your comment is a nitpick in this conversation. And yeah, I doubt I would support including base64 in std unless there were a compelling reason to do so. I can't think of one. And yes, use UTF-8. Not contesting that.
- tptacek 5y ago
  I guess I'd just say that the aggregation of these "keep base64 out of std" decisions are why every Rust thingy I put together ends up with a zillion dependencies. There might be an upside to that (I have not detected it), but there's an obvious downside. Rust gets a bunch of stuff right that other languages missed. I don't think this one of them. I think Go's strategy of having a pretty-ok version of most of the table-stakes things developers do is probably the right one, just like I think sum types and option/result are better than error types and `nil`. I share your irritation at people who drop into the middle of discussions like these to nitpick random language war arguments, but this is a thread that is actually about Rust dependency hell, and the comment I responded to, I think, sort of diagnoses why Rust has that "problem" (if you think it's a problem --- I do, but whatever). For whatever this is worth, by the way, languages that have Base64 in std appear to include Ruby, Javascript, Python, Clojure, Java, Scala, Go, Elixir, Nim, and Swift. OCaml, C++ and Haskell do not.
  - pixel_fcker 5y ago
    What’s the “obvious” downside?
    - tptacek 5y ago
      Having to find, download, review, and build a bunch of extra dependencies (and surveil all the dependencies you do explicitly want to make sure they're not pulling in fucky code to do table-stakes things that the stdlib could just stdize.) Again: these are downsides, but not dispositions. On balance there could be good reasons to have a terse, svelte, balletic std. I'm just saying, there's clearly a downside to the decision.
  - burntsushi 5y ago
    > I guess I'd just say that the aggregation of these "keep base64 out of std" decisions are why every Rust thingy I put together ends up with a zillion dependencies. Right, I partially agree, and this is a more substantive critique. But it's a much harder position to put into a pithy HN comment that sounds obviously correct. ;-) With that said, it's pretty common in my experience to wind up in exactly the same sort of situation in Go-land. Go's standard library might hold it off for a little bit, but I seem to accrue large dependency trees there too. > There might be an upside to that (I have not detected it) Aw c'mon. The comment you were originally nit-picking even called it out. I'm sure you've heard the quip, "the standard library is where things go to die." And Python's in particular is often brought up as an example. Go's standard library hasn't suffered as badly as Python's has, but there are parts that have died. If you've used Python a lot, then the fact that you have to use 'requests' and not std's HTTP client is probably not a downside. You've already digested that ecosystem knowledge. But as stewards of the standard library, this is something we see as a downside. > For whatever this is worth, by the way, languages that have Base64 in std appear to include Ruby, Javascript, Python, Clojure, Java, Scala, Go, Elixir, Nim, and Swift. OCaml, C++ and Haskell do not. Yes. Our design philosophy around std learns from some of those languages. Python in particular has influence in two respects. First is what I said above: when you have to make an HTTP request in Python, the standard advice is "use the third party requests library" despite the standard library having an HTTP client. This should make it clear that following your advice is merely a necessary criterion; it is not sufficient. Second is that "std should remain backward compatible for approximately forever." That means we have a very high bar for including things. (I am saying "we" here because I am on the Rust team that decides these things.) API evolution is more difficult in std then outside std, because we can't just release a 2.0. So if we get the API wrong, it sucks. And eventually a superior third party crate appears and everyone has to "learn" to use the third party crate instead of std. And of course, adding things to std also means a higher maintenance burden. There have to be people willing to champion and maintain these sorts of huge additions. And many of us (now speaking as the regex crate maintainer) are actively opposed to throwing it into std precisely because it becomes more difficult to evolve. Even if we did all this and put whatever pet thing it is you want to use without pulling in dependencies (I guess that's "HTTP client" here), that doesn't really solve the general problem. Eventually, you're going to pull in a dependency for _something_. And that crate's author may have decided that depending on something else instead of rolling their own thing was the better choice. Or also the opposite: perhaps they want to split apart the internals of their crate so that others can reuse those components. I did that with the regex crate. So on the one hand, I'm contributing to the high dependency count number, but on the other, I'm solving actual problems people have by providing access to internal APIs that are separately versioned. How many other regex libraries do that? Not many. Because dependencies are too annoying in the C ecosystem, which is where most regex engines live. Getting more hand wavy... In the Go world---and I speak as someone who has used and enjoyed Go for over a decade now---performance is not a #1 concern. Not like it is in the Rust ecosystem. This has huge API implications. Culturally, people expect Rust programs to be fast and they expect to be able to achieve congruent performance as they could in C. In this environment, API complexity isn't much of a reason to not doing something. Everything gets sacrificed at the alter of performance. I know, because I've been carrying sacrificial lambs to the alter for years in the Rust ecosystem. But in Go? Nope. API simplicity is valued higher relative to things like performance. Here's the punch line: if you want to write a program that sends an HTTP request and has a dependency count of 0, then Rust isn't the right language for you. It probably never will be. Also, to be clear, I am someone that works both sides of this debate. I can often be found advocating against the use of dependencies and decreasing dependency count. Precisely because I perceive and experience costs associated with them. I've done work to reduce the dependency tree size of core ecosystem libraries. But there are some fundamental trade offs that put floors on how long you can go. And you can also seen me as an advocate against using many of my crates if a "simple" solution works for you. Take aho-corasick for example. My implementation of it is several thousand lines because I sacrificed just about every lamb possible at the alter of performance. But the <10 line naive implementation of multiple substring search might be just fine for your use case. And if it is, just write that and don't bring out the big guns.
chrismorgan 5y ago
Without getting into Rust’s standard library inclusion philosophy, I would note that UTF-8 and Base64 are quite different cases, and shouldn’t be combined in this way. String encoding is pervasive, and if the standard library didn’t have strings or didn’t specify the encoding, there would be much grief throughout the ecosystem, because you can’t just patch it in at the library level, it’s a language-level feature. So string encoding is a choice that has to be made. Base64, on the other hand, is just a bunch of functions, nothing special about it that means it has to be in the standard library.
- tptacek 5y ago
  It does need to be in the standard library, because everyone needs it, almost nobody has special needs for it, and if it's not in the stdlib then you have to find, download, review, and build it and its dependencies, and that is problematic.
  - chrismorgan 5y ago
    > It does need to be in the standard library When I say that something “has to be in the standard library”, I mean that it can’t be implemented outside the standard library. That’s certainly not the case here. You’re using an outright bad definition of “need” here—subjective opinion rather than objective requirement. > because everyone needs it This is factually wildly wrong. I wrote a fair bit more here but decided it wasn’t helpful. Précis: web stuff tends to load it indirectly (though amusingly most of the time actually not use it, so that Base64 code won’t actually end up in your binary), but it’s not terribly common outside of internet stuff to reach for Base64. I’ll leave just one more remark about Base64: once things are in the standard library, breaking changes can no longer be made; the base64 crate is still experiencing breaking changes (<https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md https://github.com/marshallpierce/rust-base64/blob/master/RE...>, 0.12 and 0.13 were last year and 0.20 is not released), largely for performance reasons. Please don’t just call the thin-std approach “problematic” without acknowledging that the alternative is at least as problematic, just with a different set of caveats.
    - 5y ago