11 ms·

I tried again to use rust for something the other day. Just wanted to fetch some things via http. Nothing fancy, didn't even really care about the response bo

I tried again to use rust for something the other day. Just wanted to fetch some things via http. Nothing fancy, didn't even really care about the response body, just the timings. Consensus seemed to be that reqwest is the way to do this using rust, since the stdlib does not include an http client. Adding the dependency on reqwest tried to pull in over 100 dependencies and after building for a while, it failed somewhere along the way because my rust version was too old. Now, maybe some of these were optional dependencies, but they all got pulled in automatically. so, when TFA says > Another obvious advice is to use fewer, smaller dependencies. I just went back to go, which can do the crazy obscure task of making an http request without requiring 100 external dependencies.

ReactiveJelly 5y ago
Yeah Go just has those as internal dependencies in the stdlib. That _is_ the tradeoff - Rust builds up from nothing and makes it friction-less to add a new dependency. Go starts at a higher level and the friction for adding new stuff is high. And some of the high-level stuff, like the GC, can't be removed.
- lumost 5y ago
  Rusts ability to squeeze into small footprints is a huge part of the appeal. I don’t think that that excuses the exceptional difficulty of writing a web server or client in rust relative to other languages. How often are folks writing applications that don’t need to connect to the web these days?
  - kibwen 5y ago
    > How often are folks writing applications that don’t need to connect to the web these days? For Go, perhaps rarely. For Rust, not needing to touch the web is extremely common. Let's also keep in mind that Rust's stdlib does include IP, TCP, and UDP.
  - at_compile_time 5y ago
    I just tested this. It took me 43 second to build a new project with reqwest as a dependency. The second build took 0.55 seconds. If that's exceptional difficulty, I'm underpaid. Maybe the lesson here is to update Rust if you haven't used it in a while?
  - msbarnett 5y ago
    > How often are folks writing applications that don’t need to connect to the web these days? In Go? Rarely, I would expect. In Rust, I'd assume connecting to the web is the exception rather than the norm. Most Rust projects tend to be lower-level infrastructure & systems programming. This probably explains, and in turn is explained by, their different choices re: http in stdlib. They have much different aims and target use-case.
  - wongarsu 5y ago
    I didn't find writing a simple web server or client in Rust any harder or simpler than in e.g. Python. What is this exceptional difficulty we are talking about?
  - Ar-Curunir 5y ago
    It depends on your niche; as a person writing cryptography libraries in Rust, I have never needed to make a HTTP request.
  - masklinn 5y ago
    > How often are folks writing applications that don’t need to connect to the web these days? Every day? I think the only one I've done that in Rust it was through the program I was wrapping anyway.
- mseepgood 5y ago
  > and the friction for adding new stuff is high Not really, it's just a 'go get'.
  - yencabulator 5y ago
    Not even that really, just import a thing and the next build downloads it.
xcambar 5y ago
Funny thing is, Rust and JS are both given that comment. Seeing how they are different languages, there's not a single language feature that explains this dependency gluttony. I'd like to open up the option that it is a necessary side-effect of large ecosystems: more devs using the language, more packages published, more quality in small packages. And then, when building a large-ish package, it only makes sense to use existing packages rather than reinventing the wheel. And so, driven by nothing but good will and best practices, since you have a lot of prebuilt "assets" at your disposal ,your project ends up with 100s of deps and everybody complains.
- ReactiveJelly 5y ago
  > when building a large-ish package, it only makes sense to use existing packages rather than reinventing the wheel. Yes, for instance one of `reqwest`'s dependencies is `base64`. We can argue both that re-inventing base64 in every big library is pointless, and also that a language doesn't need base64 in its stdlib because C and C++ don't have it. There's no right way to cut this cake - Many a language has included something in the standard library, and then struggled to fix a broken interface once everything depended on it. (Like JS picking UTF-16 for strings) Many a language has also failed to include something in the stdlib and caused every application to suffer and re-invent it. (Like C++ not really saying anything about Unicode strings, which means Qt and everyone else has their own little take on it.) And the suffering is only bigger if you make dependencies artificially hard to add. Rust's approach is more flexible, at the cost of building the universe from scratch every time you make a new workspace. Personally I don't mind it. At least it's not like the problems I've heard from Haskell land. Moving from futures to async in Rust was fairly smooth.
  - tptacek 5y ago
    Rust should include Base64 in the standard library, and languages should continue to standardize on UTF-8. Doesn't seem hard to work out.
    - burntsushi 5y ago
deleted 5y ago
[deleted]
duped 5y ago
You could just use Curl bindings like you would in C or C++, which is a better comparison than Go.
serverholic 5y ago
So you gave up because your rust version was too old? Dude it's like one command to update.
- GrayShade 5y ago
  Not if you're using the Debian version. They could at least package rustup, I have no idea why they don't.
  - pornel 5y ago
    Debian's Rust is useless, sorry. Rust went for "evergreen" approach like Chrome. It focuses on making small gradual changes, quick and painless upgrades, and provides stability via backwards compatibility in the compiler. This allows everyone to only target one version: the current one. Debian's philosophy is the polar opposite, so Debian's Rust version is doomed to be perpetually obsolete.
    - j16sdiz 5y ago
      "Evergreen" compiler is enemy of reproducible build.
      - serverholic 5y ago
        I bet you don’t know what you’re talking about.
      - pornel 5y ago
        All past Rust releases are available. They can be easily installed via rustup, and multiple versions can coexist on the same machine (even run concurrently). That's even easier and more flexible than using old Debian packages.
SquishyPanda23 5y ago
> Just wanted to fetch some things via http. Nothing fancy, didn't even really care about the response body, just the timings. why not bash and curl?
- justinsaccount 5y ago
  Because I wanted to do other things with the timing data.
  - tinco 5y ago
    Why would you do it in Rust if you can do it in Go? Was there some realtime constraint in making the web request?
    - justinsaccount 5y ago
      I was prototyping something that could monitor a few api endpoints in parallel, collect timing stats using sqlite, then output a static status page, statistics and graphs. Kind of like what smokeping does, just wanted to see what a from-scratch implementation that did just what I needed might look like. Ended up being about 400 lines. Only dependencies were go-sqlite and go-chart.
sodality2 5y ago
`ureq::get("url").call().unwrap().to_string().unwrap()` with a stopwatch should get you where you want. Add ureq as a dependency, of course, but I found ureq to be better than reqwest with about half the dependencies. Or... `rustup update`...
PragmaticPulp 5y ago
Those dependencies aren't recompiled every time you change something. They will only impact your initial build or a clean build on a CI/CD system. Raw dependency counts can be shocking for those coming from other languages where pulling in a dependency is an uncommon operation, but it's not a bad thing. I'd rather have my HTTP client library re-use popular, tested packages for things like decoding JSON, handling TLS, working with IP addresses, and other common tasks. I don't think it would be better to have every package roll their own versions of all of those common pieces. You can browse the dependencies of reqwest here: https://crates.io/crates/reqwest/0.11.4/dependencies https://crates.io/crates/reqwest/0.11.4/dependencies Notably, several of the big dependencies are maintained by the same person who maintains reqwest. It's a common practice with Rust crates to split a project into reusable parts. Reqwest uses hyper (http client), http, http-body, mime, url, and several other crates that are owned by the person who owns the reqwest crate. If we instead mashed all of those dependencies together into a single crate it would appease the people who don't like seeing high dependency numbers, but it wouldn't really make the product any better. It would make it harder to re-use the individual pieces, though. > I just went back to go, which can do the crazy obscure task of making an http request without requiring 100 external dependencies. Go is famous for including HTTP utilities in the standard library. If I was building a simple app that made HTTP requests and little more, I'd probably pick Go as well. Or Python. Or you could have searched for a minimal HTTP request crate ( https://github.com/algesten/ureq https://github.com/algesten/ureq ) designed to be small instead of using the most full-featured crate. Or you could have searched for Rust libcurl bindings ( https://github.com/alexcrichton/curl-rust https://github.com/alexcrichton/curl-rust ) to use an old standby. If your goal is to reach for something quick and easy with batteries included and minimal external dependencies, Rust is not a good choice. Doesn't mean Rust is "bad", but it's not the right tool for every simple job. You also can't just pick the most popular crate and assume it's exactly what you want.
- smoldesu 5y ago
  > It's a common practice with Rust crates to split a project into reusable parts. Precisely this. Rust is not the same as Go or Java or Python: it's still a fairly low-level language. Standard practice is to build lower-level components (like http parsing and url crates) that are abstracted over by more 'friendly' crates (like reqwest). It's jarring, and certainly different from a lot of other languages, but it's a testament to the language's flexibility in my eyes.
batterylow 5y ago
Try ureq instead, like in this article: https://datacrayon.com/posts/programming/rust-notebooks/typed-arrays-from-string-arrays-for-dataset-operation/ https://datacrayon.com/posts/programming/rust-notebooks/type...
deleted 5y ago
[deleted]
mamcx 5y ago
I just wanna say why is this in Rust: Rust is a system language. For example, one shock is that you can't print! or dbg! anything (without adding Debug/Display to each type). The reasons? Where it will print in a Airpod? yeah: Rust is made to work in places where NOT exist a place to print to. Or where NOT exist a filesystem, or HTTP because not exist TCP because not exist a network card. So, in Rust all start in the deepest of the deepest bottom. --- I have a good metric to select frameworks/libraries: - If is too complex, NO except if is the only game on town - If have crazy deps setup (ejem: OpenSSL, bastard mess of dep!) NOOOOO. (except if is the only game on town) - If is too big, damm be good and solid (ej: PostgreSQL) Whatever the lang, this have served me well, and when I have been lazy (I get a deps to OpenSSL and waste 1 week fixing CI builds with it then remember this and cut that mess) the pain quickly fix it. So, next time you get a trouble like this (in Rust or whatever) ask what alternatives you can use instead!
bscphil 5y ago
I think several of the responses to your comment are misguided. If you believe them you'd think that the reason this is to be expected for Rust (and even normal! good!) is that (a) Rust is a systems language, (b) Rust deliberately has a small standard library, and (c) it doesn't matter anyway because compiles are cached. I'd say, first of all, that it does matter, because very few projects with hundreds of (transitive) dependencies are going to be able to take security sufficiently seriously. And because minimizing the number of other projects you rely on is good engineering (see: left-pad). I don't think the other two explanations are much better. Compare C as an example. With C you can use libcurl which is well-maintained, actually supports far more protocols than simply http(s), and uses far fewer dependencies to do it. It also runs on an insane number of platforms, so portability isn't the issue here either. I think it primarily comes down to cultural differences. Rust is a newer, "cooler" platform, and attracts a very different audience than C does. Because of that Rust's users will be more apt to follow common modern programming practices: using a language's built in package manager and so on. (This culture both contributes to the development of languages like Rust and is fed by them.) Remember the XKCD "Python" comic? [1] Python is what I would consider an older language in terms of culture, but it captures the idea of being able to just import a library that can already magically do everything you want. "Programming is fun again", you don't have to write boilerplate or worry about managing dependencies any more. The problem begins when everyone has that attitude, including the people writing libraries that will be used by others. Importing a library that can instantly solve your problem is just one line in your Cargo.toml away - who cares that it's got 200 dependencies! [1] https://xkcd.com/353/ https://xkcd.com/353/
- roca 5y ago
  Python's fine until you want to use a library that's not part of the standard library, then it's a nightmare. Especially if you want to distribute your code, not just run it locally. (Ditto for C and C++.)
1f60c 5y ago
In addition to what others have said, reqwest is notorious for being one of the more heavy-weight HTTP libraries, but luckily, it's far from the only option: https://www.arewewebyet.org/topics/http-clients/ https://www.arewewebyet.org/topics/http-clients/

ReactiveJelly 5y ago

Yeah Go just has those as internal dependencies in the stdlib. That _is_ the tradeoff - Rust builds up from nothing and makes it friction-less to add a new dependency. Go starts at a higher level and the friction for adding new stuff is high. And some of the high-level stuff, like the GC, can't be removed.

lumost 5y ago
Rusts ability to squeeze into small footprints is a huge part of the appeal. I don’t think that that excuses the exceptional difficulty of writing a web server or client in rust relative to other languages. How often are folks writing applications that don’t need to connect to the web these days?
- kibwen 5y ago
  > How often are folks writing applications that don’t need to connect to the web these days? For Go, perhaps rarely. For Rust, not needing to touch the web is extremely common. Let's also keep in mind that Rust's stdlib does include IP, TCP, and UDP.
- at_compile_time 5y ago
  I just tested this. It took me 43 second to build a new project with reqwest as a dependency. The second build took 0.55 seconds. If that's exceptional difficulty, I'm underpaid. Maybe the lesson here is to update Rust if you haven't used it in a while?
- msbarnett 5y ago
  > How often are folks writing applications that don’t need to connect to the web these days? In Go? Rarely, I would expect. In Rust, I'd assume connecting to the web is the exception rather than the norm. Most Rust projects tend to be lower-level infrastructure & systems programming. This probably explains, and in turn is explained by, their different choices re: http in stdlib. They have much different aims and target use-case.
- wongarsu 5y ago
  I didn't find writing a simple web server or client in Rust any harder or simpler than in e.g. Python. What is this exceptional difficulty we are talking about?
- Ar-Curunir 5y ago
  It depends on your niche; as a person writing cryptography libraries in Rust, I have never needed to make a HTTP request.
- masklinn 5y ago
  > How often are folks writing applications that don’t need to connect to the web these days? Every day? I think the only one I've done that in Rust it was through the program I was wrapping anyway.
mseepgood 5y ago
> and the friction for adding new stuff is high Not really, it's just a 'go get'.
- yencabulator 5y ago
  Not even that really, just import a thing and the next build downloads it.

xcambar 5y ago

Funny thing is, Rust and JS are both given that comment. Seeing how they are different languages, there's not a single language feature that explains this dependency gluttony. I'd like to open up the option that it is a necessary side-effect of large ecosystems: more devs using the language, more packages published, more quality in small packages. And then, when building a large-ish package, it only makes sense to use existing packages rather than reinventing the wheel. And so, driven by nothing but good will and best practices, since you have a lot of prebuilt "assets" at your disposal ,your project ends up with 100s of deps and everybody complains.

ReactiveJelly 5y ago
> when building a large-ish package, it only makes sense to use existing packages rather than reinventing the wheel. Yes, for instance one of `reqwest`'s dependencies is `base64`. We can argue both that re-inventing base64 in every big library is pointless, and also that a language doesn't need base64 in its stdlib because C and C++ don't have it. There's no right way to cut this cake - Many a language has included something in the standard library, and then struggled to fix a broken interface once everything depended on it. (Like JS picking UTF-16 for strings) Many a language has also failed to include something in the stdlib and caused every application to suffer and re-invent it. (Like C++ not really saying anything about Unicode strings, which means Qt and everyone else has their own little take on it.) And the suffering is only bigger if you make dependencies artificially hard to add. Rust's approach is more flexible, at the cost of building the universe from scratch every time you make a new workspace. Personally I don't mind it. At least it's not like the problems I've heard from Haskell land. Moving from futures to async in Rust was fairly smooth.
- tptacek 5y ago
  Rust should include Base64 in the standard library, and languages should continue to standardize on UTF-8. Doesn't seem hard to work out.
  - burntsushi 5y ago

deleted 5y ago

[deleted]

duped 5y ago

You could just use Curl bindings like you would in C or C++, which is a better comparison than Go.

serverholic 5y ago

So you gave up because your rust version was too old? Dude it's like one command to update.

GrayShade 5y ago
Not if you're using the Debian version. They could at least package rustup, I have no idea why they don't.
- pornel 5y ago
  Debian's Rust is useless, sorry. Rust went for "evergreen" approach like Chrome. It focuses on making small gradual changes, quick and painless upgrades, and provides stability via backwards compatibility in the compiler. This allows everyone to only target one version: the current one. Debian's philosophy is the polar opposite, so Debian's Rust version is doomed to be perpetually obsolete.
  - j16sdiz 5y ago
    "Evergreen" compiler is enemy of reproducible build.
    - serverholic 5y ago
      I bet you don’t know what you’re talking about.
    - pornel 5y ago
      All past Rust releases are available. They can be easily installed via rustup, and multiple versions can coexist on the same machine (even run concurrently). That's even easier and more flexible than using old Debian packages.

SquishyPanda23 5y ago

> Just wanted to fetch some things via http. Nothing fancy, didn't even really care about the response body, just the timings. why not bash and curl?

justinsaccount 5y ago
Because I wanted to do other things with the timing data.
- tinco 5y ago
  Why would you do it in Rust if you can do it in Go? Was there some realtime constraint in making the web request?
  - justinsaccount 5y ago
    I was prototyping something that could monitor a few api endpoints in parallel, collect timing stats using sqlite, then output a static status page, statistics and graphs. Kind of like what smokeping does, just wanted to see what a from-scratch implementation that did just what I needed might look like. Ended up being about 400 lines. Only dependencies were go-sqlite and go-chart.

sodality2 5y ago

`ureq::get("url").call().unwrap().to_string().unwrap()` with a stopwatch should get you where you want. Add ureq as a dependency, of course, but I found ureq to be better than reqwest with about half the dependencies. Or... `rustup update`...

PragmaticPulp 5y ago

Those dependencies aren't recompiled every time you change something. They will only impact your initial build or a clean build on a CI/CD system. Raw dependency counts can be shocking for those coming from other languages where pulling in a dependency is an uncommon operation, but it's not a bad thing. I'd rather have my HTTP client library re-use popular, tested packages for things like decoding JSON, handling TLS, working with IP addresses, and other common tasks. I don't think it would be better to have every package roll their own versions of all of those common pieces. You can browse the dependencies of reqwest here: https://crates.io/crates/reqwest/0.11.4/dependencies https://crates.io/crates/reqwest/0.11.4/dependencies Notably, several of the big dependencies are maintained by the same person who maintains reqwest. It's a common practice with Rust crates to split a project into reusable parts. Reqwest uses hyper (http client), http, http-body, mime, url, and several other crates that are owned by the person who owns the reqwest crate. If we instead mashed all of those dependencies together into a single crate it would appease the people who don't like seeing high dependency numbers, but it wouldn't really make the product any better. It would make it harder to re-use the individual pieces, though. > I just went back to go, which can do the crazy obscure task of making an http request without requiring 100 external dependencies. Go is famous for including HTTP utilities in the standard library. If I was building a simple app that made HTTP requests and little more, I'd probably pick Go as well. Or Python. Or you could have searched for a minimal HTTP request crate ( https://github.com/algesten/ureq https://github.com/algesten/ureq ) designed to be small instead of using the most full-featured crate. Or you could have searched for Rust libcurl bindings ( https://github.com/alexcrichton/curl-rust https://github.com/alexcrichton/curl-rust ) to use an old standby. If your goal is to reach for something quick and easy with batteries included and minimal external dependencies, Rust is not a good choice. Doesn't mean Rust is "bad", but it's not the right tool for every simple job. You also can't just pick the most popular crate and assume it's exactly what you want.

smoldesu 5y ago
> It's a common practice with Rust crates to split a project into reusable parts. Precisely this. Rust is not the same as Go or Java or Python: it's still a fairly low-level language. Standard practice is to build lower-level components (like http parsing and url crates) that are abstracted over by more 'friendly' crates (like reqwest). It's jarring, and certainly different from a lot of other languages, but it's a testament to the language's flexibility in my eyes.

batterylow 5y ago

Try ureq instead, like in this article: https://datacrayon.com/posts/programming/rust-notebooks/typed-arrays-from-string-arrays-for-dataset-operation/ https://datacrayon.com/posts/programming/rust-notebooks/type...

deleted 5y ago

[deleted]

mamcx 5y ago

I just wanna say why is this in Rust: Rust is a system language. For example, one shock is that you can't print! or dbg! anything (without adding Debug/Display to each type). The reasons? Where it will print in a Airpod? yeah: Rust is made to work in places where NOT exist a place to print to. Or where NOT exist a filesystem, or HTTP because not exist TCP because not exist a network card. So, in Rust all start in the deepest of the deepest bottom. --- I have a good metric to select frameworks/libraries: - If is too complex, NO except if is the only game on town - If have crazy deps setup (ejem: OpenSSL, bastard mess of dep!) NOOOOO. (except if is the only game on town) - If is too big, damm be good and solid (ej: PostgreSQL) Whatever the lang, this have served me well, and when I have been lazy (I get a deps to OpenSSL and waste 1 week fixing CI builds with it then remember this and cut that mess) the pain quickly fix it. So, next time you get a trouble like this (in Rust or whatever) ask what alternatives you can use instead!

bscphil 5y ago

I think several of the responses to your comment are misguided. If you believe them you'd think that the reason this is to be expected for Rust (and even normal! good!) is that (a) Rust is a systems language, (b) Rust deliberately has a small standard library, and (c) it doesn't matter anyway because compiles are cached. I'd say, first of all, that it does matter, because very few projects with hundreds of (transitive) dependencies are going to be able to take security sufficiently seriously. And because minimizing the number of other projects you rely on is good engineering (see: left-pad). I don't think the other two explanations are much better. Compare C as an example. With C you can use libcurl which is well-maintained, actually supports far more protocols than simply http(s), and uses far fewer dependencies to do it. It also runs on an insane number of platforms, so portability isn't the issue here either. I think it primarily comes down to cultural differences. Rust is a newer, "cooler" platform, and attracts a very different audience than C does. Because of that Rust's users will be more apt to follow common modern programming practices: using a language's built in package manager and so on. (This culture both contributes to the development of languages like Rust and is fed by them.) Remember the XKCD "Python" comic? [1] Python is what I would consider an older language in terms of culture, but it captures the idea of being able to just import a library that can already magically do everything you want. "Programming is fun again", you don't have to write boilerplate or worry about managing dependencies any more. The problem begins when everyone has that attitude, including the people writing libraries that will be used by others. Importing a library that can instantly solve your problem is just one line in your Cargo.toml away - who cares that it's got 200 dependencies! [1] https://xkcd.com/353/ https://xkcd.com/353/

roca 5y ago
Python's fine until you want to use a library that's not part of the standard library, then it's a nightmare. Especially if you want to distribute your code, not just run it locally. (Ditto for C and C++.)

1f60c 5y ago

In addition to what others have said, reqwest is notorious for being one of the more heavy-weight HTTP libraries, but luckily, it's far from the only option: https://www.arewewebyet.org/topics/http-clients/ https://www.arewewebyet.org/topics/http-clients/