7 ms·

Curl removes experimental HTTP back end in Rust

168 points by hnarn 2y ago

deleted 2y ago
[deleted]
pornel 2y ago
I think there was little traction in curl, because Rust users can just use hyper directly. https://lib.rs/curl https://lib.rs/curl is used in 1000 packages, https://lib.rs/hyper https://lib.rs/hyper is in 21,000 packages. Curl is big and supports lots of protocols, but Rust doesn't need a one-stop-shop for all of them. HTTPS covers majority of uses, and Rust has separate packages for mail, ftp, websockets, etc.
- methou 2y ago
  yeah, rust is mainly for developers,and curl are for sysadmins and their derivatives.
  - project2501a 2y ago
    That is quite the arbitrary distinction. There are plenty of sysadmins that code and/or hold compsci degrees.
    - croemer 2y ago
      Not if you ask meta.stackoverflow.com
  - meltyness 2y ago
    I could see rust catching on as a shell scripting language too, honestly. Well documented, typed abstractions over shell utilities is really quite nice.
    - colejohnson66 2y ago
      I disagree. I would hate to wrangle a borrow checker just to write a simple script.
      - burjui 2y ago
        Why it's always the borrow checker that bugs the haters? You don't even see it work most of the time, because instead of using references everywhere, like most C++ers are conditioned to do, not only you can move values (doesn't necessarily mean actually performing memory operations), but also the compiler checks that you don't use the variable from which the value has been moved. And even if you do use references everywhere, they don't usually cause any problems, unless you decide to put more than one reference in a often used struct. I often use both shared (RO) and exclusive (RW) references as function arguments and rarely have to specify lifetimes manually, because automatic lifetime elision done by the compiler is sufficient in most cases.
        colejohnson66 2y ago
        Who said I’m a hater? That was a very aggressive response. All I said is that I think Rust is not a good language for a scripting system. In scripting, I’m not always writing something “correct”, but good enough. Mutability everywhere helps do so, but the borrow checker gets in the way of that.
      - meltyness 2y ago
        I've reached some mechanical sympathy with it, I think. You get some intuition for which methods to use, many structs have a pretty breadthy pallet, and then you can mostly just ignore it; quickly write your algorithm and let the compiler walk you through any oversights with the way you may have coded processing tasks. Also helps to know that the docs support search by type signature since functions operate on or return owned, borrowed, or mutably borrowed, and generally just think of borrowed as a pointer for heuristic reasons. It makes more sense than I do, usually.
  - b5n 2y ago
    _lib_curl https://curl.se/libcurl/ https://curl.se/libcurl/ https://curl.se/docs/companies.html https://curl.se/docs/companies.html
    - xign 2y ago
      I have worked in other large companies that use libcurl and they aren't even listed above. It's pretty much the de facto way to do HTTP requests in C-land unless you really want to write your own backend for some reason. The world still primarily runs on C/C++, not Rust.
- usr1106 2y ago
  Hmm, wasn't it the other way round? Curl using a Rust library (hyper) instead of Rust programs using curl? Disclaimer: Just reading TFA, not an active Rust programmer.
  - pornel 2y ago
    Yes, but curl-with-hyper needed Rust programmers to finish and maintain the integration with the C codebase, and couldn't find anyone interested enough, which I assume is because Rust users don't need curl.
    - aragilar 2y ago
      It sounds like a lack of funding (i.e. the grant ran out) was the real issue. Given how high profile curl is, this raises the question of how sustainable rewrite-in-rust efforts driven by grants (or other short-term funding) are, if they don't have an existing rust community to take advantage of the grant.
      - pornel 2y ago
        This wasn't a rewrite-in-Rust effort, and I think that's the problem. Nothing valuable from curl has been rewritten in Rust. Only some existing Rust code has been added to curl, but the Rust ecosystem already has a better, safer way of using that code. Curl is not planning to ever require Rust, so the rewrites are limited only to optional components, and can't fully guarantee that the code is safe. The Rust components are required to expose an unsafe unchecked C interface for the rest of curl. C compilers are unable to enforce the safety invariants of the interface, like the Rust compiler would in a program fully written in Rust.
        aragilar 2y ago
        Probably I misread the original announcement, but I got the impression this was a pilot to adding more rust to curl (and is exactly what a rewrite would start to look like)?
integricho 2y ago
I don't really like the idea of mixing languages in projects, be it curl or the linux kernel, doesn't matter. If an already established and proven language was already being used for such a long time successfully, that should remain the language of the project. Forks or entirely new / independent projects may be created with whatever new language is being hyped up, and that is totally fine / should be the way to go. In the end, more harm will come from forcing these new languages into long-established projects than benefits.
- PittleyDunkin 2y ago
  > If an already established and proven language was already being used for such a long time successfully, that should remain the language of the project. Any reason why? On paper C doesn't seem to offer any benefit that Rust doesn't, nor Rust any harm that C doesn't.
  - dehrmann 2y ago
    Polyglot projects are harder to reason about, harder to onboard to, have more complex tooling, and can have weird interaction bugs.
    - PittleyDunkin 2y ago
      Normally I'd agree, but rust integration with C is so tight that this is of reduced concern.
- absentmoon 2y ago
  What counts as a project in your case? Would you oppose the idea of modules within a project mixing languages? I believe curl is over 500,000 lines of code so in the case of forks a progressive rewrite, module by module seems a lot more achievable than porting everything all at once
  - integricho 2y ago
    I view even the Linux kernel as a project in this sense, as I equally don't like the idea of mixing in Rust modules into the Linux kernel which was so far (and in my opinion should still remain) a C-only codebase. I'm ok with writing an OS in Rust, but make a separate new project based on Rust, don't mix Rust into an already established project like the Linux kernel.
rwmj 2y ago
One thing I've wished for in curl is for the backends ("handlers") to be more pluggable. This would solve a packaging problem we have in Fedora where we'd prefer to distribute a core curl with backends packaged as subpackages, so that programs that only need (eg) http support would only need to depend on curl-handler-http and curl-core. At the moment the actual solution is to compile separate curl and curl-minimal packages where curl-minimal removes the lesser-used backends, and you have to choose, system-wide, which one to install. This plus a stable ABI for handlers would also solve the writing backends in other languages problem because another team could contribute a Rust / hyper handler separately. (I'm sure Daniel has solid reasons for not doing this, not to mention that it's a bunch of work which no one is volunteering to do.)
- fulafel 2y ago
  The curl-minimal solution sounds more transparent and understandable to users, on the other hand.
  - rwmj 2y ago
    The problem with curl-minimal is if you install any program that needs full-fat curl, then you're out of luck if you want to keep the minimal install. With separate curl handlers you'd also be able to find out what protocols your system is configured to access with a command like this [simulated]: $ rpm -qa | grep curl-handler- | sort curl-handler-ftp curl-handler-http curl-handler-https curl-handler-smb and similar RPM commands might be used to find out what applications need a particular protocol.
    - fulafel 2y ago
      Yep, but still as a user I'd prefer the simpler way, since it will be more clear what's happening ("ok, curl-minimal is being replaced by the full curl when I install this package"). And there will be fewer little splinter packages. I wonder if there's a place for an additional level of abstraction here, so that the user would just see something like "subcomponent X of the curl package is being installed"...
ramon156 2y ago
They removed a backend. Rustls and QUIC are still an option
Sytten 2y ago
As a rust dev I wish we had more choice of http libraries then hyper. I have a lot of respect for the author but I consider this library a hot mess of over complicated abstraction. If you tried to use hyper recently you know what I mean, just to do anything you need 4 sister crates. The internals of hyper are both hard to read and opiniated in way that makes me swear heavily every time I have to touch it.
- diggan 2y ago
  There is hundreds if not thousands of rust crates for http to choose between. `reqwest` is just one of many other popular alternatives, all built for different sorts of tradeoffs.
  - jjice 2y ago
    Most of them are built on top of hyper, including reqwest [0] (not a bad thing, they're just still hyper). I personally wouldn't even consider hyper as an HTTP client for use in most projects, but reqwest adds a much more approachable API on top of hyper. [0] https://github.com/seanmonstar/reqwest/blob/master/Cargo.toml https://github.com/seanmonstar/reqwest/blob/master/Cargo.tom...
  - worik 2y ago
    And to get a simple https request you end up with about 200 crates in your tree
    - jdiez17 2y ago
      This is an (unfortunate?) part of the Rust ecosystem. Reminds me of the JS/NPM mess. I think it is a bit better in the Rust world, but I’m still undecided if it is acceptable to have so many crate dependencies for conceptually simple programs.
    - Klonoar 2y ago
      No, there are smaller crates out there that enable a simple https request with far less crates. Much of that is due to not needing Tokio & co.
brundolf 2y ago
I'm curious for more details about which tests were hard to align and why
pdimitar 2y ago
> We had to rethink and reassess several implementation details when we aligned HTTP behavior with hyper. libcurl parses and handles HTTP stricter now. Better. That would be interesting to dig in deeper. Wish they did that in the article. Or linked an article where they do.
bflesch 2y ago
imo rust is a great language, but the leadership/project setup is very untrustworthy. The systemic security problems of cargo package distribution opens the door for state-sponsored threat actors to performing supply chain attacks. And when asked about these issues the only people who will comment using their real identities are based in China. The security issues in the rust development experience are well-documented since several years, so my summary at [1] was no news to them. But if you offer things like rustfmt, rustdoc, and a nice-to-use package distribution platform like cargo.io you should be a bit more concered with security imo. Their lax attitude to these issues and substantial corporate interests not only from US-based but also China-based companies might be good for funding but for me raises many questions. [1] https://github.com/bf/rust-security-problems https://github.com/bf/rust-security-problems
- workingjubilee 2y ago
  I'm not based in China, bro. You're the one who went around accusing random people of being advanced persistent threat actors, and now you can't even keep your story straight. Do not post blatant lies.
- selfmodruntime 2y ago
  I agree! I actually wish crates.io or even the Rust language as a whole would embrace the "sans-io" mindset. Crates which are "sans-io should simply be "no_std + alloc". That would suffice for 99% of crates. Ideally, to be fully "sans-io", you'd only be able to depend on other "sans-io" cases.
mike_hearn 2y ago
> We have shipped hyper support in curl labeled EXPERIMENTAL for several years by now, hoping to attract attention and trigger the experimental spirit in users out there. Seeing so many people seem to want more memory-safety, surely the users would come? This analysis seems off. Memory safety isn't a feature users will explicitly ask for or opt-in to. What users want is software that isn't buggy, and they just sort of ambiently expect it. A "Rust backend" is pretty meaningless to users, a bit like expecting users to pass a --dont-be-buggy flag to every invocation of the CLI tool. As a user, I'd have questions like: if this backend is better then why isn't it the default? Why do I have to make a decision at all? I just want an HTTP library/tool that doesn't let me get hacked, how it achieves that is none of my concern really. Whether that's done using Rust or not isn't something that should appear in the user interface.
- tpoacher 2y ago
  ah, the classic "--bugs" / "--no-bugs" flag
  - binary132 2y ago
    --bugs / --other-bugs
- rmgk 2y ago
  If you want to read a bit of Rust community criticism into the Article, I think it would not be too far off to read this as “There are lots of people claiming writing things in Rust provides better software due to memory safety, but in practice users did not care and were perfectly content with the C implementation” (presumably because the latter has no noticeable bugs). I also interpreted the article a bit that “users” here refers to developers that make use of libcurl, that would overall like to ensure that the application that they are writing does not segfault or similar. It seems plausible to assume that some of the Rust goodwill would translate to developers that may also try to migrate their overall code to Rust to try out that backend, and potentially contribute improvements. The Linux kernel and Git for example seem to attract a lot of attention of people that want to work on migrations towards more Rust.
- 2y ago
deleted 2y ago
[deleted]
devops99 2y ago
https://github.com/ducaale/xh https://github.com/ducaale/xh is where it's at, haven't used curl in years.

168 points by hnarn 2y ago

deleted 2y ago

[deleted]

I think there was little traction in curl, because Rust users can just use hyper directly. https://lib.rs/curl https://lib.rs/curl is used in 1000 packages, https://lib.rs/hyper https://lib.rs/hyper is in 21,000 packages. Curl is big and supports lots of protocols, but Rust doesn't need a one-stop-shop for all of them. HTTPS covers majority of uses, and Rust has separate packages for mail, ftp, websockets, etc.

methou 2y ago
yeah, rust is mainly for developers,and curl are for sysadmins and their derivatives.
- project2501a 2y ago
  That is quite the arbitrary distinction. There are plenty of sysadmins that code and/or hold compsci degrees.
  - croemer 2y ago
    Not if you ask meta.stackoverflow.com
- meltyness 2y ago
  I could see rust catching on as a shell scripting language too, honestly. Well documented, typed abstractions over shell utilities is really quite nice.
  - colejohnson66 2y ago
    I disagree. I would hate to wrangle a borrow checker just to write a simple script.
    - burjui 2y ago
      Why it's always the borrow checker that bugs the haters? You don't even see it work most of the time, because instead of using references everywhere, like most C++ers are conditioned to do, not only you can move values (doesn't necessarily mean actually performing memory operations), but also the compiler checks that you don't use the variable from which the value has been moved. And even if you do use references everywhere, they don't usually cause any problems, unless you decide to put more than one reference in a often used struct. I often use both shared (RO) and exclusive (RW) references as function arguments and rarely have to specify lifetimes manually, because automatic lifetime elision done by the compiler is sufficient in most cases.
      - colejohnson66 2y ago
        Who said I’m a hater? That was a very aggressive response. All I said is that I think Rust is not a good language for a scripting system. In scripting, I’m not always writing something “correct”, but good enough. Mutability everywhere helps do so, but the borrow checker gets in the way of that.
    - meltyness 2y ago
      I've reached some mechanical sympathy with it, I think. You get some intuition for which methods to use, many structs have a pretty breadthy pallet, and then you can mostly just ignore it; quickly write your algorithm and let the compiler walk you through any oversights with the way you may have coded processing tasks. Also helps to know that the docs support search by type signature since functions operate on or return owned, borrowed, or mutably borrowed, and generally just think of borrowed as a pointer for heuristic reasons. It makes more sense than I do, usually.
- b5n 2y ago
  _lib_curl https://curl.se/libcurl/ https://curl.se/libcurl/ https://curl.se/docs/companies.html https://curl.se/docs/companies.html
  - xign 2y ago
    I have worked in other large companies that use libcurl and they aren't even listed above. It's pretty much the de facto way to do HTTP requests in C-land unless you really want to write your own backend for some reason. The world still primarily runs on C/C++, not Rust.
usr1106 2y ago
Hmm, wasn't it the other way round? Curl using a Rust library (hyper) instead of Rust programs using curl? Disclaimer: Just reading TFA, not an active Rust programmer.
- pornel 2y ago
  Yes, but curl-with-hyper needed Rust programmers to finish and maintain the integration with the C codebase, and couldn't find anyone interested enough, which I assume is because Rust users don't need curl.
  - aragilar 2y ago
    It sounds like a lack of funding (i.e. the grant ran out) was the real issue. Given how high profile curl is, this raises the question of how sustainable rewrite-in-rust efforts driven by grants (or other short-term funding) are, if they don't have an existing rust community to take advantage of the grant.
    - pornel 2y ago
      This wasn't a rewrite-in-Rust effort, and I think that's the problem. Nothing valuable from curl has been rewritten in Rust. Only some existing Rust code has been added to curl, but the Rust ecosystem already has a better, safer way of using that code. Curl is not planning to ever require Rust, so the rewrites are limited only to optional components, and can't fully guarantee that the code is safe. The Rust components are required to expose an unsafe unchecked C interface for the rest of curl. C compilers are unable to enforce the safety invariants of the interface, like the Rust compiler would in a program fully written in Rust.
      - aragilar 2y ago
        Probably I misread the original announcement, but I got the impression this was a pilot to adding more rust to curl (and is exactly what a rewrite would start to look like)?

integricho 2y ago

I don't really like the idea of mixing languages in projects, be it curl or the linux kernel, doesn't matter. If an already established and proven language was already being used for such a long time successfully, that should remain the language of the project. Forks or entirely new / independent projects may be created with whatever new language is being hyped up, and that is totally fine / should be the way to go. In the end, more harm will come from forcing these new languages into long-established projects than benefits.

PittleyDunkin 2y ago
> If an already established and proven language was already being used for such a long time successfully, that should remain the language of the project. Any reason why? On paper C doesn't seem to offer any benefit that Rust doesn't, nor Rust any harm that C doesn't.
- dehrmann 2y ago
  Polyglot projects are harder to reason about, harder to onboard to, have more complex tooling, and can have weird interaction bugs.
  - PittleyDunkin 2y ago
    Normally I'd agree, but rust integration with C is so tight that this is of reduced concern.
absentmoon 2y ago
What counts as a project in your case? Would you oppose the idea of modules within a project mixing languages? I believe curl is over 500,000 lines of code so in the case of forks a progressive rewrite, module by module seems a lot more achievable than porting everything all at once
- integricho 2y ago
  I view even the Linux kernel as a project in this sense, as I equally don't like the idea of mixing in Rust modules into the Linux kernel which was so far (and in my opinion should still remain) a C-only codebase. I'm ok with writing an OS in Rust, but make a separate new project based on Rust, don't mix Rust into an already established project like the Linux kernel.

rwmj 2y ago

One thing I've wished for in curl is for the backends ("handlers") to be more pluggable. This would solve a packaging problem we have in Fedora where we'd prefer to distribute a core curl with backends packaged as subpackages, so that programs that only need (eg) http support would only need to depend on curl-handler-http and curl-core. At the moment the actual solution is to compile separate curl and curl-minimal packages where curl-minimal removes the lesser-used backends, and you have to choose, system-wide, which one to install. This plus a stable ABI for handlers would also solve the writing backends in other languages problem because another team could contribute a Rust / hyper handler separately. (I'm sure Daniel has solid reasons for not doing this, not to mention that it's a bunch of work which no one is volunteering to do.)

fulafel 2y ago
The curl-minimal solution sounds more transparent and understandable to users, on the other hand.
- rwmj 2y ago
  The problem with curl-minimal is if you install any program that needs full-fat curl, then you're out of luck if you want to keep the minimal install. With separate curl handlers you'd also be able to find out what protocols your system is configured to access with a command like this [simulated]: $ rpm -qa | grep curl-handler- | sort curl-handler-ftp curl-handler-http curl-handler-https curl-handler-smb and similar RPM commands might be used to find out what applications need a particular protocol.
  - fulafel 2y ago
    Yep, but still as a user I'd prefer the simpler way, since it will be more clear what's happening ("ok, curl-minimal is being replaced by the full curl when I install this package"). And there will be fewer little splinter packages. I wonder if there's a place for an additional level of abstraction here, so that the user would just see something like "subcomponent X of the curl package is being installed"...

ramon156 2y ago

They removed a backend. Rustls and QUIC are still an option

Sytten 2y ago

As a rust dev I wish we had more choice of http libraries then hyper. I have a lot of respect for the author but I consider this library a hot mess of over complicated abstraction. If you tried to use hyper recently you know what I mean, just to do anything you need 4 sister crates. The internals of hyper are both hard to read and opiniated in way that makes me swear heavily every time I have to touch it.

diggan 2y ago
There is hundreds if not thousands of rust crates for http to choose between. `reqwest` is just one of many other popular alternatives, all built for different sorts of tradeoffs.
- jjice 2y ago
  Most of them are built on top of hyper, including reqwest [0] (not a bad thing, they're just still hyper). I personally wouldn't even consider hyper as an HTTP client for use in most projects, but reqwest adds a much more approachable API on top of hyper. [0] https://github.com/seanmonstar/reqwest/blob/master/Cargo.toml https://github.com/seanmonstar/reqwest/blob/master/Cargo.tom...
- worik 2y ago
  And to get a simple https request you end up with about 200 crates in your tree
  - jdiez17 2y ago
    This is an (unfortunate?) part of the Rust ecosystem. Reminds me of the JS/NPM mess. I think it is a bit better in the Rust world, but I’m still undecided if it is acceptable to have so many crate dependencies for conceptually simple programs.
  - Klonoar 2y ago
    No, there are smaller crates out there that enable a simple https request with far less crates. Much of that is due to not needing Tokio & co.

brundolf 2y ago

I'm curious for more details about which tests were hard to align and why

pdimitar 2y ago

> We had to rethink and reassess several implementation details when we aligned HTTP behavior with hyper. libcurl parses and handles HTTP stricter now. Better. That would be interesting to dig in deeper. Wish they did that in the article. Or linked an article where they do.

bflesch 2y ago

imo rust is a great language, but the leadership/project setup is very untrustworthy. The systemic security problems of cargo package distribution opens the door for state-sponsored threat actors to performing supply chain attacks. And when asked about these issues the only people who will comment using their real identities are based in China. The security issues in the rust development experience are well-documented since several years, so my summary at [1] was no news to them. But if you offer things like rustfmt, rustdoc, and a nice-to-use package distribution platform like cargo.io you should be a bit more concered with security imo. Their lax attitude to these issues and substantial corporate interests not only from US-based but also China-based companies might be good for funding but for me raises many questions. [1] https://github.com/bf/rust-security-problems https://github.com/bf/rust-security-problems

workingjubilee 2y ago
I'm not based in China, bro. You're the one who went around accusing random people of being advanced persistent threat actors, and now you can't even keep your story straight. Do not post blatant lies.
selfmodruntime 2y ago
I agree! I actually wish crates.io or even the Rust language as a whole would embrace the "sans-io" mindset. Crates which are "sans-io should simply be "no_std + alloc". That would suffice for 99% of crates. Ideally, to be fully "sans-io", you'd only be able to depend on other "sans-io" cases.

mike_hearn 2y ago

> We have shipped hyper support in curl labeled EXPERIMENTAL for several years by now, hoping to attract attention and trigger the experimental spirit in users out there. Seeing so many people seem to want more memory-safety, surely the users would come? This analysis seems off. Memory safety isn't a feature users will explicitly ask for or opt-in to. What users want is software that isn't buggy, and they just sort of ambiently expect it. A "Rust backend" is pretty meaningless to users, a bit like expecting users to pass a --dont-be-buggy flag to every invocation of the CLI tool. As a user, I'd have questions like: if this backend is better then why isn't it the default? Why do I have to make a decision at all? I just want an HTTP library/tool that doesn't let me get hacked, how it achieves that is none of my concern really. Whether that's done using Rust or not isn't something that should appear in the user interface.

tpoacher 2y ago
ah, the classic "--bugs" / "--no-bugs" flag
- binary132 2y ago
  --bugs / --other-bugs
rmgk 2y ago
If you want to read a bit of Rust community criticism into the Article, I think it would not be too far off to read this as “There are lots of people claiming writing things in Rust provides better software due to memory safety, but in practice users did not care and were perfectly content with the C implementation” (presumably because the latter has no noticeable bugs). I also interpreted the article a bit that “users” here refers to developers that make use of libcurl, that would overall like to ensure that the application that they are writing does not segfault or similar. It seems plausible to assume that some of the Rust goodwill would translate to developers that may also try to migrate their overall code to Rust to try out that backend, and potentially contribute improvements. The Linux kernel and Git for example seem to attract a lot of attention of people that want to work on migrations towards more Rust.
2y ago

deleted 2y ago

[deleted]

devops99 2y ago

https://github.com/ducaale/xh https://github.com/ducaale/xh is where it's at, haven't used curl in years.