7 ms·

Replacing parts of curl with Rust will not be possible. Most places want to build from source and don’t want to introduce a new compiler to the tool chain. And

Replacing parts of curl with Rust will not be possible. Most places want to build from source and don’t want to introduce a new compiler to the tool chain. And Rust supports a tiny subset of possible C targets.

galangalalgol 3y ago
Not wanting to introduce a new compiler is like wanting to introduce a new language. If you don't like the choice the package maintainer makes, roll up your sleeves. That said, we can mitigiate it using mrustc to generate c. Does anyone keep a list of all the targets libcurl is getting built for?
agwa 3y ago
Should the majority of us continue to bear the risks of memory unsafe code because of a small minority that uses exotic architectures or doesn't want to install a new compiler?
- rhdunn 3y ago
  You need to have someone, or a group of people, willing to write the code to implement a replacement that is fully compatible, and maintain it. Until then, curl written in C is what we have. With that, curl can be used on the exotic architectures and the rust/other language version can be used on other platforms.
  - moltonel3x 3y ago
    You need to have someone, or a group of people, willing to write the code to implement support for modern tooling on their platform. Until then, old harder-to-secure codebases is what these platforms have. I know reversing the burden of implementation seems flippant, but it's pragmatic. At some stage, it's less community-wide work to support Rust on a new platform than to spend extra time maintaining/securing dozens of C codebases. Curl may not be making its Rust components mandatory anytime soon, but other projects like python crypto already have, to say nothing of projects written in Rust to begin with. rustc_codegen_gcc is pretty close to ready, let's focus on getting it out of the door, and more target triples supported by rustc and llvm.
dezgeg 3y ago
"Most" is a very heavy claim. Any concrete evidence of this? Also in embedded Linux I guess we are technically "building from source", when using a build system like Buildroot or Yocto. But Rust is already supported there and in fact already in use in our project (due to python3-cryptography).
pmontra 3y ago
What are those "most places" that "want to build from source"? It's a honest question and I intentionally left out the "don’t want to introduce a new compiler to the tool chain" statement to make the set larger. I understand "most" as larger than 50% on some metric. I know about Gentoo and other Linux distributions that do build from source but it has been very uncommon for me to see a company building things from source instead of using apt, rpm, docker in this century. It's at least two orders of magnitude faster. I remember how in the 90s I had to download the source, scan the README for the dependencies, recursively so, then building the libraries, then eventually building the program I downloaded first. Security wise, we were not reading the code, except for Nethack.
- jdblair 3y ago
  The places that want to build from source are the creators of embedded devices that use curl. These devices include (but are not limited to) wifi access points, routers, home automation devices, data loggers, IP video cameras, smart TVs, set top boxes, and streaming devices. There are a lot of these devices, and there are more all the time. Even if the engineering teams do not build every dependency every time they build firmware (like busybox), they will at some point build all of the components of their system themselves (as supported by yocto).
  - dezgeg 3y ago
    Anybody using the standard embedded build solutions like Buildroot or Yocto will get Rust support without doing anything. It's already there and implemented.
  - pmontra 3y ago
    That makes sense. It was not clear in the context of the post I replied to.
    - jdblair 3y ago
      Yeah, I admit that maybe this isn't "most places," since there are almost certainly more shops running various versions of Linux servers in production than there are embedded Linux development teams. (edited to fix typo)
pizza234 3y ago
The maintainer wrote a post about this topic: https://daniel.haxx.se/blog/2020/10/09/rust-in-curl-with-hyper https://daniel.haxx.se/blog/2020/10/09/rust-in-curl-with-hyp.... "Most places want to build from source" is not something he considered; there is a brief "for users against the C implementation on the platforms you care about".
steveklabnik 3y ago
Just to be clear, since this hypothetical library exposes a C ABI, pre-built binaries wouldn’t be an issue if somehow magically curl was reimplemented in Rust. Someone would have to compile it, but that’s the same as if they didn’t want to build something written in C from source.
moltonel3x 3y ago
> Replacing parts of curl with Rust will not be possible. It's not just possible, it's been done. You can compile curl with rustls, you could for a time compile it with quiche, and work is ongoing to compile it with hyper. Curl is remarkably modular, none of those are mandatory. > And Rust supports a tiny subset of possible C targets. Gross overstatement. Rust supports the vast majority of devices that people buy today. Even if you ignore platform popularity and just count platforms supported by gcc vs rustc/llvm, there's only a handful missing from the later. And if you're talking about vendor-specific compilers, a lot of them don't support modern C or C++ either.
- axus 3y ago
  I know almost nothing about the Rust ecosystem but am interested. What Rust packages could completely replace Curl? The top result on Google suggested hyper and reqwest.
  - moltonel3x 3y ago
    You'd need a few different ones, just like curl itself uses a lot of 3rd-party libraries to provide its full feature set. Some likely first choices would be hyper (http client/server), rustls (encryption), tokio (async scheduling). The Rust ecosystem is quite rich in protocols and codecs, it shouldn't be too hard to find most (all ?) of the crates you need, but there's still work needed to bring them together into one curl-like tool. Note that Rust crates tend to be more focused than what you're used to in C, made to be composed together instead of used as a one-stop-lib. So your dependency tree would look much bigger than curl's.
  - pornel 3y ago
    Curl supports a lot of protocols, including surprising ones like LDAP, SMTP and POP3, so there is no exact curl replacement anywhere, but Rust already has libraries for every protocol that Curl supports. Hyper is a pretty robust HTTP toolkit, and reqwest is a higher-level library on top of it.
gizmo686 3y ago
I've worked on 'build the whole world from source' projects (yocto based). Adding a new compiler to the toolchain is not difficult.

galangalalgol 3y ago

Not wanting to introduce a new compiler is like wanting to introduce a new language. If you don't like the choice the package maintainer makes, roll up your sleeves. That said, we can mitigiate it using mrustc to generate c. Does anyone keep a list of all the targets libcurl is getting built for?

agwa 3y ago

Should the majority of us continue to bear the risks of memory unsafe code because of a small minority that uses exotic architectures or doesn't want to install a new compiler?

rhdunn 3y ago
You need to have someone, or a group of people, willing to write the code to implement a replacement that is fully compatible, and maintain it. Until then, curl written in C is what we have. With that, curl can be used on the exotic architectures and the rust/other language version can be used on other platforms.
- moltonel3x 3y ago
  You need to have someone, or a group of people, willing to write the code to implement support for modern tooling on their platform. Until then, old harder-to-secure codebases is what these platforms have. I know reversing the burden of implementation seems flippant, but it's pragmatic. At some stage, it's less community-wide work to support Rust on a new platform than to spend extra time maintaining/securing dozens of C codebases. Curl may not be making its Rust components mandatory anytime soon, but other projects like python crypto already have, to say nothing of projects written in Rust to begin with. rustc_codegen_gcc is pretty close to ready, let's focus on getting it out of the door, and more target triples supported by rustc and llvm.

dezgeg 3y ago

"Most" is a very heavy claim. Any concrete evidence of this? Also in embedded Linux I guess we are technically "building from source", when using a build system like Buildroot or Yocto. But Rust is already supported there and in fact already in use in our project (due to python3-cryptography).

pmontra 3y ago

What are those "most places" that "want to build from source"? It's a honest question and I intentionally left out the "don’t want to introduce a new compiler to the tool chain" statement to make the set larger. I understand "most" as larger than 50% on some metric. I know about Gentoo and other Linux distributions that do build from source but it has been very uncommon for me to see a company building things from source instead of using apt, rpm, docker in this century. It's at least two orders of magnitude faster. I remember how in the 90s I had to download the source, scan the README for the dependencies, recursively so, then building the libraries, then eventually building the program I downloaded first. Security wise, we were not reading the code, except for Nethack.

jdblair 3y ago
The places that want to build from source are the creators of embedded devices that use curl. These devices include (but are not limited to) wifi access points, routers, home automation devices, data loggers, IP video cameras, smart TVs, set top boxes, and streaming devices. There are a lot of these devices, and there are more all the time. Even if the engineering teams do not build every dependency every time they build firmware (like busybox), they will at some point build all of the components of their system themselves (as supported by yocto).
- dezgeg 3y ago
  Anybody using the standard embedded build solutions like Buildroot or Yocto will get Rust support without doing anything. It's already there and implemented.
- pmontra 3y ago
  That makes sense. It was not clear in the context of the post I replied to.
  - jdblair 3y ago
    Yeah, I admit that maybe this isn't "most places," since there are almost certainly more shops running various versions of Linux servers in production than there are embedded Linux development teams. (edited to fix typo)

pizza234 3y ago

The maintainer wrote a post about this topic: https://daniel.haxx.se/blog/2020/10/09/rust-in-curl-with-hyper https://daniel.haxx.se/blog/2020/10/09/rust-in-curl-with-hyp.... "Most places want to build from source" is not something he considered; there is a brief "for users against the C implementation on the platforms you care about".

steveklabnik 3y ago

Just to be clear, since this hypothetical library exposes a C ABI, pre-built binaries wouldn’t be an issue if somehow magically curl was reimplemented in Rust. Someone would have to compile it, but that’s the same as if they didn’t want to build something written in C from source.

moltonel3x 3y ago

> Replacing parts of curl with Rust will not be possible. It's not just possible, it's been done. You can compile curl with rustls, you could for a time compile it with quiche, and work is ongoing to compile it with hyper. Curl is remarkably modular, none of those are mandatory. > And Rust supports a tiny subset of possible C targets. Gross overstatement. Rust supports the vast majority of devices that people buy today. Even if you ignore platform popularity and just count platforms supported by gcc vs rustc/llvm, there's only a handful missing from the later. And if you're talking about vendor-specific compilers, a lot of them don't support modern C or C++ either.

axus 3y ago
I know almost nothing about the Rust ecosystem but am interested. What Rust packages could completely replace Curl? The top result on Google suggested hyper and reqwest.
- moltonel3x 3y ago
  You'd need a few different ones, just like curl itself uses a lot of 3rd-party libraries to provide its full feature set. Some likely first choices would be hyper (http client/server), rustls (encryption), tokio (async scheduling). The Rust ecosystem is quite rich in protocols and codecs, it shouldn't be too hard to find most (all ?) of the crates you need, but there's still work needed to bring them together into one curl-like tool. Note that Rust crates tend to be more focused than what you're used to in C, made to be composed together instead of used as a one-stop-lib. So your dependency tree would look much bigger than curl's.
- pornel 3y ago
  Curl supports a lot of protocols, including surprising ones like LDAP, SMTP and POP3, so there is no exact curl replacement anywhere, but Rust already has libraries for every protocol that Curl supports. Hyper is a pretty robust HTTP toolkit, and reqwest is a higher-level library on top of it.

gizmo686 3y ago

I've worked on 'build the whole world from source' projects (yocto based). Adding a new compiler to the toolchain is not difficult.