25 ms·

Curl is C

512 points by mhasbini 10y ago

empath75 10y ago
This seems like a no-brainer for a re-implementation in rust, but I wouldn't expect that someone would rewrite curl itself in rust, but a new library that does the same things.
- pionar 10y ago
  but why? What does one gain over just using libcurl?
  - eckza 10y ago
    The novelty of being "the guy that rewrote libcurl, but in Rust!". Probably.
  - mathw 10y ago
    It might be fun. I don't know that you'd gain much in the real world though. Starting such a project now? Rust, for sure, for me anyway. But is it worth rewriting Curl? I agree with the author there, it most probably isn't.
  - jjnoakes 10y ago
    Shorter code, since rust is a higher level of abstraction. Safer code, so half of the vulnerabilities wouldn't have existed. Rust's ecosystem (package manager and libraries). Of course you lose portability and you probably appeal to fewer developers, at least for now. So there is a trade-off. I wish Rust compiled to C. It would be my dream language. The only reason I can't choose rust half the time is because it doesn't support targets I need to support.
    - pjmlp 10y ago
      LLVM used to have a C generator backend, but it was dropped.
      - jjnoakes 10y ago
        LLVM still does (Julia revived it) but last I heard it was only implemented and/or tested for the subset of LLVM IR/bitcode that Julia exercises. My dream is human-readable C as well, which I wouldn't get via LLVM, i.e. a coffeescript-like mapping between Rust and C, as far as that will go. Like I said, it's a dream. I started working on something like it, but I was side-tracked, as often happens. Maybe I'll start it again...
        notriddle 10y ago
        mrustc?
        steveklabnik 10y ago
        mrustc is a rust compiler written in C. You're probably thinking of Corrode, which is a Rust -> C compiler.
        gue5t 10y ago
        You're thinking of `rrode`.
        dbaupp 10y ago
        As the sibling is implying, you flipped the arrow: Corrode is a C -> Rust compiler/transpiler.
- deleted 10y ago
  [deleted]
- floatboth 10y ago
  > a new library that does the same things Most languages already have HTTP client libraries. (In particular, Rust has Hyper. Ruby/Python/Node/Go have HTTP clients built-in in the stdlib, Haskell has http-client, etc.) Who uses libcurl really? (Spoiler alert… PHP.) Of course libcurl does FTP and Gopher and all the things, but these aren't commonly required, most applications just need HTTPS.
  - IshKebab 10y ago
    People that write C and often C++ use libcurl. A better library for C/C++ developers would be nice and I believe it could be written in Rust, although that would be a bit of a pain because then you need to integrate Rust into your build system.
    - LeonidasXIV 10y ago
      Do you? For building C projects I certainly don't have to build libcurl, it comes packaged and ready to use with my distribution. The same could be the case with a hypothetical HTTP library written in Rust.
    - floatboth 10y ago
      Er, I meant, "who uses libcurl when writing NOT in C/C++"
  - bkanber 10y ago
    > Who uses libcurl really? Thousands upon thousands of projects rely on libcurl...
  - geodel 10y ago
    Also check Rust/Cargo source trees if they use `Curl` or Hyper.
    - stusmall 10y ago
      Cargo depends on curl right now.
  - 10y ago
unwind 10y ago
Well put. Didn't know that curl was stuck back on C89, that's really optimizing for portability. If anyone is confused by the "curl sits in the boat" section header, that's basically a Swedish idiom being translated straight to English. That rarely works, of course, and I'm sure Daniel knows this. :) The closest English analog would be "curl doesn't rock the boat", I think the two expressions are equivalent (if you sit, you don't rock the boat).
- paulddraper 10y ago
  I didn't know "sit in the boat was a thing", but I liked it. "Sit in boat" is a positive expression of the stability benefits.
- cat199 10y ago
  he mentions rocking the boat later in the paragraph..
rwmj 10y ago
While this doesn't so much apply to libcurl (but see below), there is a third alternative to "write everything in C" or "write everything in <some other safer language>". That is: use a safer language to generate C code. End users, even those compiling from source, will still only need a C compiler. Only developers need to install the safer language (even Curl developers must install valgrind to run the full tests). Where can you use generated code? - For non-C language bindings (this could apply to the Curl project, but libcurl is a bit unusual in that it doesn't include other bindings, they are supplied by third parties). - To describe the API and generate header files, function prototypes, and wrappers. - To enforce type checking on API parameters (eg. all the CURL_EASY_... options could be described in the generator and then that can be turned into some kind of type checking code). - Any other time you want a single source of truth in your codebase. We use a generator (written in OCaml, generating mostly C) successfully in two projects: https://github.com/libguestfs/libguestfs/tree/master/generator https://github.com/libguestfs/libguestfs/tree/master/generat... https://github.com/libguestfs/hivex/tree/master/generator https://github.com/libguestfs/hivex/tree/master/generator
- chii 10y ago
  > generate C code. how is that different from just writing it in another language? End users who need to compile will be able to regardless of the generated C code, but the end users who need to do a _little_ modification will be given ugly generated C code! Seems stictly worse to me...
  - fiedzia 10y ago
    The difference is that you don't need compiler for this language. There are many hardware platforms that only come with C compiler.
    - tyingq 10y ago
      For something like curl, where the library is as popular as the command line tool, preserving the C ABI compatibility is probably the strongest reason.
simias 10y ago
I have no problem with Curl being written in C (I'll take battle-tested C over experimental Rust) but this point seemed odd to me: >C is not the primary reason for our past vulnerabilities >There. The simple fact is that most of our past vulnerabilities happened because of logical mistakes in the code. Logical mistakes that aren’t really language bound and they would not be fixed simply by changing language. So I looked at https://curl.haxx.se/docs/security.html https://curl.haxx.se/docs/security.html #61 -> uninitialized random : libcurl's (new) internal function that returns a good 32bit random value was implemented poorly and overwrote the pointer instead of writing the value into the buffer the pointer pointed to. #60 -> printf floating point buffer overflow #57 -> cookie injection for other servers : The issue pertains to the function that loads cookies into memory, which reads the specified file into a fixed-size buffer in a line-by-line manner using the fgets() function. If an invocation of fgets() cannot read the whole line into the destination buffer due to it being too small, it truncates the output This one is arguably not really a failure of C itself, but I'd argue that Rust encourages a more robust error handling through its Options and Results when C tends to abuse "-1" and NULL return types that need careful checking and can't usually be enforced by the compiler. #55 -> OOB write via unchecked multiplication Rust has checked multiplication enabled by default in debug builds, and regardless of that the OOB wouldn't be possible. #54 -> Double free in curl_maprintf #53 -> Double free in krb5 code #52 -> glob parser write/read out of bound And I'll stop here, so far 7 out of 11 vulnerabilities would probably have been avoided with a safer language. Looks like the vast majority of these issues wouldn't have been possible in safe Rust.
- eddieroger 10y ago
  > Of course that leaves a share of problems that could’ve been avoided if we used another language. Buffer overflows, double frees and out of boundary reads etc, but the bulk of our security problems has not happened due to curl being written in C. He addressed all of those points in the second short paragraph. None of those are C vulnerabilities, they were mistakes made on the part of the developers, not the language. Avoidance of problems in a safer language doesn't mean when things happen, it's the language's fault.
  - 10y ago
faragon 10y ago
Another reason: C is beautiful.
- madphrodite 10y ago
  It is. It is an elegant language, at least I've always found it to be.
fiatjaf 10y ago
Why are you saying this? Who asked? I always imagined it was written in C.
- tempodox 10y ago
  You know that nobody forces you to read it, right?
- grimgrin 10y ago
  Dude, it's in the first sentence. > Every once in a while someone suggests to me that curl and libcurl would do better if rewritten in a “safe language”. And so he minimally addresses that and some other reasons for sticking with (and originally choosing) C89.
  - fiatjaf 10y ago
    Oops. I read everything after the first subheader.
tannhaeuser 10y ago
Not only is curl based on C, but so are operating systems, IP stacks and network software, drivers, databases, Unix userland tools, web servers, mail servers, parts of web browsers and other network clients, language runtimes and libs of higher-level languages, compilers and almost all other infrastructure software we use daily. I know there's a sentiment here on HN against C (as evidenced by bitter comments whenever a new project dares to choose C) but I wish there'd be a more constructive approach, acknowledging the issue isn't so much new software but the large collection of existing (mostly F/OSS) software not going to be rewritten in eg. Rust or some (lets face it) esoteric/niche FP language. Even for new projects, the choice of programming language isn't clear at all if you value integration and maintainability aspects.
- erelde 10y ago
  I am student and I like C, I've tested Rust/Go, I like the feeling of C. Maybe that sentiment will change later, but for now, I like C. It's simple and sharp and there's lots of doc/books.
  - floatboth 10y ago
    Have you tried doing string manipulation in C? ;)
    - boduh 10y ago
      +1
    - erelde 10y ago
      You can see two of my current school assignement on Github. I try to avoid manipulating strings as much as possible. ;)
    - asveikau 10y ago
      Not OP, but I actually think string manipulation in C is really elegant. Many people who complain about it have too many allocations in their code and are trying to port the allocation-heavy non-C way of thinking to C. The C way I know focuses mainly on character at a time iteration with emphasis on not copying the source string. I'm reminded of a time a colleague needed something like string.split, and working in c++ he filled a std::vector<std::string> with the result. Using a more C way he'd really only have needed a couple of pointers on the stack.
ameliaquining 10y ago
I'm kind of torn on this. On the one hand, Curl is a great piece of software with a better security record than most, the engineering choices it's made thus far have served it just fine, and its developers quite reasonably view rewriting it as risky and unnecessary. On the other hand, the state of internet security is really terrible, and the only way it'll ever get fixed is if we somehow get to the point where writing networking code in a non-memory-safe language is considered professional malpractice. Because it should be; reliably not introducing memory corruption bugs without a compiler checking your work is a higher standard than programmers can realistically be held to, and in networking code such bugs often have immediate and dramatic security consequences. We need to somehow create a culture where serious programmers don't try to do this, the same way serious programmers don't write in BASIC or use tarball backups as version control. That so much existing high-profile networking software is written in C makes this a lot harder, because everyone thinks "well all those projects do it so it must be okay".
- baldfat 10y ago
  > the only way it'll ever get fixed is if we somehow get to the point where writing networking code in a non-memory-safe language is considered professional malpractice. So using Linux, Windows, BSD or MacOS servers are malpractice? I think you might have over stated your case. So are you waiting for a memory safe Herd re-write? A memory safe any OS will be decades away if someone wanted to start tackling it now.
  - freddref 10y ago
    writing, not using.
  - pjmlp 10y ago
    Microsoft does research how to improve security at OS level and sometimes those efforts do end up on Windows. Latest examples, Windows 10 secure kernel and Device Driver protection. https://myignite.microsoft.com/sessions/36925 https://myignite.microsoft.com/sessions/36925 Or the new Windows USB stack, written in the P language. https://github.com/p-org/P https://github.com/p-org/P UNIXes, not so much beyond patching C exploits.
Sir_Cmpwn 10y ago
Agreed 100%. Definitely going to be trotting this article out next time I see someone blindly arguing for rewriting xyz in Rust. I particularly like the mention of portability. No other language comes even remotely close to the portability of C. What other language runs on Linux, NT, BSD, Minix, Mach, VAX, Solaris, plan9, Hurd, eight dozen other platforms, freestanding kernels, and nearly every architecture ever made?
- cwyers 10y ago
  I mean, sure, and if you have users running VAX or the Hurd, that matters. But it turns out that most of us use one of Linux, NT or OS X. And even if you add BSD and Solaris (and a few other Unixes) you can still find languages without C's known problems that cover 100% of users. "But embedded." Embedded can maintain their own software, they do all the time. How long are we going to insist that end users run software that cannot be secure because of the lowest common denominator of programming languages?
  - Sir_Cmpwn 10y ago
    I think this is a flawed mindset for a number of reasons. First, I'd rather appeal to every user than most users. That one user I didn't have to appeal to is going to be a much more faithful and grateful user than the "normal" ones. Most of my software work is open source (remember this context is a discussion about curl), and this encourages active collaboration with users with niche situations. If I choose technologies that make using my software attainable for these people, odds are they aren't going to stop at just porting it to their platform. Limiting your platforms to Linux, OSX, and NT also stifles innovation. These platforms are all deeply flawed. Their popularity isn't due to having the best design, but rather to having a good enough design and being entrenched. They're old platforms, we've learned a lot since they were started. New or niche platforms bring a lot of value to the table. The BSDs are a great example, as it's the best suited platform for a wide variety of applications. All a new platform has to do to be able to run nearly all general purpose software is port a C compiler. Not even that - they just need a cross compiler. This is a great thing, IMO. >Embedded can maintain their own software, they do all the time This is a pretty silly argument. Most embedded developers don't ship their own implementation of HTTP, they ship curl!
coldtea 10y ago
>There. The simple fact is that most of our past vulnerabilities happened because of logical mistakes in the code. Logical mistakes that aren’t really language bound and they would not be fixed simply by changing language. That's wrong. A lot of the C mistakes are indeed "logical mistakes in the code", but most of them would be indeed fixed by changing to a language that prevents those mistakes in the first place.
oldsj 10y ago
> The plain fact, that also isn’t really about languages but is about plain old software engineering: translating or rewriting curl into a new language will introduce a lot of bugs. Bugs that we don’t have today. Don't rewrites, even in the same language usually lead to a better version of the software? I can't really imagine a seasoned C developer introducing completely new bugs in a code base they are already very familiar with
- fiedzia 10y ago
  > I can't really imagine a seasoned C developer introducing completely new bugs in a code base they are already very familiar with Get a CVE list for some Linux distribution, this will open your mind. Happens all the time.
skocznymroczny 10y ago
I don't think C is a bad language, although I think it could use lists and dictionaries in standard library. std::vector and std::map are the only things that make me pick C++ in an instant, given the choice.
davexunit 10y ago
>C is not the primary reason for our past vulnerabilities Completely false. C is a disaster.
renesd 10y ago
CPython also has many vulnerabilities in python rather than C. It's hilarious reading rust marketers talk about how people should use rust, and yet their software doesn't work as well. It has plenty of bugs. Then they go on and on about issues which post modern C doesn't have. Guess what? C has a lot of tooling, and yes, it's been improving over the years too. CQual++ exists. AFL exists. QuickCheck exists. Can your rust project from two years ago even compile? Does it have any users at all? There's a formally proven C compiler. How's that LLVM swamp going you've built your castle on? Rust brought a modern knife to a post modern gun fight -- and lost.
- steveklabnik 10y ago
  > Can your rust project from two years ago even compile? Post 1.0's release date, which is just short of two years, the vast, vast majority should, yes. We've had one or two soundness fixes in those times that would take a trivial amount of updating to do, but that only hit a very small part of the ecosystem.
- sanxiyn 10y ago
  > Can your Rust project from two years ago even compile? Eh, two years have not passed yet since Rust 1.0. But yes, it is likely that your Rust project today will compile just fine in 2019.
- wtetzner 10y ago
  > How's that LLVM swamp going you've built your castle on? http://www.cis.upenn.edu/~stevez/vellvm/ http://www.cis.upenn.edu/~stevez/vellvm/ > It has plenty of bugs. Rust doesn't claim to prevent all bugs. It only claims memory safety.
madphrodite 10y ago
This is a great little read and encapsulates the other side of the 'rethink the way' trend-ism of some HN new lang advocacy. C is fine, C is good. It is widely understood, it is a systems staple, and it is not dangerous in knowledgeable hands. Rocking the boat is fashionable.
- macintux 10y ago
  > it is not dangerous in knowledgeable hands For crying out loud, this is patently false. C isn't as dangerous if you really know what you're doing and you never make mistakes. I would bet fewer people know what they're doing than think they know what they're doing, and the set of people who never make mistakes is entirely empty.
  - madphrodite 10y ago
    Well you know..that's just like your opinion man. Opting out of this site at this point. A bunch of 6-12 year idiots (recognizably) telling people what they think they know that they don't know. It's silly recursive.
    - macintux 10y ago
      You're right, my reply was unnecessarily harsh. I apologize.
geodel 10y ago
I think Rust community increasingly behave like this[1]. They are big on suggesting others the better 'ideas' instead of implementing themselves. So they keep using 'curl' and 'openssl' but tell others to rewrite their software with Rust. 1. http://dilbert.com/strip/1994-12-17 http://dilbert.com/strip/1994-12-17
- lifthrasiir 10y ago
  I would say a portion of Rust community---that said, it is that portion that is the most visible, and I think the community well understands what it does mean.
  - sidlls 10y ago
    I'm not sure the community well understands it at all. The usual rejoinder is something along the lines of "well, I don't see that in the Rust community I'm in."
    - lifthrasiir 10y ago
      Both /r/rust [1] and the official forum [2] are reasonably regulated and I often see less informed members got warned about their use of aggressive or insulting languages, often directed to non-Rust languages. There are surely other venues with less enforcement (they still commonly observe the Rust Code of Conduct however), but at least I think the main venues and corresponding community heavily tries not to be offensive. [1] https://www.reddit.com/r/rust/ https://www.reddit.com/r/rust/ [2] https://users.rust-lang.org/ https://users.rust-lang.org/
- Manishearth 10y ago
  Like, a lot of the rust community is putting the effort into rewriting things in Rust. From within the community I don't see any coherent effort to tell folks to rewrite in Rust. We're very happy to see Rust rewrites, but not many people are pushing for it except the folks actually putting effort into it. Yes, every time a vulnerability pops up someone will say "rewrite it in Rust", but half the time it's not even a Rust programmer (often, Rust programmers come and disagree and say "Rust wouldn't fix this", IME).
jeffdavis 10y ago
"C is not a new dependency" To just use a library, rust isn't much of a dependency, either. It's designed so you don't even need to know that it's not C. Rust would obviously be a build dependency, but that's lessened somewhat because it tries to make cross-compilation easy. (But this point does apply to pretty much any other language. Curl would not be used as widely if it depended on the Go runtime, for instance.)
bitwize 10y ago
The maintainer of curl, a shitty HTTP client/library, takes a stand against the Rust Evangelism Strikeforce. The Rust Evangelism Strikeforce declares him a Suppressive Person.
- bluejekyll 10y ago
  I highly doubt anyone in the Rust community looks at it this way. As a huge Rust fan, I'm also a C fan. C existed before Rust; before Rust it was my favorite language, though I rarely chose it for work b/c of safety. The strongest argument in this piece is that Curl is written in the most portable language available. That's a great reason! And it's been a wildly successful project! Kudos to the developer(s)! The only problem with this piece is the claim on what are logic bugs vs. C bugs; but that should not detract from an excellent project. The real question being debated is this: if starting a new project like this, should you do it in a safe language to guard against security issues, or should you do it in the oldest most portable language ever. I'd argue that by the time any project becomes successful after being written in Rust, the LLVM (and Rust) will probably close that portability gap.
- sctb 10y ago
  I think we just asked you not to post like this, did we not? Please stop.
kazinator 10y ago
> The simple fact is that most of our past vulnerabilities happened because of logical mistakes in the code. Logical mistakes that aren’t really language bound and they would not be fixed simply by changing language. This statement is laughable nonsense. Shall we go into their bug history and point out counterexamples left and right? [Edit:user simias has done this; thanks!] Every single bug you ever make interacts with the language somehow. Even if you think some bug is nothing but pure, that logic is part of a program, embedded in the program's design, whose organization is driven by language.
devy 10y ago
The 7th point: "curl sits in the boat" In the curl project we’re deliberately conservative and we stick to old standards, to remain a viable and reliable library for everyone. Right now and for the foreseeable future. Things that worked in curl 15 years ago still work like that today. The same way. Users can rely on curl. We stick around. We don’t knee-jerk react to modern trends. We sit still in the boat. We don’t rock it. I see a lot of inertia in there. While it's a great record to maintain 15-year consistency but in the era of every changing InfoSec outlook, it could be a legacy and baggage if the authors resist to change. One thing we know for sure is that human will make mistakes, no matter how skillful you are. In the context of writing a fundamental piece of software with an unsafe programming language, that means we are guarantee to have memory-safety induced CVE bugs in curl in the future. Some of other points that the author raised are valid too. If there is a trade-off that we can have a safer piece of fundamental software by almost eliminating a whole category of memory safety related bugs, and with the downside of less compatibility with legacy systems, more dependencies etc., perhaps we should consider it? I believe the tradeoff is well worthy in the long run and option is ripe for explore.
- cestith 10y ago
  How is the author resistant to change? He specifically said new code should be written in a language that meets the priorities for that code. He specifically said someone has or would write a competitor to curl in Rust or some other safer language and that a good one will take off. He welcomed that. What he doesn't welcome is rewriting something that's had those bugs and the types of logic bugs not related to the language already worked out. There's a saying about a baby and bathwater. Not everything is a dichotomy, and you shouldn't be reading the article as if the author is against newer languages. He specifically says that given a fresh start with the availability of these languages he might use something besides C. Carefully weighing options is wise. Throwing away years of actual progress for the appearance of quick progress is foolish.
  - devy 10y ago
    > How is the author resistant to change? I specially quoted the section head "curl sits in the boat" and the entire section ends with "We sit still in the boat. We don’t rock it". Now read it again, and then tell me if that's welcoming changes or resisting changes. > He specifically said someone has or would write a competitor to curl in Rust or some other safer language and that a good one will take off. He welcomed that. Sure there might already be some alternatives out there. But those are not curl, they are at most forks. > He specifically says that given a fresh start with the availability of these languages he might use something besides C. Nope, he used the word "Maybe. Maybe not." Might is a stronger word.
krystiangw 10y ago
C has still huge market share. Seems that it occurs in 5% of all tech job offers: https://jobsquery.it/stats/data.technologies/C https://jobsquery.it/stats/data.technologies/C Stats also showing that average salary for C developers is above average for all tech job openings.
coding123 10y ago
I don't see why this is an issue, whoever is arguing for a change can write rurl and be done, and see if anyone takes it up in their distributions.
dmitrygr 10y ago
Software is almost a perfectly open market. If proponents of rust really think their preferred language is better in every way, they are free to rewrite the world in rust, and see the adoption numbers they get. After all, if rust is better in every way, we'd expect the adoption numbers to go up for their rust OS, with a rust http stack and rust web browser. Right? Telling others to use their language instead of putting their money where their mouth is is truly what irks me about the rust community the most. Want a rust world? Go write it and ship it. Oh, and you don't get to complain about C until your PC runs more rust than C Cool?
- steveklabnik 10y ago
  I don't think the majority of Rust users disagree.
- arcticbull 10y ago
  I'm not sure why I can't complain about problems I have today -- if nobody complained about C ever, I doubt Rust, Go or for that matter, basically any other language project, would have ever gotten started. It's okay to identify flaws in tools, that's how we make them better. It doesn't make sense to say '[car on fire] you can't complain about that Honda Civic until you develop your own better car, sir, and more people are driving it than not, now please leave the service center -- until then consider the fire normal'.
- deleted 10y ago
  [deleted]
tlrobinson 10y ago
I'm curious, of the bugs that could have been avoided by using a "safe" language, how many could have been avoided by using a bounds checking extension like as https://www.doc.ic.ac.uk/~phjk/BoundsChecking.html https://www.doc.ic.ac.uk/~phjk/BoundsChecking.html or https://github.com/Microsoft/checkedc https://github.com/Microsoft/checkedc Are such extensions popular, and if not, why not? I assume there's always some performance hit, but that might not be a big deal in an HTTP client, for example.
- pjmlp 10y ago
  Because they aren't portable.
koja86 10y ago
Quite recently I happily used libcurl for C++ project rather than any of those C++ wrappers found at github. Granted there is some non-elegance when you adapt C-style error codes to C++ exceptions and non-C++-idiomatic code style right next to any C lib. Yet libcurl is battle tested (AKA proved to be rather bug free) and has nice clean API unlike. IMHO it might eventually make sense to use other language/tech/whatever but the bar is quite high and it will quite probably take some serious sustained effort.
tombert 10y ago
While I'm definitely not suggesting we replace Curl with a rewrite in Rust (since the current Curl has had decades of good testing and auditing done on it), I am actually very curious how a rewrite in a safer language like Rust, OCaml, Haskell, or Go would fair in comparison in regards to performance and whatnot. If I were ambitious enough, I'd do it myself in Haskell, but I think it'd be too much work for a simpler curiosity.
- notriddle 10y ago
  There's hyper, if you want something that exists now, but it only does HTTP.
- fiedzia 10y ago
  Most likely I/O will take more time than whatever code is running, so in that aspect it would make no difference. Memory overhead is main concern here. Rust doesn't use GC, so you'll have full control and there should be not much difference in that aspect. Other languages do, which means more sophisticated runtime, less control and more overhead (or writing ugly code to avoid it). Libcurl written in Go/Ocaml/Haskell would require anyone using it to also include runtime of the language, which is usually rather large.
didip 10y ago
It's well within reason and capabilities for rust community to write libcurl and curl CLI libraries. The community should do it, spend a couple of years stabilizing, and then spread the words to others.
- deleted 10y ago
  [deleted]
_of 10y ago
Rust might be a great language, but it has not completed the test of time yet. C is 45 years old. Rust appeared 7 years ago.
carapace 10y ago
This has probably been said, in this thread even, but if curl is insecure (for some value of "insecure") then its ubiquity and ease of embedding are a problem rather than a feature. Fuzzy thinking.
AstralStorm 10y ago
Maybe they should attempt writing it using the Isabelle/HOL transpilers to C from SEL4 project. I don't care if it is C or machine code as long as the proof of correctness is complete, down to at least C library. Curl is small enough to make it relatively easy and used widely enough to make it worthwhile.
adynatos 10y ago
While C by itself is not safe, I would argue that no sane development environment uses C by itself. Over the decades of its production use dozens of tools have been developed that make it far safer: *grind suite, coverage tools, sanitizers, static analyzers, code formatters and so on. Those tools are external, otherwise they would make C slower. Something for something.
mgrennan 10y ago
Why does something old (C) have to be bad these days?
- wtetzner 10y ago
  It's not bad because it's old, it's bad because of how unsafe it is. There are older languages than C that are safer, but they didn't gain the popularity C did.
chousuke 10y ago
In my view, the problem with C in general is that it's a loaded gun with no safety or trigger guard. It's trivial to shoot yourself (or someone else) in the foot, and it requires knowledge, meticulous care and lots of forethought to avoid getting shot. I very much agree that rewriting existing, stable software written in C is likely not worth the trouble in many cases, but I can't accept claims that the limitations of C aren't the direct cause of tens of thousands of security vulnerabilities, either. In Rust, even a less experienced developer can fearlessly perform changes in complicated code because the language helps make sure your code is correct in ways that C does not. And you can always turn off the safeties when you need to. Experienced developers should feel all the more empowered by simply not having to always worry about things like accidental concurrent access, use-after-free, object ownership, null pointers or the myriad other trivial ways to cause your program to fail that are impossible in safe Rust. You get to worry about the non-trivial failure modes instead, which is much more productive.
kodest 10y ago
Maybe curl could be rewritten in C++ step by step like mpd (https://musicpd.org https://musicpd.org). C++ has RAII for resource management which can help a lot by itself. In my opinion the most hateful thing in C is freeing resources on all exit paths. Although, curl in C++ - the naming would became inappropriate...
derefr 10y ago
> A library in another language will add that language (and compiler, and debugger and whatever dependencies a libcurl written in that language would need) as a new dependency to a large amount of projects that are themselves written in C or C++ today. Those projects would in many cases downright ignore and reject projects written in “an alternative language”. Why would I be vendoring my own copy of libcurl in my project? Who does? This is how I (or rather, the FFI bindings my language's runtime uses) consume libcurl: dlopen("libcurl.so") I rely on a binary libcurl package. The binary shared-object file in that package needed a toolchain to build it, but I don't need said toolchain to consume it. That would still be true even if the toolchain required for compiling was C++ or Rust or Go or whatever instead of C, because either the languages themselves, or the projects, ensure that the shared-object files they ship export a C-compatible ABI. An example of a project that works the way I'm talking about: LLVM. LLVM is written in C++, but exports C symbols, and therefore "looks like" C to any FFI logic that cares about such things. LLVM is a rather heavyweight thing to compile, but I can use it just fine in my own code without even having a C++ compiler on my machine. (And an example of a project that doesn't work this way: QT. QT has no C-compatible ABI, so even though it's nominally extremely portable, many projects can't or won't link QT. QT fits the author's argument a lot better than an alternate-language libcurl would.)
lettergram 10y ago
I feel blaming a language for errors is like blaming a gun for killing people. The fact is, mistakes will happen, but in general if you follow the best practices you'll be fine. Failing to follow the best practices means you could be a better programmer. Just because the language gives you an option to do something, doesn't mean you should.
- myrrlyn 10y ago
  "I don't need a safety toggle on my gun I just totes keep it pointed at the ground and unloa-- oh god where's my foot"
  - trav4225 10y ago
    Still not the gun's fault.
    - roca 10y ago
      In some sense, sure. But in practice, people always make mistakes. Some guns/programming languages limit the damage of those mistakes more than others. All other things being equal, those guns/programming languages are better.
      - lettergram 10y ago
        I'd agree with you, but all things are not equal.
      - trav4225 10y ago
        Yup, no argument from me there. There's always room for improvement...
- roca 10y ago
  > mistakes will happen, but in general if you follow the best practices you'll be fine Everyone fails to follow best practices at times, precisely because "mistakes will happen!" Encouraging people to be better programmers can't change that. Therefore we want languages that catch as many mistakes as possible at build time and minimize the damage caused by other mistakes that slip through.
throwaway5752 10y ago
It's extremely simple. If you think Curl would be better in another language then port it, release your alternative, and maintain it for a long time. Even if your language (Rust, Erlang, LISP, Go) is "better", it's still a minimal part of the equation. A maintainer is what makes the tool. It's hard work to decide which PRs to accept (and worse yet, reject), to backport fixes to platforms for which you can't get a reliable contributor, coordinating fundraising/donations, keeping up with evolving standards... Anyway. Thank you, thank you, thank you Daniel Stenberg. Use whatever damn language you want.
- kazinator 10y ago
  > Use whatever damn language you want. On the other hand, if he didn't want his justifications for that choice examined by the world, he wouldn't have aired them, right? > If you think Curl would be better in another language then port it, release your alternative, and maintain it for a long time. That's out there; some languages have URL downloading objects that are not based on Curl. E.g. Edi Weitz's Drakma client library for Common Lisp doesn't seem to be using Curl as far as I can see. http://weitz.de/drakma/ http://weitz.de/drakma/
  - throwaway5752 10y ago
    I wouldn't presume to speak for Daniel, but I got the feeling that he just wanted to publish this to point people to rather than send the same canned response to inquiries about porting to Rust et al. Drakma sounds great. Tone is tough to get right online. I don't like people doing drive-by suggestions like "you should rewrite X in Y". But if people are really willing to roll up their sleeves, write the tool (in any language) and keep it going for the long haul, I applaud them. I just have great respect and empathy for project maintainers, I think some don't appreciate what a huge PITA it is to BDFL a successful project.
digi_owl 10y ago
A refreshing read in what seems to be a ongoing deluge of rewrites, languages and frameworks.
trav4225 10y ago
It is utterly amazing to me to see so many people's attitudes on this issue. If I cut myself by hasty use of a knife, is it the fault of the knife maker? How is that even remotely rational? If you aren't willing (or don't know how) to use the tool correctly, don't use it.
gyrgtyn 10y ago
What is everyone using curl for that it needs to be written in C (or Rust?). If it think about my usage, it's like get or post something and see what the returned json looks like. If I need to download something wget usually works without having to remember -O. But higher level things like httpie are easier to deal with, sane defaults and all that. Maybe they use libcurl... Are there any re-write userland in ${safe-high-level-lang} projects?
tete 10y ago
I think it's a bit weird that C and curl are used. If we look at C and OpenBSD or so things might look a bit different. Also one has a hard time comparing curl with another language, simply because something with curl's properties (take portability for example) doesn't exist. And no that isn't in defense of anything, just me thinking thinking that measurable points brought up in the discussions don't make sense or exist. The topic is also a bit broader, as you can easily add in static code analysis, compiler flags, stuff like W^C, stuff like seccomp, capsicum, cloudabi, pledge which might not work (well) in other cases. It's a great philosophical discussion topics and I don't wanna stop anyone, just hoping people keep that in mind, when they participate, so we don't end up with new dogmas that get thrown around for the next few year, without knowing contexts or meaning of phrases. Other than that: I really enjoy this discussion. :)

512 points by mhasbini 10y ago

This seems like a no-brainer for a re-implementation in rust, but I wouldn't expect that someone would rewrite curl itself in rust, but a new library that does the same things.

pionar 10y ago
but why? What does one gain over just using libcurl?
- eckza 10y ago
  The novelty of being "the guy that rewrote libcurl, but in Rust!". Probably.
- mathw 10y ago
  It might be fun. I don't know that you'd gain much in the real world though. Starting such a project now? Rust, for sure, for me anyway. But is it worth rewriting Curl? I agree with the author there, it most probably isn't.
- jjnoakes 10y ago
  Shorter code, since rust is a higher level of abstraction. Safer code, so half of the vulnerabilities wouldn't have existed. Rust's ecosystem (package manager and libraries). Of course you lose portability and you probably appeal to fewer developers, at least for now. So there is a trade-off. I wish Rust compiled to C. It would be my dream language. The only reason I can't choose rust half the time is because it doesn't support targets I need to support.
  - pjmlp 10y ago
    LLVM used to have a C generator backend, but it was dropped.
    - jjnoakes 10y ago
      LLVM still does (Julia revived it) but last I heard it was only implemented and/or tested for the subset of LLVM IR/bitcode that Julia exercises. My dream is human-readable C as well, which I wouldn't get via LLVM, i.e. a coffeescript-like mapping between Rust and C, as far as that will go. Like I said, it's a dream. I started working on something like it, but I was side-tracked, as often happens. Maybe I'll start it again...
      - notriddle 10y ago
        mrustc?
        steveklabnik 10y ago
        mrustc is a rust compiler written in C. You're probably thinking of Corrode, which is a Rust -> C compiler.
        gue5t 10y ago
        You're thinking of `rrode`.
        dbaupp 10y ago
        As the sibling is implying, you flipped the arrow: Corrode is a C -> Rust compiler/transpiler.
deleted 10y ago
[deleted]
floatboth 10y ago
> a new library that does the same things Most languages already have HTTP client libraries. (In particular, Rust has Hyper. Ruby/Python/Node/Go have HTTP clients built-in in the stdlib, Haskell has http-client, etc.) Who uses libcurl really? (Spoiler alert… PHP.) Of course libcurl does FTP and Gopher and all the things, but these aren't commonly required, most applications just need HTTPS.
- IshKebab 10y ago
  People that write C and often C++ use libcurl. A better library for C/C++ developers would be nice and I believe it could be written in Rust, although that would be a bit of a pain because then you need to integrate Rust into your build system.
  - LeonidasXIV 10y ago
    Do you? For building C projects I certainly don't have to build libcurl, it comes packaged and ready to use with my distribution. The same could be the case with a hypothetical HTTP library written in Rust.
  - floatboth 10y ago
    Er, I meant, "who uses libcurl when writing NOT in C/C++"
- bkanber 10y ago
  > Who uses libcurl really? Thousands upon thousands of projects rely on libcurl...
- geodel 10y ago
  Also check Rust/Cargo source trees if they use `Curl` or Hyper.
  - stusmall 10y ago
    Cargo depends on curl right now.
- 10y ago

unwind 10y ago

Well put. Didn't know that curl was stuck back on C89, that's really optimizing for portability. If anyone is confused by the "curl sits in the boat" section header, that's basically a Swedish idiom being translated straight to English. That rarely works, of course, and I'm sure Daniel knows this. :) The closest English analog would be "curl doesn't rock the boat", I think the two expressions are equivalent (if you sit, you don't rock the boat).

paulddraper 10y ago
I didn't know "sit in the boat was a thing", but I liked it. "Sit in boat" is a positive expression of the stability benefits.
cat199 10y ago
he mentions rocking the boat later in the paragraph..

rwmj 10y ago

While this doesn't so much apply to libcurl (but see below), there is a third alternative to "write everything in C" or "write everything in <some other safer language>". That is: use a safer language to generate C code. End users, even those compiling from source, will still only need a C compiler. Only developers need to install the safer language (even Curl developers must install valgrind to run the full tests). Where can you use generated code? - For non-C language bindings (this could apply to the Curl project, but libcurl is a bit unusual in that it doesn't include other bindings, they are supplied by third parties). - To describe the API and generate header files, function prototypes, and wrappers. - To enforce type checking on API parameters (eg. all the CURL_EASY_... options could be described in the generator and then that can be turned into some kind of type checking code). - Any other time you want a single source of truth in your codebase. We use a generator (written in OCaml, generating mostly C) successfully in two projects: https://github.com/libguestfs/libguestfs/tree/master/generator https://github.com/libguestfs/libguestfs/tree/master/generat... https://github.com/libguestfs/hivex/tree/master/generator https://github.com/libguestfs/hivex/tree/master/generator

chii 10y ago
> generate C code. how is that different from just writing it in another language? End users who need to compile will be able to regardless of the generated C code, but the end users who need to do a _little_ modification will be given ugly generated C code! Seems stictly worse to me...
- fiedzia 10y ago
  The difference is that you don't need compiler for this language. There are many hardware platforms that only come with C compiler.
  - tyingq 10y ago
    For something like curl, where the library is as popular as the command line tool, preserving the C ABI compatibility is probably the strongest reason.

simias 10y ago

I have no problem with Curl being written in C (I'll take battle-tested C over experimental Rust) but this point seemed odd to me: >C is not the primary reason for our past vulnerabilities >There. The simple fact is that most of our past vulnerabilities happened because of logical mistakes in the code. Logical mistakes that aren’t really language bound and they would not be fixed simply by changing language. So I looked at https://curl.haxx.se/docs/security.html https://curl.haxx.se/docs/security.html #61 -> uninitialized random : libcurl's (new) internal function that returns a good 32bit random value was implemented poorly and overwrote the pointer instead of writing the value into the buffer the pointer pointed to. #60 -> printf floating point buffer overflow #57 -> cookie injection for other servers : The issue pertains to the function that loads cookies into memory, which reads the specified file into a fixed-size buffer in a line-by-line manner using the fgets() function. If an invocation of fgets() cannot read the whole line into the destination buffer due to it being too small, it truncates the output This one is arguably not really a failure of C itself, but I'd argue that Rust encourages a more robust error handling through its Options and Results when C tends to abuse "-1" and NULL return types that need careful checking and can't usually be enforced by the compiler. #55 -> OOB write via unchecked multiplication Rust has checked multiplication enabled by default in debug builds, and regardless of that the OOB wouldn't be possible. #54 -> Double free in curl_maprintf #53 -> Double free in krb5 code #52 -> glob parser write/read out of bound And I'll stop here, so far 7 out of 11 vulnerabilities would probably have been avoided with a safer language. Looks like the vast majority of these issues wouldn't have been possible in safe Rust.

eddieroger 10y ago
> Of course that leaves a share of problems that could’ve been avoided if we used another language. Buffer overflows, double frees and out of boundary reads etc, but the bulk of our security problems has not happened due to curl being written in C. He addressed all of those points in the second short paragraph. None of those are C vulnerabilities, they were mistakes made on the part of the developers, not the language. Avoidance of problems in a safer language doesn't mean when things happen, it's the language's fault.
- 10y ago

faragon 10y ago

Another reason: C is beautiful.

madphrodite 10y ago
It is. It is an elegant language, at least I've always found it to be.

fiatjaf 10y ago

Why are you saying this? Who asked? I always imagined it was written in C.

tempodox 10y ago
You know that nobody forces you to read it, right?
grimgrin 10y ago
Dude, it's in the first sentence. > Every once in a while someone suggests to me that curl and libcurl would do better if rewritten in a “safe language”. And so he minimally addresses that and some other reasons for sticking with (and originally choosing) C89.
- fiatjaf 10y ago
  Oops. I read everything after the first subheader.

tannhaeuser 10y ago

Not only is curl based on C, but so are operating systems, IP stacks and network software, drivers, databases, Unix userland tools, web servers, mail servers, parts of web browsers and other network clients, language runtimes and libs of higher-level languages, compilers and almost all other infrastructure software we use daily. I know there's a sentiment here on HN against C (as evidenced by bitter comments whenever a new project dares to choose C) but I wish there'd be a more constructive approach, acknowledging the issue isn't so much new software but the large collection of existing (mostly F/OSS) software not going to be rewritten in eg. Rust or some (lets face it) esoteric/niche FP language. Even for new projects, the choice of programming language isn't clear at all if you value integration and maintainability aspects.

erelde 10y ago
I am student and I like C, I've tested Rust/Go, I like the feeling of C. Maybe that sentiment will change later, but for now, I like C. It's simple and sharp and there's lots of doc/books.
- floatboth 10y ago
  Have you tried doing string manipulation in C? ;)
  - boduh 10y ago
    +1
  - erelde 10y ago
    You can see two of my current school assignement on Github. I try to avoid manipulating strings as much as possible. ;)
  - asveikau 10y ago
    Not OP, but I actually think string manipulation in C is really elegant. Many people who complain about it have too many allocations in their code and are trying to port the allocation-heavy non-C way of thinking to C. The C way I know focuses mainly on character at a time iteration with emphasis on not copying the source string. I'm reminded of a time a colleague needed something like string.split, and working in c++ he filled a std::vector<std::string> with the result. Using a more C way he'd really only have needed a couple of pointers on the stack.

ameliaquining 10y ago

I'm kind of torn on this. On the one hand, Curl is a great piece of software with a better security record than most, the engineering choices it's made thus far have served it just fine, and its developers quite reasonably view rewriting it as risky and unnecessary. On the other hand, the state of internet security is really terrible, and the only way it'll ever get fixed is if we somehow get to the point where writing networking code in a non-memory-safe language is considered professional malpractice. Because it should be; reliably not introducing memory corruption bugs without a compiler checking your work is a higher standard than programmers can realistically be held to, and in networking code such bugs often have immediate and dramatic security consequences. We need to somehow create a culture where serious programmers don't try to do this, the same way serious programmers don't write in BASIC or use tarball backups as version control. That so much existing high-profile networking software is written in C makes this a lot harder, because everyone thinks "well all those projects do it so it must be okay".

baldfat 10y ago
> the only way it'll ever get fixed is if we somehow get to the point where writing networking code in a non-memory-safe language is considered professional malpractice. So using Linux, Windows, BSD or MacOS servers are malpractice? I think you might have over stated your case. So are you waiting for a memory safe Herd re-write? A memory safe any OS will be decades away if someone wanted to start tackling it now.
- freddref 10y ago
  writing, not using.
- pjmlp 10y ago
  Microsoft does research how to improve security at OS level and sometimes those efforts do end up on Windows. Latest examples, Windows 10 secure kernel and Device Driver protection. https://myignite.microsoft.com/sessions/36925 https://myignite.microsoft.com/sessions/36925 Or the new Windows USB stack, written in the P language. https://github.com/p-org/P https://github.com/p-org/P UNIXes, not so much beyond patching C exploits.

Sir_Cmpwn 10y ago

Agreed 100%. Definitely going to be trotting this article out next time I see someone blindly arguing for rewriting xyz in Rust. I particularly like the mention of portability. No other language comes even remotely close to the portability of C. What other language runs on Linux, NT, BSD, Minix, Mach, VAX, Solaris, plan9, Hurd, eight dozen other platforms, freestanding kernels, and nearly every architecture ever made?

cwyers 10y ago
I mean, sure, and if you have users running VAX or the Hurd, that matters. But it turns out that most of us use one of Linux, NT or OS X. And even if you add BSD and Solaris (and a few other Unixes) you can still find languages without C's known problems that cover 100% of users. "But embedded." Embedded can maintain their own software, they do all the time. How long are we going to insist that end users run software that cannot be secure because of the lowest common denominator of programming languages?
- Sir_Cmpwn 10y ago
  I think this is a flawed mindset for a number of reasons. First, I'd rather appeal to every user than most users. That one user I didn't have to appeal to is going to be a much more faithful and grateful user than the "normal" ones. Most of my software work is open source (remember this context is a discussion about curl), and this encourages active collaboration with users with niche situations. If I choose technologies that make using my software attainable for these people, odds are they aren't going to stop at just porting it to their platform. Limiting your platforms to Linux, OSX, and NT also stifles innovation. These platforms are all deeply flawed. Their popularity isn't due to having the best design, but rather to having a good enough design and being entrenched. They're old platforms, we've learned a lot since they were started. New or niche platforms bring a lot of value to the table. The BSDs are a great example, as it's the best suited platform for a wide variety of applications. All a new platform has to do to be able to run nearly all general purpose software is port a C compiler. Not even that - they just need a cross compiler. This is a great thing, IMO. >Embedded can maintain their own software, they do all the time This is a pretty silly argument. Most embedded developers don't ship their own implementation of HTTP, they ship curl!

coldtea 10y ago

>There. The simple fact is that most of our past vulnerabilities happened because of logical mistakes in the code. Logical mistakes that aren’t really language bound and they would not be fixed simply by changing language. That's wrong. A lot of the C mistakes are indeed "logical mistakes in the code", but most of them would be indeed fixed by changing to a language that prevents those mistakes in the first place.

oldsj 10y ago

> The plain fact, that also isn’t really about languages but is about plain old software engineering: translating or rewriting curl into a new language will introduce a lot of bugs. Bugs that we don’t have today. Don't rewrites, even in the same language usually lead to a better version of the software? I can't really imagine a seasoned C developer introducing completely new bugs in a code base they are already very familiar with

fiedzia 10y ago
> I can't really imagine a seasoned C developer introducing completely new bugs in a code base they are already very familiar with Get a CVE list for some Linux distribution, this will open your mind. Happens all the time.

skocznymroczny 10y ago

I don't think C is a bad language, although I think it could use lists and dictionaries in standard library. std::vector and std::map are the only things that make me pick C++ in an instant, given the choice.

davexunit 10y ago

>C is not the primary reason for our past vulnerabilities Completely false. C is a disaster.

renesd 10y ago

CPython also has many vulnerabilities in python rather than C. It's hilarious reading rust marketers talk about how people should use rust, and yet their software doesn't work as well. It has plenty of bugs. Then they go on and on about issues which post modern C doesn't have. Guess what? C has a lot of tooling, and yes, it's been improving over the years too. CQual++ exists. AFL exists. QuickCheck exists. Can your rust project from two years ago even compile? Does it have any users at all? There's a formally proven C compiler. How's that LLVM swamp going you've built your castle on? Rust brought a modern knife to a post modern gun fight -- and lost.

steveklabnik 10y ago
> Can your rust project from two years ago even compile? Post 1.0's release date, which is just short of two years, the vast, vast majority should, yes. We've had one or two soundness fixes in those times that would take a trivial amount of updating to do, but that only hit a very small part of the ecosystem.
sanxiyn 10y ago
> Can your Rust project from two years ago even compile? Eh, two years have not passed yet since Rust 1.0. But yes, it is likely that your Rust project today will compile just fine in 2019.
wtetzner 10y ago
> How's that LLVM swamp going you've built your castle on? http://www.cis.upenn.edu/~stevez/vellvm/ http://www.cis.upenn.edu/~stevez/vellvm/ > It has plenty of bugs. Rust doesn't claim to prevent all bugs. It only claims memory safety.

madphrodite 10y ago

This is a great little read and encapsulates the other side of the 'rethink the way' trend-ism of some HN new lang advocacy. C is fine, C is good. It is widely understood, it is a systems staple, and it is not dangerous in knowledgeable hands. Rocking the boat is fashionable.

macintux 10y ago
> it is not dangerous in knowledgeable hands For crying out loud, this is patently false. C isn't as dangerous if you really know what you're doing and you never make mistakes. I would bet fewer people know what they're doing than think they know what they're doing, and the set of people who never make mistakes is entirely empty.
- madphrodite 10y ago
  Well you know..that's just like your opinion man. Opting out of this site at this point. A bunch of 6-12 year idiots (recognizably) telling people what they think they know that they don't know. It's silly recursive.
  - macintux 10y ago
    You're right, my reply was unnecessarily harsh. I apologize.

geodel 10y ago

I think Rust community increasingly behave like this[1]. They are big on suggesting others the better 'ideas' instead of implementing themselves. So they keep using 'curl' and 'openssl' but tell others to rewrite their software with Rust. 1. http://dilbert.com/strip/1994-12-17 http://dilbert.com/strip/1994-12-17

lifthrasiir 10y ago
I would say a portion of Rust community---that said, it is that portion that is the most visible, and I think the community well understands what it does mean.
- sidlls 10y ago
  I'm not sure the community well understands it at all. The usual rejoinder is something along the lines of "well, I don't see that in the Rust community I'm in."
  - lifthrasiir 10y ago
    Both /r/rust [1] and the official forum [2] are reasonably regulated and I often see less informed members got warned about their use of aggressive or insulting languages, often directed to non-Rust languages. There are surely other venues with less enforcement (they still commonly observe the Rust Code of Conduct however), but at least I think the main venues and corresponding community heavily tries not to be offensive. [1] https://www.reddit.com/r/rust/ https://www.reddit.com/r/rust/ [2] https://users.rust-lang.org/ https://users.rust-lang.org/
Manishearth 10y ago
Like, a lot of the rust community is putting the effort into rewriting things in Rust. From within the community I don't see any coherent effort to tell folks to rewrite in Rust. We're very happy to see Rust rewrites, but not many people are pushing for it except the folks actually putting effort into it. Yes, every time a vulnerability pops up someone will say "rewrite it in Rust", but half the time it's not even a Rust programmer (often, Rust programmers come and disagree and say "Rust wouldn't fix this", IME).

jeffdavis 10y ago

"C is not a new dependency" To just use a library, rust isn't much of a dependency, either. It's designed so you don't even need to know that it's not C. Rust would obviously be a build dependency, but that's lessened somewhat because it tries to make cross-compilation easy. (But this point does apply to pretty much any other language. Curl would not be used as widely if it depended on the Go runtime, for instance.)

bitwize 10y ago

The maintainer of curl, a shitty HTTP client/library, takes a stand against the Rust Evangelism Strikeforce. The Rust Evangelism Strikeforce declares him a Suppressive Person.

bluejekyll 10y ago
I highly doubt anyone in the Rust community looks at it this way. As a huge Rust fan, I'm also a C fan. C existed before Rust; before Rust it was my favorite language, though I rarely chose it for work b/c of safety. The strongest argument in this piece is that Curl is written in the most portable language available. That's a great reason! And it's been a wildly successful project! Kudos to the developer(s)! The only problem with this piece is the claim on what are logic bugs vs. C bugs; but that should not detract from an excellent project. The real question being debated is this: if starting a new project like this, should you do it in a safe language to guard against security issues, or should you do it in the oldest most portable language ever. I'd argue that by the time any project becomes successful after being written in Rust, the LLVM (and Rust) will probably close that portability gap.
sctb 10y ago
I think we just asked you not to post like this, did we not? Please stop.

kazinator 10y ago

> The simple fact is that most of our past vulnerabilities happened because of logical mistakes in the code. Logical mistakes that aren’t really language bound and they would not be fixed simply by changing language. This statement is laughable nonsense. Shall we go into their bug history and point out counterexamples left and right? [Edit:user simias has done this; thanks!] Every single bug you ever make interacts with the language somehow. Even if you think some bug is nothing but pure, that logic is part of a program, embedded in the program's design, whose organization is driven by language.

devy 10y ago

The 7th point: "curl sits in the boat" In the curl project we’re deliberately conservative and we stick to old standards, to remain a viable and reliable library for everyone. Right now and for the foreseeable future. Things that worked in curl 15 years ago still work like that today. The same way. Users can rely on curl. We stick around. We don’t knee-jerk react to modern trends. We sit still in the boat. We don’t rock it. I see a lot of inertia in there. While it's a great record to maintain 15-year consistency but in the era of every changing InfoSec outlook, it could be a legacy and baggage if the authors resist to change. One thing we know for sure is that human will make mistakes, no matter how skillful you are. In the context of writing a fundamental piece of software with an unsafe programming language, that means we are guarantee to have memory-safety induced CVE bugs in curl in the future. Some of other points that the author raised are valid too. If there is a trade-off that we can have a safer piece of fundamental software by almost eliminating a whole category of memory safety related bugs, and with the downside of less compatibility with legacy systems, more dependencies etc., perhaps we should consider it? I believe the tradeoff is well worthy in the long run and option is ripe for explore.

cestith 10y ago
How is the author resistant to change? He specifically said new code should be written in a language that meets the priorities for that code. He specifically said someone has or would write a competitor to curl in Rust or some other safer language and that a good one will take off. He welcomed that. What he doesn't welcome is rewriting something that's had those bugs and the types of logic bugs not related to the language already worked out. There's a saying about a baby and bathwater. Not everything is a dichotomy, and you shouldn't be reading the article as if the author is against newer languages. He specifically says that given a fresh start with the availability of these languages he might use something besides C. Carefully weighing options is wise. Throwing away years of actual progress for the appearance of quick progress is foolish.
- devy 10y ago
  > How is the author resistant to change? I specially quoted the section head "curl sits in the boat" and the entire section ends with "We sit still in the boat. We don’t rock it". Now read it again, and then tell me if that's welcoming changes or resisting changes. > He specifically said someone has or would write a competitor to curl in Rust or some other safer language and that a good one will take off. He welcomed that. Sure there might already be some alternatives out there. But those are not curl, they are at most forks. > He specifically says that given a fresh start with the availability of these languages he might use something besides C. Nope, he used the word "Maybe. Maybe not." Might is a stronger word.

krystiangw 10y ago

C has still huge market share. Seems that it occurs in 5% of all tech job offers: https://jobsquery.it/stats/data.technologies/C https://jobsquery.it/stats/data.technologies/C Stats also showing that average salary for C developers is above average for all tech job openings.

coding123 10y ago

I don't see why this is an issue, whoever is arguing for a change can write rurl and be done, and see if anyone takes it up in their distributions.

dmitrygr 10y ago

Software is almost a perfectly open market. If proponents of rust really think their preferred language is better in every way, they are free to rewrite the world in rust, and see the adoption numbers they get. After all, if rust is better in every way, we'd expect the adoption numbers to go up for their rust OS, with a rust http stack and rust web browser. Right? Telling others to use their language instead of putting their money where their mouth is is truly what irks me about the rust community the most. Want a rust world? Go write it and ship it. Oh, and you don't get to complain about C until your PC runs more rust than C Cool?

steveklabnik 10y ago
I don't think the majority of Rust users disagree.
arcticbull 10y ago
I'm not sure why I can't complain about problems I have today -- if nobody complained about C ever, I doubt Rust, Go or for that matter, basically any other language project, would have ever gotten started. It's okay to identify flaws in tools, that's how we make them better. It doesn't make sense to say '[car on fire] you can't complain about that Honda Civic until you develop your own better car, sir, and more people are driving it than not, now please leave the service center -- until then consider the fire normal'.
deleted 10y ago
[deleted]

tlrobinson 10y ago

I'm curious, of the bugs that could have been avoided by using a "safe" language, how many could have been avoided by using a bounds checking extension like as https://www.doc.ic.ac.uk/~phjk/BoundsChecking.html https://www.doc.ic.ac.uk/~phjk/BoundsChecking.html or https://github.com/Microsoft/checkedc https://github.com/Microsoft/checkedc Are such extensions popular, and if not, why not? I assume there's always some performance hit, but that might not be a big deal in an HTTP client, for example.

pjmlp 10y ago
Because they aren't portable.

koja86 10y ago

Quite recently I happily used libcurl for C++ project rather than any of those C++ wrappers found at github. Granted there is some non-elegance when you adapt C-style error codes to C++ exceptions and non-C++-idiomatic code style right next to any C lib. Yet libcurl is battle tested (AKA proved to be rather bug free) and has nice clean API unlike. IMHO it might eventually make sense to use other language/tech/whatever but the bar is quite high and it will quite probably take some serious sustained effort.

tombert 10y ago

While I'm definitely not suggesting we replace Curl with a rewrite in Rust (since the current Curl has had decades of good testing and auditing done on it), I am actually very curious how a rewrite in a safer language like Rust, OCaml, Haskell, or Go would fair in comparison in regards to performance and whatnot. If I were ambitious enough, I'd do it myself in Haskell, but I think it'd be too much work for a simpler curiosity.

notriddle 10y ago
There's hyper, if you want something that exists now, but it only does HTTP.
fiedzia 10y ago
Most likely I/O will take more time than whatever code is running, so in that aspect it would make no difference. Memory overhead is main concern here. Rust doesn't use GC, so you'll have full control and there should be not much difference in that aspect. Other languages do, which means more sophisticated runtime, less control and more overhead (or writing ugly code to avoid it). Libcurl written in Go/Ocaml/Haskell would require anyone using it to also include runtime of the language, which is usually rather large.

didip 10y ago

It's well within reason and capabilities for rust community to write libcurl and curl CLI libraries. The community should do it, spend a couple of years stabilizing, and then spread the words to others.

deleted 10y ago
[deleted]

_of 10y ago

Rust might be a great language, but it has not completed the test of time yet. C is 45 years old. Rust appeared 7 years ago.

carapace 10y ago

This has probably been said, in this thread even, but if curl is insecure (for some value of "insecure") then its ubiquity and ease of embedding are a problem rather than a feature. Fuzzy thinking.

AstralStorm 10y ago

Maybe they should attempt writing it using the Isabelle/HOL transpilers to C from SEL4 project. I don't care if it is C or machine code as long as the proof of correctness is complete, down to at least C library. Curl is small enough to make it relatively easy and used widely enough to make it worthwhile.

adynatos 10y ago

While C by itself is not safe, I would argue that no sane development environment uses C by itself. Over the decades of its production use dozens of tools have been developed that make it far safer: *grind suite, coverage tools, sanitizers, static analyzers, code formatters and so on. Those tools are external, otherwise they would make C slower. Something for something.

mgrennan 10y ago

Why does something old (C) have to be bad these days?

wtetzner 10y ago
It's not bad because it's old, it's bad because of how unsafe it is. There are older languages than C that are safer, but they didn't gain the popularity C did.

chousuke 10y ago

In my view, the problem with C in general is that it's a loaded gun with no safety or trigger guard. It's trivial to shoot yourself (or someone else) in the foot, and it requires knowledge, meticulous care and lots of forethought to avoid getting shot. I very much agree that rewriting existing, stable software written in C is likely not worth the trouble in many cases, but I can't accept claims that the limitations of C aren't the direct cause of tens of thousands of security vulnerabilities, either. In Rust, even a less experienced developer can fearlessly perform changes in complicated code because the language helps make sure your code is correct in ways that C does not. And you can always turn off the safeties when you need to. Experienced developers should feel all the more empowered by simply not having to always worry about things like accidental concurrent access, use-after-free, object ownership, null pointers or the myriad other trivial ways to cause your program to fail that are impossible in safe Rust. You get to worry about the non-trivial failure modes instead, which is much more productive.

kodest 10y ago

Maybe curl could be rewritten in C++ step by step like mpd (https://musicpd.org https://musicpd.org). C++ has RAII for resource management which can help a lot by itself. In my opinion the most hateful thing in C is freeing resources on all exit paths. Although, curl in C++ - the naming would became inappropriate...

derefr 10y ago

> A library in another language will add that language (and compiler, and debugger and whatever dependencies a libcurl written in that language would need) as a new dependency to a large amount of projects that are themselves written in C or C++ today. Those projects would in many cases downright ignore and reject projects written in “an alternative language”. Why would I be vendoring my own copy of libcurl in my project? Who does? This is how I (or rather, the FFI bindings my language's runtime uses) consume libcurl: dlopen("libcurl.so") I rely on a binary libcurl package. The binary shared-object file in that package needed a toolchain to build it, but I don't need said toolchain to consume it. That would still be true even if the toolchain required for compiling was C++ or Rust or Go or whatever instead of C, because either the languages themselves, or the projects, ensure that the shared-object files they ship export a C-compatible ABI. An example of a project that works the way I'm talking about: LLVM. LLVM is written in C++, but exports C symbols, and therefore "looks like" C to any FFI logic that cares about such things. LLVM is a rather heavyweight thing to compile, but I can use it just fine in my own code without even having a C++ compiler on my machine. (And an example of a project that doesn't work this way: QT. QT has no C-compatible ABI, so even though it's nominally extremely portable, many projects can't or won't link QT. QT fits the author's argument a lot better than an alternate-language libcurl would.)

lettergram 10y ago

I feel blaming a language for errors is like blaming a gun for killing people. The fact is, mistakes will happen, but in general if you follow the best practices you'll be fine. Failing to follow the best practices means you could be a better programmer. Just because the language gives you an option to do something, doesn't mean you should.

myrrlyn 10y ago
"I don't need a safety toggle on my gun I just totes keep it pointed at the ground and unloa-- oh god where's my foot"
- trav4225 10y ago
  Still not the gun's fault.
  - roca 10y ago
    In some sense, sure. But in practice, people always make mistakes. Some guns/programming languages limit the damage of those mistakes more than others. All other things being equal, those guns/programming languages are better.
    - lettergram 10y ago
      I'd agree with you, but all things are not equal.
    - trav4225 10y ago
      Yup, no argument from me there. There's always room for improvement...
roca 10y ago
> mistakes will happen, but in general if you follow the best practices you'll be fine Everyone fails to follow best practices at times, precisely because "mistakes will happen!" Encouraging people to be better programmers can't change that. Therefore we want languages that catch as many mistakes as possible at build time and minimize the damage caused by other mistakes that slip through.

throwaway5752 10y ago

It's extremely simple. If you think Curl would be better in another language then port it, release your alternative, and maintain it for a long time. Even if your language (Rust, Erlang, LISP, Go) is "better", it's still a minimal part of the equation. A maintainer is what makes the tool. It's hard work to decide which PRs to accept (and worse yet, reject), to backport fixes to platforms for which you can't get a reliable contributor, coordinating fundraising/donations, keeping up with evolving standards... Anyway. Thank you, thank you, thank you Daniel Stenberg. Use whatever damn language you want.

kazinator 10y ago
> Use whatever damn language you want. On the other hand, if he didn't want his justifications for that choice examined by the world, he wouldn't have aired them, right? > If you think Curl would be better in another language then port it, release your alternative, and maintain it for a long time. That's out there; some languages have URL downloading objects that are not based on Curl. E.g. Edi Weitz's Drakma client library for Common Lisp doesn't seem to be using Curl as far as I can see. http://weitz.de/drakma/ http://weitz.de/drakma/
- throwaway5752 10y ago
  I wouldn't presume to speak for Daniel, but I got the feeling that he just wanted to publish this to point people to rather than send the same canned response to inquiries about porting to Rust et al. Drakma sounds great. Tone is tough to get right online. I don't like people doing drive-by suggestions like "you should rewrite X in Y". But if people are really willing to roll up their sleeves, write the tool (in any language) and keep it going for the long haul, I applaud them. I just have great respect and empathy for project maintainers, I think some don't appreciate what a huge PITA it is to BDFL a successful project.

digi_owl 10y ago

A refreshing read in what seems to be a ongoing deluge of rewrites, languages and frameworks.

trav4225 10y ago

It is utterly amazing to me to see so many people's attitudes on this issue. If I cut myself by hasty use of a knife, is it the fault of the knife maker? How is that even remotely rational? If you aren't willing (or don't know how) to use the tool correctly, don't use it.

gyrgtyn 10y ago

What is everyone using curl for that it needs to be written in C (or Rust?). If it think about my usage, it's like get or post something and see what the returned json looks like. If I need to download something wget usually works without having to remember -O. But higher level things like httpie are easier to deal with, sane defaults and all that. Maybe they use libcurl... Are there any re-write userland in ${safe-high-level-lang} projects?

tete 10y ago

I think it's a bit weird that C and curl are used. If we look at C and OpenBSD or so things might look a bit different. Also one has a hard time comparing curl with another language, simply because something with curl's properties (take portability for example) doesn't exist. And no that isn't in defense of anything, just me thinking thinking that measurable points brought up in the discussions don't make sense or exist. The topic is also a bit broader, as you can easily add in static code analysis, compiler flags, stuff like W^C, stuff like seccomp, capsicum, cloudabi, pledge which might not work (well) in other cases. It's a great philosophical discussion topics and I don't wanna stop anyone, just hoping people keep that in mind, when they participate, so we don't end up with new dogmas that get thrown around for the next few year, without knowing contexts or meaning of phrases. Other than that: I really enjoy this discussion. :)