15 ms·

Addendum to “Curl is C”

197 points by mirceasoaica 10y ago

mbrock 10y ago
The phrase "worse is better" was coined precisely to explain the success of C, and I consider that paper required reading for entering this discussion.
- shabbyrobe 10y ago
  For those curious, I am guessing this is the link: https://www.dreamsongs.com/RiseOfWorseIsBetter.html https://www.dreamsongs.com/RiseOfWorseIsBetter.html
- milesrout 10y ago
  "Worse is better" has nothing to do with it. Stop saying it every time anything you think is bad comes up. It's incredibly arrogant, the way people basically spam "worse is better" like they're in Twitch chat every time C, Unix, HTTP, or anything else that they perceive as imperfect comes up.
  - mbrock 10y ago
    But I do consider that article a good explanation of the tremendous success of C and Unix, and I think its perspective should be part of these discussions.
TorKlingberg 10y ago
Everyone will just have to accept that Daniel is not going to rewrite curl in an other language. If you want curl in Rust, a Rust programmer will have to step up and do it.
- agumonkey 10y ago
  Let's call it Crust
sapphire_tomb 10y ago
I wish people would get off Daniel's case about this. If you want to re-implement curl in rust or whatever, no-one's stopping you!
AndyKelley 10y ago
I wouldn't advocate for the curl project to experiment with rewriting some code in [Zig](http://ziglang.org/ http://ziglang.org/) yet, due to lack of maturity, but it is exactly in the target niche of the project. That is, near-seamless integration with a large pre-existing C codebase. So you could, for example, take a 400 line C file, convert it to Zig, and have it import the .h files that the previous C file was including, and export an .h file for the rest of the codebase to look at. The compiler can turn this Zig file a .o file that integrates with the rest of the build. The part of the codebase written in Zig could benefit from the safety features, generics, and error handling features of the language while presenting the same interface to the rest of the codebase. Now of course, if you already have a codebase written in 1 language, it's more complicated to introduce another. This is where the Zig build system comes into play. I'm working on this now, and it's a competitor to autotools, scons, cmake, make, and all those players. It's the Zig programming language with a declarative build system API built into the standard library with some basic tooling around it, and it ships with the compiler. So if the Zig build system gets to the point where it is high quality enough to attract users and ubiquity in package managers, then a project which already uses Zig for its build system can start using it for its actual code without the penalty of adding a dependency. In conclusion, once Zig has reached a stable enough state, I would challenge the libcurl devs to consider, perhaps it could make sense to slowly, carefully, methodically transition to a newer language, if that language was friendly enough to the process.
- stirner 10y ago
  What does Zig have to offer over Rust's FFI[1]? Also, it might be useful to know that HN's comment syntax isn't Markdown, though it shares a few features like indenting for code blocks and asterisks for italics. Markdown-style links are not supported. [1] https://doc.rust-lang.org/book/ffi.html https://doc.rust-lang.org/book/ffi.html
  - icebraining 10y ago
    As an addendum, using a Rust library in C is quite easy: https://doc.rust-lang.org/1.2.0/book/rust-inside-other-languages.html https://doc.rust-lang.org/1.2.0/book/rust-inside-other-langu...
najati83 10y ago
The reason curl is everywhere is that it's written in C. So obviously the author is not going to rewrite it in any other language.
deleted 10y ago
[deleted]
spoiler 10y ago
> These “you should switch language” remarks are strangely enough from the backseat drivers of the Internet. Those who can tell us with confidence how to run our project but who don’t actually show us any code. This is an usual problem with people participating in discussion over the Internet. I am sure those people mean well, but often they don't put enough effort into assessing the situation and underestimate its complexity before they decide to hit the keyboard. I've myself done it a number of times, but I am working on not doing it. Advice for the people who are making all these suggestions and: A good way to see if you're talking out of your ass—unintentionally, of course—is to try to do what you're saying someone should do. Just try and re-write some parts of curl in Rust (you can go with a hip name like rurl). It's a good exercise to learn about the project you're rewriting, the language you're writing it in, and the problem domain. So, at the very least you'll get something out of it, and if your project is successful: even better!
- WalterBright 10y ago
  D is compatible enough with C/C++ that I have been able to take largish multifile projects and convert them one file at a time to D. The full regression suite was run after each file was converted. This helped ensure both an accurate conversion and did not leave the project in a broken state for an extended period.
  - WalterBright 10y ago
    I've successfully converted many projects from one language to another using this method. One was a huge assembler program I converted to C back in the 1980s (ironically, so it could be ported to other platforms!).
- reitanqild 10y ago
  Were do I send proposals for "comment of the month"? More seriously speaking: IMO this is solid advice. It - is actionable - goes out of its way to be polite and respectful - has a chance to actually improve online discussions one comment (and hopefully commit) at a time
jasode 10y ago
>I think that’s missing the point I was trying to make. I think Daniel Stenberg missed the point of some of the comments. (At least the prominent ones I saw.) >We use C for a whole range of reasons as I tried to lay out there in spite of the security challenges the language brings. The availability and practicality of old ecosystems are valid justifications but it's orthogonal to the "memory bugs caused by the language vs the human". There's nuance and you have to distinguish multiple conversations: 1) curl should have been written in a memory safe language (past tense) 2) curl should be rewritten in a memory safe language (future tense -- e.g. maybe use Rust) 3) setting aside curl's 19 years of wide deployment challenges, D.S. could have been a better spokesman for the benefits memory safe languages Daniel's response is mostly to #1. However, some of the criticism is really conversation #3. Understandably, #3 is somewhat harder for D.S. to adopt because it requires distancing himself from the 19 years of curl as a successful utility and instead, consider how a language's design affect programmers' bug count. Many felt that D.S. citing curl's bugs as "human logic errors" instead of "language-induced errors" undermines his point when conversation #3 is the framework. So maybe some of the new nitpicking of counting CVE bugs is more about conversations #2 & #3. The #2 may not be realistic for another 10 years... or maybe never because of various deployment hurdles. However, we can still discuss #3.
- hsivonen 10y ago
  I think this is the disconnect. One can understand why curl is written in C, not expect Daniel to write it in another language, appreciate that the overall vulnerability count being impressively low considering how old the project is, and still worry about the section about vulnerabilities in the original post looking dismissive when it comes to vulnerabilities that are attributable to C and perhaps getting trotted out in the future by others contemplating new projects as an excuse not to use a safe language.
  - empath75 10y ago
    Yeah, I absolutely think that curl should stay a C project until the day it dies. I also think someone should write a curl-alternative in rust. And at some point, people may start using the Rust version instead of the c version.
nallerooth 10y ago
I wonder how much experience people yelling "OMG you're using C(++) - you have to rewrite everything!" actually have. Some of them can motivate their claims with good examples (at least to some extent) while others just show a serious lack of understanding of project scope and/or the reason something is written the way it is. One of my favorite examples so far: https://www.postgresql.org/message-id/CAASwCXdQUiuUnhycdRvrUmHuzk5PsaGxr54U4t34teQjcjb%3DAQ%40mail.gmail.com https://www.postgresql.org/message-id/CAASwCXdQUiuUnhycdRvrU...
- buserror 10y ago
  Ahah nice example! "I can't code, but if I were, I would like to be productive in something that doesn't require me to make any effort in understanding the problem, please advise" I don't know where all the C hate is all about. I love C, it's a great tool. Can be quite sharp, be careful with your remaining fingers. I spent the better part of last week studying rust and go, and my conclusion was 'meh, I'll have another look in another few years" and kept on trucking.
  - nallerooth 10y ago
    I don't get all the C "hate" either. Sure, C is a source of many "interesting" things - but it's also a natural part of control systems in avionics, space craft and nuclear power plants. I don't want someone who just read that "Rust is a great, safe language" to rewrite those. I really don't.
    - fiedzia 10y ago
      And I do. Planes crashed and people died because of buggy software. We have lost Mars climate orbiter due to not using type-safe language. In the cases of software that works fine, we are just relying on ridiculous amount of manual verification, and this is both risky and expensive.
      - mikulas_florek 10y ago
        Mars Climate Orbiter was lost because they forgot to convert units between different systems. That's an error that can (and probably would) be done in all type-safe languages I know of.
DCKing 10y ago
There are many reasons why curl uses C, and should continue to do so. Yes, C allows a combination of portability, performance, and accessibility of your source code by others that is not matched by any language. And yes, C was a perfectly valid choice throughout the vast majority of its life and still is. Switching is not a realistic option at all. But I think Daniel's original attempt at explaining away C's security issues has been debunked. And I think he owns it well by singing a slightly different tune in this blogpost. The lesson learnt here is that if you use C, don't try to explain away the security issues you have just opened for yourself. It will not hold up. Accept that the safety of C is a very serious issue, and that there are other reasons you choose it. Choosing C over more secure options does not mean you don't care about security, but it means that you had to make pragmatic choices. Choosing C over more secure options and subsequently attempt to minimize the security impact of this decision (not talking about Daniel here) will lead you to not addressing those issues effectively.
- fiedzia 10y ago
  Well said.
- generic_user 10y ago
  > Choosing C over more secure options and subsequently attempt to minimize the security impact of this decision (not talking about Daniel here) will lead you to not addressing those issues effectively. When you start a new C/C++ project in 2017 you can effectively and confidently address security and 'safety'. And it is done on a regular basis. There is an entire infrastructure of tools and introspection refined over the past 30+ years for C/C++ that is part of the project from the beginning. Address sanitizes, Better compilers, Valgrind, Intel's tools etc. All of the potential problems are known and there are tools to deal with them. But the responsibility is put in the hands of the programmer like many other things in C/C++.
  - pjmlp 10y ago
    CppCon 2015 as Herb Sutter asked the audience how many use regularly any form of static analysis tools, the answer was around 1% of the audience. The video is easy to find if you want to watch it. So apparently the tools might be there, but they are ignored by those that should be using them in first place.
nathan_f77 10y ago
Are there any automated tools that you can help you convert a C codebase into Rust? And is there any reason why Rust couldn't be as portable as the C code that makes up curl? Is it very difficult to transpile C into Rust? As an aside, I not like the curl style guidelines [1]. Especially "'else' on the following line", and "No space before parentheses". That doesn't look nice at all. [1] https://curl.haxx.se/dev/code-style.html https://curl.haxx.se/dev/code-style.html
- chrismorgan 10y ago
  https://github.com/jameysharp/corrode https://github.com/jameysharp/corrode is a C-to-Rust translator. It won’t be idiomatic Rust, though; not by a long shot. (Rust is much more restrictive in what it allows, for reasons of safety, though more expressive.) And it may not be correct, either; corrode is getting pretty good, but it’s not perfect yet.
- deleted 10y ago
  [deleted]
- thristian 10y ago
  For automated tools that can convert a C codebase to Rust, you're looking for is Corrode: https://github.com/jameysharp/corrode https://github.com/jameysharp/corrode Last I heard, Corrode can convert many C syntactic structures into their Rust equivalents, they usually compile and sometimes they even work. Even when the conversion works flawlessly, the resulting Rust does exactly what the original C code did—there's no safety benefit (but you can begin refactoring to clean things up).
- mrkgnao 10y ago
  https://github.com/jameysharp/corrode https://github.com/jameysharp/corrode
- fiedzia 10y ago
  And is there any reason why Rust couldn't be as portable as the C code that makes up curl? In theory no, in practice yes. Rust is using LLVM project to generate machine code, and that only supports certain architectures. There are however other safe languages that can generate C, if this is a requirement. > Is it very difficult to transpile C into Rust? If safety guarantees could be added to C by some conversion or analysis, there would be no need for Rust to exist. Rust (and other languages with similar goals) is safer for many reasons, not limited to languages itself. Stdlib and runtime play their part too. Automatic conversion that takes all that into account is either impossible or impractical. There is a tool for converting C into unsafe Rust (corrode), which might be useful for people attempting to port C code, but in itself doesn't improve code safety at all.
fiedzia 10y ago
Note: I am discussing using safe languages in general and reasoning for doing or not doing that. If you believe rewrite requests are unreasonable no matter what, don't bother reading that. Or perhaps consider this to be addressed to people deciding how to choose libraries to depend on, not to specific library authors. > curl is currently one of the most distributed and most widely used software components in the universe, be it open or proprietary and there are easily way over three billion instances of it running in appliances, servers, computers and devices across the globe. Right now. In your phone. In your car. In your TV. In your computer. Congratulations. So how many of those have unpatched vulnerabilities right now that were caused by using unsafe language? In my phone? In my computer? In my car? How many accidents of data or money theft it caused? Not to mention common non-security related bugs. > If we then have had 40, 50 or even 60 security problems because of us using C, through-out our 19 years of history, it really isn’t a whole lot given the scale and time we’re talking about here. Curl had multiple unpatched security problems for several years before they were discovered. As the author admits, it most likely still has many. dpkg -l | grep '^ii lib'|wc -l lists over 2000 libraries I have installed, lot of them written in C. An estimate of 2000 multiplied by 60 is ... scarily to many. Large amount of them avoidable by using safer languages. > Using another language would’ve caused at least some problems due to that language If it solves more problems than it creates, it is worth doing. > none of the memory safe languages anyone would suggest we should switch to have been around for 19 years. Neither have been mobile phones (at least as we know them), but they are useful, so I am using one. > We will continue to work hard on minimizing risks, detecting problems early by ourselves and work closely together with everyone who reports suspected problems to us. I prefer to work smart, not hard.
- rkangel 10y ago
  > dpkg -l | grep '^ii lib'|wc -l lists over 2000 libraries I have installed, lot of them written in C That '2000' number is not the relevant one here. The reason that security is important wrt Curl is that Curl is used to access the internet. Security in the library used to print colours to your terminal is a bit less important. It's those widely used, public facing libraries that need a good security story (and using a memory-safe language is a good start there).
simias 10y ago
I think most of the points he made in his original post are very valid ones, it's just the fatalist "devs are going to create bugs, such is life" approach to errors that I think I glimpsed in the original post that prompted my response in the previous thread. I'm tempted to call it the "C Stockholm syndrome". The reason I say that is that I could actually have written something like that a few years ago myself. I think it's a pretty common mindset amongst the "low level" C crowd. And there's a good reason for that, we've been burned before. People have been advertising "safe" alternatives to C since forever. Use Perl! Use Java! Use Lisp! Use Haskel! Use Python! Use Go! But none of these languages can replace C in all of its use cases. You have big runtimes, garbage collection, interpreters, you name it. You can't just bootstrap a stack pointer in 3 lines of ASM and call into them like you can with C. And even if we don't say it out loud there's a bit of snobbism to writing low level "unsafe" code. "Safe languages are for noobs and webdevs, real coders debug with oscilloscopes". And so I had this mindset that you couldn't have your cake and eat it too: you could be safe or you could be low level. Segfaults and NULL pointers were just a normal part of dealing with low level "unmanaged" environments and we'd just have to deal with it. People who complain about it are just not worthy. In my opinion the biggest success of Rust so far is in proving that you could be safe without needing a GC, a big runtime or complex abstractions. You could design a better C without having to compromise. Even if Rust fails in the long run, it would still be a very important "proof of concept". It showed that C was unnecessarily unsafe, that you could achieve the same feature set without throwing safety completely out of the window. That being said if there really are folks bugging devs by asking them to consider switching to language X or Y then I think it's fair to say that these people are idiots. Might as well ask JK Rowling to rewrite Harry Potter in Esperanto because "it's totally the future!". I mean, let's face it, at this point Curl probably has a much bigger userbase than Rust itself by at least an order of magnitude. Which one is more likely to be irrelevant in 10 years? Rust or Curl (and by extension, C)? If you answered the latter you should read less hacker news. And even if Rust ends up being successful it's not like C will go away any time soon, or even in our lifetimes. Curl (as a C library) will probably still be useful and relevant for decades to come no matter what.
- fiedzia 10y ago
  https://www.reddit.com/r/languagelearning/comments/1yv1i2/harry_potter_book_one_in_esperanto/ https://www.reddit.com/r/languagelearning/comments/1yv1i2/ha...
wojt_eu 10y ago
Curl should be split into micro-services.
legulere 10y ago
He writes "way over three billion instances", I read "way over three billion vulnerable instances" which is a security disaster.
spion 10y ago
I don't think the reactions and comments were meant as an attack on curl, but more as an attacks on C. Most programmers who attack C have seen the consequences of a memory-unsafe, type-unsafe language dominating systems programming throughout the years. I remember the period during the early 2000s when "buffer overflow exploits" were an entire industry. I also remember the strange allure of C and C++ in beginner programmer circles. The logic goes something like this: "they are powerful languages, all of the big and popular software projects are written in them, so they must be the best". Their next step was to go on to start a new generation of unsafe and vulnerable programs and libraries, and so on. Now that we have the tools and technology to do better, we should really be making some efforts to push for them. The goal isn't change right now, but starting the process early seems like a good idea.
- wott 10y ago
  Most programmers who attack C at the moment are frontend web javascripter kids and rust/functional language snobs, who both never used C and keep repeating the same stuff. You just have to witness the number of "OMG, you did that in C? Wow!" that are posted every time someone posts a basic C program on a forum. There is a whole crowd that thinks C is some arcane stuff only to be used by by magicians, and if you're not a guru it's gonna kill you. And then you have the Rust people who chime in and point a list of C problems that are in fact C++ problems. And then you get the "but anyway modern processors do not execute the code you tell them to execute" guys. Sigh... It goes every time the same way...
  - biskit10 10y ago
    wott
  - eeeficus 10y ago
    well it's 2017. maybe it's time we get rid of C? :)
    - endisukaj 10y ago
      Lisp is even older than C, should we get rid of that too?
spion 10y ago
I don't think the reactions and comments were meant as an attack at curl, but more as an attacks at C. Most programmers who attack C have seen the consequences of a memory-unsafe, type-unsafe language dominating systems programming throughout the years. I remember the early 2000, and the period where "buffer overflow exploits" were an entire industry. I also remember the strange allure of C and C++ in beginner programmer circles. The logic goes something like this: "they are powerful languages, all serious software is using them, so they must be the best". The next step is to go on to start a new generation of unsafe and vulnerable programs and libraries, and so on. Now that we have the tools and technology to do better, we should really be making some efforts to push them. The goal isn't change right now, but starting the process early seems like a good idea.
kazinator 10y ago
The best feature of Curl is that you can use it as a child process, driving it through the command line interface of its principal executable. Quite a lot of the API-level bells and whistles are covered. A few years ago I looked at the API and internals and went, "gack!" No way am I integrating this dung heap at that level; let the users just wrap the process with a curl function that returns a process pipe stream if they want to use this.
jeffdavis 10y ago
The original post said: "C is not the primary reason for our past vulnerabilities" So it's entirely reasonable for critics of C to challenge that claim.
zeveb 10y ago
> curl is currently one of the most distributed and most widely used software components in the universe, be it open or proprietary and there are easily way over three billion instances of it running in appliances, servers, computers and devices across the globe. Right now. In your phone. In your car. In your TV. In your computer. Etc. And every one of those three billion devices is vulnerable, due to the use of C. > I feel a need to underscore the fact that none of the memory safe languages anyone would suggest we should switch to have been around for 19 years False: Common Lisp has been around since 1994 (23 years), and in substantially the same shape for longer still. Standard ML has been around since 1990 (27 years). OCaml has been around since 1996 (21 years). Smalltalk has been around since 1984 (33 years). Each of those languages is more memory-safe than C and has facilities which help prevent other C-like errors. Each is capable of speeds approaching that of C, esp. for a problem like URl fetching (e.g. I just tried fetching http://www.google.com/ http://www.google.com/ with both curl & DRAKMA — a Common Lisp package for URL fetching: curl reliably ran in about .065 seconds & DRAKMA reliably ran in about .14 seconds; I have no reason to believe that DRAKMA is particularly well-optimised; no doubt it could get even faster if desired). I think this really is a textbook example of the Blub Paradox: someone using C thinks that for the most part it's a reasonable choice in order to achieve certain goals, while someone used to a better language is able to see that C is simply unfit for purpose: programs written in it will inevitably have security flaws which will inevitably cause harm — particularly when three billion devices, many unpatched, are running them.
- snakeanus 10y ago
  > And every one of those three billion devices is vulnerable, due to the use of C. Due to the use of unsafe C implementations. You would have the same problems if you used an unsafe CL or rust implementation.
- kazinator 10y ago
  We should stop using Pythagoras' a^2 = b^2 + c^2. The last commit to that was like 2500 years ago. (Core engine only, not counting some syntactic sugar).
- pjmlp 10y ago
  > I think this really is a textbook example of the Blub Paradox:.... It was already like that in the 90's. When I learned C, I was using Turbo Pascal 5.5 and getting up to speed with newly released Turbo Pascal 6.0. Besides the improved portability over Pascal dialects, I didn't saw any compelling feature over what Turbo Pascal was capable of.
kbenson 10y ago
curl is currently one of the most distributed and most widely used software components in the universe, be it open or proprietary and there are easily way over three billion instances of it running in appliances, servers, computers and devices across the globe. Right now. In your phone. In your car. In your TV. In your computer. Etc. If we then have had 40, 50 or even 60 security problems because of us using C, through-out our 19 years of history, it really isn’t a whole lot given the scale and time we’re talking about here. I think there's a disconnect here. Given something that's has a thousand instances running and averages 20-30 security problems a year and something else that has over three billion instances running and averages 2-3 security problems a year, which one do you think I care more about? I care about the one running everywhere, and on devices which may never see updated code in the rest of their lifetime. If curl were some unused project this wouldn't be a conversation. At the point where your code is on all those devices, you will get scrutiny. Expect it. Your choices may have some impact on a significant portion of the Earth's current population. At the same time, it's not fair to expect curl to rewrite their project in some other language. Some other project can start with the goal of providing the same API as curl (or not) that attempts to mitigate security problems though a different code architecture or language, and maybe curl devs will contribute. But expecting curl devs to start over from scratch is not feasible.
jeffdavis 10y ago
The problem is that the author listed specific reasons, which opened him up to specific challenges. He should have just said: "curl is an established project. New languages are not being seriously considered for curl." If you want to close down a conversation, it's counterproductive to make new dubious claims like "C is not the primary reason for our past vulnerabilities". Note that closing down a conversation isn't bad. Sometimes they are annoying or distracting. I love rust but here was my comment when someone wanted to rewrite postgres in rust: https://news.ycombinator.com/item?id=13354768 https://news.ycombinator.com/item?id=13354768
hzhou321 10y ago
Not to rewrite is a very sensible decision due to the fact that the current way of programming is un-rewritable. For example, people don't write FORTRAN in every language. The choice of programming language often fundamentally change the way we approach programming. Switching language not only involves addition or reduction of language constructs, it fundamentally alters programmer's highest level thinking process. That is not a merit. What we need is a meta-layer programming interface that isolates our higher thinking from lower language concerns to some degree. If you can keep the higher level understanding, then rewriting become less of a barrier. We need write pseudocode in every language. I have been exclusively programming with a meta layer, MyDef, for quite a while, and I routinely re-write my Perl program to C, not without effort, but always a worthwhile effort trading for the goal of rewriting (speed in this case). I am a researcher, and I rarely write production code. But for production code, I think most software should consider rewriting in Rust. Prototype in Perl/Python, develop in C, and produce in Rust -- makes good sense. It makes good sense in our current non-software industry, where researchers, developers, and producers are people with different focus and use different material and tools. This is not true for our software industry, where people who are not interested in researching or developing are forced to research and develop, and people who are only interested in research are forced to develop and people who only interested in developing are forced to research and produce. My example: http://hz2.org/blog/send_more_money.html http://hz2.org/blog/send_more_money.html
mirekrusin 10y ago
As a meta-comment - it's interesting to see how religiously-intense this discussion has became. Why are we - programmers - need to hold such a strong opinions about languages? Especially considering a fact that the playground is completely open. My personal opinion is that if he was wrong - we wouldn't even be reading his argument - it wouldn't make it to the top of HN - everybody would be using Xurl library instead (written in X-lang, where X is not C). At the same time Rust is shaping to be fantastic language and I personally hope that soon we'll see Rurl cargo together with kernel and what not in Rust. Unless in this decade something better will appear, who knows?

197 points by mirceasoaica 10y ago

The phrase "worse is better" was coined precisely to explain the success of C, and I consider that paper required reading for entering this discussion.

shabbyrobe 10y ago
For those curious, I am guessing this is the link: https://www.dreamsongs.com/RiseOfWorseIsBetter.html https://www.dreamsongs.com/RiseOfWorseIsBetter.html
milesrout 10y ago
"Worse is better" has nothing to do with it. Stop saying it every time anything you think is bad comes up. It's incredibly arrogant, the way people basically spam "worse is better" like they're in Twitch chat every time C, Unix, HTTP, or anything else that they perceive as imperfect comes up.
- mbrock 10y ago
  But I do consider that article a good explanation of the tremendous success of C and Unix, and I think its perspective should be part of these discussions.

TorKlingberg 10y ago

Everyone will just have to accept that Daniel is not going to rewrite curl in an other language. If you want curl in Rust, a Rust programmer will have to step up and do it.

agumonkey 10y ago
Let's call it Crust

sapphire_tomb 10y ago

I wish people would get off Daniel's case about this. If you want to re-implement curl in rust or whatever, no-one's stopping you!

AndyKelley 10y ago

I wouldn't advocate for the curl project to experiment with rewriting some code in [Zig](http://ziglang.org/ http://ziglang.org/) yet, due to lack of maturity, but it is exactly in the target niche of the project. That is, near-seamless integration with a large pre-existing C codebase. So you could, for example, take a 400 line C file, convert it to Zig, and have it import the .h files that the previous C file was including, and export an .h file for the rest of the codebase to look at. The compiler can turn this Zig file a .o file that integrates with the rest of the build. The part of the codebase written in Zig could benefit from the safety features, generics, and error handling features of the language while presenting the same interface to the rest of the codebase. Now of course, if you already have a codebase written in 1 language, it's more complicated to introduce another. This is where the Zig build system comes into play. I'm working on this now, and it's a competitor to autotools, scons, cmake, make, and all those players. It's the Zig programming language with a declarative build system API built into the standard library with some basic tooling around it, and it ships with the compiler. So if the Zig build system gets to the point where it is high quality enough to attract users and ubiquity in package managers, then a project which already uses Zig for its build system can start using it for its actual code without the penalty of adding a dependency. In conclusion, once Zig has reached a stable enough state, I would challenge the libcurl devs to consider, perhaps it could make sense to slowly, carefully, methodically transition to a newer language, if that language was friendly enough to the process.

stirner 10y ago
What does Zig have to offer over Rust's FFI[1]? Also, it might be useful to know that HN's comment syntax isn't Markdown, though it shares a few features like indenting for code blocks and asterisks for italics. Markdown-style links are not supported. [1] https://doc.rust-lang.org/book/ffi.html https://doc.rust-lang.org/book/ffi.html
- icebraining 10y ago
  As an addendum, using a Rust library in C is quite easy: https://doc.rust-lang.org/1.2.0/book/rust-inside-other-languages.html https://doc.rust-lang.org/1.2.0/book/rust-inside-other-langu...

najati83 10y ago

The reason curl is everywhere is that it's written in C. So obviously the author is not going to rewrite it in any other language.

deleted 10y ago

[deleted]

spoiler 10y ago

> These “you should switch language” remarks are strangely enough from the backseat drivers of the Internet. Those who can tell us with confidence how to run our project but who don’t actually show us any code. This is an usual problem with people participating in discussion over the Internet. I am sure those people mean well, but often they don't put enough effort into assessing the situation and underestimate its complexity before they decide to hit the keyboard. I've myself done it a number of times, but I am working on not doing it. Advice for the people who are making all these suggestions and: A good way to see if you're talking out of your ass—unintentionally, of course—is to try to do what you're saying someone should do. Just try and re-write some parts of curl in Rust (you can go with a hip name like rurl). It's a good exercise to learn about the project you're rewriting, the language you're writing it in, and the problem domain. So, at the very least you'll get something out of it, and if your project is successful: even better!

WalterBright 10y ago
D is compatible enough with C/C++ that I have been able to take largish multifile projects and convert them one file at a time to D. The full regression suite was run after each file was converted. This helped ensure both an accurate conversion and did not leave the project in a broken state for an extended period.
- WalterBright 10y ago
  I've successfully converted many projects from one language to another using this method. One was a huge assembler program I converted to C back in the 1980s (ironically, so it could be ported to other platforms!).
reitanqild 10y ago
Were do I send proposals for "comment of the month"? More seriously speaking: IMO this is solid advice. It - is actionable - goes out of its way to be polite and respectful - has a chance to actually improve online discussions one comment (and hopefully commit) at a time

jasode 10y ago

>I think that’s missing the point I was trying to make. I think Daniel Stenberg missed the point of some of the comments. (At least the prominent ones I saw.) >We use C for a whole range of reasons as I tried to lay out there in spite of the security challenges the language brings. The availability and practicality of old ecosystems are valid justifications but it's orthogonal to the "memory bugs caused by the language vs the human". There's nuance and you have to distinguish multiple conversations: 1) curl should have been written in a memory safe language (past tense) 2) curl should be rewritten in a memory safe language (future tense -- e.g. maybe use Rust) 3) setting aside curl's 19 years of wide deployment challenges, D.S. could have been a better spokesman for the benefits memory safe languages Daniel's response is mostly to #1. However, some of the criticism is really conversation #3. Understandably, #3 is somewhat harder for D.S. to adopt because it requires distancing himself from the 19 years of curl as a successful utility and instead, consider how a language's design affect programmers' bug count. Many felt that D.S. citing curl's bugs as "human logic errors" instead of "language-induced errors" undermines his point when conversation #3 is the framework. So maybe some of the new nitpicking of counting CVE bugs is more about conversations #2 & #3. The #2 may not be realistic for another 10 years... or maybe never because of various deployment hurdles. However, we can still discuss #3.

hsivonen 10y ago
I think this is the disconnect. One can understand why curl is written in C, not expect Daniel to write it in another language, appreciate that the overall vulnerability count being impressively low considering how old the project is, and still worry about the section about vulnerabilities in the original post looking dismissive when it comes to vulnerabilities that are attributable to C and perhaps getting trotted out in the future by others contemplating new projects as an excuse not to use a safe language.
- empath75 10y ago
  Yeah, I absolutely think that curl should stay a C project until the day it dies. I also think someone should write a curl-alternative in rust. And at some point, people may start using the Rust version instead of the c version.

nallerooth 10y ago

I wonder how much experience people yelling "OMG you're using C(++) - you have to rewrite everything!" actually have. Some of them can motivate their claims with good examples (at least to some extent) while others just show a serious lack of understanding of project scope and/or the reason something is written the way it is. One of my favorite examples so far: https://www.postgresql.org/message-id/CAASwCXdQUiuUnhycdRvrUmHuzk5PsaGxr54U4t34teQjcjb%3DAQ%40mail.gmail.com https://www.postgresql.org/message-id/CAASwCXdQUiuUnhycdRvrU...

buserror 10y ago
Ahah nice example! "I can't code, but if I were, I would like to be productive in something that doesn't require me to make any effort in understanding the problem, please advise" I don't know where all the C hate is all about. I love C, it's a great tool. Can be quite sharp, be careful with your remaining fingers. I spent the better part of last week studying rust and go, and my conclusion was 'meh, I'll have another look in another few years" and kept on trucking.
- nallerooth 10y ago
  I don't get all the C "hate" either. Sure, C is a source of many "interesting" things - but it's also a natural part of control systems in avionics, space craft and nuclear power plants. I don't want someone who just read that "Rust is a great, safe language" to rewrite those. I really don't.
  - fiedzia 10y ago
    And I do. Planes crashed and people died because of buggy software. We have lost Mars climate orbiter due to not using type-safe language. In the cases of software that works fine, we are just relying on ridiculous amount of manual verification, and this is both risky and expensive.
    - mikulas_florek 10y ago
      Mars Climate Orbiter was lost because they forgot to convert units between different systems. That's an error that can (and probably would) be done in all type-safe languages I know of.

DCKing 10y ago

There are many reasons why curl uses C, and should continue to do so. Yes, C allows a combination of portability, performance, and accessibility of your source code by others that is not matched by any language. And yes, C was a perfectly valid choice throughout the vast majority of its life and still is. Switching is not a realistic option at all. But I think Daniel's original attempt at explaining away C's security issues has been debunked. And I think he owns it well by singing a slightly different tune in this blogpost. The lesson learnt here is that if you use C, don't try to explain away the security issues you have just opened for yourself. It will not hold up. Accept that the safety of C is a very serious issue, and that there are other reasons you choose it. Choosing C over more secure options does not mean you don't care about security, but it means that you had to make pragmatic choices. Choosing C over more secure options and subsequently attempt to minimize the security impact of this decision (not talking about Daniel here) will lead you to not addressing those issues effectively.

fiedzia 10y ago
Well said.
generic_user 10y ago
> Choosing C over more secure options and subsequently attempt to minimize the security impact of this decision (not talking about Daniel here) will lead you to not addressing those issues effectively. When you start a new C/C++ project in 2017 you can effectively and confidently address security and 'safety'. And it is done on a regular basis. There is an entire infrastructure of tools and introspection refined over the past 30+ years for C/C++ that is part of the project from the beginning. Address sanitizes, Better compilers, Valgrind, Intel's tools etc. All of the potential problems are known and there are tools to deal with them. But the responsibility is put in the hands of the programmer like many other things in C/C++.
- pjmlp 10y ago
  CppCon 2015 as Herb Sutter asked the audience how many use regularly any form of static analysis tools, the answer was around 1% of the audience. The video is easy to find if you want to watch it. So apparently the tools might be there, but they are ignored by those that should be using them in first place.

nathan_f77 10y ago

Are there any automated tools that you can help you convert a C codebase into Rust? And is there any reason why Rust couldn't be as portable as the C code that makes up curl? Is it very difficult to transpile C into Rust? As an aside, I not like the curl style guidelines [1]. Especially "'else' on the following line", and "No space before parentheses". That doesn't look nice at all. [1] https://curl.haxx.se/dev/code-style.html https://curl.haxx.se/dev/code-style.html

chrismorgan 10y ago
https://github.com/jameysharp/corrode https://github.com/jameysharp/corrode is a C-to-Rust translator. It won’t be idiomatic Rust, though; not by a long shot. (Rust is much more restrictive in what it allows, for reasons of safety, though more expressive.) And it may not be correct, either; corrode is getting pretty good, but it’s not perfect yet.
deleted 10y ago
[deleted]
thristian 10y ago
For automated tools that can convert a C codebase to Rust, you're looking for is Corrode: https://github.com/jameysharp/corrode https://github.com/jameysharp/corrode Last I heard, Corrode can convert many C syntactic structures into their Rust equivalents, they usually compile and sometimes they even work. Even when the conversion works flawlessly, the resulting Rust does exactly what the original C code did—there's no safety benefit (but you can begin refactoring to clean things up).
mrkgnao 10y ago
https://github.com/jameysharp/corrode https://github.com/jameysharp/corrode
fiedzia 10y ago
And is there any reason why Rust couldn't be as portable as the C code that makes up curl? In theory no, in practice yes. Rust is using LLVM project to generate machine code, and that only supports certain architectures. There are however other safe languages that can generate C, if this is a requirement. > Is it very difficult to transpile C into Rust? If safety guarantees could be added to C by some conversion or analysis, there would be no need for Rust to exist. Rust (and other languages with similar goals) is safer for many reasons, not limited to languages itself. Stdlib and runtime play their part too. Automatic conversion that takes all that into account is either impossible or impractical. There is a tool for converting C into unsafe Rust (corrode), which might be useful for people attempting to port C code, but in itself doesn't improve code safety at all.

fiedzia 10y ago

Note: I am discussing using safe languages in general and reasoning for doing or not doing that. If you believe rewrite requests are unreasonable no matter what, don't bother reading that. Or perhaps consider this to be addressed to people deciding how to choose libraries to depend on, not to specific library authors. > curl is currently one of the most distributed and most widely used software components in the universe, be it open or proprietary and there are easily way over three billion instances of it running in appliances, servers, computers and devices across the globe. Right now. In your phone. In your car. In your TV. In your computer. Congratulations. So how many of those have unpatched vulnerabilities right now that were caused by using unsafe language? In my phone? In my computer? In my car? How many accidents of data or money theft it caused? Not to mention common non-security related bugs. > If we then have had 40, 50 or even 60 security problems because of us using C, through-out our 19 years of history, it really isn’t a whole lot given the scale and time we’re talking about here. Curl had multiple unpatched security problems for several years before they were discovered. As the author admits, it most likely still has many. dpkg -l | grep '^ii lib'|wc -l lists over 2000 libraries I have installed, lot of them written in C. An estimate of 2000 multiplied by 60 is ... scarily to many. Large amount of them avoidable by using safer languages. > Using another language would’ve caused at least some problems due to that language If it solves more problems than it creates, it is worth doing. > none of the memory safe languages anyone would suggest we should switch to have been around for 19 years. Neither have been mobile phones (at least as we know them), but they are useful, so I am using one. > We will continue to work hard on minimizing risks, detecting problems early by ourselves and work closely together with everyone who reports suspected problems to us. I prefer to work smart, not hard.

rkangel 10y ago
> dpkg -l | grep '^ii lib'|wc -l lists over 2000 libraries I have installed, lot of them written in C That '2000' number is not the relevant one here. The reason that security is important wrt Curl is that Curl is used to access the internet. Security in the library used to print colours to your terminal is a bit less important. It's those widely used, public facing libraries that need a good security story (and using a memory-safe language is a good start there).

simias 10y ago

I think most of the points he made in his original post are very valid ones, it's just the fatalist "devs are going to create bugs, such is life" approach to errors that I think I glimpsed in the original post that prompted my response in the previous thread. I'm tempted to call it the "C Stockholm syndrome". The reason I say that is that I could actually have written something like that a few years ago myself. I think it's a pretty common mindset amongst the "low level" C crowd. And there's a good reason for that, we've been burned before. People have been advertising "safe" alternatives to C since forever. Use Perl! Use Java! Use Lisp! Use Haskel! Use Python! Use Go! But none of these languages can replace C in all of its use cases. You have big runtimes, garbage collection, interpreters, you name it. You can't just bootstrap a stack pointer in 3 lines of ASM and call into them like you can with C. And even if we don't say it out loud there's a bit of snobbism to writing low level "unsafe" code. "Safe languages are for noobs and webdevs, real coders debug with oscilloscopes". And so I had this mindset that you couldn't have your cake and eat it too: you could be safe or you could be low level. Segfaults and NULL pointers were just a normal part of dealing with low level "unmanaged" environments and we'd just have to deal with it. People who complain about it are just not worthy. In my opinion the biggest success of Rust so far is in proving that you could be safe without needing a GC, a big runtime or complex abstractions. You could design a better C without having to compromise. Even if Rust fails in the long run, it would still be a very important "proof of concept". It showed that C was unnecessarily unsafe, that you could achieve the same feature set without throwing safety completely out of the window. That being said if there really are folks bugging devs by asking them to consider switching to language X or Y then I think it's fair to say that these people are idiots. Might as well ask JK Rowling to rewrite Harry Potter in Esperanto because "it's totally the future!". I mean, let's face it, at this point Curl probably has a much bigger userbase than Rust itself by at least an order of magnitude. Which one is more likely to be irrelevant in 10 years? Rust or Curl (and by extension, C)? If you answered the latter you should read less hacker news. And even if Rust ends up being successful it's not like C will go away any time soon, or even in our lifetimes. Curl (as a C library) will probably still be useful and relevant for decades to come no matter what.

fiedzia 10y ago
https://www.reddit.com/r/languagelearning/comments/1yv1i2/harry_potter_book_one_in_esperanto/ https://www.reddit.com/r/languagelearning/comments/1yv1i2/ha...

wojt_eu 10y ago

Curl should be split into micro-services.

legulere 10y ago

He writes "way over three billion instances", I read "way over three billion vulnerable instances" which is a security disaster.

spion 10y ago

I don't think the reactions and comments were meant as an attack on curl, but more as an attacks on C. Most programmers who attack C have seen the consequences of a memory-unsafe, type-unsafe language dominating systems programming throughout the years. I remember the period during the early 2000s when "buffer overflow exploits" were an entire industry. I also remember the strange allure of C and C++ in beginner programmer circles. The logic goes something like this: "they are powerful languages, all of the big and popular software projects are written in them, so they must be the best". Their next step was to go on to start a new generation of unsafe and vulnerable programs and libraries, and so on. Now that we have the tools and technology to do better, we should really be making some efforts to push for them. The goal isn't change right now, but starting the process early seems like a good idea.

wott 10y ago
Most programmers who attack C at the moment are frontend web javascripter kids and rust/functional language snobs, who both never used C and keep repeating the same stuff. You just have to witness the number of "OMG, you did that in C? Wow!" that are posted every time someone posts a basic C program on a forum. There is a whole crowd that thinks C is some arcane stuff only to be used by by magicians, and if you're not a guru it's gonna kill you. And then you have the Rust people who chime in and point a list of C problems that are in fact C++ problems. And then you get the "but anyway modern processors do not execute the code you tell them to execute" guys. Sigh... It goes every time the same way...
- biskit10 10y ago
  wott
- eeeficus 10y ago
  well it's 2017. maybe it's time we get rid of C? :)
  - endisukaj 10y ago
    Lisp is even older than C, should we get rid of that too?

spion 10y ago

I don't think the reactions and comments were meant as an attack at curl, but more as an attacks at C. Most programmers who attack C have seen the consequences of a memory-unsafe, type-unsafe language dominating systems programming throughout the years. I remember the early 2000, and the period where "buffer overflow exploits" were an entire industry. I also remember the strange allure of C and C++ in beginner programmer circles. The logic goes something like this: "they are powerful languages, all serious software is using them, so they must be the best". The next step is to go on to start a new generation of unsafe and vulnerable programs and libraries, and so on. Now that we have the tools and technology to do better, we should really be making some efforts to push them. The goal isn't change right now, but starting the process early seems like a good idea.

kazinator 10y ago

The best feature of Curl is that you can use it as a child process, driving it through the command line interface of its principal executable. Quite a lot of the API-level bells and whistles are covered. A few years ago I looked at the API and internals and went, "gack!" No way am I integrating this dung heap at that level; let the users just wrap the process with a curl function that returns a process pipe stream if they want to use this.

jeffdavis 10y ago

The original post said: "C is not the primary reason for our past vulnerabilities" So it's entirely reasonable for critics of C to challenge that claim.

zeveb 10y ago

> curl is currently one of the most distributed and most widely used software components in the universe, be it open or proprietary and there are easily way over three billion instances of it running in appliances, servers, computers and devices across the globe. Right now. In your phone. In your car. In your TV. In your computer. Etc. And every one of those three billion devices is vulnerable, due to the use of C. > I feel a need to underscore the fact that none of the memory safe languages anyone would suggest we should switch to have been around for 19 years False: Common Lisp has been around since 1994 (23 years), and in substantially the same shape for longer still. Standard ML has been around since 1990 (27 years). OCaml has been around since 1996 (21 years). Smalltalk has been around since 1984 (33 years). Each of those languages is more memory-safe than C and has facilities which help prevent other C-like errors. Each is capable of speeds approaching that of C, esp. for a problem like URl fetching (e.g. I just tried fetching http://www.google.com/ http://www.google.com/ with both curl & DRAKMA — a Common Lisp package for URL fetching: curl reliably ran in about .065 seconds & DRAKMA reliably ran in about .14 seconds; I have no reason to believe that DRAKMA is particularly well-optimised; no doubt it could get even faster if desired). I think this really is a textbook example of the Blub Paradox: someone using C thinks that for the most part it's a reasonable choice in order to achieve certain goals, while someone used to a better language is able to see that C is simply unfit for purpose: programs written in it will inevitably have security flaws which will inevitably cause harm — particularly when three billion devices, many unpatched, are running them.

snakeanus 10y ago
> And every one of those three billion devices is vulnerable, due to the use of C. Due to the use of unsafe C implementations. You would have the same problems if you used an unsafe CL or rust implementation.
kazinator 10y ago
We should stop using Pythagoras' a^2 = b^2 + c^2. The last commit to that was like 2500 years ago. (Core engine only, not counting some syntactic sugar).
pjmlp 10y ago
> I think this really is a textbook example of the Blub Paradox:.... It was already like that in the 90's. When I learned C, I was using Turbo Pascal 5.5 and getting up to speed with newly released Turbo Pascal 6.0. Besides the improved portability over Pascal dialects, I didn't saw any compelling feature over what Turbo Pascal was capable of.

kbenson 10y ago

curl is currently one of the most distributed and most widely used software components in the universe, be it open or proprietary and there are easily way over three billion instances of it running in appliances, servers, computers and devices across the globe. Right now. In your phone. In your car. In your TV. In your computer. Etc. If we then have had 40, 50 or even 60 security problems because of us using C, through-out our 19 years of history, it really isn’t a whole lot given the scale and time we’re talking about here. I think there's a disconnect here. Given something that's has a thousand instances running and averages 20-30 security problems a year and something else that has over three billion instances running and averages 2-3 security problems a year, which one do you think I care more about? I care about the one running everywhere, and on devices which may never see updated code in the rest of their lifetime. If curl were some unused project this wouldn't be a conversation. At the point where your code is on all those devices, you will get scrutiny. Expect it. Your choices may have some impact on a significant portion of the Earth's current population. At the same time, it's not fair to expect curl to rewrite their project in some other language. Some other project can start with the goal of providing the same API as curl (or not) that attempts to mitigate security problems though a different code architecture or language, and maybe curl devs will contribute. But expecting curl devs to start over from scratch is not feasible.

jeffdavis 10y ago

The problem is that the author listed specific reasons, which opened him up to specific challenges. He should have just said: "curl is an established project. New languages are not being seriously considered for curl." If you want to close down a conversation, it's counterproductive to make new dubious claims like "C is not the primary reason for our past vulnerabilities". Note that closing down a conversation isn't bad. Sometimes they are annoying or distracting. I love rust but here was my comment when someone wanted to rewrite postgres in rust: https://news.ycombinator.com/item?id=13354768 https://news.ycombinator.com/item?id=13354768

hzhou321 10y ago

Not to rewrite is a very sensible decision due to the fact that the current way of programming is un-rewritable. For example, people don't write FORTRAN in every language. The choice of programming language often fundamentally change the way we approach programming. Switching language not only involves addition or reduction of language constructs, it fundamentally alters programmer's highest level thinking process. That is not a merit. What we need is a meta-layer programming interface that isolates our higher thinking from lower language concerns to some degree. If you can keep the higher level understanding, then rewriting become less of a barrier. We need write pseudocode in every language. I have been exclusively programming with a meta layer, MyDef, for quite a while, and I routinely re-write my Perl program to C, not without effort, but always a worthwhile effort trading for the goal of rewriting (speed in this case). I am a researcher, and I rarely write production code. But for production code, I think most software should consider rewriting in Rust. Prototype in Perl/Python, develop in C, and produce in Rust -- makes good sense. It makes good sense in our current non-software industry, where researchers, developers, and producers are people with different focus and use different material and tools. This is not true for our software industry, where people who are not interested in researching or developing are forced to research and develop, and people who are only interested in research are forced to develop and people who only interested in developing are forced to research and produce. My example: http://hz2.org/blog/send_more_money.html http://hz2.org/blog/send_more_money.html

mirekrusin 10y ago

As a meta-comment - it's interesting to see how religiously-intense this discussion has became. Why are we - programmers - need to hold such a strong opinions about languages? Especially considering a fact that the playground is completely open. My personal opinion is that if he was wrong - we wouldn't even be reading his argument - it wouldn't make it to the top of HN - everybody would be using Xurl library instead (written in X-lang, where X is not C). At the same time Rust is shaping to be fantastic language and I personally hope that soon we'll see Rurl cargo together with kernel and what not in Rust. Unless in this decade something better will appear, who knows?