13 ms·

“I Could Rewrite Curl”

308 points by TangerineDream 5y ago

Piskvorrr 5y ago
Obviously everyone could rewrite curl*, in 2021, or so they think. Here's the magic in that trick: we have, collectively, amassed a heap of knowledge on HTTP and its behavior in the wild; this was made possible by curl, in the first place. Once this is handwaved away, of course it's easy to do. "Look at me, [standing on the shoulders of giants,] I have a really great view from up here!" (*small print: and this is speaking about the happy path, where client, its stack, network, server stack, and server all align perfectly; this happens pretty much never)
- thrower123 5y ago
  I wonder what percentage of attempts to rewrite curl in modern languages end up eventually just calling code from libcurl if you trace down through their standard libraries far enough?
  - chris_wot 5y ago
    I wonder how many attempts to rewrite curl would need to use curl to understand how the protocol works?
    - Piskvorrr 5y ago
      This is quite acceptable, IMNSHO. If you're reinventing a wheel, of course you need to know why your predecessors have not used the Obvious Simple Shortcut That Intuitively Makes Sense. Was it not available back when this was written? Was it to keep compatibility with Windows XP? Or perhaps your idea really did not occur to the developers. Most time - when reimplementing stuff - is not spent coding, or even debugging, but on research.
  - slooonz 5y ago
    What standard library uses curl for HTTP ? AFAIK, none of Go, Python or Node does.
    - meepmorp 5y ago
      "PHP," he said with a shudder.
    - jmcqk6 5y ago
      I'm not that familiar, but I believe php does
    - jsjohnst 5y ago
      Many of the commonly used Ruby HTTP gems use libcurl under the hood.
ceejayoz 5y ago
I'd love to see more people sponsor Daniel's work. (I do, because every job I've had in the last two decades has needed it.) https://github.com/sponsors/bagder https://github.com/sponsors/bagder
- qiqitori 5y ago
  It would be interesting if there were a site where people could sponsor bounty payments for security vulnerabilities in open source software. On curl: https://hackerone.com/curl?type=team https://hackerone.com/curl?type=team says "All rewards in our program are graciously donated by our sponsors." with a link to https://curl.se/sponsors.html https://curl.se/sponsors.html, and that probably works well for a large project like curl.
- npace12 5y ago
  1998 was around the time i first got into linux (slackware) and the internet. For a long time, I thought curl was one of those commands that was just a core part of linux and a team of bofhs write and maintain. It was sort of inspiring to find that it was just 1 guy and he did it fairly recently. Mad props to Daniel, whose website and tool have been invaluable to me for 23 years.
unixhero 5y ago
People are envious
- yetihehe 5y ago
  No, It's combination of Dunning-Kruger effect [0] and "Reality has a surprising amount of detail" effect [1]. [0] https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect [1] http://johnsalvatier.org/blog/2017/reality-has-a-surprising-amount-of-detail http://johnsalvatier.org/blog/2017/reality-has-a-surprising-...
  - Spivak 5y ago
    And the fact that it would be really easy to rewrite a usable subset of curl if you use your language's `import curl` which has behind the scenes had as much development effort as curl itself.
    - Piskvorrr 5y ago
      curl is mostly libcurl, `curl` the executable command is just one of the ways to interface with it. (Unless you're saying "let's pretend to reimplement curl by wrapping around curl behind the scenes," which is delightfully tautological.)
      - dtech 5y ago
        I think they meant their languages HTTP client, not literally the libcurl bindings
        Piskvorrr 5y ago
        Indeed. And that default client is, statistically, likely to use libcurl under the hood (because who has time to fix all those pesky corner cases that you wouldn't think of in 10 years? Obviously someone who's been doing that stuff for 25 years.).
tuyguntn 5y ago
I can write cURL in 1 line of bash: alias mycurl='curl' /s
- ludamad 5y ago
  You're available to fix exploits, right?
  - tuyguntn 5y ago
    of course no, I just write everything in one line and forget
- tuyguntn 5y ago
  most people underestimate huge effort under simple to use products but integrated to our lives heavily. * redis - I can do it, listen socket, serve from hashmap * uber - how come this app is so huge, I can make it in 2 weekends (google map + api + database) * cURL - easy, use my fancy library (it already implements HTTP stack) and rewrite it ....
  - eric_cc 5y ago
    What's interesting is that the next Uber may be created by somebody naive about the effort it will take. Being naive can be an advantage in this case. Sometimes people who KNOW the effort are discouraged and never try. And it's the guy who is clueless who ends up embarking on the quest.
- rvieira 5y ago
  Pff. I can write cURL _in Python_ in 1 line. import pycurl
- skrebbel 5y ago
  I think there's more truth to this comment than your presentation suggests. I bet at least a subset of people who think they could reimplement curl easily, think of curl as a CLI tool to make HTTP requests. It's just a little bit of python/requests or node/fetch code to cover some basics for that. I think these people simply aren't aware that their programming language's built-in libraries often call in to libcurl for the dirty work.
Hippocrates 5y ago
Just a little tangent but I think everyone should know cURL… not postman, not httpie or any other more “friendly” tool. cURL is the standard, pre-installed, do-everything, scriptable tool to interact with web APIs. I could say mostly the same stuff about knowing Bash. It separates the barely effective from hyper-effective individuals on simple day to day (programming) tasks.
- 1_player 5y ago
  I really wish busybox would bundle curl instead of wget. In the age of containers, I often find myself having to use wget when operating inside of one with busybox as shell, and I get it wrong on first try. What usually trips me up is the syntax: `curl -O http://host/file http://host/file` (goes to stdout without -O) instead of `wget http://host/file http://host/file` (goes to file without -O)
  - Piskvorrr 5y ago
    It doesn't even bundle a proper wget, just a massively stripped down HTTP client it aliases to wget. Sure, providing minimal tools is the point of busybox, but having to sniff what actually runs when you do `wget` is massively annoying. (e.g. until recently, busybox wget didn't believe in HTTPS by default)
  - matt-attack 5y ago
    Mentioning wget makes me embarrassed for the days (probably 20 years ago) where I used to prefer wget. Curl annoyed me because I couldn’t “simply download a file”. Needed to provide an option just to write the file to disk. Of course now I feel really icky if im ever forced to use wget. And curl’s feature of defaulting to stdout is of course a perfect thing and makes way more sense than the alternative (for many reasons).
- FriedrichN 5y ago
  I am always highly annoyed when someone has documentation for their web API and all that they provide is some postman collection of crap (or a crappy code example in an arbitrary language) and some vague abstract description of their API. Then I'll have to spend X amount of time on reverse engineering it to curl so I can see what's actually going on. Often I'll discover that they're doing really weird stuff (double base64 encoding for example) and that I'm the first to notice that it is weird. When I tell them I use curl they think I'm some hacker using some obscure tool. Too many people in our field just take the first thing that works and they won't bother to actually look at it.
tomovo 5y ago
It's both sad and funny. Saw similar comments for "fman", a simple file manager. Simple, until you start thinking about the details: https://www.catnapgames.com/2018/11/20/fman/ https://www.catnapgames.com/2018/11/20/fman/ See also: The Door Problem - https://lizengland.com/blog/2014/04/the-door-problem/ https://lizengland.com/blog/2014/04/the-door-problem/
- mherrmann 5y ago
  fman author here. Thank you for the mention :-)
FriedrichN 5y ago
I could rewrite Curl. But I certainly will not. Of all the libraries I have ever used, Curl is the one I used the most and found the most useful. I have never had the idea to rewrite it or even use another library to replace it. I'm not afraid to say that I love Curl!
nindalf 5y ago
What's frustrating is people who think that they can radically simplify something without knowing anything about it. The same people think they can implement a text editor in 1000 lines of code because that's how the first text editors were made. Why can't we go back to those simple days? We can't because modern software handles cases that those didn't. For example, rendering and editing text seems straightforward until you look closer. Text Rendering Hates You (https://gankra.github.io/blah/text-hates-you/ https://gankra.github.io/blah/text-hates-you/) and Text Editing Hates You Too (https://lord.io/blog/2019/text-editing-hates-you-too/ https://lord.io/blog/2019/text-editing-hates-you-too/) are two amazing reads that analyze this in detail. They explain the hidden complexity to the vast majority of us who never knew. We simply slapped down a <textarea> or <EditText> and called it a day. If we tried to reimplement this stuff from scratch, we would likely start with a simple implementation, then tack on bits when we tried to handle issues like right-to-left layouts, Unicode, emoji, anti-aliasing, text overlapping and so on. All this before getting into implementing features that Dennis Ritchie didn't have to, like supporting plugins, releasing on multiple operating systems, accessibility or even syntax highlighting. Problems already known to and tackled by existing text editors today. If you look at work outside of your area of expertise and you feel like starting a sentence like "why don't you just ...", then reconsider. If the person you're talking to is halfway competent (they probably are), they've already considered what you're about to say. "Have you considered using less memory/CPU or removing this feature that I personally don't need?" Why yes, we have. (This is from a post I wrote earlier - Software Isn’t Bloated - https://blog.nindalf.com/posts/software-isnt-bloated/ https://blog.nindalf.com/posts/software-isnt-bloated/)
- alpaca128 5y ago
  All those people don't realise tools like curl did start with few lines back then. And after the first two minutes of testing the devs started adding additional code to handle failures, invalid inputs, different platforms, different versions of a standard etc. One can write a lot of things in 100 lines if it doesn't have to meet higher standards than a basic chat app tutorial in the "getting started" section of a programming language.
  - 5y ago
willeh 5y ago
Every piece of software that people actually use will face criticism. But as quite honestly for some one with the clout of Mr Stenberg to out some poor Joe Codes just because they dared to have an opinion is just as pathetic if not more. Writing an HTTP 1 client in a weekend is doable and Rust is probably a better language today than C is (with some caveats obviously). Is having that opinion enough to to be publicly humiliated on Mr Stenberg's blog, I sure hope not.
- ceejayoz 5y ago
  Is he not similarly entitled to "dare to have an opinion"? What is the level of clout at which point you should be forbidden from commenting on criticism people direct at you on public forums?
- chris_wot 5y ago
  If any of those people had written such a client (in Rust even!) could you link me to their project?
Denvercoder9 5y ago
I wouldn't be surprised if the first comment is right that you could build a HTTP client sufficient for 90% of the usecases in ~100 lines or so. It's that other 10% that takes 90% of the time, effort and complexity, and for which having libraries like curl is invaluable.
- bombcar 5y ago
  I think there are two classes of cURL users - those who use it as a "let me download this file to the current folder" users who always forget the -O; and that use case is quite simple and seems it could be easily replaced. And then there are those who use all the API features and realize it's a complex program. But the missing link is that even the first use case has many complex edge-cases that aren't obvious at first.
  - swiley 5y ago
    I always use wget for the "let me download this file to the current folder" because I know the flags better. curl is for weird stuff like "POST" requests or other protocols.
stephc_int13 5y ago
Sometimes this is not completely wrong. I never used Curl or looked at the source code, I don't know about the specifics, but rewriting a simpler copycat from scratch is not always a bad idea. I've done that a few times, to prove a point or because the original was bad and too difficult to maintain.
- viraptor 5y ago
  This reminds me of bocker. (https://github.com/p8952/bocker https://github.com/p8952/bocker) A minimal docker in 100 lines of bash.
  - ceejayoz 5y ago
    A great example, in that there are six year old (important looking!) PRs sitting there without any feedback or action. Ongoing maintenance of this nature is something often ignored by the "I could rewrite that, easy" folks.
    - viraptor 5y ago
      It's a joke / PoC project. Nobody should expect it to be maintained in any way. None of those PRs are important in that way.
      - ceejayoz 5y ago
        > Nobody should expect it to be maintained in any way. Which is the exact problem of "I could rewrite this in a weekend!" scenario with curl, right?
        viraptor 5y ago
        It depends if the rewrite is meant as a joke or a serious minimal implementation.
        5y ago
secondcoming 5y ago
The problem isn't CURL, but HTTP. It's actually suprisingly complex and really needs to die. That'll never happen though.
- Piskvorrr 5y ago
  Just as we managed to do that so well with SMTP and FTP ;)
- pjmlp 5y ago
  Plenty of applications use TCP and UDP, or even custom protocols on top of raw IP. Not everything needs to come through the browser.
  - Piskvorrr 5y ago
    Not everything on top of HTTP comes through the browser, either ;)
    - bombcar 5y ago
      More and more APIs use HTTP as the transport for various reasons, but one of them is that HTTP has considered large number of edge cases and has been tested "in the fire" - rolling your own custom API with security and encryption is much more fraught with dangers.
      - pjmlp 5y ago
        Except HTTP 2.0/3.0 has very little to do with HTTP 1.0, and is more akin to use TCP/UDP directly anyway. It just happens to get a free pass over port 80.
        bombcar 5y ago
        And that's the other major reason - if your traffic LOOKS like web traffic, it will slip through firewalls and other middle-ware devices.
viraptor 5y ago
Excluding the bragging, I don't think this part of the first comment is incorrect: "You can implement a 90% usecase HTTP client in < 100 lines without golfing." We use the general purpose curl machine for convenience, but a minimal version would suffice for majority of cases. (Just an observation, I'm not suggesting we do that)
andy_ppp 5y ago
"I could write the subset of curl that I minimally understand and build something much simpler in X amount of time". I sometimes think this but then I remember all the edge-cases in even the most basic real world code.
globular-toast 5y ago
This is quite common, unfortunately. In other news, I could do the jobs of every single other person in my company comfortably. Only my job is difficult.
taf2 5y ago
#define WEEKEND 4098 Extra two days just for good measure.
parhamn 5y ago
Let's be honest here. Curl to 98% of people is http/1 requests with some post params and maybe json body, with some custom headers. Most language standard libs (which might be using libcurl) can facilitate writing that relatively quickly. And probably with a more 'modern' CLI. Curl though, is a very wide breadth project. It currently supports the following protocols: DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP. Each one of those is a massive undertaking of protocol/spec management, edge case handling, cross compatibility, hacks, and much much more to make them work. Then to put a stable C api on top of it all to be a cross language toolkit is a MASSIVE undertaking. But, again, the "idiots" who posted those comments, aren't talking about any of this. Perhaps they shouldn't care to. Most folks I know don't even know about the other protocols curl supports and have only interfaced with curl through its http. Frankly there are nicer http cli's out there with less code that _can_ be written in a weekend (assuming they piggy back on a http lib). What Daniel Stenberg achieved is giving the world a fantastic, reliable, cross protocol, cross platform and cross language network library that can be used as the foundation for many projects. I'm sure few of those cited would claim they could do that.
- eps 5y ago
  > SMB Woah. This is genuienly impressive.
  - WORMS_EAT_WORMS 5y ago
    As someone who fell into the category of: > "Most folks I know don't even know about the other protocols curl supports and have only interfaced with curl through its http." Just curious if you could elaborate on this one specifically and why you found it especially interesting.
    - mckirk 5y ago
      SMB is... not very straightforward. I needed to reverse engineer it once to write an exploit, and it's difficult to properly formulate even basic requests, let alone the more complex stuff the protocol supports.
svantana 5y ago
All the best art, whether it be books, music, movies or software, make it seem so effortless. Ironically, it's when you see something bad that you kinda realize what goes into it. Like huh, hundreds of talented professionals worked really hard on this movie, and it still turned out this bad. Now imagine if I, a complete amateur, would give it a go...
- viraptor 5y ago
  > Now imagine if I, a complete amateur, would give it a go... Now that one is actually tricky. I'd probably enjoy a mediocre amateur production more than a mediocre Hollywood production. It's a bit like the uncanny valley idea - it's easier to look at something meh than at something you know is a bit off from being good.
lgeorget 5y ago
I'd be fairly proud of myself if I manage to write a working HTTP/0.1 library and command-line client in a week-end. I can barely imagine the amount of work put in cURL.
erellsworth 5y ago
Whenever I see someone say something like "I could rewrite X in a weekend" I think, "well then fucking do it, if it's that easy." If you think you can build a curl replacement in 100 lines of code, do it, and open source it so a other randos on the Internet can complain about it.
Etheryte 5y ago
This is a problem I fight against nail and tooth almost daily, and it's closely related to the not invented here syndrome [0]. I would even go so far as to call it the Dunning–Kruger effect of technical knowledge. For context, I'm a frontend developer slash consultant who usually gets called in when development slowly grinds to a halt due to an overbearing number of bugs and regressions. Almost exclusively, at their core these problems can be reduced to underestimating complexity and trying to build something from scratch. It's a very common theme — "I can build form validation from scratch, that's easy." But then there's async validators, fields that have circular references and timezone issues. "I can build a table from scratch," but then there's endless scrolling, custom filters and performance issues. I think the common denominator is that all of these issues look simple when you address the smallest possible usable case, but quickly blow up in your face when you start building without thinking what you need to add on later. Unless you want to hire me, don't be that guy. Forms, tables, network layers and most other things are already solved problems — please don't try to build them from scratch unless you really need to and have a good plan. [0] https://en.wikipedia.org/wiki/Not_invented_here https://en.wikipedia.org/wiki/Not_invented_here
- cageface 5y ago
  This is definitely a problem. There's also the problem of devs that can't do the simplest task without pulling in 5 massive libraries. The thing that keeps programming fun is that every case is a little different.
- Oddskar 5y ago
  I'd attribute it to developers that have a complete lack of empathy and professionalism towards both their future selves and their colleagues. Once one has gotten burned sufficiently by undocumented convoluted code I think there's an understanding that starts forming on how to make choices in libraries and structuring code. The problem is devs that jump around from greenfield to greenfield project and never get to experience that pain.
JustSomeNobody 5y ago
I can (re)write X[0] in a weekend is a popular trope among devs with not enough experience and too much confidence. [0] Twitter, Facebook, etc...
AndrewStephens 5y ago
Curl is magic. Every so often (less so in recent years) I come across a bad homespun implementation of an HTTP client embedded in a larger project. Usually, a few lines of code to open a socket and POST some data to a hard coded URL. I always replace these with curl, usually over the objections of the original programmer. The problem with HTTP (v1) is that it seems very simple to write a cut down client. But your simple client better handle proxies, various encodings, chunked bodies, redirects, HTTPs (with all the machinery that implies), and a whole host of annoying little features that the client is not in control of. "But I don't need to implement talking through a proxy, we don't use one" You don't use one now, but in 6 months you might. Or your client's might install one without your knowledge. It is actually pretty simple to write an HTTP server (I wouldn't advise it) because the server controls a lot of the conversation. But clients are best left to either the OS or curl. Self-promotion time: https://sheep.horse/2019/3/using_libcurl_effectively_and_safely_in_c++.html https://sheep.horse/2019/3/using_libcurl_effectively_and_saf...
- rembo666 5y ago
  Exactly! Most people using CURL don't realize how good it actually is or how complicated HTTP can get.
  - lanstin 5y ago
    Sometime people started using expect headers (with content length) for long posts and 100-Continue and curl just handled it before I heard of it (outside of like push type apps).
jspash 5y ago
Javascript? I could write that in 9 days, Jim.
agumonkey 5y ago
I find that strange how people can be so critical about a simple utility that doesn't pose a revolutionary or boasting project .. Apologies for all the ego driven comments, people we should relax and keep doing fun and useful stuff.
ape4 5y ago
Yeah its "see URL" not "see http"
caseysoftware 5y ago
If you're a fan of curl and have benefited from it but can't contribute via code, docs, etc, I'd recommend their OpenCollective page: https://opencollective.com/curl https://opencollective.com/curl Many employers let you expense $50-100/month without much effort so get a handful of friends and it has a bigger impact than one corporate contributor.
drosan 5y ago
Amazing example of Dunning–Kruger effect.
fleaflicker 5y ago
Really good one in response to claims that stack overflow could be rewritten in a weekend: https://www.bitquabit.com/post/one-which-i-call-out-hacker-news/ https://www.bitquabit.com/post/one-which-i-call-out-hacker-n...
pipologist 5y ago
It’s not that these devs don’t understand the complexity of curl, since it does make things appear simple, it’s because they don’t understand the complexity of the protocols implemented by curl. Long ago, at ${WORK} we implemented the http 1.1 protocol on top of platform tcp sockets, it took several months to implement a correct subset of the protocol, then when we needed more we scrapped it and just used libcurl!
hawkesnest 5y ago
It seems like the correct response for these "reimplementation" types of criticism is to encourage the person to do so and share the results. I have, for personal gratification, attempted (with varied success) to reimplement several utilities. The experiences were quite eye-opening, seeing how well designed the "standard" utilities are, and identifying new ways to improve my code.
Tomte 5y ago
With one of the comments I immediately thought "this is HN". And I was right. But on a second look I'm quite certain it wasn't the wording or the sentiment, but the beige background color that I hadn't consciously noticed before.
zamadatix 5y ago
Blurring does little to hide identity, as-is the difference to find that Twitter profile without even needing any tools is literally Googling "<word> <word> <word> <word> twitter" instead of "username twitter". Doesn't hurt the profile pic is included either. Simply filling the space with a solid color is not only easier but infinitely stronger.
tutfbhuf 5y ago
Then in a few days we will see the following headline on HN: "No, You Couldn't Rewrite Curl".
gameswithgo 5y ago
This is just the usual thing where people forget that it is the final 5-10% of a good project that takes 99% of the time. It is ok, they will remember, the hard way, one day soon!
GuB-42 5y ago
I could write a simplified curl in a week-end, I could even write an HTTP server in the same week-end. The problem, they will only work together, no https of course, that "s" is really annoying. And even that is not as straightforward as one might think. I did it it C, Unix, TCP sockets, nothing unusual, but when you start putting significant load, you start getting errors left and right and you have to handle them properly. These are all documented, but easy to overlook.
- lanstin 5y ago
  With C and something like libev or won’t have issues with load before you have issues with weird http stuff that works for other clients or servers. I just had to modify my squid config to allow airflow Python to to a GET with a four length body: null (and no url-form-encoded content type).
mastrsushi 5y ago
Surprised I didn't find a post about someone saying they could rewrite it in Rust.
joaoqalves 5y ago
This happens everywhere. The devil is in details. If we use an application and the surface we see looks simple, we assume it must be simple to build :) Then we start to see all the complexity behind. Shameless plug: This kind of posts was the reason I wrote about it recently too [1] 1 - https://world.hey.com/joaoqalves/i-could-build-this-during-the-weekend-aa093c5e https://world.hey.com/joaoqalves/i-could-build-this-during-t...
notacoward 5y ago
Apparently these people never heard of Chesterton's Fence. https://fs.blog/2020/03/chestertons-fence/ https://fs.blog/2020/03/chestertons-fence/ tl;dr Don't remove something until you understand why it's there. Your needs or experience are not everyone's.
benibela 5y ago
As Delphi programmer, I have so far been avoiding libcurl successfully, and spent years rewriting curl because I need to make HTTP requests At first it was simple. I was making Windows apps and Windows has an API for HTTP requests. Just call wininet with the url, and it handles everything. No need to use someone's library if there is a standard OS API. And it is fully integrated in Windows, you get the default proxy settings and user-defined HTTPS certificates. It even shares cookies with the internet explorer (which I did not want, so I wrote my own cookie parser) Then I made a linux port. Suddenly there is no API. But I also avoid C, since it is unsafe, while Delphi/Pascal has memory safe strings and arrays. So I needed a HTTP Pascal library. There are several libraries, which also support other protocols I have never heard of. Then I used the synapse library. It is really causing a lot of problems. Incomplete cookie handling, no gzip encoding, OpenSSL messing everything up... Then I made an Android port. The standard stable system library was Apache HttpComponents, so I use that. Shortly after, Google makes a new Android version, and removes Apache HttpComponents. Then I had to find a new library. In hindsight all the porting were useless projects. I should have told people to use WINE and be done with it. A stable API is much better than a library. Or write my own HTTP client from the start, which would have been much easier than switching libraries all the time. Especially if I have to implement half of it (header parsing, OpenSSL interfacing) anyways when the libraries do not work properly
hi41 5y ago
Why do so many people indulge in casual mockery? Is it because we are told not to respect authority or institutions but instead to respect ideas? I see no reverence anymore; just casual mockery everywhere one turns. I am constantly awestruck all the time by how amazingly beautiful everything is in GNU/Linux world. What would I do without FOSS!

Obviously everyone could rewrite curl*, in 2021, or so they think. Here's the magic in that trick: we have, collectively, amassed a heap of knowledge on HTTP and its behavior in the wild; this was made possible by curl, in the first place. Once this is handwaved away, of course it's easy to do. "Look at me, [standing on the shoulders of giants,] I have a really great view from up here!" (*small print: and this is speaking about the happy path, where client, its stack, network, server stack, and server all align perfectly; this happens pretty much never)

thrower123 5y ago
I wonder what percentage of attempts to rewrite curl in modern languages end up eventually just calling code from libcurl if you trace down through their standard libraries far enough?
- chris_wot 5y ago
  I wonder how many attempts to rewrite curl would need to use curl to understand how the protocol works?
  - Piskvorrr 5y ago
    This is quite acceptable, IMNSHO. If you're reinventing a wheel, of course you need to know why your predecessors have not used the Obvious Simple Shortcut That Intuitively Makes Sense. Was it not available back when this was written? Was it to keep compatibility with Windows XP? Or perhaps your idea really did not occur to the developers. Most time - when reimplementing stuff - is not spent coding, or even debugging, but on research.
- slooonz 5y ago
  What standard library uses curl for HTTP ? AFAIK, none of Go, Python or Node does.
  - meepmorp 5y ago
    "PHP," he said with a shudder.
  - jmcqk6 5y ago
    I'm not that familiar, but I believe php does
  - jsjohnst 5y ago
    Many of the commonly used Ruby HTTP gems use libcurl under the hood.

ceejayoz 5y ago

I'd love to see more people sponsor Daniel's work. (I do, because every job I've had in the last two decades has needed it.) https://github.com/sponsors/bagder https://github.com/sponsors/bagder

qiqitori 5y ago
It would be interesting if there were a site where people could sponsor bounty payments for security vulnerabilities in open source software. On curl: https://hackerone.com/curl?type=team https://hackerone.com/curl?type=team says "All rewards in our program are graciously donated by our sponsors." with a link to https://curl.se/sponsors.html https://curl.se/sponsors.html, and that probably works well for a large project like curl.
npace12 5y ago
1998 was around the time i first got into linux (slackware) and the internet. For a long time, I thought curl was one of those commands that was just a core part of linux and a team of bofhs write and maintain. It was sort of inspiring to find that it was just 1 guy and he did it fairly recently. Mad props to Daniel, whose website and tool have been invaluable to me for 23 years.

unixhero 5y ago

People are envious

yetihehe 5y ago
No, It's combination of Dunning-Kruger effect [0] and "Reality has a surprising amount of detail" effect [1]. [0] https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect [1] http://johnsalvatier.org/blog/2017/reality-has-a-surprising-amount-of-detail http://johnsalvatier.org/blog/2017/reality-has-a-surprising-...
- Spivak 5y ago
  And the fact that it would be really easy to rewrite a usable subset of curl if you use your language's `import curl` which has behind the scenes had as much development effort as curl itself.
  - Piskvorrr 5y ago
    curl is mostly libcurl, `curl` the executable command is just one of the ways to interface with it. (Unless you're saying "let's pretend to reimplement curl by wrapping around curl behind the scenes," which is delightfully tautological.)
    - dtech 5y ago
      I think they meant their languages HTTP client, not literally the libcurl bindings
      - Piskvorrr 5y ago
        Indeed. And that default client is, statistically, likely to use libcurl under the hood (because who has time to fix all those pesky corner cases that you wouldn't think of in 10 years? Obviously someone who's been doing that stuff for 25 years.).

tuyguntn 5y ago

I can write cURL in 1 line of bash: alias mycurl='curl' /s

ludamad 5y ago
You're available to fix exploits, right?
- tuyguntn 5y ago
  of course no, I just write everything in one line and forget
tuyguntn 5y ago
most people underestimate huge effort under simple to use products but integrated to our lives heavily. * redis - I can do it, listen socket, serve from hashmap * uber - how come this app is so huge, I can make it in 2 weekends (google map + api + database) * cURL - easy, use my fancy library (it already implements HTTP stack) and rewrite it ....
- eric_cc 5y ago
  What's interesting is that the next Uber may be created by somebody naive about the effort it will take. Being naive can be an advantage in this case. Sometimes people who KNOW the effort are discouraged and never try. And it's the guy who is clueless who ends up embarking on the quest.
rvieira 5y ago
Pff. I can write cURL _in Python_ in 1 line. import pycurl
skrebbel 5y ago
I think there's more truth to this comment than your presentation suggests. I bet at least a subset of people who think they could reimplement curl easily, think of curl as a CLI tool to make HTTP requests. It's just a little bit of python/requests or node/fetch code to cover some basics for that. I think these people simply aren't aware that their programming language's built-in libraries often call in to libcurl for the dirty work.

Hippocrates 5y ago

Just a little tangent but I think everyone should know cURL… not postman, not httpie or any other more “friendly” tool. cURL is the standard, pre-installed, do-everything, scriptable tool to interact with web APIs. I could say mostly the same stuff about knowing Bash. It separates the barely effective from hyper-effective individuals on simple day to day (programming) tasks.

1_player 5y ago
I really wish busybox would bundle curl instead of wget. In the age of containers, I often find myself having to use wget when operating inside of one with busybox as shell, and I get it wrong on first try. What usually trips me up is the syntax: `curl -O http://host/file http://host/file` (goes to stdout without -O) instead of `wget http://host/file http://host/file` (goes to file without -O)
- Piskvorrr 5y ago
  It doesn't even bundle a proper wget, just a massively stripped down HTTP client it aliases to wget. Sure, providing minimal tools is the point of busybox, but having to sniff what actually runs when you do `wget` is massively annoying. (e.g. until recently, busybox wget didn't believe in HTTPS by default)
- matt-attack 5y ago
  Mentioning wget makes me embarrassed for the days (probably 20 years ago) where I used to prefer wget. Curl annoyed me because I couldn’t “simply download a file”. Needed to provide an option just to write the file to disk. Of course now I feel really icky if im ever forced to use wget. And curl’s feature of defaulting to stdout is of course a perfect thing and makes way more sense than the alternative (for many reasons).
FriedrichN 5y ago
I am always highly annoyed when someone has documentation for their web API and all that they provide is some postman collection of crap (or a crappy code example in an arbitrary language) and some vague abstract description of their API. Then I'll have to spend X amount of time on reverse engineering it to curl so I can see what's actually going on. Often I'll discover that they're doing really weird stuff (double base64 encoding for example) and that I'm the first to notice that it is weird. When I tell them I use curl they think I'm some hacker using some obscure tool. Too many people in our field just take the first thing that works and they won't bother to actually look at it.

tomovo 5y ago

It's both sad and funny. Saw similar comments for "fman", a simple file manager. Simple, until you start thinking about the details: https://www.catnapgames.com/2018/11/20/fman/ https://www.catnapgames.com/2018/11/20/fman/ See also: The Door Problem - https://lizengland.com/blog/2014/04/the-door-problem/ https://lizengland.com/blog/2014/04/the-door-problem/

mherrmann 5y ago
fman author here. Thank you for the mention :-)

FriedrichN 5y ago

I could rewrite Curl. But I certainly will not. Of all the libraries I have ever used, Curl is the one I used the most and found the most useful. I have never had the idea to rewrite it or even use another library to replace it. I'm not afraid to say that I love Curl!

nindalf 5y ago

What's frustrating is people who think that they can radically simplify something without knowing anything about it. The same people think they can implement a text editor in 1000 lines of code because that's how the first text editors were made. Why can't we go back to those simple days? We can't because modern software handles cases that those didn't. For example, rendering and editing text seems straightforward until you look closer. Text Rendering Hates You (https://gankra.github.io/blah/text-hates-you/ https://gankra.github.io/blah/text-hates-you/) and Text Editing Hates You Too (https://lord.io/blog/2019/text-editing-hates-you-too/ https://lord.io/blog/2019/text-editing-hates-you-too/) are two amazing reads that analyze this in detail. They explain the hidden complexity to the vast majority of us who never knew. We simply slapped down a <textarea> or <EditText> and called it a day. If we tried to reimplement this stuff from scratch, we would likely start with a simple implementation, then tack on bits when we tried to handle issues like right-to-left layouts, Unicode, emoji, anti-aliasing, text overlapping and so on. All this before getting into implementing features that Dennis Ritchie didn't have to, like supporting plugins, releasing on multiple operating systems, accessibility or even syntax highlighting. Problems already known to and tackled by existing text editors today. If you look at work outside of your area of expertise and you feel like starting a sentence like "why don't you just ...", then reconsider. If the person you're talking to is halfway competent (they probably are), they've already considered what you're about to say. "Have you considered using less memory/CPU or removing this feature that I personally don't need?" Why yes, we have. (This is from a post I wrote earlier - Software Isn’t Bloated - https://blog.nindalf.com/posts/software-isnt-bloated/ https://blog.nindalf.com/posts/software-isnt-bloated/)

alpaca128 5y ago
All those people don't realise tools like curl did start with few lines back then. And after the first two minutes of testing the devs started adding additional code to handle failures, invalid inputs, different platforms, different versions of a standard etc. One can write a lot of things in 100 lines if it doesn't have to meet higher standards than a basic chat app tutorial in the "getting started" section of a programming language.
- 5y ago

willeh 5y ago

Every piece of software that people actually use will face criticism. But as quite honestly for some one with the clout of Mr Stenberg to out some poor Joe Codes just because they dared to have an opinion is just as pathetic if not more. Writing an HTTP 1 client in a weekend is doable and Rust is probably a better language today than C is (with some caveats obviously). Is having that opinion enough to to be publicly humiliated on Mr Stenberg's blog, I sure hope not.

ceejayoz 5y ago
Is he not similarly entitled to "dare to have an opinion"? What is the level of clout at which point you should be forbidden from commenting on criticism people direct at you on public forums?
chris_wot 5y ago
If any of those people had written such a client (in Rust even!) could you link me to their project?

Denvercoder9 5y ago

I wouldn't be surprised if the first comment is right that you could build a HTTP client sufficient for 90% of the usecases in ~100 lines or so. It's that other 10% that takes 90% of the time, effort and complexity, and for which having libraries like curl is invaluable.

bombcar 5y ago
I think there are two classes of cURL users - those who use it as a "let me download this file to the current folder" users who always forget the -O; and that use case is quite simple and seems it could be easily replaced. And then there are those who use all the API features and realize it's a complex program. But the missing link is that even the first use case has many complex edge-cases that aren't obvious at first.
- swiley 5y ago
  I always use wget for the "let me download this file to the current folder" because I know the flags better. curl is for weird stuff like "POST" requests or other protocols.

stephc_int13 5y ago

Sometimes this is not completely wrong. I never used Curl or looked at the source code, I don't know about the specifics, but rewriting a simpler copycat from scratch is not always a bad idea. I've done that a few times, to prove a point or because the original was bad and too difficult to maintain.

viraptor 5y ago
This reminds me of bocker. (https://github.com/p8952/bocker https://github.com/p8952/bocker) A minimal docker in 100 lines of bash.
- ceejayoz 5y ago
  A great example, in that there are six year old (important looking!) PRs sitting there without any feedback or action. Ongoing maintenance of this nature is something often ignored by the "I could rewrite that, easy" folks.
  - viraptor 5y ago
    It's a joke / PoC project. Nobody should expect it to be maintained in any way. None of those PRs are important in that way.
    - ceejayoz 5y ago
      > Nobody should expect it to be maintained in any way. Which is the exact problem of "I could rewrite this in a weekend!" scenario with curl, right?
      - viraptor 5y ago
        It depends if the rewrite is meant as a joke or a serious minimal implementation.
        5y ago

secondcoming 5y ago

The problem isn't CURL, but HTTP. It's actually suprisingly complex and really needs to die. That'll never happen though.

Piskvorrr 5y ago
Just as we managed to do that so well with SMTP and FTP ;)
pjmlp 5y ago
Plenty of applications use TCP and UDP, or even custom protocols on top of raw IP. Not everything needs to come through the browser.
- Piskvorrr 5y ago
  Not everything on top of HTTP comes through the browser, either ;)
  - bombcar 5y ago
    More and more APIs use HTTP as the transport for various reasons, but one of them is that HTTP has considered large number of edge cases and has been tested "in the fire" - rolling your own custom API with security and encryption is much more fraught with dangers.
    - pjmlp 5y ago
      Except HTTP 2.0/3.0 has very little to do with HTTP 1.0, and is more akin to use TCP/UDP directly anyway. It just happens to get a free pass over port 80.
      - bombcar 5y ago
        And that's the other major reason - if your traffic LOOKS like web traffic, it will slip through firewalls and other middle-ware devices.

viraptor 5y ago

Excluding the bragging, I don't think this part of the first comment is incorrect: "You can implement a 90% usecase HTTP client in < 100 lines without golfing." We use the general purpose curl machine for convenience, but a minimal version would suffice for majority of cases. (Just an observation, I'm not suggesting we do that)

andy_ppp 5y ago

"I could write the subset of curl that I minimally understand and build something much simpler in X amount of time". I sometimes think this but then I remember all the edge-cases in even the most basic real world code.

globular-toast 5y ago

This is quite common, unfortunately. In other news, I could do the jobs of every single other person in my company comfortably. Only my job is difficult.

taf2 5y ago

#define WEEKEND 4098 Extra two days just for good measure.

parhamn 5y ago

Let's be honest here. Curl to 98% of people is http/1 requests with some post params and maybe json body, with some custom headers. Most language standard libs (which might be using libcurl) can facilitate writing that relatively quickly. And probably with a more 'modern' CLI. Curl though, is a very wide breadth project. It currently supports the following protocols: DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP. Each one of those is a massive undertaking of protocol/spec management, edge case handling, cross compatibility, hacks, and much much more to make them work. Then to put a stable C api on top of it all to be a cross language toolkit is a MASSIVE undertaking. But, again, the "idiots" who posted those comments, aren't talking about any of this. Perhaps they shouldn't care to. Most folks I know don't even know about the other protocols curl supports and have only interfaced with curl through its http. Frankly there are nicer http cli's out there with less code that _can_ be written in a weekend (assuming they piggy back on a http lib). What Daniel Stenberg achieved is giving the world a fantastic, reliable, cross protocol, cross platform and cross language network library that can be used as the foundation for many projects. I'm sure few of those cited would claim they could do that.

eps 5y ago
> SMB Woah. This is genuienly impressive.
- WORMS_EAT_WORMS 5y ago
  As someone who fell into the category of: > "Most folks I know don't even know about the other protocols curl supports and have only interfaced with curl through its http." Just curious if you could elaborate on this one specifically and why you found it especially interesting.
  - mckirk 5y ago
    SMB is... not very straightforward. I needed to reverse engineer it once to write an exploit, and it's difficult to properly formulate even basic requests, let alone the more complex stuff the protocol supports.

svantana 5y ago

All the best art, whether it be books, music, movies or software, make it seem so effortless. Ironically, it's when you see something bad that you kinda realize what goes into it. Like huh, hundreds of talented professionals worked really hard on this movie, and it still turned out this bad. Now imagine if I, a complete amateur, would give it a go...

viraptor 5y ago
> Now imagine if I, a complete amateur, would give it a go... Now that one is actually tricky. I'd probably enjoy a mediocre amateur production more than a mediocre Hollywood production. It's a bit like the uncanny valley idea - it's easier to look at something meh than at something you know is a bit off from being good.

lgeorget 5y ago

I'd be fairly proud of myself if I manage to write a working HTTP/0.1 library and command-line client in a week-end. I can barely imagine the amount of work put in cURL.

erellsworth 5y ago

Whenever I see someone say something like "I could rewrite X in a weekend" I think, "well then fucking do it, if it's that easy." If you think you can build a curl replacement in 100 lines of code, do it, and open source it so a other randos on the Internet can complain about it.

Etheryte 5y ago

This is a problem I fight against nail and tooth almost daily, and it's closely related to the not invented here syndrome [0]. I would even go so far as to call it the Dunning–Kruger effect of technical knowledge. For context, I'm a frontend developer slash consultant who usually gets called in when development slowly grinds to a halt due to an overbearing number of bugs and regressions. Almost exclusively, at their core these problems can be reduced to underestimating complexity and trying to build something from scratch. It's a very common theme — "I can build form validation from scratch, that's easy." But then there's async validators, fields that have circular references and timezone issues. "I can build a table from scratch," but then there's endless scrolling, custom filters and performance issues. I think the common denominator is that all of these issues look simple when you address the smallest possible usable case, but quickly blow up in your face when you start building without thinking what you need to add on later. Unless you want to hire me, don't be that guy. Forms, tables, network layers and most other things are already solved problems — please don't try to build them from scratch unless you really need to and have a good plan. [0] https://en.wikipedia.org/wiki/Not_invented_here https://en.wikipedia.org/wiki/Not_invented_here

cageface 5y ago
This is definitely a problem. There's also the problem of devs that can't do the simplest task without pulling in 5 massive libraries. The thing that keeps programming fun is that every case is a little different.
Oddskar 5y ago
I'd attribute it to developers that have a complete lack of empathy and professionalism towards both their future selves and their colleagues. Once one has gotten burned sufficiently by undocumented convoluted code I think there's an understanding that starts forming on how to make choices in libraries and structuring code. The problem is devs that jump around from greenfield to greenfield project and never get to experience that pain.

JustSomeNobody 5y ago

I can (re)write X[0] in a weekend is a popular trope among devs with not enough experience and too much confidence. [0] Twitter, Facebook, etc...

AndrewStephens 5y ago

Curl is magic. Every so often (less so in recent years) I come across a bad homespun implementation of an HTTP client embedded in a larger project. Usually, a few lines of code to open a socket and POST some data to a hard coded URL. I always replace these with curl, usually over the objections of the original programmer. The problem with HTTP (v1) is that it seems very simple to write a cut down client. But your simple client better handle proxies, various encodings, chunked bodies, redirects, HTTPs (with all the machinery that implies), and a whole host of annoying little features that the client is not in control of. "But I don't need to implement talking through a proxy, we don't use one" You don't use one now, but in 6 months you might. Or your client's might install one without your knowledge. It is actually pretty simple to write an HTTP server (I wouldn't advise it) because the server controls a lot of the conversation. But clients are best left to either the OS or curl. Self-promotion time: https://sheep.horse/2019/3/using_libcurl_effectively_and_safely_in_c++.html https://sheep.horse/2019/3/using_libcurl_effectively_and_saf...

rembo666 5y ago
Exactly! Most people using CURL don't realize how good it actually is or how complicated HTTP can get.
- lanstin 5y ago
  Sometime people started using expect headers (with content length) for long posts and 100-Continue and curl just handled it before I heard of it (outside of like push type apps).

jspash 5y ago

Javascript? I could write that in 9 days, Jim.

agumonkey 5y ago

I find that strange how people can be so critical about a simple utility that doesn't pose a revolutionary or boasting project .. Apologies for all the ego driven comments, people we should relax and keep doing fun and useful stuff.

ape4 5y ago

Yeah its "see URL" not "see http"

caseysoftware 5y ago

If you're a fan of curl and have benefited from it but can't contribute via code, docs, etc, I'd recommend their OpenCollective page: https://opencollective.com/curl https://opencollective.com/curl Many employers let you expense $50-100/month without much effort so get a handful of friends and it has a bigger impact than one corporate contributor.

drosan 5y ago

Amazing example of Dunning–Kruger effect.

fleaflicker 5y ago

Really good one in response to claims that stack overflow could be rewritten in a weekend: https://www.bitquabit.com/post/one-which-i-call-out-hacker-news/ https://www.bitquabit.com/post/one-which-i-call-out-hacker-n...

pipologist 5y ago

It’s not that these devs don’t understand the complexity of curl, since it does make things appear simple, it’s because they don’t understand the complexity of the protocols implemented by curl. Long ago, at ${WORK} we implemented the http 1.1 protocol on top of platform tcp sockets, it took several months to implement a correct subset of the protocol, then when we needed more we scrapped it and just used libcurl!

hawkesnest 5y ago

It seems like the correct response for these "reimplementation" types of criticism is to encourage the person to do so and share the results. I have, for personal gratification, attempted (with varied success) to reimplement several utilities. The experiences were quite eye-opening, seeing how well designed the "standard" utilities are, and identifying new ways to improve my code.

Tomte 5y ago

With one of the comments I immediately thought "this is HN". And I was right. But on a second look I'm quite certain it wasn't the wording or the sentiment, but the beige background color that I hadn't consciously noticed before.

zamadatix 5y ago

Blurring does little to hide identity, as-is the difference to find that Twitter profile without even needing any tools is literally Googling "<word> <word> <word> <word> twitter" instead of "username twitter". Doesn't hurt the profile pic is included either. Simply filling the space with a solid color is not only easier but infinitely stronger.

tutfbhuf 5y ago

Then in a few days we will see the following headline on HN: "No, You Couldn't Rewrite Curl".

gameswithgo 5y ago

This is just the usual thing where people forget that it is the final 5-10% of a good project that takes 99% of the time. It is ok, they will remember, the hard way, one day soon!

GuB-42 5y ago

I could write a simplified curl in a week-end, I could even write an HTTP server in the same week-end. The problem, they will only work together, no https of course, that "s" is really annoying. And even that is not as straightforward as one might think. I did it it C, Unix, TCP sockets, nothing unusual, but when you start putting significant load, you start getting errors left and right and you have to handle them properly. These are all documented, but easy to overlook.

lanstin 5y ago
With C and something like libev or won’t have issues with load before you have issues with weird http stuff that works for other clients or servers. I just had to modify my squid config to allow airflow Python to to a GET with a four length body: null (and no url-form-encoded content type).

mastrsushi 5y ago

Surprised I didn't find a post about someone saying they could rewrite it in Rust.

joaoqalves 5y ago

This happens everywhere. The devil is in details. If we use an application and the surface we see looks simple, we assume it must be simple to build :) Then we start to see all the complexity behind. Shameless plug: This kind of posts was the reason I wrote about it recently too [1] 1 - https://world.hey.com/joaoqalves/i-could-build-this-during-the-weekend-aa093c5e https://world.hey.com/joaoqalves/i-could-build-this-during-t...

notacoward 5y ago

Apparently these people never heard of Chesterton's Fence. https://fs.blog/2020/03/chestertons-fence/ https://fs.blog/2020/03/chestertons-fence/ tl;dr Don't remove something until you understand why it's there. Your needs or experience are not everyone's.

benibela 5y ago

As Delphi programmer, I have so far been avoiding libcurl successfully, and spent years rewriting curl because I need to make HTTP requests At first it was simple. I was making Windows apps and Windows has an API for HTTP requests. Just call wininet with the url, and it handles everything. No need to use someone's library if there is a standard OS API. And it is fully integrated in Windows, you get the default proxy settings and user-defined HTTPS certificates. It even shares cookies with the internet explorer (which I did not want, so I wrote my own cookie parser) Then I made a linux port. Suddenly there is no API. But I also avoid C, since it is unsafe, while Delphi/Pascal has memory safe strings and arrays. So I needed a HTTP Pascal library. There are several libraries, which also support other protocols I have never heard of. Then I used the synapse library. It is really causing a lot of problems. Incomplete cookie handling, no gzip encoding, OpenSSL messing everything up... Then I made an Android port. The standard stable system library was Apache HttpComponents, so I use that. Shortly after, Google makes a new Android version, and removes Apache HttpComponents. Then I had to find a new library. In hindsight all the porting were useless projects. I should have told people to use WINE and be done with it. A stable API is much better than a library. Or write my own HTTP client from the start, which would have been much easier than switching libraries all the time. Especially if I have to implement half of it (header parsing, OpenSSL interfacing) anyways when the libraries do not work properly

hi41 5y ago

Why do so many people indulge in casual mockery? Is it because we are told not to respect authority or institutions but instead to respect ideas? I see no reverence anymore; just casual mockery everywhere one turns. I am constantly awestruck all the time by how amazingly beautiful everything is in GNU/Linux world. What would I do without FOSS!