7 ms·

Let's be honest here. Curl to 98% of people is http/1 requests with some post params and maybe json body, with some custom headers. Most language standard libs

Let's be honest here. Curl to 98% of people is http/1 requests with some post params and maybe json body, with some custom headers. Most language standard libs (which might be using libcurl) can facilitate writing that relatively quickly. And probably with a more 'modern' CLI. Curl though, is a very wide breadth project. It currently supports the following protocols: DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP. Each one of those is a massive undertaking of protocol/spec management, edge case handling, cross compatibility, hacks, and much much more to make them work. Then to put a stable C api on top of it all to be a cross language toolkit is a MASSIVE undertaking. But, again, the "idiots" who posted those comments, aren't talking about any of this. Perhaps they shouldn't care to. Most folks I know don't even know about the other protocols curl supports and have only interfaced with curl through its http. Frankly there are nicer http cli's out there with less code that _can_ be written in a weekend (assuming they piggy back on a http lib). What Daniel Stenberg achieved is giving the world a fantastic, reliable, cross protocol, cross platform and cross language network library that can be used as the foundation for many projects. I'm sure few of those cited would claim they could do that.

eps 5y ago
> SMB Woah. This is genuienly impressive.
- WORMS_EAT_WORMS 5y ago
  As someone who fell into the category of: > "Most folks I know don't even know about the other protocols curl supports and have only interfaced with curl through its http." Just curious if you could elaborate on this one specifically and why you found it especially interesting.
  - mckirk 5y ago
    SMB is... not very straightforward. I needed to reverse engineer it once to write an exploit, and it's difficult to properly formulate even basic requests, let alone the more complex stuff the protocol supports.
- bjoli 5y ago
  Yeah. I once though I would have a look at what it would mean to just send a single file over SMB. What better way than to just reverse engineer the little bits of communication between a client and a server? The simplest solution would have been to give up the instant I started reading the catched traffic. I gave up after about 40 minutes of trying to figure out where to start.
flohofwoe 5y ago
Yeah, very easy to write an HTTP client on top of Berkley sockets. Until chunked responses come into play, and HTTPS, and HTTP/2, and HTTP/3... (this is actually something I'm very worried about, it used to be easy to cobble together a very basic HTTP client, not anymore with the new HTTP protocol versions, and all the bells and whistles).
- parhamn 5y ago
  Agreed. They're harder to debug, MITM, trace, etc. As a silver lining: I have been less worried recently seeing that most of the adoption of http/2-3 (at least their odd features) have been at the edge. Most developers are still writing http/1 endpoints and leaving the edge to optionally up-convert them for transport efficiency.
- ricardo81 5y ago
  And of course, CURL deals with multiple protocols as the OP mentioned. I've used libcurl for sending emails and FTP but mainly for HTTP. Using the multi interface it's relatively trivial (a few hundred lines of C) to fetch using hundreds or thousands of concurrent connections. There's even some courtesy opts for rate limiting per host. It's so good he/they built c-ares for asynchronous DNS lookups, IIRC something that wasn't so readily available in years gone by.
ephaeton 5y ago
> What Daniel Stenberg achieved is giving the world a fantastic, reliable, cross protocol, cross platform and cross language network library that can be used as the foundation for many projects. One of which, curl(1), is a constant ass-and-life saver for me personally and professionally. Of course the library itself ends up being used by me as well (in C, ++ and tcl). curl to me is in the league of sqlite aka "absolutely gorgeous and more often than not all you need"
krick 5y ago
True. I'm kind of surprised (again) by Daniel after 20 years of maintaining curl being so salty about "people not appreciating the achievement". Yes, for overwhelming majority of users curl in 2021 is just a simple http/https request client. And, yes, in 2021 that could be written over a weekend with a better CLI, mainly because most of the languages already have all important stuff implemented in a [standard] library (ironically, sometimes via libcurl).
- bombcar 5y ago
  I took it as more bemused, and given the audience I suspect it's more of a "look and laugh".
- mumblemumble 5y ago
  That was honestly my first thought. Of course it's easy to rewrite curl nowadays. It can't be anything more complicated than slapping a command line interface on top of libcurl...
  - justin_oaks 5y ago
    At first I laughed... then I thought about the sheer number of command line options that you can find in the man pages. After thinking about that, even writing the command line interface would be a non-trivial undertaking.
    - bb010g 5y ago
      Documentation takes work too.
- rkangel 5y ago
  > I'm kind of surprised (again) by Daniel after 20 years of maintaining curl being so salty about "people not appreciating the achievement" You're surprised by him being salty? I can't imagine what it's like to be on the receiving end of the water torture of people continually belittling the work you have done over 20 years and given to the community.
whymarrh 5y ago
> Let's be honest here. Curl to 98% of people is http/1 requests with some post params and maybe json body, with some custom headers. 100%, that's why we see so many smaller projects pop up (on GitHub and the like) that support basically just this. No shade to those projects, improving the UI for this subset is a worthy cause, it's just not anything near cURL.
michaelbuckbee 5y ago
In the midst of so much open source drama and angst, I'm still just profoundly grateful for Daniel Stenberg's work on curl. I'm no expert on open source management but Daniel has for years been a positive and steady force working on curl. This is him publishing the direct criticisms he's receiving as "thanks" for his efforts and it guts me he's got to put up with these ignorant comments.
dkarl 5y ago
A question that arises out of this is, should the 90% use cases be handled by a small, simple tool, or by the tip of the iceberg of a large, complex tool such as curl? I can see the advantages of standardizing on a complex, powerful tool for simple use cases. For one thing, it may be the only way to standardize: simple versions are too easy to write, and therefore you get dozens of competitors, none of whom are popular enough to shake out their edge case bugs. It's also nice not to have to find, install, and learn a new tool when you stray over that 90% boundary. With software dependencies, I think the advantages of small, simple libraries win out over generality. Supporting powerful use cases makes an API more complex and often means that simple, 90% use cases require weird incantations to make them work. Here I think of the times I've had to get into the guts of Jackson despite never doing anything remotely exotic with JSON. The 95% of your codebase that can be simple should be simple, so you can devote your attention to the things that need to be complex.
- rad_gruchalski 5y ago
  > A question that arises out of this is, should the 90% use cases be handled by a small, simple tool, or by the tip of the iceberg of a large, complex tool such as curl? But nobody is forced to use curl? People use it because it’s convenient, shoot them selves in the foot and then lash out at the author the tool for their own choices. Where’s the fault?
  - throwaway894345 5y ago
    > But nobody is forced to use curl? I like curl, but this isn't true. When you're SSH-ing onto a box, you often don't have permissions to install your favorite CLI tool, and even if you do have said permissions it's inconvenient to have to install it each time you SSH onto the box (not a major inconvenience, mind you). Moreover, in many cases you need to run a script or some other software that depends directly on curl. In general the "nobody is forced to use it" arguments rarely pan out (I remember this was a canned argument from C++ folks circa 2011: "C++ is the best language because it has every feature and if you don't like some features, you aren't forced to use them!").
spamizbad 5y ago
Why even get mad at curl? If you hate curl and all you want is a simple http/1 client just spend a few minutes to write a Python script w/requests that takes verbs and payloads from the command line rather than make the curl author's day shitty.
oxfordmale 5y ago
Curl has been a life safer for me when I had to interact with a legacy third party server. The Python request model was regularly hanging when making requests to this server, whereas I never observed these issues with curl. Curl appears to be better at dealing with such legacy edge cases as it has been around for more than two decades.
jeffrwells 5y ago
This begs the question of why all of these protocols should be in one library, instead of one each. If a project is only going to ever use http, why would I need to bundle in all of the others? Genuinely curious what the advantages are
- craftinator 5y ago
  My first thought is flexibility and interoperability. If you have one tool, cross platform, that supports a wide variety of protocols, you can use it on both ends of a connection, and easily swap out protocols without having to: 1) install a new tool on both sides 2) learn a new API for each protocol 3) bug check for it It also let's you do all this dynamically, so switching protocols on the fly is trivial.
- lanstin 5y ago
  It is a good library API with good support for async and that sort of thing. As a user it is less work to integrate the Nth protocol in a library I understand than to read the docs and try to reverse engineer some new API model of how to do things including asynchronous and configurations and TLS/cert management etc. there are a thousand ways to skin the cat, and libcurl is a good one. I want to learn a bunch of new distributed protocols because those are real things; I don’t want to learn a bunch of different APIs to do the same things because those are arbitrary wrappings to the real thing. If an API is successful enough it would be real, but cleverness in API design isn’t that useful without the popularity.
- cdelsolar 5y ago
  http://begthequestion.info/ http://begthequestion.info/
- jolmg 5y ago
  What better thing to do with a universal resource locator syntax than to have a universal client? > If a project is only going to ever use http, why would I need to bundle in all of the others? Your project might only ever think to use http, but my project might appreciate having one client for any URL, instead of having to parse URLs myself and decide on separate clients based on a part of it.
aerovistae 5y ago
I'm one of the 98%, and I don't like using cURL for this reason -- the CLI feels clunky and its gigantic number of options are distracting for my simple use case. Can anyone recommend a good alternative for the base case?
- spost 5y ago
  https://httpie.io/ https://httpie.io/ is pretty nice.
- arodyginc 5y ago
  wget maybe
- justin_oaks 5y ago
  I recognize the huge amount of time and effort that went into make cURL, but I agree that the gigantic number of options are distracting. When I read the cURL man page, I have a hard time finding what I want because there are literally dozens of screens of options. That said, I still will reach for cURL even when simpler options exist because it's ubiquitous. Same thing with Bash, grep, and sed.
- pdimitar 5y ago
  I loved using `xh` lately. https://github.com/ducaale/xh https://github.com/ducaale/xh
gavinray 5y ago
Thanks for sharing this insight, I hadn't realized it was capable of all those protocols. I fall firmly into "those 98%" you mention. Dlang's stdlib (only) implementation for making network requests is quite literally a direct "curl" wrapper: https://dlang.org/phobos/std_net_curl.html https://dlang.org/phobos/std_net_curl.html You've changed on my mind on this being an odd design decision. Baking in libcurl for the user (or optionally allowing them to dynamically link) if they want networking was maybe the most sane/pragmatic choice you could have made for a new language.
res0nat0r 5y ago
I suspect curl and rsync power more things on the internet than people realize. They are both completely amazing tools that I use almost daily.
lobocinza 5y ago
They can rewrite curl (http(s) only) in Python using requests library.

eps 5y ago

> SMB Woah. This is genuienly impressive.

WORMS_EAT_WORMS 5y ago
As someone who fell into the category of: > "Most folks I know don't even know about the other protocols curl supports and have only interfaced with curl through its http." Just curious if you could elaborate on this one specifically and why you found it especially interesting.
- mckirk 5y ago
  SMB is... not very straightforward. I needed to reverse engineer it once to write an exploit, and it's difficult to properly formulate even basic requests, let alone the more complex stuff the protocol supports.
bjoli 5y ago
Yeah. I once though I would have a look at what it would mean to just send a single file over SMB. What better way than to just reverse engineer the little bits of communication between a client and a server? The simplest solution would have been to give up the instant I started reading the catched traffic. I gave up after about 40 minutes of trying to figure out where to start.

flohofwoe 5y ago

Yeah, very easy to write an HTTP client on top of Berkley sockets. Until chunked responses come into play, and HTTPS, and HTTP/2, and HTTP/3... (this is actually something I'm very worried about, it used to be easy to cobble together a very basic HTTP client, not anymore with the new HTTP protocol versions, and all the bells and whistles).

parhamn 5y ago
Agreed. They're harder to debug, MITM, trace, etc. As a silver lining: I have been less worried recently seeing that most of the adoption of http/2-3 (at least their odd features) have been at the edge. Most developers are still writing http/1 endpoints and leaving the edge to optionally up-convert them for transport efficiency.
ricardo81 5y ago
And of course, CURL deals with multiple protocols as the OP mentioned. I've used libcurl for sending emails and FTP but mainly for HTTP. Using the multi interface it's relatively trivial (a few hundred lines of C) to fetch using hundreds or thousands of concurrent connections. There's even some courtesy opts for rate limiting per host. It's so good he/they built c-ares for asynchronous DNS lookups, IIRC something that wasn't so readily available in years gone by.

ephaeton 5y ago

> What Daniel Stenberg achieved is giving the world a fantastic, reliable, cross protocol, cross platform and cross language network library that can be used as the foundation for many projects. One of which, curl(1), is a constant ass-and-life saver for me personally and professionally. Of course the library itself ends up being used by me as well (in C, ++ and tcl). curl to me is in the league of sqlite aka "absolutely gorgeous and more often than not all you need"

krick 5y ago

True. I'm kind of surprised (again) by Daniel after 20 years of maintaining curl being so salty about "people not appreciating the achievement". Yes, for overwhelming majority of users curl in 2021 is just a simple http/https request client. And, yes, in 2021 that could be written over a weekend with a better CLI, mainly because most of the languages already have all important stuff implemented in a [standard] library (ironically, sometimes via libcurl).

bombcar 5y ago
I took it as more bemused, and given the audience I suspect it's more of a "look and laugh".
mumblemumble 5y ago
That was honestly my first thought. Of course it's easy to rewrite curl nowadays. It can't be anything more complicated than slapping a command line interface on top of libcurl...
- justin_oaks 5y ago
  At first I laughed... then I thought about the sheer number of command line options that you can find in the man pages. After thinking about that, even writing the command line interface would be a non-trivial undertaking.
  - bb010g 5y ago
    Documentation takes work too.
rkangel 5y ago
> I'm kind of surprised (again) by Daniel after 20 years of maintaining curl being so salty about "people not appreciating the achievement" You're surprised by him being salty? I can't imagine what it's like to be on the receiving end of the water torture of people continually belittling the work you have done over 20 years and given to the community.

whymarrh 5y ago

> Let's be honest here. Curl to 98% of people is http/1 requests with some post params and maybe json body, with some custom headers. 100%, that's why we see so many smaller projects pop up (on GitHub and the like) that support basically just this. No shade to those projects, improving the UI for this subset is a worthy cause, it's just not anything near cURL.

michaelbuckbee 5y ago

In the midst of so much open source drama and angst, I'm still just profoundly grateful for Daniel Stenberg's work on curl. I'm no expert on open source management but Daniel has for years been a positive and steady force working on curl. This is him publishing the direct criticisms he's receiving as "thanks" for his efforts and it guts me he's got to put up with these ignorant comments.

dkarl 5y ago

A question that arises out of this is, should the 90% use cases be handled by a small, simple tool, or by the tip of the iceberg of a large, complex tool such as curl? I can see the advantages of standardizing on a complex, powerful tool for simple use cases. For one thing, it may be the only way to standardize: simple versions are too easy to write, and therefore you get dozens of competitors, none of whom are popular enough to shake out their edge case bugs. It's also nice not to have to find, install, and learn a new tool when you stray over that 90% boundary. With software dependencies, I think the advantages of small, simple libraries win out over generality. Supporting powerful use cases makes an API more complex and often means that simple, 90% use cases require weird incantations to make them work. Here I think of the times I've had to get into the guts of Jackson despite never doing anything remotely exotic with JSON. The 95% of your codebase that can be simple should be simple, so you can devote your attention to the things that need to be complex.

rad_gruchalski 5y ago
> A question that arises out of this is, should the 90% use cases be handled by a small, simple tool, or by the tip of the iceberg of a large, complex tool such as curl? But nobody is forced to use curl? People use it because it’s convenient, shoot them selves in the foot and then lash out at the author the tool for their own choices. Where’s the fault?
- throwaway894345 5y ago
  > But nobody is forced to use curl? I like curl, but this isn't true. When you're SSH-ing onto a box, you often don't have permissions to install your favorite CLI tool, and even if you do have said permissions it's inconvenient to have to install it each time you SSH onto the box (not a major inconvenience, mind you). Moreover, in many cases you need to run a script or some other software that depends directly on curl. In general the "nobody is forced to use it" arguments rarely pan out (I remember this was a canned argument from C++ folks circa 2011: "C++ is the best language because it has every feature and if you don't like some features, you aren't forced to use them!").

spamizbad 5y ago

Why even get mad at curl? If you hate curl and all you want is a simple http/1 client just spend a few minutes to write a Python script w/requests that takes verbs and payloads from the command line rather than make the curl author's day shitty.

oxfordmale 5y ago

Curl has been a life safer for me when I had to interact with a legacy third party server. The Python request model was regularly hanging when making requests to this server, whereas I never observed these issues with curl. Curl appears to be better at dealing with such legacy edge cases as it has been around for more than two decades.

jeffrwells 5y ago

This begs the question of why all of these protocols should be in one library, instead of one each. If a project is only going to ever use http, why would I need to bundle in all of the others? Genuinely curious what the advantages are

craftinator 5y ago
My first thought is flexibility and interoperability. If you have one tool, cross platform, that supports a wide variety of protocols, you can use it on both ends of a connection, and easily swap out protocols without having to: 1) install a new tool on both sides 2) learn a new API for each protocol 3) bug check for it It also let's you do all this dynamically, so switching protocols on the fly is trivial.
lanstin 5y ago
It is a good library API with good support for async and that sort of thing. As a user it is less work to integrate the Nth protocol in a library I understand than to read the docs and try to reverse engineer some new API model of how to do things including asynchronous and configurations and TLS/cert management etc. there are a thousand ways to skin the cat, and libcurl is a good one. I want to learn a bunch of new distributed protocols because those are real things; I don’t want to learn a bunch of different APIs to do the same things because those are arbitrary wrappings to the real thing. If an API is successful enough it would be real, but cleverness in API design isn’t that useful without the popularity.
cdelsolar 5y ago
http://begthequestion.info/ http://begthequestion.info/
jolmg 5y ago
What better thing to do with a universal resource locator syntax than to have a universal client? > If a project is only going to ever use http, why would I need to bundle in all of the others? Your project might only ever think to use http, but my project might appreciate having one client for any URL, instead of having to parse URLs myself and decide on separate clients based on a part of it.

aerovistae 5y ago

I'm one of the 98%, and I don't like using cURL for this reason -- the CLI feels clunky and its gigantic number of options are distracting for my simple use case. Can anyone recommend a good alternative for the base case?

spost 5y ago
https://httpie.io/ https://httpie.io/ is pretty nice.
arodyginc 5y ago
wget maybe
justin_oaks 5y ago
I recognize the huge amount of time and effort that went into make cURL, but I agree that the gigantic number of options are distracting. When I read the cURL man page, I have a hard time finding what I want because there are literally dozens of screens of options. That said, I still will reach for cURL even when simpler options exist because it's ubiquitous. Same thing with Bash, grep, and sed.
pdimitar 5y ago
I loved using `xh` lately. https://github.com/ducaale/xh https://github.com/ducaale/xh

gavinray 5y ago

Thanks for sharing this insight, I hadn't realized it was capable of all those protocols. I fall firmly into "those 98%" you mention. Dlang's stdlib (only) implementation for making network requests is quite literally a direct "curl" wrapper: https://dlang.org/phobos/std_net_curl.html https://dlang.org/phobos/std_net_curl.html You've changed on my mind on this being an odd design decision. Baking in libcurl for the user (or optionally allowing them to dynamically link) if they want networking was maybe the most sane/pragmatic choice you could have made for a new language.

res0nat0r 5y ago

I suspect curl and rsync power more things on the internet than people realize. They are both completely amazing tools that I use almost daily.

lobocinza 5y ago

They can rewrite curl (http(s) only) in Python using requests library.