8 ms·
Curl, 17 years old today
- wging 12y agoI'm surprised it's so new. And wget is only a year older... what did people use before then?
- bagder 12y agoThose were the early days of HTTP, so no, people didn't really get a lot of HTTP "manually" like that. We used other existing tools to get gopher and ftp though. When I started my work on httpget in 1996, I didn't know about wget and I didn't become aware of it until several years later....
- hoboon 12y agoI would figure that by 1997 most people stopped using gopher/archive/veronica. I think I stopped by 1995 or so. now days we use curl/wget for web dev it seems? probably used it for the same thing back then but I also image that lots of people just used their own ad-hoc scripts or whatever; i wish i had firsthand knowledge of that. i used wget to download copies of websites when i found one that was mostly documents/images rather than webapps.
- weland 12y agoI use wget for plain, old regular downloads a lot. If I need to save a file to a specific place (e.g. datasheets to a doc/ directory in my project) and I have a terminal already open there (which I usually do), doing wget and middle-click is far more convenient than dealing with the browsers' "Save to..." dialog. Especially ever since they started considering a sane download manager to be one of those confusing features that users totally don't want.
- hoboon 12y ago> I use wget for plain, old regular downloads a lot. I loved wget for that. what are your options? Mine was always -r -p -k -nH --cut-dirs=2 -np
- barsonme 12y agoHow long have GET/POST commands been around? Probably not as long, but just curious.
- pdw 12y agoThey're mentioned in a February 1996 changelog entry for libwww-perl, so at least 19 years.
- bagder 12y agoI believe they originally come from the libwww-perl project? According to wikipedia it was first released with perl 4.0036, which was in Februrary 1993, but I can't find out if it included GET and HEAD etc already then...
- purplezky 12y agoRFC 1945 dated may 1996 mentions GET and POST methods http://www.rfc-base.org/txt/rfc-1945.txt http://www.rfc-base.org/txt/rfc-1945.txt HTTP has been in use by the World-Wide Web global information initiative since 1990. This specification reflects common usage of the protocol referred to as "HTTP/1.0".
- castell 12y agoThe first documented version of HTTP was HTTP V0.9 (1991) has only GET: http://www.w3.org/pub/WWW/Protocols/HTTP/AsImplemented.html http://www.w3.org/pub/WWW/Protocols/HTTP/AsImplemented.html Basic HTTP as defined in 1992 had GET, PUT, HEAD, POST, LINK, TEXTSEARCH, CHECKIN, etc.: http://www.w3.org/Protocols/HTTP/Methods.html http://www.w3.org/Protocols/HTTP/Methods.html More generic info: http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol
- zimpenfish 12y ago`lynx -dump` was a very useful tool.
- 72deluxe 12y agoI had to use that the other year whilst attempting to migrate an old old old Slackware box running on two Pentium 3s with an array of SCSI disks (so quite fancy for its time!). The machine had kernel sources that were different to the kernel running so no chance of compiling various bits of software, and I couldn't grab any GCC (EGCS?) binaries from anywhere. I can't remember what I eventually did but I do remember discovering lynx -dump, so thanks for the reminder!
- frabcus 12y agoI still can't stop myself typing `lynx -mime_header` to see server headers!
- heyalexej 12y agowhy not lynx -head -dump http://httpbin.org Your version dumps the body as well, which is hard to parse if you just need the headers. On another note: Thanks for curl - it's always high up in my charts somehow. Right now #2 in history 0 | awk '{print $2}' | sort | uniq -c | sort -n -r | head -n 20
- laumars 12y agoSometimes just plain telnet. It wasn't pretty, but it worked: $ telnet 80 GET / It should be noted that the above isn't a strict HTTP request header, it's missing quite a lot of detail, but it works as an example.
- josteink 12y agoPre curl/wget that was probably sufficient. HTTP/1.1 wasn't around until 1996 so back then Host: headers weren't even "optional". They simply didn't exist yet.
- gsnedders 12y agoThat's what HTTP/0.9 requests look like!
- TazeTSchnitzel 12y agoAdd two newlines and that is a perfectly valid HTTP/0.9 request.
- pjc50 12y agoIn the HTTP/1.0 days I used telnet quite a bit. Or libwww-perl, when Perl was the language of choice for web applications. LWP has copyright notice: Copyright 1995-2009, Gisle Aas Copyright 1995, Martijn Koster Round about then I wrote something similar to WWW::Mechanize in order to automate access to the Orange Web-SMS gateway, which had an inconveniently deep login procedure.
- tootie 12y agoPerl? I have only the vaguest recollection of those days. It looks like LWP is about 20 years old[1]. It's based on libwww which goes back to 1992[2]. [1] http://search.cpan.org/~ether/libwww-perl/ http://search.cpan.org/~ether/libwww-perl/ (see changelog) [2] https://en.wikipedia.org/wiki/Libwww https://en.wikipedia.org/wiki/Libwww
- w8rbt 12y agotelnet and get
- userbinator 12y agoI used netcat - and still do for interacting with protocols in ways that curl doesn't support. AFAIK it was released in 1995, so it's only three years older than curl.
- kyberias 12y ago> Rough estimates say we may have a billion users already. This cannot be true by a long shot. Or am I missing something?
- CJefferson 12y agoIf you count a user as a system which has curl installed, and has run it at some point, is curl on iOS and Android? If so, you will be in that ball park I imagine.
- Piskvorrr 12y agoI've seen it in various toasters, smart TVs and other embedded devices...so this is also a large "user" base.
- bagder 12y agoIt all depends on how you count and what a "user" is of course, but this little list helps to give a picture: http://curl.haxx.se/docs/companies.html http://curl.haxx.se/docs/companies.html
- hsx 12y agoThat wouldn't be that surprising depending on what they mean. Assuming that Google or Facebook use curl in various applications, 1 billion sounds about right.
- anonfunction 12y agoThe most common language used on websites is PHP which uses libcurl to handle HTTP requests.
- kyberias 12y agoPHP applications may use libcurl when making outbound HTTP requests, right? That is only a small subset of these web sites. Most web sites just normally process HTTP requests. This does not seem to explain the 1 billion users figure.
- escherize 12y agoI love curl so much. I just learned that you can 'copy to curl command' from the chrome inspector's network panel by right clicking on any request!! I want to make a library that reads the curl command (and maybe request syntax?) and outputs a function that will do that command.
- anonfunction 12y agoWe just built something[1] similar, but it uses the HAR request object to describe the request. [1] https://github.com/Mashape/httpsnippet https://github.com/Mashape/httpsnippet
- bagder 12y agocurl has a --libcurl option which you can add to a curl command line, and then it'll generate a libcurl-using C code template for that same operation...
- 72deluxe 12y agoThat's incredibly useful, never knew that. Thanks!
- artpar 12y agohttps://shibukawa.github.io/curl_as_dsl/ https://shibukawa.github.io/curl_as_dsl/
- sorokod 12y agoThat is very nice!
- DenisM 12y agoSo many languages supported, but no plain C?
- tlrobinson 12y agoAs someone else mentioned, this is actually built into Curl with the "--libcurl file.c" option.
- nailer 12y agoAnd still better than wget, which only does http 1.0 and this has problems due to lacking a 'host' header. Curl just works.
- leni536 12y ago> And still better than wget, which only does http 1.0 and this has problems due to lacking a 'host' header. Well I had to check, but this is not true at all: $ nc -l -p 9999 GET / HTTP/1.1 User-Agent: Wget/1.15 (linux-gnu) Accept: */* Host: localhost:9999 Connection: Keep-Alive Also wget can recursively mirror webpages and there are nice options to carefully select contents you want to download. It's quite dated though, I wish it could use an external downloader (like aria2) and only do the walking and converting links part itself.
- 72deluxe 12y agoThat mirroring option in wget has been very useful over the years, even more so if you remember to add --no-parent, else you'll end up with the ENTIRE site.
- baldfat 12y agoI love aria2c I use it all the time even as the down loader for ArchLinux's pacman (package manager).
- heyalexej 12y agoIt's also used internally in apt-fast[1] and speeds up the process by an order of magnitude for me. [1] https://github.com/ilikenwf/apt-fast https://github.com/ilikenwf/apt-fast
- 10dentious 12y agomultiple connection/source downloads is the most significant feature. Great program.
- sanqui 12y agoDo note that just a plain wget command isn't enough to properly archive/"back up" a webpage, because it doesn't save headers and other important metadata. Make sure to always use WARCs, which preserve metadata and have wonderful support, including the possibility to get absorbed into the Wayback Machine. More info: http://www.archiveteam.org/index.php?title=Wget_with_WARC_output http://www.archiveteam.org/index.php?title=Wget_with_WARC_ou... http://www.archiveteam.org/index.php?title=The_WARC_Ecosystem http://www.archiveteam.org/index.php?title=The_WARC_Ecosyste...
- pascalo 12y agoCan I just say thank you for all those hours of hard work the maintainers have put in over the years?
- xrstf 12y ago> "If it doesn't load through curl, it's broken." --someone So, so true. Thanks, curl.
- arethuza 12y agoThat's pretty much my own test for a Web based API - if I can drive it from the command line using curl then great, if I can't then it's broken.
- laurent123456 12y agoCurl is great and all but I don't think this quote makes much sense. Are there examples of services that would, for example, load in a browser but not through curl? I'd think it's rather the opposite since you can through pretty much anything at curl and it will work.
- cmg 12y agoAnything that requires JavaScript to fetch and display content.
- VMG 12y agoWell somebody tell Google and Facebook, they're probably missing out on a lot of revenue because of their broken websites.
- Piskvorrr 12y agoBroken != unprofitable.
- Millennium 12y agoWebsite != API. A cleverly-written API can exist in the same space as a Website (even the same URLs, if you differentiate by things like Accept: headers), but they're not the same thing. It's perfectly acceptable for Websites to include, and even require the use of, JavaScript. It's also perfectly acceptable to offer a JavaScript client for your API. In fact, you pretty much have to do that if you want your Website to work with the API anyway. What's not OK is for an API to require downloading and using additional JavaScript while the API is being used. As a contrived (and somewhat ludicrous) example, let's say that I queried "http://foo.com/bars" http://foo.com/bars" for some kind of collection. It's OK to return the collection. It's even OK to use an HTTP redirect if the collection actually resides elsewhere: HTTP 302, perhaps, with a Location of "http://foo.com.bazes" http://foo.com.bazes". What's not OK is to return a line of JavaScript reading "window.location = 'http://foo.com/bazes';" http://foo.com/bazes';" which might work for the browser, but wouldn't work for most other clients.
- Pephers 12y agocurl is just awesome, thanks so much to the author and all maintainers over the years! It's still my go to application for testing and debugging HTTP requests.
- noselasd 12y agoDaniel Stenberg, the maintainer, was also on the packet pusher podcast recently, talking about HTTP/2 - http://packetpushers.net/show-224-http2-its-the-biggest-network-thing-happening-on-the-internet-today-repost/ http://packetpushers.net/show-224-http2-its-the-biggest-netw...
- cdnsteve 12y agoHappy bday!
- IshKebab 12y agoCurl is pretty great to use, but be warned it has old smelly code and is probably full of security issues. I wouldn't use it for anything too critical.
- charonn0 12y agoThat's sufficiently vague to be useless. Would you care to elaborate?
- josephpmay 12y agoIt's slightly weird to me that I am older than Curl and Wget. They always seemed like Unix Monoliths to me; I had just assumed they had always existed.
- gobengo 12y agoTHIS. I have the exact same feelings. Same with the 'Kubuntu is 10 years old' link on the homepage. I think we just grew up at the perfect time!
- teamhappy 12y agoalias wget='echo "How dare you." && curl -O' brew rm wget Happy birthday.
- awalGarg 12y agoAlso, we have aria2 already :P
- T3RMINATED 12y agowho the hell is this Curl guy? who cares if hes 17 today.
- jkoudys 12y agoI was wgetting all my http requests until about 2 years ago. Curl's undeniable coolness won me over after 15 years. Now I practically live in curl when I'm setting up webservices, and libcurl for PHP does something on nearly every page request I have. For PHP people, curl_multi_exec is the new event loop.
- lloydde 12y agoI abuse curl most weeks. How many more web apps would fail at a header only request, if not for prodding from a curl user.
- tlrobinson 12y agoCurl is great, but I also recently came across HTTPie (https://github.com/jakubroztocil/httpie https://github.com/jakubroztocil/httpie) which has some nice features for playing around with HTTP APIs (JSON formatting, syntax highlighting, etc)
- deleted 12y ago[deleted]
- ddustin 12y agoI donated!
- awalGarg 12y agocUrl, I am as old as you <3