14 ms·

Curl vs. Wget

532 points by of 11y ago

geerlingguy 11y ago
Also putting this out there—for nicer REST API interaction on the CLI, and a little more user-friendliness, you might also want to add HTTPie[1] to your toolbelt. It's not going to replace curl or wget usage, but it is a nicer interface in certain circumstances. [1] https://github.com/jkbrzt/httpie https://github.com/jkbrzt/httpie
- agumonkey 11y ago
  Ha, I just got to using `jq` regularly (which is already a huge improvement). Now this ..
- wdmeldon 11y ago
  This is an extremely minor quibble, but the dependency on Python makes me less inclined to use it. I'm stuck on Windows for a lot of the work I do so configuring Python is never fun and there is no one line installation method from what I can tell. Putting it up on chocolatey[0] might be a good idea. Not sure how feasible that is however. [0] https://chocolatey.org/packages?q=httpie https://chocolatey.org/packages?q=httpie
  - recursive 11y ago
    I've never needed to do more configuration for python than whatever the installer does. What problem are you having?
    - wdmeldon 11y ago
      I've had enough problems in the past that I try to steer clear at this point. I don't do much with python so it's hard to justify spending much time on it. I think my bigger issue is the lack of being able to do cinst httpie
      - daveguy 11y ago
        The key is to stick with Python 2.7. Unfortunately pyhon 3 has significantly damaged the brand. Also installing msysgit will give you git and the msys command line which is a nice easy to install mostly UNIX command line. It is not a beast like cygwin. With msysgit (for git and msys) you can practically live in the command line on Windows and with python 2.7 the py things mostly just work. pywin32 helps for some things and if pip install doesn't work there's always the unofficial windows binaries: http://www.lfd.uci.edu/~gohlke/pythonlibs/ http://www.lfd.uci.edu/~gohlke/pythonlibs/
  - MatthewPhillips 11y ago
    That's true of running just about any command-line tool on Windows. We were promised an ecosystem of Powershell modlets that operate on objects rather than text, but that hasn't really happened.
    - wdmeldon 11y ago
      Any time I'm critical of Windows treatment of CLIs, I'm either met with "Why would I need a CLI, it's not the 80s" or "PowerShell is vastly superior to bash". The issue keeping the MS world from adopting adequate text only tools seems to be much more related to developer mindset than anything technical.
      - MatthewPhillips 11y ago
        I think it's the mindset of not being involved in open-source mostly.
  - ank_the_elder 11y ago
    Previously on HN: https://news.ycombinator.com/item?id=10418882 https://news.ycombinator.com/item?id=10418882 (that's the thread where I first found out about httpie)
  - sbierwagen 11y ago
    Should have lead that comment with "I'm on Windows, so". Of course it's going to be miserable to use any command-line tool on Windows. It's Windows.
    - kayone 11y ago
      I'm not sure why you feel like that, I switched back to Windows after ~1 year of using OSX, I wouldn't say its "miserable" there is really nothing I could do in OSX's command line that can't do in windows.
      - barbs 11y ago
        What is your Windows command-line environment? Plain CMD prompt or Cygwin? Or something else?
        kentt 11y ago
        I don't get agree with your parent, but I used cygwin when I was on Windows and it mostly does what you want if you just need basic command line tools. The major annoyances were packages were limited, compiling anything was generally a disaster and file permissions between linux/windows are a mess. I happily used it everyday though.
        gutnor 11y ago
        Powershell is very powerful and for windows it beats cygwin/mingw. Not quite sure how it measure against Linux shells running on Linux, but Microsoft has really made a proper shell for windows, too bad it looks so different. Obviously if you work in a cross-platform environment, cygwin/mingw is still the only thing that will provide you some sort of consistency in your workflow on Windows machines.
        kayone 11y ago
        I use git bash, with conemu, to be honest I don't do anything crazy but most of the linux commands I need day to day just work.
      - 11y ago
  - seanp2k2 11y ago
    +1; I likely wouldn't be able to just jump on any host and copy/paste some commands that worked from my laptop with this, but I can do that with curl and assume that it's installed and of a reasonably serviceable version basically everywhere I'd be.
  - tbrock 11y ago
    I had the same quibble, was going to write my own but found and now use bat. https://github.com/astaxie/bat/blob/master/README.md https://github.com/astaxie/bat/blob/master/README.md I don't use Windows but I also dislike python dependencies if I can get away with just needing libc.
  - vic-traill 11y ago
    I use the Babun Shell [0] on Windows desktops to provide an environment for tools more typically found and used in a Unix-like environment. I know it's not the same as installing a tool natively, but it lets me a) use some tools pretty much in the way I prefer to do so and b) check out stuff that I see on HN (and elsewhere) without a PITA install. Installing was a one line deal if pip is installed (pip install --upgrade httpie), which in turn was a one line installation (wget https://bootstrap.pypa.io/get-pip.py https://bootstrap.pypa.io/get-pip.py -O - | python) That's pretty close to one line. [0] https://babun.github.io/ https://babun.github.io/
  - jkbr 11y ago
    Relevant GitHub Issue: https://github.com/jkbrzt/httpie/issues/58 https://github.com/jkbrzt/httpie/issues/58
- Lazare 11y ago
  I've been loving HTTPie, and for my somewhat idiosyncratic usage it's completely replaced curl and wget. (Of course, most people probably don't spend as much time as me fiddling with poorly documented third party REST APIs.) Also amazing: jq.
  - broodbucket 11y ago
    I use httpie for interacting with APIs and aria2c for downloading stuff. They're both perfect at their respective jobs.
- timgilbert 11y ago
  HTTPie is amazing for working with JSON/REST interfaces. It's really succinct and generally designed with user-friendliness in mind, versus supporting every nook and cranny of the HTTP RFCs. It's installable via homebrew for Mac users. Whenever I go back to curl I feel the same minor aggravation I do when moving from homebrew back to apt (want to install? apt-get. Search? apt-cache search. List? dpkg. Remove? apt-get again, for some reason). It's just not a very user-friendly CLI and I'm constantly pulling up the curl man pages.
  - cyphar 11y ago
    Arch Linux has pacman, which is a single program that does all of the things you need from a package manager. OpenSUSE systems have zypper, which also has one command for everything (and has a much more reliable format -- rpm).
    - majewsky 11y ago
      How is RPM a more reliable format than dpkg? I've worked with both systems as part of writing a program that generates pacman/dpkg/RPM packages, and I've found dpkg to be the most sane and well-designed of the three, while the RPM format is a horrible and under-documented atrocity.
      - cyphar 11y ago
        I meant more reliable than Arch Linux's format. I haven't really worked with dpkg much, so I can't comment.
  - technomancy 11y ago
    > want to install? apt-get. Search? apt-cache search. List? dpkg. Remove? apt-get again, for some reason I think this is largely why aptitude exists; a single unified command for all common package tasks. Most of the Debian docs have been switched over to recommend its usage, but for some reason Ubuntu hasn't followed suit. That said, I just keep using the same old shell aliases I added in 2003 personally.
  - 11y ago
- hobarrera 11y ago
  I find httpie's lack of manpage quite disturbing though.
  - jkbr 11y ago
    https://github.com/jkbrzt/httpie/issues/248 https://github.com/jkbrzt/httpie/issues/248
mooreds 11y ago
"Wget can be typed in using only the left hand on a qwerty keyboard!" I love both of these, but wish that curl was just like wget in that the default behavior was to download a file, as opposed to pipe it to stdout. (Yes, aliases can help, I know.)
- Xophmeister 11y ago
  Streaming to stdout is more Unix-y, allowing you to pipe the response into further processes. For example: curl http://api.example.com/json | jq '.["someKey"]' # etc., etc.
  - mooreds 11y ago
    Fair enough. Perhaps I just need to adjust my expectations. Or maybe there's a good reason for having two tools :)
  - koolba 11y ago
    Or in modern idiomatically insecure install scripts: curl http://bogus.example.com/install | sudo bash
    - digi_owl 11y ago
      Too bad that is a reserved domain. Would be hilarious to put an actual shell script there that would echo ARF ARF ARF in a loop.
  - lomnakkus 11y ago
    It may be more unix-y, but it's less user-friendly if the expectation is to just download a file. EDIT: Wow, surprised by the downvotes. I don't think I said anything controversial (y'know principle of least surprise and all), but maybe I was being a bit too opaque: wget, by virtue of being the first on the scene, built an expectation that $THING_THAT_GETS_URLS would result in a file without any other input/arguments. Curl, to this day, surprises me because I was around when wget was all you had.
Rauchg 11y ago
I instinctively go to `wget` when I need to, uhm, get the file into my computer[1]. `curl -O` is a lot more effort :P Other than that, curl is always better. [1] Aliasing `wget` to ~`curl -O` might be a good idea :)
- cm3 11y ago
  I use wget for downloads because it follows links by default and resume is just -c. I never figured out how to make curl do the equivalent of -c.
  - mooreds 11y ago
    I don't think you can: "[Wget's] ability to recover from a prematurely broken transfer and continue downloading has no counterpart in curl."
    - JoshTriplett 11y ago
      curl -C - -O filename url -C continues from an offset; "-C -" uses the length of the output file as the offset.
      - cm3 11y ago
        That's what I tried a couple years ago and it failed. Might try it again some time.
        facetube 11y ago
        I guess it's possible that the server didn't support HTTP byte serving via Accept-Ranges:, either for that particular resource or altogether.
  - rogerbinns 11y ago
    Lets compare the length of the man page: $ man curl | wc -l 1728 $ man wget | wc -l 1096 How about the --help output? $ curl --help | wc -l 178 $ wget --help | wc -l 176 The wget help is nicer, grouping options together by category and with longer text. curl just has a long list of options in alphabetical order. How many (long) options do they have? $ curl --help | grep -- -- | wc -l 175 $ wget --help | grep -- -- | wc -l 137 I'd say it is a lot quicker to work out the flags etc you need with wget because there is less to look through.
StreamBright 11y ago
There is no other industry where tools are debated so much as in IT. We literally waste tonns of hours on arguing over minor differences and nuances that really should not matter that much.
- newjersey 11y ago
  There's a balancing act between trying to cut a tree with a blunt ax that one never resharpens versus spending all week in an ax store looking at different axes. (Speaking of which I'm doing something like that by looking at hn so I'm not trying to say bad to anyone else because I'd be a hypocrite if I did)
  - Amorymeltzer 11y ago
    "Give me six hours to chop down a tree and I will spend the first four sharpening the axe." - commonly attributed to Abraham Lincoln
- hga 11y ago
  You clearly aren't familiar with US gun culture, e.g. plastic Glock v. metal 1911, striker fired vs. Single Action, Gaston Glock vs. John Moses Browning! Light, fast 9 mm Europellet vs. heavy, slow .45 (11.5 mm) Auto Colt Pistol ... and those are just two of the most prominent right now. Let me assure you, this isn't unique to IT!
- cgh 11y ago
  I have seen guitarists spend serious time debating picks.
- Spooky23 11y ago
  It is a waste of time, clearly whet and curl are just petty toys. Emacs is obviously the one true tool to meet any purpose.
fosco 11y ago
Curl scripts allow open connection to view all new logs in a session. can wget do similar? I did not know it can or could however from my point of view if it cannot this is like comparing a philips head screwdriver to a powertool with 500pc set.
- viraptor 11y ago
  What do you even mean by logs in a session?
  - fosco 11y ago
    for example, in troubleshooting with a bluecoat proxy, I can run a curl session in conjuction with grep to check for very specific types of traffic and leave that script open while I might have an end user test.
    - viraptor 11y ago
      Sorry, can't imagine what you mean. Do you just start curl with a list of urls to process and grep for errors? Any specific examples?
      - fosco 11y ago
        it acts like a live packet capture of a log file. not sure how else to describe other than I do not think wget is capable of such functionality.
        viraptor 11y ago
        sooo... "wget -O - .... | grep ..." ?
        fosco 11y ago
        havent not worked with it in that matter, however like someone above said. typically I use wget to download, and curl to troubleshoot http https. however, if that is capable of keeping everything open I suppose thats a point for wget.
et2o 11y ago
After vi vs. emacs, this is truly the great debate of our generation.
- agumonkey 11y ago
  Come on people, who already emails with curl. Admit it.
  - notfoss 11y ago
    This would be better directed at those who use Outlook. The ones using curl to send mails will be boasting about it ;)
mynewtb 11y ago
You should all check out wpull!
- mh- 11y ago
  https://github.com/chfoo/wpull https://github.com/chfoo/wpull
- thematt 11y ago
  I love wpull! I'm actually considering using it for a large upcoming project but unfortunately there are some pretty significant bugs in their backlog. wget seems to be a bit more battle hardened.
- aendruk 11y ago
  Why?
song 11y ago
I use wget when I need to download things. curl is for everything else (love it when it comes to debugging some api)... Httpie is not bad too for debugging but most of them time I forget to use it.
ubercow 11y ago
Though only briefly mentioned in this article at the buttom, I'd like to give a huge shoutout to aria2. I use it all the time for quick torrent downloads, as it requires no daemon and just seeds until you C-c. It also does a damn good job at downloading a list of files, with multiple segments for each.
- barbs 11y ago
  +1 for aria2. It's like the VLC of command-line download tools - it Just Works with anything. https://aria2.github.io/ https://aria2.github.io/
  - toomuchtodo 11y ago
    Also can be installed with brew.
falcolas 11y ago
My favorite use of wget: mirroring web documentation to my local machine. wget -r -l5 -k -np -p https://docs.python.org/2/ Rewrites the links to point local where appropriate, and the ones which are not local remain links to the online documentation. Makes for a nice, seamless experience while browsing documentation. I also prefer wget to `curl -O` for general file downloads, simply because wget will handle redirects by default, `curl -O` will not. Yes, I could remember yet another argument to curl... but why? That said, I love curl (combined with `jq`) for playing with rest interfaces.
- look_lookatme 11y ago
  N.B., this will make curl redirect by default: echo "-L" >> ~/.curlrc
- tomphoolery 11y ago
  I bet that would be extra useful for creating Dash.app docsets...
  - pyre 11y ago
    if you look at some of the default docsets you can tell that this was most likely done. All of the directories under the file root in the docset are named after domain names (one of the things whet does when crawling a site and traversing multiple domains).
- jsmthrowaway 11y ago
  Is this the part where I tell you about pydoc and blow your mind? I had a patch at one point to make pydoc style itself just like docs.p.o, but I'm not working much in Python these days. Maybe it's not even necessary any more, which would be cool.
  - falcolas 11y ago
    Sure. Does it work with golang docs, the w3 HTML5 spec, Mozilla's JS reference, etc? ;) Python's built in documentation tools are good, but the command I referenced also gives you the full language spec locally, as well as the FAQs and packaging docs.
JonathonW 11y ago
For my usage: * Wget's the interactive, end-user tool, and my go-to if I just need to download a file. For that purpose, its defaults are more sane, its command line usage is more straightforward, its documentation is better-organized, and it can continue incomplete downloads, which curl can't. * Curl's the developer tool-- it's what I'd use if I were building a shell script that needed to download. The command line tool is more unix-y by default (outputs to stdout) and it's more flexible in terms of options. It's also present by default on more systems-- of note, OSX ships curl but not wget out of the box. Its backing library (libcurl) is also pretty nifty, but not really relevant to this comparison. This doesn't really need to be an "emacs vs. vim" or "tabs vs. spaces"-type dichotomy: wget and curl do different things well and there's no reason why both shouldn't coexist in one's workflow.
- eru 11y ago
  Looks like a frontend to curl that has wget's defaults would be useful.
  - theoh 11y ago
    Merge the code, hard link the filenames to the same executable and get it to change behavior based on the name it's invoked under.
    - kcbanner 11y ago
      Sounds like a lot of complexity for no benefit.
      - theoh 11y ago
        Either tool can provide the functionality of the other with a little work, is what I meant to imply...
      - baudehlo 11y ago
        It's what vi does.
Guruleads 11y ago
need professional hacking? contact guruleads via email : demon_teco@outlook.com
arca_vorago 11y ago
Wget is under GPLv3 so thats what I use more often. Sometimes I will use curl in certain cases, but yes, I will use a GPL product over a non-gpl product if given a choice.
keville 11y ago
Everything 6 years old is new again: https://news.ycombinator.com/item?id=1241479 https://news.ycombinator.com/item?id=1241479
- guelo 11y ago
  > Updated: February 26, 2016 17:20 (Central European, Stockholm Sweden)
xd1936 11y ago
wget has the amazing flag `--page-requisites` though, which downloads all of an html documents' css and images that you might need to display it properly. Lifesaver.
- troyvit 11y ago
  wget has another great flag, -k, which changes references to the css, js, and images to absolute URLs, resulting in a 1 page download that still looks like the original page. It's useful for making dummy pages for clients. I wish curl had this for my OSX friends who need the functionality above. Getting a wget binary onto OSX is a pain but curl is there by default.
  - hugodahl 11y ago
    Woah! Had I RTFM'd this would've saved SO. MUCH. AGGRAVATION. in recent months! Cheers!
  - goldbrick 11y ago
    wget is on homebrew.
  - arm 11y ago
    Whoa, nice, thanks for the tip! That’s amazingly useful and I seriously wish I had known about it earlier! For OS X users, you can get wget pretty easily with Homebrew¹. Just install it, then enter the following: brew install wget --with-gpgme --with-iri --with-pcre …well, those extra options aren’t strictly needed. Just what I used since I wanted wget compiled with support for those things (GnuPG Made Easy², Internationalized Resource Identifiers³, and Perl Compatible Regular Expressions⁴). You can see all the compile-time options before installing wget by typing in: brew info wget ―――――― ¹ — http://brew.sh/ http://brew.sh/ ² — https://www.gnupg.org/related_software/gpgme/ https://www.gnupg.org/related_software/gpgme/ ³ — https://en.wikipedia.org/wiki/Internationalized_resource_identifier https://en.wikipedia.org/wiki/Internationalized_resource_ide... ⁴ — https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expres...
MoSal 11y ago
I should probably write a "saldl vs. others" page someday. > Wget supports the Public Suffix List for handling cookie domains, curl does not. This is outdated info. (lib)curl can be built with libpsl support since 7.46.0.
- emj 11y ago
  Released on 2015-12-02, so it wont be in many dists for some years. :-)
  - MoSal 11y ago
    Stability and security comes first. So, let's ship an X years old curl release + patches ;)
contingencies 11y ago
If nobody's tried it, axel mentioned in the report as possibly abandoned has the awesome feature of splitting a download in to parts and then establishing that many concurrent TCP connections. Very useful on individual TCP flow rate-limited networks.
- hcarvalhoalves 11y ago
  Haven't found anything better than axel to saturate the link yet.
  - MoSal 11y ago
    Try saldl[1]. It depends on libcurl. So protocol support should be good and reliable. [1] https://github.com/saldl/saldl https://github.com/saldl/saldl
    - contingencies 11y ago
      https://github.com/saldl/saldl/wiki/saldl_vs._aria2 https://github.com/saldl/saldl/wiki/saldl_vs._aria2
      - MoSal 11y ago
        FWIW, this page is incomplete and outdated. For example, `--mirror-url` was implemented. So, it is now possible to download from two sources concurrently.
- icebraining 11y ago
  Axel is great, but nowadays I reach for aria2 instead, which can do the same, but also supports bittorrent and (s)ftp.
hartator 11y ago
> Wget requires no extra options to simply download a remote URL to a local file, while curl requires -o or -O. I think this is oddly the major reason why wget is more popular. Saving 3 chars + not having to remember the specific curl flag seems to matter more than what we can think.
- cathexis 11y ago
  I'm always amused by people who do the opposite: using wget to send get/post requests to web servers and having to add `-O /dev/null' (or, even worse `-O - > /dev/null'to keep from saving the results.
DannyBee 11y ago
Really interesting. Under curl he has: "Much more developer activity. While this can be debated, I consider three metrics here: mailing list activity, source code commit frequency and release frequency. Anyone following these two projects can see that the curl project has a lot higher pace in all these areas, and it has been so for 10+ years. Compare on openhub" Under wget he has: "GNU. Wget is part of the GNU project and all copyrights are assigned to FSF. The curl project is entirely stand-alone and independent with no organization parenting at all with almost all copyrights owned by Daniel." Daniel seems pretty wrong here. Curl does not require copyright assignment to him to contribute, and so, really, 389 people own the copyright to curl if the openhub data he points to is correct :) Even if you give it the benefit of the doubt, it's super unlikely that he owns "almost all", unless there really is not a lot of outside development activity (so this is pretty incongruous with the above statement). (I'm just about to email him with some comments about this, i just found it interesting)
- Flenser 11y ago
  "almost all copyrights owned by Daniel." and "389 people own the copyright to curl" aren't mutually exclusive. I think Daniel was saying that most of the code is copyright to him, and you are saying that the rest is copyright to 388 other people.
dorfsmay 11y ago
My usage pattern has been: - wget to download files (or entire sites even) - curl to debug everything http/https
deleted 11y ago
[deleted]
software_radio 11y ago
The war's over and Curl won. Deal with it.
jrbapna 11y ago
I like Wget's option to continue a file download if it gets interrupted. I believe you can achieve the same thing in curl but its not as simple as just setting a flag (-c).
Veratyr 11y ago
Since aria2 was only passingly mentioned, let me list some of its features: - Supports splitting and parallelising downloads. Super handy if you're on a not-so-good internet connection. - Supports bittorrent. - Can act as a server and has a really nice XML/JSON RPC interface over HTTP or WebSocket (I have a Chrome plugin that integrates with this pretty nicely). They're not super important features sure but I stick with it because it's typically the fastest tool and I hate waiting.
cushychicken 11y ago
The "only need a left hand" sways me for wget.
dustingetz 11y ago
Who funds projects like this?
ProceedsNow 11y ago
Wget just werks.
ProceedsNow 11y ago
Wget just werks.
deleted 11y ago
[deleted]
dominhhai 11y ago
>> Wget can be typed in using only the left hand on a qwerty keyboard! Great!
module17 11y ago
TLDR: curl rocks.
- lucb1e 11y ago
  Unless you want to download a web site for archiving purposes.
jonalmeida 11y ago
wget was the first one I learned how to use by trying to recursively download a professor's course website for offline use, and then learning that they hosted the solutions to the assignments there as well.. I did well in that course, granted it was an easy intro to programming one. ;)
X-Istence 11y ago
I use curl because it is generally installed. I prefer not to install wget, especially on customer machines because it stops 90% of script kiddies. For some reason wget is the only tool they will attempt to use to download their sploit.
- lucb1e 11y ago
  Pretty sure skiddies will not assume most victims have wget already, they'll just ship it with the exploit. If not installing wget is an annoyance to a hacker, they're already in too deep ;)
  - X-Istence 11y ago
    Your stock standard drive by PHP exploit attempts usually attempt to "wget" another PHP file to public_html. They try wget, fail, and move on.
dallbee 11y ago
We are forgetting our long lost cousin, fetch. http://www.unix.com/man-page/FreeBSD/1/FETCH/ http://www.unix.com/man-page/FreeBSD/1/FETCH/
nowprovision 11y ago
Unmentioned in the article - Curl supports --resolve, this single feature helps us test all sorts of scenarios for HTTPS and hostname based multiplexing where DNS isn't updated or consistent yet, e.g. transferring site, bringing up cold standbys, couldn't live without it (well I could if I wanted to edit /etc/hosts continuously)
josteink 11y ago
For me defaults matter... 99% of the time when I want to use wget or curl, I want to do it to download a file, so I can keep working on it, from the filesystem. wget does that without any parameters. Curl requires me to remember and provide parameters for this obvious usecase. So wget wins every time.
sametmax 11y ago
Nowaday I just use httpie. It's in Python, so easy to install in windows, and let me work easily with requests and responses, inspect the content, add coloration, etc. Plus the syntax is much easier.
hmsimha 11y ago
Curl gets another point for having better SNI support, as wget versions until relatively recently didn't support it. This means you can't securely download content using relatively recent (but not the newest) versions of wget (such as any in the Ubuntu 12.04 repos) from a server which uses SNI, unless the domain you're requesting happens to be the default for the server. As an example, I found the file https://redbot.org/static/style.css https://redbot.org/static/style.css only accessible with SNI. Try `wget https://redbot.org/static/style.css` https://redbot.org/static/style.css` vs. `curl -O https://redbot.org/static/style.css` https://redbot.org/static/style.css` on Ubuntu 12.04. Domain names which point to S3 buckets (and likely other CDNs) will have similar issues.
mistat 11y ago
curl for checking http headers simply with: curl -vskL http:1.2.3.4 -H "Host: example.com" > /dev/null
- ce4 11y ago
  Why don't you just use the -I flag?
  - mistat 11y ago
    For some sites a HEAD may return different headers than GET, so it is safer to return the results in full. Also using vsk shows the request headers, including IP so you can easily see if things such as roundrobin DNS is in use, again to assist with debugging.
haxpor 11y ago
For certain case like creating a Telegram bot which has no interaction with browser, do you think we can make use of curl (post request) to make PHP session works? As there's no browser interaction in Telegram bot, the script just receives response back from Telegram server. This might help to kerp track of user state without a need of db?
awjr 11y ago
Useful to know: If you use the Chrome dev tools, in the network tab, you can right click on a request and "Copy as cURL".
notfoss 11y ago
aria2 is much more reliable when downloading stuff, especially for links which involve redirections. For example here's a link to download 7zip for windows from filehippo.com. Results: * Curl doesn't download it at all. curl -O 'http://filehippo.com/download/file/bf0c7e39c244b0910cfcfaef2af45de88d8cae8cc0f55350074bf1664fbb698d/' gives: curl: Remote file name has no length! * Wget manages to download the file, but with the wrong name. wget 'http://filehippo.com/download/file/bf0c7e39c244b0910cfcfaef2af45de88d8cae8cc0f55350074bf1664fbb698d/' gives: 2016-03-03 18:08:21 (75.9 KB/s) - ‘index.html’ saved [1371668/1371668] * aria2 manages to download the file with the correct name with no additional switches. aria2c 'http://filehippo.com/download/file/bf0c7e39c244b0910cfcfaef2af45de88d8cae8cc0f55350074bf1664fbb698d/' gives: 03/03 18:08:45 [NOTICE] Download complete: /tmp/7z1514-x64.exe
- MoSal 11y ago
  The URL does not work right now. But I tried another one from the same site. No client can get this right, always. aria2c is not more reliable. It's just choosing to take the filename from the redirect URL. It appears to be the right thing to do in this case. But it would fail if the start URL was actually the one that had the right filename. Hosts can use the Content-Disposition header if they want to make sure all (capable) clients get the right filename. In saldl, I implemented `--filename-from-redirect` to handle your use-case. But It's not set by default.
  - notfoss 11y ago
    Thanks for the explanation. But generally I have found aria2 to be more reliable in such scenarios.

532 points by of 11y ago

Also putting this out there—for nicer REST API interaction on the CLI, and a little more user-friendliness, you might also want to add HTTPie[1] to your toolbelt. It's not going to replace curl or wget usage, but it is a nicer interface in certain circumstances. [1] https://github.com/jkbrzt/httpie https://github.com/jkbrzt/httpie

agumonkey 11y ago
Ha, I just got to using `jq` regularly (which is already a huge improvement). Now this ..
wdmeldon 11y ago
This is an extremely minor quibble, but the dependency on Python makes me less inclined to use it. I'm stuck on Windows for a lot of the work I do so configuring Python is never fun and there is no one line installation method from what I can tell. Putting it up on chocolatey[0] might be a good idea. Not sure how feasible that is however. [0] https://chocolatey.org/packages?q=httpie https://chocolatey.org/packages?q=httpie
- recursive 11y ago
  I've never needed to do more configuration for python than whatever the installer does. What problem are you having?
  - wdmeldon 11y ago
    I've had enough problems in the past that I try to steer clear at this point. I don't do much with python so it's hard to justify spending much time on it. I think my bigger issue is the lack of being able to do cinst httpie
    - daveguy 11y ago
      The key is to stick with Python 2.7. Unfortunately pyhon 3 has significantly damaged the brand. Also installing msysgit will give you git and the msys command line which is a nice easy to install mostly UNIX command line. It is not a beast like cygwin. With msysgit (for git and msys) you can practically live in the command line on Windows and with python 2.7 the py things mostly just work. pywin32 helps for some things and if pip install doesn't work there's always the unofficial windows binaries: http://www.lfd.uci.edu/~gohlke/pythonlibs/ http://www.lfd.uci.edu/~gohlke/pythonlibs/
- MatthewPhillips 11y ago
  That's true of running just about any command-line tool on Windows. We were promised an ecosystem of Powershell modlets that operate on objects rather than text, but that hasn't really happened.
  - wdmeldon 11y ago
    Any time I'm critical of Windows treatment of CLIs, I'm either met with "Why would I need a CLI, it's not the 80s" or "PowerShell is vastly superior to bash". The issue keeping the MS world from adopting adequate text only tools seems to be much more related to developer mindset than anything technical.
    - MatthewPhillips 11y ago
      I think it's the mindset of not being involved in open-source mostly.
- ank_the_elder 11y ago
  Previously on HN: https://news.ycombinator.com/item?id=10418882 https://news.ycombinator.com/item?id=10418882 (that's the thread where I first found out about httpie)
- sbierwagen 11y ago
  Should have lead that comment with "I'm on Windows, so". Of course it's going to be miserable to use any command-line tool on Windows. It's Windows.
  - kayone 11y ago
    I'm not sure why you feel like that, I switched back to Windows after ~1 year of using OSX, I wouldn't say its "miserable" there is really nothing I could do in OSX's command line that can't do in windows.
    - barbs 11y ago
      What is your Windows command-line environment? Plain CMD prompt or Cygwin? Or something else?
      - kentt 11y ago
        I don't get agree with your parent, but I used cygwin when I was on Windows and it mostly does what you want if you just need basic command line tools. The major annoyances were packages were limited, compiling anything was generally a disaster and file permissions between linux/windows are a mess. I happily used it everyday though.
      - gutnor 11y ago
        Powershell is very powerful and for windows it beats cygwin/mingw. Not quite sure how it measure against Linux shells running on Linux, but Microsoft has really made a proper shell for windows, too bad it looks so different. Obviously if you work in a cross-platform environment, cygwin/mingw is still the only thing that will provide you some sort of consistency in your workflow on Windows machines.
      - kayone 11y ago
        I use git bash, with conemu, to be honest I don't do anything crazy but most of the linux commands I need day to day just work.
    - 11y ago
- seanp2k2 11y ago
  +1; I likely wouldn't be able to just jump on any host and copy/paste some commands that worked from my laptop with this, but I can do that with curl and assume that it's installed and of a reasonably serviceable version basically everywhere I'd be.
- tbrock 11y ago
  I had the same quibble, was going to write my own but found and now use bat. https://github.com/astaxie/bat/blob/master/README.md https://github.com/astaxie/bat/blob/master/README.md I don't use Windows but I also dislike python dependencies if I can get away with just needing libc.
- vic-traill 11y ago
  I use the Babun Shell [0] on Windows desktops to provide an environment for tools more typically found and used in a Unix-like environment. I know it's not the same as installing a tool natively, but it lets me a) use some tools pretty much in the way I prefer to do so and b) check out stuff that I see on HN (and elsewhere) without a PITA install. Installing was a one line deal if pip is installed (pip install --upgrade httpie), which in turn was a one line installation (wget https://bootstrap.pypa.io/get-pip.py https://bootstrap.pypa.io/get-pip.py -O - | python) That's pretty close to one line. [0] https://babun.github.io/ https://babun.github.io/
- jkbr 11y ago
  Relevant GitHub Issue: https://github.com/jkbrzt/httpie/issues/58 https://github.com/jkbrzt/httpie/issues/58
Lazare 11y ago
I've been loving HTTPie, and for my somewhat idiosyncratic usage it's completely replaced curl and wget. (Of course, most people probably don't spend as much time as me fiddling with poorly documented third party REST APIs.) Also amazing: jq.
- broodbucket 11y ago
  I use httpie for interacting with APIs and aria2c for downloading stuff. They're both perfect at their respective jobs.
timgilbert 11y ago
HTTPie is amazing for working with JSON/REST interfaces. It's really succinct and generally designed with user-friendliness in mind, versus supporting every nook and cranny of the HTTP RFCs. It's installable via homebrew for Mac users. Whenever I go back to curl I feel the same minor aggravation I do when moving from homebrew back to apt (want to install? apt-get. Search? apt-cache search. List? dpkg. Remove? apt-get again, for some reason). It's just not a very user-friendly CLI and I'm constantly pulling up the curl man pages.
- cyphar 11y ago
  Arch Linux has pacman, which is a single program that does all of the things you need from a package manager. OpenSUSE systems have zypper, which also has one command for everything (and has a much more reliable format -- rpm).
  - majewsky 11y ago
    How is RPM a more reliable format than dpkg? I've worked with both systems as part of writing a program that generates pacman/dpkg/RPM packages, and I've found dpkg to be the most sane and well-designed of the three, while the RPM format is a horrible and under-documented atrocity.
    - cyphar 11y ago
      I meant more reliable than Arch Linux's format. I haven't really worked with dpkg much, so I can't comment.
- technomancy 11y ago
  > want to install? apt-get. Search? apt-cache search. List? dpkg. Remove? apt-get again, for some reason I think this is largely why aptitude exists; a single unified command for all common package tasks. Most of the Debian docs have been switched over to recommend its usage, but for some reason Ubuntu hasn't followed suit. That said, I just keep using the same old shell aliases I added in 2003 personally.
- 11y ago
hobarrera 11y ago
I find httpie's lack of manpage quite disturbing though.
- jkbr 11y ago
  https://github.com/jkbrzt/httpie/issues/248 https://github.com/jkbrzt/httpie/issues/248

mooreds 11y ago

"Wget can be typed in using only the left hand on a qwerty keyboard!" I love both of these, but wish that curl was just like wget in that the default behavior was to download a file, as opposed to pipe it to stdout. (Yes, aliases can help, I know.)

Xophmeister 11y ago
Streaming to stdout is more Unix-y, allowing you to pipe the response into further processes. For example: curl http://api.example.com/json | jq '.["someKey"]' # etc., etc.
- mooreds 11y ago
  Fair enough. Perhaps I just need to adjust my expectations. Or maybe there's a good reason for having two tools :)
- koolba 11y ago
  Or in modern idiomatically insecure install scripts: curl http://bogus.example.com/install | sudo bash
  - digi_owl 11y ago
    Too bad that is a reserved domain. Would be hilarious to put an actual shell script there that would echo ARF ARF ARF in a loop.
- lomnakkus 11y ago
  It may be more unix-y, but it's less user-friendly if the expectation is to just download a file. EDIT: Wow, surprised by the downvotes. I don't think I said anything controversial (y'know principle of least surprise and all), but maybe I was being a bit too opaque: wget, by virtue of being the first on the scene, built an expectation that $THING_THAT_GETS_URLS would result in a file without any other input/arguments. Curl, to this day, surprises me because I was around when wget was all you had.

Rauchg 11y ago

I instinctively go to `wget` when I need to, uhm, get the file into my computer[1]. `curl -O` is a lot more effort :P Other than that, curl is always better. [1] Aliasing `wget` to ~`curl -O` might be a good idea :)

cm3 11y ago
I use wget for downloads because it follows links by default and resume is just -c. I never figured out how to make curl do the equivalent of -c.
- mooreds 11y ago
  I don't think you can: "[Wget's] ability to recover from a prematurely broken transfer and continue downloading has no counterpart in curl."
  - JoshTriplett 11y ago
    curl -C - -O filename url -C continues from an offset; "-C -" uses the length of the output file as the offset.
    - cm3 11y ago
      That's what I tried a couple years ago and it failed. Might try it again some time.
      - facetube 11y ago
        I guess it's possible that the server didn't support HTTP byte serving via Accept-Ranges:, either for that particular resource or altogether.
- rogerbinns 11y ago
  Lets compare the length of the man page: $ man curl | wc -l 1728 $ man wget | wc -l 1096 How about the --help output? $ curl --help | wc -l 178 $ wget --help | wc -l 176 The wget help is nicer, grouping options together by category and with longer text. curl just has a long list of options in alphabetical order. How many (long) options do they have? $ curl --help | grep -- -- | wc -l 175 $ wget --help | grep -- -- | wc -l 137 I'd say it is a lot quicker to work out the flags etc you need with wget because there is less to look through.

StreamBright 11y ago

There is no other industry where tools are debated so much as in IT. We literally waste tonns of hours on arguing over minor differences and nuances that really should not matter that much.

newjersey 11y ago
There's a balancing act between trying to cut a tree with a blunt ax that one never resharpens versus spending all week in an ax store looking at different axes. (Speaking of which I'm doing something like that by looking at hn so I'm not trying to say bad to anyone else because I'd be a hypocrite if I did)
- Amorymeltzer 11y ago
  "Give me six hours to chop down a tree and I will spend the first four sharpening the axe." - commonly attributed to Abraham Lincoln
hga 11y ago
You clearly aren't familiar with US gun culture, e.g. plastic Glock v. metal 1911, striker fired vs. Single Action, Gaston Glock vs. John Moses Browning! Light, fast 9 mm Europellet vs. heavy, slow .45 (11.5 mm) Auto Colt Pistol ... and those are just two of the most prominent right now. Let me assure you, this isn't unique to IT!
cgh 11y ago
I have seen guitarists spend serious time debating picks.
Spooky23 11y ago
It is a waste of time, clearly whet and curl are just petty toys. Emacs is obviously the one true tool to meet any purpose.

fosco 11y ago

Curl scripts allow open connection to view all new logs in a session. can wget do similar? I did not know it can or could however from my point of view if it cannot this is like comparing a philips head screwdriver to a powertool with 500pc set.

viraptor 11y ago
What do you even mean by logs in a session?
- fosco 11y ago
  for example, in troubleshooting with a bluecoat proxy, I can run a curl session in conjuction with grep to check for very specific types of traffic and leave that script open while I might have an end user test.
  - viraptor 11y ago
    Sorry, can't imagine what you mean. Do you just start curl with a list of urls to process and grep for errors? Any specific examples?
    - fosco 11y ago
      it acts like a live packet capture of a log file. not sure how else to describe other than I do not think wget is capable of such functionality.
      - viraptor 11y ago
        sooo... "wget -O - .... | grep ..." ?
        fosco 11y ago
        havent not worked with it in that matter, however like someone above said. typically I use wget to download, and curl to troubleshoot http https. however, if that is capable of keeping everything open I suppose thats a point for wget.

et2o 11y ago

After vi vs. emacs, this is truly the great debate of our generation.

agumonkey 11y ago
Come on people, who already emails with curl. Admit it.
- notfoss 11y ago
  This would be better directed at those who use Outlook. The ones using curl to send mails will be boasting about it ;)

mynewtb 11y ago

You should all check out wpull!

mh- 11y ago
https://github.com/chfoo/wpull https://github.com/chfoo/wpull
thematt 11y ago
I love wpull! I'm actually considering using it for a large upcoming project but unfortunately there are some pretty significant bugs in their backlog. wget seems to be a bit more battle hardened.
aendruk 11y ago
Why?

song 11y ago

I use wget when I need to download things. curl is for everything else (love it when it comes to debugging some api)... Httpie is not bad too for debugging but most of them time I forget to use it.

ubercow 11y ago

Though only briefly mentioned in this article at the buttom, I'd like to give a huge shoutout to aria2. I use it all the time for quick torrent downloads, as it requires no daemon and just seeds until you C-c. It also does a damn good job at downloading a list of files, with multiple segments for each.

barbs 11y ago
+1 for aria2. It's like the VLC of command-line download tools - it Just Works with anything. https://aria2.github.io/ https://aria2.github.io/
- toomuchtodo 11y ago
  Also can be installed with brew.

falcolas 11y ago

My favorite use of wget: mirroring web documentation to my local machine. wget -r -l5 -k -np -p https://docs.python.org/2/ Rewrites the links to point local where appropriate, and the ones which are not local remain links to the online documentation. Makes for a nice, seamless experience while browsing documentation. I also prefer wget to `curl -O` for general file downloads, simply because wget will handle redirects by default, `curl -O` will not. Yes, I could remember yet another argument to curl... but why? That said, I love curl (combined with `jq`) for playing with rest interfaces.

look_lookatme 11y ago
N.B., this will make curl redirect by default: echo "-L" >> ~/.curlrc
tomphoolery 11y ago
I bet that would be extra useful for creating Dash.app docsets...
- pyre 11y ago
  if you look at some of the default docsets you can tell that this was most likely done. All of the directories under the file root in the docset are named after domain names (one of the things whet does when crawling a site and traversing multiple domains).
jsmthrowaway 11y ago
Is this the part where I tell you about pydoc and blow your mind? I had a patch at one point to make pydoc style itself just like docs.p.o, but I'm not working much in Python these days. Maybe it's not even necessary any more, which would be cool.
- falcolas 11y ago
  Sure. Does it work with golang docs, the w3 HTML5 spec, Mozilla's JS reference, etc? ;) Python's built in documentation tools are good, but the command I referenced also gives you the full language spec locally, as well as the FAQs and packaging docs.

JonathonW 11y ago

For my usage: * Wget's the interactive, end-user tool, and my go-to if I just need to download a file. For that purpose, its defaults are more sane, its command line usage is more straightforward, its documentation is better-organized, and it can continue incomplete downloads, which curl can't. * Curl's the developer tool-- it's what I'd use if I were building a shell script that needed to download. The command line tool is more unix-y by default (outputs to stdout) and it's more flexible in terms of options. It's also present by default on more systems-- of note, OSX ships curl but not wget out of the box. Its backing library (libcurl) is also pretty nifty, but not really relevant to this comparison. This doesn't really need to be an "emacs vs. vim" or "tabs vs. spaces"-type dichotomy: wget and curl do different things well and there's no reason why both shouldn't coexist in one's workflow.

eru 11y ago
Looks like a frontend to curl that has wget's defaults would be useful.
- theoh 11y ago
  Merge the code, hard link the filenames to the same executable and get it to change behavior based on the name it's invoked under.
  - kcbanner 11y ago
    Sounds like a lot of complexity for no benefit.
    - theoh 11y ago
      Either tool can provide the functionality of the other with a little work, is what I meant to imply...
    - baudehlo 11y ago
      It's what vi does.

Guruleads 11y ago

need professional hacking? contact guruleads via email : demon_teco@outlook.com

arca_vorago 11y ago

Wget is under GPLv3 so thats what I use more often. Sometimes I will use curl in certain cases, but yes, I will use a GPL product over a non-gpl product if given a choice.

keville 11y ago

Everything 6 years old is new again: https://news.ycombinator.com/item?id=1241479 https://news.ycombinator.com/item?id=1241479

guelo 11y ago
> Updated: February 26, 2016 17:20 (Central European, Stockholm Sweden)

xd1936 11y ago

wget has the amazing flag `--page-requisites` though, which downloads all of an html documents' css and images that you might need to display it properly. Lifesaver.

troyvit 11y ago
wget has another great flag, -k, which changes references to the css, js, and images to absolute URLs, resulting in a 1 page download that still looks like the original page. It's useful for making dummy pages for clients. I wish curl had this for my OSX friends who need the functionality above. Getting a wget binary onto OSX is a pain but curl is there by default.
- hugodahl 11y ago
  Woah! Had I RTFM'd this would've saved SO. MUCH. AGGRAVATION. in recent months! Cheers!
- goldbrick 11y ago
  wget is on homebrew.
- arm 11y ago
  Whoa, nice, thanks for the tip! That’s amazingly useful and I seriously wish I had known about it earlier! For OS X users, you can get wget pretty easily with Homebrew¹. Just install it, then enter the following: brew install wget --with-gpgme --with-iri --with-pcre …well, those extra options aren’t strictly needed. Just what I used since I wanted wget compiled with support for those things (GnuPG Made Easy², Internationalized Resource Identifiers³, and Perl Compatible Regular Expressions⁴). You can see all the compile-time options before installing wget by typing in: brew info wget ―――――― ¹ — http://brew.sh/ http://brew.sh/ ² — https://www.gnupg.org/related_software/gpgme/ https://www.gnupg.org/related_software/gpgme/ ³ — https://en.wikipedia.org/wiki/Internationalized_resource_identifier https://en.wikipedia.org/wiki/Internationalized_resource_ide... ⁴ — https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expres...

MoSal 11y ago

I should probably write a "saldl vs. others" page someday. > Wget supports the Public Suffix List for handling cookie domains, curl does not. This is outdated info. (lib)curl can be built with libpsl support since 7.46.0.

emj 11y ago
Released on 2015-12-02, so it wont be in many dists for some years. :-)
- MoSal 11y ago
  Stability and security comes first. So, let's ship an X years old curl release + patches ;)

contingencies 11y ago

If nobody's tried it, axel mentioned in the report as possibly abandoned has the awesome feature of splitting a download in to parts and then establishing that many concurrent TCP connections. Very useful on individual TCP flow rate-limited networks.

hcarvalhoalves 11y ago
Haven't found anything better than axel to saturate the link yet.
- MoSal 11y ago
  Try saldl[1]. It depends on libcurl. So protocol support should be good and reliable. [1] https://github.com/saldl/saldl https://github.com/saldl/saldl
  - contingencies 11y ago
    https://github.com/saldl/saldl/wiki/saldl_vs._aria2 https://github.com/saldl/saldl/wiki/saldl_vs._aria2
    - MoSal 11y ago
      FWIW, this page is incomplete and outdated. For example, `--mirror-url` was implemented. So, it is now possible to download from two sources concurrently.
icebraining 11y ago
Axel is great, but nowadays I reach for aria2 instead, which can do the same, but also supports bittorrent and (s)ftp.

hartator 11y ago

> Wget requires no extra options to simply download a remote URL to a local file, while curl requires -o or -O. I think this is oddly the major reason why wget is more popular. Saving 3 chars + not having to remember the specific curl flag seems to matter more than what we can think.

cathexis 11y ago
I'm always amused by people who do the opposite: using wget to send get/post requests to web servers and having to add `-O /dev/null' (or, even worse `-O - > /dev/null'to keep from saving the results.

DannyBee 11y ago

Really interesting. Under curl he has: "Much more developer activity. While this can be debated, I consider three metrics here: mailing list activity, source code commit frequency and release frequency. Anyone following these two projects can see that the curl project has a lot higher pace in all these areas, and it has been so for 10+ years. Compare on openhub" Under wget he has: "GNU. Wget is part of the GNU project and all copyrights are assigned to FSF. The curl project is entirely stand-alone and independent with no organization parenting at all with almost all copyrights owned by Daniel." Daniel seems pretty wrong here. Curl does not require copyright assignment to him to contribute, and so, really, 389 people own the copyright to curl if the openhub data he points to is correct :) Even if you give it the benefit of the doubt, it's super unlikely that he owns "almost all", unless there really is not a lot of outside development activity (so this is pretty incongruous with the above statement). (I'm just about to email him with some comments about this, i just found it interesting)

Flenser 11y ago
"almost all copyrights owned by Daniel." and "389 people own the copyright to curl" aren't mutually exclusive. I think Daniel was saying that most of the code is copyright to him, and you are saying that the rest is copyright to 388 other people.

dorfsmay 11y ago

My usage pattern has been: - wget to download files (or entire sites even) - curl to debug everything http/https

deleted 11y ago

[deleted]

software_radio 11y ago

The war's over and Curl won. Deal with it.

jrbapna 11y ago

I like Wget's option to continue a file download if it gets interrupted. I believe you can achieve the same thing in curl but its not as simple as just setting a flag (-c).

Veratyr 11y ago

Since aria2 was only passingly mentioned, let me list some of its features: - Supports splitting and parallelising downloads. Super handy if you're on a not-so-good internet connection. - Supports bittorrent. - Can act as a server and has a really nice XML/JSON RPC interface over HTTP or WebSocket (I have a Chrome plugin that integrates with this pretty nicely). They're not super important features sure but I stick with it because it's typically the fastest tool and I hate waiting.

cushychicken 11y ago

The "only need a left hand" sways me for wget.

dustingetz 11y ago

Who funds projects like this?

ProceedsNow 11y ago

Wget just werks.

ProceedsNow 11y ago

Wget just werks.

deleted 11y ago

[deleted]

dominhhai 11y ago

>> Wget can be typed in using only the left hand on a qwerty keyboard! Great!

module17 11y ago

TLDR: curl rocks.

lucb1e 11y ago
Unless you want to download a web site for archiving purposes.

jonalmeida 11y ago

wget was the first one I learned how to use by trying to recursively download a professor's course website for offline use, and then learning that they hosted the solutions to the assignments there as well.. I did well in that course, granted it was an easy intro to programming one. ;)

X-Istence 11y ago

I use curl because it is generally installed. I prefer not to install wget, especially on customer machines because it stops 90% of script kiddies. For some reason wget is the only tool they will attempt to use to download their sploit.

lucb1e 11y ago
Pretty sure skiddies will not assume most victims have wget already, they'll just ship it with the exploit. If not installing wget is an annoyance to a hacker, they're already in too deep ;)
- X-Istence 11y ago
  Your stock standard drive by PHP exploit attempts usually attempt to "wget" another PHP file to public_html. They try wget, fail, and move on.

dallbee 11y ago

We are forgetting our long lost cousin, fetch. http://www.unix.com/man-page/FreeBSD/1/FETCH/ http://www.unix.com/man-page/FreeBSD/1/FETCH/

nowprovision 11y ago

Unmentioned in the article - Curl supports --resolve, this single feature helps us test all sorts of scenarios for HTTPS and hostname based multiplexing where DNS isn't updated or consistent yet, e.g. transferring site, bringing up cold standbys, couldn't live without it (well I could if I wanted to edit /etc/hosts continuously)

josteink 11y ago

For me defaults matter... 99% of the time when I want to use wget or curl, I want to do it to download a file, so I can keep working on it, from the filesystem. wget does that without any parameters. Curl requires me to remember and provide parameters for this obvious usecase. So wget wins every time.

sametmax 11y ago

Nowaday I just use httpie. It's in Python, so easy to install in windows, and let me work easily with requests and responses, inspect the content, add coloration, etc. Plus the syntax is much easier.

hmsimha 11y ago

Curl gets another point for having better SNI support, as wget versions until relatively recently didn't support it. This means you can't securely download content using relatively recent (but not the newest) versions of wget (such as any in the Ubuntu 12.04 repos) from a server which uses SNI, unless the domain you're requesting happens to be the default for the server. As an example, I found the file https://redbot.org/static/style.css https://redbot.org/static/style.css only accessible with SNI. Try `wget https://redbot.org/static/style.css` https://redbot.org/static/style.css` vs. `curl -O https://redbot.org/static/style.css` https://redbot.org/static/style.css` on Ubuntu 12.04. Domain names which point to S3 buckets (and likely other CDNs) will have similar issues.

mistat 11y ago

curl for checking http headers simply with: curl -vskL http:1.2.3.4 -H "Host: example.com" > /dev/null

ce4 11y ago
Why don't you just use the -I flag?
- mistat 11y ago
  For some sites a HEAD may return different headers than GET, so it is safer to return the results in full. Also using vsk shows the request headers, including IP so you can easily see if things such as roundrobin DNS is in use, again to assist with debugging.

haxpor 11y ago

For certain case like creating a Telegram bot which has no interaction with browser, do you think we can make use of curl (post request) to make PHP session works? As there's no browser interaction in Telegram bot, the script just receives response back from Telegram server. This might help to kerp track of user state without a need of db?

awjr 11y ago

Useful to know: If you use the Chrome dev tools, in the network tab, you can right click on a request and "Copy as cURL".

notfoss 11y ago

aria2 is much more reliable when downloading stuff, especially for links which involve redirections. For example here's a link to download 7zip for windows from filehippo.com. Results: * Curl doesn't download it at all. curl -O 'http://filehippo.com/download/file/bf0c7e39c244b0910cfcfaef2af45de88d8cae8cc0f55350074bf1664fbb698d/' gives: curl: Remote file name has no length! * Wget manages to download the file, but with the wrong name. wget 'http://filehippo.com/download/file/bf0c7e39c244b0910cfcfaef2af45de88d8cae8cc0f55350074bf1664fbb698d/' gives: 2016-03-03 18:08:21 (75.9 KB/s) - ‘index.html’ saved [1371668/1371668] * aria2 manages to download the file with the correct name with no additional switches. aria2c 'http://filehippo.com/download/file/bf0c7e39c244b0910cfcfaef2af45de88d8cae8cc0f55350074bf1664fbb698d/' gives: 03/03 18:08:45 [NOTICE] Download complete: /tmp/7z1514-x64.exe

MoSal 11y ago
The URL does not work right now. But I tried another one from the same site. No client can get this right, always. aria2c is not more reliable. It's just choosing to take the filename from the redirect URL. It appears to be the right thing to do in this case. But it would fail if the start URL was actually the one that had the right filename. Hosts can use the Content-Disposition header if they want to make sure all (capable) clients get the right filename. In saldl, I implemented `--filename-from-redirect` to handle your use-case. But It's not set by default.
- notfoss 11y ago
  Thanks for the explanation. But generally I have found aria2 to be more reliable in such scenarios.