7 ms·

Curl vs Wget

• dlsspy 17y ago
  "libcurl does not put any restrictions on the program that uses the library." I wonder how much of the activity could be attributed to such things.
• henning 17y ago
  Probably wget's major reason for being is mirroring websites. It does this well. Curl is a more general HTTP library with a command line front end. And of course there are GUI frontends for wget, but it's not really intended to be used as a library. I don't see a problem here.
• ighost 17y ago
  kinda banal if you ask me. it's two simple tools that do approximately the same thing. even grep vs ack would be more interesting. no offense to the authors of course, they did a great job in both cases.
  • sorbits 17y ago
    As someone who have always used curl and wondered why I often see wget quoted in READMEs (as how to obtain some remote file), I quite enjoyed reading the write-up. My take-away was that a) I am not missing out by staying with curl and b) the use of wget is likely due to being part of GNU.
    • Maskawanian 17y ago
      I use wget for one shot downloads because it is less to type. I use libcURL when developing. This is the difference between "wget http://www.google.ca/ http://www.google.ca/ and "curl -O http://www.google.ca/ http://www.google.ca/ Just a simple small but significant difference.
    • AlisdairO 17y ago
      Well, plus the recursive download facilities. This is the reason why I mostly use wget on the command line, although I'd use curl in scripts.
    • ZeroGravitas 17y ago
      You did notice this was written by curl's author? I only noticed after I read it and without that knowledge thought it was rather a pathetically biased article. But you can't expect developers to be impartial about their own code and it was my fault for not noticing the disclaimer the first time. It reads completely different with that framing in mind, but still if I was looking for the definitive reason that one tool was recommended over another I'd consider the point of view of the developers as a starting point rather than an open and shut case.
  • arethuza 17y ago
    I'm pretty sure awk is Turing Complete while grep isn't.
    • ZeroGravitas 17y ago
      He said ack, not awk: "ack is a tool like grep, designed for programmers with large trees of heterogeneous source code." http://betterthangrep.com/ http://betterthangrep.com/
• mikeytown2 17y ago
  What I do * curl for any bash scripts * wget for downloading via command line The error codes that curl output make it very powerful when creating a script. Something that gets run on cron & you want to know why it failed if it did. Wget is simple, and for a shell it's what I always turn to. The only time I would use wget in a script is if I was making some sort of simple web crawler script.
  • sliverstorm 17y ago
    same here. I use wget to do stuff, and I use curl to do complicated/fancy stuff. One is simple, one is powerful. I'll use simple whenever I can get away with it.
• statictype 17y ago
  >Wget requires no extra options to simply download a remote URL to a local file, while curl requires -o or -O. However trivial, this fact is often mentioned to me when people explain why they prefer downloading with wget. Funny because that's exactly the reason I installed macports and then wget. It's stupidly trivial, but there you have it.
  • telemachos 17y ago
    I'm not criticizing, but wouldn't a simple alias be easier? alias co='curl -O'
    • statictype 17y ago
      True, much simpler. I guess a larger reason would be that 7+ years of using wget has wired me to its commandline. I'm so used to typing `wget -c "<middle-click>"` or something on the command line, that I just never considered creating aliases for alternatives.
    • mahmud 17y ago
      'co' is already taken though :-)
      • X-Istence 17y ago
        With what? I've got no such alias here.
        • randallsquared 17y ago
          Maybe he aliases that to a vcs checkout command, like 'svn co'?
        • cabacon 17y ago
          co and ci are the old RCS commands, and they're why you can do "cvs co" and "cvs ci" with CVS.
      • pavel_lishin 17y ago
• X-Istence 17y ago
  I use curl for almost everything. Especially for debugging caching and whatnot, using curl -D - -o /dev/null http://example.net/ http://example.net/ makes it extremely simple to see the headers the server is sending for a certain file. Not only that, but curl is installed standard on Mac OS X whereas wget is not, also, most attacks that attack PHP tend to use wget to download further files to the target system, which means that when wget is not available no extra files are downloaded and it generally stops the exploit cold in its track as the attacker does not try again.
  • vog 17y ago
    > Not only that, but curl is installed standard on Mac OS X whereas wget is not On a vanilla Debian, Wget is installed but Curl is not.
  • vog 17y ago
    > most attacks that attack PHP tend to use wget to download further files Nowadays, neither Wget nor Curl is required for downloading in PHP, because functions like file_get_contents() are able to handle URL arguments! So a missing Wget protects only against script kiddies who don't know PHP very well and copied over some outdated code snippets. Also, not installing Wget "protects" you just from those kind of primitive attacks to which your system shouldn't be vulnerable in the first place. Security is achieved by sealing entry doors, not by removing tools.
    • X-Istence 17y ago
      file_get_contents on most hosts does not allow external URL's, including the one where I am an administrator. http://www.php.net/manual/en/filesystem.configuration.php#ini.allow-url-fopen http://www.php.net/manual/en/filesystem.configuration.php#in... Also, installing wget is just one more way to defend a server that is hosting multiple sites with multiple scripts all of which can not be trusted all the time. I don't know if tomorrow a new bug is found in phpBB that allows a remote exploit that runs as the user php runs under, if not having wget stops one such exploit it is a win in my book. The other thing I have in my firewall is all outgoing access is blocked for the user that PHP runs under so that in the event something does happen and something is executed under that user it won't be able to make an outbound connection or get inbound connections. Yes, security is achieved by sealing entry doors, but removing an entry door in the first place may be even better, wget being an entry door, unfortunately the front door is a huge gaping mess you could drive a truck through, but that is unfortunately required for the business I am in, administrating a server with multiple websites running various PHP scripts.
• pibefision 17y ago
  Does anyone knows how to mirror a site using wget or curl, and get all the assets? (specially images linked on stylesheets).
  • Tichy 17y ago
    Some time ago I filed a bug report because of the stylesheet images. So it still isn't fixed? It's simply a bug. I have tried and liked pavuk in the past, don't know if it supports stylesheet images.
  • megamark16 17y ago
    I use HTTrack, usually handles the job just fine.
• mjs 17y ago
  wget has a better-designed command line API and documentation, although its inability to PUT or POST is rather unhelpful. For example, curl uses --include to output full headers (wget uses --save-headers) and, more inexplicably, --request to change the request method, even though --method is available.
  • barrkel 17y ago
    Use --post-data or --post-file to POST with wget.
    • mjs 17y ago
      --post-* is pretty sensible for POST. Unfortunately wget doesn't do PUT or DELETE.
• cowsandmilk 17y ago
  lwp-download should be in this comparison. I've found lots of random cases where it does stuff right and the others don't.
  • godDLL 17y ago
    As should aria2. But that would make it a completely different type of an article, more like a Wikipedia sub-section. We already have Wikipedia.
• Groxx 17y ago
  That's a nice breakdown... I've been a curl user, and never really knew what the major differences were. And it doesn't seem overly biased to me. Many thanks to the author!
• barrkel 17y ago
  I find wget is almost always what you want, unless you're interested in development details like headers. If I want output to a pipe, I use wget -O -. In scripts, I use wget -o <log-file> (or -a) so that I can tail -f to keep track of it. I don't see off hand what curl's log file support is, so I can't comment too deeply on it.
• whyme 17y ago
  When downloading via HTTPS curl seems to barf error codes often so I try wget first, and use curl only if wget doesn't work or curl is easier (which isn't often).
• 1amzave 17y ago
  My main complaint about curl is that it prints a bunch of crap (well, statistics technically) to stderr by default. Especially for something claimed to be (at least in this article) "traditional unix style", that seems undesirable. It's annoying when I do 'curl $URL | less' and end up with garbage in my pager that I have to scroll around a bit to make disappear. It seems like default behavior should be more like 'curl -s -S' -- silent, but print errors if they occur, just like...well, every other sensible command-line tool in the world (so that's what I have curl aliased to). wget of course prints statistics too, but it's not such "pipe-oriented" tool, as the author points out, so this isn't as big of a deal.
  • sophacles 17y ago
    Actually, it is very much unix style. With any modern shell you can simply turn off stderr, or only put stdout in the pipe, or put 2>/dev/null and so on. If both went to stdout then it would be a reasonable complaint.
    • 1amzave 17y ago
      > With any modern shell you can simply turn off stderr, or only put stdout in the pipe, or put 2>/dev/null and so on. Or I could type fewer extra characters by using the '-s' flag, as I described in my comment. My point was about its default mode of operation: "noisy" (which is not, as I see it, a characteristic of "traditional unix" tools). Edit: Additionally, if both went to stdout the tool would be completely unusable -- I'd simply file that under "data corruption".
      • sophacles 17y ago
        That is not really a good assertion. Look at grep: default output is filename, line number, relevant line. This is far from a minimalist output. Or du, which by default lists all the files and subdirectories and their sizes. Minimal, quiet output doesn't define "traditional unix" style -- the ability to taylor the output to the situation, and further make the output as machine readable as possible is what makes it "unixy".
• bhseo 17y ago
  If I want to download a single file, I just use wget and paste the URL in my Putty window. If you want something better than curl and wget, try Mulk. It's based on libcurl, does parallel downloading, parses html, does recursive fetching like wget and more. Check it out: http://mulk.sourceforge.net/ http://mulk.sourceforge.net/