4 ms·
Interesting article, but do many people use curl for binary downloads? I thought most sysadmins used it for viewing headers and/or parsing raw text. For binary
by laumars 13y ago
Interesting article, but do many people use curl for binary downloads? I thought most sysadmins used it for viewing headers and/or parsing raw text. For binary downloads wget seems to be favoured. So with that in mind, I'd be curious to see how wget compared with the other solutions in those benchmarks.
DISCLAIMER: I'm not trying to imply that my way is "better" nor the only way HTTP GETs should be called. I'm only commenting on how I generally work and the trend I've anecdotally observed. Though I'd be interested to see if many others on here do prefer curl over wget for pulling binary files.
- maggit 13y ago> I'd be interested to see if many others on here do prefer curl over wget for pulling binary files. This is how I ended up consistently prefering curl: Last time I looked closely, I had better control with curl, and curl had better support for stuff. For example HTTP 1.1 was supported by curl, but not wget. This makes curl a far better choice for testing HTTP servers. And next, since I now know curl pretty well, and have it installed all over the place, it is my go-to HTTP command line client. I don't know of any reason to use both, or any reason to prefer wget, although in some situations maybe "wget <url>" acts slightly more like you want than "curl <url>". (Which is of course easy to work around by using "curl -OL <url>", if that's what you want, but convenient defaults are still convenient)
- laumars 13y ago> "Last time I looked closely, I had better control with curl, and curl had better support for stuff. For example HTTP 1.1 was supported by curl, but not wget. > "This makes curl a far better choice for testing HTTP servers." Well yes. You're just reiterating what I said about curl being great for server testing. But this article is about download performance. When just downloading a largish binary file (eg a tarball), it's easier to just: wget example.com/source.tar.gz With that, you get a progress bar, it saves the file to disk and basically just does all the sane options you'd want for downloading by default (which I think is why most INSTALL / README docs tend to recommend wget in their instructions). Again, I'm not saying one is better than the other, but the convenience of wget's defaults (which I think you also highlighted later in you post) does tend to make me favour it for downloading binary content (with curl being my go-to for any server testing or text resources). > "And next, since I now know curl pretty well, and have it installed all over the place, it is my go-to HTTP command line client." wget also comes as part of the basic install with almost all Linux and other UNIX-like OSs as well. In fact wget actually pre-dates curl (albeit they're both >= 15 years old, so there's not really much between the two relatively speaking) so I'd have expected even your oldest live systems would still have wget. I can totally relate to the "know[ing] curl pretty well" point though. There is definitely an argument for reusing the tools that you're already familiar with.
- maggit 13y agoWe seem to be in full agreement. I only wanted to spell out my own reasoning for using curl and not wget, since you explicitly said you were interested in this information, and I suppose there are others that think like me in this regard as well :) One popular UNIX-like OS that comes with curl and not wget is OS X. That might also drive adoption. (I seem to remember having had to install both curl and wget on Ubuntu, but I am unable to verify this now)
- laumars 13y ago> "We seem to be in full agreement. I only wanted to spell out my own reasoning for using curl and not wget, since you explicitly said you were interested in this information, and I suppose there are others that think like me in this regard as well :)" I'm grateful for your input. Sorry if I came across as elitist - that wasn't my intention but in reflection, in does read a little that way. > "One popular UNIX-like OS that comes with curl and not wget is OS X. That might also drive adoption." Ahh interesting point. I guess that would have an impact.
- kalleboo 13y agoWell cURL is also an HTTP library which makes it a better comparison with other HTTP libraries than a file download utility.
- laumars 13y agoYeah, I've spent some time playing with libcurl as well. However I hadn't considered your point about wget being out of scope because this is about libraries rather than utilities. Good point :)
- kalleboo 13y agoThat said, including JVM startup time is a very odd way to benchmark HTTP libraries. Is it a common enough use case to make Java apps that just start up and download a single file?
- klausa 13y ago[13:17:18] i13:klausa:~% uname -a Darwin i13.local 12.4.0 Darwin Kernel Version 12.4.0: Wed May 1 17:57:12 PDT 2013; root:xnu-2050.24.15~1/RELEASE_X86_64 x86_64 [13:17:19] i13:klausa:~% curl curl: try 'curl --help' or 'curl --manual' for more information [13:17:22] i13:klausa:~% wget wget: Command not found.
- Achshar 13y agoand this is...?
- laumars 13y agoA demonstration about how OS X ships curl but not wget. Essentially he's just running which {curl,wget} to demonstrate that curl is in $PATH where as the shell cannot find wget (since we're talking about base installs it's safe to assume that the aforementioned utilities should be in $PATH by default - if included at all).
- Achshar 13y agoThanks, that makes sense now.
- zeckalpha 13y agoI typically use curl and prefer it, but one feature I find it lacking which wget has is recursive downloading. https://www.gnu.org/software/wget/manual/html_node/Recursive-Retrieval-Options.html https://www.gnu.org/software/wget/manual/html_node/Recursive...
- ethomson 13y agoDo people use curl from the command line directly? Maybe, maybe not. But this article is really about libcurl's performance. And libcurl is used quite extensively to download all sorts of things, including binaries. Git, for example, makes heavy use of curl for http remotes.
- ssmoot 13y agoCurl is the default on most systems I've used. The only place wget seems to be a default is on Ubuntu (and probably some other flavors of Linux). But most places where I need such a tool: My desktop (Mac), the load-balancers (OpenBSD), the database servers (FreeBSD), the hypervisors (SmartOS), the NFS servers (FreeBSD), basically all the infrastructurey stuff, curl is the default. I wouldn't swear to it, but I don't think wget is installed on most of those systems.
- X-Istence 13y agoI use curl across the board. I can't even begin to count the amount of times that I have seen script kiddies use some pre-made script to hack {Wordpress, Drupal, Joomla, whatever other PHP script} and try to use wget to download their new code. wget isn't installed on my servers, so it fails. curl is installed, but they never seem to reach for that. Keeping wget off my servers has stopped countless attacks, yes the PHP scripts should be updated... but shared hosting isn't all that great as a sys-admin.