13 ms·
I would also add at least "sane default options", "continues downloads" and "retries on error" to the Wget column. I recently had to write a script that downloa
by vesinisa 3y ago
I would also add at least "sane default options", "continues downloads" and "retries on error" to the Wget column. I recently had to write a script that downloads a very large file over a somewhat unreliable connection. The common wisdom among the engineers is that you need to use Wget for this job. I tried using curl but out of the box it could not resume or retry the download. I would have to study the manual and specify multiple options with arguments for this behaviour that really sounds something that should just work out of the box.
Wget needed one option to enable resuming in all conditions, even after a crash: --continue
Wget's introdcution in the manual page also states: "Wget has been designed for robustness over slow or unstable network connections; if a download fails due to a network problem, it will keep retrying until the whole file has been retrieved."
I was sold. Even if I by some miracle managed to get all the options for curl to enable reliable performance over poor connection right, Wget seems to have those on by default and the sane defaults make me believe it will also have this expected correct behaviour enabled even for those error scenarios that I did not think to test myself. Or - if the HTTP protocol ever receives updates, newer versions of Wget will also support those by default, but curl will require new switches to enable the enhanced behaviour - something which I can not add after the product has been shipped.
To me it often seems like curl is a good and extremely versatile low-level tool, and the CLI reflects this. But for my everyday work, I prefer to use Wget as it seems to work much better out of the box. And the manual page is much faster to navigate - probalby in part due to just not supporting all the obscure protocols called out on this page.
- OskarS 3y ago> I tried using curl but out of the box it could not resume or retry the download. Maybe I'm misunderstanding, but curl has exactly that feature, it's the `-C` flag. If you want retries, there's `--retry`. I find curls defaults pretty sane, personally, I wouldn't want either of those by default for a tool like curl.
- deleted 3y ago[deleted]
- inopinatus 3y agoStrong agree. The only misbehaviour I believe curl displays out of the box is globbing, which has burned me enough times that I’ve come to believe it would’ve been better disabled by default and enabled with -g instead of vice versa.
- grayhatter 3y agodo you have an example of the globbing that burns you?
- vesinisa 3y agoYou recall incorrectly. curl's -C flag does not work as-is. You must specify the offset from where it should continue. Why doesn't it take the resumed file's existing length as the guess by default? What else could the user want outside of some very exotic cases? Yes, I want retries. They should be the default for a user-facing tool. Try searching the curl's manual page for "retry". There are no less than 5 different interdependent flags for specifying retry behaviour: --retry-all-errors, --retry-connrefused, --retry-delay <seconds>, --retry-max-time <seconds> and --retry <num> If I "just" want it to retry, surely there's a boolean flag like "--retry" that enables sane defaults? Nope! --retry takes a mandatory integer argument of maximum number of retries. Surely I can set it to zero for a sane default? Nope again: "Setting the number to 0 makes curl do no retries." curl is a good tool if you know it through & through and want exact control over the transfer behaviour. I don't think it's a good tool if you just want to fetch a file and except your tool of choice to apply some sane behaviours for you to that end, that would probably make sense if you are a human rather than an application using a library.
- sherburt3 3y agoI thought that `-C -` checks the output file for the offset
- madfucker69 3y agoYes it does. How does the wget --continue work?
- sherburt3 3y agoI don't know
- OskarS 3y ago> You recall incorrectly. curl's -C flag does not work as-is. You must specify the offset from where it should continue. Why doesn't it take the resumed file's existing length as the guess by default? What else could the user want outside of some very exotic cases? But... it does, though. From the man page ( https://curl.se/docs/manpage.html#-C https://curl.se/docs/manpage.html#-C ) > Use "-C -" to tell curl to automatically find out where/how to resume the transfer. It then uses the given output/input files to figure that out. I just tried it, works perfectly. I don't really see a difference between writing `--continue` in wget and `-C -` in curl. And the use case for specifying it is not so exotic, you might want to do range requests for all sorts of reasons. Look, it's fine if you prefer wget's command line: I don't think retrying a request is a reasonable default for a tool like curl, but reasonable people can disagree on that for sure. But curl is perfectly capable of resuming downloads automatically, you're just (very arrogantly) wrong on that one.
- deleted 3y ago[deleted]
- TheAceOfHearts 3y agoBoth tools have their use-cases. I think with the advent of LLMs like ChatGPT it has also become a lot easier to get the proper command line incantation for whatever tool you're using. Even if you've read the manual previously, it's easy to forget the exact flags you want to use, and validating the generated command line is usually less effort than having to build it up yourself from scratch by reading the manual. With the modern web, sometimes it's easier to use a tool like Puppeteer from a custom script. Especially if the sites you're interacting with are using a lot of JS.
- bdhcuidbebe 3y agoI just look at my curl / wget note for some sane examples. i cant believe one needed chatgpt to be productive
- cinntaile 3y agoIf chatgpt makes someone more productive, then why not. It's just a tool as any other tool.
- freeopinion 3y agoI agree with this sentiment. But I also have personal antidotes that make me cautious about it. When I was a child I memorized dozens of phone numbers of people I called a couple times a month. Now my phone memorizes any phone number for me. We used to memorize poems and such. I can't give you any scientific reason why it is good to have memorized the Gettysburg Address. But I feel intuitively that there is some benefit to exercise your brain similar to the intuition I have about exercising muscles. If a person thinks they benefit from 100 situps a day, I'm not going to disagree. And if they think there is some benefit in reading man pages, well, thanks to those who take the time to write all that ducumentation.
- linhns 3y agoWhen it backfires you're screwed, I have experienced it one using curl -L
- 3y ago
- thrdbndndn 3y agoFor downloading large files I would rather just use aria2c TBH.
- augusto-moura 3y agoAria2c currently looks unmaintained https://github.com/aria2/aria2/pulse https://github.com/aria2/aria2/pulse
- b0afc375b5 3y agoAlso here https://github.com/aria2/aria2/issues/2078 https://github.com/aria2/aria2/issues/2078
- HarHarVeryFunny 3y agoI only used aria2c once a long time ago, but it was awesome for huge files. As I recollect one thing it does is download different sections of a file in parallel over multiple connections, which speeds up downloads from servers that throttle per connection.
- systems_glitch 3y agoRetry with `wget` was one of the most incredible Linux distro included features when I started running it at home. Pretty crucial thing on 56K dialup, and it worked better than the Windows tools I was aware of at the time.
- soneil 3y agoOur dialup used to disconnect every 2 hours by design. wget and wvdial were the only alternative to mail-ordering CDs
- systems_glitch 3y agoWe were way out in the sticks, so interruptions were very common. Still nothing but dialup, satellite, and LTE (no 5G) out there.
- soneil 3y agoI think our setup was very particular to the UK. We didn't (and still don't?) have free local calls like the US, so we paid per minute for ISPs. Almost all ISPs went through a scheme setup by British Telecom - you could either have free internet but you paid for your calls, or you could pay for your internet, and have access via a freefone number - so effectively flat-rate. But the flat-rate option disconnected after two hours, on the dot. Which was hugely frustrating because we had a voicemail variant that was hosted by the telco, and let you know you had messages waiting by pulsing the dialtone. And my modem did not recognise the pulsed dialtone as a valid dialtone, and refused to connect until we called the number and marked them read. Which lead to one of my most UK-centric retro stories. I tried to connect to the internet, and it refused to dial. I blew away my wvdial config, and it refused to dial. I blew away my ppp config, and it refused to dial. I grepped / for the error message and it didn't exist. I ended up blowing away my OS (and accidentally installing onto the wrong drive, and blowing away everything non-OS too), and it still wouldn't dial. So I dragged the modem & extension cord to my mother's PC, and shot off a mail to my preferred mailing list (one hosted by John @ linuxemporium, my preferred source of mail-order distros), and swiftly received the response that in order to be certified by BT to operate on their network, one of the rules equipment had to obey was to refuse to redial the same number x many times. And that all I needed to do was power-cycle the modem. Which I'd done by dragging it upstairs to my mother's PC. And my own machine had been wiped twice over needlessly. Aside, there was a lady named Helen on that mailing list who knew everything about everything, and is everything I aspire to be today. She had opinions on which harddrives best survived salt/sea air, and why they weren't deathstars. Just an incredible amount of lived experience. I miss mailing lists.
- sureglymop 3y agoI would use GNU Wget2 for this. It's supposed to be a wget successor.
- twic 3y agoAlso add -i which lets wget read URLs from a file. In particular wget -i - which makes it read from standard input, and is very useful in pipelines. curl cannot, AFAIK, do this. People usually suggest using xargs, which is a mediocre substitute because it waits for all the URLs to arrive before invoking curl, giving up any chance at parallelism between the command generating the URLs and the one downloading them.
- quyse 3y ago`--config filename` allows this. `--config -` for stdin. Not only urls, but any config options echo '--url https://google.com/' | curl --config -
- dspillett 3y agoxargs doesn't have to wait, you can specify the number of items to include in a single sub-command and it'll batch things as they come in. For instance: ds@swann3:~# (for x in {1..100}; do sleep 0.1s; echo $x >&2; echo $x; done) | xargs -L5 echo 1 2 3 4 5 1 2 3 4 5 6 7 8 9 10 6 7 8 9 10 11 12 [... and so on ...] If the xargs call uses -I then --max-lines=1 is implied anyway. If you replace echo with something that sleeps you'll see that the pipe doesn't stall waiting on xargs so the process producing the list can keep pushing new items to it as they are found: ds@swann3:~# (for x in {1..100}; do sleep 0.1s; echo $x >&2; echo $x; done) | xargs --max-lines=5 ./echosleepecho 1 2 3 4 5 starting 1 2 3 4 5 6 7 8 9 10 11 12 13 14 done 1 2 3 4 5 starting 6 7 8 9 10 15 16 17 18 19 [... and so on until ...] 98 99 100 done 46 47 48 49 50 sleeping for 51 52 53 54 55 done 51 52 53 54 55 sleeping for 56 57 58 59 60 [... and so on until xarg's stdin is exhausted] And you can stop the calls made by xargs being sequential too for more parallelism with the --max-procs option (or use parallel instead of xargs): ds@swann3:~# (for x in {1..100}; do sleep 0.1s; echo $x >&2; echo $x; done) | xargs --max-lines=3 --max-procs=10 ./echosleepecho 1 2 3 sleeping for 1 2 3 4 5 6 sleeping for 4 5 6 7 8 9 sleeping for 7 8 9 10 11 12 sleeping for 10 11 12 done 1 2 3 13 14 15 sleeping for 13 14 15 done 4 5 6 16 [... and so on ...] (I adjusted max-lines in that last example because my current timings made things line up in a manner that made the effect less obvious, adjusting the timings would have been equally valid, in a less artificial example like calling curl to get many resources timings will of course be less regular, perhaps these examples can be improved by randomising the sleeps) I'm not sure what you would do about error handling in all this though, more experimentation necessary there before I'd ever do this in production!
- dicytea 3y agoAnother pet peeve of mine is that curl's URL parser is a lot more strict compared to wget. For example: $ curl -sSLOJ 'example.com/file name.txt' curl: (3) URL using bad/illegal format or missing URL $ curl -sSLOJ 'example.com/file%20name.txt' $ ls file%20name.txt On the other hand, wget (without any additional flags) will produce a file called "file name.txt" for both URLs. Well, technically you also need to add a --content-on-error flag to wget because this example URL 404s.
- grayhatter 3y agoI thought curl had the ability to encode html entities if you asked it to?
- hunter2_ 3y agoHTML entities (&.....;) are distinct from URL encoding (%..).
- forgotmypw17 3y agoI agree on the "sane defaults". Just the fact that `wget url` downloads a URL and saves it makes it a winner for me in command-line use.
- taway1237 3y agoWell, the point of the article is that they are not cpmpetitors and are used differently. For me, 99% of the time i'm curl-ing some API and I definitely don't want to save the result to disk (but often want to pipe it to grep/jq).
- thiht 3y agoAgree. I also use both: wget to download files, and curl to talk to APIs, and as a "no magic" HTTP client. This is not about "sane defaults", but about use cases.
- bla3 3y agoAlso following redirects by default. "Run curl, be confused run curl -L" is something I do many times a month.
- tomatocracy 3y agoOne point where I would argue wget doesn't have a sensible default is on filenames - it really should make --content-disposition the default, at least for single file downloads. Otherwise it will often use the wrong name if there is a redirect or similar in the chain, which seems increasingly common.
- joshspankit 3y agoAlso (annoyingly) URL params
- yencabulator 3y agoI'll argue not trusting the server to dictate the saved file name is the only correct default behavior, and taking the filename from the user input (URL) is reasonable.
- hunter2_ 3y agoA server configured with a docroot to serve a static site will map requested URLs to hierarchical filesystem paths, but that isn't the only possibility; it's a common but quite loose coupling of ideas. But the filename directive of the content-disposition response header is entirely coupled to the idea of a filename. Therefore, it ought to take precedence.
- yencabulator 3y agoI don't want the stranger deciding to save the content as .bash_profile or such.
- hunter2_ 3y agoFair enough, but do you have that fear when using a browser? I guess a Downloads folder is lower stakes than whatever other working directory you're wgetting from, though.
- rlpb 3y agoIndeed - and curl requires `-L` to follow redirects whereas wget just does that by default too. So for ad-hoc CLI use, I turn to wget rather than remember all the curl options required.
- yjftsjthsd-h 3y agoI think of it like this: curl makes HTTP requests, wget downloads files. (Though both can be made to do the other thing in some capacity)
- ComputerGuru 3y agoI mean, it all depends on who you are and what you are primarily doing. “Sane defaults” for one person could be everything another doesn’t want.
- ziml77 3y agocurl is an excellently powerful library and utility but I agree that wget has better defaults. I am almost certain to get the behavior that I want by just throwing a URL at wget, including retrying from the point where it had an issue. I actually ran into a case where our corporate firewall was a little too eager to block a download being performed by the Visual Studio installer because of a signature match partway through a specific download. All I had to do to grab the file was have wget download it. No magic incantations, it was just smart enough to not start the download from the beginning after being cut off, and since it started midway it no longer tripped the signature match rule.