29 ms·
The curl-wget Venn diagram
- azatom 3y agodon't forget the weekly security fix on the right side ;)
- Karellen 3y agoThat seems a bit harsh https://curl.se/docs/releases.html https://curl.se/docs/releases.html
- rany_ 3y agoI don't know about weekly but the security record seems alright for something as complex as curl: https://curl.se/docs/security.html https://curl.se/docs/security.html
- acdha 3y agoCurl is very widely used and has a ton of features which means that it gets a lot of CVEs, but their severity is often significantly overstated for users outside of specific niche configurations - for marketing purposes, it’s nice to be able to say that you found a HIGH in libcurl without mentioning that it only affected Windows domain authentication on ARM. The lead developer has written about this providing a lot of noise without much tangible security benefit: https://daniel.haxx.se/blog/2023/08/26/cve-2020-19909-is-everything-that-is-wrong-with-cves/ https://daniel.haxx.se/blog/2023/08/26/cve-2020-19909-is-eve...
- flotwig 3y agoLooks like cURL and SQLite have the same woes: https://www.sqlite.org/cves.html https://www.sqlite.org/cves.html Previously I worked on an open source project that pulled in many third party libraries. Users would run their corpo vulnerability scanners on the project and find dependencies with open CVEs and demand fixes, not understanding that in our usage of the libraries, the vulnerability is not exposed. I think in 4 years, we had users open roughly 50 issues like this, which corresponded to exactly 0 real world exploitable issues. A central vuln DB makes sense for sysadmins, but too many make it the end-all-be-all.
- acdha 3y agoI think this ends up devolving to Goodhart’s law: once CVEs became marketing, a ton of people had a huge incentive to game their stats at the expense of everyone else’s time.
- johnchristopher 3y agoWhat are happy eyeballs in the curl circle ?
- Spidler 3y agoA fast(er) & standards compatible way of doing ipv6+ipv4 connections. - https://en.wikipedia.org/wiki/Happy_Eyeballs https://en.wikipedia.org/wiki/Happy_Eyeballs
- thriftwy 3y agoInterestingly, it would make more sense for Wget supporting these, since it is more of an "user" agent.
- deleted 3y ago[deleted]
- k255 3y agowarc support is smth wget specific, worth mentioning
- jaimehrubiks 3y agoI guess the most common usage is the overlap between the two. That's why I'd love to see a Venn diagram of where (OS and docker images) each is installed by default!
- nicce 3y agoMandatory mention for the comparison made by Daniel Stenberg https://daniel.haxx.se/docs/curl-vs-wget.html https://daniel.haxx.se/docs/curl-vs-wget.html
- enriquto 3y agoFor me the killer feature of wget is that by default it downloads a file with a name derived from the url. You do: wget url://to/file.htm and a file named "file.htm" appears in your cwd. Using curl, you would have to do curl url://to/file.htm > file.htm or some other, less ergonomical, incantation.
- dwb 3y agocurl -O https://curl.se/docs/manpage.html#-O https://curl.se/docs/manpage.html#-O
- OskarS 3y agoI much prefer the fact that curl doesn't do this by default (but has the option), it much closer matches the behavior of most unix-y tools. Makes it so much easier to put it into pipelines.
- throw0101a 3y agoWhat's "better" depends on whether you 'just' want to download a file, or if you want to build an larger workflow. For me, I usually want to download files and I'm usually not doing any more processing, so I tend to type in *wget" first.
- TheFuzzball 3y agoI usually use `curl -OL <url>` or `curl -L -o <local-filename> <url> instead of wget. wget is better for download lists, but I'd accept the argument that a simple shell script is similarly easy.
- mananaysiempre 3y agoAlso -OJ : with -O you get the name derived from the URL (the initial one, I think, even if redirects are being followed), with -OJ you get the one from the Content-Disposition header or the final URL, the way browsers do it. Of course, plain -O is safer. (For parity with Wget, you might also want to add -R to set the downloaded file’s mtime according to the Last-Modified header.)
- vesinisa 3y agoI would also add at least "sane default options", "continues downloads" and "retries on error" to the Wget column. I recently had to write a script that downloads a very large file over a somewhat unreliable connection. The common wisdom among the engineers is that you need to use Wget for this job. I tried using curl but out of the box it could not resume or retry the download. I would have to study the manual and specify multiple options with arguments for this behaviour that really sounds something that should just work out of the box. Wget needed one option to enable resuming in all conditions, even after a crash: --continue Wget's introdcution in the manual page also states: "Wget has been designed for robustness over slow or unstable network connections; if a download fails due to a network problem, it will keep retrying until the whole file has been retrieved." I was sold. Even if I by some miracle managed to get all the options for curl to enable reliable performance over poor connection right, Wget seems to have those on by default and the sane defaults make me believe it will also have this expected correct behaviour enabled even for those error scenarios that I did not think to test myself. Or - if the HTTP protocol ever receives updates, newer versions of Wget will also support those by default, but curl will require new switches to enable the enhanced behaviour - something which I can not add after the product has been shipped. To me it often seems like curl is a good and extremely versatile low-level tool, and the CLI reflects this. But for my everyday work, I prefer to use Wget as it seems to work much better out of the box. And the manual page is much faster to navigate - probalby in part due to just not supporting all the obscure protocols called out on this page.
- OskarS 3y ago> I tried using curl but out of the box it could not resume or retry the download. Maybe I'm misunderstanding, but curl has exactly that feature, it's the `-C` flag. If you want retries, there's `--retry`. I find curls defaults pretty sane, personally, I wouldn't want either of those by default for a tool like curl.
- deleted 3y ago[deleted]
- inopinatus 3y ago
- jiofj 3y agowget's "downloads recursively" is worth half the features of curl.
- jmmcd 3y agoI read “downloads recursively GPLv3 licensed” and wondered whether even Stallman would really claim that a file downloaded by wget becomes retroactively GPLv3.
- tyingq 3y agoSeems maybe dated. For example, it excludes both of these from wget in the diagram > HTTP PUT wget --method=PUT --body-data=<STRING> > proxies ... HTTPS wget --use-proxy=on --https_proxy=https://example.com https://example.com Curl consistently has more options and flexibility, but there's several things on the right side of the venn diagram where wget does have some capability.
- aragilar 3y agoLooks like there's FTP support also (based on the man page).
- ducktective 3y agoDaniel Stenberg is among those rare breed of developers who put their heart and soul into their creation, a fading trait in the modern world of big tech that shadowy developers seem to be replaceable cogs of a money-making machine. It's as if he treats curl as his mark on the world of IT.
- jiofj 3y agoMaybe if you work for a company you don't put your heart into your creation, but if you have a popular personal project that brings you a lot of cash I'm sure you'll be as dedicated as him.
- freeopinion 3y agoCurl is a money maker? It makes me very happy to hear that. I hope it is true.
- latexr 3y agoFrom Daniel’s homepage¹: > I work for wolfSSL doing commercial curl support. If you need help to fix curl problems, fix your app's use of libcurl, add features to curl, fix curl bugs, optimize your curl use or libcurl education for your developers... Then I'm your man. Contact us! From Wikipedia’s wolfSSL page²: > In February 2019, Daniel Stenberg, the creator of cURL, joined the wolfSSL project. Given that, saying cURL is “a popular personal project that brings [Daniel] a lot of cash” seems like a bit of a stretch. ¹ https://daniel.haxx.se https://daniel.haxx.se ² https://en.wikipedia.org/wiki/WolfSSL#History https://en.wikipedia.org/wiki/WolfSSL#History
- globular-toast 3y agoFree software is full of people like this. That's why I use free software even if it's technically inferior. Of course, a lot of it is actually technically superior these days which makes it an even easier choice.
- tm11zz 3y agocurl is the ffmpeg of url fetching
- red_admiral 3y agoFor many of us, I bet the key distinction is "the one that writes to stdout by default" vs "the one that makes a file by default".
- Karellen 3y agoor "the one that can be piped to `sh` by default" ;-)
- quickthrower2 3y agoI find wget is more likely to be on a given system than curl by default so I usually reach for that first. But I am squarely in the middle of the venn.
- alexchamberlain 3y agoI recently found [axel], which is very impressive wget-like tool for larger files. [axel]: https://github.com/axel-download-accelerator/axel https://github.com/axel-download-accelerator/axel
- midasuni 3y agoUsed to use parallel downloading of axel a lot 20 years ago whe it’s long fat pipes, window scaling wasn’t always enabled, and it wasn’t on the corporate proxy I had to use.
- coldtea 3y agoIn the olden times we used wget when we wanted to mirror a website. It is a specialized tool. Curl is a general purpose request library with a cli frontend (also used embedded from other programs, or as a standard library API in PHP etc).
- hunter2_ 3y agoPersonally I'm a fan of httrack for mirroring, although wget has some href/src translation capabilities that are occasionally a better match for particular goals.
- kriro 3y agoThe best part of the post for me is: """I have contributed code to wget. Several wget maintainers have contributed to curl. We are all friends."""
- KaiserPro 3y agoI've never seen them as competitors! wget is my goto if I need to download a file now, with the minimum of fuss. curl is used when I need to do something fancy with a url to make it work, or when I'm fiddling with params to make an API work/debug it.
- hunter2_ 3y agoIt's even in the name: wget will do everything reasonable to just get something to you, while curl will do a huge portion of things that use URLs.
- bandergirl 3y agocurl is a connection tool while wget is an app. At a basic level they do the same thing, but they excel in different areas. This diagram is clearly and unapologetically biased towards curl. Feels strange that the author of curl doesn’t know what wget actually offers.
- safety1st 3y agoCurl can fetch over imap/imaps? Can I use it to download and back up my entire mailbox?
- bravetraveler 3y agoOn the cURL side; ridiculous manual I regularly forget the order for the values for --resolve, try searching for that word and figuring it out quickly I've been relegated to grepping a flippin' manual
- fellerts 3y agoA trick I've found useful when searching large man pages for a flag --foo is to search for `␣␣--foo` (note the two leading spaces). In my experience this always hits the line where the flag is defined instead of irrelevant mentions of that flag, and it's faster than paging through the manual by hand.
- bravetraveler 3y agoAh, good call - I've tried a variation of that with one space and been left disappointed; two works a treat!
- dicytea 3y agoOr for something more flexible: ^ *--foo. But unfortunately this kind of pattern will be defeated by -f, --foo indexes.
- deleted 3y ago[deleted]
- Gud 3y agoFor the FreeBSD users out there also ‘fetch’ is available. Don’t know what the advantages/disadvantages are, but it comes with the default install. It’s usually what I use.
- Aissen 3y agoAnother thing I forget that wasn't supported in wget (but worked in curl) last I checked: IPv6 link-local address scopes (interface names on linux).
- wizofaus 3y agoCan anyone explain "happy eyeballs"? Did find one page about it, but wasn't 100% clear what the use case for it being an option was, or where on earth the name came from...
- jlokier 3y agoHappy Eyeballs makes a simultaneous connection over IPv4 and IPv6 to an HTTP server, and uses the first connection that gets a server acknowledgement. This is useful because many networks have noticeably different response times for IPv4 and IPv6, and many have one of them configured but not working properly (usually IPv6). Without Happy Eyeballs web browsers can be slow fetching some web pages, for some users, waiting for a request timeout on IP addresses that don't work before trying one that works, or working but with the slower IP. It's called Happy Eyeballs because it improves the visible page load time in web browsers for many users. https://en.m.wikipedia.org/wiki/Happy_Eyeballs https://en.m.wikipedia.org/wiki/Happy_Eyeballs
- wizofaus 3y agoThanks...so curl always does this but wget never does? I got the impression initially it was an option.
- jgalt212 3y agoFor the intersection area, I see no reason to use curl or wget over requests / urllib. Assuming one is inside a script.
- jessekv 3y agoWithin the python ecosystem, I find httpx to be more similar to curl, and requests to be more like wget. For example, when following redirects or handling connection issues.
- ketanmaheshwari 3y agoI have never seen an example of curl working with SFTP. Does anyone know or have used curl over SFTP?
- geocrasher 3y agoTo me the real takeaway here isn't related to wget or curl. It's related to using the right tool for the job, whatever that is.
- specialist 3y agoNeat. Love it. Is there a feature matrix to Venn diagram converter? (Deep down) on my To Do list is comparing Ansible, Puppet, Chef, Docker, etc. Which ultimately means some kind of feature matrix, right? With a converter, we'd get Venns for free.
- jpeeler 3y agoDoes the webpage parsing functionality of wget only come into play when doing something like an entire site backup?
- rob74 3y agoOk, wow, I didn't know that curl supported so many protocols - but the fact remains that that small intersection area is probably what > 90% of curl/Wget users are using the tools for. So, from a developer's perspective, the overlap is not that big, but from a user's perspective it might appear much bigger...
- jiehong 3y agoLooks like wget 2 introduces an equivalent to libcurl: libwget [0][1]. [0]: https://gitlab.com/gnuwget/wget2 https://gitlab.com/gnuwget/wget2 [1]: https://en.wikipedia.org/wiki/Wget#Wget2 https://en.wikipedia.org/wiki/Wget#Wget2
- lurtbancaster 3y agoCouple more things wget can do that curl can't. 1. wget can resolve onion links. curl can't(yet). You'll get a curl: (6) Not resolving .onion address (RFC 7686) 2. curl has problems parsing unicode characters curl -s -A "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0" https://old.reddit.com/r/GonewildAudible/comments/wznkop/f4m_mi_coño_esta_mojada_summer22tomboy/.json will give you a {"message": "Bad Request", "error": 400} wget on the other hand, automatically converts the ñ to UTF-8 hex - %C3%B1 - and resolves the link perfectly. I've searched the curl manpage and couldn't find a way to solve this. Please help. I'm having to use `xh --curl` [1] to "fix" the links before I pass them to curl. [1] https://github.com/ducaale/xh https://github.com/ducaale/xh