3 ms·

With the recent wget vs. curl discussions fresh in my mind, how would I download all these .zip files with wget? Curl would of course require some sort of parsi

With the recent wget vs. curl discussions fresh in my mind, how would I download all these .zip files with wget? Curl would of course require some sort of parsing, but I would think that wget's mirror option could somehow download just the .zip files.

tyingq 3y ago
The mirror option depends on a hierarchy where it can exclude parents. Since the actual downloads are on a different domain, it seems this won't work. A hacky script to do it: #!/bin/bash for page in $(curl -s https://damieng.com/sitemap.xml | perl -ne 's/<loc>/\n<loc>/g;while ($_ =~ s#<loc>(https://damieng.com/typography/zx-origins/[^\<]+)</loc>##) {print $1."\n"}'); do echo "Page is [$page]" download=$(curl -sL "$page" | perl -ne 'm#(download\.damieng\.com/fonts/zx-origins/[^"]*)#g && print "https://$1"') echo "Download is [$download]" wget -qN "$download" done
- INTPenis 3y ago
  Sure but wget has a -D option to specify transitioning to other domains. So my command was actually something more like wget -m -D download.damieng.com,damieng.com -A .zip https://damieng.com/typography/zx-origins/ https://damieng.com/typography/zx-origins/ But it still wouldn't download the .zip files. I think this is one case where wget should, on paper, be superior to curl.
  - tyingq 3y ago
    I think that would work if that page directly had references to the zip files, but it doesn't. The references are on the sub pages for each font.
    - INTPenis 3y ago
      Yeah but wget -m sets -l (depth limit) to unlimited, I don't see why it shouldn't work out of box. But for some reason it won't get those .zip files.
      - tyingq 3y ago
        Doesn't the -A zip cause it to skip the subpages? That's what I'm saying...the filter is keeping you from traversing main page -> sub page -> zip file.
        INTPenis 3y ago
        No if you check the output, or even enable -d, you'll see that it goes through sub-pages. It downloads all the index files into a tmp file so it can parse them for more sub-pages, then deletes them after because they don't match the pattern. It even goes as far as to download download.damieng.com/fonts/conversions/Micropack.zip, proving that it does traverse domains and sub-dirs, but nothing under download.damieng.com/fonts/zx-origins. It's really strange, I'm thinking it has something to do with all the .zip archives being linked without a protocol (//download.damieng.com), might be a bug but I'd be shocked if no one had encountered it before.

tyingq 3y ago

The mirror option depends on a hierarchy where it can exclude parents. Since the actual downloads are on a different domain, it seems this won't work. A hacky script to do it: #!/bin/bash for page in $(curl -s https://damieng.com/sitemap.xml | perl -ne 's/<loc>/\n<loc>/g;while ($_ =~ s#<loc>(https://damieng.com/typography/zx-origins/[^\<]+)</loc>##) {print $1."\n"}'); do echo "Page is [$page]" download=$(curl -sL "$page" | perl -ne 'm#(download\.damieng\.com/fonts/zx-origins/[^"]*)#g && print "https://$1"') echo "Download is [$download]" wget -qN "$download" done

INTPenis 3y ago
Sure but wget has a -D option to specify transitioning to other domains. So my command was actually something more like wget -m -D download.damieng.com,damieng.com -A .zip https://damieng.com/typography/zx-origins/ https://damieng.com/typography/zx-origins/ But it still wouldn't download the .zip files. I think this is one case where wget should, on paper, be superior to curl.
- tyingq 3y ago
  I think that would work if that page directly had references to the zip files, but it doesn't. The references are on the sub pages for each font.
  - INTPenis 3y ago
    Yeah but wget -m sets -l (depth limit) to unlimited, I don't see why it shouldn't work out of box. But for some reason it won't get those .zip files.
    - tyingq 3y ago
      Doesn't the -A zip cause it to skip the subpages? That's what I'm saying...the filter is keeping you from traversing main page -> sub page -> zip file.
      - INTPenis 3y ago
        No if you check the output, or even enable -d, you'll see that it goes through sub-pages. It downloads all the index files into a tmp file so it can parse them for more sub-pages, then deletes them after because they don't match the pattern. It even goes as far as to download download.damieng.com/fonts/conversions/Micropack.zip, proving that it does traverse domains and sub-dirs, but nothing under download.damieng.com/fonts/zx-origins. It's really strange, I'm thinking it has something to do with all the .zip archives being linked without a protocol (//download.damieng.com), might be a bug but I'd be shocked if no one had encountered it before.