4 ms·
The mirror option depends on a hierarchy where it can exclude parents. Since the actual downloads are on a different domain, it seems this won't work. A hacky
by tyingq 3y ago
The mirror option depends on a hierarchy where it can exclude parents. Since the actual downloads are on a different domain, it seems this won't work. A hacky script to do it:
#!/bin/bash
for page in $(curl -s https://damieng.com/sitemap.xml | perl -ne 's/<loc>/\n<loc>/g;while ($_ =~ s#<loc>(https://damieng.com/typography/zx-origins/[^\<]+)</loc>##) {print $1."\n"}'); do
echo "Page is [$page]"
download=$(curl -sL "$page" | perl -ne 'm#(download\.damieng\.com/fonts/zx-origins/[^"]*)#g && print "https://$1"')
echo "Download is [$download]"
wget -qN "$download"
done
- INTPenis 3y agoSure but wget has a -D option to specify transitioning to other domains. So my command was actually something more like wget -m -D download.damieng.com,damieng.com -A .zip https://damieng.com/typography/zx-origins/ https://damieng.com/typography/zx-origins/ But it still wouldn't download the .zip files. I think this is one case where wget should, on paper, be superior to curl.
- tyingq 3y agoI think that would work if that page directly had references to the zip files, but it doesn't. The references are on the sub pages for each font.
- INTPenis 3y agoYeah but wget -m sets -l (depth limit) to unlimited, I don't see why it shouldn't work out of box. But for some reason it won't get those .zip files.
- tyingq 3y agoDoesn't the -A zip cause it to skip the subpages? That's what I'm saying...the filter is keeping you from traversing main page -> sub page -> zip file.
- INTPenis 3y agoNo if you check the output, or even enable -d, you'll see that it goes through sub-pages. It downloads all the index files into a tmp file so it can parse them for more sub-pages, then deletes them after because they don't match the pattern. It even goes as far as to download download.damieng.com/fonts/conversions/Micropack.zip, proving that it does traverse domains and sub-dirs, but nothing under download.damieng.com/fonts/zx-origins. It's really strange, I'm thinking it has something to do with all the .zip archives being linked without a protocol (//download.damieng.com), might be a bug but I'd be shocked if no one had encountered it before.