3 ms·
Is anyone aware of a 'modern wget'? Something that has a more modern spider (wget misses a lot of deeplinks because the are not standard conform or js) with a s
by tluyben2 4y ago
Is anyone aware of a 'modern wget'? Something that has a more modern spider (wget misses a lot of deeplinks because the are not standard conform or js) with a smart switch to a browser based headless spider if it detects a frontend js framework? I tried quite a lot of them (including colly) and there seems to be nothing that's just apt install nextwget; nextwget -r https://site.com https://site.com and then see the results appear? The ones I tried needed a lot of tweaking or would miss as much as wget/httrack and in that case, wget/httrack are really quite good.