4 ms·
A Basic Scraper/Aggregator Site in Next.js, Go Cloud Functions and Supabase
- nesarkvechnep 4y agoCan this be more overengineered? Wouldn’t cron jobs do the job just fine?
- tluyben2 4y agoIs anyone aware of a 'modern wget'? Something that has a more modern spider (wget misses a lot of deeplinks because the are not standard conform or js) with a smart switch to a browser based headless spider if it detects a frontend js framework? I tried quite a lot of them (including colly) and there seems to be nothing that's just apt install nextwget; nextwget -r https://site.com https://site.com and then see the results appear? The ones I tried needed a lot of tweaking or would miss as much as wget/httrack and in that case, wget/httrack are really quite good.