3 ms·
The issue with scrapping is the intensity and volume of bots. I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to
by sixtyj 3mo ago
The issue with scrapping is the intensity and volume of bots.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
- ccgreg 3mo agoA lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
- sumedh 3mo agoCloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.
- m463 3mo agocloudflare routinely denies my human-piloted browser now, on many sites.
- ambigious7777 3mo agoyou're not a bot thats irrelevant
- pocksuppet 3mo agoI wonder if you'd have more reliable internet browsing with a browser that was a CF verified bot.
- pineapplepizza6 3mo ago[dead]
- 2000UltraDeluxe 3mo ago> I think that nobody would care if I use wget or curl for few pages If only you were the only one doing it...