3 ms·
OP didn't say... But were they trying to access a single page repeatedly, or were they just doing a scrape of the web? People who are doing a 'get all conte
by londons_explore 4y ago
OP didn't say... But were they trying to access a single page repeatedly, or were they just doing a scrape of the web?
People who are doing a 'get all content on the web' scrape, I try to not block, because there are plenty of reasons to want an offline copy of the internet.
I try to make sure all my webapps have sensible URL schemes to prevent such (IMO legitimate) activities ending up doing an infinite crawl because for example the URL has a session identifier in it and the bot keeps getting assigned a new session.
- luckylion 4y ago> trying to access a single page repeatedly That's what I saw from Azure IPs recently. A single URL being requested from some 5k IPs, each one requesting it every few minutes. I had reported it to MS, they said (after two weeks) "it looks like abuse" and they'd forward it to their cert. I haven't heard any update since.
- eli 4y agoIs there a chance the URL appeared in an email newsletter or marketing message or similar that was sent to many people? I have a theory that this is anti-spam and anti-malware appliances scanning individually for each recipient.
- luckylion 4y agoNo, it was an HTML sitemap, just a long list of URLs that's supposedly helping with discovery for crawlers (or link structure? idk, some SEO thing). And it was on a website that you wouldn't usually see in a newsletter.