3 ms·
Does it honour robots.txt when scraping data?
by gumballindie 4y ago
Does it honour robots.txt when scraping data?
- squarefoot 4y ago"No, I do not honor robots.txt files when scraping data."
- gumballindie 4y agoWondering if there's a list of IPs or a reliable technique to block them else our hosting bills will be inflated by script kiddies running their crawlers. Curious how much extra it's costing us to have them keep sucking images, audio, video and text over and over.