3 ms·
Do any of you who scrape fear retaliation from the sites you scrape? Maybe you are violating a ToS or scraping copyrighted text, and they cut off your IP. Thoug
by andrew311 19y ago
Do any of you who scrape fear retaliation from the sites you scrape? Maybe you are violating a ToS or scraping copyrighted text, and they cut off your IP. Thoughts?
- inovica 19y agoI think you have to take into consideration the TOS, copyright and also robots.txt. If you ignore these then its well within the site owners rights to do something about it - blocking you or further. We always look at the robots.txt file first and use that as our benchmark in terms of what they (the site) wish robots/crawlers to look at