I guess it depends on how you're scraping. For general web crawling, simply implementing a response time based crawl back off per origin and identifying yoursel

I guess it depends on how you're scraping. For general web crawling, simply implementing a response time based crawl back off per origin and identifying yourself appropriately in User Agent goes a long way. If you are instead automating Facebook's SPA to pull comments for analysis, then yeah you need to emulate a human, because that's not how they intend you to do it.