This comment seems like a non-sequitur. My comment had nothing to do with a particular site, much less "your" site. I was making a general comment about automa

This comment seems like a non-sequitur. My comment had nothing to do with a particular site, much less "your" site. I was making a general comment about automation and detection. If the detection gets better than the automation, it could change the dynamic. There is no fixed rule that says that content providers will or will not allow scraping based on robots.txt or other guidelines. Some could elect to disallow any/all robot behavior, if they have the capability.