It seems to me that the moment you'd need to do anything interesting with a website, you'd need to crawl a lot of its pages and you'd hit robots.txt limitations

It seems to me that the moment you'd need to do anything interesting with a website, you'd need to crawl a lot of its pages and you'd hit robots.txt limitations very quickly.