I suppose there is no harm in adding things to robots.txt but I would not expect anyone to honor it. People that work on bleeding edge technology are also comm

I suppose there is no harm in adding things to robots.txt but I would not expect anyone to honor it. People that work on bleeding edge technology are also commonly the risk takers that move fast, break things while sometimes acting a bit cavalier. Adding to this I assume perhaps incorrectly that these companies have the funding to write their scrapers using Chrome headless so the same techniques I used to block archive.is and other bots will not be useful. If something should not be scraped I think it should be placed behind strong authentication and the AUP/ToS must state in clear and concise language explicitly what the sites content may be used for. Even then I would expect risk takers to simply ignore it and accept any legal findings to be the cost of doing business. Or perhaps I have become a bit too skeptical and jaded with time and experience. I will go run a lap.