The person who owns a server can restrict access by adding a login form and creating user accounts. Or taking other measures (for example, banning IP networks o

The person who owns a server can restrict access by adding a login form and creating user accounts. Or taking other measures (for example, banning IP networks or countries or implementing a captcha). I think that the only thing that has to be regulated is the load (number of request per unit of time) on the server. So that it doesn't prevent server from serving pages to other visitors. Regarding robots.txt, I am not sure if ignoring it should have legal consequences. It is just a hint to robots, don't visit these pages or we will ban you. If there is no harm done to the server I don't see any problem with scraping.