robots.txt takes as assumptions that are well-meant, and respectful to the site intentions, major players, that try to index/mirror sites while avoiding overwhe

robots.txt takes as assumptions that are well-meant, and respectful to the site intentions, major players, that try to index/mirror sites while avoiding overwhelming it and accessing only what is supposed to be freely accessed. Using a visible user-agent, having a clearly defined IP block for doing those scans, and a method of scanning goes in the same direction of cooperating with the site owner to both get visibility while not affecting (a lot) functionality. But that doesn't mean that there aren't bad players, that ignore the robots.txt, give random user agent strings, or connects from IPs from all the world to avoid being blocked. LLMs has changed a bit the landscape, mostly because far more players want to get everything or have automated tools to search your information on specific requests. But that doesn't rule out that still exist well-behaved players.