3 ms·
robots.txt did not work?
by throwaway2037 2y ago
robots.txt did not work?
- lemper 2y agobro, since when vc funded ai companies have the courtesy to respect robots.txt?
- seabird 2y agoOf course it didn't work. At best, the dorks doing this think there's a gamechanging LLM application to justify the insane valuations right around the corner if they just scrape every backwater site they can find. At worst, they're doing it because it's paying good money. Either way, they don't care, they're just going to ignore robots.txt.
- hsbauauvhabzb 2y agoI have not monitored traffic in this way, but I imagine most AI companies would explicitly follow links listed in robots, even if not mentioned elsewhere on the site.
- epc 2y agoI’ve been doing web sites for thirty years, robots.txt is at best a request to polite user agents to respect the server’s desires. None of the malicious crawlers respect it. None of the AI crawlers respect it. I’ve resorted to returning xml and zip bombs in canary pages. At best it slows them down until I block their network.