It's not about secret vs public, it's about resource overload on the websites. Existing crawlers so far mostly respected robots.txt, the LLM crawlers don't. Yo

It's not about secret vs public, it's about resource overload on the websites. Existing crawlers so far mostly respected robots.txt, the LLM crawlers don't. You, as a user, might not care, but as servers keep going down, more and more website owners start blocking LLMs. Good riddance, hopefully all good stuff gets locked down. Or to use an analogy, your comment is similar to: "sure those delivery vans violate speed limits and occasionally hit the pedestrians. I don't care, those fast deliveries have been proven wildly useful to me"