4 ms·
No thank you, when I define a robots.txt file I expect all automated systems to respect it.
by mvdtnz 2y ago
No thank you, when I define a robots.txt file I expect all automated systems to respect it.
- beeflet 2y agoSomeone should call the robots.txt police then, there's a bandit on the loose!
- TheDudeMan 2y agoBut this isn't automated. This is user-driven.
- jrflowers 2y agoIf this feature isn’t already part of the Claude API it likely will be at some point, in which case many Claude requests will be automated with no way to distinguish between user-driven or otherwise.
- pixl97 2y agoSimply put, at the end of the day you lose, AI blocking will not work. I mean, currently the AI request comes from the datacenter running the AI, but eventually one of two things will happen. AI models will get small/fast enough to run on user hardware and use the users resources: End result? You lose. The user will set their own headers and sites will play the impossible game of identifying AI. AI sites will figure out how to route the requests via any number of potential methods so the requests appear to come from the user anyway: End result? You lose. The sites attempting to block will play the cat and mouse game of figuring out what is AI or not AI. Note, this doesn't mean AI blocking isn't worth doing, if nothing else to reduce load on the servers. It's just not a long term winning strategy.
- ipaddr 2y agoWelcome to the world of CAPTCHAs
- pixl97 2y agoHeh, I'm always reminded of dunkey when captchas are brought up. It seems AI gets better faster at them than humans do. https://www.youtube.com/watch?v=WqnXp6Saa8Y https://www.youtube.com/watch?v=WqnXp6Saa8Y
- jimbokun 2y agoDepends if the legal system survives. You may not be able to stop AIs from crawling web sites through technological means. But you can confiscate all the resources of the company that owns the AI.
- hooverd 2y agoIdeally bad behavior by AI companies should trigger crushing fines and jail time for executives.
- jimbokun 2y agoIt’s clearly not. The human user is not requesting the resource. The AI is.
- wraptile 2y agoIt's clearly not. The human user is not requesting the resouce. The web browser is. Where do we stop here? at "please drink a verification can and maintain eye contact at all times"?
- victorbjorklund 2y agoA browser is automated too.
- goatlover 2y agoBrowser don't automatically browse, unless they are being automated.
- Sargos 2y agoAny AI tool I make will ignore robots.txt on principle. Artificial humans should have equal rights as real humans.
- navigate8310 2y agoThink of the "searching" LLM as a peon of the user, the user asks, the peon performs. In that essence, searching by the LLM should be human-driven and must not be blocked. It's just an automated system doing the search not your personal peon.
- bcrosby95 2y agoCan't you make the same argument for a crawler? The user wants information, the peon (crawler) just compiles it for them.
- theshackleford 2y agoThen you’ve fundamentally misunderstood what a robots.txt file does or is even intended to do and should reevaluate if you should be in charge of how access is or is not prevented to such systems. Absolutely nothing has to obey robots.txt. It’s a politeness guideline for crawlers, not a rule, and anyone expecting bots to universally respect it is misunderstanding its purpose.
- usrbinbash 2y ago> Absolutely nothing has to obey robots.txt And absolutely no one needs to reply to every random request from an unknown source. robots.txt is the POLITE way of telling a crawler, or other automated system, to get lost. And as is so often the case, there is a much less polite way to do that, which is to block them. So, the way I see it, crawlers and other automated systems have 2 options: They can honor the polite way of doing things, or they can get their packets dropped by the firewall.