3 ms·
Can you compare a robots.txt file on a website to a sign on a physical door that says "authorized admittance only"? Both the website and the building are open t
by dylan604 3y ago
Can you compare a robots.txt file on a website to a sign on a physical door that says "authorized admittance only"? Both the website and the building are open to the public, but there are designated areas for authorized visitors only.
- falcolas 3y agoIMO, yes. And you can also trespass users - prevent them from ever coming back - even as you provide a business generally open to the public.
- dylan604 3y agomuch like a physical business having the right to refuse service, a website should also be able to refuse serving data to users that have abused the service. a bartender says you've had too many, so no more service. websites can rate limit as well. we recently saw Musk test this concept.
- jraph 3y agoWhere I live, you can't really legally refuse serving an individual without a strong reason. And if you did, you'd better be able to prove that you didn't do it for discriminatory reasons. Here, bartenders can say you've had too many, they are supposed to stop selling you alcohol if they think it could be dangerous for you or other people. This is one of the cases where not only the business can refuse to serve you, it also has to. Now, I'm not a lawyer but I wouldn't expect robots.txt to be legally enforceable. Of course, in any case, the website can totally refuse to fulfill a request, rate limit, etc. I'd say this is not comparable to the bartender or a business refusing to serve you though. Responding to an HTTP request is not offering a commercial service. And if it is, whatever legal document that promises you the service can totally specify that they won't promise to answer your robot.
- dylan604 3y agoIf you're being an nuisance and causing the other patrons to have a less than ideal experience, absolutely, you should be asked to leave. If your site is constantly getting scraped and your service is affected due to it which causes other users to have slowed responses, you should absolutely be able to refuse requests from known knuckleheads. To this day, bars have a list of "refuse entry". Sporting teams have life time bans to hooligans. Sure, the ban hammer should be slow to swing, but it shouldn't a never allowed to swing situation.
- jraph 3y ago100% agree :-)
- jstarfish 3y agorobots.txt is such a shitty implementation, it's like putting the "authorized admittance only" sign on the back of the door you're not allowed to traverse. Were it a standard, it wouldn't be so easily neglected without consequence. In practice it's an arbitrary playground rule, like declaring the floor is actually lava. Unless you're directly involved with coming up with these "standards," you can go your entire career in total ignorance of all of them.
- jraph 3y agorobots.txt does not say "authorized admittance only". It says "I'd like you to abide by these rules pretty please". It's there for entities willing to (try to) respect website owners wishes and preferences. It's useful when you need accessing stuff in an automated way, while staying on good terms. By downloading robots.txt, the entity is asking the website owners their preferences regarding crawling. It's polite to do so. robots.txt can also offer useful guidance ("yeah, don't go there... [it's useless, too costly, etc], but you can go there..."). It's not a guarantee. And entities can totally decide that the request is unreasonable. Their call. The risk is being shamed or blocked. Personally, I find it unreasonable / harmful to allow Google and deny everything else. If I built a crawler for a search engine, I would probably not follow this. And Twitter might find out and block me, of course. robots.txt is not a security measure / measure against abuse. This is what IP ban and other kinds of blocking are for. But it's still useful. Not flawless, but useful.
- dylan604 3y agothere's no way you can convince me that if you're building a bot to scrape websites, you've never heard of robots.txt.