4 ms·
robots.txt is such a shitty implementation, it's like putting the "authorized admittance only" sign on the back of the door you're not allowed to traverse. Wer
by jstarfish 3y ago
robots.txt is such a shitty implementation, it's like putting the "authorized admittance only" sign on the back of the door you're not allowed to traverse.
Were it a standard, it wouldn't be so easily neglected without consequence. In practice it's an arbitrary playground rule, like declaring the floor is actually lava. Unless you're directly involved with coming up with these "standards," you can go your entire career in total ignorance of all of them.
- jraph 3y agorobots.txt does not say "authorized admittance only". It says "I'd like you to abide by these rules pretty please". It's there for entities willing to (try to) respect website owners wishes and preferences. It's useful when you need accessing stuff in an automated way, while staying on good terms. By downloading robots.txt, the entity is asking the website owners their preferences regarding crawling. It's polite to do so. robots.txt can also offer useful guidance ("yeah, don't go there... [it's useless, too costly, etc], but you can go there..."). It's not a guarantee. And entities can totally decide that the request is unreasonable. Their call. The risk is being shamed or blocked. Personally, I find it unreasonable / harmful to allow Google and deny everything else. If I built a crawler for a search engine, I would probably not follow this. And Twitter might find out and block me, of course. robots.txt is not a security measure / measure against abuse. This is what IP ban and other kinds of blocking are for. But it's still useful. Not flawless, but useful.
- dylan604 3y agothere's no way you can convince me that if you're building a bot to scrape websites, you've never heard of robots.txt.