robots.txt does not say "authorized admittance only". It says "I'd like you to abide by these rules pretty please". It's there for entities willing to (try to)

robots.txt does not say "authorized admittance only". It says "I'd like you to abide by these rules pretty please". It's there for entities willing to (try to) respect website owners wishes and preferences. It's useful when you need accessing stuff in an automated way, while staying on good terms. By downloading robots.txt, the entity is asking the website owners their preferences regarding crawling. It's polite to do so. robots.txt can also offer useful guidance ("yeah, don't go there... [it's useless, too costly, etc], but you can go there..."). It's not a guarantee. And entities can totally decide that the request is unreasonable. Their call. The risk is being shamed or blocked. Personally, I find it unreasonable / harmful to allow Google and deny everything else. If I built a crawler for a search engine, I would probably not follow this. And Twitter might find out and block me, of course. robots.txt is not a security measure / measure against abuse. This is what IP ban and other kinds of blocking are for. But it's still useful. Not flawless, but useful.