3 ms·
The interesting thing about robots.txt is that there really isn't a standard for it. This [0] is the closest thing to one and almost every modern website deviat
by wybiral 7y ago
The interesting thing about robots.txt is that there really isn't a standard for it. This [0] is the closest thing to one and almost every modern website deviates from it.
For instance it explicitly says "To exclude all files except one: This is currently a bit awkward, as there is no "Allow" field."
And the behavior is so different between different parsers and website implementations that, for instance, the default parser in Python can't even successfully parse twitter.com's robots.txt file because of the newlines.
Most search engines obey it as a matter of principle but not all crawlers or archivers [1] do.
It's a good example of missing standards in the wild.
[0] https://www.robotstxt.org/robotstxt.html https://www.robotstxt.org/robotstxt.html
[1] https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- pbowyer 7y ago> The interesting thing about robots.txt is that there really isn't a standard for it. This [0] is the closest thing to one and almost every modern website deviates from it. That is changing, and was announced today: https://news.ycombinator.com/item?id=20326067 https://news.ycombinator.com/item?id=20326067
- wybiral 7y agoYeah, my first reaction to Google heading yet another standard was to cringe but this is one of the situations where I think it makes a lot of sense. They're dominant in the search industry and most other engines tend to take their cue so having them spearhead it seems like a good move.
- egypturnash 7y agohttps://datatracker.ietf.org/doc/draft-rep-wg-topic/ https://datatracker.ietf.org/doc/draft-rep-wg-topic/ looks to be the current draft; that post doesn’t have a link to it I can find.
- C1sc0cat 7y agoGoogle used to have (must check the code) have problems with robots.txt files that had a BOM - If the First line was Disallow The D got mangled and that disallow directive got ignored.