4 ms·
> The interesting thing about robots.txt is that there really isn't a standard for it. This [0] is the closest thing to one and almost every modern website devi
by pbowyer 7y ago
> The interesting thing about robots.txt is that there really isn't a standard for it. This [0] is the closest thing to one and almost every modern website deviates from it.
That is changing, and was announced today: https://news.ycombinator.com/item?id=20326067 https://news.ycombinator.com/item?id=20326067
- wybiral 7y agoYeah, my first reaction to Google heading yet another standard was to cringe but this is one of the situations where I think it makes a lot of sense. They're dominant in the search industry and most other engines tend to take their cue so having them spearhead it seems like a good move.
- egypturnash 7y agohttps://datatracker.ietf.org/doc/draft-rep-wg-topic/ https://datatracker.ietf.org/doc/draft-rep-wg-topic/ looks to be the current draft; that post doesn’t have a link to it I can find.