3 ms·
“Because the REP was only a de-facto standard for the past 25 years, different implementers implement parsing of robots.txt slightly differently, leading to con
by noncoml 7y ago
“Because the REP was only a de-facto standard for the past 25 years, different implementers implement parsing of robots.txt slightly differently, leading to confusion. This project aims to fix that by releasing the parser that Google uses.”
The amount of arrogance in this sentence is insane.
Because Google way is the only one true way?
- gbear605 7y agoIn terms of “what should a robots.txt file look like to be parsed correctly,” yes, because they’re the ones who are going to be doing most of that parsing. Yes, ideally it would be an entirely independent standardization process, but it’s not arrogant of them.
- deleted 7y ago[deleted]
- tick_tock_tick 7y agoBecause google is the only search engine most people care about....