3 ms·
robots.txt is well... for robots. It is not a binding agreement, it is a preference by website owner not to index some pages in search engines. Public pages ar
by rcdwealth 6y ago
robots.txt is well... for robots. It is not a binding agreement, it is a preference by website owner not to index some pages in search engines.
Public pages are still public pages, and regardless if they are indexed in search engines, everyone is free to to hyperlink to such.
- gumby 6y agoI agree with you but apparently some do not! As for robots, well it’s expressing a preference which I shy I said “be kind”. In particular I believe the OP is mechanically building the indexes so it makes sense. But yes those pages are provided for the public’s use, else the people posting could have taken steps to change that. I don’t feel the OP did anything at all wrong.