28 ms·
Do we need more features that are generally ignored? What has robots.txt gotten us? What has Do Not Track gotten us?
by annoyingnoob 3y ago
Do we need more features that are generally ignored? What has robots.txt gotten us? What has Do Not Track gotten us?
- shadowgovt 3y ago> What has robots.txt gotten us A standard protocol for reputable crawlers to semantically understand some high-level page navigation rules. Actual, useful crawling (i.e. to build search indices) would be much messier and more useless without most interesting sites putting up meaningful robots.txt guide-rails. Look at facebook.com/robots.txt and consider how much crap both Facebook and indexers would have to deal with lacking that information.