3 ms·
Won't this robots.txt update stop major search engines from indexing Twitter? Seems like a great way to kill revenue as Twitter vanishes from search results.
by grumple 3y ago
Won't this robots.txt update stop major search engines from indexing Twitter? Seems like a great way to kill revenue as Twitter vanishes from search results.
- jraph 3y agoI wonder if search engines actually use rules for Googlebot when parsing some (specific?) robots.txt. "Googlebot" might as well be the "Mozilla/5.0" of search index bot user agents.
- wiktor-k 3y agoTwitter is benefiting from network effects by now. I'm not even sure when have I encountered a tweet in search results last time.
- deleted 3y ago[deleted]
- jeroenhd 3y agoI encounter tweets all the time when I'm trying to find out more about recent events.
- thisismyhna 3y agoDoes twitter rely on that? Most of the time I go to twitter is from news stories, comments or other social media.
- sundaeofshock 3y agoAny revenue damage from this will be faulty minor compared to the damage that has already been done.
- dahfizz 3y agoThey have a section in the robots.txt that allows Google. They must think no other search engine matters.