5 ms·
Twitter just updated its robots.txt to exclude all scrapers
- sb057 11y agoSame file as of a few hours ago: https://web.archive.org/web/20150715164726/https://twitter.com/robots.txt https://web.archive.org/web/20150715164726/https://twitter.c...
- kenkasan 11y agoso what does that mean?
- minimaxir 11y agoAbsolutely nothing from Twitter should be appearing in search engines.
- chc 11y agoExcept don't they have a special deal with Google to use the firehose?
- guiambros 11y agoIMO, that's exactly the reason. Before search engines could scrape the data and load the content for free. Now they'll need to reach firehose data agreements.
- Paulods 11y agoTake a look at the robots file without the "www" subdomain. Its likely to prevent content duplication/push for only the non "www" url to appear on search engines. https://twitter.com/robots.txt https://twitter.com/robots.txt
- guiambros 11y agoUh... Excellent point. Seems the entire speculation is/was unfounded.
- teaneedz 11y agoSo only firehose agreements get data? What's the impact to general SEO or 3rd parties? Doesn't sound good.
- Paulods 11y agoActually i believe its only from the "www" subdomain. Take a look at the robots without the "www" https://www.twitter.com/robots.txt https://www.twitter.com/robots.txt https://twitter.com/robots.txt https://twitter.com/robots.txt This is likely just to prevent content duplication/nudge users to visit without the "www".
- thenextcorner 11y agoyep makes a huge difference... as explained here as well http://webmarketingschool.com/no-twitter-did-not-just-de-index/ http://webmarketingschool.com/no-twitter-did-not-just-de-ind...
- sp332 11y agoYou should never use Google's estimate as a real estimate, especially once it gets past 100. There are better ways to move content, like 301 redirects or rel=canonical.
- searchmartin 11y agoNope. No. They didn't. What they did was some perfectly legitimate duplicate content protection. Will write it up in a bit more detail...
- mattbasta 11y agoNah. https://twitter.com/robots.txt https://twitter.com/robots.txt They blocked robots on their marketing pages.
- searchmartin 11y agoagain, no, they blocked it on WWW. only: http://webmarketingschool.com/no-twitter-did-not-just-de-index/ http://webmarketingschool.com/no-twitter-did-not-just-de-ind...
- searchmartin 11y agoHere's a write up as to why they have made the changes. Was going to write it yesterday, but life got in the way: http://webmarketingschool.com/no-twitter-did-not-just-de-index/ http://webmarketingschool.com/no-twitter-did-not-just-de-ind...
- sp332 11y agoWhy would you not use 301 redirects or just rel=canonical?
- searchmartin 11y agoThere must be some platform issue is my best guess...
- sp332 11y agoThey could at least have allowed the Internet Archive in the robots.txt, since the way things stand all www.twitter.com links will be unavailable from the Wayback Machine. That will obviously be a huge loss to researchers.
- sp332 11y ago(Update: the Wayback Machine will be fine, using the twitter.com/robots.txt)