8 ms·

Facebook employee responds on robots.txt controversy

54 points by petewarden 16y ago

Aaronontheweb 16y ago
Pete's right-on about this: "You've chosen to leave all that information out in the open so you can benefit from the search traffic, and instead try to change the established rules of the web so you can selectively sue anyone you decide is a threat. " Speaking as someone who's working on leveraging the Facebook API in a commercial product, this leaves me feeling like I'm opening myself to a lot of legal exposure if Facebook subjectively decides that my service poses even a minor threat to them. Given that I'm bootstrapping, there's no way I'd be able to put up any sort of legal fight what so ever against a company as well-funded as Facebook.
- sounddust 16y ago
  I think that Facebook has a right to protect their data, but leaving robots.txt wide open and trying to enforce some random TOS that a developer might not even know about is the wrong way to do it. I think a much better method would be to ban all bots from robots.txt (and crawling) except the major search engines and provide data to others through APIs or other means that are controlled (and the TOS can apply to the API rather than random crawling). Maybe not what a developer wants to hear, but the truth is that the internet is infested with bots that do nefarious things, and I think it's better for the privacy and security of their userbase to control the spread of their data as much as possible.
  - Aaronontheweb 16y ago
    And I absolutely agree with you - it's the subjectivity in their choice of whether or not something actually violates their policies which gives me pause. If you can't figure out in good faith if you're actually complying with their rules or not then that's a problem for developers, myself and Pete included.
  - scott_s 16y ago
    I am unfamiliar with the details of a robots.txt file. Is it possible to specify "I only want bots x, y and z to crawl my site"?
    - curio 16y ago
      yes, using the user-agent: http://en.wikipedia.org/wiki/Robots_exclusion_standard http://en.wikipedia.org/wiki/Robots_exclusion_standard
      - scott_s 16y ago
        How difficult is it for a bot to lie? Though if you've made it clear that only x, y and z can crawl your site, and someone spoofs, say, y, then it would be easy to demonstrate that someone has done something they know they shouldn't.
        curio 16y ago
        incredibly easy. and not only can the bot lie, it can disregard the robots.txt file altogether. just like the terms of service document for humans, you can choose to disregard it & deal w/ the consequences (blocked IP's, lawsuit, etc). robots.txt is just a version of the TOS that computers can read.
  - keltex 16y ago
    They could also make the pages NOT indexable and then have a side agreement with Google to give Google API access to index the pages they want.
    - duck 16y ago
      What? So you want to just make Google more powerful? How about all the other search engines, even the less known ones like Duck Duck Go?
  - gbhn 16y ago
    Robots.txt governs automated retrieval of a page's contents. I don't think it makes any comments about what the retriever does with that data. Those uses are governed by other sets of rules. For instance, just because a site's robots.txt invites automated retrieval doesn't make it a legitimate use to just re-host that content elsewhere as your own. I think Facebook's ToS is an appropriate way for them to send messages about what they'll do if you make certain uses of the content you retrieve from their site. However, these ToS documents aren't omnipotent: they can't restrict some fair uses to which you might put the data, for instance, or bind you to silly terms. IANAL, but I think the ToS is probably best understood as an intent to use still other sets of rules (perhaps selectively) if you do certain things with retrieved data. Disagreeing with that policy is certainly possible, but I don't think robots.txt has much to do with it.
tjmaxal 16y ago
this is essentially the same argument people use against the iOS platform. But clearly from Apple's example it is possible to build a profitable product on a closed system. Admittedly it doesn't seem "fair" or true to the spirit of the internet. but it shouldn't be any surprise that a company is going to do what is best for that company forsaking all others.
curio 16y ago
this is a really important issue. this new model that facebook is trying to push isn't scalable. it favors the big guys and it's bad for the open web. you shouldn't have a TOS that contradicts your robots.txt. period.
jrussbowman 16y ago
Sorry, as implied to my comment on the first story about this posted to Hacker News, I'm siding with Facebook on this one. Of course they tune their page so that search crawlers can best index the information, so does everyone else on the web. However, they also provide an API, with clearly defined terms of use, which you may use to get information. Basically your complaint boils down to you don't want to use the methods they've set up for you to access their data, and you're complaining about it. As for comments about fair, true spirit of the internet what have you... I don't think the true spirit of the internet has ever been everyone has to give everything away in every possible imaginable way. And, in the end, it is Facebook's data, they make that clear before you ever start adding data to their servers. So does just about everyone else who allows you to submit data to their servers. This shouldn't be confused with how difficult it is to remove data and accounts from their system, which is a giant pain in the butt, or the fact that they've made drastic changes to the public nature of the data after keeping it private for so long. That's all just a giant mess.
- jdrock 16y ago
  It's not Facebook's data. It's the users' data.
  - xtacy 16y ago
    I am just logically arguing here, not in support for anyone. But hasn't Facebook worked "hard" to setup the infrastructure to "collect" users' data? If you want users' data, go ask the users; not via scraping Facebook.
    - curio 16y ago
      but their robots.txt says that you're welcome to scrape it: User-agent: * Disallow: /ac.php Disallow: /ae.php Disallow: /album.php Disallow: /ap.php Disallow: /feeds/ Disallow: /o.php Disallow: /p.php Disallow: /photo_comments.php Disallow: /photo_search.php Disallow: /photos.php
- natrius 16y ago
jdrock 16y ago
In my personal opinion, the Facebook employee's response shows that he doesn't understand the company's high-level business goals. Their developers may think they are building an open web, but it's very clear the company is only interested in a closed web they control. Based on my interaction with website API developers, most of them honestly believe that they are building an open web. But let's face it, the API is a benefit to them and creates a critical dependency for the user of the API. It's basically vendor lock-in.
- natrius 16y ago
  As far as I know, any data that people post to Facebook without restricted visibility can be pulled through the API. If a user has logged into your site and given you access to their data, you can access restricted data as well. You can also write to Facebook via the API. I'm not sure what you want Facebook to change.
gkoberger 16y ago
"Facebook has always been a closed system where developers are expected to live in a culture of asking permission before doing anything, and existing at the whim of the company's management. [...] The web I love is an open world where you are free to innovate as long as you stick to the mutually agreed rules." Facebook's policies aside, it's interesting to note that this criticism was written in response to a letter from Blake Ross- the founder of Firefox.
ojbyrne 16y ago
Facebook isn't the only site with that language. In fact I think you'll have to look hard to find any major site that doesn't have it. Not that it doesn't suck.
finiteloop 16y ago
This is Bret Taylor, CTO of Facebook. There are a couple of things I want to clarify. First, we genuinely support data portability: we want users to be able to use their data in other applications without restriction. Our new data policies, which we deployed at f8, clearly reflect this (http://developers.facebook.com/policy/ http://developers.facebook.com/policy/): "Users give you their basic account information when they connect with your application. For all other data, you must obtain explicit consent from the user who provided the data to us before using it for any purpose other than displaying it back to the user on your application." Basically, users have complete control over their data, and as long as user gives an application explicit consent, Facebook doesn't get in the way of the user using their data in your applications beyond basic protections like selling data to ad networks and other sleazy data collectors. Crawling is a bit of special case. We have a privacy control enabling users to decide whether they want their profile page to show up in search engines. Many of the other "crawlers" don't really meet user expectations. As Blake mentioned in his response on Pete's blog post, some sleazy crawlers simply aggregate user data en masse and then sell it, which we view as a threat to user privacy. Pete's post did bring up some real issues with the way we were handling things. In particular, I think it was bad for us to stray from Internet standards and conventions by having an robots.txt that was open and a separate agreement with additional restrictions. This was just a lapse of judgment. We are updating our robots.txt to explicitly allow the crawlers of search engines that we currently allow to index Facebook content and disallow all other crawlers. We will whitelist crawlers when legitimate companies contact us who want to crawl us (presumably search engines). For other purposes, we really want people using our API because it has explicit controls around privacy and has important additional requirements that we feel are important when a company is using users' data from Facebook (e.g., we require that you have a privacy policy and offer users the ability to delete their data from your service). This robots.txt change should be deployed today. The change will make our robots.txt abide by conventions and standards, which I think is the main legitimate complaint in Pete's post.
- petewarden 16y ago
  Thanks Bret, I appreciate you taking the time to dig into this. You're right, my complaint is the disconnect between the terms-of-service and robots.txt, so I'm very happy you're addressing this. I just wish we could have had this conversation a few months ago, before you guys threw your lawyers at me.
sd273 16y ago
I'm tired of hearing people complain about private companies' rules for using their information/platforms. Facebook and Apple can do whatever they want on their own platforms and if people don't like it they can leave/not use the service. That's how a free market and free web works. People complained about Microsoft's dominance years ago and now their ascendancy is ending due to this idea called the "market"...
- pyre 16y ago
  The issue that a lot of people have is that they would like to see change happen in their lifetime. Microsoft is in a decline now, but that doesn't mean that: 1) it will continue to decline or 2) it will completely fail within our lifetimes. Once companies hit a certain level, they take a long time to completely fail due to the accumulated money and power during their 'good years.' Change, in general, doesn't come about by people sitting around doing nothing but saying, 'the market will decide.' If you take the time to become an advocate for a certain 'side,' you can affect the market through public perception. [aside: People are not always logical agents with access to full information like most economic theories seem to assume. To me this comes off like a lot of physics where things are assumed to be on a 'perfectly frictionless surface,' 'in a vaccum', etc.]

54 points by petewarden 16y ago

Pete's right-on about this: "You've chosen to leave all that information out in the open so you can benefit from the search traffic, and instead try to change the established rules of the web so you can selectively sue anyone you decide is a threat. " Speaking as someone who's working on leveraging the Facebook API in a commercial product, this leaves me feeling like I'm opening myself to a lot of legal exposure if Facebook subjectively decides that my service poses even a minor threat to them. Given that I'm bootstrapping, there's no way I'd be able to put up any sort of legal fight what so ever against a company as well-funded as Facebook.

sounddust 16y ago
I think that Facebook has a right to protect their data, but leaving robots.txt wide open and trying to enforce some random TOS that a developer might not even know about is the wrong way to do it. I think a much better method would be to ban all bots from robots.txt (and crawling) except the major search engines and provide data to others through APIs or other means that are controlled (and the TOS can apply to the API rather than random crawling). Maybe not what a developer wants to hear, but the truth is that the internet is infested with bots that do nefarious things, and I think it's better for the privacy and security of their userbase to control the spread of their data as much as possible.
- Aaronontheweb 16y ago
  And I absolutely agree with you - it's the subjectivity in their choice of whether or not something actually violates their policies which gives me pause. If you can't figure out in good faith if you're actually complying with their rules or not then that's a problem for developers, myself and Pete included.
- scott_s 16y ago
  I am unfamiliar with the details of a robots.txt file. Is it possible to specify "I only want bots x, y and z to crawl my site"?
  - curio 16y ago
    yes, using the user-agent: http://en.wikipedia.org/wiki/Robots_exclusion_standard http://en.wikipedia.org/wiki/Robots_exclusion_standard
    - scott_s 16y ago
      How difficult is it for a bot to lie? Though if you've made it clear that only x, y and z can crawl your site, and someone spoofs, say, y, then it would be easy to demonstrate that someone has done something they know they shouldn't.
      - curio 16y ago
        incredibly easy. and not only can the bot lie, it can disregard the robots.txt file altogether. just like the terms of service document for humans, you can choose to disregard it & deal w/ the consequences (blocked IP's, lawsuit, etc). robots.txt is just a version of the TOS that computers can read.
- keltex 16y ago
  They could also make the pages NOT indexable and then have a side agreement with Google to give Google API access to index the pages they want.
  - duck 16y ago
    What? So you want to just make Google more powerful? How about all the other search engines, even the less known ones like Duck Duck Go?
- gbhn 16y ago
  Robots.txt governs automated retrieval of a page's contents. I don't think it makes any comments about what the retriever does with that data. Those uses are governed by other sets of rules. For instance, just because a site's robots.txt invites automated retrieval doesn't make it a legitimate use to just re-host that content elsewhere as your own. I think Facebook's ToS is an appropriate way for them to send messages about what they'll do if you make certain uses of the content you retrieve from their site. However, these ToS documents aren't omnipotent: they can't restrict some fair uses to which you might put the data, for instance, or bind you to silly terms. IANAL, but I think the ToS is probably best understood as an intent to use still other sets of rules (perhaps selectively) if you do certain things with retrieved data. Disagreeing with that policy is certainly possible, but I don't think robots.txt has much to do with it.

tjmaxal 16y ago

this is essentially the same argument people use against the iOS platform. But clearly from Apple's example it is possible to build a profitable product on a closed system. Admittedly it doesn't seem "fair" or true to the spirit of the internet. but it shouldn't be any surprise that a company is going to do what is best for that company forsaking all others.

curio 16y ago

this is a really important issue. this new model that facebook is trying to push isn't scalable. it favors the big guys and it's bad for the open web. you shouldn't have a TOS that contradicts your robots.txt. period.

jrussbowman 16y ago

Sorry, as implied to my comment on the first story about this posted to Hacker News, I'm siding with Facebook on this one. Of course they tune their page so that search crawlers can best index the information, so does everyone else on the web. However, they also provide an API, with clearly defined terms of use, which you may use to get information. Basically your complaint boils down to you don't want to use the methods they've set up for you to access their data, and you're complaining about it. As for comments about fair, true spirit of the internet what have you... I don't think the true spirit of the internet has ever been everyone has to give everything away in every possible imaginable way. And, in the end, it is Facebook's data, they make that clear before you ever start adding data to their servers. So does just about everyone else who allows you to submit data to their servers. This shouldn't be confused with how difficult it is to remove data and accounts from their system, which is a giant pain in the butt, or the fact that they've made drastic changes to the public nature of the data after keeping it private for so long. That's all just a giant mess.

jdrock 16y ago
It's not Facebook's data. It's the users' data.
- xtacy 16y ago
  I am just logically arguing here, not in support for anyone. But hasn't Facebook worked "hard" to setup the infrastructure to "collect" users' data? If you want users' data, go ask the users; not via scraping Facebook.
  - curio 16y ago
    but their robots.txt says that you're welcome to scrape it: User-agent: * Disallow: /ac.php Disallow: /ae.php Disallow: /album.php Disallow: /ap.php Disallow: /feeds/ Disallow: /o.php Disallow: /p.php Disallow: /photo_comments.php Disallow: /photo_search.php Disallow: /photos.php
natrius 16y ago

jdrock 16y ago

In my personal opinion, the Facebook employee's response shows that he doesn't understand the company's high-level business goals. Their developers may think they are building an open web, but it's very clear the company is only interested in a closed web they control. Based on my interaction with website API developers, most of them honestly believe that they are building an open web. But let's face it, the API is a benefit to them and creates a critical dependency for the user of the API. It's basically vendor lock-in.

natrius 16y ago
As far as I know, any data that people post to Facebook without restricted visibility can be pulled through the API. If a user has logged into your site and given you access to their data, you can access restricted data as well. You can also write to Facebook via the API. I'm not sure what you want Facebook to change.

gkoberger 16y ago

"Facebook has always been a closed system where developers are expected to live in a culture of asking permission before doing anything, and existing at the whim of the company's management. [...] The web I love is an open world where you are free to innovate as long as you stick to the mutually agreed rules." Facebook's policies aside, it's interesting to note that this criticism was written in response to a letter from Blake Ross- the founder of Firefox.

ojbyrne 16y ago

Facebook isn't the only site with that language. In fact I think you'll have to look hard to find any major site that doesn't have it. Not that it doesn't suck.

finiteloop 16y ago

This is Bret Taylor, CTO of Facebook. There are a couple of things I want to clarify. First, we genuinely support data portability: we want users to be able to use their data in other applications without restriction. Our new data policies, which we deployed at f8, clearly reflect this (http://developers.facebook.com/policy/ http://developers.facebook.com/policy/): "Users give you their basic account information when they connect with your application. For all other data, you must obtain explicit consent from the user who provided the data to us before using it for any purpose other than displaying it back to the user on your application." Basically, users have complete control over their data, and as long as user gives an application explicit consent, Facebook doesn't get in the way of the user using their data in your applications beyond basic protections like selling data to ad networks and other sleazy data collectors. Crawling is a bit of special case. We have a privacy control enabling users to decide whether they want their profile page to show up in search engines. Many of the other "crawlers" don't really meet user expectations. As Blake mentioned in his response on Pete's blog post, some sleazy crawlers simply aggregate user data en masse and then sell it, which we view as a threat to user privacy. Pete's post did bring up some real issues with the way we were handling things. In particular, I think it was bad for us to stray from Internet standards and conventions by having an robots.txt that was open and a separate agreement with additional restrictions. This was just a lapse of judgment. We are updating our robots.txt to explicitly allow the crawlers of search engines that we currently allow to index Facebook content and disallow all other crawlers. We will whitelist crawlers when legitimate companies contact us who want to crawl us (presumably search engines). For other purposes, we really want people using our API because it has explicit controls around privacy and has important additional requirements that we feel are important when a company is using users' data from Facebook (e.g., we require that you have a privacy policy and offer users the ability to delete their data from your service). This robots.txt change should be deployed today. The change will make our robots.txt abide by conventions and standards, which I think is the main legitimate complaint in Pete's post.

petewarden 16y ago
Thanks Bret, I appreciate you taking the time to dig into this. You're right, my complaint is the disconnect between the terms-of-service and robots.txt, so I'm very happy you're addressing this. I just wish we could have had this conversation a few months ago, before you guys threw your lawyers at me.

sd273 16y ago

I'm tired of hearing people complain about private companies' rules for using their information/platforms. Facebook and Apple can do whatever they want on their own platforms and if people don't like it they can leave/not use the service. That's how a free market and free web works. People complained about Microsoft's dominance years ago and now their ascendancy is ending due to this idea called the "market"...

pyre 16y ago
The issue that a lot of people have is that they would like to see change happen in their lifetime. Microsoft is in a decline now, but that doesn't mean that: 1) it will continue to decline or 2) it will completely fail within our lifetimes. Once companies hit a certain level, they take a long time to completely fail due to the accumulated money and power during their 'good years.' Change, in general, doesn't come about by people sitting around doing nothing but saying, 'the market will decide.' If you take the time to become an advocate for a certain 'side,' you can affect the market through public perception. [aside: People are not always logical agents with access to full information like most economic theories seem to assume. To me this comes off like a lot of physics where things are assumed to be on a 'perfectly frictionless surface,' 'in a vaccum', etc.]