3 ms·

I think that Facebook has a right to protect their data, but leaving robots.txt wide open and trying to enforce some random TOS that a developer might not even

I think that Facebook has a right to protect their data, but leaving robots.txt wide open and trying to enforce some random TOS that a developer might not even know about is the wrong way to do it. I think a much better method would be to ban all bots from robots.txt (and crawling) except the major search engines and provide data to others through APIs or other means that are controlled (and the TOS can apply to the API rather than random crawling). Maybe not what a developer wants to hear, but the truth is that the internet is infested with bots that do nefarious things, and I think it's better for the privacy and security of their userbase to control the spread of their data as much as possible.

Aaronontheweb 16y ago
And I absolutely agree with you - it's the subjectivity in their choice of whether or not something actually violates their policies which gives me pause. If you can't figure out in good faith if you're actually complying with their rules or not then that's a problem for developers, myself and Pete included.
scott_s 16y ago
I am unfamiliar with the details of a robots.txt file. Is it possible to specify "I only want bots x, y and z to crawl my site"?
- curio 16y ago
  yes, using the user-agent: http://en.wikipedia.org/wiki/Robots_exclusion_standard http://en.wikipedia.org/wiki/Robots_exclusion_standard
  - scott_s 16y ago
    How difficult is it for a bot to lie? Though if you've made it clear that only x, y and z can crawl your site, and someone spoofs, say, y, then it would be easy to demonstrate that someone has done something they know they shouldn't.
    - curio 16y ago
      incredibly easy. and not only can the bot lie, it can disregard the robots.txt file altogether. just like the terms of service document for humans, you can choose to disregard it & deal w/ the consequences (blocked IP's, lawsuit, etc). robots.txt is just a version of the TOS that computers can read.
keltex 16y ago
They could also make the pages NOT indexable and then have a side agreement with Google to give Google API access to index the pages they want.
- duck 16y ago
  What? So you want to just make Google more powerful? How about all the other search engines, even the less known ones like Duck Duck Go?
gbhn 16y ago
Robots.txt governs automated retrieval of a page's contents. I don't think it makes any comments about what the retriever does with that data. Those uses are governed by other sets of rules. For instance, just because a site's robots.txt invites automated retrieval doesn't make it a legitimate use to just re-host that content elsewhere as your own. I think Facebook's ToS is an appropriate way for them to send messages about what they'll do if you make certain uses of the content you retrieve from their site. However, these ToS documents aren't omnipotent: they can't restrict some fair uses to which you might put the data, for instance, or bind you to silly terms. IANAL, but I think the ToS is probably best understood as an intent to use still other sets of rules (perhaps selectively) if you do certain things with retrieved data. Disagreeing with that policy is certainly possible, but I don't think robots.txt has much to do with it.

Aaronontheweb 16y ago

And I absolutely agree with you - it's the subjectivity in their choice of whether or not something actually violates their policies which gives me pause. If you can't figure out in good faith if you're actually complying with their rules or not then that's a problem for developers, myself and Pete included.

scott_s 16y ago

I am unfamiliar with the details of a robots.txt file. Is it possible to specify "I only want bots x, y and z to crawl my site"?

curio 16y ago
yes, using the user-agent: http://en.wikipedia.org/wiki/Robots_exclusion_standard http://en.wikipedia.org/wiki/Robots_exclusion_standard
- scott_s 16y ago
  How difficult is it for a bot to lie? Though if you've made it clear that only x, y and z can crawl your site, and someone spoofs, say, y, then it would be easy to demonstrate that someone has done something they know they shouldn't.
  - curio 16y ago
    incredibly easy. and not only can the bot lie, it can disregard the robots.txt file altogether. just like the terms of service document for humans, you can choose to disregard it & deal w/ the consequences (blocked IP's, lawsuit, etc). robots.txt is just a version of the TOS that computers can read.

keltex 16y ago

They could also make the pages NOT indexable and then have a side agreement with Google to give Google API access to index the pages they want.

duck 16y ago
What? So you want to just make Google more powerful? How about all the other search engines, even the less known ones like Duck Duck Go?

gbhn 16y ago

Robots.txt governs automated retrieval of a page's contents. I don't think it makes any comments about what the retriever does with that data. Those uses are governed by other sets of rules. For instance, just because a site's robots.txt invites automated retrieval doesn't make it a legitimate use to just re-host that content elsewhere as your own. I think Facebook's ToS is an appropriate way for them to send messages about what they'll do if you make certain uses of the content you retrieve from their site. However, these ToS documents aren't omnipotent: they can't restrict some fair uses to which you might put the data, for instance, or bind you to silly terms. IANAL, but I think the ToS is probably best understood as an intent to use still other sets of rules (perhaps selectively) if you do certain things with retrieved data. Disagreeing with that policy is certainly possible, but I don't think robots.txt has much to do with it.