4 ms·
User-agent: * Disallow: /directory/people/* Disallow: /directory/pages/* Done. Google crawls Facebook. Every social media monitoring company crawls Faceboo
by jdrock 17y ago
User-agent: *
Disallow: /directory/people/*
Disallow: /directory/pages/*
Done. Google crawls Facebook. Every social media monitoring company crawls Facebook (and every other social network). Tons of other companies do the same every day. I know this because many of these companies are our customers.
What really irks me is that Pete collected valuable data and whoever he shared it with was probably able to derive added value from that data - in ways that Facebook is not doing. Facebook is prohibiting value creation.
- Vivtek 17y agoGoogle can do it because Facebook doesn't really want to go up against money in making this a test case, and Google has baskets of cash. Facebook doesn't give a tinker's damn about prohibiting your value creation.
- rapind 17y agoOr because Google would simply comply with their demands and remove all mention of facebook from their indexes... That would be pretty interesting.
- Vivtek 17y agoHa! Yeah, I hadn't thought of that aspect. Half of their userbase wouldn't even be able to sign on any more!
- bluesmoon 17y agorather more than half I think
- robryan 17y agoI guess when they have direct gain, ie- people accessing facebook profiles through google searches, they are more than happy for this to occur.