When I worked at Spock we had pretty much the same policy. Spock mined everything it could, including corporate bios, news articles, phone directories, etc. It

When I worked at Spock we had pretty much the same policy. Spock mined everything it could, including corporate bios, news articles, phone directories, etc. It wasn't deeply thought out; we just didn't have time to work it out fully. There was a time when people were regularly upset and surprised at what search engines came across. It's not that the data are public, but how they are collated and presented. Over time a balance was struck, mostly in favor of the search engines. I'm not sure what the right balance is here. At the very least there should be some sort of opt-out in the same manner as robots.txt or removing yourself from Facebook search results.