4 ms·
You are not entitled to scrape content from other sites, even if they contain publicly submitted data. I do support this sort of thing in the hacker spirit (as
by naner 14y ago
You are not entitled to scrape content from other sites, even if they contain publicly submitted data. I do support this sort of thing in the hacker spirit (as long as you aren't being too agressive) but selling an application using these techniques is downright unethical.
You have zero control over what you are selling to people. They don't know that you aren't using an offical means to get the data and the application could stop working at any time. And if you were scraping via the client (as opposed to collating data on a server of your own) you also put your customers in a position where they might be violating the site's TOS and risk reprecussions (ip ban, etc).
- jcampbell1 14y ago> You are not entitled to scrape content from other sites, even if they contain publicly submitted data. Isn't this Google's business model? They take snapshots of the public web and show organized and relevant snippets. Last time I checked, they never asked me for permission.
- nhebb 14y agoThey don't ask your permission, but they do respect robots.txt files as well as noindex, nofollow.
- jcampbell1 14y agoI am sure the orignal author does as well. This is more a question of "fair use", not robots.txt
- chii 14y agoIndeed - and i believe scraping data off a publically accessible webpage (e.g., one where you do not require registration and login/password) falls under fair use, provided you do not take up more bandwidth/resources than the average user of that site. I recall there being some precedent for this sort of fair use - something like a phone directory - the information is not copyrighted, but the arrangement and layout is. So hence, you can't just iframe a site and present it, but obtaining the data, and deriving a new work from it should fall under fair use.
- benmccann 14y agoThat court case was Feist v. Rural. The court ruled that copying the listings in a phone book was permitted because facts cannot be copyrighted.
- intended 14y agoAnd they can be asked to remove listings from their search. Ask rupert Murdoch
- deleted 14y ago[deleted]