8 ms·
They don't ask your permission, but they do respect robots.txt files as well as noindex, nofollow.
by nhebb 14y ago
They don't ask your permission, but they do respect robots.txt files as well as noindex, nofollow.
- jcampbell1 14y agoI am sure the orignal author does as well. This is more a question of "fair use", not robots.txt
- chii 14y agoIndeed - and i believe scraping data off a publically accessible webpage (e.g., one where you do not require registration and login/password) falls under fair use, provided you do not take up more bandwidth/resources than the average user of that site. I recall there being some precedent for this sort of fair use - something like a phone directory - the information is not copyrighted, but the arrangement and layout is. So hence, you can't just iframe a site and present it, but obtaining the data, and deriving a new work from it should fall under fair use.
- benmccann 14y agoThat court case was Feist v. Rural. The court ruled that copying the listings in a phone book was permitted because facts cannot be copyrighted.