4 ms·
How do you deal with the legality of scraping? I was hired to scrape some economic related pages and build an excel file and email it, I got that done but not
by jcun4128 9y ago
How do you deal with the legality of scraping?
I was hired to scrape some economic related pages and build an excel file and email it, I got that done but not sure if I should try to host this and turn it into a service or just set it up for the client and let them deal with it. It's just personal use on their part.
- dotdi 9y agoI would not equate scraping with periodic searching for a keyword. The problematic part is when you scrape data off of websites, and the owners don't want you to do it; as in, they would not even be happy if you manually copied that stuff into an excel file for fun or profit.
- jcun4128 9y agoI can see that, although automating to decrease views of a page(from checking by refresh)... I suppose the end result is sign up in this case. So for me I'll set it up for them and them run it.
- amelius 9y ago> How do you deal with the legality of scraping? How did Google deal with it, when they started their search engine business?
- jcun4128 9y agoMy counter to this is, without Google's "algorithm" the search results would not be great/find what you want. And in this case websites want to volunteer/agree to index their site for visibility but then you still have to visit the site because conveniently The site summary ends before the content that you want. I don't know how I would find websites unless referred on some website. I kinda felt the same way about how Netflix started out but I think they probably had a deal to pay a portion of revenue from dvd's that they rented out.
- benji-york 9y agorobots.txt
- gspetr 9y agohttps://www.google.com/killer-robots.txt https://www.google.com/killer-robots.txt