4 ms·

What sorts of real-world and legitimate/ethical use cases are there for wholesale repeated scraping?

macsj200 9y ago
Internet archival initiatives are a legitimate use case, as well as things like sneakernet in Cuba.
mickeyp 9y ago
Scraping websites like AliExpress and Ebay for trademark and brand infringement. There are many vendors who sell fake tat on the internet, and there are companies out there who track it down on behalf of the brand owners. These companies tend to have armies of lawyers who can swat away even the likes of eBay when it comes to justifying web scraping. Nevertheless, the work required is still the same tedium that others deal with: CAPTCHAs; throttling; IP bans; etc.
salawat 9y ago
I'm kind of interested in an answer to this as well. I know the typical "travel site" or "comparison shop" use case. There is also the "darn it, I want this" use case. However, automated, periodic web scraping that mutates (I.e. ticket or reservation grabbing bots) has always felt a bit squicky to me in the same way DNS squatting does.
- megous 9y ago
  Website you like that invites discussion has a very shitty comment system, that doesn't notify you of the responses to your comments. Or perhaps you want to monitor mentions of your name, and join the discussion. Or you want to not lose your past thoughts, because discussions were deep and some may be interesting to re-read in the future. Or a website is known to allow users delete their comments, or website itself bans users and hides all their content from others for flimsy reasons like "ISIS" in the title, no matter whether it's pro/against/neutral/irrelevant. Or you want to organize the information differently than the site does. Or the website is bloated and slow as hell, and you want to use it over gprs, so you create a lightweight/fast/better organized/filtered mirror. Or the website doesn't have search or it is crap/slooow, and is not indexed in google, like some IRC logs out there. Or you want to have content available offline. ...
lazycouchpotato 9y ago
Statistics I'd say is one of the useful cases of scraping. Back in 2013, a guy scraped the results of about 150,000 students giving their 10th grade finals of a particular examination board in India. He showed that not only was there no privacy of student's marks because the roll numbers were all linearly incremented, but there was also mass-scale manipulation of marks going on. The concept is simple but it's a very interesting read. https://deedy.quora.com/Hacking-into-the-Indian-Education-System?srid=d5K&share=1 https://deedy.quora.com/Hacking-into-the-Indian-Education-Sy... I was one of the 150,000 kids that gave those exams back in 2013 :)
- reallymental 9y ago
  Facinating read, I cannot find out more about what happened to him after he was accused of 'hacking' the govt systems, do you have any more sources that sheds some light on that?
  - lazycouchpotato 9y ago
    I tweeted out to him and he responded. https://twitter.com/debarghya_das/status/988178000914022400 https://twitter.com/debarghya_das/status/988178000914022400
sheeshkebab 9y ago
Scrape government data... it’s all free but try to find it in any downloadable and consistent format.
time0ut 9y ago
I've used it to convert publicly available data to a more useful format. A local government agency made the data freely available as a series of thousands of HTML pages, but not in any other form. I didn't need to use any evasive tactics, but it was still wholesale repeated scraping.

Internet archival initiatives are a legitimate use case, as well as things like sneakernet in Cuba.

Scraping websites like AliExpress and Ebay for trademark and brand infringement. There are many vendors who sell fake tat on the internet, and there are companies out there who track it down on behalf of the brand owners. These companies tend to have armies of lawyers who can swat away even the likes of eBay when it comes to justifying web scraping. Nevertheless, the work required is still the same tedium that others deal with: CAPTCHAs; throttling; IP bans; etc.

salawat 9y ago

I'm kind of interested in an answer to this as well. I know the typical "travel site" or "comparison shop" use case. There is also the "darn it, I want this" use case. However, automated, periodic web scraping that mutates (I.e. ticket or reservation grabbing bots) has always felt a bit squicky to me in the same way DNS squatting does.

megous 9y ago
Website you like that invites discussion has a very shitty comment system, that doesn't notify you of the responses to your comments. Or perhaps you want to monitor mentions of your name, and join the discussion. Or you want to not lose your past thoughts, because discussions were deep and some may be interesting to re-read in the future. Or a website is known to allow users delete their comments, or website itself bans users and hides all their content from others for flimsy reasons like "ISIS" in the title, no matter whether it's pro/against/neutral/irrelevant. Or you want to organize the information differently than the site does. Or the website is bloated and slow as hell, and you want to use it over gprs, so you create a lightweight/fast/better organized/filtered mirror. Or the website doesn't have search or it is crap/slooow, and is not indexed in google, like some IRC logs out there. Or you want to have content available offline. ...

lazycouchpotato 9y ago

Statistics I'd say is one of the useful cases of scraping. Back in 2013, a guy scraped the results of about 150,000 students giving their 10th grade finals of a particular examination board in India. He showed that not only was there no privacy of student's marks because the roll numbers were all linearly incremented, but there was also mass-scale manipulation of marks going on. The concept is simple but it's a very interesting read. https://deedy.quora.com/Hacking-into-the-Indian-Education-System?srid=d5K&share=1 https://deedy.quora.com/Hacking-into-the-Indian-Education-Sy... I was one of the 150,000 kids that gave those exams back in 2013 :)

reallymental 9y ago
Facinating read, I cannot find out more about what happened to him after he was accused of 'hacking' the govt systems, do you have any more sources that sheds some light on that?
- lazycouchpotato 9y ago
  I tweeted out to him and he responded. https://twitter.com/debarghya_das/status/988178000914022400 https://twitter.com/debarghya_das/status/988178000914022400

sheeshkebab 9y ago

Scrape government data... it’s all free but try to find it in any downloadable and consistent format.

time0ut 9y ago

I've used it to convert publicly available data to a more useful format. A local government agency made the data freely available as a series of thousands of HTML pages, but not in any other form. I didn't need to use any evasive tactics, but it was still wholesale repeated scraping.