5 ms·

It all makes sense if you believe in an information is universal idea where it should all be freely available. Though, you you get cases like celebritynetworth

It all makes sense if you believe in an information is universal idea where it should all be freely available. Though, you you get cases like celebritynetworth.com (no affiliation to me) that was featured on HN where Google wanted to APIize the data and when that was not available they decided to scrape it. It effectively killed the business. I think if a company decides to offer an API or other structured data then fine, they've decided that their data can be shared in a way that makes it OK to collect en masse. I wouldn't say we should "learn to love" scraping when there are sites that put a lot of man hours into the data that's seen on public html pages only for someone else to spend a few hours scraping it and repurpose it. Now, if the major traffic drivers like search engines and social networks were able to differentiate who the true authoritative source is, that'd be great. But they don't.

cik2e 6y ago
Link to the celebritynetworth.com discussion: https://news.ycombinator.com/item?id=24105465 https://news.ycombinator.com/item?id=24105465
wolco 6y ago
The big problem is once google killed it no new information will come and google can't scrape what doesn't exist
pjc50 6y ago
The EU sui generis copyright in databases: https://europa.eu/youreurope/business/running-business/intellectual-property/database-protection/index_en.htm https://europa.eu/youreurope/business/running-business/intel... Not part of the Berne convention, and not applicable in the US.
wzdd 6y ago
There are a significant number of missing steps, in terms of moral justifiability, between "scraping information from the web for academic research purposes" and "using the scraped information to add value to our primary generator of advertisement-derived revenue".
- Nasrudith 6y ago
  What is there that needs justification for what the proverbial eyes can plainly see? That aside what is the moral license to academic research and what gives the right to gatekeep what everyone can see if they took an uncommon action? Both are in fact doing so to make a living indirectly for their institutions by using what was already out there and rely upon arcane means to derive income from the data through layers of separation.
lazyjeff 6y ago
The article avoids directly addressing this, but I think is the real elephant in the room -- whether you have the right to web scrape for research. The article says, "You might be able to use what you scrape, but it’s worth checking that you can also legally share it. Ideally, the website content licence [sic] will be readily available." In practice, the data you want to scrape either has 1) no license/information about whether they are okay with you scraping it (and if they are okay with it, they usually offer the source data anyways), or more commonly they 2) strictly prohibit it in the terms of service, in which case it's not clear whether a scraper which mimics a browser falls under fair use. It's an area where when you ask your university attorney, they exchange a few emails with you and then avoid making any decision. I think it's just because it's not well tested in court (at least when I encountered these situations), and depends on whether you'd be subject to a takedown notice or lawsuit.
- elwell 6y ago
  > it's not clear whether a scraper which mimics a browser falls under fair use Or what about a 'human scraper' a la Amazon MTurk?
  - hansvm 6y ago
    It's rarely the scraper that runs afoul of legal complications (as always, with exceptions); it's what you do with the scraped data.
- dvdkon 6y ago
  "licence" is the non-USA spelling.
  - abandonliberty 6y ago
    Possibly the more common spelling, given that there's more English speakers in India, Pakistan, and Nigeria than the USA. Very strange fact. Too bad I can't find a measure of English content written by country. https://en.wikipedia.org/wiki/List_of_countries_by_English-speaking_population https://en.wikipedia.org/wiki/List_of_countries_by_English-s...

cik2e 6y ago

Link to the celebritynetworth.com discussion: https://news.ycombinator.com/item?id=24105465 https://news.ycombinator.com/item?id=24105465

wolco 6y ago

The big problem is once google killed it no new information will come and google can't scrape what doesn't exist

pjc50 6y ago

The EU sui generis copyright in databases: https://europa.eu/youreurope/business/running-business/intellectual-property/database-protection/index_en.htm https://europa.eu/youreurope/business/running-business/intel... Not part of the Berne convention, and not applicable in the US.

wzdd 6y ago

There are a significant number of missing steps, in terms of moral justifiability, between "scraping information from the web for academic research purposes" and "using the scraped information to add value to our primary generator of advertisement-derived revenue".

Nasrudith 6y ago
What is there that needs justification for what the proverbial eyes can plainly see? That aside what is the moral license to academic research and what gives the right to gatekeep what everyone can see if they took an uncommon action? Both are in fact doing so to make a living indirectly for their institutions by using what was already out there and rely upon arcane means to derive income from the data through layers of separation.

lazyjeff 6y ago

The article avoids directly addressing this, but I think is the real elephant in the room -- whether you have the right to web scrape for research. The article says, "You might be able to use what you scrape, but it’s worth checking that you can also legally share it. Ideally, the website content licence [sic] will be readily available." In practice, the data you want to scrape either has 1) no license/information about whether they are okay with you scraping it (and if they are okay with it, they usually offer the source data anyways), or more commonly they 2) strictly prohibit it in the terms of service, in which case it's not clear whether a scraper which mimics a browser falls under fair use. It's an area where when you ask your university attorney, they exchange a few emails with you and then avoid making any decision. I think it's just because it's not well tested in court (at least when I encountered these situations), and depends on whether you'd be subject to a takedown notice or lawsuit.

elwell 6y ago
> it's not clear whether a scraper which mimics a browser falls under fair use Or what about a 'human scraper' a la Amazon MTurk?
- hansvm 6y ago
  It's rarely the scraper that runs afoul of legal complications (as always, with exceptions); it's what you do with the scraped data.
dvdkon 6y ago
"licence" is the non-USA spelling.
- abandonliberty 6y ago
  Possibly the more common spelling, given that there's more English speakers in India, Pakistan, and Nigeria than the USA. Very strange fact. Too bad I can't find a measure of English content written by country. https://en.wikipedia.org/wiki/List_of_countries_by_English-speaking_population https://en.wikipedia.org/wiki/List_of_countries_by_English-s...