3 ms·
> My experience is that many people want to buy scraped data that is against a large website's TOS such as Google, Amazon or Craigslist and the act of selling t
by ggm 3y ago
> My experience is that many people want to buy scraped data that is against a large website's TOS such as Google, Amazon or Craigslist and the act of selling that data directly in a marketplace is likely to draw legal action from these companies as it's someone else's copyrighted data.
I came here to make substantively the same point. Scraping for your own personal use and re-use as a derived product is reasonably cool but can border into IPR concerns. Scraping to re-sell as a product, thats really stepping over a line.
This has been a problem since the days of the yellow pages phonebook. There are rules around when a catalog is in the public domain, page contents with data I don't think cross into the "its ok to just copy and resell" space.
I scrape several sources. World population and GDP stats are highly useful but finding good canonical sources was hard for a while there. I now use a Python API provided by the world bank, for most of them. Now I just have the problem that by economy, the inputs span 2019-2022 and I can't get a single authority for population estimates right "now" worldwide. But, scraping to resell that? I think it would be wrong.
- bloppe 3y agoScraping internet data is perfectly fine and legal. It may be subject to copyright or ToS (which are not the same) but as long as you don't redistribute copyrighted content, or affirmatively accept any ToS, you're doing nothing wrong. So the potential risk lies entirely with the seller, not the buyer, and could potentially be reduced or circumvented in various ways. Just saying
- ggm 3y agoYes. but the article was almost entirely about being that seller. The premise was, "why don't more people scrape the web and sell it" -And if I was an IPR owner and detected a scraper, and then tracked back from their origin IP robot to their web service, I'd be sending my legals round. Offering a brokering service to "share" access to the investment in scraping, has to be done carefully. If for example its brokering access to the page specific structure in some meta notation and you as a consumer scrape direct? thats actually quite cool. If its running a cache of the data re-presented in JSON form for you to consume, or monetising an Elastic Stack of the data and you read it from them? I think its getting iffy to be that seller.
- Havoc 3y ago> Scraping internet data is perfectly fine and legal. I doubt that is true for all jurisdictions
- squatrito23 3y agoIt all comes down to what data: PII (personally Identifiable Info) or IP (Intellectual Property) are strongly protected. There are a lot of useful info not PII or IP that can be collected, used and sold, like prices (Ryanair case against Expedia being a crucial case)
- Havoc 3y agoI'm somewhat ignorant on this matter, but I was under the impression that in some jurisdictions (e.g. UK) scraping as a whole sale concept is pretty grey area at best not just on PII & IP
- kmeisthax 3y agoRegarding the legality of scraping factual data... In the EU there's a 'database right' that covers specific collections of facts. The US has categorically refused to issue any kind of "IP"[0] on facts, and doesn't have 'database rights', but in practice you can sprinkle a few fictitious bits of information in your dataset, retain copyright on the lies, and sue people who don't independently verify the factuality of every bit of information in the dataset. So the US has database rights. No clue if people are sticking fictional data on their websites to sue scrapers with, but I wouldn't be surprised if they were. [0] Intellectual property is the right to dictate to competitors the conditions upon which they are allowed to compete, if any.