3 ms·
Web scraping is a legal gray area in many or most jurisdictions. In some jurisdictions, depending on the tos of the web site itself, scraping it might be illega
by bjourne 5y ago
Web scraping is a legal gray area in many or most jurisdictions. In some jurisdictions, depending on the tos of the web site itself, scraping it might be illegal. In others republishing the scraped information in any form might be illegal. In others still you might not be allowed to use the scraped data for any commercial purpose.
"But what about Google?" Google is worth 100 billion dollars and can play by completely different rules than scraping startups.
- tempestn 5y agoYeah, I was going to say something like this. And one step better than Google, but still challenging, there are also a lot of sites out there that prohibit scraping but make an exception for "general purpose search engines". So if you're providing a specialized scraper that's used for specific purposes, you'd likely run into problems there too. As well as the actual legal challenges, you'll also have the perception thereof, which could make businesses wary of relying on your services.
- stef25 5y agoGoogle also plays nice. I manage a few sites that get hammered by scrapers to the extent that it causes big spikes in CPU, something that doesn't happen with G.
- that_guy_iain 5y agoSo many people don't even think about number of requests and often want the job to end as quickly as possible. For me, that is thinking short term, I've seen requests per second as low as 4 cause performance issues for a company. If you want to scrape long term you need to think requests per minute. That is unless you're dealing with a mega site then it's keeping a low request per second.
- pb7 5y ago>Google is worth 100 billion dollars $100B from $2T*. :-)