4 ms·
I need to add their IP range to my blocklist.
by chinathrow 11y ago
I need to add their IP range to my blocklist.
- kuschku 11y agoWhy? People will just scrape your data anyway. What you put online is public. Treat the web the same as a public bulletin board. Everything you post will be readable by everyone. Trying to prevent this is pointless.
- pingswept 11y agoIt might be desirable to make data available to humans, but not bots, no?
- atemerev 11y agoOK, how about hiring hordes of cheap human scrapers from Mechanical Turk? Everything visible can and will be scraped if need arises, the only variable is price.
- Retra 11y agoYou're making a distinction that, if you were to actually explain it in a way that makes sense, would reveal a much more relevant solution to this problem. What's the difference between a machine scraper and a human scraper? Humans use machines to access websites. All your users are machines. And behind every machine is a human. If you don't want someone to access all your data in a short time-frame, then don't give it all away in a short time-frame.
- troymc 11y agoThere have been lawsuits where the company running the site-being-scraped sued the scraper and won. So while it's technically possible, there can be legal issues. See https://en.wikipedia.org/wiki/Web_scraping#Legal_issues https://en.wikipedia.org/wiki/Web_scraping#Legal_issues
- kuschku 11y agoIf I republish, yes. If I just download a site – for local use, or similar cases – it’s not an issue. And if I provide a scraper where the user customly has to enter the URL, it’s also technically just a very advanced webproxy, and also okay.
- troymc 11y agoThat is incorrect. Even if you don't republish the data, the scraping alone can be a violation of the site's Terms of Service, and you can be sued (and lose). See the Wikipedia article I linked above. Think about it from the point of view of the site being scraped. Suddenly your webserver is being slammed with traffic that's way beyond what any human would cause. It's costing you bandwidth and compute resources. You have every right to say no, scraping is not allowed (in your Terms of Service). It's then up to a court to decide if those Terms of Service are legally binding. Some courts have decided that yes, they are.
- kuschku 11y agoAnd where I am, courts decided that no, ToS are never binding. ToS are effectively worthless.
- brwnll 11y agoWell that sure is an arrogant reply, and not even close to true. In the US everything published (online or not) is copyrighted. Since you're clearly not a content creator, and have decided since you didn't work on it it has no value, but putting something online does not make content public domain.
- jumperjake 11y agoYou're confusing redistribution with scraping. Scraping publicly-accessible content is legal. Redistributing it is not (at least not in the US). Google scraps webpages as a core competency.
- chinathrow 11y agoGoogle also adheres to the robots.txt standard. Most of the scrapers I block don't.
- jumperjake 11y agoNot correct. Google will completely ignore the rules in robots.txt if it deems it acceptable. I think there's a link to this somewhere in this comment page.
- chinathrow 11y agoThey do not index the content but might add the URL, correct. You can have a meta noindex present and they won't index even the URL.
- kuschku 11y agoPutting it online allows me to scrape it, but not to republish it. I might just want to train a neural network on extracting data out of websites, and, for that usecase, wget -m every website I can find. How I use the data you present to me, as long as I don’t republish it, is my decision. If you give me a license to read it, you also give me the license to copy it onto up to 7 different media at the same time, and to show it to up to 7 friends at the same time.
- chinathrow 11y agoWhy? My site, my robots.txt - if the scrapers do not obey, then I'll block them.
- kuschku 11y agoNo. Your site is public. If it is available via HTTP without authorization, I can scrape it (but not republish). There have been in many countries (except the US) court decisions about this. There are ways to force authorization, if you don’t use them, your fault. If I just use TOR or VPNs or similar to get around your block, also, again, your issue. Unless you require authorization, your service is, by law, public. And if I want to store the site offline for later reading, or read it now in a browser, is nothing that you have to decide.
- chinathrow 11y agoWell, copyright does prohibit redistributing the scraped content. And that is what all 3rd party content scraping services essentially do. That violates copyright law in quite a few countries. If you scrape with your own tools, own servers and use it only within your company, I would think it's a different legal matter. No, it's not "by law" public, or could you cite a few laws for that, both US and EU. Would appreciate that.
- kuschku 11y agoIt is public because the HTTP standard defines it as public, and, if you claim to use this standard, its definitions apply to you (if you claim to support a standard, you have to accept that its definitions will be used in court against you) HTTP directly says you should use authorization to make stuff non-public. And third-party content scraping tools are NOT violating copyright law, as they do not redistribute it freely. Otherwise services like Opera Turbo – which scrapes a website, removes cruft, compresses data, and sends it to your phone – would also break copyright law, and there is a nice exception for this case, under US law, it’s covered under fair use. (Unless the scraping service makes your content accessible publicly – and even then, fair use applies usually, as in the case of pure archival like archive.org or archive.is)
- wrath 11y agoIf these guys are serious about their business they are using proxies to obfuscate who they are. There are many services now that are claiming to have a ton of IP addresses Luminati, Shadio.io and Nohodo are just a few examples.
- dchuk 11y agoSorry, I'm having trouble finding that second link: shadio. Can you confirm that's their address? Interested in checking them out
- wrath 11y agoSorry, my bad... it's shader.io
- dchuk 11y agoAwesome, thank you. They do look interesting, but their pricing confuses me. Are they just selling individual proxies? Or does $1.80 get you single exit nodes or threads basically?
- wrath 11y agoAs far as I know, shader.io sells individual proxies. Luminati and Nohodo are exit nodes priced by the amount of bandwidth that you use.
- chinathrow 11y agoIf these guys are serious about their business, then they should behave like a good netizen and obey robots.txt directives.