5 ms·
When scraping just behave as to not piss off the site owner - whatever that means. Eg. not cause excessive load or making sure you don't leak out sensitive data
by snidane 7y ago
When scraping just behave as to not piss off the site owner - whatever that means. Eg. not cause excessive load or making sure you don't leak out sensitive data.
Next put yourself in their shoes and realize they don't usually monitor their traffic that much or simply don't care as long as you don't slow down their site. It's usually only certain big sites with heavy bot traffic such as linkedin or sneaker shoe sites which implement bot protections. Most others don't care.
Some websites are created almost as if they want to be scraped. The json api used by frontend is ridiculously clean and accessible. Perhaps they benefit when people see their results and invest in their stock. You never fully know if the site wants to be scraped or not.
The reality of scraping industry related to your question is this
1. scraping companies generally don't use real user agent such as 'my friendly data science bot' but they hide behind a set of fake ones and/or route the traffic through a proxy network. You don't want to get banned so stupidly easily by revealing user agent when you know your competitors don't reveal theirs.
2. This one is obvious. The general rule is to scrape over long time period continuously and add large delays between requests of at least 1 second. If you go below 1 second be careful.
3. robots.txt is controversial and doesn't serve its original purpose. It should be renamed to google_instructions.txt because site owners use it to guide googlebot to navigate their site. It is generally ignored by the industry again because you know your competitors ignore it.
Just remember the rule of 'not to piss off the site owner' and then just go ahead and scrape. Also keep in mind that you are in a free country and we don't discriminate here whether it is of racial or gender reasons or whether you are a biological or mechanical website visitor.
I simply described the reality of data science industry around scraping after several years of being in it. Note that this will probably not be liked by HN audience as they are mostly website devs and site owners.
- wizzwizz4 7y ago1. is the only one I don't like. I think you should use your real user agent first on any given site, as a courtesy; whether you give up or change to a more "normal" user agent if you get banned is up to you. Oh, and for 3.: if you can, apply some heuristics to your reading of the robots.txt. If it's just "deny everything", then ignore it, but you really don't want to be responsible for crawling all of the GET /delete/:id pages of a badly-designed site… (those should definitely be POST, and authenticated, by the way).
- mpclark 7y agoAlso, if a target site is behind Cloudflare then you probably won’t be able to masquerade as any of the popular bots - they block fake google/yandex bots.
- chatmasta 7y agoI disagree. The risks are similar to those of disclosing a security vulnerability to a company without a bug bounty. You cannot know how litigious or technically illiterate the company will be. What if they decide you're "hacking" them and call the FBI with the helpful information you included in your user agent? Crazier things have happened. Anonymity is part of the right to privacy; IMO, such a right should extend to bots as well. There should be no shame in anonymously accessing a website, whether via automated means or otherwise.
- a1369209993 7y ago> such a right should extend to bots as well No, it very much shouldn't, but (as you probably meant) it should extend to the person (not, eg, company) using a bot, which amounts to the same thing in this case.
- gilad 7y agoAs for delete, use authenticated DELETE, not POST, it’s why its there in the first place
- aspyct 7y agoWell, having a few websites of my own, I really do think that point 1 is the worst. I can't filter bots that disguise as users from my access logs, and they actually hurt my work (i.e. figuring out what people read). Totally agree with the rest though. Maybe adapt the "large delay" of 1 second to the kind of website I'm scraping though. Thanks for your feedback!
- the8472 7y ago> I can't filter bots that disguise as users from my access logs, and they actually hurt my work (i.e. figuring out what people read). If the bots aren't querying from residential IPs you could match their IPs to ASNs and then filter based on that to separate domestic and data center origins.
- aspyct 7y agoHa, that's a good idea! Is there a list somewhere of the cidr blocks that are assigned to residential vs server farms? I mean, how can I tell an IP is residential?
- the8472 7y agoThe other way around may be easier, i.e. excluding known datacenter ranges. There are some commercial databases for that, i'm not sure if there are any free ones. But you can also do this manually by running a whois on an IP and then extracting the ranges from the whois response and caching then. Then you can look at the orgname or something like that. You can also download the whois databases from the RIRs, but they don't contain the information what kind of entities they are. $ dig +short reddit.com 151.101.1.140 $ whois 151.101.1.140 NetRange: 151.101.0.0 - 151.101.255.255 CIDR: 151.101.0.0/16 OrgName: Fastly [...] So if you see a known hoster here then you can exclude it from your statistics.
- capableweb 7y agoWhat I've done in the past is to pull down all the IPs of request I see, filter by unique, do whois for each one of them (you're gonna need to have a backoff/rate limit here as whois services are usually rate limited) and save the organization name, ASN and CIDR blocks, again filter by uniqueness, then create a new list with the organizations of interest and match with the CIDR blocks. Now you have an allow/blocklist you can use.
- erdos4d 7y agoPerhaps I am behind the curve here, but why would sneaker shoe sites get scraped hard?
- abannin 7y agoThere is a very active secondary market for sneakers. If you can buy before supply is exhausted, you can make some decent money.
- deleted 7y ago[deleted]
- codingdave 7y agoYou are correct that I don't like this advice... not because I find it to be wrong, but because you are approaching it solely from a competitive perspective -- "Your competitors don't have ethics, so you shouldn't either." That doesn't help someone who is engaging in research and trying to hold themselves to a higher standard.
- jdc 7y agoIt seems to me that you are conflating ethics with compliance.
- codingdave 7y agoWhen the compliance we are talking about is whether or not to communicate clearly to others who you are, and whether or not to respect their directions, are you so sure they are not correlated?
- hutzlibu 7y ago"or making sure you don't leak out sensitive data" If sensitive data can be scraped, it is not really stored sensitive. So I would not care too much about it and just notify the owner if I notice it.
- HenryBemis 7y agoKeep in mind that if you end up with data that are protected under GDPR, merely having them puts you in a damning position. The intended owner will be fried for not protecting it adequately, but you violate GDPR since "I never agreed to you collecting, processing, etc" the data. And imagine the world of pain if you are caught with children's data.
- wolco 7y agoThe site might be if located in Europe but if it's public you are not bound by those terms as long as you don't share with Europeans.
- fiddlerwoaroof 7y agoIn the US I’d be more concerned about the CCPA.
- lordgrenville 7y agoI'm neither a web dev nor a site owner, but OP literally asked for tips on ethical web scraping, not "what's the most I can get away with".
- trickstra 7y agoThat's because OP is operating under the wrong assumption that sites won't ban an ethical scraper. The reality is that they will, and much faster than an unethical one. They don't care about your science project, they want ad revenue, conversions...
- amelius 7y agoYou forgot to mention to use different IP addresses.