8 ms·
Google2Csv is a simple Google scraper that saves the results on a CSV file
- xur17 6y agoNeat! Have you run into issues with getting blocked by Google / issued captchas?
- damowangcy 6y agoDid something similar using requests, there's a rated limit around 80+ request (not sure how long it lasted though) no matter how I delay each request. My work around is to run in on the cloud with multiple IPs, so I just switch between. Mine's a one time thing so this is workable. If you're querying for news, Reddit might be a wise choice, they have quite a bit of free APIs afaik.
- jll29 6y agoFurther to the practical/technical issue of being blocked, there is a legal issue: this way of coding it (= not using an API key) violates Google's Terms and Conditions.
- xur17 6y agoAre Google's terms legally binding if you are just searching? I don't recall ever agreeing to them in that case.
- ggggtez 6y ago> is the ToS legally binding if you are just searching Based on the LinkedIn case my guess is no, because it could be considered "generally public". But that ruling is highly suspect and likely to go to the supreme court, imo.
- dvfjsdhgfv 6y agoThe onus is on them to prove that I agreed to some ToS.
- deleted 6y ago[deleted]
- 4636760295 6y agoIf it's publicly accessible, it's pretty much a public space, thus Google ToS is BS. It's the equivalent of walking by the Google office and taking a photo of the building, IMO. This wouldn't hold up in court.
- tvjunky 6y agoAdditionally in a court decision from last year, accessing public data is legal. https://parsers.me/us-court-fully-legalized-website-scraping-and-technically-prohibited-it/ https://parsers.me/us-court-fully-legalized-website-scraping...
- ErikAugust 6y agoThat's pretty rich coming from a company that scrapes websites for a living.
- stinos 6y agoNice, until the CSV part really. I mean you have a nice DataFrame there but then it gets serialized using probably the worst format out there for totally variable data like what you scrape from a site. Unless the file uses the actual ASCII record separator you'll end up with a CSV file which can only be read by a handful of software, after telling it explicitly what the separator and quoting rules are. And even then it's hit or miss. And likely it does not use the RS because even though the chances it's unambiguous greatly increase and the RS was actually meant for that, software doesn't typically use it because when CSV was invented it's existence was unknown or ignored beacue it's not human readable (I guess, don't really know actually) and so the sad story began. As you can see: I'm not a fan of CSV :) Just today I again had to waste time because at one point in the development of this otherwise fine piece of software I'm working on - even though I knew I'd regret it - I allowed it to export CSV files. Customer moved software to another machine, forgot that they once told the CSV exporter part to use the system settings, and now has CSV files with a comma separator (you know, the C in CSV). Oh the irony, that's not what they wanted.
- hetspookjee 6y agoWhile I couldnt agree more with your disliking of CSV, I have yet to find another more generic human-readable table file. Of all the pandas dumping methods there is no more widely available format. I find the times a CSV is utterly broken and I have to manually write the parsing logic for it quite rare actually, but then again it varies per field. What format would you rather prefer?
- rabidrat 6y agoJSONL (jsonlines.org)
- micael_dias 6y agohttp://jsonlines.org http://jsonlines.org
- brokensegue 6y agopandas can directly write to parquet
- elektor 6y agoIf you're looking to do any serious search engine scraping, you'll find yourself needing to use proxies. For my thesis which required millions of datapoints, I used this tool: http://www.scrapebox.com/ http://www.scrapebox.com/
- deleted 6y ago[deleted]
- mateus1 6y agoThis seems like what I was lookingf. What kind of proxy were you using?
- nurettin 6y agoThis is a great way of getting your ip permanently blacklisted and swamped by captchas.
- dvfjsdhgfv 6y agoIt's not going to be permanently blocked, but the low rate limits are annoying, so for any seriosu use a set of proxies is must.
- nurettin 6y agoI don't know about being blocked. I have a vps where I used a similar naive script and it got blacklisted by google. I sometimes tunnel through it. It has been getting captchas from google the past 7 years. It is pretty permanent.
- lbj 6y agoI made a wonderful little Google scraper in Clojure once. I was surprised to see I got IP blocked after only 20 or so searches.
- tantalor 6y agoA rite of passage.
- lazyjones 6y ago> Scraping google search results is illegal. IANAL, can someone please elaborate? This sounds wrong in several ways, one of which being that Google results are almost exclusively scraped from somewhere else already.
- ttul 6y agoIt is decidedly not illegal to scrape web sites. This was recently established by a court [1]. [1] https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
- TechBro8615 6y agoTangentially related, this would be a fun target for the anti-trust lawsuits. Force google to open its search results as an API service. Allow them to charge for it, sure. But make it an open API. (Isn’t it weird they don’t offer this already? Makes you think.)
- Minor49er 6y agoThis is a cool project. I've used Scraperr[0] for years, but it's always great to have alternatives. [0] http://scraperr.com/ http://scraperr.com/