4 ms·

And google marks your ip as a bot. I did this three years ago in one of my DO proxies. Google still asks me to confirm a captcha when I do a search.

bake 9y ago
Indeed. If you're looking for programmatic web search, I'd suggest you go the Bing API route.
- charred_toast 9y ago
  I commented earlier on this post about it being impossible for a bot to do search on Google. Turns out I turned to the same resource you're recommending. I'd also recommend sending a python bot to your favorite news sites once a day for updates instead. Same deal, Beautiful Soup. On the other hand, Google has Custom Search, which is $100 a year for 20k queries. I use that as well.
  - surfingdino 9y ago
    If you want news, use RSS. Almost every news site is using it.
  - bottled_poe 9y ago
    Google CSE seems like the most fitting solution, which appears to do pretty much the same thing as the article, but without all the hackery nonsense: Step 1 - Setup a CSE to search entire web: https://support.google.com/customsearch/answer/2631040?hl=en https://support.google.com/customsearch/answer/2631040?hl=en Step 2 - Use the CSE API: https://developers.google.com/custom-search/json-api/v1/overview https://developers.google.com/custom-search/json-api/v1/over...
  - defenestration 9y ago
    It looks like Google is shutting down the paid version of Custom Search Engine: "On April 1st, 2017, Google will discontinue the sales of Google Site Search, the paid version of Custom Search Engine. All new purchases and renewals must take place before this date. This product will be completely shut down by April 1st, 2018." Source: https://cse.google.com/cse/ https://cse.google.com/cse/
slackingoff2017 9y ago
My company is behind a giant honking NAT and we get blocked all the time. Unusual volume for our "residential" IP block I suppose. Google seems to do some averaging across CIDR blocks to detect likely bots. Any chance somebody is abusing your proxy IP? Is your proxy part of an IP allocation thats underutilized? An easy way to get around it is to login to a Google account. Google allows you to use even the most trashed and abused TOR exit nodes as long as you're logged in.
gh1 9y ago
This depends on the rate at which you are making requests. If the rate is low, then Google doesn't bother. I have successfully scraped Google by waiting longer between requests.

bake 9y ago

Indeed. If you're looking for programmatic web search, I'd suggest you go the Bing API route.

charred_toast 9y ago
I commented earlier on this post about it being impossible for a bot to do search on Google. Turns out I turned to the same resource you're recommending. I'd also recommend sending a python bot to your favorite news sites once a day for updates instead. Same deal, Beautiful Soup. On the other hand, Google has Custom Search, which is $100 a year for 20k queries. I use that as well.
- surfingdino 9y ago
  If you want news, use RSS. Almost every news site is using it.
- bottled_poe 9y ago
  Google CSE seems like the most fitting solution, which appears to do pretty much the same thing as the article, but without all the hackery nonsense: Step 1 - Setup a CSE to search entire web: https://support.google.com/customsearch/answer/2631040?hl=en https://support.google.com/customsearch/answer/2631040?hl=en Step 2 - Use the CSE API: https://developers.google.com/custom-search/json-api/v1/overview https://developers.google.com/custom-search/json-api/v1/over...
- defenestration 9y ago
  It looks like Google is shutting down the paid version of Custom Search Engine: "On April 1st, 2017, Google will discontinue the sales of Google Site Search, the paid version of Custom Search Engine. All new purchases and renewals must take place before this date. This product will be completely shut down by April 1st, 2018." Source: https://cse.google.com/cse/ https://cse.google.com/cse/

slackingoff2017 9y ago

My company is behind a giant honking NAT and we get blocked all the time. Unusual volume for our "residential" IP block I suppose. Google seems to do some averaging across CIDR blocks to detect likely bots. Any chance somebody is abusing your proxy IP? Is your proxy part of an IP allocation thats underutilized? An easy way to get around it is to login to a Google account. Google allows you to use even the most trashed and abused TOR exit nodes as long as you're logged in.

gh1 9y ago

This depends on the rate at which you are making requests. If the rate is low, then Google doesn't bother. I have successfully scraped Google by waiting longer between requests.