7 ms·

Slightly OT: Does anyone know how RankScanner works? My guess was this but it still seems to work. Their site seems to suggest they have loads of servers, but s

Slightly OT: Does anyone know how RankScanner works? My guess was this but it still seems to work. Their site seems to suggest they have loads of servers, but seems unlikely, I'd have thought Google would have captached them to hell by now. Very good product. I'm curious how it works though.

woodruffw 11y ago
I'm not acquainted with RankScanner, but my educated guess is that they use one of Google's (many) authenticated APIs[1] for search/content categorization/ranking analysis. [1]: https://developers.google.com/custom-search/ https://developers.google.com/custom-search/
- martinald 11y ago
  Thanks. I was under the search impression that this was only for building your own internal search. do you know how much the pricing is?
  - woodruffw 11y ago
    There are some prices listed here: https://developers.google.com/custom-search/json-api/v1/overview#pricing https://developers.google.com/custom-search/json-api/v1/over... These seem to be for products in a slightly different domain than plain old search, though.
Jake232 11y ago
Having built a ranktracker before, and still being good friends with people who run one of the biggest rank trackers on the market, using the google API's simply isn't an option. Simply put, the API never returned the results in the exact same order as the actual search results did. The best / most reliable rank trackers did (and still do) simply use proxies to get around the google captchas. I've scraped millions of pages from google over the years, and with enough proxies, the correct proxy delays/timeouts, and other little tricks you pick up along the way; you can actually scrape google pretty easily. This is especially true with the dropping cost of proxies. I'm obviously an exception considering I run several scraping SaaS and have generally specialized in it for years, but I can he hitting x.com with 30,000 different IP's within the hour.
- hn_username 11y ago
  Any additional tricks you want to share and/or cheap proxy services you'd recommend? Do you make use of selenium and/or phantomjs?
- coderdude 11y ago
  Haven't all the web search APIs been gimped? In a past life this interested me greatly and I turned to both Google's API and to Yahoo's BOSS and they both basically sucked. You could tell you were not getting the type of results a proper search would return.
- visarga 11y ago
  > with enough proxies, the correct proxy delays/timeouts, and other little tricks you pick up along the way; you can actually scrape google pretty easily Try putting a site: in the search and it becomes suspicious fast.
  - Jake232 11y ago
    You're correct. However for Rank Tracking these types of queries aren't needed.
- agopaul 11y ago
  That is probably why some rank trackers give different results. Some probably use the API and others scrape Google's result page.
Ixiaus 11y ago
I also built a rank tracker (WhooshTraffic at the time - no longer around) and we scraped the Search Result Pages as the user would see them, spoofed the session cookie and used human captcha solvers. We would then cache the cookies generated from the captcha solve. It was highly effective and very scalable because you can stimulate the captcha pages easily and out-of-band generate a very large cookie pool by captcha solving. Then when you go to actually scrape google you can balance your pool of IPs and cookies, chilling them before they trigger another captcha, to handle spiking demand. Contant, sustained demand was very easy to plan for. Anyway, we stopped using the API after the third day and realized it was not only not accurate and you couldn't turn many knobs (want to scrape results as they look for a different geographical region?)
- dchuk 11y ago
  This is super interesting to me, is there anything else you can share about how you approached this? In my scraping Google experience I have found roughly the same thing where once you've passed the captcha test, you can scrape a lot more. Were you scraping with real browsers or something like Mechanize/Curl? Rate limiting at all? Proxies or real servers?
  - Ixiaus 11y ago
    We had a pool of IPs we were leasing on our own, proxy services get abused and are poisoned. We didn't rate limit, we would just increase the size of the cookie pool if a captcha was hit, which was rare because we would scrape n-pages till a threshold was met to prevent that session from being captcha'ed so we wouldn't have to captcha solve it. We had two pools, the primary pool and the "chilling" pool, cookies near their captcha life would cool off for a few hours before returning to the active pool which behaves just like any other resource pool, every page scraped would "borrow" a cookie out of the pool, customize the encrypted location key, and make the request with a common user agent string. Scaling it was difficult but once we had it figured out, Erlang was invaluable to us and our dependence on IPs dropped once we figured out the cookie methodology. Solving captchas is cheaper than renting IPs.
    - 11y ago

woodruffw 11y ago

I'm not acquainted with RankScanner, but my educated guess is that they use one of Google's (many) authenticated APIs[1] for search/content categorization/ranking analysis. [1]: https://developers.google.com/custom-search/ https://developers.google.com/custom-search/

martinald 11y ago
Thanks. I was under the search impression that this was only for building your own internal search. do you know how much the pricing is?
- woodruffw 11y ago
  There are some prices listed here: https://developers.google.com/custom-search/json-api/v1/overview#pricing https://developers.google.com/custom-search/json-api/v1/over... These seem to be for products in a slightly different domain than plain old search, though.

Jake232 11y ago

Having built a ranktracker before, and still being good friends with people who run one of the biggest rank trackers on the market, using the google API's simply isn't an option. Simply put, the API never returned the results in the exact same order as the actual search results did. The best / most reliable rank trackers did (and still do) simply use proxies to get around the google captchas. I've scraped millions of pages from google over the years, and with enough proxies, the correct proxy delays/timeouts, and other little tricks you pick up along the way; you can actually scrape google pretty easily. This is especially true with the dropping cost of proxies. I'm obviously an exception considering I run several scraping SaaS and have generally specialized in it for years, but I can he hitting x.com with 30,000 different IP's within the hour.

hn_username 11y ago
Any additional tricks you want to share and/or cheap proxy services you'd recommend? Do you make use of selenium and/or phantomjs?
coderdude 11y ago
Haven't all the web search APIs been gimped? In a past life this interested me greatly and I turned to both Google's API and to Yahoo's BOSS and they both basically sucked. You could tell you were not getting the type of results a proper search would return.
visarga 11y ago
> with enough proxies, the correct proxy delays/timeouts, and other little tricks you pick up along the way; you can actually scrape google pretty easily Try putting a site: in the search and it becomes suspicious fast.
- Jake232 11y ago
  You're correct. However for Rank Tracking these types of queries aren't needed.
agopaul 11y ago
That is probably why some rank trackers give different results. Some probably use the API and others scrape Google's result page.

Ixiaus 11y ago

I also built a rank tracker (WhooshTraffic at the time - no longer around) and we scraped the Search Result Pages as the user would see them, spoofed the session cookie and used human captcha solvers. We would then cache the cookies generated from the captcha solve. It was highly effective and very scalable because you can stimulate the captcha pages easily and out-of-band generate a very large cookie pool by captcha solving. Then when you go to actually scrape google you can balance your pool of IPs and cookies, chilling them before they trigger another captcha, to handle spiking demand. Contant, sustained demand was very easy to plan for. Anyway, we stopped using the API after the third day and realized it was not only not accurate and you couldn't turn many knobs (want to scrape results as they look for a different geographical region?)

dchuk 11y ago
This is super interesting to me, is there anything else you can share about how you approached this? In my scraping Google experience I have found roughly the same thing where once you've passed the captcha test, you can scrape a lot more. Were you scraping with real browsers or something like Mechanize/Curl? Rate limiting at all? Proxies or real servers?
- Ixiaus 11y ago
  We had a pool of IPs we were leasing on our own, proxy services get abused and are poisoned. We didn't rate limit, we would just increase the size of the cookie pool if a captcha was hit, which was rare because we would scrape n-pages till a threshold was met to prevent that session from being captcha'ed so we wouldn't have to captcha solve it. We had two pools, the primary pool and the "chilling" pool, cookies near their captcha life would cool off for a few hours before returning to the active pool which behaves just like any other resource pool, every page scraped would "borrow" a cookie out of the pool, customize the encrypted location key, and make the request with a common user agent string. Scaling it was difficult but once we had it figured out, Erlang was invaluable to us and our dependence on IPs dropped once we figured out the cookie methodology. Solving captchas is cheaper than renting IPs.
  - 11y ago