4 ms·

Basic question before I can recommend this to my boss: can it scrape G2? (or any other page behind CF)

• tomcam 4y ago
  What are G2 and CF?
  • duckmysick 4y ago
    G2 is a software (as a service) comparison website: https://www.g2.com/ https://www.g2.com/ CF is Cloudflare, which offers an anti-scraping protection for websites (among other things): https://www.cloudflare.com/ https://www.cloudflare.com/
    • tomcam 4y ago
      Thank you!
• buffer_overflow 4y ago
  I could depending on the proxy I got from my provider. I'm currently working on adding the ability to select a higher quality proxy for difficult to scrape websites and to add captcha solvers as well. But if I have to be honest, I can not guarantee it at the present time. This app is the side-project I starter 2 months ago, it's evolving fast but I still need to add some key features for enterprise customers.
• jetter 4y ago
  scrapeninja.net /scrape-js endpoint scrapes company pages of g2 without big troubles (with "us"/"eu" proxy geo in their online sandbox: https://scrapeninja.net/scraper-sandbox https://scrapeninja.net/scraper-sandbox ). They also have /scrape which is much faster because it does not bootstrap real browser, and bypasses CloudFlare TLS fingeprint check: https://pixeljets.com/blog/bypass-cloudflare/ https://pixeljets.com/blog/bypass-cloudflare/