4 ms·

I thought scraping is kind of dead given all the CAPTCHAs and auth walls everywhere. The article does mention proxies and rate limiting, but could anyone with (

I thought scraping is kind of dead given all the CAPTCHAs and auth walls everywhere. The article does mention proxies and rate limiting, but could anyone with (recent) practical experience elaborate on dealing with such challenges?

caesil 3y ago
Not only is scraping not dead but it has won the arms race. There are ways around every defense, and this will only accelerate as AI advances. The CAPTCHAs and walls are more of a desperate, doomed retreat.
- annowiki 3y ago
  How do you get around 403/401's from WSJ/Reuters/Axios? Because I've tried user agent manipulation and it seems like I'd have to use selenium and headless to deal with them.
  - eddd-ddde 3y ago
    Sometimes you also need "Accept: html" I have noticed.
  - jonatron 3y ago
    If curl-impersonate works, it's probably TLS fingerprinting.
- accidbuddy 3y ago
  Some months ago, I had problems with captcha. I tried to write an application to access many drugstores and compare the price, but captcha with login system fail the mission. Do you have any piece of advice for me?
  - nico 3y ago
    Not sure about the currently available tools, given the break-neck speed of AI progress, but a couple of years ago I built a scraper that used a captcha-solving service, they sell something like 1000 solutions for $10, it was super cheap. The process was a bit slow because they were using humans to solve the captchas, but it worked really well
  - jawerty 3y ago
    Try 2captcha https://2captcha.com/2captcha-api https://2captcha.com/2captcha-api
nlunbeck 3y ago
I've noticed many smaller and medium sites only use client-side CAPTCHAs/paywalls
leumon 3y ago
If you have a decent gpu (16gb+ vram) and are using Linux, then this tool I wrote some days ago might do the trick. (at least for googles recaptcha). Also, for now, you have to call the main.py every time you see a captcha on a site and you need the gui since I am only using vision via Screenshots, no HTML or similar. (Sorry that it's not yet that well optimized. I am currently very busy with lots of other things, but next week I should have time to improve this further. But it should still work for basic scraping.) https://github.com/notune/captcha-solver/ https://github.com/notune/captcha-solver/
pocket_cheese 3y ago
A few different techniques - 1. use mobile phone proxies. Because of how mobile phone networks do NAT, basically it means that thousands of people share IPs and are much less like to get blocked. 2. Reverse engineer APIs if the data you want is returned in an ajax call. 3. Use a captcha solving service to defeat captchas. There's many and they are cheap. 4. Use an actual phone or get really good at convincing the server you are a mobile phone. 5. Buy 1000s of fake emails to simulate multiple accounts. 6. Experiment. Experiment. Experiment. Get some burner accounts. Figure out if they have request per min/hour/day throttling. See what behavior triggers a cloudflare captchas. Check if different variables such as email domain, useragent, voip vs non-voip sms based 2fa. your goal is to simulate a human. So if you sequentially enumerate through every document - that might be what get's you flagged. Best of luck and happy scraping!

caesil 3y ago

Not only is scraping not dead but it has won the arms race. There are ways around every defense, and this will only accelerate as AI advances. The CAPTCHAs and walls are more of a desperate, doomed retreat.

annowiki 3y ago
How do you get around 403/401's from WSJ/Reuters/Axios? Because I've tried user agent manipulation and it seems like I'd have to use selenium and headless to deal with them.
- eddd-ddde 3y ago
  Sometimes you also need "Accept: html" I have noticed.
- jonatron 3y ago
  If curl-impersonate works, it's probably TLS fingerprinting.
accidbuddy 3y ago
Some months ago, I had problems with captcha. I tried to write an application to access many drugstores and compare the price, but captcha with login system fail the mission. Do you have any piece of advice for me?
- nico 3y ago
  Not sure about the currently available tools, given the break-neck speed of AI progress, but a couple of years ago I built a scraper that used a captcha-solving service, they sell something like 1000 solutions for $10, it was super cheap. The process was a bit slow because they were using humans to solve the captchas, but it worked really well
- jawerty 3y ago
  Try 2captcha https://2captcha.com/2captcha-api https://2captcha.com/2captcha-api

nlunbeck 3y ago

I've noticed many smaller and medium sites only use client-side CAPTCHAs/paywalls

leumon 3y ago

If you have a decent gpu (16gb+ vram) and are using Linux, then this tool I wrote some days ago might do the trick. (at least for googles recaptcha). Also, for now, you have to call the main.py every time you see a captcha on a site and you need the gui since I am only using vision via Screenshots, no HTML or similar. (Sorry that it's not yet that well optimized. I am currently very busy with lots of other things, but next week I should have time to improve this further. But it should still work for basic scraping.) https://github.com/notune/captcha-solver/ https://github.com/notune/captcha-solver/

pocket_cheese 3y ago

A few different techniques - 1. use mobile phone proxies. Because of how mobile phone networks do NAT, basically it means that thousands of people share IPs and are much less like to get blocked. 2. Reverse engineer APIs if the data you want is returned in an ajax call. 3. Use a captcha solving service to defeat captchas. There's many and they are cheap. 4. Use an actual phone or get really good at convincing the server you are a mobile phone. 5. Buy 1000s of fake emails to simulate multiple accounts. 6. Experiment. Experiment. Experiment. Get some burner accounts. Figure out if they have request per min/hour/day throttling. See what behavior triggers a cloudflare captchas. Check if different variables such as email domain, useragent, voip vs non-voip sms based 2fa. your goal is to simulate a human. So if you sequentially enumerate through every document - that might be what get's you flagged. Best of luck and happy scraping!