3 ms·
Not only is scraping not dead but it has won the arms race. There are ways around every defense, and this will only accelerate as AI advances. The CAPTCHAs and
by caesil 3y ago
Not only is scraping not dead but it has won the arms race. There are ways around every defense, and this will only accelerate as AI advances.
The CAPTCHAs and walls are more of a desperate, doomed retreat.
- annowiki 3y agoHow do you get around 403/401's from WSJ/Reuters/Axios? Because I've tried user agent manipulation and it seems like I'd have to use selenium and headless to deal with them.
- accidbuddy 3y agoSome months ago, I had problems with captcha. I tried to write an application to access many drugstores and compare the price, but captcha with login system fail the mission. Do you have any piece of advice for me?
- nico 3y agoNot sure about the currently available tools, given the break-neck speed of AI progress, but a couple of years ago I built a scraper that used a captcha-solving service, they sell something like 1000 solutions for $10, it was super cheap. The process was a bit slow because they were using humans to solve the captchas, but it worked really well
- jawerty 3y agoTry 2captcha https://2captcha.com/2captcha-api https://2captcha.com/2captcha-api