4 ms·
Seems like a lot of interesting sites recognise that I am scraping and block access with a capture. I've no idea how to get around that.
by uhtred 6y ago
Seems like a lot of interesting sites recognise that I am scraping and block access with a capture. I've no idea how to get around that.
- gmac 6y agoHow are you doing it? Automating an actual browser using the developer tools is probably the most under-the-radar way, and also quite nice to work with. See: https://github.com/jawj/web-scraping-for-researchers https://github.com/jawj/web-scraping-for-researchers
- ivoecpereira 6y agoWhat out for "Devtools detect" :) https://github.com/sindresorhus/devtools-detect https://github.com/sindresorhus/devtools-detect
- gmac 6y agoThis is a good point, and I've noticed this phenomenon on some video sites lately. Although the caveats for this particular library[1] imply enough false positives and false negatives that it seems mostly useless. Sites that take this seriously must be doing something smarter. [1] "Doesn't work if DevTools is undocked and will show false positive if you toggle any kind of sidebar."
- umaar 6y agoI've put an example here, for doing it in Node.js: https://github.com/umaar/learn-browser-testing/tree/master/2-scraping/10-captcha https://github.com/umaar/learn-browser-testing/tree/master/2...