4 ms·
When you said you were faking a proper agent with `requests`, do you mean you were setting the headers to look like a browser, as in here?: https://stackoverflo
by cottager 6y ago
When you said you were faking a proper agent with `requests`, do you mean you were setting the headers to look like a browser, as in here?: https://stackoverflow.com/questions/27652543/how-to-use-python-requests-to-fake-a-browser-visit https://stackoverflow.com/questions/27652543/how-to-use-pyth...
That was going to be my suggestion for how to get around the anti-robot responses.
- catwind7 6y agoyeah so I have a user-agent generator that uses a random (valid) agent for each request. that, combined with cookies enabled, still didn't seem enough to get around the anti-robot barrier 100% of the time. 20% of the time, I get stopped. My guess is that they must do some kind of pattern matching on their end that kicks for some sets of requests but not others? But maybe it's not that sophisticated and I'm just missing something else in my headers
- x3blah 6y agoNo agent at all is required. I got past the anti-robot response using no user-agent header and a simple delay.