yeah so I have a user-agent generator that uses a random (valid) agent for each request. that, combined with cookies enabled, still didn't seem enough to get a

yeah so I have a user-agent generator that uses a random (valid) agent for each request. that, combined with cookies enabled, still didn't seem enough to get around the anti-robot barrier 100% of the time. 20% of the time, I get stopped. My guess is that they must do some kind of pattern matching on their end that kicks for some sets of requests but not others? But maybe it's not that sophisticated and I'm just missing something else in my headers