3 ms·

“Of course clicks and keyboard events needs to emulate the timings and movements of a human, but that's another problem entirely.” I don’t think that’s a diffe

“Of course clicks and keyboard events needs to emulate the timings and movements of a human, but that's another problem entirely.” I don’t think that’s a different problem. As you know, real browsers are operated by humans, and humans have limited processing capacity and somewhat predictable navigation patterns. To be undetectable, you have to mimic both (almost) perfectly. So, you have to both limit the speed at which a fake user requests pages _and_ request then in a believable sequence. The speed issue can be circumvented by using multiple machines to make requests from, but the second is the real problem. A collection of fake users would have to either stay under the radar by not making many requests or follow human patterns. Combined, I would think it’s not possible to efficiently scrape lots of pages in a way that cannot be detected.

throwawayadvsec 3y ago
From my experience, it's a completely different problem. For 99% of websites there is no serious behavioral analysis, and when there it's easily bypassable. It's as simple as: wait some random time between 500 and 3000ms before clicking and always make the clicks randomly slightly off from the center of the element wait for some random time between 50 and 200ms between keystrokes use ghost-cursor to make realistic movements using bezier curves Most of the time you get spotted because of some obscure value in some random part of the browser or request headers isn't what it's supposed to be, not because your actions aren't realistic
- OkayPhysicist 3y ago
  Or, save yourself the hassle of pretending to be using a mouse, by pretending to be using a touchscreen.
  - throwawayadvsec 3y ago
    well it's also pretty hard to simulate a phone, even a lot harder than a browser imo for example I'm pretty sure the GPU/canvas fingerprint can't be faked

throwawayadvsec 3y ago

From my experience, it's a completely different problem. For 99% of websites there is no serious behavioral analysis, and when there it's easily bypassable. It's as simple as: wait some random time between 500 and 3000ms before clicking and always make the clicks randomly slightly off from the center of the element wait for some random time between 50 and 200ms between keystrokes use ghost-cursor to make realistic movements using bezier curves Most of the time you get spotted because of some obscure value in some random part of the browser or request headers isn't what it's supposed to be, not because your actions aren't realistic

OkayPhysicist 3y ago
Or, save yourself the hassle of pretending to be using a mouse, by pretending to be using a touchscreen.
- throwawayadvsec 3y ago
  well it's also pretty hard to simulate a phone, even a lot harder than a browser imo for example I'm pretty sure the GPU/canvas fingerprint can't be faked