4 ms·

I think there might be a market for "human crawlers". Just like people use Mechanical Turk to get humans to beat CAPTCHAs, you could use it to get humans to vis

I think there might be a market for "human crawlers". Just like people use Mechanical Turk to get humans to beat CAPTCHAs, you could use it to get humans to visit a web page for you, and return its HTML source. There are of course residential proxy services (ie HolaVPN), but they're still technically can be detected.

TomMarius 7y ago
Why would you do that when you can automate it?
- AznHisoka 7y ago
  Because of the issues the article described: detection of headless crawlers/bots/etc
  - driverdan 7y ago
    You can automate a regular browser. It doesn't have to be headless.
    - AznHisoka 7y ago
      Unfortunately, there are some sites that can even detect regular automated browser sessions.
  - TomMarius 7y ago
    You can still simulate mouse and keyboard.