4 ms·

Try faking your User-Agent header.

• banana_giraffe 5y ago
  The article touches on that a bit in the "Avoid being blocked" section. Sometimes user-agent isn't enough, I've run into other headers that can trigger a block or a change in behavior. The last one I ran into was gating on accept-language to not control the language, but serve up a honey-pot type page for automated crawlers.
  • guskel 5y ago
    Haha, that's devious but ultimately futile against a targeted scraper.