4 ms·

You would be surprised to find out that some web servers are capable of detecting browser emulation through curl or Python’s requests lib. Try programmatically

You would be surprised to find out that some web servers are capable of detecting browser emulation through curl or Python’s requests lib. Try programmatically scrolling through Instagram photos. It will work if you use curl, but it will not work using Python’s requests lib. Not sure how they detect it - maybe related to timing of packets.

octoberfranklin 5y ago
The clowns who run the Seattle Times's website block all non-browser user-agent requests to their RSS feeds. Except curl. You can "curl" their RSS feed. You can open it in a browser. Anything else that doesn't lie about its User-Agent will fail. W T F. Somebody please go strangle those people. I had to set my RSS feed reader to impersonate curl's User-Agent.
chockchocschoir 5y ago
The most trivial check a website owner can do is checking the user-agent, which Python requests automatically sets to show its name, unless you configure your own. Trivial way to work around is to set your own user-agent to one that looks like a browser. Specifically regarding Instagram, you can take a look at the implementation of https://github.com/dilame/instagram-private-api https://github.com/dilame/instagram-private-api to understand more workarounds, as Instagram is getting better and better at working against the workarounds.
- jmt_ 5y ago
  In the particular case of Instagram as GP mentions, I'm guessing the devs don't go off of user agent since curl's default user agent is "curl/<installed version num>". Even if they are going off user agent, seems strange to block requests but not curl. GP doesn't mention if they tried to changing the user agent, would be interested to know if Instagram can guess if the client is curl or requests based off other heuristics
  - RF_Savage 5y ago
    I wonder if they have some internal tooling or monitoring that use curl. And thus blocking it would break things.
abdusco 5y ago
This tool claims to replicate Firefox/Chrome's TLS handshake signature: https://github.com/lwthiker/curl-impersonate https://github.com/lwthiker/curl-impersonate I haven't tried it, haven't really come across a service that blocks curl, but I'll be keeping an eye on it in case I need it.

octoberfranklin 5y ago

The clowns who run the Seattle Times's website block all non-browser user-agent requests to their RSS feeds. Except curl. You can "curl" their RSS feed. You can open it in a browser. Anything else that doesn't lie about its User-Agent will fail. W T F. Somebody please go strangle those people. I had to set my RSS feed reader to impersonate curl's User-Agent.

chockchocschoir 5y ago

The most trivial check a website owner can do is checking the user-agent, which Python requests automatically sets to show its name, unless you configure your own. Trivial way to work around is to set your own user-agent to one that looks like a browser. Specifically regarding Instagram, you can take a look at the implementation of https://github.com/dilame/instagram-private-api https://github.com/dilame/instagram-private-api to understand more workarounds, as Instagram is getting better and better at working against the workarounds.

jmt_ 5y ago
In the particular case of Instagram as GP mentions, I'm guessing the devs don't go off of user agent since curl's default user agent is "curl/<installed version num>". Even if they are going off user agent, seems strange to block requests but not curl. GP doesn't mention if they tried to changing the user agent, would be interested to know if Instagram can guess if the client is curl or requests based off other heuristics
- RF_Savage 5y ago
  I wonder if they have some internal tooling or monitoring that use curl. And thus blocking it would break things.

abdusco 5y ago

This tool claims to replicate Firefox/Chrome's TLS handshake signature: https://github.com/lwthiker/curl-impersonate https://github.com/lwthiker/curl-impersonate I haven't tried it, haven't really come across a service that blocks curl, but I'll be keeping an eye on it in case I need it.