3 ms·

Curl’s TLS Fingerprint

27 points by Amorymeltzer 4y ago

LinuxBender 4y ago
The interesting bit I took from the article was the curl-impersonate project. I could see some benefit in developers starting to implement this in other projects such as yt-dlp [2] to reduce the risk of being blocked in the future. They mention Cat and Mouse in the article. That has been the case with yt-dlp and other video download utilities. [1] - https://github.com/lwthiker/curl-impersonate https://github.com/lwthiker/curl-impersonate [2] - https://github.com/yt-dlp/yt-dlp https://github.com/yt-dlp/yt-dlp
mdaniel 4y ago
> As early as only a few years ago I had never heard of any site that blocked clients because of their TLS handshake. Through recent years I have seen it happen and the use of it seems to have increased. Cloudflare has entered the chat Although I can appreciate I may be placing too much emphasis on the word "blocked," since strictly speaking they intercept your request and return text/html with a captcha in it, and I guess maybe if I were to then load that html in a browser, complete the captcha, preserve the cookies, and resubmit the request it may go through but from the "can I quickly just curl this URL that uses cloudflare?" it behaves as if it were blocked I had actually forgotten about curl-impersonate so I'm definitely going to give that a try and see if it helps
zinekeller 4y ago
> I have recently talked with customers and users who have faced servers that refused them access when they connected to sites using curl, but allowed them access to the site when they instead use one of the popular browsers – or if curl was tweaked to look like one of those browsers. It might be a trend in the making. There might be more sites out there now that reject clients that produce the wrong fingerprint then there used to be. As someone working in a bank, yeah we do use this. The draconian application is for our iOS and Android apps - if they don't match Apple's or our fingerprints (respectively, yes you could bake a different cryptographic library in Android) the connection is summarily terminated. For the web app, it is used as a fingerprinting component, but is less draconian in such that it merely checks if the TLS ciphers and the user agent (and some JS magic) is sensible (and the approach is to block known bots outside of robots.txt and the robot-readable login page - to be blunt it includes wget and curl), but I knew that one of our competitors is so much more draconian that it blocks Firefox due to its consistent TLS ciphers (we don't because Firefox is way more accessibility-friendly than Chrome). > Another popular way to work around this kind of blocking is to simply program a browser to do the job. Either a headless browser or with tools like Selenium. Since these methods make the TLS handshake using a browser “engine”, they are unlikely to get blocked by these filters. And yes, we already know about pupeteering (that's why we are now focusing to provide business clients with our own FIDO stick while clients are encourage to activate 2FA, preferably FIDO but at the insistent of the management still includes TOTP and SMS (!) verification).
denysvitali 4y ago
Faced a similar issue while I was trying to programmatically interact with a website that I'll not mention. Surprisingly the website had a protection for which I was able to use their API via the browser, but not from cURL or Go. I was puzzled at first, then I understood what was going on. Basically what's being described in this article. Cloudflare was adding this "protection" against "unauthorized TLS fingerprints", and of course the Go HTTP / TLS client and cURL weren't on that list. To work around the issue (in Go), I simply used a customized version of this: https://github.com/Danny-Dasilva/CycleTLS https://github.com/Danny-Dasilva/CycleTLS
tbyehl 4y ago
I have a script that impersonates curl because the web site being called uses that as a signal to return a small piece of raw data instead of a much larger HTML document.
x3sphere 4y ago
I've encountered some APIs that use this. My question is, why? It doesn't really improve security - someone determined enough can match the fingerprint or send requests using the same library that the originating app uses. It feels like a security through obscurity tactic to me.
- booi 4y ago
  It feels like something similar to running sshd on a different port. Yes, it is security through obscurity and not really any more secure.. but practically speaking it might filter out tons of unsophisticated/spamming scripts that won't be bothered to take the extra step. Depending on how easy and how accurate it is, I can easily see this being included by default in security libraries.

27 points by Amorymeltzer 4y ago

The interesting bit I took from the article was the curl-impersonate project. I could see some benefit in developers starting to implement this in other projects such as yt-dlp [2] to reduce the risk of being blocked in the future. They mention Cat and Mouse in the article. That has been the case with yt-dlp and other video download utilities. [1] - https://github.com/lwthiker/curl-impersonate https://github.com/lwthiker/curl-impersonate [2] - https://github.com/yt-dlp/yt-dlp https://github.com/yt-dlp/yt-dlp

mdaniel 4y ago

> As early as only a few years ago I had never heard of any site that blocked clients because of their TLS handshake. Through recent years I have seen it happen and the use of it seems to have increased. Cloudflare has entered the chat Although I can appreciate I may be placing too much emphasis on the word "blocked," since strictly speaking they intercept your request and return text/html with a captcha in it, and I guess maybe if I were to then load that html in a browser, complete the captcha, preserve the cookies, and resubmit the request it may go through but from the "can I quickly just curl this URL that uses cloudflare?" it behaves as if it were blocked I had actually forgotten about curl-impersonate so I'm definitely going to give that a try and see if it helps

zinekeller 4y ago

> I have recently talked with customers and users who have faced servers that refused them access when they connected to sites using curl, but allowed them access to the site when they instead use one of the popular browsers – or if curl was tweaked to look like one of those browsers. It might be a trend in the making. There might be more sites out there now that reject clients that produce the wrong fingerprint then there used to be. As someone working in a bank, yeah we do use this. The draconian application is for our iOS and Android apps - if they don't match Apple's or our fingerprints (respectively, yes you could bake a different cryptographic library in Android) the connection is summarily terminated. For the web app, it is used as a fingerprinting component, but is less draconian in such that it merely checks if the TLS ciphers and the user agent (and some JS magic) is sensible (and the approach is to block known bots outside of robots.txt and the robot-readable login page - to be blunt it includes wget and curl), but I knew that one of our competitors is so much more draconian that it blocks Firefox due to its consistent TLS ciphers (we don't because Firefox is way more accessibility-friendly than Chrome). > Another popular way to work around this kind of blocking is to simply program a browser to do the job. Either a headless browser or with tools like Selenium. Since these methods make the TLS handshake using a browser “engine”, they are unlikely to get blocked by these filters. And yes, we already know about pupeteering (that's why we are now focusing to provide business clients with our own FIDO stick while clients are encourage to activate 2FA, preferably FIDO but at the insistent of the management still includes TOTP and SMS (!) verification).

denysvitali 4y ago

Faced a similar issue while I was trying to programmatically interact with a website that I'll not mention. Surprisingly the website had a protection for which I was able to use their API via the browser, but not from cURL or Go. I was puzzled at first, then I understood what was going on. Basically what's being described in this article. Cloudflare was adding this "protection" against "unauthorized TLS fingerprints", and of course the Go HTTP / TLS client and cURL weren't on that list. To work around the issue (in Go), I simply used a customized version of this: https://github.com/Danny-Dasilva/CycleTLS https://github.com/Danny-Dasilva/CycleTLS

tbyehl 4y ago

I have a script that impersonates curl because the web site being called uses that as a signal to return a small piece of raw data instead of a much larger HTML document.

x3sphere 4y ago

I've encountered some APIs that use this. My question is, why? It doesn't really improve security - someone determined enough can match the fingerprint or send requests using the same library that the originating app uses. It feels like a security through obscurity tactic to me.

booi 4y ago
It feels like something similar to running sshd on a different port. Yes, it is security through obscurity and not really any more secure.. but practically speaking it might filter out tons of unsophisticated/spamming scripts that won't be bothered to take the extra step. Depending on how easy and how accurate it is, I can easily see this being included by default in security libraries.