7 ms·
Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake
- wswope 5y agoVery cool! Thanks for sharing - it’s always nice to learn about fingerprinting tricks and workarounds, from both a privacy and a “don’t unintentionally look like a bot” perspective. What inspired the project?
- oefrha 5y agoMotivation is in the blog post: https://lwthiker.com/reversing/2022/02/17/curl-impersonate-firefox.html https://lwthiker.com/reversing/2022/02/17/curl-impersonate-f...
- jart 5y agoGood blog post. Stuff like this makes me wonder if by 2030 (1) the internet will mostly consist of machine generated content; (2) machines written by normal people in Python won't be authorized to access the machine-generated content anymore due to Protectify; (3) most client traffic will originate from Protectify's network, so people like bloggers won't have any visibility into whether their readers are humans or machines; (4) video compression algorithms will become indistinguishable from deepfakes; and (5) airborne pathogens will make alternatives to the above impractical.
- deleted 5y ago[deleted]
- gruez 5y ago>impersonate Firefox 95 you should really be impersonating an ESR version (eg. 91). Versions from the release channel is updated every month or so, and everyone has autoupdate enabled. Therefore unless you keep it up to date, your fingerprint is going to stick out like a sore thumb in a few months. On the other hand, ESR sticks to one version and shouldn't change significantly during its one year lifetime. It's still going to stick out to some extent (most people don't use ESR), but at least you have some enterprises who use ESR to blend into.
- kalleboo 5y agoThey should really be impersonating Chrome. If this takes off, Firefox has such a small user share that I could see sites just banning Firefox altogether, like they do with Tor
- jve 5y agoI suspect Tor is being banned not because of a small user share. Perhaps you may get broken sites with Firefox, because no-one cared. But banning? Seems like a stretch.
- kalleboo 5y agoTor is banned (or rather, tar-pitted in endless CAPTCHAs) because the amount of legit users is massively dwarfed by the abuse If everyone running scrapers and attack probes start showing up as Firefox, then they'll end up in the same situation.
- oefrha 5y agoIf there are a lot of abuse masquerading as Firefox, outstripping legit users, they can totally throw up a CAPTCHA for Firefox but not for Chrome. An outright ban isn’t the only annoying outcome.
- LanternLight83 5y agoI think ESR is the way to go too, but either way, I wonder if some tests can be written to confirm the coverage/similarity of the requests? It would entail automating a both Firefox session and the recording of network traffic, and feels like it might end up as bikeshedding.
- lwthiker 5y agoThanks for the suggestion, I had no idea ESR was a thing. I've just added support for Firefox ESR 91 (it was pretty similar and required adding one cipher to the cipher list and changing the user agent).
- 0xbkt 5y agoThis might also be an interesting read for those curious about TLS fingerprinting: https://news.ycombinator.com/item?id=29472624 https://news.ycombinator.com/item?id=29472624
- vincent-toups 5y agoCool, can't wait for anti-bot protection to start rejecting me because I use firefox.
- deleted 5y ago[deleted]
- npteljes 5y agoOnly a matter of time I'm afraid :( Firefox usage share is already low enough for many sites to make pages for Chrome and maybe Safari only.
- oefrha 5y agoVery cool. I would have used Puppeteer/Playwright in a similar scenario, but thanks for sharing the bot detection trick they employ.
- javajosh 5y agoHandy. Is the TCP handshake, or other details about socket behavior, ever get used for assessing the remote process, and in turn libraries written to mimic known patterns?
- tenebrisalietum 5y agoYes. https://nmap.org/book/osdetect-methods.html https://nmap.org/book/osdetect-methods.html
- tootahe45 5y agoWould be cool if there was something like this for Python. Last time i tried to scrape something interesting i found that one of Cloudflare's enterprise options was easily blocking all of the main http libraries due to the identifiable TLS handshake.
- tyingq 5y agoI think most of the scraping libraries have stagnated since it's hard to scrape without a headless browser these days...too many sites with client-side rendered content.
- ricardo81 5y agoAre you sure they blocked you because of the handshake? Always thought it was the myriad of cookies and expiry time of said cookies that tend to make non-browser clients more obvious to CF.
- tootahe45 5y agoThe site wasn't using it to block me, just to prompt a captcha, without doing so to 'real' browsers. The HTTP requests were exact copies of browser requests (in terms of how the server would've seen them), so it was something below HTTP. I ended up finding a lot of info about Cloudflare and the TLS stuff on StackOverflow, with others having similar issues. Someone even made an API to do the TLS stuff as a service, but was too expensive for me. https://pixeljets.com/blog/scrape-ninja-bypassing-cloudflare-403-code-1020-errors/ https://pixeljets.com/blog/scrape-ninja-bypassing-cloudflare...
- ricardo81 5y agoThanks for the response, never came across the particular behaviour. fwiw I think when it comes to the 'copy as curl', the HTTP header ordering may be different and it's worth loading up a page twice as some of the cookies are replaced. I've used puppeteer as the article talks about. Manages the cookies better. Managed to do continuous requests without getting further CF blocks as opposed to a couple of hundred with cURL (due to cookies different from what CF expect over a time) IIRC CF does have a sliding scale of how protected you want a site to be, so perhaps the TLS stuff belongs further up the scale.
- userbinator 5y agoI run a MITM proxy for adblocking/general filtering and within the past little while I've noticed CloudFlare and other "bot protection" tends to get me blocked out of increasingly more sites I come across in search results, so this will be very useful for fixing that. However, I should caution that in this era of companies being particularly user-hostile and authoritarian, especially Big Tech, I would be more careful with sharing stuff like this. Being forced to run JS is bad enough; profiling users based on other traits, and essentially determining if they are using "approved" software, is a dystopia we should fight strongly against. Stallman's Right To Read comes to mind as a very relevant warning story.
- oh_sigh 5y agoMaybe I'm just a techno-optimist, but I suspect big tech companies don't give a hoot about you running "unapproved" software, but rather care about their services being abused and "unapproved" software is just a useful signal that fails on a tiny percentage of total legit users.
- llampx 5y agoYou are a lot more charitable than I am. I believe the big tech companies use dark patterns to get us to sign up, improve their metrics and hoover up our data.
- saurik 5y agoThis is a distinction without a difference.
- bayindirh 5y agoJust trying to keep services operational is a fine goal to pursue as an operator, but forcing users to small inbound funnels for the service is detrimental too. There needs to be better research to be done to allow simpler ways of operation to continue working. A browser is becoming a universal agent by itself, but many people (maybe increasingly) use terminal to access to the resources, and stonewalling these paths are never OK in my book.
- 5y ago
- jandrese 5y agoGiven the relative market shares it might make more sense to impersonate Chrome.
- 7v3x3n3sem9vv 5y agoand make it seem like Firefox has less market share? sounds like a good way to kill Firefox even faster, my 2 cents.
- flawi 5y agoCounter argument is service providers just choosing to block anything that looks like Firefox since the market share is so small and it's being used to circumvent their precious protections.
- Handytinge 5y agoWhilst it's not as big as it once was, the idea of a service provider blocking all Firefox user agents is still ludicrous to the point that I can't believe you're not trolling here.
- lwthiker 5y agoI will try to impersonate Chrome next, However, I suspect this is going to be more challenging. Chrome uses BoringSSL, which curl does not support. So it means either enforcing curl to compile with BoringSSL or modifying NSS to look like BoringSSL.
- pabs3 5y agoDo you plan on getting this merged back into curl with an option to enable it? I can see that being useful for some people.
- lwthiker 5y agoI hope to do so in the future, for now the implementation is extremely hacky so I doubt it can get accepted into curl.
- bburky 5y agoThere was a conversation on their mailing list contemplating dropping NSS support. https://curl.se/mail/lib-2022-01/0120.html https://curl.se/mail/lib-2022-01/0120.html If you have a use case for NSS in curl, you may want to speak up. Perhaps "I want curl to look exactly like a browser" is a significant use case?
- pabs3 5y agoAgreed, it is very important to bring this up on the mailing list. It might also be plausible to make curl look like Chrome if curl had BoringSSL support.
- thompson1 5y ago[flagged]
- 1vuio0pswjnm7 5y ago"Some web services therefore use the TLS handshake to fingerprint which HTTP client is accessing them. Notably, some bot protection platforms use this to identify curl and block it." As a user of non-browser clients (not curl though) I have not run into this in the wild.^1 Anyone have an example of a site that blocks non-browser clients based on TLS fingerprint. 1. As far as I know. The only site I know of today that is blocking non-browser clients appears to be www.startpage.com. Perhaps this is the heuristic they are using. More likely it is something simpler I have not figured out yet.
- deleted 5y ago[deleted]
- octoberfranklin 5y agoThis is cool, but is it really needed that often? There are some industries (virtually all of Wall Street, for example, and certain parts of government) where the company needs to surveil 100% of what their employees do on the web from inside the office. These companies have been running MITM proxies for decades. Wouldn't any website that rejects a non-browsery TLS client be blocking out these people as well?
- lwthiker 5y agoThey don't block you completely, just present you with a JS challenge that delays your access to the site. A browser, even if behind a MITM proxy, would be able to solve this challenge.
- sylware 5y agoCurrently, I cannot think about anything else other than "noscript/basic (x)html" /IRC to get us out of this, at least for sites where such protocols are "good enough" to provide their services to users over internet. But how? Enlighten the "javascript web" brain-washed devs to make them realize how toxic what they do is? regulations (at least for critical sites)? And how to deal with the other sites: those which devs are scammers and perfectly aware of how toxic they are and keep doing it. In my own country, for critical sites, I will probably have go to court since 'noscript/basic (x)html" interop was broken in the last few years.
- kinderjaje 5y agoGreat work mate, one of my team-mates showed me this library and we might use it in near future.