9 ms·
Curl-impersonate: Special build of curl that can impersonate the major browsers
- pvg 2y agoShowhn at the time https://news.ycombinator.com/item?id=30378562 https://news.ycombinator.com/item?id=30378562
- croemer 2y agoBack then (2022) it was Firefox only
- jchw 2y agoI'm rooting for Ladybird to gain traction in the future. Currently, it is using cURL proper for networking. That is probably going to have some challenges (I think cURL is still limited in some ways, e.g. I don't think it can do WebSockets over h2 yet) but on the other hand, having a rising browser engine might eventually remove this avenue for fingerprinting since legitimate traffic will have the same fingerprint as stock cURL.
- rhdunn 2y agoIt would be good to see Ladybird's cURL usage improve cURL itself, such as the WebSocket over h2 example you mention. It is also a good test of cURL to see and identify what functionality cURL is missing w.r.t. real-world browser workflows.
- eesmith 2y agoI'm hoping this means Ladybird might support ftp URLs.
- navanchauhan 2y agoand even the Gopher protocol!
- nonrandomstring 2y agoWhen I spoke to these guys [0] we touched on those quirks and foibles that make a signature (including TCP stack stuff beyond control of any userspace app). I love this curl, but I worry that if a component takes on the role of deception in order to "keep up" it accumulates a legacy of hard to maintain "compatibility" baggage. Ideally it should just say... "hey I'm curl, let me in" The problem of course lies with a server that is picky about dress codes, and that problem in turn is caused by crooks sneaking in disguise, so it's rather a circular chicken and egg thing. [0] https://cybershow.uk/episodes.php?id=39 https://cybershow.uk/episodes.php?id=39
- immibis 2y agoWhat should instead happen is that Chrome should stop sending as much of a fingerprint, so that sites won't be able to fingerprint. That won't happen, since it's against Google's interests.
- gruez 2y agoThis is a fundamental misunderstanding of how TLS fingerprinting works. The "fingerprint" isn't from chrome sending a "fingerprint: [random uuid]" attribute in every TLS negotiation. It's derived from various properties of the TLS stack, like what ciphers it can accept. You can't make "stop sending as much of a fingerprint", without every browser agreeing on the same TLS stack. It's already minimal as it is, because there's basically no aspect of the TLS stack that users can configure, and chrome bundles its own, so you'd expect every chrome user to have the same TLS fingerprint. It's only really useful to distinguish "fake" chrome users (eg. curl with custom header set, or firefox users with user agent spoofer) from "real" chrome users.
- dochtman 2y agoPart of the fingerprint is stuff like the ordering of extensions, which Chrome could easily do but AFAIK doesn’t. (AIUI Google’s Play Store is one of the biggest TLS fingerprinting culprits.)
- userbinator 2y agobut on the other hand, having a rising browser engine might eventually remove this avenue for fingerprinting If what I've seen from CloudFlare et.al. are any indication, it's the exact opposite --- the amount of fingerprinting and "exploitation" of implementation-defined behaviour has increased significantly in the past few months, likely in an attempt to kill off other browser engines; the incumbents do not like competition at all. The enemy has been trying to spin it as "AI bots DDoSing" but one wonders how much of that was their own doing...
- hansvm 2y agoHold up, one of those things is not like the other. Are we really blaming webmasters for 100x increases in costs from a huge wave of poorly written and maliciously aggressive bots?
- refulgentis 2y ago> Are we really blaming... No, they're discussing increased fingerprinting / browser profiling recently and how it affects low-market-share browsers.
- hansvm 2y agoI saw that, but I'm still not sure how this fits in: > The enemy has been trying to spin it as "AI bots DDoSing" but one wonders how much of that was their own doing... I'm reading that as `enemy == fingerprinters`, `that == AI bots DDoSing`, and `their own == webmasters, hosting providers, and CDNs (i.e., the fingerprinters)`, which sounds pretty straightforwardly like the fingerprinters are responsible for the DDoSing they're receiving. That interpretation doesn't seem to match the rest of the post though. Do you happen to have a better one?
- userbinator 2y ago"their own" = CloudFlare and/or those who have vested interests in closing up the Internet.
- johnisgood 2y agoI used to call it "cURL", but apparently officially it is curl, correct?
- cruffle_duffle 2y agoAs in “See-URL”? I’ve always called it curl but “see url” makes a hell of a lot of sense too! I’ve just never considered it and it’s one of those things you rarely say out loud.
- johnisgood 2y agoI prefer cURL as well, but according to official sources it is curl. :D Not sure how it is pronounced though, I pronounce it as "see-url" and/or "see-U-R-L". It might be pronounced as "curl" though.
- bdhcuidbebe 2y agoI’d guess Daniel pronounce it as ”kurl”, with a hard C like in ”crust”, since hes swedish.
- devwastaken 2y agoladybird does not have the resources to be a contender to current browsers. its well marketed but has no benefits or reason to exist over chromium. its also a major security risk as it is designed yet again in demonstrably unsafe c++.
- ec109685 2y agoThere are API’s that chrome provides that allows servers to validate whether the request came from an official chrome browser. That would detect that this curl isn’t really chrome. It’d be nice if something could support curl’s arguments but drive an actual headless chrome browser.
- binarymax 2y agoI’m interested in learning more about this. Are these APIs documented anywhere and are there server side implementation examples that you know of? EDIT: this is the closest I could find. https://developers.google.com/chrome/verified-access/overview https://developers.google.com/chrome/verified-access/overvie... ...but it's not generic enough to lead me to the declaration you made.
- KTibow 2y agoI think they confused Chrome and Googlebot.
- bowmessage 2y agoThere’s no way this couldn’t be replicated by a special build of curl.
- darrenf 2y agoAre you referring to the Web Environment Integrity[0] stuff, or something else? 'cos WEI was abandoned in late 2023. [0] https://github.com/explainers-by-googlers/Web-Environment-Integrity/ https://github.com/explainers-by-googlers/Web-Environment-In...
- do_not_redeem 2y agoSiblings are being more charitable about this, but I just don't think what you're suggesting is even possible. An HTTP client sends a request. The server sends a response. The request and response are made of bytes. Any bytes Chrome can send, curl-impersonate could also send. Chromium is open source. If there was some super secret handshake, anyone could copy that code to curl-impersonate. And if it's only in closed-source Chrome, someone will disassemble it and copy it over anyway.
- anon6362 2y agoSet a UA and any headers and/or cookies with regular cURL compiled with HTTP/3. This can be done with wrapper scripts very easily. 99.999% of problems solved with no special magic buried in an unclean fork.
- andrewmcwatters 2y agoThat’s not the point of this fork. And “unclean fork” is such an unnecessary and unprofessional comment. There’s an entire industry of stealth browser technologies out there that this falls under.
- psanford 2y agoThat doesn't solve the problem of TLS handshake fingerprinting, which is the whole point of this project.
- mmh0000 2y agoYou should really read the "Why" section of the README before jumping to conclusions: ``` some web services use the TLS and HTTP handshakes to fingerprint which client is accessing them, and then present different content for different clients. These methods are known as TLS fingerprinting and HTTP/2 fingerprinting respectively. Their widespread use has led to the web becoming less open, less private and much more restrictive towards specific web clients With the modified curl in this repository, the TLS and HTTP handshakes look exactly like those of a real browser. ``` For example, this will get you past Cloudflare's bot detection.
- 01HNNWZ0MV43FF 2y agoThe README indicates that this fork is compiled with nss (from Firefox) and BoringSSL (from Chromium) to resist fingerprinting based on the TLS lib. CLI flags won't do that.
- VladVladikoff 2y agoWait a sec… if the TLS handshakes look different, would it be possible to have an nginx level filter for traffic that claims to be a web browser (eg chrome user agent), yet really is a python/php script? Because this would account for the vast majority of malicious bot traffic, and I would love to just block it.
- gruez 2y agoThat's basically what security vendors like cloudflare does, except with even more fingerprinting, like a javascript challenge that checks the js interpreter/DOM.
- walrus01 2y agoJS to check user agent things like screen window dimensions as well, which legit browsers will have and bots will also present but with a more uniform and predictable set of x and y dimensions per set of source IPs. Lots of possibilities for js endpoint fingerprinting.
- Fripplebubby 2y agoI also present a uniform and predictable set of x and y dimensions per source IPs as a human user who maximizes my browser window
- gruez 2y agoMaximizing reduces the variations, but there's still quite a bit of variation because of different display resolution + scaling settings + OS configuration (eg. short or tall taskbars).
- walrus01 2y agoOr settings like auto-hide MacOS dock vs not auto hide, affecting the vertical size of the browser window.
- bossyTeacher 2y agoCool tool but it shouldn't matter whether the client is a browser or not. I feel sad that we need such a tool in the real world
- brutal_chaos_ 2y agoYou may enter our site iff you use software we approve. Anything else will be seen as malicious. Papers please! I, too, am saddened by this gatekeeping. IIUC custom browsers (or user-agent) from scratch will never work on cloudflare sites and the like until the UA has enough clout (money, users, etc) to sway them.
- DrillShopper 2y agoThis was sadly always going to be the outcome of the Internet going commercial. There's too much lost revenue in open things for companies to embrace fully open technology anymore.
- throwawaytodey 2y ago[dead]
- jrockway 2y agoIt's kind of the opposite problem as well; huge well-funded companies bringing down open source project websites. See Xe's journey here: https://xeiaso.net/blog/2025/anubis/ https://xeiaso.net/blog/2025/anubis/ One may posit "maybe these projects should cache stuff so page loads aren't actually expensive" but these things are best-effort and not the core focus of these projects. You install some Git forge or Trac or something and it's Good Enough for your contributors to get work done. But you have to block the LLM bots because they ignore robots.txt and naively ask for the same expensive-to-render page over and over again. The commercial impact is also not to be understated. I remember when I worked for a startup with a cloud service. It got talked about here, and suddenly every free-for-open-source CI provider IP range was signing up for free trials in a tight loop. These mechanical users had to be blocked. It made me sad, but we wanted people to use our product, not mine crypto ;)
- Jotalea 2y ago[flagged]
- 1100001111 2y ago[flagged]
- ryao 2y agoDid they also set IP_TTL to set the TTL value to match the platform being impersonated? If not, then fingerprinting could still be done to some extent at the IP layer. If the TTL value in the IP layer is below 64, it is obvious this is either not running on modern Windows or is running on a modern Windows machine that has had its default TTL changed, since by default the TTL of packets on modern Windows starts at 128 while most other platforms start it at 64. Since the other platforms do not have issues communicating over the internet, so IP packets from modern Windows will always be seen by the remote end with TTLs at or above 64 (likely just above). That said, it would be difficult to fingerprint at the IP layer, although it is not impossible.
- xrisk 2y agoWouldn’t the TTL value of received packets depend on network conditions? Can you recover the client’s value from the server?
- ralferoo 2y agoThe argument is that if the many (maybe the majority) of systems are sending packets with a TTL of 64 and they don't experience problems on the internet, then it stands to reason that almost everywhere on the internet is reachable in less than 64 hops (personally, I'd be amazed if it any routes are actually as high as 32 hops). If everywhere is reachable in under 64 hops, then packets sent from systems that use a TTL of 128 will arrive at the destination with a TTL still over 64 (or else they'd have been discarded for all the other systems already).
- ryao 2y agoWindows 9x used a TTL of 32. I vaguely recall hearing that it caused problems in extremely exotic cases, but that could have been misinformation. I imagine that >99.999% of the time, 32 is enough. This makes fingerprinting via TTL to distinguish between those who set it at 32, 64, 128 and 255 (OpenSolaris and derivatives) viable. That said, almost nobody uses Windows 9x or OpenSolaris derivatives on the internet these days, so I used values from systems that they do use for my argument that fingerprinting via TTL is possible.
- jruohonen 2y agoThe notion of real-world TLS/HTTP fingerprinting was somewhat new to me, and it looks interesting in theory, but I wonder what the build's use case really is? I mean you have the heavy-handed JavaScript running everywhere now.
- jamal-kumar 2y agoThis tool is pretty sweet in little bash scripts combo'd up with gnu parallel on red team engagements for mapping https endpoints within whatever scoped address ranges that will only respond to either proper browsers due to whatever, or with the SNI stuff in order. Been finding it super sweet for that. Can do all the normal curl switches like -H for header spoofing
- davidsojevic 2y agoThere's a fork of this that has some great improvements over to the top of the original and it is also actively maintained: https://github.com/lexiforest/curl-impersonate https://github.com/lexiforest/curl-impersonate There's also Python bindings for the fork for anyone who uses Python: https://github.com/lexiforest/curl_cffi https://github.com/lexiforest/curl_cffi
- nyanpasu64 2y agoI suppose it does make sense that a "make curl look like a browser" program would get sponsored by "bypass bot detection" services...
- ImHereToVote 2y agoEasy. Just make a small fragment shader to produce a token in your client. No bot is going to waste GPU resources to compile your shader.
- kelsey978126 2y agoWhy do people even think this? Bots almost always just use headful instrumented browsers now. if a human sitting at a keyboard can load the content, so can a bot.
- simpaticoder 2y agoSecurity measures never prevent all abuse. They raise the cost of abuse above an acceptable threshold. Many things work like this. Cleaning doesn't eliminate dirt, it dilutes the dirt below an acceptable threshold. Same for "repairing" and "defects", and some other pairs of things that escape me atm.
- abofh 2y agoThat's the same argument as CAPTCHA's - as far as I know there are no bots protesting them making their lives harder, but as a human - my life is much harder than it needs to be because things need me to prove I'm a human. Clean for data ingestion usually means complicated for data creation - optimizing for the advertisers has material cash value downstream, but customers are upstream, and making it harder is material too.
- unit149 2y ago[dead]
- gj035571 2y ago[flagged]
- userbinator 2y agoI'm always ambivalent about things like this showing up here. On one hand, it's good to let others know that there is still that bit of rebelliousness and independence alive amongst the population. On the other hand, much like other "freedom is insecurity" projects, attracting unwanted attention may make it worse for those who rely on them. Writing a browser is hard, and the incumbents are continually making it harder.
- jolmg 2y agoYour comment makes it sound like a browser being fingerprintable is a desired property by browser developers. It's just something that happens on its own from different people doing things differently. I don't see this as being about rebelliousness. Software being fingerprintable erodes privacy and software diversity.
- gkbrk 2y agoNot all browsers, but Chrome certainly desires to be fingerprintable. They even try to cryptographically prove that the current browser is an unmodified Chrome with Web Environment Integrity [1]. Doesn't get more fingerprintable than that. They provide an un-falsifiable certificate that "the current browser is an unmodified Chrome build, running on an unmodified Android phone with secure boot". If they didn't want to fingerprintable, they could just not do that and spend all the engineering time and money on something else. [1]: https://en.wikipedia.org/wiki/Web_Environment_Integrity https://en.wikipedia.org/wiki/Web_Environment_Integrity
- matt-p 2y agoI do kind of yern for the simpler days when if a website didn't mind bots it allowed it and if they did they blocked your user agent.
- andrethegiant 2y agoBack then websites weren’t so resource intensive. The negative backlash towards bots is kind of a side effect of how demanding expectations of web experiences has become.
- thunderfork 2y ago[dead]
- yard2010 2y agoNow I'm waiting for the MCP version of this.. :)
- andrethegiant 2y agohttps://github.com/puremd/puremd-mcp https://github.com/puremd/puremd-mcp handles this, probably some other MCP servers out there that handle this too
- doctor_radium 2y agoKudos to the coder and the poster. I'm involved in a browser project that runs on OpenSSL, and figured I'd have to dig through WireShark myself at some point to figure this stuff out. Well, I may still need to, but now have many points of reference. If the most common use of OpenSSL is Python, then in the age of Cloudflare, a Firefox TLS spoofing option isn't just a good idea, it's a necessity.
- INTPenis 2y agoOnly three patches and shell wrappers, this should get Daniel coding. Imho this should definitely be in mainline curl.
- deleted 2y ago[deleted]
- lcfcjs6 2y agoive been using puppeteer to query and read responses from deepseek.com, it works really well but i have to use a stealth mode and "headed" version to make it think its a person
- GNOMES 2y agoI had to do something like this with Ansible's get_url module once. Was having issues getting module to download an installer from a vendors site. Played with Curl/WGET, but was running into the same, while it worked from a browser. I ended up getting both Curl + get_url to work by passing the same headers my browser sent such as User-Agent, encoding, etc
- ck2 2y agoGood luck getting past imperva If you thought cloudflare challenge can be bad, imperva doesn't even want most humans through
- 1vuio0pswjnm7 2y ago"For these reasons, some web services use the TLS and HTTP handshakes to fingerprint which client is accessing them, and then present different content for different clients." Examples: [missing]