17 ms·
New headless Chrome has been released and has a near-perfect browser fingerprint
- transitivebs 4y agoThe cat & mouse game continues...
- Mimmy 4y agoI’m naive here but why would Chrome release a headless browser that makes it easier for bot developers to avoid detection?
- ethbr0 4y agoBecause none of the people complaining about headless bots (read probably: content and retail) are major stakeholders from Chrome's viewpoint.
- dmix 4y agoThis blog post is written that way because the guy works in the bot detection business so it's what he cares most about. But there are still plenty of legitimate use cases for wanting a headless browser that perfectly replicates a normal browser environment. The obvious ones are automated frontend testing tools like https://playwright.dev/ https://playwright.dev/
- hoistbypetard 4y agoExactly. And as the blog post mentioned, people who have a strong need to block bots have tools other than browser fingerprinting at their disposal. Quoth the post: > It’s important to leverage other signals such as: > > * Behavior (client-side and server-side) > * Different kinds of reputations (IP, sessions, user) > * Proxy detection, in particular, residential proxy detection > * Contextual information: time of the day, country, etc > * TLS fingerprinting. Having a headless browser that behaves exactly like a normal one is tremendously useful for making things. And people who really *need* to block bots also need to contend with "mechanical turk" style attackers anyway. These techniques are also very useful against that approach, which still may be cheaper than making an undetectable bot even with a near-perfect Chrome fingerprint available headless.
- aabbcc11 4y ago> * Behavior (client-side and server-side) Imagine count of false positives. > * Different kinds of reputations (IP, sessions, user) Almost all use mobile network right now. One IP can be sticked to thousands of users. Imagine count of false positives. > * Proxy detection, in particular, residential proxy detection Most residential proxy are just common ISP ips bought by a face or led by botnet. Imagine false positives of simple home users that are IP ranged like on 4chan. > * Contextual information: time of the day, country, etc script.execute(() => navigator.dateOffset = Math.random()...) script.execute(() => navigator.country = Math.random()...) script.execute(() => navigator.etc = Math.random()...) > * TLS fingerprinting. Imagine count of false positives, especially because there are 4 common tls fingerprints across browsers. Just cope and seethe that your antispam filters will never work, antibot measures are fail. Cloudflare turnstile is fail. Bots won as usual.
- mercurialuser 4y agoWe use a headless browser to load an internal webpage (with content that may be updated several times per day) and generate a pdf on-demand.
- dataviz1000 4y agoAs a bot developer, without taking legal steps (I do not break the law) there is no stopping me regardless.
- aabbcc11 4y agoBased. Make companies cope further, by scrapping their prices and make your own prices 5% lower so customers buy from you. Their sites lag even on 32GB ram devices, with animations everywhere and zero optimization. This is the only way to compensate it. Even if you have nothing to sell, you still should abuse their antispam filters.
- TAKEMYMONEY 4y agoheadless browsers or are faster than the normal browsers (no GUI) so your tests run faster
- deleted 4y ago[deleted]
- natorion 4y agoPM working on Headless here. Masking bots is not the reason why the new Headless mode was created. The goal is to provide an headless browser that can be used in web tests. The original Headless is essentially a separate browser implemented in parallel to "proper" Chromium. That results in all sorts of subtle reproducibility problems for developers using Headless for their tests.
- ilyt 4y ago> PM working on Headless here. Masking bots is not the reason why the new Headless mode was created. Right. But it will be massively used just for that.
- literallyroy 4y agoYes, same as many technologies with legitimate uses. Tor is largely used for illegal activities, yet many would say the anonymity it provides for the general public is worth it being created (or the anonymity it provided for US intelligence).
- ilyt 4y agoI'm not chasting anyone for building a piece of cool tech but that does seem like something like a holy grail for bots.
- SeanAnderson 4y ago
- hashseed 4y agoChrome sets navigator.webdriver to true when controlled by automation. Until now, bots could simply use headful mode to achieve the same effect that is now made available through the new headless implementation.
- deleted 4y ago[deleted]
- TAKEMYMONEY 4y ago> the new headless Chrome can still be detected using JS browser fingerprinting techniques [...] however, the task has become more challenging [...] I’m not going to share any new detection signals Any guesses?
- botflyguy 4y agoIn the bot detection methods I've seen so far on this, a large part of it is timing analyses where there is a significant difference between headed and headless, e.g. graphical operations, audio processing.
- bornfreddy 4y agoThat, or making sure that mouse really moved somewhere (in a sensible way) before the click occured.
- mwill 4y agoThis would have false positives for some accessibility software, I believe
- vntok 4y agoTrue, that's why you don't want to block the pageload on this signal alone, just use it to trigger a captcha.
- hoistbypetard 4y agoIt's pretty awful to make people who need accessibility software go through more captchas. Those are an accessibility nightmare.
- ryandrake 4y agoOr even non-disabled people who typically browse using the keyboard only. Please stop sending users who you find inconvenient to captchas!
- shp0ngle 4y agoThe best way to catch a robot is just to slap a captcha there. Everything else is kind of useless and not effective.
- luckylion 4y agoCaptchas also tell apart the average human visitor from the very committed human visitor that really, really, really needs to do whatever they can do on your website.
- bornfreddy 4y agoHa ha, good point. When presented with captcha I often decide I don't care that much and just close the page.
- Symbiote 4y agoI do the same, but sometimes I wish I could give better feedback. "Dear British Airways, I booked with SAS instead because you assumed a Linux user with Firefox was a bot." (Or maybe it was the other way round, I forgot.)
- lupire 4y agoThey are also very good at distinguishing paying Google users who get the fast-pass to Google captchas.
- luckylion 4y agoIs that a thing? What services do you need to buy to bypass recaptcha?
- sfe22 4y agoThat means way more captchas after this release, yay
- phiresky 4y agoGetting captchas solved reliably via a service costs around $1 per 1000 captchas so captchas are kinda useless as well if there's a tiny monetary incentive to get to whatever is behind the captcha.
- nullifidian 4y agoAre there non-headless browsers modified specifically to have extremely generic fingerprints? Hiding OS, GPU, fonts everything.
- matterhorn2000 4y agoFirefox (and probably others) have fingerprint protection. https://support.mozilla.org/en-US/kb/firefox-protection-against-fingerprinting https://support.mozilla.org/en-US/kb/firefox-protection-agai...
- nullifidian 4y agoAny chromium based forks?
- Eisenstein 4y agoBrave.
- krmbzds 4y ago+1. Also make sure to disable all cryptocurrency and "web3" related plugins for a pleasant experience.
- jeroenhd 4y agoGoing by the amount of upset advertising/cyberstalking companies that Brave is indistinguishable from Chrome, I think this may be the answer. I don't like the way they pretend(ed) to send funds to websites using their cryptocurrency services, though. Good software, sketchy company.
- zelphirkalt 4y agoTor browser (based on Firefox) seems to fit that bill.
- worksonmine 4y agoNot a browser but Arkenfox[1] hardens standard firefox. But it's not for everyone and using something this specific can be a problem in itself. [1]: https://github.com/arkenfox/user.js/ https://github.com/arkenfox/user.js/
- graderjs 4y agoI built a remote browser based on headless Chrome^0 and this is going to make things way easier. It's also great to see Google supporting Chrome use cases beyond "consumer browsing", and perhaps that's in large part been pushed by the "grass roots popularity" of things like puppeteer and playwright. One thing I'm hoping for (but have heard it would require extensive rejigging of almost absolutely everything) is Extensions support in this new headless. However, if I'm reading the winds, it seems as if things might be going there, because: - Tamper scripts now work on Firefox mobile - Non-webkit iOS browsers are in the works - It's technically possible to "shim" much of the chrome.extension APIs using RDP (the low-level protocol that pptr and its ilk are based on) which would lead essentially to a "parallel extensions runtime" and "alt-Webstore" with less restrictions, something which Google may not look merrily upon Anyway, back to "headless detection", for the remote isolated browser, I have been using an extensive bot detection evasion script that proxied many of the normal properties on navigator (like plugins, etc), and tested extensively against detectors like luca.gg/headless^1 Interestingly one of the most effective way to defeat "first wave" / non-sophisticated bots used to be simply throwing up a JS modal (alert, confirm, prompt) -- for the convenient way it kills the JS runtime until dismissed, and how you have to explicitly dismiss it. ^0 = https://github.com/crisdosyago/BrowserBox https://github.com/crisdosyago/BrowserBox ^1 = https://luca.gg/headless/ https://luca.gg/headless/
- deleted 4y ago[deleted]
- eazyson 4y ago[dead]
- mike_hearn 4y agoThe game continues. Back in 2010 when I was writing the first in-browser bot detection signals for Google (so BotGuard could spot embedded Internet Explorers) I wondered how long they might last. Surely at some point embedded browsers would become undetectable? It never happened - browsers are so complex that there will probably always be ways to detect when they're being automated. There are some less obvious aspects to this that matter a lot in practice: 1. You have to force the code to actually run inside a real browser in the first place, not simply inside a fast emulator that sends back a clean response. This is by itself a big part of the challenge. 2. Doing so is useful even if you miss some automated browsers, because adversaries are often CPU and RAM constrained in ways you may not expect. 3. You have to do something sensible if the User-Agent claims to be something obscure, old or alternatively, too new for you to have seen before. 4. The signals have to be well protected, otherwise bot authors will just read your JS to see what they have to patch next. Signal collection and obfuscation work best when the two are tightly integrated together. These days there are quite a few companies doing JS based bot detection but I noticed from write-ups by reverse engineers that they don't seem to be obfuscating what they're doing as well as they could. It's like they heard that a custom VM is a good form of obfuscation but missed some of the reasons why. I wrote a bit about why the pattern is actually useful a month ago when TikTok's bot detector was being blogged about: https://www.reddit.com/r/programming/comments/10755l2/reverse_engineering_tiktoks_vm_obfuscation_part_2/ https://www.reddit.com/r/programming/comments/10755l2/revers... tl;dr you want to use a mesh oriented obfuscation and a custom VM makes that easier. It's a means, not an end. Ad: Occasionally I do private consulting on this topic, mostly for tech firms. Bot detectors tend to be either something home-grown by tech/social networking firms, or these days sold as a service by companies like DataDome, HUMAN etc. Companies that want to own their anti-abuse stack have to start from scratch every time, and often end up with something subpar because it's very difficult to hire for this set of skills. You often end up hiring people with a generic ML background but then they struggle to obtain good enough signals and the model produces noise. You do want some ML in the mix (or just statistics) to establish a base level of protection and to ensure that when bots are caught their resources are burned, but it's not enough by itself anymore. I offer training courses on how to construct high quality JS anti-bot systems and am thinking of maybe in future offering a reference codebase you can license and then fork. If anyone reading this is interested, drop me an email: mike@plan99.net
- chuckwolfe 4y agoI tried with akami and it still didn’t work. Still need the stealth plugin and some additional tweaks to bypass
- londons_explore 4y agoIf you add DRM video playback to the fingerprint, it is pretty much impossible to fake... Either they have a real TPM with a real nvidia graphics card able to decrypt content with a real serial number... Or they don't... If one graphics card or TPM serial number starts acting bot-like, you can ban just that one.
- redox99 4y agoI don't believe DRM fingerprinting is used in the wild. Firefox shows when DRM is being used (like Netflix) and I've never seen it used outside that.
- xnx 4y agoHow does this work? Wouldn't a lot of real user-agents not have this capability and therefore not be able to be fingerprinted and banned in this way?
- beagle3 4y agoAlso shutting out a lot of older and weird devices (internet fridges, dumb smart tvs, and more, many Linux and bsd users) who can’t play DRM. Some sites won’t care, but for some this will be too high a price for avoiding headless bots.
- azalemeth 4y agoI browse with DRM disabled. Every time it gives me a notification about it, I view it as a "hah, fingerprinting avoided!" signal. Sites that use it get my anti-traffic. I don't buy, support, or condone DRM'd media and I actively disable EME on every browser I come across...
- natorion 4y agoI am the PM working on Headless. Feel free to ask questions in this thread and I will try to answer them if I can. Edit: Please also note that we have not released New Headless yet. We "merely" landed the source code.
- LilyFrenchPants 4y ago[flagged]
- natorion 4y agoWhat rumors? Can you provide any links or context?
- mike_hearn 4y agoDo you guys ever think about abusive automation at all, or do you just consider that other people's problem?
- scotty79 4y agoYou call it abuse. Other people might call it use.
- hackernewds 4y agoYou call it use. Other people might call it abuse.
- mike_hearn 4y ago
- ilyt 4y agoWhy my first reaction on the last part is "oh no!"? Seems something that would have more illegitimate/annoying use cases than good
- nine_k 4y agoCan't you say the same about a real browser with Selenium driving it? It's been available for years; has it been hugely detrimental for something?
- jeroenhd 4y agoIt's not like spam farms can't use their own version of Chromium that already mimicks a real browser. Relying on client side indicators for your bot detection will only catch the bots that don't care about being caught in the first place. Show an alert that says "welcome to my site!" for any browsers originating from a data center and you've probably filtered most of those out. I like automating menial tasks in shitty web UIs (i.e. clearing out a list of sessions/search history/ad providers that only allow removing a single entry at a time). Simply using Firefox also gets flagged by a lot of these shitty bot detection services. I've never seen them do any useful work. The only exception is maybe reCAPTCHA or Cloudflare's alternative; that seems to be quite good at catching actual bots, but I do hate most websites that use them because in Firefox you end up clicking on boats twenty times. They're also trivially bypassed by delegating your spamming to click farms, as 1000 minimum wage workers in a faraway country can be cheaper than paying for dev time to work around the minor nuisances of bot detection.
- harrisonjackson 4y agoWe have a chatbot that can send users screenshots of their CMS views (kanban, calendar, tables, gallery, etc) from inside of Slack. The screenshotting uses puppeteer and chromium and a read-only session to impersonate the user and screenshot their dashboard. It uses the old version of chromium and there were many gotchas that required a lot of extra scaffolding to actually render ours and other websites like they would on my laptop. This will hopefully make it easier for us to maintain once implemented.
- supriyo-biswas 4y agoI'm assuming the next step will be to bring to Cloudflare's pet project of TPM attestation into Chrome, otherwise known as PATs[1]. And just like that, not only would headless be defeated, but all of you using rooted devices and small time browsers would be left high and dry. It's "Right to read"[2] all over again. [1] https://www.ietf.org/archive/id/draft-private-access-tokens-01.html https://www.ietf.org/archive/id/draft-private-access-tokens-... [2] https://www.gnu.org/philosophy/right-to-read.en.html https://www.gnu.org/philosophy/right-to-read.en.html
- judge2020 4y agoWhat is the solution to automation then? What do I do when someone hits my content-rich Wordpress blog with a scraper that hits 100 pages a second to download my content, and my database falls over leading to real, legitimate users being unable to use my site? What if it’s not a legitimate scraper but someone with hundreds of proxies uses them to DDOS my site for days? Should I sacrifice my uptime to protect the freedom of those unwilling to attest that they’re running on real hardware?
- aumerle 4y agorate limit. Or paywall.
- supriyo-biswas 4y agoProof-of-work schemes such as Hashcash[1] and simple ratelimiting algorithms can act as deterrents to spamming and scraping attacks. There are other kinds of non-invasive bot management you can do as well, however, due to various reasons I'm not in a position to talk about it. A few other methods are mentioned at the end of the post being discussed[2]. [1] https://en.wikipedia.org/wiki/Hashcash https://en.wikipedia.org/wiki/Hashcash [2] https://antoinevastel.com/bot%20detection/2023/02/19/new-headless-chrome.html#does-it-mean-it-has-become-impossible-to-detect-the-new-headless-chrome https://antoinevastel.com/bot%20detection/2023/02/19/new-hea...
- geokon 4y ago
- chirau 4y agoHow do i set the new part of the headless flag in Python? The article mentions that to use this you need to specify the --headless=new flag. I know that to set the headless flag i can just use this code: from selenium.webdriver.chrome.options import Options options = Options() options.headless = True But how would I specify the new part of the flag/option?
- mixedCase 4y ago[flagged]
- pRusya 4y agoThere's a mention to this in the recent Selenium blog post https://www.selenium.dev/blog/2023/headless-is-going-away/#after https://www.selenium.dev/blog/2023/headless-is-going-away/#a... Basically omit options.headless and use options.add_argument("--headless=new") instead.
- novaleaf 4y agoI am using the new headless Chrome for my Browser-Automation SaaS (PhantomJsCloud.com) and it is working great. It fixes some nagging compatibilities with certain websites. I don't bother with anti-bot mitigations, and I don't expect this to be useful in that regard. commercial Anti-Bot doesn't care about how much you spoof your browser fingerprint. feel free to AMA
- newhotelowner 4y agoCan you share the code for how to launch a new headless chrome?
- novaleaf 4y agoIf you are using Puppeteer, it's easy, just add "headless":"new" as described here: https://pptr.dev/guides/chrome-extensions https://pptr.dev/guides/chrome-extensions fyi, The new version just helps with compatibility, I have not seen it impact anti-bot detection at all.
- newhotelowner 4y agoI just realized that I use your service to get pool monitoring data. I configured everything (login & scraping) and start fetching data using your serivce. Then I discoverd that you have to login to load the dashboard but the their get API only requires the serial number to fetch the data .
- novaleaf 4y agoIf you want to get billing details automatically (without logging in) just inspect the Http Response sent back with every api request. it says how many credits are remaining, the api cost, etc.
- mnutt 4y agoDoes phantomjscloud still run phantomjs? How do you keep it up to date with security patches? Have you benchmarked how this new chrome performs against the old headless chrome and against phantomjs?
- cratermoon 4y ago> As you can imagine, given my position at DataDome (a bot detection company), I’m not going to share any new detection signals as I used to do Here comes the sales pitch....
- BonoboIO 4y agoAt the end we come to a browser and we have to emulate a mouse that does all the clicking.
- eimrine 4y ago> navigator.plugins.length = 0 So, any website on the Internets can know how many plugins my browser has? Ridiculously!
- joshschreuder 4y agoIt would seem like no, in recent times at least. In recent browser versions (Chrome 94+, Firefox 99+, etc.) it's been changed to only report the default PDF plugins https://developer.mozilla.org/en-US/docs/Web/API/Navigator/plugins#browser_compatibility https://developer.mozilla.org/en-US/docs/Web/API/Navigator/p...
- deleted 4y ago[deleted]
- PascLeRasc 4y agoThis is off topic but when did we get the ability to use spaces in URLs?
- pixelesque 4y agoBrowsers have automatically done the "correct" thing (converting to "bot%20detection") under-the-hood for years in my experience. I remember MS FrontPage-made sites with spaces in the name and IE would work with them.
- syrrim 4y agoin what sense? spaces as mod encoded (%20) values have been around ever since I've used the web. those spaces are occassionally displayed as spaces in the url bar, depending on the context.
- jaimex2 4y agoNo one stopped a Chromium fork from this earlier.
- thekingshorses 4y agoI wish I can automate some of the banking tasks. I tried but couldn't automate Chase, Citi or CapitalOne. If anyone has a working script to login and perform simple task on one of these sites, please share it.
- user3939382 4y agoLast time I was able to automate Chase by targeting their mobile site which, at the time anyway, had a dedicated URI. Mobile site was simple HTML and easy to scrape.
- jasmer 4y agoWe should assume anyone visiting a site without some kind of credentialed login is a 'bot'. Or for all intents and purposes 'noise' traffic. It'd be nice for the powers that be develop an anonymous cookie standard to allow people to flag themselves as 'humans' without enabling the host to know anything about them. We are fighting wars over problems that we have created for ourselves.
- aabbcc11 4y agoNo one is gonna use your malware tracking cookies, I always block them and Chrome as good browser also detects such usage. Also, by european law, you have no force to make "anonymous" cookie. We already lived after evercookies, now there are no any evercookies. If I ever detect that site uses "anonymous" cookies without my consent, well you will have to pay a lot of $$ for me as a compensation. Enjoy, try your luck, man. I need money, anyways and I love jurisdiction. Move on.