4 ms·

The cat & mouse game continues...

Mimmy 4y ago
I’m naive here but why would Chrome release a headless browser that makes it easier for bot developers to avoid detection?
- ethbr0 4y ago
  Because none of the people complaining about headless bots (read probably: content and retail) are major stakeholders from Chrome's viewpoint.
- dmix 4y ago
  This blog post is written that way because the guy works in the bot detection business so it's what he cares most about. But there are still plenty of legitimate use cases for wanting a headless browser that perfectly replicates a normal browser environment. The obvious ones are automated frontend testing tools like https://playwright.dev/ https://playwright.dev/
  - hoistbypetard 4y ago
    Exactly. And as the blog post mentioned, people who have a strong need to block bots have tools other than browser fingerprinting at their disposal. Quoth the post: > It’s important to leverage other signals such as: > > * Behavior (client-side and server-side) > * Different kinds of reputations (IP, sessions, user) > * Proxy detection, in particular, residential proxy detection > * Contextual information: time of the day, country, etc > * TLS fingerprinting. Having a headless browser that behaves exactly like a normal one is tremendously useful for making things. And people who really *need* to block bots also need to contend with "mechanical turk" style attackers anyway. These techniques are also very useful against that approach, which still may be cheaper than making an undetectable bot even with a near-perfect Chrome fingerprint available headless.
    - aabbcc11 4y ago
      > * Behavior (client-side and server-side) Imagine count of false positives. > * Different kinds of reputations (IP, sessions, user) Almost all use mobile network right now. One IP can be sticked to thousands of users. Imagine count of false positives. > * Proxy detection, in particular, residential proxy detection Most residential proxy are just common ISP ips bought by a face or led by botnet. Imagine false positives of simple home users that are IP ranged like on 4chan. > * Contextual information: time of the day, country, etc script.execute(() => navigator.dateOffset = Math.random()...) script.execute(() => navigator.country = Math.random()...) script.execute(() => navigator.etc = Math.random()...) > * TLS fingerprinting. Imagine count of false positives, especially because there are 4 common tls fingerprints across browsers. Just cope and seethe that your antispam filters will never work, antibot measures are fail. Cloudflare turnstile is fail. Bots won as usual.
  - mercurialuser 4y ago
    We use a headless browser to load an internal webpage (with content that may be updated several times per day) and generate a pdf on-demand.
- dataviz1000 4y ago
  As a bot developer, without taking legal steps (I do not break the law) there is no stopping me regardless.
  - aabbcc11 4y ago
    Based. Make companies cope further, by scrapping their prices and make your own prices 5% lower so customers buy from you. Their sites lag even on 32GB ram devices, with animations everywhere and zero optimization. This is the only way to compensate it. Even if you have nothing to sell, you still should abuse their antispam filters.
- TAKEMYMONEY 4y ago
  headless browsers or are faster than the normal browsers (no GUI) so your tests run faster
- deleted 4y ago
  [deleted]
natorion 4y ago
PM working on Headless here. Masking bots is not the reason why the new Headless mode was created. The goal is to provide an headless browser that can be used in web tests. The original Headless is essentially a separate browser implemented in parallel to "proper" Chromium. That results in all sorts of subtle reproducibility problems for developers using Headless for their tests.
- ilyt 4y ago
  > PM working on Headless here. Masking bots is not the reason why the new Headless mode was created. Right. But it will be massively used just for that.
  - literallyroy 4y ago
    Yes, same as many technologies with legitimate uses. Tor is largely used for illegal activities, yet many would say the anonymity it provides for the general public is worth it being created (or the anonymity it provided for US intelligence).
    - ilyt 4y ago
      I'm not chasting anyone for building a piece of cool tech but that does seem like something like a holy grail for bots.
      - richwater 4y ago
        Nice to know you are the arbiter of what is and what isn't a "cool piece of tech"
        ilyt 4y ago
        I don't know whether you're illiterate or just maliciously misinterpreted what I wrote.
- SeanAnderson 4y ago
hashseed 4y ago
Chrome sets navigator.webdriver to true when controlled by automation. Until now, bots could simply use headful mode to achieve the same effect that is now made available through the new headless implementation.

I’m naive here but why would Chrome release a headless browser that makes it easier for bot developers to avoid detection?

ethbr0 4y ago
Because none of the people complaining about headless bots (read probably: content and retail) are major stakeholders from Chrome's viewpoint.
dmix 4y ago
This blog post is written that way because the guy works in the bot detection business so it's what he cares most about. But there are still plenty of legitimate use cases for wanting a headless browser that perfectly replicates a normal browser environment. The obvious ones are automated frontend testing tools like https://playwright.dev/ https://playwright.dev/
- hoistbypetard 4y ago
  Exactly. And as the blog post mentioned, people who have a strong need to block bots have tools other than browser fingerprinting at their disposal. Quoth the post: > It’s important to leverage other signals such as: > > * Behavior (client-side and server-side) > * Different kinds of reputations (IP, sessions, user) > * Proxy detection, in particular, residential proxy detection > * Contextual information: time of the day, country, etc > * TLS fingerprinting. Having a headless browser that behaves exactly like a normal one is tremendously useful for making things. And people who really *need* to block bots also need to contend with "mechanical turk" style attackers anyway. These techniques are also very useful against that approach, which still may be cheaper than making an undetectable bot even with a near-perfect Chrome fingerprint available headless.
  - aabbcc11 4y ago
    > * Behavior (client-side and server-side) Imagine count of false positives. > * Different kinds of reputations (IP, sessions, user) Almost all use mobile network right now. One IP can be sticked to thousands of users. Imagine count of false positives. > * Proxy detection, in particular, residential proxy detection Most residential proxy are just common ISP ips bought by a face or led by botnet. Imagine false positives of simple home users that are IP ranged like on 4chan. > * Contextual information: time of the day, country, etc script.execute(() => navigator.dateOffset = Math.random()...) script.execute(() => navigator.country = Math.random()...) script.execute(() => navigator.etc = Math.random()...) > * TLS fingerprinting. Imagine count of false positives, especially because there are 4 common tls fingerprints across browsers. Just cope and seethe that your antispam filters will never work, antibot measures are fail. Cloudflare turnstile is fail. Bots won as usual.
- mercurialuser 4y ago
  We use a headless browser to load an internal webpage (with content that may be updated several times per day) and generate a pdf on-demand.
dataviz1000 4y ago
As a bot developer, without taking legal steps (I do not break the law) there is no stopping me regardless.
- aabbcc11 4y ago
  Based. Make companies cope further, by scrapping their prices and make your own prices 5% lower so customers buy from you. Their sites lag even on 32GB ram devices, with animations everywhere and zero optimization. This is the only way to compensate it. Even if you have nothing to sell, you still should abuse their antispam filters.
TAKEMYMONEY 4y ago
headless browsers or are faster than the normal browsers (no GUI) so your tests run faster
deleted 4y ago
[deleted]

natorion 4y ago

PM working on Headless here. Masking bots is not the reason why the new Headless mode was created. The goal is to provide an headless browser that can be used in web tests. The original Headless is essentially a separate browser implemented in parallel to "proper" Chromium. That results in all sorts of subtle reproducibility problems for developers using Headless for their tests.

ilyt 4y ago
> PM working on Headless here. Masking bots is not the reason why the new Headless mode was created. Right. But it will be massively used just for that.
- literallyroy 4y ago
  Yes, same as many technologies with legitimate uses. Tor is largely used for illegal activities, yet many would say the anonymity it provides for the general public is worth it being created (or the anonymity it provided for US intelligence).
  - ilyt 4y ago
    I'm not chasting anyone for building a piece of cool tech but that does seem like something like a holy grail for bots.
    - richwater 4y ago
      Nice to know you are the arbiter of what is and what isn't a "cool piece of tech"
      - ilyt 4y ago
        I don't know whether you're illiterate or just maliciously misinterpreted what I wrote.
SeanAnderson 4y ago

hashseed 4y ago

Chrome sets navigator.webdriver to true when controlled by automation. Until now, bots could simply use headful mode to achieve the same effect that is now made available through the new headless implementation.