9 ms·

I am the PM working on Headless. Feel free to ask questions in this thread and I will try to answer them if I can. Edit: Please also note that we have not rele

I am the PM working on Headless. Feel free to ask questions in this thread and I will try to answer them if I can. Edit: Please also note that we have not released New Headless yet. We "merely" landed the source code.

LilyFrenchPants 4y ago
[flagged]
- natorion 4y ago
  What rumors? Can you provide any links or context?
mike_hearn 4y ago
Do you guys ever think about abusive automation at all, or do you just consider that other people's problem?
- scotty79 4y ago
  You call it abuse. Other people might call it use.
  - hackernewds 4y ago
    You call it use. Other people might call it abuse.
    - scotty79 4y ago
      That's my point exactly.
    - _moof 4y ago
      Are we just misquoting the Eurythmics now?
  - mike_hearn 4y ago
    I've not yet encountered anyone who doesn't consider spam to be a form of abuse.
    - scotty79 4y ago
      [flagged]
      - dang 4y ago
        Would you please stop posting in the flamewar style? We've had to ask you this in the past as well. It's not what this site is for, and destroys what it is for. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
        scotty79 4y ago
        I'm sorry. I'll try to bite my tongue more often when I'm in combative mood. Thanks for putting up with me so far.
    - aabbcc11 4y ago
      I am that anyone you mentioned. For example, autoposting on 4chan works very well for me. I spam goods on 4chan to buy or create opinions that I force.
    - account42 4y ago
      Spam can be an effective way around censorship. What is and isn't abuse often isn't as objective as some people want to pretend.
- lupire 4y ago
  Abusive how? Headed chrome can be automated, as can wget. Its bizarre to ask a client side program to implement server-side controls for users you want to allow on your site but throttle.
  - parker_mountain 4y ago
    Headed chrome adds a huge amount of overhead, and can also be fingerprinted more easily. This is a lot more declarative and makes it easier to run an abuse farm. Although, per my other comment, I don't see Headless as a tool that will particularly move the needle on abuse cases.
    - squeaky-clean 4y ago
      Isn't headed chrome usually fingerprinted by variables inserted by the chromedriver? You can rename these variables and be undetectable (you don't even have to recompile chromedriver, you can use a hex editor or a perl replacement). At least I've never gotten detected.
      - runlevel1 4y ago
        There are even Puppeteer plugins that will do it for you. [^1] The best detection I've come across so far (i.e. before this release) has just required I run headless Chrome in headed mode. Granted, I don't do a ton of scraping -- mostly just pulling data out of websites so that I can play with it in aggregate using more civilized tools. [1]: https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth https://github.com/berstend/puppeteer-extra/tree/master/pack...
- pdntspa 4y ago
  The implications of your question are beyond dystopian
  - DangitBobby 4y ago
    Please elaborate.
    - supriyo-biswas 4y ago
      See my comment[1] on this very thread. [1] https://news.ycombinator.com/item?id=34858232 https://news.ycombinator.com/item?id=34858232
    - pdntspa 4y ago
      Because it suggests adding usage controls, possibly enforced via cloud connectivity, to add restrictions that will inevitably make legitimate usage more difficult, frustrating, and most importantly, subject to outside control. Extend this far enough and the world starts to look like Doctorow's "Unauthorized Bread". This is an awful world, one designed to reinforce class divide and protect the entrenched and the rich by deliberately handicapping easily-accessible tools, because of a few bad actors. It creates a world where the code for literally everything is the most hideously complex version of itself because it is riddled with constant checks, phone-homes, and arbitrary usage limits. It further pushes us towards a disempowering future where our computing is limited exclusively to appliance-like devices whos inner workings are controlled for it. It stands against the very principle of general-purpose computing.
      - robertlagrant 4y ago
        That's not beyond dystopian. It's just dystopian. And implications of a question aren't either. Just your imagined implications. Questions aren't bad.
- parker_mountain 4y ago
  For what it's worth, the large "players" already seem to have this capability. They've forced pretty much everyone to roll out captchas, waf-level throttling, proof of work interstitials, and behavior-based fingerprinting. While my immediate response was the same as yours, I think this actually won't really change much in the way of bad actors. It's unfortunate, but basic controls (such as throttling, etc) are pretty much a floor-required feature - one way to avoid this burden is to do things like use 3rd party idp (aka google login). I'm not happy with the state of things but I don't think headless will particularly contribute to a material increase in abuse cases.
- aabbcc11 4y ago
  If you are soy developer who thinks cloudflare is god that should solve problems for you and use O(n^2) or even worse algorithms in your code so you can't even optimize it, it is only your problem, correct. In 2000 sites were running where code has been precisely made such way DDoS attack was impossible. Now it is heckin sauce of js malware obfuscated proprietary code. If your site like this, you deserved it. Cloudflare and such companies just need your money for solving 5-minutes problem like AWF that is just a regex, and you have limits even for user agent filtering, lol. Stop making shitcode and learn HTTP and TCP/IP theory, and you will make antispam filter that is 200% better than any cloudflare shit that is simply malware that runs cryptominer as a "IUAM" mode for their own benefit and you even pay for it.
Ono-Sendai 4y ago
Can this replace chromium embedded framework (CEF)?
- natorion 4y ago
  I fail to see the connection. Can you elaborate?
skybrian 4y ago
Can you talk about your team's motivations for improving headless mode? Any particular use cases in mind?
- natorion 4y ago
  Here are two of them: -Test reproducibility -Automated configuration rollouts in enterprise environments
  - ccooffee 4y ago
    Improving test environments is a huge upside. I haven't worked on browser automation in nearly a decade, but finding ways to work around shortcomings in the headless environment used to burn a lot of time on that team. I know of many small teams which made deliberate decisions NOT to do any browser automation tests (e.g. Selenium) because some issues required testing hooks in production code.
LinuxBender 4y ago
There are many comments about potential abuse. I would be curious to know if your team have ever challenged each other to look like a real person accessing a site and the other part of the team tries to detect and block them? If there is anyone that could do this it would be the creators of Headless. Why go through the exercise, one may ask? I believe it would be a critical thinking exercise to improve Headless even more while giving website maintainers a way to opt out of receiving traffic from it. If not your team, have you reached out to see if people from project zero would take on that challenge in their abundance of spare time? [1] [1] - https://googleprojectzero.blogspot.com/ https://googleprojectzero.blogspot.com/
- natorion 4y ago
  We regularly get feature requests for Headless to provide a field or property that can be polled by JS frameworks to detect if Headless is active e.g. windows.isBot. Well, Headless is open source, which means anybody could build a Headless version with such a property set to "I am a human, trust me!" and employ such a modified binary ... ;-)
  - LinuxBender 4y ago
    Oh absolutely, relying on a header would be a placebo at best. I was thinking more along the line of having two teams, one that develops Headless and another team at Google that try to defeat it non stop. An official game of cat and mouse. Project: Tom and Jerry? I guess legal would never buy into that name. My own personal method for my silly hobby sites is just to put passwords on things with an auth prompt delay.
    - dmix 4y ago
      Why should Google redteam their headless browser though? As other comments point out there's plenty of ways for bot detectors to id bots even with a browser which mirrors a normal one: https://news.ycombinator.com/item?id=34858056 https://news.ycombinator.com/item?id=34858056 Almost all of those are things are outside of the scope of the browser itself. And anyone doing serious bot attacks already have scripts/forks that modify these signals. I don't see how the chrome team could do much to help stop that at that level.
rmorey 4y ago
what makes the new one “Native” ?
- natorion 4y ago
  It's real Chromium, not emulating a Chromium browser. "Old" Headless was merely pretending to be a Chromium browser, the "New" Headless is a Chromium browser. "Old" Headless requires a parallel/duplicate implementation of features, which leads to subtle behavior differences or infeasability to support certain features e.g. extensions proper.
  - mh- 4y ago
    Does this mean we might see proper extension support in "New" Headless?
  - rmorey 4y ago
    wow, i had no idea the old headless was a reimplementation. congrats on landing the new one
oh_sigh 4y ago
Is it too late to change the name from "new headless"? It won't be new forever, and then there will need to be a new new mode, or a differently named one that people think is older because it isn't the new mode.
- dylan604 4y ago
  No, obviously, the next version will be called Newer Headless. Then you get the More Newer or Even Newer release. Or my personal favorite NewV2. /s Using the word "new" in naming conventions is the most moronic and shortsighted way to name things in something that is quite obviously going to be changing in the somewhat near future.
  - oh_sigh 4y ago
    It reminds me of "pont neuf"("new bridge" in French), which is the oldest bridge in Paris crossing the seine.
  - plugin-baby 4y ago
    See also: report_final_draft(1).doc
  - robertlagrant 4y ago
    New College is doing fine even with its name. It's just a name. Doesn't really matter.
    - dboreham 4y ago
      Also New Forest.
  - int_19h 4y ago
    By all rights, it ought to be EvenLessHead. ~
- natorion 4y ago
  You would be surprised how much we talked about that . New/old are just relevant for the transition period.
starik36 4y ago
Any chance of an build for the Raspberry Pi?
andrewstuart 4y ago
So this argument can be used these ways: --headless --headless=new --headless=chrome And each mean something different - but what? Not documented, very frustrating. Can you explain the difference between each of the above arguments?
- quenix 4y ago
  There are two headless mode: "chrome" and "new". --headless enables the default, which is "chrome". --headless=new enables "new".
  - deleted 4y ago
    [deleted]
nobu-mori 4y ago
Now that headless mode is a "real" Chromium instance, is it possible to add extension support to Chrome running in headless mode?
- rektide 4y ago
  I didn't know this was a restriction before! Interesting. I would have assumed old headless had a profile, that typical command-line efforts[1] would let one load extensions. Are we sure that your question is valid? Are we sure that previous headless Chrome didn't have profiles or couldn't load extensions? I'm not sure this question is valid. I think maybe the assumptions here are incorrect. The new Chrome headless certainly purports to be "just Chrome" "without actually rendering." One of the notable differences in the new headless mode is that it at least shows the stock/built-in extensions. From the submission: > Similarly, when it comes to plugins, the old headless Chrome used to return no plugins with navigator.plugins, which is a technique that used to be exploited for detection when Headless Chrome got released 6 years ago, cf this blog post. The new headless Chrome returns the same plugins as a headful Chrome, and that’s the same for the mimeTypes obtained with navigator.mimeTypes: Maybe perhaps the new headless is faking it, but my impression is that extensions definitely work as normal in the new headless Chrome. How or whether they worked before is another very very interesting question I'd like answers to. I do wish the AMA dev had actually replied to this. My hope is that this wasn't an issue before (but default plugins just weren't installed, and now they are, just to alter fingerprinting), and that now the situation is unchanged but default plugins are installed. [1] https://stackoverflow.com/questions/16800696/how-install-crx-chrome-extension-via-command-line https://stackoverflow.com/questions/16800696/how-install-crx...
  - nobu-mori 4y ago
    https://bugs.chromium.org/p/chromium/issues/detail?id=706008 https://bugs.chromium.org/p/chromium/issues/detail?id=706008 It looks like the new headless mode does support extensions.
andrewstuart 4y ago
What is the actual difference between old and new? Is there a list, an explanation anywhere?
tmm1 4y ago
So the --headless=new doesn't work on any released version of Chrome yet?

LilyFrenchPants 4y ago

[flagged]

natorion 4y ago
What rumors? Can you provide any links or context?

mike_hearn 4y ago

Do you guys ever think about abusive automation at all, or do you just consider that other people's problem?

scotty79 4y ago
You call it abuse. Other people might call it use.
- hackernewds 4y ago
  You call it use. Other people might call it abuse.
  - scotty79 4y ago
    That's my point exactly.
  - _moof 4y ago
    Are we just misquoting the Eurythmics now?
- mike_hearn 4y ago
  I've not yet encountered anyone who doesn't consider spam to be a form of abuse.
  - scotty79 4y ago
    [flagged]
    - dang 4y ago
      Would you please stop posting in the flamewar style? We've had to ask you this in the past as well. It's not what this site is for, and destroys what it is for. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
      - scotty79 4y ago
        I'm sorry. I'll try to bite my tongue more often when I'm in combative mood. Thanks for putting up with me so far.
  - aabbcc11 4y ago
    I am that anyone you mentioned. For example, autoposting on 4chan works very well for me. I spam goods on 4chan to buy or create opinions that I force.
  - account42 4y ago
    Spam can be an effective way around censorship. What is and isn't abuse often isn't as objective as some people want to pretend.
lupire 4y ago
Abusive how? Headed chrome can be automated, as can wget. Its bizarre to ask a client side program to implement server-side controls for users you want to allow on your site but throttle.
- parker_mountain 4y ago
  Headed chrome adds a huge amount of overhead, and can also be fingerprinted more easily. This is a lot more declarative and makes it easier to run an abuse farm. Although, per my other comment, I don't see Headless as a tool that will particularly move the needle on abuse cases.
  - squeaky-clean 4y ago
    Isn't headed chrome usually fingerprinted by variables inserted by the chromedriver? You can rename these variables and be undetectable (you don't even have to recompile chromedriver, you can use a hex editor or a perl replacement). At least I've never gotten detected.
    - runlevel1 4y ago
      There are even Puppeteer plugins that will do it for you. [^1] The best detection I've come across so far (i.e. before this release) has just required I run headless Chrome in headed mode. Granted, I don't do a ton of scraping -- mostly just pulling data out of websites so that I can play with it in aggregate using more civilized tools. [1]: https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth https://github.com/berstend/puppeteer-extra/tree/master/pack...
pdntspa 4y ago
The implications of your question are beyond dystopian
- DangitBobby 4y ago
  Please elaborate.
  - supriyo-biswas 4y ago
    See my comment[1] on this very thread. [1] https://news.ycombinator.com/item?id=34858232 https://news.ycombinator.com/item?id=34858232
  - pdntspa 4y ago
    Because it suggests adding usage controls, possibly enforced via cloud connectivity, to add restrictions that will inevitably make legitimate usage more difficult, frustrating, and most importantly, subject to outside control. Extend this far enough and the world starts to look like Doctorow's "Unauthorized Bread". This is an awful world, one designed to reinforce class divide and protect the entrenched and the rich by deliberately handicapping easily-accessible tools, because of a few bad actors. It creates a world where the code for literally everything is the most hideously complex version of itself because it is riddled with constant checks, phone-homes, and arbitrary usage limits. It further pushes us towards a disempowering future where our computing is limited exclusively to appliance-like devices whos inner workings are controlled for it. It stands against the very principle of general-purpose computing.
    - robertlagrant 4y ago
      That's not beyond dystopian. It's just dystopian. And implications of a question aren't either. Just your imagined implications. Questions aren't bad.
parker_mountain 4y ago
For what it's worth, the large "players" already seem to have this capability. They've forced pretty much everyone to roll out captchas, waf-level throttling, proof of work interstitials, and behavior-based fingerprinting. While my immediate response was the same as yours, I think this actually won't really change much in the way of bad actors. It's unfortunate, but basic controls (such as throttling, etc) are pretty much a floor-required feature - one way to avoid this burden is to do things like use 3rd party idp (aka google login). I'm not happy with the state of things but I don't think headless will particularly contribute to a material increase in abuse cases.
aabbcc11 4y ago
If you are soy developer who thinks cloudflare is god that should solve problems for you and use O(n^2) or even worse algorithms in your code so you can't even optimize it, it is only your problem, correct. In 2000 sites were running where code has been precisely made such way DDoS attack was impossible. Now it is heckin sauce of js malware obfuscated proprietary code. If your site like this, you deserved it. Cloudflare and such companies just need your money for solving 5-minutes problem like AWF that is just a regex, and you have limits even for user agent filtering, lol. Stop making shitcode and learn HTTP and TCP/IP theory, and you will make antispam filter that is 200% better than any cloudflare shit that is simply malware that runs cryptominer as a "IUAM" mode for their own benefit and you even pay for it.

Ono-Sendai 4y ago

Can this replace chromium embedded framework (CEF)?

natorion 4y ago
I fail to see the connection. Can you elaborate?

skybrian 4y ago

Can you talk about your team's motivations for improving headless mode? Any particular use cases in mind?

natorion 4y ago
Here are two of them: -Test reproducibility -Automated configuration rollouts in enterprise environments
- ccooffee 4y ago
  Improving test environments is a huge upside. I haven't worked on browser automation in nearly a decade, but finding ways to work around shortcomings in the headless environment used to burn a lot of time on that team. I know of many small teams which made deliberate decisions NOT to do any browser automation tests (e.g. Selenium) because some issues required testing hooks in production code.

LinuxBender 4y ago

There are many comments about potential abuse. I would be curious to know if your team have ever challenged each other to look like a real person accessing a site and the other part of the team tries to detect and block them? If there is anyone that could do this it would be the creators of Headless. Why go through the exercise, one may ask? I believe it would be a critical thinking exercise to improve Headless even more while giving website maintainers a way to opt out of receiving traffic from it. If not your team, have you reached out to see if people from project zero would take on that challenge in their abundance of spare time? [1] [1] - https://googleprojectzero.blogspot.com/ https://googleprojectzero.blogspot.com/

natorion 4y ago
We regularly get feature requests for Headless to provide a field or property that can be polled by JS frameworks to detect if Headless is active e.g. windows.isBot. Well, Headless is open source, which means anybody could build a Headless version with such a property set to "I am a human, trust me!" and employ such a modified binary ... ;-)
- LinuxBender 4y ago
  Oh absolutely, relying on a header would be a placebo at best. I was thinking more along the line of having two teams, one that develops Headless and another team at Google that try to defeat it non stop. An official game of cat and mouse. Project: Tom and Jerry? I guess legal would never buy into that name. My own personal method for my silly hobby sites is just to put passwords on things with an auth prompt delay.
  - dmix 4y ago
    Why should Google redteam their headless browser though? As other comments point out there's plenty of ways for bot detectors to id bots even with a browser which mirrors a normal one: https://news.ycombinator.com/item?id=34858056 https://news.ycombinator.com/item?id=34858056 Almost all of those are things are outside of the scope of the browser itself. And anyone doing serious bot attacks already have scripts/forks that modify these signals. I don't see how the chrome team could do much to help stop that at that level.

rmorey 4y ago

what makes the new one “Native” ?

natorion 4y ago
It's real Chromium, not emulating a Chromium browser. "Old" Headless was merely pretending to be a Chromium browser, the "New" Headless is a Chromium browser. "Old" Headless requires a parallel/duplicate implementation of features, which leads to subtle behavior differences or infeasability to support certain features e.g. extensions proper.
- mh- 4y ago
  Does this mean we might see proper extension support in "New" Headless?
- rmorey 4y ago
  wow, i had no idea the old headless was a reimplementation. congrats on landing the new one

oh_sigh 4y ago

Is it too late to change the name from "new headless"? It won't be new forever, and then there will need to be a new new mode, or a differently named one that people think is older because it isn't the new mode.

dylan604 4y ago
No, obviously, the next version will be called Newer Headless. Then you get the More Newer or Even Newer release. Or my personal favorite NewV2. /s Using the word "new" in naming conventions is the most moronic and shortsighted way to name things in something that is quite obviously going to be changing in the somewhat near future.
- oh_sigh 4y ago
  It reminds me of "pont neuf"("new bridge" in French), which is the oldest bridge in Paris crossing the seine.
- plugin-baby 4y ago
  See also: report_final_draft(1).doc
- robertlagrant 4y ago
  New College is doing fine even with its name. It's just a name. Doesn't really matter.
  - dboreham 4y ago
    Also New Forest.
- int_19h 4y ago
  By all rights, it ought to be EvenLessHead. ~
natorion 4y ago
You would be surprised how much we talked about that . New/old are just relevant for the transition period.

starik36 4y ago

Any chance of an build for the Raspberry Pi?

andrewstuart 4y ago

So this argument can be used these ways: --headless --headless=new --headless=chrome And each mean something different - but what? Not documented, very frustrating. Can you explain the difference between each of the above arguments?

quenix 4y ago
There are two headless mode: "chrome" and "new". --headless enables the default, which is "chrome". --headless=new enables "new".
- deleted 4y ago
  [deleted]

nobu-mori 4y ago

Now that headless mode is a "real" Chromium instance, is it possible to add extension support to Chrome running in headless mode?

rektide 4y ago
I didn't know this was a restriction before! Interesting. I would have assumed old headless had a profile, that typical command-line efforts[1] would let one load extensions. Are we sure that your question is valid? Are we sure that previous headless Chrome didn't have profiles or couldn't load extensions? I'm not sure this question is valid. I think maybe the assumptions here are incorrect. The new Chrome headless certainly purports to be "just Chrome" "without actually rendering." One of the notable differences in the new headless mode is that it at least shows the stock/built-in extensions. From the submission: > Similarly, when it comes to plugins, the old headless Chrome used to return no plugins with navigator.plugins, which is a technique that used to be exploited for detection when Headless Chrome got released 6 years ago, cf this blog post. The new headless Chrome returns the same plugins as a headful Chrome, and that’s the same for the mimeTypes obtained with navigator.mimeTypes: Maybe perhaps the new headless is faking it, but my impression is that extensions definitely work as normal in the new headless Chrome. How or whether they worked before is another very very interesting question I'd like answers to. I do wish the AMA dev had actually replied to this. My hope is that this wasn't an issue before (but default plugins just weren't installed, and now they are, just to alter fingerprinting), and that now the situation is unchanged but default plugins are installed. [1] https://stackoverflow.com/questions/16800696/how-install-crx-chrome-extension-via-command-line https://stackoverflow.com/questions/16800696/how-install-crx...
- nobu-mori 4y ago
  https://bugs.chromium.org/p/chromium/issues/detail?id=706008 https://bugs.chromium.org/p/chromium/issues/detail?id=706008 It looks like the new headless mode does support extensions.

andrewstuart 4y ago

What is the actual difference between old and new? Is there a list, an explanation anywhere?

tmm1 4y ago

So the --headless=new doesn't work on any released version of Chrome yet?