10 ms·

Google Chrome experiment crashes browser tabs, impacts companies worldwide

155 points by pasharayan 7y ago

voiper1 7y ago
>Many didn't know that Chrome engineers could run experiments on their tightly-controlled Chrome installations, let alone that Google engineers could just ship changes to everyone's browsers without any prior approval. Uhm, isn't that the entire auto-update feature, that google ships changes without you even being aware?
- esquire_900 7y ago
  Strange indeed that "tightly controlled enterprise environments" are so blissfully unaware of the functionality of chrome.
  - tomc1985 7y ago
    There aren't a lot of alternatives to the current trend of shipping shit fast and furious.
    - esquire_900 7y ago
      Firefox does: https://support.mozilla.org/en-US/kb/firefox-esr-release-cycle https://support.mozilla.org/en-US/kb/firefox-esr-release-cyc...
      - mrspuratic 7y ago
        ESR user for many years, plus useragent-switcher for sites that pretend to care about my "security" (but mostly care about CSS eye candy and the latest whizzbang). Free beta tester for other people isn't in my job description ;-)
        progval 7y ago
        > plus useragent-switcher for sites that pretend to care about my "security" What kind of site does that? I'm on ESR too (from Debian's repos) and never seen anything like it
        krageon 7y ago
        Specifically this example I don't know, but there are pretty big differences between look and feel on websites when sending a Chrome UA vs a Firefox UA. The most obvious example is Google.
        mrspuratic 7y ago
        I just lie about the version: an ESR can lag current by 8 major versions, ~12 months (with full security patching of course)
        mrspuratic 7y ago
        Citrix support kb is one. I've heard Twitter & Spotify too.
      - Freak_NL 7y ago
        I always recommend Firefox ESR to corporate clients that use our web-based product. Long-term stability, automatic security updates, and a yearly, well-documented release cycle that allows their IT department to test the upcoming ESR release and upgrade with confidence, while we get the benefit of a modern web browser to target.
      - sciurus 7y ago
        Also relevant is that there are additional management features targeted at Enterprise needs like group policy. https://www.mozilla.org/en-US/firefox/enterprise/ https://www.mozilla.org/en-US/firefox/enterprise/
    - dragonwriter 7y ago
      > There aren't a lot of alternatives to the current trend of shipping shit fast and furious. Sure there are, especially in enterprise, where IE is still a thing. There aren't good alternatives, but...
- tozeur 7y ago
  The experiments released to the general public should be stable. Unstable experiments are released in their canary builds.
  - voiper1 7y ago
    Article said they already ran it in beta and even experimented in stable! >"The experiment/flag has been on in beta for ~5 months," said David Bienvenu, a Google Chrome engineer. "It was turned on for stable (e.g., M77, M78) via an experiment that was pushed to released Chrome Tuesday morning." >"Prior to that, it had been on for about 1% of M77 and M78 users for a month with no reports of issues, unfortunately," he added.
    - omh 7y ago
      This indicates a flaw in their testing strategy. The bug affected Citrix-style environments, where perhaps 10-20 users are running Chrome on the same machine. If 1% of users are running the experiment the chance of two of those 10 users running the experiment is tiny, and when they experience the bug it's unlikely to get reported back to Google.
- kaared 7y ago
  My interpretation of the below quote from the article is that Google enabled the feature through pushing a configuration change to active clients. So if I'm a sysadmin and have done my due diligence and successfully tested the new build (with my configuration) before deploying it, I would be very unhappy if/when Google decides to turn this on without my consent or knowledge. Ref. quote from article: "Chrome engineers operate a system called Finch that lets them push updated Chrome settings to active installs, such as enabling or disabling experimental flags."
  - voiper1 7y ago
    Thanks, I didn't grok that was something different than the auto-update that I'm used to seeing.
- omh 7y ago
  This change didn't come from a binary update. We were affected and our version hadn't changed (in fact we weren't quite on the latest version - we were still testing it). We have updates disabled and are very much aware of how to manage this. Google changed a feature flag that was automatically picked up by existing copies of Chrome and changed their behaviour.
  - voiper1 7y ago
    Whoa, that wasn't clear in the article. Thanks for enlightening me!
    - dudus 7y ago
      That was crystal clear on the article. Be honest you just read the title.
      - voiper1 7y ago
        Right, I put a quote from the end of the article but I just read the title. /s
    - vanadium 7y ago
      In the bug thread, multiple orgs stated they tried to downgrade to the last version and encountered the exact same problem. This alone is going to leave orgs circumspect about their security and Chrome’s reliability, at the very least.
  - ehsankia 7y ago
    If Chrome is critical to your company, shouldn't you be testing on Beta before it goes stable? In this case, the feature was enabled on Beta for 5 months, yet not a single impacted company caught it. Even with 1% enabled, no one caught and reported it.
  - ikiris 7y ago
    This is explicit. Binary changes and flag flips are very strongly split. This is almost always a good thing, as it means you don't need to roll back binaries to revert problems.
BuckRogers 7y ago
I’ve seen Chrome in so many corporate environments and never understood why. Native browsers work fine and are crafted to work reliably in the environment they’re used in. My guess is some admins simply like Chrome and thus decided everyone should. I like Firefox but I keep it at home, at work I use Microsoft Edge.
- IggleSniggle 7y ago
  Perhaps easier to manage Chrome than managing a different browser for each operating system?
  - BuckRogers 7y ago
    Not really, that would be an excuse if someone said that to you. Most enterprises use Windows Server and manage this through Group Policy, which supports all browsers for all operating systems. If you're not using Windows Server it can be more difficult.
- mschuster91 7y ago
  People are using Chrome at home and demanding it at work is ome reason, many developers exclusively testing on Chrome the other.
- ocdtrekkie 7y ago
  An unfortunate number of services now ask people to use them only via Chrome. In my environment, I tell them to get over themselves, and we use Firefox. But a lot of services specifically state an expectation of using Chrome. It really is the new IE6.
- prepend 7y ago
  My work uses Microsoft Edge and it’s just not very good. Not as many sites work on Edge as Chrome. I hope this changes when Edge goes to Chromiun but from a UX perspective, too much stuff doesn’t work on Edge.
  - BuckRogers 7y ago
Animats 7y ago
Enterprise environments don't click on ads, so they don't matter to Google. Enterprise IT management should realize this.
- andybak 7y ago
  > Enterprise environments don't click on ads, Not sure where you've worked!
  - Animats 7y ago
    The article seemed to be about an enterprise where people in some job used the browser mostly to access in-house systems. That generates no revenue for Google.
- fgnkj 7y ago
  Thanks, this is the kind of insightful comments I come to HN for.
transreal 7y ago
This is the trouble with Google's aggressive auto-update policy. "Everyone is always on the latest" is mandated because everyone should have the latest security updates, but it works great except when it doesn't. Another example is a recent issue where a Chrome update broke WebView based Android apps and stopped them from being able to make certain types of network requests. It was fixed in 2 weeks, but that 2 week period was full of unhappy customers and lost revenue. I'm hoping the upcoming Chromium based Edge from Microsoft will allow IT Admins to control when a browser update is rolled out and give them more control over the update process. Chrome auto-updates can be disabled for networks behind a firewall by blocking the update server address, but that's a very crude way, and doesn't allow for updating a test machine to see how the new version works, or updating to the latest minus 1 version.
- isostatic 7y ago
  You could whitelist the test machine
- mimsee 7y ago
  Couldn’t one setup to use Chrome Enterprise with group plolicy rules on when certain updates can happen?
- senderista 7y ago
  This is not about auto-updates. Organizations which had disabled auto-updates were still affected. In this case Google changed Chrome behavior without changing the release version. That’s absolutely unacceptable.
lessname 7y ago
Even though it might break sometimes, it is one of the reasons why I choose Chrome over Firefox: The updates on Firefox take just too long (mostly before starting) on Chrome they just happen... However, it would be nice if they asked before.
omh 7y ago
I've been dealing with this bug for the last couple of days and it's been hugely frustrating. This was not a new version of Chrome or a software update. None of our software was updated at all (and we spent a long time checking!). But apparently Google have an ability to change a setting and globally affect the behaviour of all Chrome browsers by enabling experimental features. We carefully manage our software updates and patching so that we can test it and roll back if it impacts the business. Google had been good at understanding "enterprise" requirements - disabling automatic updates, setting policies etc. But this shows that they're really focused on consumers and business users will always be an afterthought.
- staktrace 7y ago
  As an admin why would you not also run the beta version of the browser in a limited setting? That would have helped you find out about this issue sooner, since the experiment was enabled there for a while.
  - yabadabadoes 7y ago
    I'm sure plenty of admins do, that doesn't mean you will test with the same flags chosen in an experiment it only means you will see changes going into stable permanently enabled a bit early. My own experience with ChromeOs is that I switch to beta, report bugs via interfaces provided (by Google) and then the chromium team acts surprised when the regression hits stable. If you ignore signals don't ask for them. Edit- clarity
- blub 7y ago
  That's putting it lightly. All your computers have a Google backdoor. It's not about businesses versus personal users, people are for Google an afterthought, they are mere information generators.
chaz6 7y ago
With Edge being based on Chromium I expect many corporations will start switching away from Google Chrome, but it's just swapping one corporate overlord for another.
- pluma 7y ago
  It's switching two corporate overlords for one, so I think they'll still see it as a net win.
Slartie 7y ago
I chuckled at this: > "Do you see the impact you created for thousands of us without any warning or explanation? We are not your test subjects," said an angry sysadmin. "We are running professional services for multi million dollar programs. Do you understand how many hours of resources were wasted by your 'experiment'?" Did you, dear sysadmin, pay anything for Google Chrome? No? Are you in any contractual relation to Google that covers your use of Chrome? No? Well, there you have it. You are not a customer, hence you aren't really in any position to demand anything. You basically agreed to take whatever Google shoves down your throat for free, and if that includes "experiments", then that's what it is. If your multi-million dollar programs move so much money around, maybe take some of that to either invest in the necessary software - including the browser - so you are a paying customer and may actually demand anything, or pay some people to be up to date on the intricacies of Google Chrome and test them under your environment or disable them if undesired. The experiments are not exactly a secret program and have existed for a while. Firefox has a similar thing going. The Firefox one can definitely be disabled. The Chrome one unfortunately not (at least I don't know of a way at the moment, there might be one, maybe it's also only available for the Enterprise subscription that I have never personally heard of any business to be using, but for which money is actually paid, hence there might be leverage in that case to force Google to offer an opt-out). Or ultimately you could also just compile yourself a Chromium from scratch and update it regularly. I've done it, it's not that hard, and that gets you the ultimate level of control over that nice free piece of software that you depend your business upon.
- tyingq 7y ago
  Does Google run these experiments on the $50/year per device Chrome Enterprise?
  - ehsankia 7y ago
    "experiments" is just new features that are rolled out gradually". This "experiment" was enabled on the beta branch for 5 months and no on ran into it. It was also enabled at 1% on stable and no one caught it. If these "experiments" didn't exist, the alternative would be that the new stable version would come out and it would break everyone, so we're back to square 1.
lightedman 7y ago
Its about time Google had a CFAA charge brought against them. I'm pretty sure most network admins didn't know Google had such a whopper of a backdoor deliberately snuck in through their security measures.
cimnine 7y ago
If something is critical infrastructure for you, better make sure you have a backup. I've seen this in a Swiss government agency that has Firefox installed exactly for this purpose. Their main browser is the built-in one, but if it can't or must not be used (e.g. because of a zero-day threat), they can quickly roll out a change to make Firefox the default browser until the problem/threat is over.
- lordfoom 7y ago
  >they can quickly roll out a change to make Firefox the default browser until the problem/threat is over. ...why not just make firefox the default browser full-time?
  - lotu 7y ago
    What if Firefox has a problem?
asah 7y ago
There's lots of browsers out there, why are these environments single-sourced to Chrome? Hope is not a strategy.
- rchaud 7y ago
  Because for years, IT managers have been fending off requests from users to give them something other than IE6. Since 2008, Google Chrome has been the "winner" of the desktop browser wars, with IE falling away completely and Firefox's word of mouth impact plateauing against Google's relentless marketing of Chrome.
CJefferson 7y ago
What's the alternative? Google thought this feature was polished, finished, and ready for release. What used to happen was it would be pushed out in an update, and then everyone on a server would have seen the crash. This "experiment" is just slow releasing.
faceplantNapalm 7y ago
Man, I just do not get what's happening here. It sounds like there are just SO MANY things WRONG about how people could let problems like this affect them. First: it's harming terminal servers... What? Why would you need to use a browser on a terminal server? I just don't get that, first and foremost. By and large, a GUI should be a luxury, if you're remoting into a server. Even for loopback purposes, a browser should only be needed for a web server. Second: Why is Chrome, but most importantly GOOGLE Chrome left as the only option available to users? Obviously, it's been made clear in recent news that Microsoft has adopted Chrome as its browser core, but the Microsoft distribution should be on a different release channel, no? Third: Why is Google not firewalled and blacklisted, such that Chrome can't phone home. Seems like an obvious disease vector. We all know that Chrome routes around policy controls, by installing to user home directories, so that it can thwart frisky, tyrannical administrators that would seek to prevent users from installing wonky appz, progz and warez. Fourth: Why no enterprise LTS install. Google used to supply this to Windows Enterprise deployments. And if they did, it's, by definition, supposed to be a slower, more stable release channel (long term, often meaning six months, or perhaps years, with only incremental urgent security patches, for emergency zero day coverage). Shame on Google for crossing that line, but shame on enterprise subscribers if they installed non-professional consumer software on not just enterprise machines, but servers no less. Fifth: If you need browser on your terminal server, in order to get things done, you're doing something wrong. But really, if that browser is mission critical to performing actions on said server, and you are dependent on it, then I question your skill level, talent and wit in general. Browsers just aren't that special. There's no way Chrome should be such a crucial component, while dialed in to a server, such that only Chrome can accomplish things that cannot be performed by some other means. Please check yourself, because it sounds like you made your own bed, and you should be made to lie in it, because you are bad at your job. (or dump the whole situation, quit, find some other place to work, if you are subject to the bad decisions made by someone else, because this sounds stupid) Last: I bring this up, because I can't imagine the Chrome dependency being rooted in anything else. Intranets and JavaScript. Chrome's singular claim to fame is essentially it being a consistent JavaScript wrapper for client-side UI delivery. If you need to look at page elements that only work via JavaScript, again, you've created your own problem. And if JS is needed on a terminal server because intranet? Holy shit! You did it to yourself, and there's no two ways about it. You need to correct your situation, and extricate any such workflows that happen to (A.) rely on JS (B.) in a browser (C.) a browser you aren't really trying to control (D.) while remoting into another machine (E.) one that is indeed a server no less. Listen, remote desktop, firewalls, VPN access, and web technology are all great, but this is no way to run a business. But most of all, this is no such way to run a server. No company should be able to pull the rug out from under you by making uncontrolled contact with your server, in ANY way. Not because they aren't supposed to. Not because of some imaginary honor system that might forbid such a thing. But really, because it is impossible for the network traffic to even occur. Google should be unreachable, even if you've chosen to use their software. If Google is able to push changes to your shared access machine, you're the one in the wrong.
hieloz 7y ago
I came arcoss the loading circle tab suspending on Chrome 78.0.3904.97.Even if I close the tab,the circle was still on browser,Finally I had to restore down the chrome in order to clear it up.
sjustinas 7y ago
While Google might have fucked up, what could possibly be your workflow where a single crashed tab costs you one and a half day's of work?
- micael_dias 7y ago
  Well it wasn't just one tab. At my company all work is done on the browser (Web app) and the browser would just crash and there was no indication what could be the problem. So although it didn't cost the entirety of one a half days work per user it severely diminished productivity all around.
user740372 7y ago
Can we stop the "not paying, no demanding" BS? First it's unfair to silence people just because they are not paying. Second, we are always paying. If not with money with our privacy.
- m45t3r 7y ago
  The problem is the tone of voice. One thing is you asking, humbling, why did this happen, if it can be reverted since I had a problem in my specific setup and how I (as the non paying customer) can disable it. The other is to be arrogant saying that I am a multi zillion company, you're idiotic for enabling this feature and I demand you to disable it ASAP and never do this again. The quote from this article is clearly from the later case, even if I am sure that other people reported it more humbly.
rbdeveloper 7y ago
Thank god Brave has field trials disabled by default [1], among many other Google "services" [2]. [1] https://github.com/brave/brave-browser/issues/4283 https://github.com/brave/brave-browser/issues/4283 [2] https://github.com/brave/brave-browser/wiki/Deviations-from-Chromium-(features-we-disable-or-remove) https://github.com/brave/brave-browser/wiki/Deviations-from-...
a3n 7y ago
Except for the intent, it sounds like an attack. "The attack is only possible through a carefully crafted browser ..."
Shorel 7y ago
The problem I see here is that companies are not moving to Firefox en masse.
slenk 7y ago
So when Firefox did this earlier in the year for the expired cert, everyone was freaking out that Mozilla had that ability. Where is all the freaking out at Chrome? My theory is that is was mostly Chrome users (the majority) complaining last time when they had no personal experience.
chris_wot 7y ago
I think we are now seeing that the attitude of Google in the way they treat Chrome users is seriously worrying. There are now a litany if issues - from adversely affecting ad blockers, linking Google sync to logons to Gmail and to now rolling out “experiments” unannounced... surely by now it is time to seriously consider Firefox? Of course, Mozilla needs to make sure they never enable extensions like the way they did for the Mr Robot one many years ago...
datavirtue 7y ago
Reminds me of 2014 when drop down boxes in forms stopped working for two weeks. Left a bad taste in my mouth causing me to view Chrome as a toy. I never run one browser. Usually run three or four. Especially in an rds.

155 points by pasharayan 7y ago

>Many didn't know that Chrome engineers could run experiments on their tightly-controlled Chrome installations, let alone that Google engineers could just ship changes to everyone's browsers without any prior approval. Uhm, isn't that the entire auto-update feature, that google ships changes without you even being aware?

esquire_900 7y ago
Strange indeed that "tightly controlled enterprise environments" are so blissfully unaware of the functionality of chrome.
- tomc1985 7y ago
  There aren't a lot of alternatives to the current trend of shipping shit fast and furious.
  - esquire_900 7y ago
    Firefox does: https://support.mozilla.org/en-US/kb/firefox-esr-release-cycle https://support.mozilla.org/en-US/kb/firefox-esr-release-cyc...
    - mrspuratic 7y ago
      ESR user for many years, plus useragent-switcher for sites that pretend to care about my "security" (but mostly care about CSS eye candy and the latest whizzbang). Free beta tester for other people isn't in my job description ;-)
      - progval 7y ago
        > plus useragent-switcher for sites that pretend to care about my "security" What kind of site does that? I'm on ESR too (from Debian's repos) and never seen anything like it
        krageon 7y ago
        Specifically this example I don't know, but there are pretty big differences between look and feel on websites when sending a Chrome UA vs a Firefox UA. The most obvious example is Google.
        mrspuratic 7y ago
        I just lie about the version: an ESR can lag current by 8 major versions, ~12 months (with full security patching of course)
        mrspuratic 7y ago
        Citrix support kb is one. I've heard Twitter & Spotify too.
    - Freak_NL 7y ago
      I always recommend Firefox ESR to corporate clients that use our web-based product. Long-term stability, automatic security updates, and a yearly, well-documented release cycle that allows their IT department to test the upcoming ESR release and upgrade with confidence, while we get the benefit of a modern web browser to target.
    - sciurus 7y ago
      Also relevant is that there are additional management features targeted at Enterprise needs like group policy. https://www.mozilla.org/en-US/firefox/enterprise/ https://www.mozilla.org/en-US/firefox/enterprise/
  - dragonwriter 7y ago
    > There aren't a lot of alternatives to the current trend of shipping shit fast and furious. Sure there are, especially in enterprise, where IE is still a thing. There aren't good alternatives, but...
tozeur 7y ago
The experiments released to the general public should be stable. Unstable experiments are released in their canary builds.
- voiper1 7y ago
  Article said they already ran it in beta and even experimented in stable! >"The experiment/flag has been on in beta for ~5 months," said David Bienvenu, a Google Chrome engineer. "It was turned on for stable (e.g., M77, M78) via an experiment that was pushed to released Chrome Tuesday morning." >"Prior to that, it had been on for about 1% of M77 and M78 users for a month with no reports of issues, unfortunately," he added.
  - omh 7y ago
    This indicates a flaw in their testing strategy. The bug affected Citrix-style environments, where perhaps 10-20 users are running Chrome on the same machine. If 1% of users are running the experiment the chance of two of those 10 users running the experiment is tiny, and when they experience the bug it's unlikely to get reported back to Google.
kaared 7y ago
My interpretation of the below quote from the article is that Google enabled the feature through pushing a configuration change to active clients. So if I'm a sysadmin and have done my due diligence and successfully tested the new build (with my configuration) before deploying it, I would be very unhappy if/when Google decides to turn this on without my consent or knowledge. Ref. quote from article: "Chrome engineers operate a system called Finch that lets them push updated Chrome settings to active installs, such as enabling or disabling experimental flags."
- voiper1 7y ago
  Thanks, I didn't grok that was something different than the auto-update that I'm used to seeing.
omh 7y ago
This change didn't come from a binary update. We were affected and our version hadn't changed (in fact we weren't quite on the latest version - we were still testing it). We have updates disabled and are very much aware of how to manage this. Google changed a feature flag that was automatically picked up by existing copies of Chrome and changed their behaviour.
- voiper1 7y ago
  Whoa, that wasn't clear in the article. Thanks for enlightening me!
  - dudus 7y ago
    That was crystal clear on the article. Be honest you just read the title.
    - voiper1 7y ago
      Right, I put a quote from the end of the article but I just read the title. /s
  - vanadium 7y ago
    In the bug thread, multiple orgs stated they tried to downgrade to the last version and encountered the exact same problem. This alone is going to leave orgs circumspect about their security and Chrome’s reliability, at the very least.
- ehsankia 7y ago
  If Chrome is critical to your company, shouldn't you be testing on Beta before it goes stable? In this case, the feature was enabled on Beta for 5 months, yet not a single impacted company caught it. Even with 1% enabled, no one caught and reported it.
- ikiris 7y ago
  This is explicit. Binary changes and flag flips are very strongly split. This is almost always a good thing, as it means you don't need to roll back binaries to revert problems.

BuckRogers 7y ago

I’ve seen Chrome in so many corporate environments and never understood why. Native browsers work fine and are crafted to work reliably in the environment they’re used in. My guess is some admins simply like Chrome and thus decided everyone should. I like Firefox but I keep it at home, at work I use Microsoft Edge.

IggleSniggle 7y ago
Perhaps easier to manage Chrome than managing a different browser for each operating system?
- BuckRogers 7y ago
  Not really, that would be an excuse if someone said that to you. Most enterprises use Windows Server and manage this through Group Policy, which supports all browsers for all operating systems. If you're not using Windows Server it can be more difficult.
mschuster91 7y ago
People are using Chrome at home and demanding it at work is ome reason, many developers exclusively testing on Chrome the other.
ocdtrekkie 7y ago
An unfortunate number of services now ask people to use them only via Chrome. In my environment, I tell them to get over themselves, and we use Firefox. But a lot of services specifically state an expectation of using Chrome. It really is the new IE6.
prepend 7y ago
My work uses Microsoft Edge and it’s just not very good. Not as many sites work on Edge as Chrome. I hope this changes when Edge goes to Chromiun but from a UX perspective, too much stuff doesn’t work on Edge.
- BuckRogers 7y ago

Animats 7y ago

Enterprise environments don't click on ads, so they don't matter to Google. Enterprise IT management should realize this.

andybak 7y ago
> Enterprise environments don't click on ads, Not sure where you've worked!
- Animats 7y ago
  The article seemed to be about an enterprise where people in some job used the browser mostly to access in-house systems. That generates no revenue for Google.
fgnkj 7y ago
Thanks, this is the kind of insightful comments I come to HN for.

transreal 7y ago

This is the trouble with Google's aggressive auto-update policy. "Everyone is always on the latest" is mandated because everyone should have the latest security updates, but it works great except when it doesn't. Another example is a recent issue where a Chrome update broke WebView based Android apps and stopped them from being able to make certain types of network requests. It was fixed in 2 weeks, but that 2 week period was full of unhappy customers and lost revenue. I'm hoping the upcoming Chromium based Edge from Microsoft will allow IT Admins to control when a browser update is rolled out and give them more control over the update process. Chrome auto-updates can be disabled for networks behind a firewall by blocking the update server address, but that's a very crude way, and doesn't allow for updating a test machine to see how the new version works, or updating to the latest minus 1 version.

isostatic 7y ago
You could whitelist the test machine
mimsee 7y ago
Couldn’t one setup to use Chrome Enterprise with group plolicy rules on when certain updates can happen?
senderista 7y ago
This is not about auto-updates. Organizations which had disabled auto-updates were still affected. In this case Google changed Chrome behavior without changing the release version. That’s absolutely unacceptable.

lessname 7y ago

Even though it might break sometimes, it is one of the reasons why I choose Chrome over Firefox: The updates on Firefox take just too long (mostly before starting) on Chrome they just happen... However, it would be nice if they asked before.

omh 7y ago

I've been dealing with this bug for the last couple of days and it's been hugely frustrating. This was not a new version of Chrome or a software update. None of our software was updated at all (and we spent a long time checking!). But apparently Google have an ability to change a setting and globally affect the behaviour of all Chrome browsers by enabling experimental features. We carefully manage our software updates and patching so that we can test it and roll back if it impacts the business. Google had been good at understanding "enterprise" requirements - disabling automatic updates, setting policies etc. But this shows that they're really focused on consumers and business users will always be an afterthought.

staktrace 7y ago
As an admin why would you not also run the beta version of the browser in a limited setting? That would have helped you find out about this issue sooner, since the experiment was enabled there for a while.
- yabadabadoes 7y ago
  I'm sure plenty of admins do, that doesn't mean you will test with the same flags chosen in an experiment it only means you will see changes going into stable permanently enabled a bit early. My own experience with ChromeOs is that I switch to beta, report bugs via interfaces provided (by Google) and then the chromium team acts surprised when the regression hits stable. If you ignore signals don't ask for them. Edit- clarity
blub 7y ago
That's putting it lightly. All your computers have a Google backdoor. It's not about businesses versus personal users, people are for Google an afterthought, they are mere information generators.

chaz6 7y ago

With Edge being based on Chromium I expect many corporations will start switching away from Google Chrome, but it's just swapping one corporate overlord for another.

pluma 7y ago
It's switching two corporate overlords for one, so I think they'll still see it as a net win.

Slartie 7y ago

I chuckled at this: > "Do you see the impact you created for thousands of us without any warning or explanation? We are not your test subjects," said an angry sysadmin. "We are running professional services for multi million dollar programs. Do you understand how many hours of resources were wasted by your 'experiment'?" Did you, dear sysadmin, pay anything for Google Chrome? No? Are you in any contractual relation to Google that covers your use of Chrome? No? Well, there you have it. You are not a customer, hence you aren't really in any position to demand anything. You basically agreed to take whatever Google shoves down your throat for free, and if that includes "experiments", then that's what it is. If your multi-million dollar programs move so much money around, maybe take some of that to either invest in the necessary software - including the browser - so you are a paying customer and may actually demand anything, or pay some people to be up to date on the intricacies of Google Chrome and test them under your environment or disable them if undesired. The experiments are not exactly a secret program and have existed for a while. Firefox has a similar thing going. The Firefox one can definitely be disabled. The Chrome one unfortunately not (at least I don't know of a way at the moment, there might be one, maybe it's also only available for the Enterprise subscription that I have never personally heard of any business to be using, but for which money is actually paid, hence there might be leverage in that case to force Google to offer an opt-out). Or ultimately you could also just compile yourself a Chromium from scratch and update it regularly. I've done it, it's not that hard, and that gets you the ultimate level of control over that nice free piece of software that you depend your business upon.

tyingq 7y ago
Does Google run these experiments on the $50/year per device Chrome Enterprise?
- ehsankia 7y ago
  "experiments" is just new features that are rolled out gradually". This "experiment" was enabled on the beta branch for 5 months and no on ran into it. It was also enabled at 1% on stable and no one caught it. If these "experiments" didn't exist, the alternative would be that the new stable version would come out and it would break everyone, so we're back to square 1.

lightedman 7y ago

Its about time Google had a CFAA charge brought against them. I'm pretty sure most network admins didn't know Google had such a whopper of a backdoor deliberately snuck in through their security measures.

cimnine 7y ago

If something is critical infrastructure for you, better make sure you have a backup. I've seen this in a Swiss government agency that has Firefox installed exactly for this purpose. Their main browser is the built-in one, but if it can't or must not be used (e.g. because of a zero-day threat), they can quickly roll out a change to make Firefox the default browser until the problem/threat is over.

lordfoom 7y ago
>they can quickly roll out a change to make Firefox the default browser until the problem/threat is over. ...why not just make firefox the default browser full-time?
- lotu 7y ago
  What if Firefox has a problem?

asah 7y ago

There's lots of browsers out there, why are these environments single-sourced to Chrome? Hope is not a strategy.

rchaud 7y ago
Because for years, IT managers have been fending off requests from users to give them something other than IE6. Since 2008, Google Chrome has been the "winner" of the desktop browser wars, with IE falling away completely and Firefox's word of mouth impact plateauing against Google's relentless marketing of Chrome.

CJefferson 7y ago

What's the alternative? Google thought this feature was polished, finished, and ready for release. What used to happen was it would be pushed out in an update, and then everyone on a server would have seen the crash. This "experiment" is just slow releasing.

faceplantNapalm 7y ago

Man, I just do not get what's happening here. It sounds like there are just SO MANY things WRONG about how people could let problems like this affect them. First: it's harming terminal servers... What? Why would you need to use a browser on a terminal server? I just don't get that, first and foremost. By and large, a GUI should be a luxury, if you're remoting into a server. Even for loopback purposes, a browser should only be needed for a web server. Second: Why is Chrome, but most importantly GOOGLE Chrome left as the only option available to users? Obviously, it's been made clear in recent news that Microsoft has adopted Chrome as its browser core, but the Microsoft distribution should be on a different release channel, no? Third: Why is Google not firewalled and blacklisted, such that Chrome can't phone home. Seems like an obvious disease vector. We all know that Chrome routes around policy controls, by installing to user home directories, so that it can thwart frisky, tyrannical administrators that would seek to prevent users from installing wonky appz, progz and warez. Fourth: Why no enterprise LTS install. Google used to supply this to Windows Enterprise deployments. And if they did, it's, by definition, supposed to be a slower, more stable release channel (long term, often meaning six months, or perhaps years, with only incremental urgent security patches, for emergency zero day coverage). Shame on Google for crossing that line, but shame on enterprise subscribers if they installed non-professional consumer software on not just enterprise machines, but servers no less. Fifth: If you need browser on your terminal server, in order to get things done, you're doing something wrong. But really, if that browser is mission critical to performing actions on said server, and you are dependent on it, then I question your skill level, talent and wit in general. Browsers just aren't that special. There's no way Chrome should be such a crucial component, while dialed in to a server, such that only Chrome can accomplish things that cannot be performed by some other means. Please check yourself, because it sounds like you made your own bed, and you should be made to lie in it, because you are bad at your job. (or dump the whole situation, quit, find some other place to work, if you are subject to the bad decisions made by someone else, because this sounds stupid) Last: I bring this up, because I can't imagine the Chrome dependency being rooted in anything else. Intranets and JavaScript. Chrome's singular claim to fame is essentially it being a consistent JavaScript wrapper for client-side UI delivery. If you need to look at page elements that only work via JavaScript, again, you've created your own problem. And if JS is needed on a terminal server because intranet? Holy shit! You did it to yourself, and there's no two ways about it. You need to correct your situation, and extricate any such workflows that happen to (A.) rely on JS (B.) in a browser (C.) a browser you aren't really trying to control (D.) while remoting into another machine (E.) one that is indeed a server no less. Listen, remote desktop, firewalls, VPN access, and web technology are all great, but this is no way to run a business. But most of all, this is no such way to run a server. No company should be able to pull the rug out from under you by making uncontrolled contact with your server, in ANY way. Not because they aren't supposed to. Not because of some imaginary honor system that might forbid such a thing. But really, because it is impossible for the network traffic to even occur. Google should be unreachable, even if you've chosen to use their software. If Google is able to push changes to your shared access machine, you're the one in the wrong.

hieloz 7y ago

I came arcoss the loading circle tab suspending on Chrome 78.0.3904.97.Even if I close the tab,the circle was still on browser,Finally I had to restore down the chrome in order to clear it up.

sjustinas 7y ago

While Google might have fucked up, what could possibly be your workflow where a single crashed tab costs you one and a half day's of work?

micael_dias 7y ago
Well it wasn't just one tab. At my company all work is done on the browser (Web app) and the browser would just crash and there was no indication what could be the problem. So although it didn't cost the entirety of one a half days work per user it severely diminished productivity all around.

user740372 7y ago

Can we stop the "not paying, no demanding" BS? First it's unfair to silence people just because they are not paying. Second, we are always paying. If not with money with our privacy.

m45t3r 7y ago
The problem is the tone of voice. One thing is you asking, humbling, why did this happen, if it can be reverted since I had a problem in my specific setup and how I (as the non paying customer) can disable it. The other is to be arrogant saying that I am a multi zillion company, you're idiotic for enabling this feature and I demand you to disable it ASAP and never do this again. The quote from this article is clearly from the later case, even if I am sure that other people reported it more humbly.

rbdeveloper 7y ago

Thank god Brave has field trials disabled by default [1], among many other Google "services" [2]. [1] https://github.com/brave/brave-browser/issues/4283 https://github.com/brave/brave-browser/issues/4283 [2] https://github.com/brave/brave-browser/wiki/Deviations-from-Chromium-(features-we-disable-or-remove) https://github.com/brave/brave-browser/wiki/Deviations-from-...

a3n 7y ago

Except for the intent, it sounds like an attack. "The attack is only possible through a carefully crafted browser ..."

Shorel 7y ago

The problem I see here is that companies are not moving to Firefox en masse.

slenk 7y ago

So when Firefox did this earlier in the year for the expired cert, everyone was freaking out that Mozilla had that ability. Where is all the freaking out at Chrome? My theory is that is was mostly Chrome users (the majority) complaining last time when they had no personal experience.

chris_wot 7y ago

I think we are now seeing that the attitude of Google in the way they treat Chrome users is seriously worrying. There are now a litany if issues - from adversely affecting ad blockers, linking Google sync to logons to Gmail and to now rolling out “experiments” unannounced... surely by now it is time to seriously consider Firefox? Of course, Mozilla needs to make sure they never enable extensions like the way they did for the Mr Robot one many years ago...

datavirtue 7y ago

Reminds me of 2014 when drop down boxes in forms stopped working for two weeks. Left a bad taste in my mouth causing me to view Chrome as a toy. I never run one browser. Usually run three or four. Especially in an rds.