6 ms·

I chuckled at this: > "Do you see the impact you created for thousands of us without any warning or explanation? We are not your test subjects," said an angry

I chuckled at this: > "Do you see the impact you created for thousands of us without any warning or explanation? We are not your test subjects," said an angry sysadmin. "We are running professional services for multi million dollar programs. Do you understand how many hours of resources were wasted by your 'experiment'?" Did you, dear sysadmin, pay anything for Google Chrome? No? Are you in any contractual relation to Google that covers your use of Chrome? No? Well, there you have it. You are not a customer, hence you aren't really in any position to demand anything. You basically agreed to take whatever Google shoves down your throat for free, and if that includes "experiments", then that's what it is. If your multi-million dollar programs move so much money around, maybe take some of that to either invest in the necessary software - including the browser - so you are a paying customer and may actually demand anything, or pay some people to be up to date on the intricacies of Google Chrome and test them under your environment or disable them if undesired. The experiments are not exactly a secret program and have existed for a while. Firefox has a similar thing going. The Firefox one can definitely be disabled. The Chrome one unfortunately not (at least I don't know of a way at the moment, there might be one, maybe it's also only available for the Enterprise subscription that I have never personally heard of any business to be using, but for which money is actually paid, hence there might be leverage in that case to force Google to offer an opt-out). Or ultimately you could also just compile yourself a Chromium from scratch and update it regularly. I've done it, it's not that hard, and that gets you the ultimate level of control over that nice free piece of software that you depend your business upon.

tyingq 7y ago
Does Google run these experiments on the $50/year per device Chrome Enterprise?
- ehsankia 7y ago
  "experiments" is just new features that are rolled out gradually". This "experiment" was enabled on the beta branch for 5 months and no on ran into it. It was also enabled at 1% on stable and no one caught it. If these "experiments" didn't exist, the alternative would be that the new stable version would come out and it would break everyone, so we're back to square 1.
jve 7y ago
Yeah. > We are running professional services for multi million dollar programs. Do you understand how many hours of resources were wasted by your 'experiment'? -- Angry sysadmin Well, do they pay Google for testing in their environment? Chrome is a free-to-use product. Their rollout strategy is good. Not that they experiment in prod - the flag was in beta for 5 months. And then they turned that on in prod for %1 users, still no reports. Well what better could they do?
- oarsinsync 7y ago
  > Their rollout strategy is good. I think you've misunderstood what they did > the flag was in beta for 5 months. And then they turned that on in prod for %1 users, still no reports. It was in beta, had no reports, and then they turned it on for a subset of users, and created a flood of problems. > Well what better could they do? Release it in a new version that doesn't auto-enable for small subsets of people, but is enabled for everyone if they've deployed that version. It's how software releases used to work. Enterprise environments can then test before it gets rolled, and save this problem from occurring.
- vanadium 7y ago
  Making experiments opt-in seems like a good first step, no? As it is, there wasn’t even a clear opt-OUT.
  - jacquesm 7y ago
    Was there any opt-out at all? Even an unclear one would do, then at least users can point each other to the option.
- curryst 7y ago
  > Their rollout strategy is good. Their rollout strategy was bad. They tested it on 1% of beta users for a month. They should have ramped it to 100% in beta before thinking about prod. Maybe 1% for a week, 25% for a week, 75% for a week then 100% for a week. Then think about enabling it in prod. Testing on a subset of the subset of people that run beta is not enough to validate the functionality. Even then, they probably should have done a ramp in prod as well. There are certain configurations that seem unlikely to be tested in beta. VDI is one that comes to mind. Headless operations also seem less likely to be running the beta build.
omh 7y ago
Google have made many attempts to sell Google services, and Chrome in particular[1], as professional software that businesses can rely on. I've been at conferences where senior Google staff went to great lengths to present the benefits of switching to Chrome. > The experiments are not exactly a secret program and have existed for a while I've been responsible for managing Chrome in a professional environment for several years and looked in detail at many Chrome management settings. I was not aware of the experiments. They don't seem to be mentioned in the Chrome Enterprise documentation or policies. > Both can be disabled How can we disable them? This is on my to-do list for today, so would genuinely appreciate your help! [1] https://cloud.google.com/chrome-enterprise/ https://cloud.google.com/chrome-enterprise/
- Slartie 7y ago
  Whoa, I just set out to find the way to disable them. I thought it was possible, but I seem to have confused that with Firefox. Firefox calls it "studies", and you can easily opt out of them in the browser's options. Chrome (and chromium) calls the feature "field trials" and there doesn't really seem to be a way to opt out. I'm seriously baffled by the audacity of Google here. Seems as if the only way to actually get rid of them is to modify the Chromium codebase and compile it yourself. It might be worth to investigate whether there is some kind of host name that could be nullified on the network, of the server that the trials are loaded from. Sorry for that misinformation, I clearly have to update my original rant.
  - omh 7y ago
    > I'm seriously baffled by the audacity of Google here Join the club! > It might be worth to investigate whether there is some kind of host name that could be nullified on the network, of the server that the trials are loaded from. It looks like they're downloaded from https://clients4.google.com/chrome-variations/seed https://clients4.google.com/chrome-variations/seed Blocking the entire hostname would cause problems with other features of Chrome. But I think blocking the particular URL would need SSL interception which isn't very appealing.
    - 7y ago
m-p-3 7y ago
Does being a GSuite customer counts as paying for Google Chrome? If that doesn't count, there there is no scenario where you can say you've paid for it.
- chris_wot 7y ago
  It does. Under “Can I get phone support” it states: Yes, if you have a G Suite or G Suite for Education account, or if your company purchased Chrome Enterprise Support. https://support.google.com/chrome/a/answer/188447?hl=en https://support.google.com/chrome/a/answer/188447?hl=en
user740372 7y ago
So no pay, no voice?
- Slartie 7y ago
  Of course it's not that simple. There are other ways to gain leverage over Google and the Chrome development process than money. But money is the easiest one for enterprises, especially if they boast their multi-million-dollar programs, so there at least seems to be a cashflow of considerable size depending on Chrome, of which a tiny part may possibly be diverted. And for a lot of enterprises that are not exactly at Google-scale, paying for it and hence being able to request support (and maybe even make Google pay for some of the monetary damages, depending on how the contracts are written) is the only feasible option to motivate Google. However, Chromium being open-source, there is always the possibility of taking matters into your own hand here as an alternative to the above. That doesn't come free of charge as well, of course, the money just ends up somewhere else than Google.
- m45t3r 7y ago
  One thing is "well, your experiment broke our <detailed information of setup>, is there some instructions on how to disable it?" So, well, showing humbleness since this is a software that you're using for free. The other thing is the way that was described by the article: "I am a sysadmin from a multi zillion enterprise, your idiotic experiment broke our setup. Disable this ASAP and never do something similar ever again".
- ehsankia 7y ago
  These people had a voice, the problem was solved and reverted in 1.5 day. If they had paid, they would've had direct line and it potentially would've been fixed within hours instead.
jacquesm 7y ago
> Did you, dear sysadmin, pay anything for Google Chrome? No? Are you in any contractual relation to Google that covers your use of Chrome? The idea that because you did not pay anything for a product translates into 'you have no rights' really is getting old. Even if you don't pay you have plenty of rights and the provider of the software should not - and in many places can not - walk away claiming that because you did not pay you have no rights. It may not be the rights that you want, and it may very well be that certain behavior even if you don't like it is allowed but it just simply isn't true that you have no rights. And that contractual bit runs both ways: companies have tried to argue time and again that free users of websites and software products were bound by EULAs, that means you can't now turn around and suddenly claim that there is no contractual relationship. Yes, there is no paper contract. But you are in a relationship and if Google or any other provider claims that they have rights by the definition of the word 'consideration' so do you.
- Slartie 7y ago
  That is a straw man argument. We are not talking about an incident that may in any way be covered by the rights you might have against Google even though you did only use a free piece of software - like for example the right to not be actively and intentionally harmed by Google (think of using Chrome as trojan horse to infiltrate your network). I am not doubting that such rights exist. But I am doubting that you have any legal right to demand Google to not enable a generally non-destructive and well-intentioned feature in some randomly chosen installations of the software that just by bad luck happens to have a bug which makes the software non-functional (but not actively harmful) in your particular terminal server environment. There clearly was no malicious intent behind this particular issue, and Google is not obliged to test every single feature on every possible combinations of systems, including every kind of remote terminal solution imaginable. And, having established that the legal system does not help you here, I additionally doubt that you have any other leverage over Google to make them not do such feature enablements in randomly chosen installations if you don't pay for the software and if you don't happen to be a company of huge size (which might allow you to threaten to switch your hundreds of thousands of users to a different browser). But if you paid for it, Google just might be interested in keeping that cashflow flowing, and thus might be inclined to put the additional effort in to create an option for you to disable these random feature enablements, and/or to disable them outright for the paid Enterprise installations.
chris_wot 7y ago
Some of them do. If a company purchases GSuite, GSuite for Education or Chrome Enterprise Support then they have paid for this. https://support.google.com/chrome/a/answer/188447?hl=en https://support.google.com/chrome/a/answer/188447?hl=en
davvolun 7y ago
Exactly, and when the workaround is "just use one of the other 3 or 4 major browsers that are probably already installed on your work computer" (and if you're using an enterprise setup as mentioned in the article, you've probably got 2 or 3 things that are still locked into IE, probably IE6 for some stupid reason). If your business is so locked into using Chrome, and specifically Chrome, and you aren't paying money to Google to ensure it keeps working as you need it, then you have only yourself to blame. (And yes, being unable to disable the experiments thing, and not giving any warning before-hand, is a mistake on Google and the Chrome teams part, and hurts Chrome's enterprise-ready image, but that doesn't refute the point above). If an essential piece of your business relies on something you have no control over, then you only have yourself to blame when that reliability fails and you can't fix it. Cost of doing business, at worst.
- lykr0n 7y ago
  Chrome is the new IE. There are a bunch of SaaS tools and self hosted tools that only work in Chrome. The problem with Chrome is that developers push each other to use Chrome, so they only develop for Chrome, and the stuff they make might only work in Chrome.
  - davvolun 7y ago
    Absolutely, and if your business is repeating the same mistakes in Chrome as we saw 10-20 years ago from IE....
blub 7y ago
This self-defeating attitude and Schadenfreude is puzzling, if not downright disgusting. Google cheated their way into the #1 browser position by bundling Chrome with other installers (similar to the spyware of old), advertising for it on its other properties and misleading users into thinking that their current browser was worse. Now that they're #1 they're hard at work implementing anti-features such as disabling an API which was used for ad filtering, trying to keep users always logged in, tracking visited URLs, etc. Chrome is Google's trojan horse, they're not investing millions into it out of the goodness of their hearts. They're doing it because it gives them something priceless: control over the window people have to the internet. Once the above is clear, it becomes easy to understand why Chrome auto-updates and is nearly impossible to stop from doing so. With these experiments Google has a backdoor installed into most computers in the world. And you're saying people should not be upset about the backdoor, because they didn't pay for the trojan horse. It's not the people that should pay. It's Google for spying on them and for running "tests" on their computers for free.

tyingq 7y ago

Does Google run these experiments on the $50/year per device Chrome Enterprise?

ehsankia 7y ago
"experiments" is just new features that are rolled out gradually". This "experiment" was enabled on the beta branch for 5 months and no on ran into it. It was also enabled at 1% on stable and no one caught it. If these "experiments" didn't exist, the alternative would be that the new stable version would come out and it would break everyone, so we're back to square 1.

jve 7y ago

Yeah. > We are running professional services for multi million dollar programs. Do you understand how many hours of resources were wasted by your 'experiment'? -- Angry sysadmin Well, do they pay Google for testing in their environment? Chrome is a free-to-use product. Their rollout strategy is good. Not that they experiment in prod - the flag was in beta for 5 months. And then they turned that on in prod for %1 users, still no reports. Well what better could they do?

oarsinsync 7y ago
> Their rollout strategy is good. I think you've misunderstood what they did > the flag was in beta for 5 months. And then they turned that on in prod for %1 users, still no reports. It was in beta, had no reports, and then they turned it on for a subset of users, and created a flood of problems. > Well what better could they do? Release it in a new version that doesn't auto-enable for small subsets of people, but is enabled for everyone if they've deployed that version. It's how software releases used to work. Enterprise environments can then test before it gets rolled, and save this problem from occurring.
vanadium 7y ago
Making experiments opt-in seems like a good first step, no? As it is, there wasn’t even a clear opt-OUT.
- jacquesm 7y ago
  Was there any opt-out at all? Even an unclear one would do, then at least users can point each other to the option.
curryst 7y ago
> Their rollout strategy is good. Their rollout strategy was bad. They tested it on 1% of beta users for a month. They should have ramped it to 100% in beta before thinking about prod. Maybe 1% for a week, 25% for a week, 75% for a week then 100% for a week. Then think about enabling it in prod. Testing on a subset of the subset of people that run beta is not enough to validate the functionality. Even then, they probably should have done a ramp in prod as well. There are certain configurations that seem unlikely to be tested in beta. VDI is one that comes to mind. Headless operations also seem less likely to be running the beta build.

omh 7y ago

Google have made many attempts to sell Google services, and Chrome in particular[1], as professional software that businesses can rely on. I've been at conferences where senior Google staff went to great lengths to present the benefits of switching to Chrome. > The experiments are not exactly a secret program and have existed for a while I've been responsible for managing Chrome in a professional environment for several years and looked in detail at many Chrome management settings. I was not aware of the experiments. They don't seem to be mentioned in the Chrome Enterprise documentation or policies. > Both can be disabled How can we disable them? This is on my to-do list for today, so would genuinely appreciate your help! [1] https://cloud.google.com/chrome-enterprise/ https://cloud.google.com/chrome-enterprise/

Slartie 7y ago
Whoa, I just set out to find the way to disable them. I thought it was possible, but I seem to have confused that with Firefox. Firefox calls it "studies", and you can easily opt out of them in the browser's options. Chrome (and chromium) calls the feature "field trials" and there doesn't really seem to be a way to opt out. I'm seriously baffled by the audacity of Google here. Seems as if the only way to actually get rid of them is to modify the Chromium codebase and compile it yourself. It might be worth to investigate whether there is some kind of host name that could be nullified on the network, of the server that the trials are loaded from. Sorry for that misinformation, I clearly have to update my original rant.
- omh 7y ago
  > I'm seriously baffled by the audacity of Google here Join the club! > It might be worth to investigate whether there is some kind of host name that could be nullified on the network, of the server that the trials are loaded from. It looks like they're downloaded from https://clients4.google.com/chrome-variations/seed https://clients4.google.com/chrome-variations/seed Blocking the entire hostname would cause problems with other features of Chrome. But I think blocking the particular URL would need SSL interception which isn't very appealing.
  - 7y ago

m-p-3 7y ago

Does being a GSuite customer counts as paying for Google Chrome? If that doesn't count, there there is no scenario where you can say you've paid for it.

chris_wot 7y ago
It does. Under “Can I get phone support” it states: Yes, if you have a G Suite or G Suite for Education account, or if your company purchased Chrome Enterprise Support. https://support.google.com/chrome/a/answer/188447?hl=en https://support.google.com/chrome/a/answer/188447?hl=en

user740372 7y ago

So no pay, no voice?

Slartie 7y ago
Of course it's not that simple. There are other ways to gain leverage over Google and the Chrome development process than money. But money is the easiest one for enterprises, especially if they boast their multi-million-dollar programs, so there at least seems to be a cashflow of considerable size depending on Chrome, of which a tiny part may possibly be diverted. And for a lot of enterprises that are not exactly at Google-scale, paying for it and hence being able to request support (and maybe even make Google pay for some of the monetary damages, depending on how the contracts are written) is the only feasible option to motivate Google. However, Chromium being open-source, there is always the possibility of taking matters into your own hand here as an alternative to the above. That doesn't come free of charge as well, of course, the money just ends up somewhere else than Google.
m45t3r 7y ago
One thing is "well, your experiment broke our <detailed information of setup>, is there some instructions on how to disable it?" So, well, showing humbleness since this is a software that you're using for free. The other thing is the way that was described by the article: "I am a sysadmin from a multi zillion enterprise, your idiotic experiment broke our setup. Disable this ASAP and never do something similar ever again".
ehsankia 7y ago
These people had a voice, the problem was solved and reverted in 1.5 day. If they had paid, they would've had direct line and it potentially would've been fixed within hours instead.

jacquesm 7y ago

> Did you, dear sysadmin, pay anything for Google Chrome? No? Are you in any contractual relation to Google that covers your use of Chrome? The idea that because you did not pay anything for a product translates into 'you have no rights' really is getting old. Even if you don't pay you have plenty of rights and the provider of the software should not - and in many places can not - walk away claiming that because you did not pay you have no rights. It may not be the rights that you want, and it may very well be that certain behavior even if you don't like it is allowed but it just simply isn't true that you have no rights. And that contractual bit runs both ways: companies have tried to argue time and again that free users of websites and software products were bound by EULAs, that means you can't now turn around and suddenly claim that there is no contractual relationship. Yes, there is no paper contract. But you are in a relationship and if Google or any other provider claims that they have rights by the definition of the word 'consideration' so do you.

Slartie 7y ago
That is a straw man argument. We are not talking about an incident that may in any way be covered by the rights you might have against Google even though you did only use a free piece of software - like for example the right to not be actively and intentionally harmed by Google (think of using Chrome as trojan horse to infiltrate your network). I am not doubting that such rights exist. But I am doubting that you have any legal right to demand Google to not enable a generally non-destructive and well-intentioned feature in some randomly chosen installations of the software that just by bad luck happens to have a bug which makes the software non-functional (but not actively harmful) in your particular terminal server environment. There clearly was no malicious intent behind this particular issue, and Google is not obliged to test every single feature on every possible combinations of systems, including every kind of remote terminal solution imaginable. And, having established that the legal system does not help you here, I additionally doubt that you have any other leverage over Google to make them not do such feature enablements in randomly chosen installations if you don't pay for the software and if you don't happen to be a company of huge size (which might allow you to threaten to switch your hundreds of thousands of users to a different browser). But if you paid for it, Google just might be interested in keeping that cashflow flowing, and thus might be inclined to put the additional effort in to create an option for you to disable these random feature enablements, and/or to disable them outright for the paid Enterprise installations.

chris_wot 7y ago

Some of them do. If a company purchases GSuite, GSuite for Education or Chrome Enterprise Support then they have paid for this. https://support.google.com/chrome/a/answer/188447?hl=en https://support.google.com/chrome/a/answer/188447?hl=en

davvolun 7y ago

Exactly, and when the workaround is "just use one of the other 3 or 4 major browsers that are probably already installed on your work computer" (and if you're using an enterprise setup as mentioned in the article, you've probably got 2 or 3 things that are still locked into IE, probably IE6 for some stupid reason). If your business is so locked into using Chrome, and specifically Chrome, and you aren't paying money to Google to ensure it keeps working as you need it, then you have only yourself to blame. (And yes, being unable to disable the experiments thing, and not giving any warning before-hand, is a mistake on Google and the Chrome teams part, and hurts Chrome's enterprise-ready image, but that doesn't refute the point above). If an essential piece of your business relies on something you have no control over, then you only have yourself to blame when that reliability fails and you can't fix it. Cost of doing business, at worst.

lykr0n 7y ago
Chrome is the new IE. There are a bunch of SaaS tools and self hosted tools that only work in Chrome. The problem with Chrome is that developers push each other to use Chrome, so they only develop for Chrome, and the stuff they make might only work in Chrome.
- davvolun 7y ago
  Absolutely, and if your business is repeating the same mistakes in Chrome as we saw 10-20 years ago from IE....

blub 7y ago

This self-defeating attitude and Schadenfreude is puzzling, if not downright disgusting. Google cheated their way into the #1 browser position by bundling Chrome with other installers (similar to the spyware of old), advertising for it on its other properties and misleading users into thinking that their current browser was worse. Now that they're #1 they're hard at work implementing anti-features such as disabling an API which was used for ad filtering, trying to keep users always logged in, tracking visited URLs, etc. Chrome is Google's trojan horse, they're not investing millions into it out of the goodness of their hearts. They're doing it because it gives them something priceless: control over the window people have to the internet. Once the above is clear, it becomes easy to understand why Chrome auto-updates and is nearly impossible to stop from doing so. With these experiments Google has a backdoor installed into most computers in the world. And you're saying people should not be upset about the backdoor, because they didn't pay for the trojan horse. It's not the people that should pay. It's Google for spying on them and for running "tests" on their computers for free.