9 ms·

Hey. Google Keystone tech lead here. We are aware of the issue, and we've stopped the release. We're building a replacement that fixes the problem. In the meant

Hey. Google Keystone tech lead here. We are aware of the issue, and we've stopped the release. We're building a replacement that fixes the problem. In the meantime, to fix affected machines: sudo rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle sudo ln -F /private/var /var This deletes the affected version of Keystone and reinstates the damaged /var symlink. The version of Keystone packaged with Chrome is not affected by this bug, so allowing it to reinstall Keystone will not recreate the issue.

dreamcompiler 7y ago
After that, you can also do what I do to prevent Google from reinstalling Keystone ever again: touch ~/Library/Google/GoogleSoftwareUpdate touch /Library/Google/GoogleSoftwareUpdate chmod 000 ~/Library/Google/GoogleSoftwareUpdate chmod 000 /Library/Google/GoogleSoftwareUpdate
- Shank 7y ago
  Honestly, if you're going to go this far, why not switch to Firefox or another Chromium/Blink-based browser, like Brave? It seems kind of counter productive to kill off the auto update system when you can just as easily switch to a browser that just doesn't do what Keystone does.
  - amiga-workbench 7y ago
    Its a bit like throwing the baby out with the bathwater isn't it? Chrome is a fine browser, botnet "features" aside.
    - dewey 7y ago
      It’s so easy to switch browsers so why even bother trying to fix some big ad company’s browser that is not acting in your interest?
  - dreamcompiler 7y ago
    Yes indeed but FF bogs down quicker than Chrome with lots of tabs, so I use both. More important, I like Google Earth and it tries to install Keystone too.
    - cbsmith 7y ago
      The next release of FF looks to be a game changer in that regard.
      - wilkystyle 7y ago
        It is much better (using v70 beta 8), but still has areas where performance lags behind Chrome. On a large board in https://miro.com/ https://miro.com/, for example, Firefox is laggy and jittery, whereas Chrome is buttery smooth.
        yoasif_ 7y ago
        You can report a performance problem -- I have had good luck with fixes: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Reporting_a_Performance_Problem https://developer.mozilla.org/en-US/docs/Mozilla/Performance...
      - nkozyra 7y ago
        Unfortunately I think we've heard this so many separate times that it's beginning to be the boy who cried wolf. I've heard "Firefox is better than it was" only for me to reinstall the latest and find it's still way cludgier than chrome.
        cbsmith 7y ago
        Sure. I hear that, but there have been some specific MacOS issues that have lead to it performing worse on MacOS than on other platforms, and they seem to be getting addressed in the Nightly builds. In general, I've found it to be much better than Chrome, but as always YMMV.
        7y ago
- jacobsenscott 7y ago
  I certainly understand the desire to rage kill google software update because they messed up, but people shouldn't actually do this because they'll be vulnerable to all future malware that targets chrome. And this varsectomany bug will never happen again.
  - dreamcompiler 7y ago
    This is not rage-killing. I've been doing this for several years because Keystone is a ridiculous resource hog and I fundamentally disagree with the notion that any software should be allowed to run (much less change the configuration of) my machine without my explicit permission. I'm willing to stay on top of the malware situation and update Chrome manually. I wish I didn't have to, but Google leaves me no other option.
  - robin_reala 7y ago
    Only if you continue using Chrome.
  - prepend 7y ago
    I think the probability of Google freaking out and pushing ads to my system is higher than the probability of me a chrome zero day that I give a shit. They are both very low, but I’d rather programs not change my stuff against my will, even if they are trying to protect me.
  - Gibbon1 7y ago
    Apple needs to provide the user with the ability to ban software from google and other malign companies. Problem partly solved.
- Scapeghost 7y ago
  I would change that to just /Library/Google/ to prevent Google from putting ANYthing outside its .app bundle or the normal user preferences folders.
  - saagarjha 7y ago
    TIL that Google puts a "brand" code in that folder to identify how you downloaded Chrome.
- apostacy 7y ago
  I usually do chflags schg instead of chmod 000. I know it might seem like overkill, but Google is very sneaky, and I would not put it past Keystone to just change the permissions for itself.
  - dreamcompiler 7y ago
    That's very hardcore, but I agree with your logic.
  - masklinn 7y ago
    > I would not put it past Keystone to just change the permissions for itself. From experience, they absolutely do it.
    - apostacy 7y ago
      They would call it "repairing permissions". OK, so I didn't break into someone's house, I just "repaired" their door that had locked me out.
  - Scapeghost 7y ago
    > I usually do chflags schg instead of chmod 000. Thank you. This was the best tip to come out of this whole discussion. I'll identify every location Google apps write to, and lock them out with this.
- Tinfoilhat666 7y ago
  Thank you! Because of Keystone, I have decided to treat Google Chrome as malware. I won't install it unless I really have to. One reason is that I have to test websites on Chrome. I can either run it on a virtual machine or disable the updater as you suggest.
norberg 7y ago
We have a revised set of commands that fix the symlink more correctly. These can only be run from macOS Recovery Console: rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle ln -shf /private/var /var chflags -h restricted /var chflags -h hidden /var xattr -sw com.apple.rootless "" /var
- AdieuToLogic 7y ago
  > We have a revised set of commands that fix the symlink more correctly. Tell your team and your supervisors this: My computer is not your playground.
  - cududa 7y ago
    I’m sure most engineers on the team feel awful. They’re clearly trying, and maybe in a day or so we should figure out the nags ember breakdown. But for the time being, let’s let engineers do their job?
    - AdieuToLogic 7y ago
      > I’m sure most engineers on the team feel awful. There is no legitimate reason for a user-space install to manipulate system directories. So for an install to do so, there must have been an conscious decision made and code written to make real. Therefore, for this system manipulation to have both been introduced and released, "most engineers on the team" either raised no problems with it or did not consider the implications of this decision. > But for the time being, let’s let engineers do their job? They did their job, which resulted in the release of this system destabilizing product. Perhaps the job they should have done was to consider their work product be one which did not assume complete control of the machine onto which it runs?
      - compuguy 7y ago
        Ok, but with newer macOS releases, SIP is enabled. I'm assuming the Google developers working on this are doing their developer work on newer SIP enabled releases....
wtfrmyinitials 7y ago
Why did your team deem it appropriate to mess with core system components like /var?
- justinclift 7y ago
  What's the bet Google disclaim any and all liability for this? eg the time taken to fix this, loss of income, etc. Seems an awful lot of work related computers (eg Avid systems, and more) have been rendered inoperatable until someone manually boots and fixes each one.
- savoytruffle 7y ago
  Especially since the OS will prevent the attempt from succeeding on most Mac installations. Presumably it is a sloppy mistake, but one in an attempt to do … something … that is probably nefarious.
  - ancarda 7y ago
    I suppose that's how it happened; some code to tamper with `/var` was accidentally (most likely - I doubt this was intentional/malicious) added into the update script. When this was tested and run through QA, everything looked OK because everyone is running Mac OS with SIP enabled Makes me wonder if other software might be attempting to damage the system (totally by mistake) but SIP is preventing it, making it quite deadly to use said good software if you happen to turn off SIP for stuff like debugging
Scapeghost 7y ago
@norberg or any other Google Chrome/Keystone engineers: WHY can you not make Chrome update like every other sane, well-behaved app? Update notification -> User confirmation (or an OPTION for auto-updating) -> Download status. Why do you insist on installing things into our startup sequence without our permission? If your intent is to "protect" users, increase the nagging. I'd be fine with Chrome refusing to load any website until Chrome is updated to the latest version. Even Apple, who is notorious for making users' decisions for them, lets us choose when to update apps and operating systems.
- apostacy 7y ago
  Obviously they could have a consensual and transparent updating mechanism. This was not some sort of oversight. Google's software is a cascade of lies and deceptions. Think about this: when you start to tamper with Keystone agent, it never says anything to you, it just silently reinstalls itself somewhere else like spyware. It will keep asking over and over for root access, without explaining why. They make it seem like your installation is incomplete without root access, but that is a lie. It will function fine running out of ~/Library/ as /Library. But there is no way to make it stop asking. Google Earth, Google Drive, or many other Google products will re-install Keystone agent. If I try deleting it, then that means I probably want it gone. They should prompt me to repair it or leave it alone. You would think that Google would want to show off their updater. Even just a growl notification that an update has occurred. But it makes sense why they don't want users thinking about it. If they were more transparent, they would say: We've installed this software that will monitor your filesystem and make irreversible changes whenever we feel like it. Sometimes we will break things, but most of the time we won't and if we do break something, we will fix it. It is possible to disable, but you will have to search for it, because you will never discover it yourself. Oh, we could just have a checkbox in Preferences, but we want to make you work for it. And all you are doing is requesting that we stop updating, but we'll still be running.
  - pwr-n 7y ago
    Most accurate summation of Google I've seen. It boggles the mind how many users on HN defend google out of some sort of Stockholm Syndrome.
AdieuToLogic 7y ago
> Hey. Google Keystone tech lead here. We are aware of the issue, and we've stopped the release. There is no legitimate reason for any install other than an OS upgrade to modify /var or any other system-related directory. Ever. > We're building a replacement that fixes the problem. The fact that your team would allow any code which modifies a machine at the OS-level only reifies the concerns regarding Google's products.
jbverschoor 7y ago
2 days ago keystone and the updater was pumping 100% cpu Killing it resulted in a relaunch and 100% cpu. There is no way to stop this except for unloading the launch agent, AND launchdaemon. Removing the application and killing the instance. The os platform providers updates.. use that instead of crafting your own malware. How would you like it if your car suddenly has a top speed of 15mph, and no power steering, because someone wanted to update the number of radio presets.
- shantly 7y ago
  Huh. My wife uses Chrome (won't switch to Safari, even as she constantly complains about her battery life—go figure) and the last couple days she'd been saying that her battery life on her Macbook Air had suddenly dropped to like 25% of what it had been, leaving her seeking wall power every hour or so. Wonder if it was that. Of course then it stopped booting at all yesterday so if it was that then it must have pushed the 4.5yr old battery over the edge and killed it. Or overheated something until it died. I don't think those fans have ever been cleaned.
Doctor_Fegg 7y ago
You're missing the word "sorry" from your response. My wife's a primary school headteacher (or K-12 as you say in the States). Her MacBook was disabled by this. Yes, she takes weekly backups, but schools don't have free money to spend on spare laptops for a few days' work, nor on unnecessary technician time to fix it. Fortunately I spotted this posting (thanks, HN poster!) on blearily checking HN this morning and instantly recognised this was what's happening. Have some decency for the people whose lives you've just affected and apologise to them.
- IshKebab 7y ago
  To be fair why does she have SIP disabled?
  - Doctor_Fegg 7y ago
    Pre-SIP OS (10.10).
    - compuguy 7y ago
      Ok, why hasn't she updated to a supported macOS version then? Support ended for 10.10 in August 2017....
- Already__Taken 7y ago
  You're getting immediate tech support about a very specific issue in one for the first places you'd look. Don't be a dickhead. Do you want only PR people on HN trying to talk to you? Because this is how that happens.
  - Doctor_Fegg 7y ago
    I'm addressing Google corporately. I presume @norberg is posting on behalf of his employers given that he states his job title immediately. One of the first places "I'd" look? It's not my Mac. I'm not sure how many primary headteachers read Hacker News or have a spouse who does. I'm guessing <1%. When the world's biggest software company actually bricks people's Macs with a software update, then "sorry" is the least I expect, frankly. But if you want to dismiss this with "dickhead", you do you.
NietTim 7y ago
Why did this happen in the first place? Why are you modifying system directories to the point where you can make an oopsie and brick entire machines? In what world is this okay?
StreamBright 7y ago
My mom says she does not use sudo, please advise.
ecf 7y ago
Can we get a straight answer why these files are being changed in the first place?
protomyth 7y ago
Why does Keystone exist? Everyone else can do updates without having a launch agent, so why does Google insist on doing it this way? Given it deleted such a vital link, security looks to be compromised with this method.
techslave 7y ago
why is this a keystone bug? (besides that you shouldn’t be touching /var. WTF dude) i’m having a hard time understanding why this isn’t a mac bug. trivial kernel panic.
techslave 7y ago
“allowing it”. lol

dreamcompiler 7y ago

After that, you can also do what I do to prevent Google from reinstalling Keystone ever again: touch ~/Library/Google/GoogleSoftwareUpdate touch /Library/Google/GoogleSoftwareUpdate chmod 000 ~/Library/Google/GoogleSoftwareUpdate chmod 000 /Library/Google/GoogleSoftwareUpdate

Shank 7y ago
Honestly, if you're going to go this far, why not switch to Firefox or another Chromium/Blink-based browser, like Brave? It seems kind of counter productive to kill off the auto update system when you can just as easily switch to a browser that just doesn't do what Keystone does.
- amiga-workbench 7y ago
  Its a bit like throwing the baby out with the bathwater isn't it? Chrome is a fine browser, botnet "features" aside.
  - dewey 7y ago
    It’s so easy to switch browsers so why even bother trying to fix some big ad company’s browser that is not acting in your interest?
- dreamcompiler 7y ago
  Yes indeed but FF bogs down quicker than Chrome with lots of tabs, so I use both. More important, I like Google Earth and it tries to install Keystone too.
  - cbsmith 7y ago
    The next release of FF looks to be a game changer in that regard.
    - wilkystyle 7y ago
      It is much better (using v70 beta 8), but still has areas where performance lags behind Chrome. On a large board in https://miro.com/ https://miro.com/, for example, Firefox is laggy and jittery, whereas Chrome is buttery smooth.
      - yoasif_ 7y ago
        You can report a performance problem -- I have had good luck with fixes: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Reporting_a_Performance_Problem https://developer.mozilla.org/en-US/docs/Mozilla/Performance...
    - nkozyra 7y ago
      Unfortunately I think we've heard this so many separate times that it's beginning to be the boy who cried wolf. I've heard "Firefox is better than it was" only for me to reinstall the latest and find it's still way cludgier than chrome.
      - cbsmith 7y ago
        Sure. I hear that, but there have been some specific MacOS issues that have lead to it performing worse on MacOS than on other platforms, and they seem to be getting addressed in the Nightly builds. In general, I've found it to be much better than Chrome, but as always YMMV.
      - 7y ago
jacobsenscott 7y ago
I certainly understand the desire to rage kill google software update because they messed up, but people shouldn't actually do this because they'll be vulnerable to all future malware that targets chrome. And this varsectomany bug will never happen again.
- dreamcompiler 7y ago
  This is not rage-killing. I've been doing this for several years because Keystone is a ridiculous resource hog and I fundamentally disagree with the notion that any software should be allowed to run (much less change the configuration of) my machine without my explicit permission. I'm willing to stay on top of the malware situation and update Chrome manually. I wish I didn't have to, but Google leaves me no other option.
- robin_reala 7y ago
  Only if you continue using Chrome.
- prepend 7y ago
  I think the probability of Google freaking out and pushing ads to my system is higher than the probability of me a chrome zero day that I give a shit. They are both very low, but I’d rather programs not change my stuff against my will, even if they are trying to protect me.
- Gibbon1 7y ago
  Apple needs to provide the user with the ability to ban software from google and other malign companies. Problem partly solved.
Scapeghost 7y ago
I would change that to just /Library/Google/ to prevent Google from putting ANYthing outside its .app bundle or the normal user preferences folders.
- saagarjha 7y ago
  TIL that Google puts a "brand" code in that folder to identify how you downloaded Chrome.
apostacy 7y ago
I usually do chflags schg instead of chmod 000. I know it might seem like overkill, but Google is very sneaky, and I would not put it past Keystone to just change the permissions for itself.
- dreamcompiler 7y ago
  That's very hardcore, but I agree with your logic.
- masklinn 7y ago
  > I would not put it past Keystone to just change the permissions for itself. From experience, they absolutely do it.
  - apostacy 7y ago
    They would call it "repairing permissions". OK, so I didn't break into someone's house, I just "repaired" their door that had locked me out.
- Scapeghost 7y ago
  > I usually do chflags schg instead of chmod 000. Thank you. This was the best tip to come out of this whole discussion. I'll identify every location Google apps write to, and lock them out with this.
Tinfoilhat666 7y ago
Thank you! Because of Keystone, I have decided to treat Google Chrome as malware. I won't install it unless I really have to. One reason is that I have to test websites on Chrome. I can either run it on a virtual machine or disable the updater as you suggest.

norberg 7y ago

We have a revised set of commands that fix the symlink more correctly. These can only be run from macOS Recovery Console: rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle ln -shf /private/var /var chflags -h restricted /var chflags -h hidden /var xattr -sw com.apple.rootless "" /var

AdieuToLogic 7y ago
> We have a revised set of commands that fix the symlink more correctly. Tell your team and your supervisors this: My computer is not your playground.
- cududa 7y ago
  I’m sure most engineers on the team feel awful. They’re clearly trying, and maybe in a day or so we should figure out the nags ember breakdown. But for the time being, let’s let engineers do their job?
  - AdieuToLogic 7y ago
    > I’m sure most engineers on the team feel awful. There is no legitimate reason for a user-space install to manipulate system directories. So for an install to do so, there must have been an conscious decision made and code written to make real. Therefore, for this system manipulation to have both been introduced and released, "most engineers on the team" either raised no problems with it or did not consider the implications of this decision. > But for the time being, let’s let engineers do their job? They did their job, which resulted in the release of this system destabilizing product. Perhaps the job they should have done was to consider their work product be one which did not assume complete control of the machine onto which it runs?
    - compuguy 7y ago
      Ok, but with newer macOS releases, SIP is enabled. I'm assuming the Google developers working on this are doing their developer work on newer SIP enabled releases....

wtfrmyinitials 7y ago

Why did your team deem it appropriate to mess with core system components like /var?

justinclift 7y ago
What's the bet Google disclaim any and all liability for this? eg the time taken to fix this, loss of income, etc. Seems an awful lot of work related computers (eg Avid systems, and more) have been rendered inoperatable until someone manually boots and fixes each one.
savoytruffle 7y ago
Especially since the OS will prevent the attempt from succeeding on most Mac installations. Presumably it is a sloppy mistake, but one in an attempt to do … something … that is probably nefarious.
- ancarda 7y ago
  I suppose that's how it happened; some code to tamper with `/var` was accidentally (most likely - I doubt this was intentional/malicious) added into the update script. When this was tested and run through QA, everything looked OK because everyone is running Mac OS with SIP enabled Makes me wonder if other software might be attempting to damage the system (totally by mistake) but SIP is preventing it, making it quite deadly to use said good software if you happen to turn off SIP for stuff like debugging

Scapeghost 7y ago

@norberg or any other Google Chrome/Keystone engineers: WHY can you not make Chrome update like every other sane, well-behaved app? Update notification -> User confirmation (or an OPTION for auto-updating) -> Download status. Why do you insist on installing things into our startup sequence without our permission? If your intent is to "protect" users, increase the nagging. I'd be fine with Chrome refusing to load any website until Chrome is updated to the latest version. Even Apple, who is notorious for making users' decisions for them, lets us choose when to update apps and operating systems.

apostacy 7y ago
Obviously they could have a consensual and transparent updating mechanism. This was not some sort of oversight. Google's software is a cascade of lies and deceptions. Think about this: when you start to tamper with Keystone agent, it never says anything to you, it just silently reinstalls itself somewhere else like spyware. It will keep asking over and over for root access, without explaining why. They make it seem like your installation is incomplete without root access, but that is a lie. It will function fine running out of ~/Library/ as /Library. But there is no way to make it stop asking. Google Earth, Google Drive, or many other Google products will re-install Keystone agent. If I try deleting it, then that means I probably want it gone. They should prompt me to repair it or leave it alone. You would think that Google would want to show off their updater. Even just a growl notification that an update has occurred. But it makes sense why they don't want users thinking about it. If they were more transparent, they would say: We've installed this software that will monitor your filesystem and make irreversible changes whenever we feel like it. Sometimes we will break things, but most of the time we won't and if we do break something, we will fix it. It is possible to disable, but you will have to search for it, because you will never discover it yourself. Oh, we could just have a checkbox in Preferences, but we want to make you work for it. And all you are doing is requesting that we stop updating, but we'll still be running.
- pwr-n 7y ago
  Most accurate summation of Google I've seen. It boggles the mind how many users on HN defend google out of some sort of Stockholm Syndrome.

AdieuToLogic 7y ago

> Hey. Google Keystone tech lead here. We are aware of the issue, and we've stopped the release. There is no legitimate reason for any install other than an OS upgrade to modify /var or any other system-related directory. Ever. > We're building a replacement that fixes the problem. The fact that your team would allow any code which modifies a machine at the OS-level only reifies the concerns regarding Google's products.

jbverschoor 7y ago

2 days ago keystone and the updater was pumping 100% cpu Killing it resulted in a relaunch and 100% cpu. There is no way to stop this except for unloading the launch agent, AND launchdaemon. Removing the application and killing the instance. The os platform providers updates.. use that instead of crafting your own malware. How would you like it if your car suddenly has a top speed of 15mph, and no power steering, because someone wanted to update the number of radio presets.

shantly 7y ago
Huh. My wife uses Chrome (won't switch to Safari, even as she constantly complains about her battery life—go figure) and the last couple days she'd been saying that her battery life on her Macbook Air had suddenly dropped to like 25% of what it had been, leaving her seeking wall power every hour or so. Wonder if it was that. Of course then it stopped booting at all yesterday so if it was that then it must have pushed the 4.5yr old battery over the edge and killed it. Or overheated something until it died. I don't think those fans have ever been cleaned.

Doctor_Fegg 7y ago

You're missing the word "sorry" from your response. My wife's a primary school headteacher (or K-12 as you say in the States). Her MacBook was disabled by this. Yes, she takes weekly backups, but schools don't have free money to spend on spare laptops for a few days' work, nor on unnecessary technician time to fix it. Fortunately I spotted this posting (thanks, HN poster!) on blearily checking HN this morning and instantly recognised this was what's happening. Have some decency for the people whose lives you've just affected and apologise to them.

IshKebab 7y ago
To be fair why does she have SIP disabled?
- Doctor_Fegg 7y ago
  Pre-SIP OS (10.10).
  - compuguy 7y ago
    Ok, why hasn't she updated to a supported macOS version then? Support ended for 10.10 in August 2017....
Already__Taken 7y ago
You're getting immediate tech support about a very specific issue in one for the first places you'd look. Don't be a dickhead. Do you want only PR people on HN trying to talk to you? Because this is how that happens.
- Doctor_Fegg 7y ago
  I'm addressing Google corporately. I presume @norberg is posting on behalf of his employers given that he states his job title immediately. One of the first places "I'd" look? It's not my Mac. I'm not sure how many primary headteachers read Hacker News or have a spouse who does. I'm guessing <1%. When the world's biggest software company actually bricks people's Macs with a software update, then "sorry" is the least I expect, frankly. But if you want to dismiss this with "dickhead", you do you.

NietTim 7y ago

Why did this happen in the first place? Why are you modifying system directories to the point where you can make an oopsie and brick entire machines? In what world is this okay?

StreamBright 7y ago

My mom says she does not use sudo, please advise.

ecf 7y ago

Can we get a straight answer why these files are being changed in the first place?

protomyth 7y ago

Why does Keystone exist? Everyone else can do updates without having a launch agent, so why does Google insist on doing it this way? Given it deleted such a vital link, security looks to be compromised with this method.

techslave 7y ago

why is this a keystone bug? (besides that you shouldn’t be touching /var. WTF dude) i’m having a hard time understanding why this isn’t a mac bug. trivial kernel panic.

techslave 7y ago

“allowing it”. lol