22 ms·

Google Chrome Keystone is modifying /var symlink on non SIP Macs causing k-panic

462 points by frandroid 7y ago

TheSwordsman 7y ago
I think this about sums it up: https://i.imgur.com/fc6nwzB.gifv https://i.imgur.com/fc6nwzB.gifv
- dang 7y ago
  Not here, please.
  - mitchtbaum 7y ago
    dang
mitchtbaum 7y ago
https://giphy.com/gifs/the-x-files-3oEdvcPNJJCBxquJz2 https://giphy.com/gifs/the-x-files-3oEdvcPNJJCBxquJz2
mitchtbaum 7y ago
https://imgur.com/jHXjUVr https://imgur.com/jHXjUVr https://californication.fandom.com/wiki/Slip_of_the_Tongue https://californication.fandom.com/wiki/Slip_of_the_Tongue
neonate 7y ago
https://web.archive.org/web/20190924204425/https://mrmacintosh.com/google-chrome-keystone-is-modifying-var-symlink-on-non-sip-macs-causing-boot-issues/ https://web.archive.org/web/20190924204425/https://mrmacinto...
dang 7y ago
Related thread: https://news.ycombinator.com/item?id=21057157 https://news.ycombinator.com/item?id=21057157
michaelt 7y ago
Reminds me of the Steam bug back in 2015 [1] where on Linux, if you tried to move where Steam stored downloaded games, it would wipe your hard drive by running "rm -rf "$STEAMROOT/"*" with $STEAMROOT being null. [1] https://github.com/valvesoftware/steam-for-linux/issues/3671#issuecomment-70021818 https://github.com/valvesoftware/steam-for-linux/issues/3671...
- outworlder 7y ago
  There was a similar issue with EVE Online deleting boot.ini https://www.eveonline.com/article/about-the-boot.ini-issue https://www.eveonline.com/article/about-the-boot.ini-issue
- olafure 7y ago
  I start bash scripts, by setting the holy trinity: set -eou That prevents end-of-the-world scenario like the one above, if the script derails.
  - saagarjha 7y ago
    Maybe also add "pipefail" to the end of that?
    - scarejunba 7y ago
      Haha right, I feel like these should always be popularized as `set -eu` and `set -o pipefail` rather than making it look like you an `set pipefail`. I wonder if that chap has been uselessly printing options at the beginning of scripts for a while now.
  - loeg 7y ago
    alias rm='rm --no-preserve-root'
- jen20 7y ago
  Of course, Linux could do what Solaris did decades ago and define the directory order in which `rm -rf /` works to start with `pwd` - and thus fail immediately. That would fix that problem completely.
parliament32 7y ago
Why do userland installers need root again?
- londons_explore 7y ago
  In the case of Chrome it's because some of the sandboxing can't be done by a regular user. Same with both Linux and windows. Bit of a design flaw with the OS - in all cases a process should be allowed to restrict itself to have fewer permissions and access to fewer API's without being root, but sadly that isn't universally the case.
  - saagarjha 7y ago
    You can't apply a sandbox profile against yourself if you're not root?
  - the8472 7y ago
    On windows and linux most installers are intended for system-wide instead of per-user installation. You can get most of the sandboxing functionality without admin/root.
  - mehrdadn 7y ago
    > Same with both Linux and windows. Could you clarify how this is the case on Windows? I thought Google Chrome installs and runs just fine without admin privileges. I'm not aware of any security downsides for doing so.
    - judge2020 7y ago
      Maybe it's possible but the current installer has windows pop up a UAC prompt before it continues.
      - mehrdadn 7y ago
        That's just because it wants to install machine-wide if possible. You can just tell it to continue without admin permissions and it tells you explicitly that it can be installed without that. Note that Windows doesn't work like Linux with setuid bits and whatnot. The permissions a file is installed with don't dictate what permissions the program that executes it has. That's entirely a function of the program's security context. Hence, for a machine-wide installation to actually make a difference security-wise, Google would actually have to install e.g. a high-privilege service that would run when you try to start Chrome. I don't think it does such a thing. So I think Windows is already designed correctly in this regard and hence I don't think this is an issue on Windows as claimed.
jontro 7y ago
Looks like the issue has been reported here: https://bugs.chromium.org/p/chromium/issues/detail?id=1007358 https://bugs.chromium.org/p/chromium/issues/detail?id=100735...
londons_explore 7y ago
What's the link with AVID?
- orf 7y ago
  Maybe AVID requires SIP to be off?
- 693471 7y ago
  There was a comment that AVID may need SIP disabled for some video cards, which left this open for Chrome to do
- msbarnett 7y ago
  It's coincidental: the first widespread reports of this were from Hollywood editing shops: https://variety.com/2019/digital/news/avid-mac-pro-corrupted-hollywood-1203347033/ https://variety.com/2019/digital/news/avid-mac-pro-corrupted.... This lead to the initial presumption that it was a being caused by a bug in AVID. The reason the AVID community popped it first seems to come down to the fact that this is their busy season, so a lot of their machines were active last night as this Keystone update was rolling out, editors for whatever reason (technical issues or superstition) reboot their workstations fairly often, and, crucially, a lot of editing workstations are using third-party GPUs that require them to disable SIP (whether this is particular to AVID or just an intrinsic property of using the Mac Pros with third-party GPUs, I don't know).
  - Mindwipe 7y ago
    > (whether this is particular to AVID or just an intrinsic property of using the Mac Pros with third-party GPUs, I don't know) Definitely more the latter.
teamspirit 7y ago
So this caused me to have to reinstall the OS yesterday. Glad to know what the issue was. And if anyone wants to know, I have to disable SIP because Apple won't let me use an eGPU on my Macbook with TB2.
- mehrdadn 7y ago
  Interesting, why can't you do that with SIP? (not a Mac user)
  - saagarjha 7y ago
    Presumably the driver is not signed with a kext developer certificate, which means SIP must be disabled for macOS to load it.
  - teamspirit 7y ago
    From what I understand, Apple removed the ability for TB2. In order to use an eGPU, some system files need to be patched [0]. 0. https://github.com/mayankk2308/purge-wrangler https://github.com/mayankk2308/purge-wrangler
    - dictum 7y ago
      Can't they be patched once, and then re-patched when system updates change make changes to system files?
      - month13 7y ago
        Could, it's just a hassle. I'm in a similar boat with a kext that enables unsupported Thunderbolt 3 docks.
        dictum 7y ago
        Yep. I make some changes that SIP would catch, but I'm mostly comfortable with the boot into Recovery -> run a script -> boot back again. It's not kext stuff, though. I might have to use PurgeWrangler for an older iMac. Apparently, can keep it mostly enabled, but you need to mind updates and be ready to recover if your modifications are invalidated https://github.com/mayankk2308/purge-wrangler/issues/2#issuecomment-379283737 https://github.com/mayankk2308/purge-wrangler/issues/2#issue...
deleted 7y ago
[deleted]
norberg 7y ago
Hey. Google Keystone tech lead here. We are aware of the issue, and we've stopped the release. We're building a replacement that fixes the problem. In the meantime, to fix affected machines: sudo rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle sudo ln -F /private/var /var This deletes the affected version of Keystone and reinstates the damaged /var symlink. The version of Keystone packaged with Chrome is not affected by this bug, so allowing it to reinstall Keystone will not recreate the issue.
- dreamcompiler 7y ago
  After that, you can also do what I do to prevent Google from reinstalling Keystone ever again: touch ~/Library/Google/GoogleSoftwareUpdate touch /Library/Google/GoogleSoftwareUpdate chmod 000 ~/Library/Google/GoogleSoftwareUpdate chmod 000 /Library/Google/GoogleSoftwareUpdate
  - Shank 7y ago
    Honestly, if you're going to go this far, why not switch to Firefox or another Chromium/Blink-based browser, like Brave? It seems kind of counter productive to kill off the auto update system when you can just as easily switch to a browser that just doesn't do what Keystone does.
    - amiga-workbench 7y ago
      Its a bit like throwing the baby out with the bathwater isn't it? Chrome is a fine browser, botnet "features" aside.
      - dewey 7y ago
        It’s so easy to switch browsers so why even bother trying to fix some big ad company’s browser that is not acting in your interest?
    - dreamcompiler 7y ago
      Yes indeed but FF bogs down quicker than Chrome with lots of tabs, so I use both. More important, I like Google Earth and it tries to install Keystone too.
saagarjha 7y ago
I have SIP disabled and Chrome installed…will my /var be broken by sometime tomorrow when Keystone runs? Can I just disable the launch agent to fix this?
- norberg 7y ago
  We have stopped pushing the affected version of Keystone, so if your computer has not been broken yet, it won't be. If it has been broken but not rebooted: sudo rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle sudo ln /private/var /var should fix it. That deletes Keystone and fixes the symlink. If it has been rebooted, these commands at a recovery console should repair the computer. Chrome will subsequently ask for admin credentials to reinstall the updater next time you run it. This will not re-break the computer; the version of Keystone bundled with Chrome is older, and we have stopped serving the version affected by this issue.
  - im3w1l 7y ago
    Please don't run that command as stated. It deletes /var (because HN merged adjacent lines) Edit: It has since been fixed.
    - saagarjha 7y ago
      I won't, don't worry ;) That code should probably be indented two spaces, like this: sudo rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle sudo ln /private/var /var
      - norberg 7y ago
        yikes! thanks, fixed.
        akie 7y ago
        I see a pattern here...
SloopJon 7y ago
I've been wary of Gatekeeper and SIP as moving Macs towards an iOS-style walled garden, but this is a perfect case of SIP protecting the user from bad software.
- userbinator 7y ago
  On the contrary, I think that sort of protection just hides problems --- like this one. As a general rule, bugs with the highest impact are also the ones which are most likely to be fixed quickly. If you tested with SIP on, it'd try to remove /var but wouldn't succeed, and you'd think everything is OK when the application's logic is actually faulty.
  - xvector 7y ago
    I find that logic faulty. SIP is justified by these incidents. It is not the user’s job to isolate application faults. That is on the part of the app developer’s.
  - dkh 7y ago
    Shouldn’t Google also be able to run their tests against a machine with SIP disabled before deploying?
    - kevingadd 7y ago
      Yeah. Pre-SIP OSes were affected too, so they literally just did not test this on any non-SIP version of OS X... or if they did they didn't notice that it was nuking /var. Fresh mac VM wiped after every test run and no 'did we just destroy the OS' smoke test?
      - compuguy 7y ago
        To be fair, none of the Pre-SIP macOS releases are supported by Apple anymore...
        7y ago
Scapeghost 7y ago
I've always hated that "service" (more like malware given this news) like everything else that installs itself into the autolaunch sequence without permission, and remove* it whenever I notice/remember it, but it keeps coming back whenever I touch Google Chrome, which I prefer not to use in favor of Safari/FireFox because of reasons like this. Things like these (including secretly signing you into Search when you sign into YouTube† or refusing to support PiP on iPadOS/macOS) just solidify Google's image in my mind as a forever scummy, intrusive company that I wish I could leave behind like I did Microsoft, but sadly Google Search and YouTube still don't have good enough alternatives yet. * (startup items usually reside in the LaunchAgents/ and LaunchDaemons/ folders in your user ~/Library/, the root /Library/ and /System/Library/) † (you can fix this by deleting all Google cookies after signing into YouTube, on any OS)
- sneak 7y ago
  Yeah, the Google autoupdater’s insistence on reinstalling itself to maintain RCE on my machine even after being explicitly removed is really irksome.
  - techslave 7y ago
    interesting that apple brought the hammer out for zoom but doesn’t do that for chrome
- toyg 7y ago
  Yeah, I removed Chrome precisely because it kept re-adding its crap to the login sequence. It really drove home the notion that Google will not respect any boundary or privacy - your machine is their machine, your data is their data, and screw you if you don't agree. Now I live in Firefox and it's just as good as Chrome, at least for my needs. I've dropped pretty much all Google stuff except for GMail, mostly out of laziness (I would have to update hundreds of accounts).
  - w0utert 7y ago
    >> I've dropped pretty much all Google stuff except for GMail, mostly out of laziness (I would have to update hundreds of accounts). I held off for two years moving away from GMail for this reason. A year ago I decided to pull the plug anyway, and it turned out to be much less annoying than expected. My strategy was as follows: first enable a forward from GMail to your new mail address, then directly migrate the ~10 vital/daily accounts, then just leave the rest pointing to the GMail account. After that, change each remaining account immediately (no exceptions) the moment I either log in to it, or receive an e-mail from it that refers to the GMail address. It took me about two months migrating away from GMail, and as a bonus I was able to identify quite a few old login I didn't really have a use for anymore, so I closed them. For regular mail I put an auto-reply in GMail that says I don't use it anymore and the address will be closed at some point in the future. But honestly, I don't think anyone ever saw it as nobody sends regular email anymore these days. All in all the process was pretty painless, and I feel very happy about ditching the last Google service I was still using (except the rare Google query of DDG fails to return useful results)
shortformblog 7y ago
This is also affecting Hackintosh users. A fix for them is listed here: https://www.reddit.com/r/hackintosh/comments/d8tm8z/psa_google_chrome_updaterkeystone_rendering/ https://www.reddit.com/r/hackintosh/comments/d8tm8z/psa_goog... I wonder what role Apple’s aversion to working with Nvidia played in all those Avid users having SIP turned off.
- presidentscroob 7y ago
  Hackintosh or not isn't relevant because it's a macOS + Google update service issue. PS: Written on a Hackintosh
lostmsu 7y ago
Can somebody explain in simple terms what Keystone is, what was it trying to do, which caused /var unlinking, and why does it cause MacOS to panic?
- yjftsjthsd-h 7y ago
  Super short version: Keystone appears to be the Google auto updater service. It has a bug that causes it to unlink /var, and since that's sort of an important piece of the OS, its absence breaks stuff.
- dreamcompiler 7y ago
  Keystone is Google's auto-updater program. It updates not only Chrome but also Earth and other Google programs. It's a notorious resource hog and it tries very hard not to let you ever turn it off. If you manage to uninstall it, it will try even harder to reinstall itself the next time you run a Google app. Keystone is malware made by Google. The incident this week was the first time it contained an actual destructive payload, but it's been malware for years.
  - kovrik 7y ago
    Where can I find a manual on how to uninstall Keystone, Chrome etc. on Mac properly? Has anyone done it?
olliej 7y ago
Ok, for those who don't use Chrome (unless absolutely necessary): * what the heck is keystone? * why is it running any time Chrome isn't? * why is a browser installing a root service? * why is a piece of software changing root level symlinks in the first place? Clearly it doesn't need to because SIP prevents that nonsense * Finally: is this enough to explain why SIP/rootless is a good feature?
- saagarjha 7y ago
  > what the heck is keystone? Keystone is Google's updater service for their software. > why is it running any time Chrome isn't? It runs updates in the background, so it needs to run when Chrome doesn't. > why is a browser installing a root service? ¯\_(ツ)_/¯ > why is a piece of software changing root level symlinks in the first place? Clearly it doesn't need to because SIP prevents that nonsense Probably a bug. > Finally: is this enough to explain why SIP/rootless is a good feature? Well, a number of people decided that SIP was hindering them enough to turn it off, so I'm not sure…
  - olliej 7y ago
    > > why is it running any time Chrome isn't? > It runs updates in the background, so it needs to run when Chrome doesn't. Gnah > > why is a browser installing a root service? > ¯\_(ツ)_/¯ Ok so it needs to replace the bundle - I feel Apple should add support for replacing binary A with binary B if A and B has the same signing key, although obviously there are a bunch of fun issues involved, I think that case shouldn't necessitate an update service running as root :-/ > > why is a piece of software changing root level symlinks in the first place? Clearly it doesn't need to because SIP prevents that nonsense > Probably a bug. Wah wah > > Finally: is this enough to explain why SIP/rootless is a good feature? > Well, a number of people decided that SIP was hindering them enough to turn it off, so I'm not sure… The general problem is that it's still easier for developers to say "disable SIP by doing ..." without saying "we haven't written our [drivers/application/whatever] properly", rather than just writing the software properly. Which you know is possible because even in kernel driver land you hardly ever see driver's claiming that it's necessary. e.g. its necessary from an end-user PoV but only because companies don't want to pay devs to put effort into working with SIP enabled when there's a much cheaper "tell the user to disable security" option available.
fortran77 7y ago
Why does the Mac OS allow this?
- kyralis 7y ago
  Modern MacOS in default configuration does not.
  - fortran77 7y ago
    So did all these Hollywood people change the "default configuration" or did the standard AVID installer disable it?
    - gumby 7y ago
      Some people wanted to use video cards with unsigned drivers. The hardware Mfr said “disable this malware blocking security feature in order to use our hardware”. As it happens most people with exotic video cards are avid users. They disabled the malware protection and got killed by the malware.
      - fortran77 7y ago
        On other platforms, those cards aren't "Exotic" (I run Premiere Pro on Windows 10 with a pair of 2080Ti cards for rendering. Premiere and After Effcts _fly_! One problem is that Apple abandoned the Pro market, but some users are very loyal.
        gumby 7y ago
        They're exotic in that their manufacturers can't be bothered signing their drivers. But at least there are drivers. They're also exotic in the sense that only a very small proportion of the overall user base cares/requires them. There are supported video cards / TB3 video systems that are natively supported but at the moment they are possibly even more exotic in the sense of shipping in low volume (per your second point). I really don't like "apple abandoned the XXX market, which I am in and wish they had just the right product for ME" statements but in this case, I think your comment is unfortunately correct.
pier25 7y ago
Does Chromium also install keystone?
jbverschoor 7y ago
Google, please tell me how to update Chrome without keystone. Found non-functional system update engine. Please reinstall Google Software Update from https://dl.google.com/mac/install/googlesoftwareupdate.dmg https://dl.google.com/mac/install/googlesoftwareupdate.dmg KSUpdateEngine no ticket to update for the specified product ID. I don't want a "System update engine"... This is baked into Apple's AppStore. It works very well. Use that. You don't need access to my system.
- masklinn 7y ago
  > Google, please tell me how to update Chrome without keystone. That's easy, just regularly download a new Chrome. The difficulty is managing to stop keystone from reinstalling and re-enabling itself. > This is baked into Apple's AppStore. It works very well. Use that. I hate keystone with a passion, but TBF getting a modern browser into the appstore is not possible, even ignoring all the limitations the store puts upon its software, there's no way you can actually get a browser (as opposed to a shell UI around the platform webkit) in the appstore by its rules.
jbverschoor 7y ago
Does chromium suffer the same malware?
jbverschoor 7y ago
This is the same shit Adobe is doing.
forgotmypw3 7y ago
Chrome hasn't been present on my desktops for a couple of years, and I don't miss it.
mlang23 7y ago
To sum up most of the comments here: Google is the new Microsoft. Fascinating, how a company can go from "we will not do evil" to "fuck you all" in just 5 years.
- shantly 7y ago
  I thought they started to get kinda crappy back in '08 or '09, it just took until recently for the consensus to catch up.
  - jm4 7y ago
    Agreed. That's around the time I started using some weird setup where my google searches would go through Tor and I was blocking their cookies. It was overboard, probably unnecessary and probably didn't accomplish much, but it was the result of getting a creepy vibe from google. Once I discovered DDG, I switched and never looked back. It took a little while to get used to it, but the results are good enough that I keep using it. I run maybe a few google searches a year if I can't find what I need on DDG and I usually don't find it there either. I don't think google is significantly better, although it is noticeably faster. There's nothing about it that's appealing enough that I want to accept the bad things that also come with it.
goatinaboat 7y ago
Anything Google is the very definition of malware: if you install it on your computer it’s not your computer anymore.
J5892 7y ago
Holy crap, that's what's happening? It took me hours to figure out that the failed boot was caused by the `/var` symlink being removed. I was literally a minute away from reinstalling the OS when I saw a post from 2014 that had a passing reference to the `/var` symlink. Then I went through and disabled every conceivable startup program, and even created a bash script to fix `/var` when it randomly disappeared. I didn't even consider that it could be Chrome causing it. This all happened with SIP enabled, btw.
- kovrik 7y ago
  How come SIP didn't catch it? Is Chrome asking for a root access during installation?
  - J5892 7y ago
    I have no idea. After I fixed the issue, I even tried disabling SIP, then enabling it again. It's still randomly removing the symlink. (well, as of last night around 10pm. I haven't opened the computer since then) That computer has been through a lot, though. So it's totally possible that I did something stupid to permafuck SIP.
    - NobodyNada 7y ago
      When you recreated the symlink, did you add the SIP flags to it? https://news.ycombinator.com/item?id=21066472 https://news.ycombinator.com/item?id=21066472
kzrdude 7y ago
Switch to Linux or stop using Chrome, unfortunately.
Animats 7y ago
The "fix" looks more like a virus removal job. It includes rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle and doesn't replace it with anything.
rgovostes 7y ago
There are a few legitimate reasons to disable SIP, but too often I see people turning it entirely off, rather than just disabling the parts that are in the way: csrutil enable --without kext csrutil enable --without fs csrutil enable --without debug csrutil enable --without dtrace csrutil enable --without nvram If you want to load some untrusted kernel extension, the first one will let you do so, but still keep all the other SIP protections on. If you want to use DTrace, use the corresponding flag. Etc. You can mix and match flags.
- stunt 7y ago
  Thanks for sharing. Is it documented anywhere? man page didn't help.
  - rgovostes 7y ago
    I do not think that they are documented officially. The only "supported" configurations are all on/all off.
1GR3 7y ago
Since this is a thread where men can talk about their feelings, I'd like to say that I feel angry and frustrated! If I wanted a machine for browsing web and watching youtube videos while always been safely signed into my google account, I'd bought a f@©4!n9 Chromebook.
will_hoskings 7y ago
It's funny how Google Chrome even needs this in the first place, to be honest. This is another great reason to move to Firefox, with the rest of us :p

462 points by frandroid 7y ago

TheSwordsman 7y ago

I think this about sums it up: https://i.imgur.com/fc6nwzB.gifv https://i.imgur.com/fc6nwzB.gifv

dang 7y ago
Not here, please.
- mitchtbaum 7y ago
  dang

mitchtbaum 7y ago

https://giphy.com/gifs/the-x-files-3oEdvcPNJJCBxquJz2 https://giphy.com/gifs/the-x-files-3oEdvcPNJJCBxquJz2

mitchtbaum 7y ago

https://imgur.com/jHXjUVr https://imgur.com/jHXjUVr https://californication.fandom.com/wiki/Slip_of_the_Tongue https://californication.fandom.com/wiki/Slip_of_the_Tongue

neonate 7y ago

https://web.archive.org/web/20190924204425/https://mrmacintosh.com/google-chrome-keystone-is-modifying-var-symlink-on-non-sip-macs-causing-boot-issues/ https://web.archive.org/web/20190924204425/https://mrmacinto...

dang 7y ago

Related thread: https://news.ycombinator.com/item?id=21057157 https://news.ycombinator.com/item?id=21057157

michaelt 7y ago

Reminds me of the Steam bug back in 2015 [1] where on Linux, if you tried to move where Steam stored downloaded games, it would wipe your hard drive by running "rm -rf "$STEAMROOT/"*" with $STEAMROOT being null. [1] https://github.com/valvesoftware/steam-for-linux/issues/3671#issuecomment-70021818 https://github.com/valvesoftware/steam-for-linux/issues/3671...

outworlder 7y ago
There was a similar issue with EVE Online deleting boot.ini https://www.eveonline.com/article/about-the-boot.ini-issue https://www.eveonline.com/article/about-the-boot.ini-issue
olafure 7y ago
I start bash scripts, by setting the holy trinity: set -eou That prevents end-of-the-world scenario like the one above, if the script derails.
- saagarjha 7y ago
  Maybe also add "pipefail" to the end of that?
  - scarejunba 7y ago
    Haha right, I feel like these should always be popularized as `set -eu` and `set -o pipefail` rather than making it look like you an `set pipefail`. I wonder if that chap has been uselessly printing options at the beginning of scripts for a while now.
- loeg 7y ago
  alias rm='rm --no-preserve-root'
jen20 7y ago
Of course, Linux could do what Solaris did decades ago and define the directory order in which `rm -rf /` works to start with `pwd` - and thus fail immediately. That would fix that problem completely.

parliament32 7y ago

Why do userland installers need root again?

londons_explore 7y ago
In the case of Chrome it's because some of the sandboxing can't be done by a regular user. Same with both Linux and windows. Bit of a design flaw with the OS - in all cases a process should be allowed to restrict itself to have fewer permissions and access to fewer API's without being root, but sadly that isn't universally the case.
- saagarjha 7y ago
  You can't apply a sandbox profile against yourself if you're not root?
- the8472 7y ago
  On windows and linux most installers are intended for system-wide instead of per-user installation. You can get most of the sandboxing functionality without admin/root.
- mehrdadn 7y ago
  > Same with both Linux and windows. Could you clarify how this is the case on Windows? I thought Google Chrome installs and runs just fine without admin privileges. I'm not aware of any security downsides for doing so.
  - judge2020 7y ago
    Maybe it's possible but the current installer has windows pop up a UAC prompt before it continues.
    - mehrdadn 7y ago
      That's just because it wants to install machine-wide if possible. You can just tell it to continue without admin permissions and it tells you explicitly that it can be installed without that. Note that Windows doesn't work like Linux with setuid bits and whatnot. The permissions a file is installed with don't dictate what permissions the program that executes it has. That's entirely a function of the program's security context. Hence, for a machine-wide installation to actually make a difference security-wise, Google would actually have to install e.g. a high-privilege service that would run when you try to start Chrome. I don't think it does such a thing. So I think Windows is already designed correctly in this regard and hence I don't think this is an issue on Windows as claimed.

jontro 7y ago

Looks like the issue has been reported here: https://bugs.chromium.org/p/chromium/issues/detail?id=1007358 https://bugs.chromium.org/p/chromium/issues/detail?id=100735...

londons_explore 7y ago

What's the link with AVID?

orf 7y ago
Maybe AVID requires SIP to be off?
693471 7y ago
There was a comment that AVID may need SIP disabled for some video cards, which left this open for Chrome to do
msbarnett 7y ago
It's coincidental: the first widespread reports of this were from Hollywood editing shops: https://variety.com/2019/digital/news/avid-mac-pro-corrupted-hollywood-1203347033/ https://variety.com/2019/digital/news/avid-mac-pro-corrupted.... This lead to the initial presumption that it was a being caused by a bug in AVID. The reason the AVID community popped it first seems to come down to the fact that this is their busy season, so a lot of their machines were active last night as this Keystone update was rolling out, editors for whatever reason (technical issues or superstition) reboot their workstations fairly often, and, crucially, a lot of editing workstations are using third-party GPUs that require them to disable SIP (whether this is particular to AVID or just an intrinsic property of using the Mac Pros with third-party GPUs, I don't know).
- Mindwipe 7y ago
  > (whether this is particular to AVID or just an intrinsic property of using the Mac Pros with third-party GPUs, I don't know) Definitely more the latter.

teamspirit 7y ago

So this caused me to have to reinstall the OS yesterday. Glad to know what the issue was. And if anyone wants to know, I have to disable SIP because Apple won't let me use an eGPU on my Macbook with TB2.

mehrdadn 7y ago
Interesting, why can't you do that with SIP? (not a Mac user)
- saagarjha 7y ago
  Presumably the driver is not signed with a kext developer certificate, which means SIP must be disabled for macOS to load it.
- teamspirit 7y ago
  From what I understand, Apple removed the ability for TB2. In order to use an eGPU, some system files need to be patched [0]. 0. https://github.com/mayankk2308/purge-wrangler https://github.com/mayankk2308/purge-wrangler
  - dictum 7y ago
    Can't they be patched once, and then re-patched when system updates change make changes to system files?
    - month13 7y ago
      Could, it's just a hassle. I'm in a similar boat with a kext that enables unsupported Thunderbolt 3 docks.
      - dictum 7y ago
        Yep. I make some changes that SIP would catch, but I'm mostly comfortable with the boot into Recovery -> run a script -> boot back again. It's not kext stuff, though. I might have to use PurgeWrangler for an older iMac. Apparently, can keep it mostly enabled, but you need to mind updates and be ready to recover if your modifications are invalidated https://github.com/mayankk2308/purge-wrangler/issues/2#issuecomment-379283737 https://github.com/mayankk2308/purge-wrangler/issues/2#issue...

deleted 7y ago

[deleted]

norberg 7y ago

Hey. Google Keystone tech lead here. We are aware of the issue, and we've stopped the release. We're building a replacement that fixes the problem. In the meantime, to fix affected machines: sudo rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle sudo ln -F /private/var /var This deletes the affected version of Keystone and reinstates the damaged /var symlink. The version of Keystone packaged with Chrome is not affected by this bug, so allowing it to reinstall Keystone will not recreate the issue.

dreamcompiler 7y ago
After that, you can also do what I do to prevent Google from reinstalling Keystone ever again: touch ~/Library/Google/GoogleSoftwareUpdate touch /Library/Google/GoogleSoftwareUpdate chmod 000 ~/Library/Google/GoogleSoftwareUpdate chmod 000 /Library/Google/GoogleSoftwareUpdate
- Shank 7y ago
  Honestly, if you're going to go this far, why not switch to Firefox or another Chromium/Blink-based browser, like Brave? It seems kind of counter productive to kill off the auto update system when you can just as easily switch to a browser that just doesn't do what Keystone does.
  - amiga-workbench 7y ago
    Its a bit like throwing the baby out with the bathwater isn't it? Chrome is a fine browser, botnet "features" aside.
    - dewey 7y ago
      It’s so easy to switch browsers so why even bother trying to fix some big ad company’s browser that is not acting in your interest?
  - dreamcompiler 7y ago
    Yes indeed but FF bogs down quicker than Chrome with lots of tabs, so I use both. More important, I like Google Earth and it tries to install Keystone too.

saagarjha 7y ago

I have SIP disabled and Chrome installed…will my /var be broken by sometime tomorrow when Keystone runs? Can I just disable the launch agent to fix this?

norberg 7y ago
We have stopped pushing the affected version of Keystone, so if your computer has not been broken yet, it won't be. If it has been broken but not rebooted: sudo rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle sudo ln /private/var /var should fix it. That deletes Keystone and fixes the symlink. If it has been rebooted, these commands at a recovery console should repair the computer. Chrome will subsequently ask for admin credentials to reinstall the updater next time you run it. This will not re-break the computer; the version of Keystone bundled with Chrome is older, and we have stopped serving the version affected by this issue.
- im3w1l 7y ago
  Please don't run that command as stated. It deletes /var (because HN merged adjacent lines) Edit: It has since been fixed.
  - saagarjha 7y ago
    I won't, don't worry ;) That code should probably be indented two spaces, like this: sudo rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle sudo ln /private/var /var
    - norberg 7y ago
      yikes! thanks, fixed.
      - akie 7y ago
        I see a pattern here...

SloopJon 7y ago

I've been wary of Gatekeeper and SIP as moving Macs towards an iOS-style walled garden, but this is a perfect case of SIP protecting the user from bad software.

userbinator 7y ago
On the contrary, I think that sort of protection just hides problems --- like this one. As a general rule, bugs with the highest impact are also the ones which are most likely to be fixed quickly. If you tested with SIP on, it'd try to remove /var but wouldn't succeed, and you'd think everything is OK when the application's logic is actually faulty.
- xvector 7y ago
  I find that logic faulty. SIP is justified by these incidents. It is not the user’s job to isolate application faults. That is on the part of the app developer’s.
- dkh 7y ago
  Shouldn’t Google also be able to run their tests against a machine with SIP disabled before deploying?
  - kevingadd 7y ago
    Yeah. Pre-SIP OSes were affected too, so they literally just did not test this on any non-SIP version of OS X... or if they did they didn't notice that it was nuking /var. Fresh mac VM wiped after every test run and no 'did we just destroy the OS' smoke test?
    - compuguy 7y ago
      To be fair, none of the Pre-SIP macOS releases are supported by Apple anymore...
      - 7y ago

Scapeghost 7y ago

I've always hated that "service" (more like malware given this news) like everything else that installs itself into the autolaunch sequence without permission, and remove* it whenever I notice/remember it, but it keeps coming back whenever I touch Google Chrome, which I prefer not to use in favor of Safari/FireFox because of reasons like this. Things like these (including secretly signing you into Search when you sign into YouTube† or refusing to support PiP on iPadOS/macOS) just solidify Google's image in my mind as a forever scummy, intrusive company that I wish I could leave behind like I did Microsoft, but sadly Google Search and YouTube still don't have good enough alternatives yet. * (startup items usually reside in the LaunchAgents/ and LaunchDaemons/ folders in your user ~/Library/, the root /Library/ and /System/Library/) † (you can fix this by deleting all Google cookies after signing into YouTube, on any OS)

sneak 7y ago
Yeah, the Google autoupdater’s insistence on reinstalling itself to maintain RCE on my machine even after being explicitly removed is really irksome.
- techslave 7y ago
  interesting that apple brought the hammer out for zoom but doesn’t do that for chrome
toyg 7y ago
Yeah, I removed Chrome precisely because it kept re-adding its crap to the login sequence. It really drove home the notion that Google will not respect any boundary or privacy - your machine is their machine, your data is their data, and screw you if you don't agree. Now I live in Firefox and it's just as good as Chrome, at least for my needs. I've dropped pretty much all Google stuff except for GMail, mostly out of laziness (I would have to update hundreds of accounts).
- w0utert 7y ago
  >> I've dropped pretty much all Google stuff except for GMail, mostly out of laziness (I would have to update hundreds of accounts). I held off for two years moving away from GMail for this reason. A year ago I decided to pull the plug anyway, and it turned out to be much less annoying than expected. My strategy was as follows: first enable a forward from GMail to your new mail address, then directly migrate the ~10 vital/daily accounts, then just leave the rest pointing to the GMail account. After that, change each remaining account immediately (no exceptions) the moment I either log in to it, or receive an e-mail from it that refers to the GMail address. It took me about two months migrating away from GMail, and as a bonus I was able to identify quite a few old login I didn't really have a use for anymore, so I closed them. For regular mail I put an auto-reply in GMail that says I don't use it anymore and the address will be closed at some point in the future. But honestly, I don't think anyone ever saw it as nobody sends regular email anymore these days. All in all the process was pretty painless, and I feel very happy about ditching the last Google service I was still using (except the rare Google query of DDG fails to return useful results)

shortformblog 7y ago

This is also affecting Hackintosh users. A fix for them is listed here: https://www.reddit.com/r/hackintosh/comments/d8tm8z/psa_google_chrome_updaterkeystone_rendering/ https://www.reddit.com/r/hackintosh/comments/d8tm8z/psa_goog... I wonder what role Apple’s aversion to working with Nvidia played in all those Avid users having SIP turned off.

presidentscroob 7y ago
Hackintosh or not isn't relevant because it's a macOS + Google update service issue. PS: Written on a Hackintosh

lostmsu 7y ago

Can somebody explain in simple terms what Keystone is, what was it trying to do, which caused /var unlinking, and why does it cause MacOS to panic?

yjftsjthsd-h 7y ago
Super short version: Keystone appears to be the Google auto updater service. It has a bug that causes it to unlink /var, and since that's sort of an important piece of the OS, its absence breaks stuff.
dreamcompiler 7y ago
Keystone is Google's auto-updater program. It updates not only Chrome but also Earth and other Google programs. It's a notorious resource hog and it tries very hard not to let you ever turn it off. If you manage to uninstall it, it will try even harder to reinstall itself the next time you run a Google app. Keystone is malware made by Google. The incident this week was the first time it contained an actual destructive payload, but it's been malware for years.
- kovrik 7y ago
  Where can I find a manual on how to uninstall Keystone, Chrome etc. on Mac properly? Has anyone done it?

olliej 7y ago

Ok, for those who don't use Chrome (unless absolutely necessary): * what the heck is keystone? * why is it running any time Chrome isn't? * why is a browser installing a root service? * why is a piece of software changing root level symlinks in the first place? Clearly it doesn't need to because SIP prevents that nonsense * Finally: is this enough to explain why SIP/rootless is a good feature?

saagarjha 7y ago
> what the heck is keystone? Keystone is Google's updater service for their software. > why is it running any time Chrome isn't? It runs updates in the background, so it needs to run when Chrome doesn't. > why is a browser installing a root service? ¯\_(ツ)_/¯ > why is a piece of software changing root level symlinks in the first place? Clearly it doesn't need to because SIP prevents that nonsense Probably a bug. > Finally: is this enough to explain why SIP/rootless is a good feature? Well, a number of people decided that SIP was hindering them enough to turn it off, so I'm not sure…
- olliej 7y ago
  > > why is it running any time Chrome isn't? > It runs updates in the background, so it needs to run when Chrome doesn't. Gnah > > why is a browser installing a root service? > ¯\_(ツ)_/¯ Ok so it needs to replace the bundle - I feel Apple should add support for replacing binary A with binary B if A and B has the same signing key, although obviously there are a bunch of fun issues involved, I think that case shouldn't necessitate an update service running as root :-/ > > why is a piece of software changing root level symlinks in the first place? Clearly it doesn't need to because SIP prevents that nonsense > Probably a bug. Wah wah > > Finally: is this enough to explain why SIP/rootless is a good feature? > Well, a number of people decided that SIP was hindering them enough to turn it off, so I'm not sure… The general problem is that it's still easier for developers to say "disable SIP by doing ..." without saying "we haven't written our [drivers/application/whatever] properly", rather than just writing the software properly. Which you know is possible because even in kernel driver land you hardly ever see driver's claiming that it's necessary. e.g. its necessary from an end-user PoV but only because companies don't want to pay devs to put effort into working with SIP enabled when there's a much cheaper "tell the user to disable security" option available.

fortran77 7y ago

Why does the Mac OS allow this?

kyralis 7y ago
Modern MacOS in default configuration does not.
- fortran77 7y ago
  So did all these Hollywood people change the "default configuration" or did the standard AVID installer disable it?
  - gumby 7y ago
    Some people wanted to use video cards with unsigned drivers. The hardware Mfr said “disable this malware blocking security feature in order to use our hardware”. As it happens most people with exotic video cards are avid users. They disabled the malware protection and got killed by the malware.
    - fortran77 7y ago
      On other platforms, those cards aren't "Exotic" (I run Premiere Pro on Windows 10 with a pair of 2080Ti cards for rendering. Premiere and After Effcts _fly_! One problem is that Apple abandoned the Pro market, but some users are very loyal.
      - gumby 7y ago
        They're exotic in that their manufacturers can't be bothered signing their drivers. But at least there are drivers. They're also exotic in the sense that only a very small proportion of the overall user base cares/requires them. There are supported video cards / TB3 video systems that are natively supported but at the moment they are possibly even more exotic in the sense of shipping in low volume (per your second point). I really don't like "apple abandoned the XXX market, which I am in and wish they had just the right product for ME" statements but in this case, I think your comment is unfortunately correct.

pier25 7y ago

Does Chromium also install keystone?

jbverschoor 7y ago

Google, please tell me how to update Chrome without keystone. Found non-functional system update engine. Please reinstall Google Software Update from https://dl.google.com/mac/install/googlesoftwareupdate.dmg https://dl.google.com/mac/install/googlesoftwareupdate.dmg KSUpdateEngine no ticket to update for the specified product ID. I don't want a "System update engine"... This is baked into Apple's AppStore. It works very well. Use that. You don't need access to my system.

masklinn 7y ago
> Google, please tell me how to update Chrome without keystone. That's easy, just regularly download a new Chrome. The difficulty is managing to stop keystone from reinstalling and re-enabling itself. > This is baked into Apple's AppStore. It works very well. Use that. I hate keystone with a passion, but TBF getting a modern browser into the appstore is not possible, even ignoring all the limitations the store puts upon its software, there's no way you can actually get a browser (as opposed to a shell UI around the platform webkit) in the appstore by its rules.

jbverschoor 7y ago

Does chromium suffer the same malware?

jbverschoor 7y ago

This is the same shit Adobe is doing.

forgotmypw3 7y ago

Chrome hasn't been present on my desktops for a couple of years, and I don't miss it.

mlang23 7y ago

To sum up most of the comments here: Google is the new Microsoft. Fascinating, how a company can go from "we will not do evil" to "fuck you all" in just 5 years.

shantly 7y ago
I thought they started to get kinda crappy back in '08 or '09, it just took until recently for the consensus to catch up.
- jm4 7y ago
  Agreed. That's around the time I started using some weird setup where my google searches would go through Tor and I was blocking their cookies. It was overboard, probably unnecessary and probably didn't accomplish much, but it was the result of getting a creepy vibe from google. Once I discovered DDG, I switched and never looked back. It took a little while to get used to it, but the results are good enough that I keep using it. I run maybe a few google searches a year if I can't find what I need on DDG and I usually don't find it there either. I don't think google is significantly better, although it is noticeably faster. There's nothing about it that's appealing enough that I want to accept the bad things that also come with it.

goatinaboat 7y ago

Anything Google is the very definition of malware: if you install it on your computer it’s not your computer anymore.

J5892 7y ago

Holy crap, that's what's happening? It took me hours to figure out that the failed boot was caused by the `/var` symlink being removed. I was literally a minute away from reinstalling the OS when I saw a post from 2014 that had a passing reference to the `/var` symlink. Then I went through and disabled every conceivable startup program, and even created a bash script to fix `/var` when it randomly disappeared. I didn't even consider that it could be Chrome causing it. This all happened with SIP enabled, btw.

kovrik 7y ago
How come SIP didn't catch it? Is Chrome asking for a root access during installation?
- J5892 7y ago
  I have no idea. After I fixed the issue, I even tried disabling SIP, then enabling it again. It's still randomly removing the symlink. (well, as of last night around 10pm. I haven't opened the computer since then) That computer has been through a lot, though. So it's totally possible that I did something stupid to permafuck SIP.
  - NobodyNada 7y ago
    When you recreated the symlink, did you add the SIP flags to it? https://news.ycombinator.com/item?id=21066472 https://news.ycombinator.com/item?id=21066472

kzrdude 7y ago

Switch to Linux or stop using Chrome, unfortunately.

Animats 7y ago

The "fix" looks more like a virus removal job. It includes rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle and doesn't replace it with anything.

rgovostes 7y ago

There are a few legitimate reasons to disable SIP, but too often I see people turning it entirely off, rather than just disabling the parts that are in the way: csrutil enable --without kext csrutil enable --without fs csrutil enable --without debug csrutil enable --without dtrace csrutil enable --without nvram If you want to load some untrusted kernel extension, the first one will let you do so, but still keep all the other SIP protections on. If you want to use DTrace, use the corresponding flag. Etc. You can mix and match flags.

stunt 7y ago
Thanks for sharing. Is it documented anywhere? man page didn't help.
- rgovostes 7y ago
  I do not think that they are documented officially. The only "supported" configurations are all on/all off.

1GR3 7y ago

Since this is a thread where men can talk about their feelings, I'd like to say that I feel angry and frustrated! If I wanted a machine for browsing web and watching youtube videos while always been safely signed into my google account, I'd bought a f@©4!n9 Chromebook.

will_hoskings 7y ago

It's funny how Google Chrome even needs this in the first place, to be honest. This is another great reason to move to Firefox, with the rest of us :p