4 ms·
People from outside Google freak out about this because at their company, in 99.9% of companies, running code on an engineer's workstation would immediately be
by jeffbee 4y ago
People from outside Google freak out about this because at their company, in 99.9% of companies, running code on an engineer's workstation would immediately be the highest possible level of breach. Said process could silently insert code into repos, corrupt the build environment, replace packages in production, or whatever. If you never worked at a company that took it seriously it is hard to imagine that there are people who do take it seriously, and that it is possible to have technical defenses against committing unreviewed and unapproved code, poisoning the official build toolchain, or surreptitiously changing production software images.
- ec109685 4y agoRunning malicious code on your workstation is a vulnerability worth fixing. Especially when it is as easy as namespacing internal packages.
- numbsafari 4y agoIf accurate, at least one of the hosts in question indicated the user was root. Perhaps it was running in a container. Regardless, this malicious package could be used for data exfiltration or other nefarious things.
- riceart 4y ago> user was root In a well designed zero trust network this makes little difference. The traditional posix security model is bogus from the start anyway. A lot more useful stuff you can exfiltrate as a regular user usually.
- eternalban 4y agoBut they have already explained (at the end) that in any event, "Googlers can download and run arbitrary code on their machines", with the implication being that Google already thought about and had to deal with this general issue a long time ago. What this exploit does is invert the how of this "arbitrary code" running on google machines, but it makes perfect sense that Google's security protections couldn't care less how the code ended up on some dev's machine since they themselves could have just explicitly pulled it from the net.
- Initial_BP 4y agoNot directly relevant but interesting... https://github.com/google/santa https://github.com/google/santa This is a product developed by Google that has at least been utilized internally to some extent. It's not perfect, but my previous company used it and it does prevent unexpected unknown code from running in the background. What it does not do is prevent someone from intentionally downloading and executing a library unless the upvoter actually comes to some demand that you do so. I found that it quickly became a bit of a "alert fatigue" where you approve things your coworkers send you so they can get back to work without properly vetting.
- saagarjha 4y agoMany developers have this set to “only block known malware” for obvious reasons.
- eternalban 4y agoThank you for sharing this! This is very cool. I'd use this on my personal laptop. It has a pure local mode.
- mjg59 4y agoIf an attacker has arbitrary code execution as a user with sudo access (ie, basically any developer system), then the user/root distinction is basically meaningless.
- smsm42 4y ago> would immediately be the highest possible level of breach Obviously not true, in fact none of the companies I worked in that was the case. Now, if we're talking about somebody like network admin that may be more juicy, but in a large company rarely a single person has the "highest possible level". > If you never worked at a company that took it seriously it is hard to imagine that there are people who do take it seriously This sounds quite unsufferably smug, and unwarrantedly so. Yes, we get it, in Google they have access levels. Newsflash: this concept wasn't invented by Google. A lot of companies use it. Running code on engineers workstation does not give you keys to the whole castle. But it does get you into a first perimeter, after which you can do some recon, not available outside the big firewall, launch other things against corporate sites not available to mere mortals, look for "experimental" servers which are poorly secured because they're not production yet and are behind corporate firewall (that happened in pretty much ever company I've worked with), get their hands on some juicy browser cookies containing authorizations to some internal services, steal some code finally... There are a lot of interesting things a person can do with engineer's access and some of them may land a stepping stone to a next step access maybe. If you don't understand why such a thing may be important even without giving somebody the keys to the castle - maybe you don't know enough about how secure systems are built to smugly dismiss other people.
- dub 4y ago> Obviously not true, in fact none of the companies I worked in that was the case I once offered a bet to the large security team at a well-known decacorn tech company I worked at: I offered to make a personal, reasonable-sized cash bet with any member of the security team that I would win if I could deploy malicious, unreviewed code to any service or machine of their choice without it being prevented or proactively noticed by them. The members of the security team all declined my bet. We're talking about a team of probably at least a dozen people, many of who had been working at the company far longer than I and who had been shaping and reviewing the company's security design for years. They knew perfectly well that I would be able to win the bet. Not because their security was unusually bad, but because it was bad in the common, usual ways. Securing the supply chain is hard, and real security is almost impossibly expensive to add to a system late in the game if you didn't design it in from the beginning.
- devinj 4y agoOP compromised more than one user's machine. So they could write the change on one machine, and send it to be reviewed and approved by another user. (I work at Google but have no special insight. My opinions are my own.)
- jonplackett 4y agoIsn’t it a different thing for a googler to run some code deliberately, presumably having some idea where it came from and what it does VS some code being run on their computer without their consent or knowledge of what it is. Seems weird to group those two as the same thing
- jeffbee 4y agoI don't fool myself into believing that when I install some package from CRAN, or whatever your least-favorite insecure software repository is, that I know exactly what it is or does. So to me it seems the same risk either way. Anyway I am only trying to address the question raised in the article about why Google security did not flip out over this report in the way the author expected them to.
- moonchrome 4y ago> Said process could silently insert code into repos, corrupt the build environment, replace packages in production, or whatever. So just like pretty much any package from public repo out there ?
- tmsbrg 4y agoFrom my experience most of these defences take the form of limiting privileges and implementing the four eye principle. Consider the fact he could take over 15 Google developers systems within a few weeks with little effort. What if there's two people who can approve each other's production deployments, etc.? Even if you only take over one developer's system, it's a great starting point for pivoting into the network and starting a more sophisticated attack. I'm sure an advanced threat actor would know how to take advantage of the opportunity against Google.
- ikiris 4y agoAdvanced threat actors would have developers working for Google. This isn't a high access bar.
- stingraycharles 4y agoAnd yet, almost always, infiltrating large organisations like this for whatever purpose (e.g. state level actors, corporate espionage, whatever) involves multiple chains of vulnerability. The type of vulnerability that the OP reported is the initial entry door, and as such, extremely valuable. Also, about “code reviews”: here’s a story from last year about how a massive refactoring of some webkit code in 2016 resurrected a massive exploit that was actually fixed in 2013, but went unnoticed. It was only discovered and patched in 2022, as it was being exploited in the wild by, among others, the NSO group. https://www.theregister.com/2022/06/21/apple-safari-zombie-exploit/ https://www.theregister.com/2022/06/21/apple-safari-zombie-e...
- btown 4y agoDoesn’t Google use a monorepo though? Even if there are guards against poisoning builds and commits, the risk of an RCE causing silent exfiltration of confidential trade secrets must be astronomical.
- saagarjha 4y agoGoogle doesn’t really seem to be able to prevent this; instead they monitor to see what ended up leaked.
- UncleMeat 4y agoIncredibly sensitive portions of google3 are siloed and only a small part of the company can access them. Tools like code search and the source checkout process also both check for accessing unusually large portions of the codebase, making it only possible to exfiltrate small portions of the codebase at once.
- deleted 4y ago[deleted]
- dontupvoteme 4y ago[flagged]
- summerlight 4y ago> Said process could silently insert code into repos, corrupt the build environment, replace packages in production, or whatever Haha I hope this to be true :sigh: The reality is that all those security hardening measures already surpass the level where it significantly undermines the overall productivity... Engineers cannot even have a test run on production data without an explicit review from colleagues.
- mjg59 4y agoThe two main protections are requiring multiple signoffs (for landing code) and requiring proof of physical presence (for various things, including any sort of access to prod). Having code exec on a large number of systems means the first is weakened significantly, and the strength of the second really depends to an extent on how much effort and patience the attacker is willing to put in (eg, alias a command that requires gnubby press to something under your control, use the press to do whatever you want, then trigger the real command - user ends up being prompted twice, but how many people would actually flag that as a security concern?). Someone with no understanding of how things work in Google would probably screw up in a way that got them noticed pretty quickly, but a well-skilled adversary who's not looking for a quick win could definitely circumvent many controls if they had code exec on a decent number of systems. tl;dr - if I were still gLinux security, I might not be freaking out about this, but it would definitely fall into the set of stuff I'd be making space for in next quarter's OKRs.
- saagarjha 4y agoThis is an ad hominem dressed up as a comment about company policy: “if only you worked at a company that didn’t suck, like I did, you would have actual security and wouldn’t file stupid bugs like these!” We both know that Google does better than most at endpoint security. In some cases it’s possible to argue that they are the best. What we definitely don’t need is you to be superior about it: it’s part of the reason why (ex-)Googlers have a poor reputation. In this case, having technical measures that avoid sensitive things ending up on developer machines is an excellent way to help improve your security posture. That said, it definitely doesn’t mean you shouldn’t be unconcerned about code execution on developer machines. There’s a reason that internal red team exercises distinguish between external access and already having a foothold on a machine. In this case isn’t obviously not a specific failure in Google’s security policy that package managers don’t do namespacing, but if I was on the team and I received this report I would at least think about whether there is something I would want to do here to improve the situation, similar to existing efforts to prevent attacks like paste bracketing or trivial keylogging.
- davidshepherd7 4y agoThis sounds really interesting! Can you explain or link to an explanation of the technical defenses that Google has? (Or, is this comment https://news.ycombinator.com/item?id=35585453 https://news.ycombinator.com/item?id=35585453 complete and accurate?)