4 ms·

How do you suggest they prevent this?

mixdup 4y ago
Are you saying it's impossible to mitigate/eliminate package source confusion attacks? There's no way in Python to pin the source exactly where Google wants it?
- aflag 4y ago
  It's possible to mitigate, but it's impossible to realistically prevent unless you disconnect your employees from the internet. Which is probably costlier than any losses they are ever likely to end up having due to this.
  - Brian_K_White 4y ago
    There is a universe of difference between "employee is allowed to download from the internet" and "something unintended from an unauthorized external party runs automatically" come the fuck on
    - juunpp 4y ago
      And "employee ignores the threat despite being told repeatedly about it". I don't know where some HN folks get their logic from.
    - robertlagrant 4y ago
      It's not very big when it comes to package management.
    - Xylakant 4y ago
      Any package from pypi can at any time run code that’s malicious. IIRC, even at install time. So installing any package, even in a python virtual environment bears the risk of arbitrary code execution - before you have a chance to inspect it. (Same for php, ruby, rust, …) Basing your security concept on that never happening is like playing whack-a-mole in hardcore mode.
    - aflag 4y ago
      What are you going to do? Block access to any pypi.org package except for whitelisted ones? You can try to mitigate things by perhaps blocking access to any pypi repos that's also available internally but that haven't been released to the public. At least it would get rid of this one vector. But that doesn't mean that people won't just "pip install whatever" anyway.
      - Brian_K_White 4y ago
        To quote John Oliver: "What can you do?" "Something. You can do, something." The possible ways to respond to that process are open ended and infinite. You can do anything you want about it. Doing nothing about it is approximately the least defensible.
        Xylakant 4y ago
        > Doing nothing about it is approximately the least defensible. Everything you do has a cost. Increased friction which makes people work around it. Lower productivity. Time spent implementing it that cannot be used to implement something more useful. Don’t just do something. Do something that improves the situation - and google does. Their statement indicates that they consider the developers machine as fundamentally not trusted - and I’d consider that a correct assumption. Some of the thousands of machines will be compromised at any given time. Whether it’s via this exploit, or another or by bribing the engineer doesn’t matter. What matter is that they attempt to contain the issue at that boundary.
        Brian_K_White 4y ago
        I don't know why you interpret that that way, but I sure hope you don't write the autopilot to any planes I might fly in or anythimg else important with such logic. I did not say or imply or suggest to "do something, anything" without caring if it's sensible or effective. "anything" simply means there is no limit to the possible suitable things. There is also no limit to the possible unsuitable things, but so what? It is beyond stupid to take that starting point and conclude that anyone suggested "Maybe Gooogle should issue Tarot decks to all employees to determine if they should press enter at the end of every shell command." just because, after all, that is something and included in "anything". I can't know which of the infinite possible detailed measures make sense within Google's environment. But I don't have to to still know that they exist. The details will depend on internal details only they know. If I say "wrap the pip command in an internal wrapper that performs various checks" surely there is some reason that is not practical or not effective enough, exactly as stated. Or maybe that would exactly clear it all up. But if not, ok so something else then. Have some imagination. But that does not remotely imply random nonsense.
- Xylakant 4y ago
  Essentially it’s impossible. You can make it harder, but if you grant developer machines access to the internet, they can download an untrustworthy package from anywhere and use it as a local dependency. You can protect production and CI systems by restricting their internet access, but who can reasonably do work with such a restriction.
  - jacquesm 4y ago
    You could work with a whitelist or a build environment that is sandboxed somehow.
    - Xylakant 4y ago
      As I understand the article, the problem is that an employee machine is misconfigured and uses the public package repo for a private package. The package in question seems to be explicitly pinned in the requirements.txt - that’s essentially a whitelist. Whitelisting also suffers from the issue that a good public packages could at any time turn into a malicious package - so you’d need to pin the version. A task that seems infeasible for developer machines at googles scale. Sandboxing the development environment could be done, but would only help against this attack if the sandbox cannot connect to the public internet- which again would be painful. Googles strategy of accepting that this kind of breach will happen and rather focus on mitigation of the resulting damage seems like the better way.
  - smsm42 4y ago
    There's a big difference from a person downloading a package from a random site and running it, and a person adding a known internal package name to requirements.txt. The first is usually a bad idea (unless you know what you're doing and ready for the consequences) and every SWE worth Google money knows that. The second is a standard practice and everybody does that. You can't make stupid behavior impossible - but you can make common behavior safe, and that's the basis of good security.
    - Xylakant 4y ago
      I agree with your general thrust, but I don’t see how your point refutes mine. First, it’s not required to actually run the package. Installing it is sufficient. All package managers that I worked with so far can run code at install time. Second, the issue here seems to be a misconfiguration that makes pip look up a private package that should be retrieved from a private repository on the public repository. The attacker then just registers a malicious package on the public repo with the same name. Preventing this attack requires that python is correctly configured on each an every developer machine - something that I’d never rely on as cornerstone of my security. Third: This is one example of smuggling a malicious package on a developers machine. Another vector is that a good package turns into a malicious package with an update. That’s even harder to defend against - pulling in the update with the programming languages standard tooling may run malicious code. You can certainly first download the package, unpack and inspect it and then pull the update - but would you rely on thousands of developers diligently doing that? Last: this class of error affects almost all programming language package managers out there. So it’s better to assume that this will happen, take a local compromise of a developers machine as a matter of when, and mitigate what the attacker can do with the capabilities they gain from this compromise.
smsm42 4y ago
Namespacing packages plus private (or public) pip patch that hardcodes the link of certain namespaces to certain repos would be the most obvious one (I spent 5 seconds thinking about it so far not I'm not saying it's the best ever). Internal package registry that knows all internal package names and makes pip reject colliding names from other sources would be another possibility. Explicitly verifying the hash of the internal package based on the registry (again) and refusing to install packages that don't match the hash would be another option. I'm sure if a person smarter than me (Google probably has thousands) spends a day thinking about it, they could think of a dozen better ways.

Are you saying it's impossible to mitigate/eliminate package source confusion attacks? There's no way in Python to pin the source exactly where Google wants it?

aflag 4y ago
It's possible to mitigate, but it's impossible to realistically prevent unless you disconnect your employees from the internet. Which is probably costlier than any losses they are ever likely to end up having due to this.
- Brian_K_White 4y ago
  There is a universe of difference between "employee is allowed to download from the internet" and "something unintended from an unauthorized external party runs automatically" come the fuck on
  - juunpp 4y ago
    And "employee ignores the threat despite being told repeatedly about it". I don't know where some HN folks get their logic from.
  - robertlagrant 4y ago
    It's not very big when it comes to package management.
  - Xylakant 4y ago
    Any package from pypi can at any time run code that’s malicious. IIRC, even at install time. So installing any package, even in a python virtual environment bears the risk of arbitrary code execution - before you have a chance to inspect it. (Same for php, ruby, rust, …) Basing your security concept on that never happening is like playing whack-a-mole in hardcore mode.
  - aflag 4y ago
    What are you going to do? Block access to any pypi.org package except for whitelisted ones? You can try to mitigate things by perhaps blocking access to any pypi repos that's also available internally but that haven't been released to the public. At least it would get rid of this one vector. But that doesn't mean that people won't just "pip install whatever" anyway.
    - Brian_K_White 4y ago
      To quote John Oliver: "What can you do?" "Something. You can do, something." The possible ways to respond to that process are open ended and infinite. You can do anything you want about it. Doing nothing about it is approximately the least defensible.
      - Xylakant 4y ago
        > Doing nothing about it is approximately the least defensible. Everything you do has a cost. Increased friction which makes people work around it. Lower productivity. Time spent implementing it that cannot be used to implement something more useful. Don’t just do something. Do something that improves the situation - and google does. Their statement indicates that they consider the developers machine as fundamentally not trusted - and I’d consider that a correct assumption. Some of the thousands of machines will be compromised at any given time. Whether it’s via this exploit, or another or by bribing the engineer doesn’t matter. What matter is that they attempt to contain the issue at that boundary.
        Brian_K_White 4y ago
        I don't know why you interpret that that way, but I sure hope you don't write the autopilot to any planes I might fly in or anythimg else important with such logic. I did not say or imply or suggest to "do something, anything" without caring if it's sensible or effective. "anything" simply means there is no limit to the possible suitable things. There is also no limit to the possible unsuitable things, but so what? It is beyond stupid to take that starting point and conclude that anyone suggested "Maybe Gooogle should issue Tarot decks to all employees to determine if they should press enter at the end of every shell command." just because, after all, that is something and included in "anything". I can't know which of the infinite possible detailed measures make sense within Google's environment. But I don't have to to still know that they exist. The details will depend on internal details only they know. If I say "wrap the pip command in an internal wrapper that performs various checks" surely there is some reason that is not practical or not effective enough, exactly as stated. Or maybe that would exactly clear it all up. But if not, ok so something else then. Have some imagination. But that does not remotely imply random nonsense.
Xylakant 4y ago
Essentially it’s impossible. You can make it harder, but if you grant developer machines access to the internet, they can download an untrustworthy package from anywhere and use it as a local dependency. You can protect production and CI systems by restricting their internet access, but who can reasonably do work with such a restriction.
- jacquesm 4y ago
  You could work with a whitelist or a build environment that is sandboxed somehow.
  - Xylakant 4y ago
    As I understand the article, the problem is that an employee machine is misconfigured and uses the public package repo for a private package. The package in question seems to be explicitly pinned in the requirements.txt - that’s essentially a whitelist. Whitelisting also suffers from the issue that a good public packages could at any time turn into a malicious package - so you’d need to pin the version. A task that seems infeasible for developer machines at googles scale. Sandboxing the development environment could be done, but would only help against this attack if the sandbox cannot connect to the public internet- which again would be painful. Googles strategy of accepting that this kind of breach will happen and rather focus on mitigation of the resulting damage seems like the better way.
- smsm42 4y ago
  There's a big difference from a person downloading a package from a random site and running it, and a person adding a known internal package name to requirements.txt. The first is usually a bad idea (unless you know what you're doing and ready for the consequences) and every SWE worth Google money knows that. The second is a standard practice and everybody does that. You can't make stupid behavior impossible - but you can make common behavior safe, and that's the basis of good security.
  - Xylakant 4y ago
    I agree with your general thrust, but I don’t see how your point refutes mine. First, it’s not required to actually run the package. Installing it is sufficient. All package managers that I worked with so far can run code at install time. Second, the issue here seems to be a misconfiguration that makes pip look up a private package that should be retrieved from a private repository on the public repository. The attacker then just registers a malicious package on the public repo with the same name. Preventing this attack requires that python is correctly configured on each an every developer machine - something that I’d never rely on as cornerstone of my security. Third: This is one example of smuggling a malicious package on a developers machine. Another vector is that a good package turns into a malicious package with an update. That’s even harder to defend against - pulling in the update with the programming languages standard tooling may run malicious code. You can certainly first download the package, unpack and inspect it and then pull the update - but would you rely on thousands of developers diligently doing that? Last: this class of error affects almost all programming language package managers out there. So it’s better to assume that this will happen, take a local compromise of a developers machine as a matter of when, and mitigate what the attacker can do with the capabilities they gain from this compromise.

smsm42 4y ago

Namespacing packages plus private (or public) pip patch that hardcodes the link of certain namespaces to certain repos would be the most obvious one (I spent 5 seconds thinking about it so far not I'm not saying it's the best ever). Internal package registry that knows all internal package names and makes pip reject colliding names from other sources would be another possibility. Explicitly verifying the hash of the internal package based on the registry (again) and refusing to install packages that don't match the hash would be another option. I'm sure if a person smarter than me (Google probably has thousands) spends a day thinking about it, they could think of a dozen better ways.