3 ms·

I consider this kind of reporting (and underlying detection by companies capitalizing on supply chain security fears) borderline irresponsible: anybody can uplo

I consider this kind of reporting (and underlying detection by companies capitalizing on supply chain security fears) borderline irresponsible: anybody can upload a package to a public index, by design, and the presence of a blatantly maliciously package doesn’t imply any degree of actual download, use of, or compromise via it. Removing malware from the index is a laudable goal, but the language of threats, attacks, thefts, etc. is essentially unsubstantiated: there’s no evidence that this goes beyond opportunistic, untargeted script kiddy behavior. Edit: FTA, the “rapid evolution” in question is that the malicious packages switched from using base64 for obfuscation to…double base64.

SethMLarson 3y ago
I am looking forward to PyPI's "answer" to this class of malicious packages with automated third-party reporting (and hopefully long-term, quarantining or even deletion). Hopefully this project will swing the narrative closer to the /actual/ security threat that these packages represent to their respective package ecosystems. The PyPI Safety and Security Engineer Mike Fiedler is working on this project :)
- woodruffw 3y ago
  Yes! I think an under-considered aspect here is that this kind of high-volume malicious package activity isn’t particularly dangerous in and of itself, but does represent a operational/sustainability risk w/r/t compromising the index’s abilities to respond to actual urgent issues in a timely manner. More automation (reporting, quarantining) goes a long way towards addressing that. It’s more the language of compromise I resent: I think treating every low-effort malicious spam package as a fundamental risk unnecessarily “spends” the security mindshare budget that engineers keep in reserve. But I think my opinions there are already well-known :-)
015a 3y ago
I... disagree. I feel that the differentiation between "lifting keys" and "maliciously using them" is not a useful one to make. Both actions are very bad. If you want to wait until we have evidence that the compromised keys were used to pull e.g. proprietary code or configuration from Github; why not also wait until that compromised data is sold? Or until a company is formed leveraging it? Ultimately, it's a slippery slope of "nothing will satisfy you", and there's clearly a genesis event here that is reprehensible and should be known about, even if it is, as you say, just opportunistic script kiddies. Speaking of which; I loath that characterization of threat actors. This is not the 90s anymore, and it's time to grow up and recognize that the opponents in this space can be extremely sophisticated and well-funded. This is, genuinely, something that many people in the industry refuse to accept; because it implies that we absolutely need to hold ourselves and our work to a higher standard, treat what we produce with severe gravity, and ultimately this means the pace of innovation will slow down and we'll need to ask the mirror serious questions like "wait, so you just, npm installed critical code that was handling sensitive user data? without even auditing it?" NPM (and, by extension, Github) are extremely blame-worthy here. Reports like this one absolutely happen so Sonatype can sell their product; no doubt on that coming from me whatsoever. But, they also put pressure on key infrastructure providers. > FTA, the “rapid evolution” in question is that the malicious packages switched from using base64 for obfuscation to…double base64. Ok, yeah; describing that as "rapid evolution" is overselling it. But let's play in that playground; it's not a rapid evolution, it's an unsophisticated additional layer of easily-reversible obfuscation, which npm did not catch. Why? Why is sonatype catching this, and not npm?
- woodruffw 3y ago
  > I feel that the differentiation between "lifting keys" and "maliciously using them" is not a useful one to make. You misunderstood: the differentiation was between malicious packages being uploaded and actually being used, not between the keys being stolen and being used. I agree that the latter would not be a useful or productive distinction to make. To be clear: supply chain security is serious, and there are sophisticated (organized crime, nation state) actors in the space. But that isn’t what this is; the lack of sophistication makes that clear. It’s true that indices could probably be doing more to stop this kind of stuff, and many are; see the adjacent comment by ‘SethMLarson.
jimt1234 3y ago
A few years ago, my company found hundreds of internal projects using the malicious "python3-dateutil" library. One instance was ultimately used to get full ATO of an AWS account and launch hundreds of EC2 instances to mine crypto. So, the threat is real, even if it is overstated by "companies capitalizing on supply chain security fears".
- woodruffw 3y ago
  I agree that the threat is real. But I think the threat needs to be quantifiable in terms of metrics that aren’t solely the attacker’s ability to spam the index. Put another way: what matters is that the attacker can get arbitrary code to run on your machine, not that they can unload arbitrary package-shaped things to a publicly accessible package index. The former can be solved; the latter isn’t solvable in the general case.
  - delusional 3y ago
    I liken it to hosting malware on the open internet. Yes the internet can serve malware, duh, that's the point. You can share whatever you want over the internet and we wouldn't want it any other way. What matters is who you can get to download and execute it. And what they can do with that remote code execution. As always, the most secure system is the system that doesn't exist. Every other security argument has to contend with our desire to do useful stuff, and therefore it has to begin with a credible attack vector. Not just a hunch or some notion that it's "security best practice".
datadeft 3y ago
Better yet, we need a new model of running computer programs. I think the good old running a python file through an interpreter that has at least read-only access to ~/ is not cutting it anymore. This is why Apple's additional access (explicitly granting read access to certain user folders to certain applications) is a good step to the right direction.
- simonw 3y ago
  The thing I most want from computing right now is the ability to safely run untrusted code in a sandbox on my own devices. It feels like we're constantly getting closer to this. We've been mostly able to do this in the browser for years now. I continue to have high hopes for non-browser WebAssembly. I really, really want to be able to run Python, JavaScript, Rust code etc on my phone and laptop inside a WebAssembly sandbox that tightly limits the filesystem, network, memory and CPU access of that code. I'm seeing glimpses of this being possible - https://til.simonwillison.net/webassembly/python-in-a-wasm-sandbox https://til.simonwillison.net/webassembly/python-in-a-wasm-s... and https://til.simonwillison.net/deno/pyodide-sandbox https://til.simonwillison.net/deno/pyodide-sandbox for example - but it's still not as easy as I want it to be.
  - bomewish 3y ago
    Docker doesn’t solve this?
    - Tknl 3y ago
      Docker provides virtually no security guarantees on its own and container rights escalation vulnerabilities are a serious threat vector. The default containers provided by many applications are often insecure and ignore recommended security hardening practices. Seehttps://kubernetes.io/docs/concepts/security/overview/ https://kubernetes.io/docs/concepts/security/overview/ 4 c's of container security.
    - simonw 3y ago
      Sadly it doesn't - Docker isn't specifically designed as a security tool and there are plenty of warnings against using it in this way. Firecracker is the best I've seen in terms of container security - it was designed by AWS for Lambda and is trusted by people I trust. It's not really packaged for easily running on macOS etc though.

SethMLarson 3y ago

I am looking forward to PyPI's "answer" to this class of malicious packages with automated third-party reporting (and hopefully long-term, quarantining or even deletion). Hopefully this project will swing the narrative closer to the /actual/ security threat that these packages represent to their respective package ecosystems. The PyPI Safety and Security Engineer Mike Fiedler is working on this project :)

woodruffw 3y ago
Yes! I think an under-considered aspect here is that this kind of high-volume malicious package activity isn’t particularly dangerous in and of itself, but does represent a operational/sustainability risk w/r/t compromising the index’s abilities to respond to actual urgent issues in a timely manner. More automation (reporting, quarantining) goes a long way towards addressing that. It’s more the language of compromise I resent: I think treating every low-effort malicious spam package as a fundamental risk unnecessarily “spends” the security mindshare budget that engineers keep in reserve. But I think my opinions there are already well-known :-)

015a 3y ago

I... disagree. I feel that the differentiation between "lifting keys" and "maliciously using them" is not a useful one to make. Both actions are very bad. If you want to wait until we have evidence that the compromised keys were used to pull e.g. proprietary code or configuration from Github; why not also wait until that compromised data is sold? Or until a company is formed leveraging it? Ultimately, it's a slippery slope of "nothing will satisfy you", and there's clearly a genesis event here that is reprehensible and should be known about, even if it is, as you say, just opportunistic script kiddies. Speaking of which; I loath that characterization of threat actors. This is not the 90s anymore, and it's time to grow up and recognize that the opponents in this space can be extremely sophisticated and well-funded. This is, genuinely, something that many people in the industry refuse to accept; because it implies that we absolutely need to hold ourselves and our work to a higher standard, treat what we produce with severe gravity, and ultimately this means the pace of innovation will slow down and we'll need to ask the mirror serious questions like "wait, so you just, npm installed critical code that was handling sensitive user data? without even auditing it?" NPM (and, by extension, Github) are extremely blame-worthy here. Reports like this one absolutely happen so Sonatype can sell their product; no doubt on that coming from me whatsoever. But, they also put pressure on key infrastructure providers. > FTA, the “rapid evolution” in question is that the malicious packages switched from using base64 for obfuscation to…double base64. Ok, yeah; describing that as "rapid evolution" is overselling it. But let's play in that playground; it's not a rapid evolution, it's an unsophisticated additional layer of easily-reversible obfuscation, which npm did not catch. Why? Why is sonatype catching this, and not npm?

woodruffw 3y ago
> I feel that the differentiation between "lifting keys" and "maliciously using them" is not a useful one to make. You misunderstood: the differentiation was between malicious packages being uploaded and actually being used, not between the keys being stolen and being used. I agree that the latter would not be a useful or productive distinction to make. To be clear: supply chain security is serious, and there are sophisticated (organized crime, nation state) actors in the space. But that isn’t what this is; the lack of sophistication makes that clear. It’s true that indices could probably be doing more to stop this kind of stuff, and many are; see the adjacent comment by ‘SethMLarson.

jimt1234 3y ago

A few years ago, my company found hundreds of internal projects using the malicious "python3-dateutil" library. One instance was ultimately used to get full ATO of an AWS account and launch hundreds of EC2 instances to mine crypto. So, the threat is real, even if it is overstated by "companies capitalizing on supply chain security fears".

woodruffw 3y ago
I agree that the threat is real. But I think the threat needs to be quantifiable in terms of metrics that aren’t solely the attacker’s ability to spam the index. Put another way: what matters is that the attacker can get arbitrary code to run on your machine, not that they can unload arbitrary package-shaped things to a publicly accessible package index. The former can be solved; the latter isn’t solvable in the general case.
- delusional 3y ago
  I liken it to hosting malware on the open internet. Yes the internet can serve malware, duh, that's the point. You can share whatever you want over the internet and we wouldn't want it any other way. What matters is who you can get to download and execute it. And what they can do with that remote code execution. As always, the most secure system is the system that doesn't exist. Every other security argument has to contend with our desire to do useful stuff, and therefore it has to begin with a credible attack vector. Not just a hunch or some notion that it's "security best practice".

datadeft 3y ago

Better yet, we need a new model of running computer programs. I think the good old running a python file through an interpreter that has at least read-only access to ~/ is not cutting it anymore. This is why Apple's additional access (explicitly granting read access to certain user folders to certain applications) is a good step to the right direction.

simonw 3y ago
The thing I most want from computing right now is the ability to safely run untrusted code in a sandbox on my own devices. It feels like we're constantly getting closer to this. We've been mostly able to do this in the browser for years now. I continue to have high hopes for non-browser WebAssembly. I really, really want to be able to run Python, JavaScript, Rust code etc on my phone and laptop inside a WebAssembly sandbox that tightly limits the filesystem, network, memory and CPU access of that code. I'm seeing glimpses of this being possible - https://til.simonwillison.net/webassembly/python-in-a-wasm-sandbox https://til.simonwillison.net/webassembly/python-in-a-wasm-s... and https://til.simonwillison.net/deno/pyodide-sandbox https://til.simonwillison.net/deno/pyodide-sandbox for example - but it's still not as easy as I want it to be.
- bomewish 3y ago
  Docker doesn’t solve this?
  - Tknl 3y ago
    Docker provides virtually no security guarantees on its own and container rights escalation vulnerabilities are a serious threat vector. The default containers provided by many applications are often insecure and ignore recommended security hardening practices. Seehttps://kubernetes.io/docs/concepts/security/overview/ https://kubernetes.io/docs/concepts/security/overview/ 4 c's of container security.
  - simonw 3y ago
    Sadly it doesn't - Docker isn't specifically designed as a security tool and there are plenty of warnings against using it in this way. Firecracker is the best I've seen in terms of container security - it was designed by AWS for Lambda and is trusted by people I trust. It's not really packaged for easily running on macOS etc though.