7 ms·

And all this is happening just as after the public release of a serious exploit which allows malicious code to do all sorts of nefarious things when it is someh

And all this is happening just as after the public release of a serious exploit which allows malicious code to do all sorts of nefarious things when it is somehow installed on the target machine. Hmm. Given that there's hints, at least, that the problems were caused by some particular developer's actions, I wonder about the security model for package-managed platforms altogether now. If I were a big cybercrime ring, the first thing I'd do would be, get a bunch of thugs together and knock on the front door of a developer of a widely-used package; "help us launch [the sort of attack we're seeing here] or we'll [be very upset with you] with this wrench." Is there a valid defense for a platform whose security relies on the unanimous cooperation of a widely-scattered developer base?

tetha 9y ago
> Is there a valid defense for a platform whose security relies on the unanimous cooperation of a widely-scattered developer base? The defense is staged deployment and active users. This obviously depends on the blutness of the malicious code. If I may assume easily noticed effects of the malicious code: A dev at our place - using java with maven - would update the library, his workstation would get owned. This could have impacts, but if we notice, we'd wipe that workstation, re-image from backup and get in contact with sonatype to kill that version. This version would never touch staging, the last step before prod. If we don't notice on the workstation, there's a good chance we or our IDS would notice trouble either on our testing servers or our staging servers, since especially staging is similar to prod and subject to load tests similar to prod load. Once we're there, it's back to bug reports with the library and contact with sonatype to handle that version. If we can't notice the malicious code at all until due to really really smart activation mechanisms... well then we're in NSA conspiracy land again.
- paulryanrogers 9y ago
  Sounds like good hygiene, though it seems burdensome if everyone must do it or seriously risk infection. Ideally there would be at least minimal sanity checks and a formal process before a package can be claimed by someone else.
- maxander 9y ago
  > If we can't notice the malicious code at all until due to really really smart activation mechanisms... well then we're in NSA conspiracy land again. What about really dumb activation methods? I.e., a condition that only triggers malicious behavior several months after the date the package was subverted. You don’t have to be the NSA to write that. What’s scary here is that there are simpleminded attacks that, AFAIK, we don’t know how to defend against.
  - tetha 9y ago
    Mh, I have a rather aggressive stance on these kind of incidents, no matter if they are availability or security related. You can fish for them, you can test for them, and there are entire classes of malicious code you cannot find. For everything you do, turing complete code can circumvent it. There's a lot of interesting reading material going on in the space of malware analysis regarding sandbox detection, for example. So stop worrying. Try to catch as much as feasible before prod. Then focus on detecting, alerting and ending the actual incident. If code causes an incident, it't probably measurable and detectable. And even then you won't be able to catch everything. As long as a server has behavior observable from the internet, it could be exfiltrating data.
    - didibus 9y ago
      What if it encrypts user data?
      - eropple 9y ago
        You have your tested backups, yeah?
        tetha 9y ago
        Tested restores with at most 59 minutes of data loss for prod clusters within 90 minutes after order. 30ish minutes of downtime. We could even inspect binlogs for a full restore afterwards on a per-request basis for our big customers. Cryptolocker on prod is not my primary issue.
jannotti 9y ago
That's a scary scenario, and all too possible.
dmitriid 9y ago
On top of that, they way countless packages are used everywhere is potentially exploitable: https://medium.com/@david.gilbertson/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5 https://medium.com/@david.gilbertson/im-harvesting-credit-ca...
- kcolford 9y ago
  That's actually even more scary than what's going on now... At least most of us are noticing and can check what's going on...
- razorunreal 9y ago
  In case anyone was considering sending him $10, no, his hypothetical code would not be running on the Google login page. Google does not pull in external dependencies willy nilly like that.
  - didibus 9y ago
    I'd be surprised if they ran a thorough security audit on all code they import, but I'd rather believe they do.
    - jpalomaki 9y ago
      On Google scale you quite certainly want to do that. Not just for security, but for legal reasons. You really don't want to end up using for example AGPL licensed stuff in wrong places and if you just blindly pull stuff with dependencies from package manager, this could easily happen.
      - BeeOnRope 9y ago
        Sure a legal audit is standard and usually much simpler than a full source audit for security, which has a complexity proportional to the project size.
      - thanatropism 9y ago
ioquatix 9y ago
You could sign packages and record their signatures along with the version. Which, coincidentally, is basically what https://teapot.nz https://teapot.nz does, e.g.: https://github.com/kurocha/geometry/blob/master/development-lock.yml https://github.com/kurocha/geometry/blob/master/development-... Although, I've never considered this in the case of an actual attack. It would make sense to actually fingerprint the entire source tree and record this too somewhere, so when you build it you know you are getting the right thing. Teapot basically defers this to git.
- deleted 9y ago
  [deleted]
wybiral 9y ago
An ipfs model would help. People would use a strong hash if the package or something.
- zbentley 9y ago
  I'm not sure if it would help much. That means you either have to have users be able to recognize and eyeball-validate hashes ("sure, this is left-pad-5ffc991e; that's what I want! Wait, shit, it's actually left-pad-5ffd991e, never mind; wrong package), or you need pre-existing databases of trusted hashes (which either puts you right back at a registry a la NPM, or leaves you reliant on a package.lock file or similar, which doesn't cover many common use cases for secure package signing).
  - wybiral 9y ago
    I just meant as a solution to the fact that people can typosquat or jack a name when a package is deleted. If the developers can't get the hash right then there's not much that can be done.
MagicWishMonkey 9y ago
Wouldn't you need to install those packages as root for the code to have privileges to take advantage of that exploit?
- jbg_ 9y ago
  I've known plenty of developers whose automatic response to `packagemanager install packagename` failing is `sudo packagemanager install packagename`.
  - semi-extrinsic 9y ago
    I sincerely hope all modern package managers, when invoked with sudo, immediately spawn a very-low-privilege process that does most of the work sandboxed to /tmp/whatnot, and the root process just copies files to the right place and runs some system calls to update databases etc.
    - bennofs 9y ago
      Most package managers I know support Turing complete install hooks. How would a package manager detect what parts of those require/are safe to run with root?
- zbentley 9y ago
  No. Packages would not need to be installed as root. Additionally, many possible ways to use the exploit in GP could run as unprivileged users.
  - MagicWishMonkey 9y ago
    Wouldn't the package need to be executed as root, though? Or does spectre/meltdown not require privileged access?
    - Natanael_L 9y ago
      No, that's the entire point. They need almost nothing at all but the ability to run code fast in a loop with memory calls. The entire point is that they bypass privilege checks.
ytpete 9y ago
With cases like the current one, or the leftpad incident in 2016, I'm surprised package registries still allow recycling old package names after a package was deleted. Really seems like deleted packages should be frozen forever - if the original author never recreates it or transfers ownership, then people would have to explicitly choose to move to some new fork with a new id. But your point about pressuring or bribing package authors still stands as a scary issue. Similar things have already happened: for example, Kite quietly buying code-editor plugins from their original authors and then adding code some consider spyware (see https://news.ycombinator.com/item?id=14902630 https://news.ycombinator.com/item?id=14902630). I believe there were cases where a similar thing happened with some Chrome extensions too...
- mst 9y ago
  > With cases like the current one, or the leftpad incident in 2016, I'm surprised package registries still allow recycling old package names after a package was deleted. CPAN requires the old author to explicitly transfer or mark it abandoned-and-available-to-new-owner. For all the things wrong with perl5 (and I love it dearly but have spent enough time I can probably list more things wrong with it than the people who hate it ;) it's always a trifle depressing to watch other ecosystems failing to steal the things we got right.
  - Moru 9y ago
    This happens all the time. The new generation creates something cool because what our parents created isnt cool any more, only to fail exactly on the same spot as our parents. Only, it was already solved in the parents last version. This goes for cloth design, cars, houses, kitchen wares and so on, as well as software. Just look at the microwave oven earlier...
    - ragesh 9y ago
      Genuine question... What happened with the microwave oven?
      - 9y ago

tetha 9y ago

> Is there a valid defense for a platform whose security relies on the unanimous cooperation of a widely-scattered developer base? The defense is staged deployment and active users. This obviously depends on the blutness of the malicious code. If I may assume easily noticed effects of the malicious code: A dev at our place - using java with maven - would update the library, his workstation would get owned. This could have impacts, but if we notice, we'd wipe that workstation, re-image from backup and get in contact with sonatype to kill that version. This version would never touch staging, the last step before prod. If we don't notice on the workstation, there's a good chance we or our IDS would notice trouble either on our testing servers or our staging servers, since especially staging is similar to prod and subject to load tests similar to prod load. Once we're there, it's back to bug reports with the library and contact with sonatype to handle that version. If we can't notice the malicious code at all until due to really really smart activation mechanisms... well then we're in NSA conspiracy land again.

paulryanrogers 9y ago
Sounds like good hygiene, though it seems burdensome if everyone must do it or seriously risk infection. Ideally there would be at least minimal sanity checks and a formal process before a package can be claimed by someone else.
maxander 9y ago
> If we can't notice the malicious code at all until due to really really smart activation mechanisms... well then we're in NSA conspiracy land again. What about really dumb activation methods? I.e., a condition that only triggers malicious behavior several months after the date the package was subverted. You don’t have to be the NSA to write that. What’s scary here is that there are simpleminded attacks that, AFAIK, we don’t know how to defend against.
- tetha 9y ago
  Mh, I have a rather aggressive stance on these kind of incidents, no matter if they are availability or security related. You can fish for them, you can test for them, and there are entire classes of malicious code you cannot find. For everything you do, turing complete code can circumvent it. There's a lot of interesting reading material going on in the space of malware analysis regarding sandbox detection, for example. So stop worrying. Try to catch as much as feasible before prod. Then focus on detecting, alerting and ending the actual incident. If code causes an incident, it't probably measurable and detectable. And even then you won't be able to catch everything. As long as a server has behavior observable from the internet, it could be exfiltrating data.
  - didibus 9y ago
    What if it encrypts user data?
    - eropple 9y ago
      You have your tested backups, yeah?
      - tetha 9y ago
        Tested restores with at most 59 minutes of data loss for prod clusters within 90 minutes after order. 30ish minutes of downtime. We could even inspect binlogs for a full restore afterwards on a per-request basis for our big customers. Cryptolocker on prod is not my primary issue.

jannotti 9y ago

That's a scary scenario, and all too possible.

dmitriid 9y ago

On top of that, they way countless packages are used everywhere is potentially exploitable: https://medium.com/@david.gilbertson/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5 https://medium.com/@david.gilbertson/im-harvesting-credit-ca...

kcolford 9y ago
That's actually even more scary than what's going on now... At least most of us are noticing and can check what's going on...
razorunreal 9y ago
In case anyone was considering sending him $10, no, his hypothetical code would not be running on the Google login page. Google does not pull in external dependencies willy nilly like that.
- didibus 9y ago
  I'd be surprised if they ran a thorough security audit on all code they import, but I'd rather believe they do.
  - jpalomaki 9y ago
    On Google scale you quite certainly want to do that. Not just for security, but for legal reasons. You really don't want to end up using for example AGPL licensed stuff in wrong places and if you just blindly pull stuff with dependencies from package manager, this could easily happen.
    - BeeOnRope 9y ago
      Sure a legal audit is standard and usually much simpler than a full source audit for security, which has a complexity proportional to the project size.
    - thanatropism 9y ago

ioquatix 9y ago

You could sign packages and record their signatures along with the version. Which, coincidentally, is basically what https://teapot.nz https://teapot.nz does, e.g.: https://github.com/kurocha/geometry/blob/master/development-lock.yml https://github.com/kurocha/geometry/blob/master/development-... Although, I've never considered this in the case of an actual attack. It would make sense to actually fingerprint the entire source tree and record this too somewhere, so when you build it you know you are getting the right thing. Teapot basically defers this to git.

deleted 9y ago
[deleted]

wybiral 9y ago

An ipfs model would help. People would use a strong hash if the package or something.

zbentley 9y ago
I'm not sure if it would help much. That means you either have to have users be able to recognize and eyeball-validate hashes ("sure, this is left-pad-5ffc991e; that's what I want! Wait, shit, it's actually left-pad-5ffd991e, never mind; wrong package), or you need pre-existing databases of trusted hashes (which either puts you right back at a registry a la NPM, or leaves you reliant on a package.lock file or similar, which doesn't cover many common use cases for secure package signing).
- wybiral 9y ago
  I just meant as a solution to the fact that people can typosquat or jack a name when a package is deleted. If the developers can't get the hash right then there's not much that can be done.

MagicWishMonkey 9y ago

Wouldn't you need to install those packages as root for the code to have privileges to take advantage of that exploit?

jbg_ 9y ago
I've known plenty of developers whose automatic response to `packagemanager install packagename` failing is `sudo packagemanager install packagename`.
- semi-extrinsic 9y ago
  I sincerely hope all modern package managers, when invoked with sudo, immediately spawn a very-low-privilege process that does most of the work sandboxed to /tmp/whatnot, and the root process just copies files to the right place and runs some system calls to update databases etc.
  - bennofs 9y ago
    Most package managers I know support Turing complete install hooks. How would a package manager detect what parts of those require/are safe to run with root?
zbentley 9y ago
No. Packages would not need to be installed as root. Additionally, many possible ways to use the exploit in GP could run as unprivileged users.
- MagicWishMonkey 9y ago
  Wouldn't the package need to be executed as root, though? Or does spectre/meltdown not require privileged access?
  - Natanael_L 9y ago
    No, that's the entire point. They need almost nothing at all but the ability to run code fast in a loop with memory calls. The entire point is that they bypass privilege checks.

ytpete 9y ago

With cases like the current one, or the leftpad incident in 2016, I'm surprised package registries still allow recycling old package names after a package was deleted. Really seems like deleted packages should be frozen forever - if the original author never recreates it or transfers ownership, then people would have to explicitly choose to move to some new fork with a new id. But your point about pressuring or bribing package authors still stands as a scary issue. Similar things have already happened: for example, Kite quietly buying code-editor plugins from their original authors and then adding code some consider spyware (see https://news.ycombinator.com/item?id=14902630 https://news.ycombinator.com/item?id=14902630). I believe there were cases where a similar thing happened with some Chrome extensions too...

mst 9y ago
> With cases like the current one, or the leftpad incident in 2016, I'm surprised package registries still allow recycling old package names after a package was deleted. CPAN requires the old author to explicitly transfer or mark it abandoned-and-available-to-new-owner. For all the things wrong with perl5 (and I love it dearly but have spent enough time I can probably list more things wrong with it than the people who hate it ;) it's always a trifle depressing to watch other ecosystems failing to steal the things we got right.
- Moru 9y ago
  This happens all the time. The new generation creates something cool because what our parents created isnt cool any more, only to fail exactly on the same spot as our parents. Only, it was already solved in the parents last version. This goes for cloth design, cars, houses, kitchen wares and so on, as well as software. Just look at the microwave oven earlier...
  - ragesh 9y ago
    Genuine question... What happened with the microwave oven?
    - 9y ago