5 ms·

Hi folks, npm COO here. This was an operational issue that we worked to correct. All packages are now restored: https://status.npmjs.org/incidents/41zfb8qpvrdj

by seldo 9y ago

Hi folks, npm COO here. This was an operational issue that we worked to correct. All packages are now restored: https://status.npmjs.org/incidents/41zfb8qpvrdj https://status.npmjs.org/incidents/41zfb8qpvrdj

seldo 9y ago
Update: (this is not the post-mortem, this is just more detail) http://blog.npmjs.org/post/169432444640/npm-operational-incident-6-jan-2018 http://blog.npmjs.org/post/169432444640/npm-operational-inci...
xwvvvvwx 9y ago
What was the root cause of the issue?
- chrisfosterelli 9y ago
  Yes I'd be very curious to see a debrief on what the technical cause was. Thanks to the npm team for a quick weekend fix, at any rate!
  - seldo 9y ago
    We're working on a full post-mortem now. Until then we don't want to give out misleading/partial information.
    - fl00d 9y ago
      Any update on the post-mortem? How long have the binaries been replaced? Is there evidence that malware was injected into the binaries? Additionally, you should brush up on your code signing implementations. Had you signed it with a trusted code signing cert, consumers could have verified that you produced the binaries...and not a malicious user. Assuming they didnt have access to the private key material of your code signing key.
      - chrisfosterelli 9y ago
        Not sure if you saw but they did post this: http://blog.npmjs.org/post/169432444640/npm-operational-incident-6-jan-2018 http://blog.npmjs.org/post/169432444640/npm-operational-inci...
- drdrey 9y ago
  Or rather: what were the contributing factors of the issue?
thsowers 9y ago
Any chance of a technical write-up so that we can all learn from whatever happened?
- seldo 9y ago
  Absofuckinglutely. It's being done as we speak.
  - sillysaurus3 9y ago
    Ha, thanks for speaking plainly. It's so refreshing. Sometimes a fuckbomb is the best way. Rarely and tastefully, but still.
    - mrlatinos 9y ago
      It's unprofessional in circumstances such as this imo, but to each their own.
      - sillysaurus3 9y ago
        "The less confident you are, the more serious you have to act."
        darkerside 9y ago
        Seems to make the opposite case here. Why the need to swear? Seems not to indicate anything but disingenuous tribal signaling of outrage. At whom?
  - danjoc 9y ago
    Good luck explaining this https://news.ycombinator.com/item?id=16087079 https://news.ycombinator.com/item?id=16087079 in the face of this https://news.ycombinator.com/item?id=14905870 https://news.ycombinator.com/item?id=14905870 Literally nothing was done for 158 days. You yourself asked: https://github.com/node-forward/discussions/issues/29#issuecomment-320170082 https://github.com/node-forward/discussions/issues/29#issuec... "How would package signing prevent people from requesting the wrong package? The malware author could also sign their package." And here is a perfect example. Someone replaced a legit package with a malicious one. Had the original author signed the package, then then NPM users could have defended against the new malicious author, because the new author's signing key would not be in their truststore. Unsigned packages leave NPM package users defenseless. I hope that is crystal clear now.
BetterThanSlave 9y ago
Isaac is subhuman filth.
nnutter 9y ago
Seems like you should have froze publishing instead of saying, "Please do not attempt to republish packages, as this will hinder our progress in restoring them." Especially, to prevent, even temporary, hijacking.
yashap 9y ago
Were any of the deleted packages temporarily hijacked? It seems strongly like this was the case. If so, please confirm immediately so people who installed packages during this time can start scanning for malware. Even if the answer is “yes, 1+ packages were hijacked by not-the-original author, but we’re still investigating if there was malware”, tell people immediately. Don’t wait a few days for your investigation and post mortem if it’s possible that some users’ systems have already been compromised.
- f4rker 9y ago
  "Don’t wait a few days for your investigation " Mate, one can't come to conclusions without an investigation, ya?
- electric_sheep 9y ago
  I would also hope for and expect this to be communicated ASAP from the NPM org to its users. @seldo, I understand that you don't want to disseminate misleading info, but an abundance of caution seems warranted in this case as my understanding of the incident lines up with what @yashap has said. If we're wrong, straighten us out --- if we're not, please sound an advisory, because this is major.
  - yashap 9y ago
    Yeah, these were some core, widely used packages that were deleted. If they were temporarily hijacked, lots of dev machines (including mine) may have been compromised. There’s a major security risk here, if there was any hijacking now is not the timing for information hiding and PR.

by seldo 9y ago

seldo 9y ago

Update: (this is not the post-mortem, this is just more detail) http://blog.npmjs.org/post/169432444640/npm-operational-incident-6-jan-2018 http://blog.npmjs.org/post/169432444640/npm-operational-inci...

xwvvvvwx 9y ago

What was the root cause of the issue?

chrisfosterelli 9y ago
Yes I'd be very curious to see a debrief on what the technical cause was. Thanks to the npm team for a quick weekend fix, at any rate!
- seldo 9y ago
  We're working on a full post-mortem now. Until then we don't want to give out misleading/partial information.
  - fl00d 9y ago
    Any update on the post-mortem? How long have the binaries been replaced? Is there evidence that malware was injected into the binaries? Additionally, you should brush up on your code signing implementations. Had you signed it with a trusted code signing cert, consumers could have verified that you produced the binaries...and not a malicious user. Assuming they didnt have access to the private key material of your code signing key.
    - chrisfosterelli 9y ago
      Not sure if you saw but they did post this: http://blog.npmjs.org/post/169432444640/npm-operational-incident-6-jan-2018 http://blog.npmjs.org/post/169432444640/npm-operational-inci...
drdrey 9y ago
Or rather: what were the contributing factors of the issue?

thsowers 9y ago

Any chance of a technical write-up so that we can all learn from whatever happened?

seldo 9y ago
Absofuckinglutely. It's being done as we speak.
- sillysaurus3 9y ago
  Ha, thanks for speaking plainly. It's so refreshing. Sometimes a fuckbomb is the best way. Rarely and tastefully, but still.
  - mrlatinos 9y ago
    It's unprofessional in circumstances such as this imo, but to each their own.
    - sillysaurus3 9y ago
      "The less confident you are, the more serious you have to act."
      - darkerside 9y ago
        Seems to make the opposite case here. Why the need to swear? Seems not to indicate anything but disingenuous tribal signaling of outrage. At whom?
- danjoc 9y ago
  Good luck explaining this https://news.ycombinator.com/item?id=16087079 https://news.ycombinator.com/item?id=16087079 in the face of this https://news.ycombinator.com/item?id=14905870 https://news.ycombinator.com/item?id=14905870 Literally nothing was done for 158 days. You yourself asked: https://github.com/node-forward/discussions/issues/29#issuecomment-320170082 https://github.com/node-forward/discussions/issues/29#issuec... "How would package signing prevent people from requesting the wrong package? The malware author could also sign their package." And here is a perfect example. Someone replaced a legit package with a malicious one. Had the original author signed the package, then then NPM users could have defended against the new malicious author, because the new author's signing key would not be in their truststore. Unsigned packages leave NPM package users defenseless. I hope that is crystal clear now.

BetterThanSlave 9y ago

Isaac is subhuman filth.

nnutter 9y ago

Seems like you should have froze publishing instead of saying, "Please do not attempt to republish packages, as this will hinder our progress in restoring them." Especially, to prevent, even temporary, hijacking.

yashap 9y ago

Were any of the deleted packages temporarily hijacked? It seems strongly like this was the case. If so, please confirm immediately so people who installed packages during this time can start scanning for malware. Even if the answer is “yes, 1+ packages were hijacked by not-the-original author, but we’re still investigating if there was malware”, tell people immediately. Don’t wait a few days for your investigation and post mortem if it’s possible that some users’ systems have already been compromised.

f4rker 9y ago
"Don’t wait a few days for your investigation " Mate, one can't come to conclusions without an investigation, ya?
electric_sheep 9y ago
I would also hope for and expect this to be communicated ASAP from the NPM org to its users. @seldo, I understand that you don't want to disseminate misleading info, but an abundance of caution seems warranted in this case as my understanding of the incident lines up with what @yashap has said. If we're wrong, straighten us out --- if we're not, please sound an advisory, because this is major.
- yashap 9y ago
  Yeah, these were some core, widely used packages that were deleted. If they were temporarily hijacked, lots of dev machines (including mine) may have been compromised. There’s a major security risk here, if there was any hijacking now is not the timing for information hiding and PR.