6 ms·

What the hell are npm doing that this caused a prod outage. Either way, really poor from Smyte. No reason to immediately turn off access

sbr464 8y ago
I could imagine (just guessing): submit new repo -> scan with Smyte -> on success, publish repo. Maybe not handling an error properly and skipping Smyte. This problem creeps up with external mfa/2 factor auth APIs that go down, bringing down the main login. Some choose to skip mfa if down, some don’t
- astonex 8y ago
  It's the not handling an API failure correctly that surprises me. That seems like it'd be quite common
  - cwyers 8y ago
    I mean, sometimes handling an API failure correctly is failing over. If an API call actually matters, then when the API stops responding, throw an error. Even if you're probably catching that and giving out an error message, I'd still call that a production outage.
    - astonex 8y ago
      We have a differing opinions on what outage means then. Catching the error and showing it to the user is obviously the right thing to do, but not what I'd call an outage.
      - drchickensalad 8y ago
        I can't say I've ever met a developer who thinks that a 100% error rate to valid requests isn't an outage. Not sure why you have such a strong view of your semantics.
      - function_seven 8y ago
        Sure it is, if it affects all users at once, and makes your system unusable. If I dial a phone number and get a busy signal, that's an "error" of sorts. If every Verizon user gets that busy tone on every call, that's an outage.
  - geofft 8y ago
    This is a spam filter, so failing closed seems reasonable. (Failing open is also reasonable too.)
    - testplzignore 8y ago
      There are lots of potential security issues with npm (compromised accounts, spam packages, etc). I think failing closed is the right thing to do for them. It's a temporary annoyance and loss of productivity for a day - a good time to put your feet up and relax. Failing open could lead to a costly security breach that affects many people, and could lead to npm's demise.
p1necone 8y ago
If I was including an anti spam/security service as part of my pipeline I probably would want my system to stop if it went down, rather than just skipping the protection. Especially as cutting me off from said service is a possible attack option for a bad actor.
- RIMR 8y ago
  Seriously. Shutting down a security product without warning is just reckless...
iainmerrick 8y ago
npm gets a lot of flak on HN, but this seems fine (or rather, it should have been fine) given that they and others apparently had multi-year contracts with Smyte. What kind of situation would you be in if Github went away, for comparison? This seems like a clear breach of contract. I can’t imagine how anybody thought it was a good idea. Maybe it was a mistake, although that wouldn’t be a whole lot better.

sbr464 8y ago

I could imagine (just guessing): submit new repo -> scan with Smyte -> on success, publish repo. Maybe not handling an error properly and skipping Smyte. This problem creeps up with external mfa/2 factor auth APIs that go down, bringing down the main login. Some choose to skip mfa if down, some don’t

astonex 8y ago
It's the not handling an API failure correctly that surprises me. That seems like it'd be quite common
- cwyers 8y ago
  I mean, sometimes handling an API failure correctly is failing over. If an API call actually matters, then when the API stops responding, throw an error. Even if you're probably catching that and giving out an error message, I'd still call that a production outage.
  - astonex 8y ago
    We have a differing opinions on what outage means then. Catching the error and showing it to the user is obviously the right thing to do, but not what I'd call an outage.
    - drchickensalad 8y ago
      I can't say I've ever met a developer who thinks that a 100% error rate to valid requests isn't an outage. Not sure why you have such a strong view of your semantics.
    - function_seven 8y ago
      Sure it is, if it affects all users at once, and makes your system unusable. If I dial a phone number and get a busy signal, that's an "error" of sorts. If every Verizon user gets that busy tone on every call, that's an outage.
- geofft 8y ago
  This is a spam filter, so failing closed seems reasonable. (Failing open is also reasonable too.)
  - testplzignore 8y ago
    There are lots of potential security issues with npm (compromised accounts, spam packages, etc). I think failing closed is the right thing to do for them. It's a temporary annoyance and loss of productivity for a day - a good time to put your feet up and relax. Failing open could lead to a costly security breach that affects many people, and could lead to npm's demise.

p1necone 8y ago

If I was including an anti spam/security service as part of my pipeline I probably would want my system to stop if it went down, rather than just skipping the protection. Especially as cutting me off from said service is a possible attack option for a bad actor.

RIMR 8y ago
Seriously. Shutting down a security product without warning is just reckless...

iainmerrick 8y ago

npm gets a lot of flak on HN, but this seems fine (or rather, it should have been fine) given that they and others apparently had multi-year contracts with Smyte. What kind of situation would you be in if Github went away, for comparison? This seems like a clear breach of contract. I can’t imagine how anybody thought it was a good idea. Maybe it was a mistake, although that wouldn’t be a whole lot better.