4 ms·

This is incredibly fragile ecosystem if one obscure API getting shutdown causes so much damage. This could be prevented by a simple periodic check that would de

This is incredibly fragile ecosystem if one obscure API getting shutdown causes so much damage. This could be prevented by a simple periodic check that would determine if API is available and some fail-safe alternative to remain online. Instead this is like a house of cards that breaks with any of the cards fold. I'm not excusing the abrupt shutdown, its obviously a wrong way to end service, but being prepared for it is much better.

im3w1l 8y ago
I can see cases where you want an anti-abuse system to "fail closed".
sethammons 8y ago
When I work on systems, I ask, "how does it scale and how does it fail?" Everything I work on considers handling of failure cases, especially 3rd party APIs. I'm surprised by how many people assume that the network will just work and their provider will be available.
necubi 8y ago
This is a hard call when the system is question is your anti-abuse provider. If you "fail open" when they're down, you risk allowing a flood of bad users during outages. Depending on your business, that may be much worse than having an outage yourself. (Disclaimer: I work for a smyte competitor).
- FrozenVoid 8y ago
  You can turn on something like manual verification of new users, an alternative security service, or just temporary halt new account registration. All of which don't result in system wide failure. Even a few abusive users is a small and temporary cost to absorb vs complete outage.
  - krainboltgreene 8y ago
    Ah, yes, just the casual manual verification of checks notes a fire hose of data.
  - BinaryIdiot 8y ago
    > just temporary halt new account registration I'm confused; how do you know these companies did not do that? In my experience even small companies will typically abstract away a third party service so that if it does fail it fails in an expected way. What you seem to be suggesting is that all of the companies failed in expected ways. I didn't see that in any of the articles. Did you have a source? I'm curious if it hit some harder than others.
  - Klathmon 8y ago
    >just temporary halt new account registration. That's an outage... >manual verification of new users So is that, unless you have tons of customer support reps sitting around doing nothing when shit hits the fan.
BinaryIdiot 8y ago
What you're suggesting is something for large companies. Most of these companies affected probably don't have time and resources to develop and pay for alternatives to every single third party service they require. And why would they? They have contracts you'd expect to be honored. Seems unrealistic especially since the majority of times when these services go down it's likely just a short time so they probably have abstracted around them to avoid unexpected failure and just have routine, down time failures.
msie 8y ago
I'm surprised that npm was affected so much since they provide service to so many people.

im3w1l 8y ago

I can see cases where you want an anti-abuse system to "fail closed".

sethammons 8y ago

When I work on systems, I ask, "how does it scale and how does it fail?" Everything I work on considers handling of failure cases, especially 3rd party APIs. I'm surprised by how many people assume that the network will just work and their provider will be available.

necubi 8y ago

This is a hard call when the system is question is your anti-abuse provider. If you "fail open" when they're down, you risk allowing a flood of bad users during outages. Depending on your business, that may be much worse than having an outage yourself. (Disclaimer: I work for a smyte competitor).

FrozenVoid 8y ago
You can turn on something like manual verification of new users, an alternative security service, or just temporary halt new account registration. All of which don't result in system wide failure. Even a few abusive users is a small and temporary cost to absorb vs complete outage.
- krainboltgreene 8y ago
  Ah, yes, just the casual manual verification of checks notes a fire hose of data.
- BinaryIdiot 8y ago
  > just temporary halt new account registration I'm confused; how do you know these companies did not do that? In my experience even small companies will typically abstract away a third party service so that if it does fail it fails in an expected way. What you seem to be suggesting is that all of the companies failed in expected ways. I didn't see that in any of the articles. Did you have a source? I'm curious if it hit some harder than others.
- Klathmon 8y ago
  >just temporary halt new account registration. That's an outage... >manual verification of new users So is that, unless you have tons of customer support reps sitting around doing nothing when shit hits the fan.

BinaryIdiot 8y ago

What you're suggesting is something for large companies. Most of these companies affected probably don't have time and resources to develop and pay for alternatives to every single third party service they require. And why would they? They have contracts you'd expect to be honored. Seems unrealistic especially since the majority of times when these services go down it's likely just a short time so they probably have abstracted around them to avoid unexpected failure and just have routine, down time failures.

msie 8y ago

I'm surprised that npm was affected so much since they provide service to so many people.