6 ms·

Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC,

Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one minute and 100% untrustworthy the next minute. Having the failure rate ramp up slowly would give advance warning before everything has gone completely pear-shaped. In the case of Ericsson, this might have allowed an emergency certificate update before all the O2 systems could no longer be automatically updated. Once your network is completely down, bringing it back up remotely is hard.

deleted 8y ago
[deleted]
codingminds 8y ago
Certificate monitoring would be a good idea.
gred 8y ago
I think this could also be a good idea for phasing out public APIs -- instead of just taking an API offline, start to fail requests early at low probability, ramping the probability up to 100% over the course of a month or so.
- hinkley 8y ago
  Adding delays to responses can also work too. Years ago, during the tail end of the Age of XML, I had a conversation with one of the admins over at W3C. To make your app work you're supposed to cache all schemas locally but an inordinate amount of people don't. Their apps will be slow because of that, but work anyway. So tons of traffic goes to w3.org. They can't remove those schemas because then nobody can use them the right way (download them 1x and keep the result). But how do you get people's attention? I pointed out to him that a human downloading the schema isn't going to notice if the request takes .3 seconds or 3 seconds, but someone loading that schema 100x in a loop is sure going to be incentivized to figure out where the extra 5 minutes of processing time came from. He allowed that I might be onto something there but I never heard back about whether he tried it. You want to kill off an API, assign it resources (hardware, bandwidth) that get progressively smaller, or add artificial waits that get progressively bigger. Depending on how horizontally scalable your app is (is it easier to add a sleep or to pull servers out of the cluster?).
- anilakar 8y ago
  How long would it take before developers started wrapping their API and CDN calls in rapid-firing loops because of this?
  - doesnt_know 8y ago
    It already exists and has borrowed the name "resilience engineering" from the construction and engineering fields. Netflix has some interesting blog posts on how they deal with transient faults and resilience in general. Implementing concepts like circuit breakers. Have a search for libraries in your favorite language, I'm sure something will already exist. I've personally used Polly in .NET. https://github.com/App-vNext/Polly https://github.com/App-vNext/Polly
olliej 8y ago
That can be achieved by having multiple carts issued with slightly different expiries. Without weakening enforcement of all other certs (incl. those belonging to companies that manage them properly). This is immediately results in the “it took us more than 24 hours to work out everything was failing, it needs to be 72 hours”. The best thing about that is that it results in a self extending policy - the longer it takes to age out an expired cert, the longer it takes for the failure volume to become noticeable, and then the longer to work out that the problem is cert expiry, and so the more likely they’ll need an extension (again). The correct solution for this problem is for people to correctly manage their certs, which they should be doing anyway because the private keys are sufficiently important you should know exactly which are in use and where they are in use.
desc 8y ago
No, this is not a good idea. Failures which are random are harder to diagnose. Something which works or not is much quicker to track down. The simple fact is that this sort of problem should've been dealt wiht a lot sooner, and a failure to do so is sheer incompetence. Where that incompetence lies, is up for debate. But given how these things usually run, it lies some levels above the people who didn't have the time to properly follow a poorly-described process in the face of demands to do other stuff instead.
- mhandley 8y ago
  Of course certificates should be managed properly, so this sort of thing should never happen. My suggestion is not instead of that, but in addition. The question is when your certificate management process has failed, what then? It's defense in depth. In this case the certs expiry seems to have rendered it hard or impossible to remotely resurrect the systems. That's a very brutal failure when your whole network goes down in one go. Having random failures, with alarms going off before everything is down, is definitely better that having alarms going off with the entire network already down.
_jal 8y ago
Maybe it would be a good idea for people to put an expiration date check in whatever they use to monitor the rest of the machines. Or put the expiry date on a shared calendar. We do both, and they start alerting two months in advance. This isn't rocket surgery, new, or much different than remembering people's birthdays. But a surprising number of technical people are simply incapable of managing events farther out than next week, and a surprising number of their managers are similarly incapable of making sure someone treats it as important. Makes me almost want to start a betting pool on who will be the next HugeCo to down themselves this way. Edit: And let me join the chorus in saying deliberately inducing random errors as a way to draw attention is a terrible idea.
- slimsag 8y ago
  > whatever they use to monitor the rest of the machines. "Ah, are we supposed to do that too?"
megy 8y ago
Are you insane? That would be fun to diagnose.
- drannex 8y ago
  While I can totally see why this would be annoying to diagnose, throwing a specific form of a certificate expired error code could mitigate this issue.
closeparen 8y ago
At this point I wouldn’t even contemplate using X.509 without a fully automated PKI issuing ephemeral certs. Paging humans is not the answer.

deleted 8y ago

[deleted]

codingminds 8y ago

Certificate monitoring would be a good idea.

gred 8y ago

I think this could also be a good idea for phasing out public APIs -- instead of just taking an API offline, start to fail requests early at low probability, ramping the probability up to 100% over the course of a month or so.

hinkley 8y ago
Adding delays to responses can also work too. Years ago, during the tail end of the Age of XML, I had a conversation with one of the admins over at W3C. To make your app work you're supposed to cache all schemas locally but an inordinate amount of people don't. Their apps will be slow because of that, but work anyway. So tons of traffic goes to w3.org. They can't remove those schemas because then nobody can use them the right way (download them 1x and keep the result). But how do you get people's attention? I pointed out to him that a human downloading the schema isn't going to notice if the request takes .3 seconds or 3 seconds, but someone loading that schema 100x in a loop is sure going to be incentivized to figure out where the extra 5 minutes of processing time came from. He allowed that I might be onto something there but I never heard back about whether he tried it. You want to kill off an API, assign it resources (hardware, bandwidth) that get progressively smaller, or add artificial waits that get progressively bigger. Depending on how horizontally scalable your app is (is it easier to add a sleep or to pull servers out of the cluster?).
anilakar 8y ago
How long would it take before developers started wrapping their API and CDN calls in rapid-firing loops because of this?
- doesnt_know 8y ago
  It already exists and has borrowed the name "resilience engineering" from the construction and engineering fields. Netflix has some interesting blog posts on how they deal with transient faults and resilience in general. Implementing concepts like circuit breakers. Have a search for libraries in your favorite language, I'm sure something will already exist. I've personally used Polly in .NET. https://github.com/App-vNext/Polly https://github.com/App-vNext/Polly

olliej 8y ago

That can be achieved by having multiple carts issued with slightly different expiries. Without weakening enforcement of all other certs (incl. those belonging to companies that manage them properly). This is immediately results in the “it took us more than 24 hours to work out everything was failing, it needs to be 72 hours”. The best thing about that is that it results in a self extending policy - the longer it takes to age out an expired cert, the longer it takes for the failure volume to become noticeable, and then the longer to work out that the problem is cert expiry, and so the more likely they’ll need an extension (again). The correct solution for this problem is for people to correctly manage their certs, which they should be doing anyway because the private keys are sufficiently important you should know exactly which are in use and where they are in use.

desc 8y ago

No, this is not a good idea. Failures which are random are harder to diagnose. Something which works or not is much quicker to track down. The simple fact is that this sort of problem should've been dealt wiht a lot sooner, and a failure to do so is sheer incompetence. Where that incompetence lies, is up for debate. But given how these things usually run, it lies some levels above the people who didn't have the time to properly follow a poorly-described process in the face of demands to do other stuff instead.

mhandley 8y ago
Of course certificates should be managed properly, so this sort of thing should never happen. My suggestion is not instead of that, but in addition. The question is when your certificate management process has failed, what then? It's defense in depth. In this case the certs expiry seems to have rendered it hard or impossible to remotely resurrect the systems. That's a very brutal failure when your whole network goes down in one go. Having random failures, with alarms going off before everything is down, is definitely better that having alarms going off with the entire network already down.

_jal 8y ago

Maybe it would be a good idea for people to put an expiration date check in whatever they use to monitor the rest of the machines. Or put the expiry date on a shared calendar. We do both, and they start alerting two months in advance. This isn't rocket surgery, new, or much different than remembering people's birthdays. But a surprising number of technical people are simply incapable of managing events farther out than next week, and a surprising number of their managers are similarly incapable of making sure someone treats it as important. Makes me almost want to start a betting pool on who will be the next HugeCo to down themselves this way. Edit: And let me join the chorus in saying deliberately inducing random errors as a way to draw attention is a terrible idea.

slimsag 8y ago
> whatever they use to monitor the rest of the machines. "Ah, are we supposed to do that too?"

megy 8y ago

Are you insane? That would be fun to diagnose.

drannex 8y ago
While I can totally see why this would be annoying to diagnose, throwing a specific form of a certificate expired error code could mitigate this issue.

closeparen 8y ago

At this point I wouldn’t even contemplate using X.509 without a fully automated PKI issuing ephemeral certs. Paging humans is not the answer.