5 ms·

Let's Encrypt: Active Incident, DNS errors causing service disruption

97 points by antouank 11y ago

aparadja 11y ago
Isn't the "DNS errors causing service disruption" an old incident from April 6th? There is something strange going on at the moment, though, but not sure it has to do with DNS. I actually tried set up my first cert about 10 hours ago with Let's Encrypt, and got a variety of ungraceful errors with the same configuration. Only one was related to DNS. Most were code 500 from the api servers.
- viraptor 11y ago
  Their status page is broken. You can see the actual issue on the history page instead, or specifically https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/570fe5aafe92599160001098 https://letsencrypt.status.io/pages/incident/55957a99e800baa... April 14, 2016 12:47PM MDT April 14, 2016 6:47PM UTC [Investigating] We have noticed an increased number of errors. We are investigating now. April 14, 2016 1:21PM MDT April 14, 2016 7:21PM UTC [Resolved] Systems have fully recovered and all services appear to be operating nominally. Cause seems to have been a transient hardware failure and further investigation is under way. Or at least the most recent issue... they both marked it as resolved and "further investigation is under way". Should be "investigating/monitoring" instead?
- antouank 11y ago
  I don't know what the exact title should be. If you see here, https://letsencrypt.status.io/pages/history/55957a99e800baa4470002da https://letsencrypt.status.io/pages/history/55957a99e800baa4... , there's a "Investigating interruption of issuances" event from yesterday. I had problem with getting certificates yesterday. Seems to be working today, but I just thought it's interesting to see what happened.
giancarlostoro 11y ago
I figure there should exist alternatives to LE, otherwise they're a pretty big target. If anything too harmful happened to LE what would it's users do? I don't know enough about it, but right now they're going to be the only player in the free SSL certs market which puts them in a scary enough location.
- ultramancool 11y ago
  StartSSL and CAcert have existed long before Let's Encrypt was even started.
  - RadioactiveMan 11y ago
    If CAcert isn't usually trusted and StartSSL is not free for commercial use, are they really alternatives?
  - pmlnr 11y ago
    StartSSL is a b* to use, the free tier. I can't renew a cert before it expires.
    - breser 11y ago
      That's not exactly true. You can renew it before it expires. You just can't renew it until it is nearly expired. I think it's like a 2 week window or something like that. The bigger complaint about StartSSL should be that if you need to revoke the certificate you have to pay for it. For a free service though I don't think their requirements are too terrible.
      - geofft 11y ago
        That wouldn't be such a problem given that most SSL stacks (correctly, IMO) ignore revocation checking... except that you can't get a new cert without revoking the old one. It would otherwise be reasonable to determine you don't care about getting it revoked (and therefore don't want to fund the revocation infrastructure, like a highly-available OCSP responder) but you just want a new cert.
  - nly 11y ago
    And WoSign.
    - niersi 11y ago
      Right, and CNNIC.
nereid666 11y ago
Is causing a problem to the pages protected by certificate due to OCSP error? Or the impact is only afecting issue and renewals?
jldugger 11y ago
By my estimation, Lets Encrypt has at least 16 million dollars in annual sponsorship.[1][2] That's quite a bit of money, but I wonder how much of it goes to operations, versus other overhead of various kinds (legal, marketing, administrative). Many of our clients are excited about LE, and as we figure out how to support it without disrupting our infrastructure too much, it's concerning to imagine that there's substantial daily risk that 4-5 certs will fail to renew. I suppose since it's free and automated, you just renew a month earlier than required. [1] https://letsencrypt.org/sponsors/ https://letsencrypt.org/sponsors/ [2] https://letsencrypt.org/become-a-sponsor/ https://letsencrypt.org/become-a-sponsor/
- JonathonW 11y ago
  That's their idea-- the reference client (by default) will renew any certificates expiring within 30 days, and they recommend you script it to run at least daily. That gives it many opportunities to retry in case of network problems or server outages (on your end or on theirs).
mholt 11y ago
I believe this issue was originally reported via Twitter, here: https://twitter.com/_rsc/status/717777241296543744 https://twitter.com/_rsc/status/717777241296543744 -- and that for the most part things are working, but there are just occasional errors.
iamgopal 11y ago
I have wondered many times, so let me just ask it, Why not platform providers like Microsoft, Google or Apple are also secure certificate providers ? Is not will it be immensely easier for both party ?
- ikeboy 11y ago
  Amazon does, for one.
roblooman 11y ago
Guys i wanna add something here that i think we should all understand...perspective wise...1 - Encryption should NOT be an open sourced task -- NEVER, --and before the flame wall of comments comes down on me, ask your self and honest answer why would YOU trust your data security to anything someone else could easily ( and already ) knows how to break --ssl, or any other algo--- that has open sourced code! ( i know the idea of open sourced encryption is great i even use it for some tasks, but relying on it for everything is LITERALLY telling the burgler out side your house where spare set of keys are to get in while your away!!! .... and 2 -- if correctly implemented in the first place data security should never be trusted to a STATIC library of tools, just like learning (martial arts) you learn this skill to be useful should you be attacked to defend your self, why would you rely on one punch in ANY scenario where you have to dynamicaly defend yourself,,, the one strategy would never work and more over you will be laying on the ground bleeding and your wallet gone....EVERY time. SSL is great only if it is regularly changed every 24 hours, some thing can do that....i have personally seem people retain the certs for over several years,,,not to mention the other static library of encryption alternatives that are basically trying to bail water out of a sinking boat with a bucket with a whole in it... there are better alternatives and one that are unbreakable. i know i invented one - roblooman2010@gmail.com
- zmarty 11y ago
  Is there a Report Spam button anywhere?
  - maxerickson 11y ago
    If you click on "5 minutes ago" there is a flag link/button. That comment isn't really worth flagging though, it's already been downvoted a couple of times which will sort it to the bottom of the page, and it isn't egregious, just odd. If it was part of a pattern, then it would be worth flagging. It takes 500 karma to be able to downvote comments.
    - roblooman 11y ago
      ok why do that? think of it as a means to learn and then proceed from there, down voting is just a means to lower an opinion because you don't agree, fine don't agree, but your opinion is no better they are all like asses, we all have them.
esterly 11y ago
We created certs with Let's Encrypt today, worked like a champ.
pg_is_a_butt 11y ago
but but but but, a single point of failure NEVER fails... it's too big to fail... it can't fail. this isn't happening.
mehrzad 11y ago
When will GitHub Pages support this?

97 points by antouank 11y ago

Isn't the "DNS errors causing service disruption" an old incident from April 6th? There is something strange going on at the moment, though, but not sure it has to do with DNS. I actually tried set up my first cert about 10 hours ago with Let's Encrypt, and got a variety of ungraceful errors with the same configuration. Only one was related to DNS. Most were code 500 from the api servers.

viraptor 11y ago
Their status page is broken. You can see the actual issue on the history page instead, or specifically https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/570fe5aafe92599160001098 https://letsencrypt.status.io/pages/incident/55957a99e800baa... April 14, 2016 12:47PM MDT April 14, 2016 6:47PM UTC [Investigating] We have noticed an increased number of errors. We are investigating now. April 14, 2016 1:21PM MDT April 14, 2016 7:21PM UTC [Resolved] Systems have fully recovered and all services appear to be operating nominally. Cause seems to have been a transient hardware failure and further investigation is under way. Or at least the most recent issue... they both marked it as resolved and "further investigation is under way". Should be "investigating/monitoring" instead?
antouank 11y ago
I don't know what the exact title should be. If you see here, https://letsencrypt.status.io/pages/history/55957a99e800baa4470002da https://letsencrypt.status.io/pages/history/55957a99e800baa4... , there's a "Investigating interruption of issuances" event from yesterday. I had problem with getting certificates yesterday. Seems to be working today, but I just thought it's interesting to see what happened.

giancarlostoro 11y ago

I figure there should exist alternatives to LE, otherwise they're a pretty big target. If anything too harmful happened to LE what would it's users do? I don't know enough about it, but right now they're going to be the only player in the free SSL certs market which puts them in a scary enough location.

ultramancool 11y ago
StartSSL and CAcert have existed long before Let's Encrypt was even started.
- RadioactiveMan 11y ago
  If CAcert isn't usually trusted and StartSSL is not free for commercial use, are they really alternatives?
- pmlnr 11y ago
  StartSSL is a b* to use, the free tier. I can't renew a cert before it expires.
  - breser 11y ago
    That's not exactly true. You can renew it before it expires. You just can't renew it until it is nearly expired. I think it's like a 2 week window or something like that. The bigger complaint about StartSSL should be that if you need to revoke the certificate you have to pay for it. For a free service though I don't think their requirements are too terrible.
    - geofft 11y ago
      That wouldn't be such a problem given that most SSL stacks (correctly, IMO) ignore revocation checking... except that you can't get a new cert without revoking the old one. It would otherwise be reasonable to determine you don't care about getting it revoked (and therefore don't want to fund the revocation infrastructure, like a highly-available OCSP responder) but you just want a new cert.
- nly 11y ago
  And WoSign.
  - niersi 11y ago
    Right, and CNNIC.

nereid666 11y ago

Is causing a problem to the pages protected by certificate due to OCSP error? Or the impact is only afecting issue and renewals?

jldugger 11y ago

By my estimation, Lets Encrypt has at least 16 million dollars in annual sponsorship.[1][2] That's quite a bit of money, but I wonder how much of it goes to operations, versus other overhead of various kinds (legal, marketing, administrative). Many of our clients are excited about LE, and as we figure out how to support it without disrupting our infrastructure too much, it's concerning to imagine that there's substantial daily risk that 4-5 certs will fail to renew. I suppose since it's free and automated, you just renew a month earlier than required. [1] https://letsencrypt.org/sponsors/ https://letsencrypt.org/sponsors/ [2] https://letsencrypt.org/become-a-sponsor/ https://letsencrypt.org/become-a-sponsor/

JonathonW 11y ago
That's their idea-- the reference client (by default) will renew any certificates expiring within 30 days, and they recommend you script it to run at least daily. That gives it many opportunities to retry in case of network problems or server outages (on your end or on theirs).

mholt 11y ago

I believe this issue was originally reported via Twitter, here: https://twitter.com/_rsc/status/717777241296543744 https://twitter.com/_rsc/status/717777241296543744 -- and that for the most part things are working, but there are just occasional errors.

iamgopal 11y ago

I have wondered many times, so let me just ask it, Why not platform providers like Microsoft, Google or Apple are also secure certificate providers ? Is not will it be immensely easier for both party ?

ikeboy 11y ago
Amazon does, for one.

roblooman 11y ago

Guys i wanna add something here that i think we should all understand...perspective wise...1 - Encryption should NOT be an open sourced task -- NEVER, --and before the flame wall of comments comes down on me, ask your self and honest answer why would YOU trust your data security to anything someone else could easily ( and already ) knows how to break --ssl, or any other algo--- that has open sourced code! ( i know the idea of open sourced encryption is great i even use it for some tasks, but relying on it for everything is LITERALLY telling the burgler out side your house where spare set of keys are to get in while your away!!! .... and 2 -- if correctly implemented in the first place data security should never be trusted to a STATIC library of tools, just like learning (martial arts) you learn this skill to be useful should you be attacked to defend your self, why would you rely on one punch in ANY scenario where you have to dynamicaly defend yourself,,, the one strategy would never work and more over you will be laying on the ground bleeding and your wallet gone....EVERY time. SSL is great only if it is regularly changed every 24 hours, some thing can do that....i have personally seem people retain the certs for over several years,,,not to mention the other static library of encryption alternatives that are basically trying to bail water out of a sinking boat with a bucket with a whole in it... there are better alternatives and one that are unbreakable. i know i invented one - roblooman2010@gmail.com

zmarty 11y ago
Is there a Report Spam button anywhere?
- maxerickson 11y ago
  If you click on "5 minutes ago" there is a flag link/button. That comment isn't really worth flagging though, it's already been downvoted a couple of times which will sort it to the bottom of the page, and it isn't egregious, just odd. If it was part of a pattern, then it would be worth flagging. It takes 500 karma to be able to downvote comments.
  - roblooman 11y ago
    ok why do that? think of it as a means to learn and then proceed from there, down voting is just a means to lower an opinion because you don't agree, fine don't agree, but your opinion is no better they are all like asses, we all have them.

esterly 11y ago

We created certs with Let's Encrypt today, worked like a champ.

pg_is_a_butt 11y ago

but but but but, a single point of failure NEVER fails... it's too big to fail... it can't fail. this isn't happening.

mehrzad 11y ago

When will GitHub Pages support this?