5 ms·

It's a good thing we have ever-shrinking certificate lifetimes and automation never breaks. That's what I've been told, anyway.

bigbuppo 8mo ago
Yeah, this could end up as the actual root cause of The Great Oops that I've been raving about for years. And Google probably would be the right company to fuck it up in the worst way possible since Google Knows Best In All Situations.
- ocdtrekkie 8mo ago
  I can't wait for the Great Oops.
- LPisGood 8mo ago
  Please tell me more about The Great Oops
  - bigbuppo 8mo ago
    It's inevitable that one of the major cloud providers will irrecoverably delete all customer data with one single fat-fingered command. Though in google's case I'll also consider the prophecy to be fulfilled if they delete their own data. It will forever be known as The Great Oops.
    - tokyobreakfast 8mo ago
      That seems unlikely. Is Google run by one Homer Simpson?
      - bigbuppo 8mo ago
        Yes.
    - Arainach 8mo ago
      It's not inevitable, it's essentially impossible. There are a few things that can cause tremendously widespread outages, essentially all of them network configuration changes. Actually deleting customer data is dramatically more difficult to the point of impossible - there are so many different services in so many different locations with so many layers of access control. There is no "one command" that can do such a thing - at the scale of a worldwide network of data centers there is no "rm -rf /".
      - ocdtrekkie 8mo ago
        Ah, but you fail to account for Google's incredible knack for building tools designed to do things at scale. Or put AI in things that don't need it. The possibility Google will either manage to unleash a malicious AI on their infrastructure and/or develop a way to destroy a lot of data at scale quite efficiently or some combination of the two is far from zero. Bear in mind, this "Little Oops" should also have been impossible: https://www.techspot.com/news/103207-google-reveals-how-blank-parameter-mistake-erased-135.html https://www.techspot.com/news/103207-google-reveals-how-blan...
        Arainach 8mo ago
        .....no? "We deployed this private cloud with a missing parameter and it wasn't caught" is as different from "we wiped out all customer data" as hello world is from Kubernetes. No one promised this "should be impossible". Did you confuse "we'll take steps to ensure this never happens again"?
        ocdtrekkie 8mo ago
        It's pretty much half the puzzle actually. You contend there's no global rm rf for a global cloud provider, but clearly a missing parameter can rm rf a customer in an irrecoverable manner. The only half you're missing is... how every major cloud outage happens today... a bad configuration update. These companies have hundreds of thousands of servers, but they also use orchestration tools to distribute sets of changes to all of them. You only need a command to rm rf one box, if you are distributing that command to every box. Now sure, there are tons of security precautions and checks and such to prevent this! But pretending it's impossible is delusional. People do stupid stuff, at scale, every day. The most likely scenario is a zero day in an environment necessitating an extremely rapid global rollout, combined with a plain, simple error.
    - JyB 8mo ago
      I don’t know if you’re being serious but that’s laughable
      - SchemaLoad 8mo ago
        The idea that all customer data will be deleted is far fetched, but I feel like there have been some massive incidents. Crowdstrike comes to mind, but I feel its entirely possible that Apple/Google/etc could push out some kind of config update which bricks phones in a way they are unable to download another update to fix them. Though I'm sure the major players are all over this risk which is why it hasn't happened.
        aragilar 8mo ago
        Google wiped all of UniSuper not too long ago by mistake, I don't see why such a occurrence couldn't happen more widely.
- tokyobreakfast 8mo ago
  I don't subscribe to your newsletter. What about the Oops?
- _qib3 8mo ago
  Do you have a blog post on the oops? I'd love to read it.
msie 8mo ago
I was thinking about the time some software influencer said that if you are afraid to deploy on Friday then there's something wrong with you. Eff that! Murphy's Law! (allen holub - https://x.com/allenholub/status/1637111242610610182 https://x.com/allenholub/status/1637111242610610182)
- tzs 8mo ago
  I often deployed on Friday evening. Several factors contributed to this decision. 1. Sales volume was lowest on weekends so if something went wrong it would affect fewer customers. 2. If something went wrong and I needed to revert, nobody was at work on weekends so it would not disrupt coworkers. 3. I always made it so reverting would be easy. 4. Most of my weekends were just relaxing at home, mostly doing online stuff (games, reading, videos) or doing offline stuff at my computer (programming my personal projects). It wasn't much of a bother at all to have an ssh open to something at work monitoring the new deployment for problems for the rest of Friday night and Saturday.
  - deleted 8mo ago
    [deleted]
jsheard 8mo ago
There's at least five free ACME CAs, with failover it doesn't matter all that much if one of them falls over. If all of them fall over at once there's probably a more pressing issue like nuclear holocaust or alien invasion going on.
- tokyobreakfast 8mo ago
  How many servers are set up with CA redundancy? I've yet to see one let alone hear of this practice.
  - jsheard 8mo ago
    For one, Cloudflare uses four different CAs almost interchangeably. Caddy also makes it easy to configure ACME failover if you're self-hosting, and defaults to using two different CAs if you don't specify any. Frankly even with no CA redundancy, downtime would have to drag on for weeks to actually disrupt renewals. ACME certs usually get rotated after about 2/3rds of their duration has expired, so the upcoming 45 day certs will still have about 15 days of wiggle room.
    - thayne 8mo ago
      They aren't all drop in replacements for each other though. For example, Let's Encrypt offers free wildcard certs (with dns verification), but for ZeroSSL, it requires a paid subscription.
      - jsheard 8mo ago
        ZeroSSL is weird, if you use their classic non-ACME interface then the free tier is indeed limited to 3 active certs which can't be wildcards, but if you use ACME then there's no limits and wildcards are allowed. https://zerossl.com/documentation/acme/ https://zerossl.com/documentation/acme/ > By using ZeroSSL's ACME feature, you will be able to generate an unlimited amount of 90-day SSL certificates at no charge, also supporting multi-domain certificates and wildcards.
    - 8mo ago

bigbuppo 8mo ago

Yeah, this could end up as the actual root cause of The Great Oops that I've been raving about for years. And Google probably would be the right company to fuck it up in the worst way possible since Google Knows Best In All Situations.

ocdtrekkie 8mo ago
I can't wait for the Great Oops.
LPisGood 8mo ago
Please tell me more about The Great Oops
- bigbuppo 8mo ago
  It's inevitable that one of the major cloud providers will irrecoverably delete all customer data with one single fat-fingered command. Though in google's case I'll also consider the prophecy to be fulfilled if they delete their own data. It will forever be known as The Great Oops.
  - tokyobreakfast 8mo ago
    That seems unlikely. Is Google run by one Homer Simpson?
    - bigbuppo 8mo ago
      Yes.
  - Arainach 8mo ago
    It's not inevitable, it's essentially impossible. There are a few things that can cause tremendously widespread outages, essentially all of them network configuration changes. Actually deleting customer data is dramatically more difficult to the point of impossible - there are so many different services in so many different locations with so many layers of access control. There is no "one command" that can do such a thing - at the scale of a worldwide network of data centers there is no "rm -rf /".
    - ocdtrekkie 8mo ago
      Ah, but you fail to account for Google's incredible knack for building tools designed to do things at scale. Or put AI in things that don't need it. The possibility Google will either manage to unleash a malicious AI on their infrastructure and/or develop a way to destroy a lot of data at scale quite efficiently or some combination of the two is far from zero. Bear in mind, this "Little Oops" should also have been impossible: https://www.techspot.com/news/103207-google-reveals-how-blank-parameter-mistake-erased-135.html https://www.techspot.com/news/103207-google-reveals-how-blan...
      - Arainach 8mo ago
        .....no? "We deployed this private cloud with a missing parameter and it wasn't caught" is as different from "we wiped out all customer data" as hello world is from Kubernetes. No one promised this "should be impossible". Did you confuse "we'll take steps to ensure this never happens again"?
        ocdtrekkie 8mo ago
        It's pretty much half the puzzle actually. You contend there's no global rm rf for a global cloud provider, but clearly a missing parameter can rm rf a customer in an irrecoverable manner. The only half you're missing is... how every major cloud outage happens today... a bad configuration update. These companies have hundreds of thousands of servers, but they also use orchestration tools to distribute sets of changes to all of them. You only need a command to rm rf one box, if you are distributing that command to every box. Now sure, there are tons of security precautions and checks and such to prevent this! But pretending it's impossible is delusional. People do stupid stuff, at scale, every day. The most likely scenario is a zero day in an environment necessitating an extremely rapid global rollout, combined with a plain, simple error.
  - JyB 8mo ago
    I don’t know if you’re being serious but that’s laughable
    - SchemaLoad 8mo ago
      The idea that all customer data will be deleted is far fetched, but I feel like there have been some massive incidents. Crowdstrike comes to mind, but I feel its entirely possible that Apple/Google/etc could push out some kind of config update which bricks phones in a way they are unable to download another update to fix them. Though I'm sure the major players are all over this risk which is why it hasn't happened.
      - aragilar 8mo ago
        Google wiped all of UniSuper not too long ago by mistake, I don't see why such a occurrence couldn't happen more widely.
tokyobreakfast 8mo ago
I don't subscribe to your newsletter. What about the Oops?
_qib3 8mo ago
Do you have a blog post on the oops? I'd love to read it.

msie 8mo ago

I was thinking about the time some software influencer said that if you are afraid to deploy on Friday then there's something wrong with you. Eff that! Murphy's Law! (allen holub - https://x.com/allenholub/status/1637111242610610182 https://x.com/allenholub/status/1637111242610610182)

tzs 8mo ago
I often deployed on Friday evening. Several factors contributed to this decision. 1. Sales volume was lowest on weekends so if something went wrong it would affect fewer customers. 2. If something went wrong and I needed to revert, nobody was at work on weekends so it would not disrupt coworkers. 3. I always made it so reverting would be easy. 4. Most of my weekends were just relaxing at home, mostly doing online stuff (games, reading, videos) or doing offline stuff at my computer (programming my personal projects). It wasn't much of a bother at all to have an ssh open to something at work monitoring the new deployment for problems for the rest of Friday night and Saturday.
- deleted 8mo ago
  [deleted]

jsheard 8mo ago

There's at least five free ACME CAs, with failover it doesn't matter all that much if one of them falls over. If all of them fall over at once there's probably a more pressing issue like nuclear holocaust or alien invasion going on.

tokyobreakfast 8mo ago
How many servers are set up with CA redundancy? I've yet to see one let alone hear of this practice.
- jsheard 8mo ago
  For one, Cloudflare uses four different CAs almost interchangeably. Caddy also makes it easy to configure ACME failover if you're self-hosting, and defaults to using two different CAs if you don't specify any. Frankly even with no CA redundancy, downtime would have to drag on for weeks to actually disrupt renewals. ACME certs usually get rotated after about 2/3rds of their duration has expired, so the upcoming 45 day certs will still have about 15 days of wiggle room.
  - thayne 8mo ago
    They aren't all drop in replacements for each other though. For example, Let's Encrypt offers free wildcard certs (with dns verification), but for ZeroSSL, it requires a paid subscription.
    - jsheard 8mo ago
      ZeroSSL is weird, if you use their classic non-ACME interface then the free tier is indeed limited to 3 active certs which can't be wildcards, but if you use ACME then there's no limits and wildcards are allowed. https://zerossl.com/documentation/acme/ https://zerossl.com/documentation/acme/ > By using ZeroSSL's ACME feature, you will be able to generate an unlimited amount of 90-day SSL certificates at no charge, also supporting multi-domain certificates and wildcards.
  - 8mo ago