7 ms·

New TLS certificate for .herokuapp.com hostnames

• abhisuri97 6y ago
  Wow. I really hope they’ll get it done before the expiration date but I always thought they’d be renewing months in advance at minimum. Are they trying to negotiate something?
  • PedroBatista 6y ago
    Most likely. I'm pretty sure the money they want is something ridiculous because in their business wisdom they know Heroku has no power if they don't want to rock the boat. The question is: Is Heroku willing to rock the boat?
    • jrockway 6y ago
      Let's Encrypt issues wildcard certificates, so Heroku could easily pay $0 for one. I don't think they are negotiating anything.
      • tempay 6y ago
        The issue is that some of their customers have pinned DigiCert so they have two choices: * pay whatever DigiCert demands for a new certificate * accept that some of their customers will break Doing this two weeks before the old certificate expires puts them in a difficult situation for negotiating, especially now they've committed to getting a new DigiCert certificate.
        • yjftsjthsd-h 6y ago
          It seems like the solution to this is to implement both but charge customers to use the DigiCert chain. "Oh, you went and pinned something that you shouldn't have? That's fine; you can either fix it yourself or pay us to support your mistake."
  • tialaramex 6y ago
    Get what done? Heroku replaced their wildcard certificate in plenty of time but many customers do not anticipate anything changing ever and will fiercely resist this simple fact, so for those customers stuff blew up. Remember the Y2K problem is the result of software written not in 1901 or even just 1981 but even well into the 1990s with the calm certainty that all years begin 19xx. Heroku can try brown out policies, but this sort of customer intransigence is very difficult to defeat. The customer is quite certain they're right, who ever heard of "change" anyway? Everyone knows that the world is a flat plane, fixed in space, eternal and unchanging, this pinning rule I wrote in 2019 worked then, therefore it is still correct now.
    • tinus_hn 6y ago
      This is called ‘enterprise’ and everyone in it will tell you a million excuses why it’s the only way.
• jamil7 6y ago
  Heroku also doesn't enforce any verification that you own a domain name. Another user can simply add any domain they like to their app if you haven't claimed it by adding it to your app first. Regardless of ownership and you will no longer be able to add your own domain to your app getting a generic "domain is already in use" error. Happened to me a few years ago, had to reach out to support and prove I owned the domain. They made me verify I owned it and fixed it but said theres nothing they can do going forward. Granted it's a total edge case but was still an unnexpected experience, maybe it's fixed now who knows. Edit: Looks like this is fairly common on PAAS so my original comment isn't that relevant.
  • swiley 6y ago
    How does GitHub’s gh pages handle this? I don’t remember them doing anything either.
    • derision 6y ago
      Pretty sure you have to set some DNS records for gh pages
      • swiley 6y ago
        Yes you have to point a record at the gh pages if you want to use dns with it, but I don’t think their server checks for that.
        • awirth 6y ago
          Is it possible to subscribe to a firehose of .COM NS record updates through one of those fancy things like dnsdb? If so, perhaps there's an opportunity here for exploiting that race condition en-masse for services that support direct NS-style delegation, like netlify.
    • jamil7 6y ago
      I actually don't know as I haven't used it in a while. But I did just test netlify and the same issue exists there, domains need to be unique, perhaps there isn't a nice way of dealing with that edge case.
• _raul 6y ago
  There's more context in this Heroku Changelog https://devcenter.heroku.com/changelog-items/1813 https://devcenter.heroku.com/changelog-items/1813 "On Tuesday June 9th 2020 Heroku changed the certificate used for terminating TLS for built-in <appname>.herokuapp.com hostnames from a certificate issued by DigiCert to one issued by Starfield/AWS. This change was rolled back on June 10th because a small subset of Heroku customers had pinned apps to the DigiCert certificate or had apps that could not establish a chain of trust with the new certificate for other reasons. A new DigiCert-signed certificate will replace the current one before June 22nd (when it expires)."
• photonios 6y ago
  It already broke a few days ago: https://status.heroku.com/incidents/2045 https://status.heroku.com/incidents/2045
  • tuna-piano 6y ago
    Yep, we were down during that time. I don't know what Heroku is doing with this certificate thing, but seems like a mess.
    • jdxcode 6y ago
      If you're pinning the SSL cert then it's not something they can fix, it's something you need to change on your end.
      • photonios 6y ago
        We went down. We're not pinning the SSL cert. We're behind CloudFront, and CloudFront didn't trust the new cert and stopped forwarding traffic.
• bad_user 6y ago
  Why would you pin a certificate that you did not generate for a domain that's not yours?
  • godzillabrennus 6y ago
    My guess is that it is some larger corporate client with a middleware app they pinned the cert into. An app they built on Heroku hurriedly because it was fast and cheap to get started and they didn’t expect to need to scale. Then as can happen it became important and they scaled anyway. They probably lost the talent that built it so they don’t even remember how it works. All of this is assumption based on how they seem to buy enough compute at Heroku to have sway over rolling back this change of cert provider.
    • yjftsjthsd-h 6y ago
      Ugh, yes, I assure you that some corporate clients will even try to pin the actual leaf certificate; pinning an intermediate or root is almost good behavior for them. (Honestly, the number of times I had to tell our support people that no, we would not support customers trying to pin our AWS-issued certificates, and no, I couldn't promise to notify them even if I wanted to since AWS could just rotate them at will...)
  • teh_klev 6y ago
    Once upon a time a UK model train manufacturer had a warning inside the box of your new loco or rolling stock that said something like "Do not dismantle out of idle curiosity". Some people despite the warnings and consequences can't help themselves :)
    • pbhjpbhj 6y ago
      What a clever way to make more sales. 1) people are dismantling goods, then ringing for advice but being embarrassed they broke it, then rebuying when we refuse advice, 2) tell people not to dismantle it, they'll get curious and do the opposite, 3) ... 4) pro-fit!
  • 6y ago
• jrockway 6y ago
  Something that's nice about Let's Encrypt is that it forces you to change something every few months. After the first couple months, you'll probably get your issues worked out. If you just change certs every few years, then every few years you have some sort of disaster because of the "well we fixed it, we don't have to worry for two years" effect. A broader lesson is the importance of "trying out" rare events, even before that rare event actually happens. If depends on a service with a certain SLA, it's pretty dangerous when that service has 100% uptime. You never get to see what happens when it does go down, which it did promise you it will. Some people track their error budget, and at the end of the accounting period, break their service in accordance with the SLA. Then you get to see what happens when it does go down. (Ref: https://queue.acm.org/detail.cfm?id=2371516 https://queue.acm.org/detail.cfm?id=2371516)
  • tialaramex 6y ago
    Although, speaking of Let's Encrypt, there will be a series of disruptive events over the next 18 months or so. * Soon (although when exactly I'm not sure because it has been delayed at least once) the Let's Encrypt systems will tell compliant ACME clients that the "correct" intermediate is Let's Encrypt's ISRG-signed X3 intermediate. This is a different certificate for the same X3 private key you're used to but not signed by the same trust root. If you use a correct client and have done things properly, this may cut off TLS clients for your systems that don't trust ISRG (the charity which runs Let's Encrypt). Six year old Android phones, the Windows XP system you know should have been retired, a VoIP desk phone running out-of-date firmware, stuff like that. * In March 2021 the X3 Intermediate expires. If your certificate software was not compliant with ACME, or you manually overrode it to use the old certificates to avoid the problem in the previous item, things break now. More things, and worse. Although... * Maybe before March 2021 the Let's Encrypt systems stop issuing from those soon-to-be-obsolete Intermediates and use newer ones instead perhaps named Y3 and Y4. In this case if you've jury rigged things (in an ACME non-compliant way) to keep using the old X3 intermediate that'll break suddenly after your renewal. Common web browsers may not trust the nonsense you're emitting, exactly which browsers break may vary depending on exactly what stupid things you did, but chances are you haven't tested and don't know. If you are using a compliant client then modern browsers are all fine, but archaic stuff breaks suddenly. * In September 2021 the DST Root X3 root expires. If you have somehow clung on to trust via this root, whether through your own effort or via trust path discovery code inside client systems, that goes away instantly. Any systems that don't trust ISRG will refuse to trust your certificates, no matter how often you re-issue them and reconfigure things, those clients themselves need updating urgently and you probably have no way to do that. Oops.