9 ms·

GitHub's User Content certificate has expired

126 points by GOATS- 4y ago

GOATS- 4y ago
This also applies to their avatars subdomain, causing them not to load anymore.
- sha-3 4y ago
  Are you sure? avatars.githubusercontent.com works fine for me.
  - guessmyname 4y ago
    > Are you sure? avatars.githubusercontent.com works fine for me. That’s because they already resolved the issue → https://www.githubstatus.com/incidents/x7njwb481j9b https://www.githubstatus.com/incidents/x7njwb481j9b This is what I saw an hour ago: $ echo | openssl s_client -connect avatars.githubusercontent.com:443 CONNECTED(00000005) depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA verify return:1 depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1 verify return:1 depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io >>> verify error:num=10:certificate has expired notAfter=Mar 21 23:59:59 2023 GMT verify return:1 depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io notAfter=Mar 21 23:59:59 2023 GMT verify return:1
dz0ny 4y ago
Also serving wrong certificates for a lot of content domains. https://news.ycombinator.com/item?id=35295191 https://news.ycombinator.com/item?id=35295191
koolba 4y ago
The cert for objects.githubusercontent.com has also expired: $ openssl s_client -connect objects.githubusercontent.com:443 CONNECTED(00000005) depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA verify return:1 depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1 verify return:1 depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io verify error:num=10:certificate has expired notAfter=Mar 21 23:59:59 2023 GMT verify return:1 depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io notAfter=Mar 21 23:59:59 2023 GMT verify return:1 --- Certificate chain 0 s:C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io i:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1 1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1 i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA What are the odds this happens the same day they rotate their SSH keys?
- brandur 4y ago
  It's also the domain used for releases and other artifacts (after a redirect from github.com). There's going to be a lot of broken builds today: $ curl -i -L https://github.com/kyleconroy/sqlc/releases/download/v1.17.0/sqlc_1.17.0_linux_amd64.tar.gz HTTP/2 302 server: GitHub.com date: Fri, 24 Mar 2023 20:51:56 GMT content-type: text/html; charset=utf-8 location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/193160679/09048595-c7f4-45b5-858a-7f55baa2fd7d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230324%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230324T205156Z&X-Amz-Expires=300&X-Amz-Signature=772d0aa8c5c19b0a5ef84d718d2faf0d81f24b224a4ef634d2410787e8f50bad&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=193160679&response-content-disposition=attachment%3B%20filename%3Dsqlc_1.17.0_linux_amd64.tar.gz&response-content-type=application%2Foctet-stream curl: (60) SSL certificate problem: certificate has expired More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. > What are the odds this happens the same day they rotate their SSH keys? Definitely a bad for them. When it rains, it pours.
  - nwatson 4y ago
    They laid off the wrong person.
    - AviationAtom 4y ago
      This is so comical because it's so relevant. Do you think people architect poorly designed systems more often than not as a means of job security or just a failure to put much forethought in whilst planning it?
      - tomrod 4y ago
        Failure of forethought with the wrong deadlines in place, soup to nuts.
      - 4y ago
ksml 4y ago
They're serving the wrong cert on pkg-containers.githubusercontent.com (it's for *.githubassets.com) and their support site also expired 3/21... https://support.github.com/ https://support.github.com/ What is going on over there?
gorjusborg 4y ago
And today of all days I have a moment to upgrade homebrew stuff.
carrina 4y ago
Not Before Fri, 18 Mar 2022 00:00:00 GMT Not After Tue, 21 Mar 2023 23:59:59 GMT 3-day certs.
- GOATS- 4y ago
  There's a whole year between those dates.
- warent 4y ago
  3 day plus 365 days ;) you're missing a full year
- AkshatJ27 4y ago
  the year is different
- vehementi 4y ago
  Short lived certs are a thing, also
tonto 4y ago
got a "RequestError: certificate has expired" doing a release just now...as usual, not a good idea to release on a friday
bvogelzang 4y ago
It looks as though it's back for me now. Status page is now showing the problem: https://www.githubstatus.com/ https://www.githubstatus.com/
- deathanatos 4y ago
  … well, the status page is back to green now, but AFAICT, the domains are still serving the expired cert. » TIMEZONE=UTC date; openssl s_client -connect support.github.com:443 2>&1 | grep 'cert.*has.*ex' Fri Mar 24 17:40:28 EDT 2023 verify error:num=10:certificate has expired Verify return code: 10 (certificate has expired) The previous incident seems pretty clearly to be this … so it seems like they think they fixed it…
Kelamir 4y ago
Previously I had the same issue, but it works for me now, as well as for a friend in another EU country.
deleted 4y ago
[deleted]
GOATS- 4y ago
It's back now!
ccheney 4y ago
EDIT: this specific issue is resolved Failing for us in GitHub Actions For SEO purposes: npm ERR! code ERR_TLS_CERT_ALTNAME_INVALID npm ERR! errno ERR_TLS_CERT_ALTNAME_INVALID npm ERR! request to https://pkg- npm.githubusercontent.com/npmregistryv2prod/blobs/\*\* failed, reason: Hostname/IP does not match certificate's altnames: Host: pkg-npm.githubusercontent.com. is not in the cert's altnames: DNS:\*.githubassets.com, DNS:githubassets.com
deleted 4y ago
[deleted]
apetresc 4y ago
Seems to be resolved now. My `brew update` works again.
lytedev 4y ago
Back up now, it looks like.
dz0ny 4y ago
Still some weird stuff around (* subject: CN=apistatus.chorus.co.nz). curl https://www.githubstatus.com/ -vvvv -I \* Trying 52.215.192.131:443... \* Connected to www.githubstatus.com (52.215.192.131) port 443 (#0) \* ALPN: offers h2 \* ALPN: offers http/1.1 ... \* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384 \* ALPN: server accepted h2 \* Server certificate: \* subject: CN=apistatus.chorus.co.nz \* start date: Mar 6 23:10:30 2023 GMT \* expire date: Jun 4 23:10:29 2023 GMT \* subjectAltName: host "www.githubstatus.com" matched cert's "www.githubstatus.com" \* issuer: C=US; O=Let's Encrypt; CN=R3 \* SSL certificate verify ok. \* Using HTTP2, server supports multiplexing
jjice 4y ago
Well I'm kind of just waiting on PRs for the rest of the day today and it's a Friday, so I'll consider this a modern equivalent of https://xkcd.com/303/ https://xkcd.com/303/
ollemasle 4y ago
Here is the report for this incident: https://www.githubstatus.com/incidents/x7njwb481j9b https://www.githubstatus.com/incidents/x7njwb481j9b
artyom 4y ago
ChatGPT, rotate my certs
- pmontra 4y ago
  It needs a plugin for that.
  - bink 4y ago
    Did we learn nothing from Son of Anton?
- slowmovintarget 4y ago
  Why? Don't most cloud providers have auto-renewing certs now?
jmspring 4y ago
Sounds like whoever is in charge of certificates at GH must have come over from MSFT. Afterall, I think Microsoft has had 2-3 certificate expiry issues in the last several years.
- jiggawatts 4y ago
  Azure had several global outages because of issues with certificates. One outage was caused by an incorrect date computation: the certificates last for one year, and this was computed with: "new DateTime(now.Year+1,now.Month,now.Day)". If you do that on Feb 29th of a leap year, it'll throw an exception because the next year doesn't have a Feb 29th! Oops. They "fixed" it and promptly had another related outage the very next day.
  - jmspring 4y ago
    That was one of them I recall.
alexanderscott 4y ago
didn’t they announce a bunch of layoffs recently?
gunshai 4y ago
For us dumb dumbs what does this mean?
mattbillenstein 4y ago
I built a free monitoring service some years ago if anyone doesn't want to be the victim of this... https://ismycertexpired.com/check?domain=objects.githubusercontent.com https://ismycertexpired.com/check?domain=objects.githubuserc...
radicalbyte 4y ago
I wonder if this has anything to do with the recent SNAFU from a Senior Security Engineer* there? https://twitter.com/viibeeng/status/1639374358287118336 https://twitter.com/viibeeng/status/1639374358287118336 (*yeah we can all make mistakes, but it's 2023, if you've not build controls into your workflows by now you don't deserve to be a Senior anything)

126 points by GOATS- 4y ago

GOATS- 4y ago

This also applies to their avatars subdomain, causing them not to load anymore.

sha-3 4y ago
Are you sure? avatars.githubusercontent.com works fine for me.
- guessmyname 4y ago
  > Are you sure? avatars.githubusercontent.com works fine for me. That’s because they already resolved the issue → https://www.githubstatus.com/incidents/x7njwb481j9b https://www.githubstatus.com/incidents/x7njwb481j9b This is what I saw an hour ago: $ echo | openssl s_client -connect avatars.githubusercontent.com:443 CONNECTED(00000005) depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA verify return:1 depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1 verify return:1 depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io >>> verify error:num=10:certificate has expired notAfter=Mar 21 23:59:59 2023 GMT verify return:1 depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io notAfter=Mar 21 23:59:59 2023 GMT verify return:1

dz0ny 4y ago

Also serving wrong certificates for a lot of content domains. https://news.ycombinator.com/item?id=35295191 https://news.ycombinator.com/item?id=35295191

koolba 4y ago

The cert for objects.githubusercontent.com has also expired: $ openssl s_client -connect objects.githubusercontent.com:443 CONNECTED(00000005) depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA verify return:1 depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1 verify return:1 depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io verify error:num=10:certificate has expired notAfter=Mar 21 23:59:59 2023 GMT verify return:1 depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io notAfter=Mar 21 23:59:59 2023 GMT verify return:1 --- Certificate chain 0 s:C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io i:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1 1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1 i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA What are the odds this happens the same day they rotate their SSH keys?

brandur 4y ago
It's also the domain used for releases and other artifacts (after a redirect from github.com). There's going to be a lot of broken builds today: $ curl -i -L https://github.com/kyleconroy/sqlc/releases/download/v1.17.0/sqlc_1.17.0_linux_amd64.tar.gz HTTP/2 302 server: GitHub.com date: Fri, 24 Mar 2023 20:51:56 GMT content-type: text/html; charset=utf-8 location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/193160679/09048595-c7f4-45b5-858a-7f55baa2fd7d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230324%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230324T205156Z&X-Amz-Expires=300&X-Amz-Signature=772d0aa8c5c19b0a5ef84d718d2faf0d81f24b224a4ef634d2410787e8f50bad&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=193160679&response-content-disposition=attachment%3B%20filename%3Dsqlc_1.17.0_linux_amd64.tar.gz&response-content-type=application%2Foctet-stream curl: (60) SSL certificate problem: certificate has expired More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. > What are the odds this happens the same day they rotate their SSH keys? Definitely a bad for them. When it rains, it pours.
- nwatson 4y ago
  They laid off the wrong person.
  - AviationAtom 4y ago
    This is so comical because it's so relevant. Do you think people architect poorly designed systems more often than not as a means of job security or just a failure to put much forethought in whilst planning it?
    - tomrod 4y ago
      Failure of forethought with the wrong deadlines in place, soup to nuts.
    - 4y ago

ksml 4y ago

They're serving the wrong cert on pkg-containers.githubusercontent.com (it's for *.githubassets.com) and their support site also expired 3/21... https://support.github.com/ https://support.github.com/ What is going on over there?

gorjusborg 4y ago

And today of all days I have a moment to upgrade homebrew stuff.

carrina 4y ago

Not Before Fri, 18 Mar 2022 00:00:00 GMT Not After Tue, 21 Mar 2023 23:59:59 GMT 3-day certs.

GOATS- 4y ago
There's a whole year between those dates.
warent 4y ago
3 day plus 365 days ;) you're missing a full year
AkshatJ27 4y ago
the year is different
vehementi 4y ago
Short lived certs are a thing, also

tonto 4y ago

got a "RequestError: certificate has expired" doing a release just now...as usual, not a good idea to release on a friday

bvogelzang 4y ago

It looks as though it's back for me now. Status page is now showing the problem: https://www.githubstatus.com/ https://www.githubstatus.com/

deathanatos 4y ago
… well, the status page is back to green now, but AFAICT, the domains are still serving the expired cert. » TIMEZONE=UTC date; openssl s_client -connect support.github.com:443 2>&1 | grep 'cert.*has.*ex' Fri Mar 24 17:40:28 EDT 2023 verify error:num=10:certificate has expired Verify return code: 10 (certificate has expired) The previous incident seems pretty clearly to be this … so it seems like they think they fixed it…

Kelamir 4y ago

Previously I had the same issue, but it works for me now, as well as for a friend in another EU country.

deleted 4y ago

[deleted]

GOATS- 4y ago

It's back now!

ccheney 4y ago

EDIT: this specific issue is resolved Failing for us in GitHub Actions For SEO purposes: npm ERR! code ERR_TLS_CERT_ALTNAME_INVALID npm ERR! errno ERR_TLS_CERT_ALTNAME_INVALID npm ERR! request to https://pkg- npm.githubusercontent.com/npmregistryv2prod/blobs/\*\* failed, reason: Hostname/IP does not match certificate's altnames: Host: pkg-npm.githubusercontent.com. is not in the cert's altnames: DNS:\*.githubassets.com, DNS:githubassets.com

deleted 4y ago

[deleted]

apetresc 4y ago

Seems to be resolved now. My `brew update` works again.

lytedev 4y ago

Back up now, it looks like.

dz0ny 4y ago

Still some weird stuff around (* subject: CN=apistatus.chorus.co.nz). curl https://www.githubstatus.com/ -vvvv -I \* Trying 52.215.192.131:443... \* Connected to www.githubstatus.com (52.215.192.131) port 443 (#0) \* ALPN: offers h2 \* ALPN: offers http/1.1 ... \* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384 \* ALPN: server accepted h2 \* Server certificate: \* subject: CN=apistatus.chorus.co.nz \* start date: Mar 6 23:10:30 2023 GMT \* expire date: Jun 4 23:10:29 2023 GMT \* subjectAltName: host "www.githubstatus.com" matched cert's "www.githubstatus.com" \* issuer: C=US; O=Let's Encrypt; CN=R3 \* SSL certificate verify ok. \* Using HTTP2, server supports multiplexing

jjice 4y ago

Well I'm kind of just waiting on PRs for the rest of the day today and it's a Friday, so I'll consider this a modern equivalent of https://xkcd.com/303/ https://xkcd.com/303/

ollemasle 4y ago

Here is the report for this incident: https://www.githubstatus.com/incidents/x7njwb481j9b https://www.githubstatus.com/incidents/x7njwb481j9b

artyom 4y ago

ChatGPT, rotate my certs

pmontra 4y ago
It needs a plugin for that.
- bink 4y ago
  Did we learn nothing from Son of Anton?
slowmovintarget 4y ago
Why? Don't most cloud providers have auto-renewing certs now?

jmspring 4y ago

Sounds like whoever is in charge of certificates at GH must have come over from MSFT. Afterall, I think Microsoft has had 2-3 certificate expiry issues in the last several years.

jiggawatts 4y ago
Azure had several global outages because of issues with certificates. One outage was caused by an incorrect date computation: the certificates last for one year, and this was computed with: "new DateTime(now.Year+1,now.Month,now.Day)". If you do that on Feb 29th of a leap year, it'll throw an exception because the next year doesn't have a Feb 29th! Oops. They "fixed" it and promptly had another related outage the very next day.
- jmspring 4y ago
  That was one of them I recall.

alexanderscott 4y ago

didn’t they announce a bunch of layoffs recently?

gunshai 4y ago

For us dumb dumbs what does this mean?

mattbillenstein 4y ago

I built a free monitoring service some years ago if anyone doesn't want to be the victim of this... https://ismycertexpired.com/check?domain=objects.githubusercontent.com https://ismycertexpired.com/check?domain=objects.githubuserc...

radicalbyte 4y ago

I wonder if this has anything to do with the recent SNAFU from a Senior Security Engineer* there? https://twitter.com/viibeeng/status/1639374358287118336 https://twitter.com/viibeeng/status/1639374358287118336 (*yeah we can all make mistakes, but it's 2023, if you've not build controls into your workflows by now you don't deserve to be a Senior anything)