3 ms·

Possibly related? https://github.blog/changelog/2023-11-03-multi-account-support-on-github-com/ https://github.blog/changelog/2023-11-03-multi-account-suppo...

by ezekg 3y ago

Possibly related? https://github.blog/changelog/2023-11-03-multi-account-support-on-github-com/ https://github.blog/changelog/2023-11-03-multi-account-suppo... Implementing that type of multi-account support likely had some big authz changes (I've done a couple of those projects before), and GitHub defaults to a 404 for authz errors instead of leaking information with a 403. Just a wild guess.

jrochkind1 3y ago
The fact that https://github.com/ https://github.com/ (which requires no specific authz check) seems to be up, and that some users report logging out and in again at least temporarily restores access, seems to support that hypothesis. Still seems odd that no CI/QA would have caught such a widespread bug.
- grepfru_it 3y ago
  Oh if you only knew the code base they are up against. Or in other words “what QA <laughing.emoji>”
netcraft 3y ago
spot on: > Update - We’re in the process of rolling back an authorization-related change that is causing 404s and other errors.
- ezekg 3y ago
  Like I said, not my first rodeo for implementing a multi-account system on top of a single-account system lol.
- hirsin 3y ago
  No, it was an authorization internals upgrade we were testing, not the account switcher.
d3mcfadden 3y ago
nailed it man
ta1243 3y ago
Nobody would deploy such a change on a Friday afternoon though
- xeonmc 3y ago
  #famouslastwords
- alfanick 3y ago
  It kinda makes sense though to do it on Friday afternoon, as on the weekend enterprises and most people just go skiing or do other things so less of customer base is affected in case of (unfortunate) outage.
  - ta1243 3y ago
    Even if that were a fact, then that's the one time github should not be updating. Their customers will be doing deployments on a Friday afternoon, thus require their repositories for their internal processes to work
- paulddraper 3y ago
  What day is the best day to have downtime?
  - ta1243 3y ago
    Depends on your business. For my workflow we are happiest with risk for weekdays 0930-1130 and 1400-1600, and Saturday from 0500-1100, but it varies depending on the affected systems and workflows. If I'm doing work on one systems used by Department A then a Saturday is great. If it's on Department B though Saturday is the worst possible time. Department C's systems are mainly used from 2000-0700. Department D's systems are used 24/7, which is why we have geographic resilience. But you don't get to choose this with XaaS. It goes down when it goes down, it comes up when it comes up. Our connectivity supplier sends us a notice that "circuit from X to Y will be going down from 0200-0500" and we have no choice but to accept it. Typically updates are best done on Tuesday morning given that most people are awake and working and thus if there are problems they can fix it, but you wouldn't go ahead while the resilience is down (say due to a fibre cut, or a power outage)
deleted 3y ago
[deleted]

by ezekg 3y ago

jrochkind1 3y ago

The fact that https://github.com/ https://github.com/ (which requires no specific authz check) seems to be up, and that some users report logging out and in again at least temporarily restores access, seems to support that hypothesis. Still seems odd that no CI/QA would have caught such a widespread bug.

grepfru_it 3y ago
Oh if you only knew the code base they are up against. Or in other words “what QA <laughing.emoji>”

netcraft 3y ago

spot on: > Update - We’re in the process of rolling back an authorization-related change that is causing 404s and other errors.

ezekg 3y ago
Like I said, not my first rodeo for implementing a multi-account system on top of a single-account system lol.
hirsin 3y ago
No, it was an authorization internals upgrade we were testing, not the account switcher.

d3mcfadden 3y ago

nailed it man

ta1243 3y ago

Nobody would deploy such a change on a Friday afternoon though

xeonmc 3y ago
#famouslastwords
alfanick 3y ago
It kinda makes sense though to do it on Friday afternoon, as on the weekend enterprises and most people just go skiing or do other things so less of customer base is affected in case of (unfortunate) outage.
- ta1243 3y ago
  Even if that were a fact, then that's the one time github should not be updating. Their customers will be doing deployments on a Friday afternoon, thus require their repositories for their internal processes to work
paulddraper 3y ago
What day is the best day to have downtime?
- ta1243 3y ago
  Depends on your business. For my workflow we are happiest with risk for weekdays 0930-1130 and 1400-1600, and Saturday from 0500-1100, but it varies depending on the affected systems and workflows. If I'm doing work on one systems used by Department A then a Saturday is great. If it's on Department B though Saturday is the worst possible time. Department C's systems are mainly used from 2000-0700. Department D's systems are used 24/7, which is why we have geographic resilience. But you don't get to choose this with XaaS. It goes down when it goes down, it comes up when it comes up. Our connectivity supplier sends us a notice that "circuit from X to Y will be going down from 0200-0500" and we have no choice but to accept it. Typically updates are best done on Tuesday morning given that most people are awake and working and thus if there are problems they can fix it, but you wouldn't go ahead while the resilience is down (say due to a fibre cut, or a power outage)

deleted 3y ago

[deleted]