5 ms·

Hearts out to all the GitHub SREs who get to fix this, and all the SREs at GitHub’s customers who get to spend the next week convincing management and their pee

Hearts out to all the GitHub SREs who get to fix this, and all the SREs at GitHub’s customers who get to spend the next week convincing management and their peers that moving git in house won’t result in fewer outages.

andrewfromx 3y ago
so true. At this moment everyone's like, well! This is UNACCEPTABLE. We'll just, we'll just... what, what are you going to do better than github?
- no_wizard 3y ago
  I always wondered if some % of GitHub orgs move to Gitlab when this happens. I'd be interested to see if there is any correlation there.
  - slg 3y ago
    Yeah, because Gitlab would never go down. https://hn.algolia.com/?q=gitlab+down https://hn.algolia.com/?q=gitlab+down
    - dmonitor 3y ago
      Gitlab can be self hosted
      - dmattia 3y ago
        as can Github: https://docs.github.com/en/enterprise-server@3.5/admin/overview/about-github-enterprise-server https://docs.github.com/en/enterprise-server@3.5/admin/overv...
      - slg 3y ago
        And we all know self-hosted software never goes down. Although I guess the benefit is that the CTO gets a psychological lift of looking over Jeremy from IT's shoulder when he tries to fix the issue rather than being at the mercy of some engineers which the company can't fire.
        Alupis 3y ago
        People like to make it sound like running your own server/service is impossible - completely forgetting that's how everything was done not that long ago. Some wiz-bang new graduates calling themselves Engineers probably haven't a clue - sure, but for everyone else, you probably still have folks employed that did just that on a professional level. It's not hard - and often it's less expensive. Particularly for a turn-key paid product like Github Enterprise or GitLab.
        apple4ever 3y ago
        > And we all know self-hosted software never goes down. I've been running self-hosted software, and GitLab in particularly, for over a decade at various jobs. It almost never goes down. It fact, its up way way way more often that GitHub!
- gnarbarian 3y ago
  there's always gitlab
- robopsychology 3y ago
  We self-host Gitlab and haven't had an unplanned outage in a year since I joined.
- choeger 3y ago
  >what are you going to do better than github? Not add new features every week? Seriously, you cannot tell me that you cannot run a minimal version of a git repo service much more stable than that. I blame the need for new shiny stuff.
  - joshmanders 3y ago
    > I blame the need for new shiny stuff. Ya'll make me laugh. Don't add any features and "This thing is dead, nothing new in years, find an alternative" But a company actively adds new features their customers are asking for and you're upset that they're adding new features. Pick a lane.
    - doodlesdev 3y ago
      Curiosity: Not everyone in Hacker News has the same opinions.
      - joshmanders 3y ago
        Yet in every thread on HN when a service has an outage, the same opinion is spewed. I know the comments of these posts before I even open them. HN is an echo chamber of predictability.
        capableweb 3y ago
        That you keep opening the discussion page and even participate might say more about you than HN :)
    - lowercased 3y ago
      There's a middle ground between "nothing new in years" and "new feature or changes every week".
vb-8448 3y ago
It depends on your needs and team size. For few tens of dev that need a versioning system with some sugar, a self hosted gitlab can acieve better uptime.
- paulddraper 3y ago
  > a self hosted gitlab can acieve better uptime. Gonna call BS on that. It can achieve better uptime, if you ignore the downtime for upgrades, the downtime for configuration errors, the downtime when the disk fills up...
  - vb-8448 3y ago
    As I said, it depends on your needs. I don't know if you have ever managed a small gitlab self-hosted installation, but it's really trivial, you set up it the first time, and you forget about it. Also, the upgrade process takes few minutes to complete. Btw, you cannot compare a scheduled maintenance window with an unplanned downtime ... maybe in the middle of an important deployment.
    - paulddraper 3y ago
      > I don't know if you have ever managed a small gitlab self-hosted installation Bitbucket Server
  - capableweb 3y ago
    > It can achieve better uptime, if you ignore the downtime for upgrades, the downtime for configuration errors, the downtime when the disk fills up... Besides the last one, the others you can schedule when developers are not actually working on something, or give a headsup so developers can be prepared in case of errors. In the case of GitHub, Microsoft deploys changes whenever they want, whenever you want it or not.
    - joshmanders 3y ago
      > In the case of GitHub, Microsoft deploys changes whenever they want, whenever you want it or not. That's because GitHub is used by millions of people every day all over the world. Business hours for them are around the clock. No matter when they update, if it breaks something, many people will hear about it regardless of when the break was deployed.
bogantech 3y ago
Our on-prem GH enterprise hasn't gone down in the last year. Additionally any potentially disruptive work like upgrades etc can be scheduled outside of business hours so that they won't impact users
- DougN7 3y ago
  I self-host Subversion and haven’t had an unplanned outage in more than a decade :) (almost two decades?)
  - COMMENT___ 3y ago
    How’s your experience with Subversion? What kind of content do store in SVN, BTW?
    - DougN7 3y ago
      I really like it. I keep source code, 3rd party libraries, images, binary build output and installers. I can easily go back and debug and crash a customer ever sees because I have all the binaries and debugging files.
- BossingAround 3y ago
  In general, paying GH tends to be much cheaper than employing a team of SREs that'll keep your on-prem Git instance up and running 24/7 (esp. including things like backups, disaster recovery, and similar good practices necessary for any enterprise-ready SaaS). Something tells me that if you can schedule upgrades "outside of business hours", your scale is small (because in a global company, it's always "business hours" somewhere), which makes the cost argument even more prominent.
  - bogantech 3y ago
    You don't employ a team of SRE's with the sole day-job of git maintenance though, your existing SRE/sysadmin team manages it. And any large business will already be doing Backups, DR etc for their existing systems.

andrewfromx 3y ago

so true. At this moment everyone's like, well! This is UNACCEPTABLE. We'll just, we'll just... what, what are you going to do better than github?

no_wizard 3y ago
I always wondered if some % of GitHub orgs move to Gitlab when this happens. I'd be interested to see if there is any correlation there.
- slg 3y ago
  Yeah, because Gitlab would never go down. https://hn.algolia.com/?q=gitlab+down https://hn.algolia.com/?q=gitlab+down
  - dmonitor 3y ago
    Gitlab can be self hosted
    - dmattia 3y ago
      as can Github: https://docs.github.com/en/enterprise-server@3.5/admin/overview/about-github-enterprise-server https://docs.github.com/en/enterprise-server@3.5/admin/overv...
    - slg 3y ago
      And we all know self-hosted software never goes down. Although I guess the benefit is that the CTO gets a psychological lift of looking over Jeremy from IT's shoulder when he tries to fix the issue rather than being at the mercy of some engineers which the company can't fire.
      - Alupis 3y ago
        People like to make it sound like running your own server/service is impossible - completely forgetting that's how everything was done not that long ago. Some wiz-bang new graduates calling themselves Engineers probably haven't a clue - sure, but for everyone else, you probably still have folks employed that did just that on a professional level. It's not hard - and often it's less expensive. Particularly for a turn-key paid product like Github Enterprise or GitLab.
      - apple4ever 3y ago
        > And we all know self-hosted software never goes down. I've been running self-hosted software, and GitLab in particularly, for over a decade at various jobs. It almost never goes down. It fact, its up way way way more often that GitHub!
gnarbarian 3y ago
there's always gitlab
robopsychology 3y ago
We self-host Gitlab and haven't had an unplanned outage in a year since I joined.
choeger 3y ago
>what are you going to do better than github? Not add new features every week? Seriously, you cannot tell me that you cannot run a minimal version of a git repo service much more stable than that. I blame the need for new shiny stuff.
- joshmanders 3y ago
  > I blame the need for new shiny stuff. Ya'll make me laugh. Don't add any features and "This thing is dead, nothing new in years, find an alternative" But a company actively adds new features their customers are asking for and you're upset that they're adding new features. Pick a lane.
  - doodlesdev 3y ago
    Curiosity: Not everyone in Hacker News has the same opinions.
    - joshmanders 3y ago
      Yet in every thread on HN when a service has an outage, the same opinion is spewed. I know the comments of these posts before I even open them. HN is an echo chamber of predictability.
      - capableweb 3y ago
        That you keep opening the discussion page and even participate might say more about you than HN :)
  - lowercased 3y ago
    There's a middle ground between "nothing new in years" and "new feature or changes every week".

vb-8448 3y ago

It depends on your needs and team size. For few tens of dev that need a versioning system with some sugar, a self hosted gitlab can acieve better uptime.

paulddraper 3y ago
> a self hosted gitlab can acieve better uptime. Gonna call BS on that. It can achieve better uptime, if you ignore the downtime for upgrades, the downtime for configuration errors, the downtime when the disk fills up...
- vb-8448 3y ago
  As I said, it depends on your needs. I don't know if you have ever managed a small gitlab self-hosted installation, but it's really trivial, you set up it the first time, and you forget about it. Also, the upgrade process takes few minutes to complete. Btw, you cannot compare a scheduled maintenance window with an unplanned downtime ... maybe in the middle of an important deployment.
  - paulddraper 3y ago
    > I don't know if you have ever managed a small gitlab self-hosted installation Bitbucket Server
- capableweb 3y ago
  > It can achieve better uptime, if you ignore the downtime for upgrades, the downtime for configuration errors, the downtime when the disk fills up... Besides the last one, the others you can schedule when developers are not actually working on something, or give a headsup so developers can be prepared in case of errors. In the case of GitHub, Microsoft deploys changes whenever they want, whenever you want it or not.
  - joshmanders 3y ago
    > In the case of GitHub, Microsoft deploys changes whenever they want, whenever you want it or not. That's because GitHub is used by millions of people every day all over the world. Business hours for them are around the clock. No matter when they update, if it breaks something, many people will hear about it regardless of when the break was deployed.

bogantech 3y ago

Our on-prem GH enterprise hasn't gone down in the last year. Additionally any potentially disruptive work like upgrades etc can be scheduled outside of business hours so that they won't impact users

DougN7 3y ago
I self-host Subversion and haven’t had an unplanned outage in more than a decade :) (almost two decades?)
- COMMENT___ 3y ago
  How’s your experience with Subversion? What kind of content do store in SVN, BTW?
  - DougN7 3y ago
    I really like it. I keep source code, 3rd party libraries, images, binary build output and installers. I can easily go back and debug and crash a customer ever sees because I have all the binaries and debugging files.
BossingAround 3y ago
In general, paying GH tends to be much cheaper than employing a team of SREs that'll keep your on-prem Git instance up and running 24/7 (esp. including things like backups, disaster recovery, and similar good practices necessary for any enterprise-ready SaaS). Something tells me that if you can schedule upgrades "outside of business hours", your scale is small (because in a global company, it's always "business hours" somewhere), which makes the cost argument even more prominent.
- bogantech 3y ago
  You don't employ a team of SRE's with the sole day-job of git maintenance though, your existing SRE/sysadmin team manages it. And any large business will already be doing Backups, DR etc for their existing systems.