8 ms·

GitHub has ruined my lunch or dinner time at least once a week or so for the last three months in some capacity. I am absolutely fucked off with it now. Centra

by uuyi 4y ago

GitHub has ruined my lunch or dinner time at least once a week or so for the last three months in some capacity. I am absolutely fucked off with it now. Centralising on this infra was a stupid technical decision.

chrisseaton 4y ago
If it goes down I would have thought you'd get a longer lunch or dinner break!
- uuyi 4y ago
  No I have to field a thousand questions about why nothing is working from just about every part of the organisation. Then spend several hours after it has recovered assigning blame back to the engineers when they do something stupid rather than blaming it on the GitHub outage.
  - mxuribe 4y ago
    It reminds me of a time many years ago - before SaaS days - that a co-worker was similarly disrupted due to unreliability of a vendor system...and his anger towards internal senior staff for (what many obviously) considered to be a poor decision to go with said vendor. So my buddy was tempted to automate email forwards to a partiocular senior leader...so they somewhat feel the pain of all the messages being sent on a partoicular topic...plus my buddy acvited his email's out of office message to both assure folks that he was aware of the issue and to contact the senior leader most responsible for the poor vendor decision...again, to share the paion. Ultimately he didn't do that direct of a blame for fear of getting fired...but he did implement the out of office in a general fashion at least to lessen/avoid him having to answer questions about the numerous outages, etc. He didn't name any names...but some savvy folks in the org who understood the management hierarchy eventually figured things out and knew to start pionging that senior leader...and my friend didn't get fired...So there could be some mildly automated ways to lessen disruptions to your life. ;-)
    - uuyi 4y ago
      Unfortunately the guy who decided this is gone. I should print and mail everything to him instead :)
Helitioo 4y ago
The little bit of downtime on GitHub is still much better for me to have to maintain my own gitlab. GitHub is still a no brainer for me.
- dahfizz 4y ago
  This is like the third downtime in a month. Hosting your own infra is easier than people think, but it is completely trivial if weekly downtime is the expectation.
  - Helitioo 4y ago
    I have been on both sides. GitHub downtime is something I can just tell managment and prepare things to put it out of high critical path. Like I would not pull images from GitHub on my k8s cluster. Gitlab works absolutely but you know I don't want to do gitlab updates, do backup, do regular backup/restore test, deleting backups etc. I prefer to create new things.
- dncornholio 4y ago
  Maintaining gitlab has been nothing more than sudo apt-get update && sudo apt-get install gitlab-ee. Their releases often have security vulnerabilities though. But other than that, even the community edition is packed with features, it's a no brainer for me ;)
  - Helitioo 4y ago
    The security issues in the last year were shocking to me. You also need to do backups, try restore, you might need to announce a maintenance window. And for what? Getting stress from mgmnt when it breaks? With a GitHub downtime that's just it. No one cares, everyone accepts it as given.
- mac-chaffee 4y ago
  Agreed. The people saying self-hosting is easy would do well to read this: https://utcc.utoronto.ca/~cks/space/blog/sysadmin/RunningServersNotTrivial https://utcc.utoronto.ca/~cks/space/blog/sysadmin/RunningSer...

by uuyi 4y ago

chrisseaton 4y ago

If it goes down I would have thought you'd get a longer lunch or dinner break!

uuyi 4y ago
No I have to field a thousand questions about why nothing is working from just about every part of the organisation. Then spend several hours after it has recovered assigning blame back to the engineers when they do something stupid rather than blaming it on the GitHub outage.
- mxuribe 4y ago
  It reminds me of a time many years ago - before SaaS days - that a co-worker was similarly disrupted due to unreliability of a vendor system...and his anger towards internal senior staff for (what many obviously) considered to be a poor decision to go with said vendor. So my buddy was tempted to automate email forwards to a partiocular senior leader...so they somewhat feel the pain of all the messages being sent on a partoicular topic...plus my buddy acvited his email's out of office message to both assure folks that he was aware of the issue and to contact the senior leader most responsible for the poor vendor decision...again, to share the paion. Ultimately he didn't do that direct of a blame for fear of getting fired...but he did implement the out of office in a general fashion at least to lessen/avoid him having to answer questions about the numerous outages, etc. He didn't name any names...but some savvy folks in the org who understood the management hierarchy eventually figured things out and knew to start pionging that senior leader...and my friend didn't get fired...So there could be some mildly automated ways to lessen disruptions to your life. ;-)
  - uuyi 4y ago
    Unfortunately the guy who decided this is gone. I should print and mail everything to him instead :)

Helitioo 4y ago

The little bit of downtime on GitHub is still much better for me to have to maintain my own gitlab. GitHub is still a no brainer for me.

dahfizz 4y ago
This is like the third downtime in a month. Hosting your own infra is easier than people think, but it is completely trivial if weekly downtime is the expectation.
- Helitioo 4y ago
  I have been on both sides. GitHub downtime is something I can just tell managment and prepare things to put it out of high critical path. Like I would not pull images from GitHub on my k8s cluster. Gitlab works absolutely but you know I don't want to do gitlab updates, do backup, do regular backup/restore test, deleting backups etc. I prefer to create new things.
dncornholio 4y ago
Maintaining gitlab has been nothing more than sudo apt-get update && sudo apt-get install gitlab-ee. Their releases often have security vulnerabilities though. But other than that, even the community edition is packed with features, it's a no brainer for me ;)
- Helitioo 4y ago
  The security issues in the last year were shocking to me. You also need to do backups, try restore, you might need to announce a maintenance window. And for what? Getting stress from mgmnt when it breaks? With a GitHub downtime that's just it. No one cares, everyone accepts it as given.
mac-chaffee 4y ago
Agreed. The people saying self-hosting is easy would do well to read this: https://utcc.utoronto.ca/~cks/space/blog/sysadmin/RunningServersNotTrivial https://utcc.utoronto.ca/~cks/space/blog/sysadmin/RunningSer...