3 ms·

Is there any reason why Github needs 99.99% uptime? You can continue working with your local repo.

by sgt 9mo ago

nullstyle 9mo ago
The money i pay them is the reason
badgersnake 9mo ago
Lots of teams embraced actions to run their CI/CD, and GitHub reviews as part of their merge process. And copilot. Basically their SOC2 (or whatever) says they have to use GitHub. I’m guessing they’re regretting it.
- swiftcoder 9mo ago
  > Basically their SOC2 (or whatever) says they have to use GitHub Our SOC2 doesn't specify GitHub by name, but it does require we maintain a record of each PR having been reviewed. I guess in extremis we could email each other patch diffs, and CC the guy responsible for the audit process with the approval...
  - onraglanroad 9mo ago
    The Linux kernel uses an email based workflow. You can digitally sign email and add it to an immutable store that can be reviewed.
  - sgt 9mo ago
    Does SOC2 itself require that or just yours? I'm not too familiar with SOC2 but I know ISO 27001 quite well, and there's no PR specific "requirements" to speak of. But it is something that could be included in your secure development policy.
    - badgersnake 9mo ago
      Yeah, it’s what you write in the policy.
      - swiftcoder 9mo ago
        And it's pretty common to write in the policy, because its pretty much a gimme, and lets you avoid writing a whole bunch of other equivalent quality measures in the policy.
  - bostik 9mo ago
    Every product vendor, especially those that are even within a shouting distance from security, has a wet dream: to have their product explicitly named in corporate policies. I have cleaned up more than enough of them.
esafak 9mo ago
Are you kidding? I need my code to pass CI, and get reviewed, so I can move on, otherwise the PRs just keep piling. You might as well say the lights could go out, you can do paperwork.
- koreth1 9mo ago
  > otherwise the PRs just keep piling Good news! You can't create new PRs right now anyway, so they won't pile.
  - deleted 9mo ago
    [deleted]
- munk-a 9mo ago
  When in doubt - schedule a meeting about how you're unable to do work to keep doing work!
- 1718627440 9mo ago
  Are you technically required to do it that way, or is it just your company that want things to be done that way?
degenerate 9mo ago
Many teams work exclusively in GitHub (ticketing, boards, workflows, dev builds). People also have entire production build systems on GitHub. There's a lot more than git repo hosting.
- munk-a 9mo ago
  It's especially painful for anyone who uses Github actions for CI/CD - maybe the release you just cut never actually got deployed to prod because their internal trigger didn't fire... you need to watch it like a hawk.
  - jamiecurle 9mo ago
    I waited 2.5 hours for a webhook from the registry_packages endpoint today. I'm grateful it arrived, but two and half hours feels less than ideal.
theappsecguy 9mo ago
What if you need to deploy to production urgently...
- 1718627440 9mo ago
  SSH, rsync or FTP.
babo 9mo ago
As an example, Go build could fail anywhere if a dependency module from Github is not available.
- sethops1 9mo ago
  Any module that is properly tagged and contains an OSS license gets stored in Google's module cache indefinitely. As long as it was go-get-ed once before, you can pull it again without going to GitHub (or any other VCS host).
- toastal 9mo ago
  Does go build not support mirrors so you can define a fallback repository? If not, why?
ajross 9mo ago
I think this is being downvoted unfairly. I mean, sure, as a company accepting payment for services, being down for a few hours every few months is notably bad by modern standards. But the inward-looking point is correct: git itself is a distributed technology, and development using it is distributed and almost always latency-tolerant. To the extent that github's customers have processes that are dependent on services like bug tracking and reporting and CI to keep their teams productive, that's a bug with the customer's processes. It doesn't have to be that way and we as a community can recognize that even if the service provider kinda sucks.
- progmetaldev 9mo ago
  There are still some processes that require a waterfall method for development, though. One example would be if you have a designer, and also have a front-end developer that is waiting for a feature to be complete to come in and start their development. I know on HN it's common for people to be full-stack developers, or for front-end developers to be able to work with a mockup and write the code before a designer gets involved, but there are plenty of companies that don't work that way. Even if a company is working in an agile manner, there still may come a time where work stalls until some part of a system is finished by another team/team-member, especially in a monorepo. Of course they could change the organization of their project, but the time suck of doing that (like going with microservices) is probably going to waste quite a bit more time than how often GitHub is down.
  - ajross 9mo ago
    > There are still some processes that require a waterfall method for development, though Not on the 2-4 hour latency scale of a GitHub outage though. I mean, sure, if you have a process that requires the engineering talent to work completely independently on day-plus timescales and/or do all their coordination offline, then you're going to have a ton of trouble staffing[1] that team. But if your folks can't handle talking with the designers over chat or whatnot to backfill the loss of the issue tracker for an afternoon, then that's on you. [1] It can obviously be done! But it's isomorphic to "put together a Linux-style development culture", very non-trivial.
amonith 9mo ago
I'm a firm believer that almost nothing except public services needs that kind of uptime... We've introduced ridiculous amounts of complexity to our infra to achieve this and we've contributed to the increasing costs of both services and development itself (the barrier of entry for current juniors is insane compared to what I've had to deal with in my early 20s).
- nosman 9mo ago
  What do you mean by public services? All kinds of companies lose millions of dollars of revenue per day if not hour if their sites are not stable.... apple, amazon, google, Shopify, uber, etc etc. Those companies have decided the extra complexity is worth the reliability. Even if you're operating a tech company that doesn't need to have that kind of uptime, your developers probably need those services to be productive, and you don't want them just sitting there either.
  - amonith 9mo ago
    By public services I mean only important things like healthcare, law enforcement, fire department. Definitely not stores and food delivery. You can wait an hour or even a couple of hours for that. > Those companies have decided the extra complexity is worth the reliability. Companies always want more money and yes it makes sense economically. I'm not disagreeing with that. I'm just saying that nobody needs this. I grew up in a world where this wasn't a thing and no, life wasn't worse at all.
    - zinodaur 9mo ago
      Eh, if I'm paying someone to host my git webui, and they are as shitty about it as github has been recently, I'd rather pay someone else to host it or go back to hosting it myself. It is not absolutely required, but it's a differentiating feature I'm happy to pay for

Is there any reason why Github needs 99.99% uptime? You can continue working with your local repo.

by sgt 9mo ago

Is there any reason why Github needs 99.99% uptime? You can continue working with your local repo.

nullstyle 9mo ago

The money i pay them is the reason

badgersnake 9mo ago

Lots of teams embraced actions to run their CI/CD, and GitHub reviews as part of their merge process. And copilot. Basically their SOC2 (or whatever) says they have to use GitHub. I’m guessing they’re regretting it.

swiftcoder 9mo ago
> Basically their SOC2 (or whatever) says they have to use GitHub Our SOC2 doesn't specify GitHub by name, but it does require we maintain a record of each PR having been reviewed. I guess in extremis we could email each other patch diffs, and CC the guy responsible for the audit process with the approval...
- onraglanroad 9mo ago
  The Linux kernel uses an email based workflow. You can digitally sign email and add it to an immutable store that can be reviewed.
- sgt 9mo ago
  Does SOC2 itself require that or just yours? I'm not too familiar with SOC2 but I know ISO 27001 quite well, and there's no PR specific "requirements" to speak of. But it is something that could be included in your secure development policy.
  - badgersnake 9mo ago
    Yeah, it’s what you write in the policy.
    - swiftcoder 9mo ago
      And it's pretty common to write in the policy, because its pretty much a gimme, and lets you avoid writing a whole bunch of other equivalent quality measures in the policy.
- bostik 9mo ago
  Every product vendor, especially those that are even within a shouting distance from security, has a wet dream: to have their product explicitly named in corporate policies. I have cleaned up more than enough of them.

esafak 9mo ago

Are you kidding? I need my code to pass CI, and get reviewed, so I can move on, otherwise the PRs just keep piling. You might as well say the lights could go out, you can do paperwork.

koreth1 9mo ago
> otherwise the PRs just keep piling Good news! You can't create new PRs right now anyway, so they won't pile.
- deleted 9mo ago
  [deleted]
munk-a 9mo ago
When in doubt - schedule a meeting about how you're unable to do work to keep doing work!
1718627440 9mo ago
Are you technically required to do it that way, or is it just your company that want things to be done that way?

degenerate 9mo ago

Many teams work exclusively in GitHub (ticketing, boards, workflows, dev builds). People also have entire production build systems on GitHub. There's a lot more than git repo hosting.

munk-a 9mo ago
It's especially painful for anyone who uses Github actions for CI/CD - maybe the release you just cut never actually got deployed to prod because their internal trigger didn't fire... you need to watch it like a hawk.
- jamiecurle 9mo ago
  I waited 2.5 hours for a webhook from the registry_packages endpoint today. I'm grateful it arrived, but two and half hours feels less than ideal.

theappsecguy 9mo ago

What if you need to deploy to production urgently...

1718627440 9mo ago
SSH, rsync or FTP.

babo 9mo ago

As an example, Go build could fail anywhere if a dependency module from Github is not available.

sethops1 9mo ago
Any module that is properly tagged and contains an OSS license gets stored in Google's module cache indefinitely. As long as it was go-get-ed once before, you can pull it again without going to GitHub (or any other VCS host).
toastal 9mo ago
Does go build not support mirrors so you can define a fallback repository? If not, why?

ajross 9mo ago

I think this is being downvoted unfairly. I mean, sure, as a company accepting payment for services, being down for a few hours every few months is notably bad by modern standards. But the inward-looking point is correct: git itself is a distributed technology, and development using it is distributed and almost always latency-tolerant. To the extent that github's customers have processes that are dependent on services like bug tracking and reporting and CI to keep their teams productive, that's a bug with the customer's processes. It doesn't have to be that way and we as a community can recognize that even if the service provider kinda sucks.

progmetaldev 9mo ago
There are still some processes that require a waterfall method for development, though. One example would be if you have a designer, and also have a front-end developer that is waiting for a feature to be complete to come in and start their development. I know on HN it's common for people to be full-stack developers, or for front-end developers to be able to work with a mockup and write the code before a designer gets involved, but there are plenty of companies that don't work that way. Even if a company is working in an agile manner, there still may come a time where work stalls until some part of a system is finished by another team/team-member, especially in a monorepo. Of course they could change the organization of their project, but the time suck of doing that (like going with microservices) is probably going to waste quite a bit more time than how often GitHub is down.
- ajross 9mo ago
  > There are still some processes that require a waterfall method for development, though Not on the 2-4 hour latency scale of a GitHub outage though. I mean, sure, if you have a process that requires the engineering talent to work completely independently on day-plus timescales and/or do all their coordination offline, then you're going to have a ton of trouble staffing[1] that team. But if your folks can't handle talking with the designers over chat or whatnot to backfill the loss of the issue tracker for an afternoon, then that's on you. [1] It can obviously be done! But it's isomorphic to "put together a Linux-style development culture", very non-trivial.

amonith 9mo ago

I'm a firm believer that almost nothing except public services needs that kind of uptime... We've introduced ridiculous amounts of complexity to our infra to achieve this and we've contributed to the increasing costs of both services and development itself (the barrier of entry for current juniors is insane compared to what I've had to deal with in my early 20s).

nosman 9mo ago
What do you mean by public services? All kinds of companies lose millions of dollars of revenue per day if not hour if their sites are not stable.... apple, amazon, google, Shopify, uber, etc etc. Those companies have decided the extra complexity is worth the reliability. Even if you're operating a tech company that doesn't need to have that kind of uptime, your developers probably need those services to be productive, and you don't want them just sitting there either.
- amonith 9mo ago
  By public services I mean only important things like healthcare, law enforcement, fire department. Definitely not stores and food delivery. You can wait an hour or even a couple of hours for that. > Those companies have decided the extra complexity is worth the reliability. Companies always want more money and yes it makes sense economically. I'm not disagreeing with that. I'm just saying that nobody needs this. I grew up in a world where this wasn't a thing and no, life wasn't worse at all.
  - zinodaur 9mo ago
    Eh, if I'm paying someone to host my git webui, and they are as shitty about it as github has been recently, I'd rather pay someone else to host it or go back to hosting it myself. It is not absolutely required, but it's a differentiating feature I'm happy to pay for