11 ms·

GitHub Broken Download URLs

206 points by wyoh 5y ago

sharmin123 5y ago
Learn Ethical Hacking And Save The World: Hacking Benefits: https://www.hackerslist.co/learn-ethical-hacking-and-save-the-world-hacking-benefits/ https://www.hackerslist.co/learn-ethical-hacking-and-save-th...
gudmundur 5y ago
Hi everyone. I'm an engineer at GitHub and I just posted a response to this issue here: https://github.com/github/feedback/discussions/8149#discussioncomment-1712006 https://github.com/github/feedback/discussions/8149#discussi....
- Rabidgremlin 5y ago
  Is this issue also related? https://github.com/explosion/spaCy/issues/9606 https://github.com/explosion/spaCy/issues/9606
- kzrdude 5y ago
  Thanks for fixing! Sorry for being curious.. does Github have an office in Copenhagen? That would be cool.
  - gudmundur 5y ago
    Yes, we do. I’ve been at GitHub longer than we’ve had that office, so I tend to work from home anyways.
    - Liquidor 5y ago
      That's amazing. Had no idea we had a GitHub office here, let alone in the country! If you're working remotely, are you employed by Californian standards/regulations/benefits or by local ones here in Denmark? If you don't mind me asking.
      - gudmundur 5y ago
        I am employed here in Denmark, wouldn’t have it any other way.
- dehrmann 5y ago
  > A change in the handling of URL schemes was deployed a couple of days ago This was deployed Thanksgiving week? I realize Github isn't a consumer company, so it doesn't face the same pressures as Amazon, but I'm surprised there wasn't a code freeze so people can have a quiet holiday weekend.
  - bengale 5y ago
    Is everyone at GitHub based in America?
    - mdoms 5y ago
      No, they are not. I know at least one Kiwi who worked remotely for Github from New Zealand.
    - jsnell 5y ago
      No. But this appears to now be the most important online shopping week of the year in all of Europe as well. Shops are abnormally sensitive to outages. You'd expect all kinds of service providers to be extremely conservative with code and config pushes this week due to that. Maybe GitHub is far enough removed from the actual consumers that they don't feel the pressure, but it does seem surprising.
    - novok 5y ago
      If a good chunk of them are, it's still good to do a code freeze. If %30-90 of your company is on holiday, not a good time to do major things.
intunderflow 5y ago
Combined with the outage yesterday this hasn't been a good weekend for GitHub SRE's I'm surprised something like this happened on a weekend though since I wouldn't expect anyone to be changing anything in the codebase (then again it could be some infra has just ran out of storage or etc)
- yardstick 5y ago
  Perhaps there’s a lack of attention due to the holiday weekend in the US? Seems like major incidents in infrastructure and services tend to happen over holidays (general observation, not GitHub specific).
  - deleted 5y ago
    [deleted]
junon 5y ago
Another gripe; unrelated, but since we're piling on... My username ends in a hyphen. Apparently, that's no longer allowed, though my username appears to be grandfathered in. Trying to give feedback about new experimental features lands me on the GitHub communities site, which is treated as a standalone app and thus requires you to log in via GitHub (it doesn't re-use the existing session token). However, Communities won't let me sign up with my username since it has a hanging hyphen, and I can't change the username in the form. So I effectively can't sign up. Support has not responded for over a month. Feels like things are inching toward getting worse with GitHub.
- favadi 5y ago
  I still remember the night when logrus's author decided to rename his Github account and broke our production build (https://github.com/sirupsen/logrus/pull/384 https://github.com/sirupsen/logrus/pull/384). Since then, I always vendor third party dependencies.
  - junon 5y ago
    To be fair, this is a side effect of poor module system design in Go. Not really github's fault here.
- Eikon 5y ago
  Sounds like a very important and critical issue that should be prioritised.
- belter 5y ago
  I am surprised a Microsoft MVP Certified Professional expert did not pop up yet in the newsgroup, asking you to reboot your machine and review the steps in a certain technote ;-)
  - Causality1 5y ago
    Hey now, that's libel. There are problems that can be solved by rebooting, and a Microsoft MVP would never post a genuine potential solution. I'm pretty sure you meant to say "an MVP popped up in the newsgroup to copy/paste a paragraph of random intro text and then ask if he's solved your problem".
the_duke 5y ago
At least now I know why my nix builds are failing... Odd for something like this to slip through and not be rolled back immediately. Unless it was intentional, in which case it would be even more odd to not communicate this widely beforehand.
- KennyBlanken 5y ago
  If Github doesn't at least monitor their 404 error rate for large-scale spikes, whoever is in charge of SRE should be fired. With no announcements and no response to a now two day old bug report, I see two possibilities: 1)Their monitoring of their infrastructure and monitoring of issues is shockingly incompetent for a company of their size and importance (the fact that it is a US holiday is irrelevant.) 2)This was 100% intentional and they're purposefully looking "incompetent" to get people to shift to using other services for downloads. My money is on the latter, given others in this discussion are reporting random download link failures starting a month or two ago. A huge number of projects seem to use GitHub as a sort of free file hosting service. I imagine the opex for both storage and bandwidth is a not insignificant amount of money and someone has been told to shoo the freeloaders off the grass. Announcing they're ending free file hosting for unpaid projects would generate a lot of noise and PR. Instead they just make it unreliable, and people go elsewhere. Multiple people in this discussion have described moving downloads of Github in response, which is exactly what Github likely wants.
  - tata71 5y ago
    > If Github doesn't at least monitor their 404 error rate for large-scale spikes "Is that a service we can charge for?!"
  - skyeto 5y ago
    From the discussion thread on GitHub[0]: A change in the handling of URL schemes was deployed a couple of days ago that caused the regression being discussed here. Due to the amount of traffic that the archive endpoints see, and the high baseline of 404s on them, this regression did not cause an unusual increase of errors that would've caused our alerting to kick in. The change has just been rolled back, so the issue is fixed. We will investigate this issue further after the weekend and take the appropriate steps to make sure similar regressions don't happen in the future. [0] https://github.com/github/feedback/discussions/8149#discussioncomment-1712006 https://github.com/github/feedback/discussions/8149#discussi...
AshamedCaptain 5y ago
Say what you want about "cloud" reliability, but my little home server in a residential ISP has been up for more time than Github.com in (at least!) the past year.
- seoaeu 5y ago
  And if github.com tried to host their website entirely off your little home server, they’d surely have 24x7 outages from being bombarded by too much load. All your anecdote proves is that it is easier to keep a single server online than operate a big distributed system, which has been obvious for quite a while
  - dvdkon 5y ago
    Obvious, maybe, but there are many who will criticise anyone self-hosting, saying that a cloud solution will inherently be more stable, since it has a bigger ops team.
    - donny2018 5y ago
      Well, GitHub came back online without me doing anything. GitHub’s life depends on providing a good service to customers. I think occasional downtime is a good tradeoff for what I am getting, as opposed to having to manage my own server.
      - mistrial9 5y ago
        > GitHub came back online without me doing anything the Executive Privilege for the common committer ! just keep your keys "valid" and don't talk to anyone we ban up next -- Github social scores
  - AshamedCaptain 5y ago
    No, it is not "obvious" to everyone at all. You can still see people here claiming that one should move to a centralized provider since they can guarantee nine nines of whatever, and that self-hosting is way too hard to make reliable. (Which is double irony when the centralized provider goes down and then the excuse is "well, that's because they're big!". If only...). In any case, the point was that Github.com just sucks, rather than everything cloud sucks. For the past year, they have been down a couple of magnitudes more time than I have spent managing my server.
mfashby 5y ago
This has also broken a bunch of packages in the arch user repository, for example https://aur.archlinux.org/packages/dendrite/ https://aur.archlinux.org/packages/dendrite/ :(
- Semaphor 5y ago
  Yeah, I thought the package I tried to install had an issue, but didn’t have the time to investigate. This explains it.
- viccuad 5y ago
  This sounds for the better. Not having code mirrors (as other distribution channels) sounds not just insecure, but borderly malicious.
  - themusicgod1 5y ago
    Using github, period, is not just insecure, but actively malicious. Stop using NSA/Microsoft.
  - turminal 5y ago
    I suspect you have no idea what AUR is and how it works and furthermore, you have no experience with software packaging. If a project is using github to publish releases, where else are consumers of that software going to get them from? Having all sources of everything that is packaged backed up is a must for the official repository of a competent distro, but even in that case there is no reason not to use github in normal operation.
  - mfashby 5y ago
    Depends what you mean by better. It's kind of annoying for me when trying to install some software I want to actually use, and I just can't.
_zywo 5y ago
Whether this specific problem is intentional or not, these kinds of problems show the issue with using a single centralized service for distribution of third-party dependencies. But it's just so much more darned convenient than hosting your own Git server! It would be super cool if there was a decentralized alternative to GitHub, that used Git under the hood. Perhaps one would upload their repositories to a node, which would then be synchronized with all other nodes, and all you would need to do to use it is to specify any_node.com/author/project. This would keep GitHub's discoverability, while allowing all the benefits of decentralization.
- adeelk93 5y ago
  Breaking changes aren’t only an issue with centralization. A breaking change with git itself would mess up your decentralized scenario as well.
  - TobTobXX 5y ago
    Maybe for one or two minutes, but if it costs more time, I'd just rollback my git installation. Good Luck rolling back GitHub.
    - adeelk93 5y ago
      But what about rolling back someone else’s installation? What if you’re depending on a different user that has a buggy installation?
- viccuad 5y ago
  I would love to see a Git+Matrix forge, completely decentralised. With the spaces and the upcoming threads, seems like it wouldn't be so difficult to create a client that exposes that. It would be easier to make it closer to Gerrit than Github review process, which would be a step up!
  - nshntarora 5y ago
    Gitlab is uniquely positioned to do this Activity Pub integration issue on Gitlab: https://gitlab.com/gitlab-org/gitlab/-/issues/21582 https://gitlab.com/gitlab-org/gitlab/-/issues/21582
grumple 5y ago
We ran into intermittent failure with Github download urls around a month or two ago that caused our builds to fail (there was no github status, but we replicated the failure easily manually). In response to the failure, we started self-hosting the dependency. The fewer external dependencies you depend on at the actual point of builds/deploys, the better. Since you're using a fixed version of the dependency anyway, you might as well self-host or include it in the ami or container.
sashk 5y ago
Ran into this issue when was updating package. But then, I did not find anywhere mentioned this url documented anywhere. I.e currently broken link to archives https://github.com/USER/REPO/archive/TAG/REPO-TAG.tar.gz https://github.com/USER/REPO/archive/TAG/REPO-TAG.tar.gz vs currently working and documented URL https://github.com/USER/REPO/archive/refs/tags/TAG.tar.gz https://github.com/USER/REPO/archive/refs/tags/TAG.tar.gz Is there a list of pre-defined URLs supported by GitHub?
Karellen 5y ago
> Arch Linux as well for every package downloading tarballs from GitHub. Packages checking out the git source tree are not affected, but they are a minority. It feels like people are using git wrong.
- anjbe 5y ago
  When building distro packages from source, the full revision history of a Git repo is unnecessary. Downloading just the source code of a specific commit or tag with curl is simpler, faster, has fewer dependencies, and is less prone to breakage (well, unless your URLs change under you, as happened here).
  - jeroenhd 5y ago
    Git allows checking out only a limited set of changes with the --depth flag: git clone --branch v1.0 --depth 1 https://github.com/example/example.git The parameter is called --branch but it also takes tags. It's not as fast as fetching a ZIP file, but it gets pretty close. From my count, this method only requires one dependency (git) whereas the curl + unzip method requires, well, both curl and unzip. The zip download method (download, decompress, build, compress into package, decompress onto system) is already silly enough, the first decompression part can easily be dropped.
    - anjbe 5y ago
      > It's not as fast as fetching a ZIP file, but it gets pretty close. In the context of distro packages (the bug report mentioned OpenBSD and Fedora) you might be building tens of thousands of packages of which thousands are likely to come from GitHub. A small difference becomes greatly magnified. > From my count, this method only requires one dependency (git) whereas the curl + unzip method requires, well, both curl and unzip. You’re forgetting that git itself depends on curl.
  - Karellen 5y ago
    The first time, sure. But the second time, a `git pull` should normally be significantly quicker. Heck, you could check if the tag you want already exists in the repo you have and skip the `git pull` a lot of the time.

206 points by wyoh 5y ago

Learn Ethical Hacking And Save The World: Hacking Benefits: https://www.hackerslist.co/learn-ethical-hacking-and-save-the-world-hacking-benefits/ https://www.hackerslist.co/learn-ethical-hacking-and-save-th...

gudmundur 5y ago

Hi everyone. I'm an engineer at GitHub and I just posted a response to this issue here: https://github.com/github/feedback/discussions/8149#discussioncomment-1712006 https://github.com/github/feedback/discussions/8149#discussi....

Rabidgremlin 5y ago
Is this issue also related? https://github.com/explosion/spaCy/issues/9606 https://github.com/explosion/spaCy/issues/9606
kzrdude 5y ago
Thanks for fixing! Sorry for being curious.. does Github have an office in Copenhagen? That would be cool.
- gudmundur 5y ago
  Yes, we do. I’ve been at GitHub longer than we’ve had that office, so I tend to work from home anyways.
  - Liquidor 5y ago
    That's amazing. Had no idea we had a GitHub office here, let alone in the country! If you're working remotely, are you employed by Californian standards/regulations/benefits or by local ones here in Denmark? If you don't mind me asking.
    - gudmundur 5y ago
      I am employed here in Denmark, wouldn’t have it any other way.
dehrmann 5y ago
> A change in the handling of URL schemes was deployed a couple of days ago This was deployed Thanksgiving week? I realize Github isn't a consumer company, so it doesn't face the same pressures as Amazon, but I'm surprised there wasn't a code freeze so people can have a quiet holiday weekend.
- bengale 5y ago
  Is everyone at GitHub based in America?
  - mdoms 5y ago
    No, they are not. I know at least one Kiwi who worked remotely for Github from New Zealand.
  - jsnell 5y ago
    No. But this appears to now be the most important online shopping week of the year in all of Europe as well. Shops are abnormally sensitive to outages. You'd expect all kinds of service providers to be extremely conservative with code and config pushes this week due to that. Maybe GitHub is far enough removed from the actual consumers that they don't feel the pressure, but it does seem surprising.
  - novok 5y ago
    If a good chunk of them are, it's still good to do a code freeze. If %30-90 of your company is on holiday, not a good time to do major things.

intunderflow 5y ago

Combined with the outage yesterday this hasn't been a good weekend for GitHub SRE's I'm surprised something like this happened on a weekend though since I wouldn't expect anyone to be changing anything in the codebase (then again it could be some infra has just ran out of storage or etc)

yardstick 5y ago
Perhaps there’s a lack of attention due to the holiday weekend in the US? Seems like major incidents in infrastructure and services tend to happen over holidays (general observation, not GitHub specific).
- deleted 5y ago
  [deleted]

junon 5y ago

Another gripe; unrelated, but since we're piling on... My username ends in a hyphen. Apparently, that's no longer allowed, though my username appears to be grandfathered in. Trying to give feedback about new experimental features lands me on the GitHub communities site, which is treated as a standalone app and thus requires you to log in via GitHub (it doesn't re-use the existing session token). However, Communities won't let me sign up with my username since it has a hanging hyphen, and I can't change the username in the form. So I effectively can't sign up. Support has not responded for over a month. Feels like things are inching toward getting worse with GitHub.

favadi 5y ago
I still remember the night when logrus's author decided to rename his Github account and broke our production build (https://github.com/sirupsen/logrus/pull/384 https://github.com/sirupsen/logrus/pull/384). Since then, I always vendor third party dependencies.
- junon 5y ago
  To be fair, this is a side effect of poor module system design in Go. Not really github's fault here.
Eikon 5y ago
Sounds like a very important and critical issue that should be prioritised.
belter 5y ago
I am surprised a Microsoft MVP Certified Professional expert did not pop up yet in the newsgroup, asking you to reboot your machine and review the steps in a certain technote ;-)
- Causality1 5y ago
  Hey now, that's libel. There are problems that can be solved by rebooting, and a Microsoft MVP would never post a genuine potential solution. I'm pretty sure you meant to say "an MVP popped up in the newsgroup to copy/paste a paragraph of random intro text and then ask if he's solved your problem".

the_duke 5y ago

At least now I know why my nix builds are failing... Odd for something like this to slip through and not be rolled back immediately. Unless it was intentional, in which case it would be even more odd to not communicate this widely beforehand.

KennyBlanken 5y ago
If Github doesn't at least monitor their 404 error rate for large-scale spikes, whoever is in charge of SRE should be fired. With no announcements and no response to a now two day old bug report, I see two possibilities: 1)Their monitoring of their infrastructure and monitoring of issues is shockingly incompetent for a company of their size and importance (the fact that it is a US holiday is irrelevant.) 2)This was 100% intentional and they're purposefully looking "incompetent" to get people to shift to using other services for downloads. My money is on the latter, given others in this discussion are reporting random download link failures starting a month or two ago. A huge number of projects seem to use GitHub as a sort of free file hosting service. I imagine the opex for both storage and bandwidth is a not insignificant amount of money and someone has been told to shoo the freeloaders off the grass. Announcing they're ending free file hosting for unpaid projects would generate a lot of noise and PR. Instead they just make it unreliable, and people go elsewhere. Multiple people in this discussion have described moving downloads of Github in response, which is exactly what Github likely wants.
- tata71 5y ago
  > If Github doesn't at least monitor their 404 error rate for large-scale spikes "Is that a service we can charge for?!"
- skyeto 5y ago
  From the discussion thread on GitHub[0]: A change in the handling of URL schemes was deployed a couple of days ago that caused the regression being discussed here. Due to the amount of traffic that the archive endpoints see, and the high baseline of 404s on them, this regression did not cause an unusual increase of errors that would've caused our alerting to kick in. The change has just been rolled back, so the issue is fixed. We will investigate this issue further after the weekend and take the appropriate steps to make sure similar regressions don't happen in the future. [0] https://github.com/github/feedback/discussions/8149#discussioncomment-1712006 https://github.com/github/feedback/discussions/8149#discussi...

AshamedCaptain 5y ago

Say what you want about "cloud" reliability, but my little home server in a residential ISP has been up for more time than Github.com in (at least!) the past year.

seoaeu 5y ago
And if github.com tried to host their website entirely off your little home server, they’d surely have 24x7 outages from being bombarded by too much load. All your anecdote proves is that it is easier to keep a single server online than operate a big distributed system, which has been obvious for quite a while
- dvdkon 5y ago
  Obvious, maybe, but there are many who will criticise anyone self-hosting, saying that a cloud solution will inherently be more stable, since it has a bigger ops team.
  - donny2018 5y ago
    Well, GitHub came back online without me doing anything. GitHub’s life depends on providing a good service to customers. I think occasional downtime is a good tradeoff for what I am getting, as opposed to having to manage my own server.
    - mistrial9 5y ago
      > GitHub came back online without me doing anything the Executive Privilege for the common committer ! just keep your keys "valid" and don't talk to anyone we ban up next -- Github social scores
- AshamedCaptain 5y ago
  No, it is not "obvious" to everyone at all. You can still see people here claiming that one should move to a centralized provider since they can guarantee nine nines of whatever, and that self-hosting is way too hard to make reliable. (Which is double irony when the centralized provider goes down and then the excuse is "well, that's because they're big!". If only...). In any case, the point was that Github.com just sucks, rather than everything cloud sucks. For the past year, they have been down a couple of magnitudes more time than I have spent managing my server.

mfashby 5y ago

This has also broken a bunch of packages in the arch user repository, for example https://aur.archlinux.org/packages/dendrite/ https://aur.archlinux.org/packages/dendrite/ :(

Semaphor 5y ago
Yeah, I thought the package I tried to install had an issue, but didn’t have the time to investigate. This explains it.
viccuad 5y ago
This sounds for the better. Not having code mirrors (as other distribution channels) sounds not just insecure, but borderly malicious.
- themusicgod1 5y ago
  Using github, period, is not just insecure, but actively malicious. Stop using NSA/Microsoft.
- turminal 5y ago
  I suspect you have no idea what AUR is and how it works and furthermore, you have no experience with software packaging. If a project is using github to publish releases, where else are consumers of that software going to get them from? Having all sources of everything that is packaged backed up is a must for the official repository of a competent distro, but even in that case there is no reason not to use github in normal operation.
- mfashby 5y ago
  Depends what you mean by better. It's kind of annoying for me when trying to install some software I want to actually use, and I just can't.

_zywo 5y ago

Whether this specific problem is intentional or not, these kinds of problems show the issue with using a single centralized service for distribution of third-party dependencies. But it's just so much more darned convenient than hosting your own Git server! It would be super cool if there was a decentralized alternative to GitHub, that used Git under the hood. Perhaps one would upload their repositories to a node, which would then be synchronized with all other nodes, and all you would need to do to use it is to specify any_node.com/author/project. This would keep GitHub's discoverability, while allowing all the benefits of decentralization.

adeelk93 5y ago
Breaking changes aren’t only an issue with centralization. A breaking change with git itself would mess up your decentralized scenario as well.
- TobTobXX 5y ago
  Maybe for one or two minutes, but if it costs more time, I'd just rollback my git installation. Good Luck rolling back GitHub.
  - adeelk93 5y ago
    But what about rolling back someone else’s installation? What if you’re depending on a different user that has a buggy installation?
viccuad 5y ago
I would love to see a Git+Matrix forge, completely decentralised. With the spaces and the upcoming threads, seems like it wouldn't be so difficult to create a client that exposes that. It would be easier to make it closer to Gerrit than Github review process, which would be a step up!
- nshntarora 5y ago
  Gitlab is uniquely positioned to do this Activity Pub integration issue on Gitlab: https://gitlab.com/gitlab-org/gitlab/-/issues/21582 https://gitlab.com/gitlab-org/gitlab/-/issues/21582

grumple 5y ago

We ran into intermittent failure with Github download urls around a month or two ago that caused our builds to fail (there was no github status, but we replicated the failure easily manually). In response to the failure, we started self-hosting the dependency. The fewer external dependencies you depend on at the actual point of builds/deploys, the better. Since you're using a fixed version of the dependency anyway, you might as well self-host or include it in the ami or container.

sashk 5y ago

Ran into this issue when was updating package. But then, I did not find anywhere mentioned this url documented anywhere. I.e currently broken link to archives https://github.com/USER/REPO/archive/TAG/REPO-TAG.tar.gz https://github.com/USER/REPO/archive/TAG/REPO-TAG.tar.gz vs currently working and documented URL https://github.com/USER/REPO/archive/refs/tags/TAG.tar.gz https://github.com/USER/REPO/archive/refs/tags/TAG.tar.gz Is there a list of pre-defined URLs supported by GitHub?

Karellen 5y ago

> Arch Linux as well for every package downloading tarballs from GitHub. Packages checking out the git source tree are not affected, but they are a minority. It feels like people are using git wrong.

anjbe 5y ago
When building distro packages from source, the full revision history of a Git repo is unnecessary. Downloading just the source code of a specific commit or tag with curl is simpler, faster, has fewer dependencies, and is less prone to breakage (well, unless your URLs change under you, as happened here).
- jeroenhd 5y ago
  Git allows checking out only a limited set of changes with the --depth flag: git clone --branch v1.0 --depth 1 https://github.com/example/example.git The parameter is called --branch but it also takes tags. It's not as fast as fetching a ZIP file, but it gets pretty close. From my count, this method only requires one dependency (git) whereas the curl + unzip method requires, well, both curl and unzip. The zip download method (download, decompress, build, compress into package, decompress onto system) is already silly enough, the first decompression part can easily be dropped.
  - anjbe 5y ago
    > It's not as fast as fetching a ZIP file, but it gets pretty close. In the context of distro packages (the bug report mentioned OpenBSD and Fedora) you might be building tens of thousands of packages of which thousands are likely to come from GitHub. A small difference becomes greatly magnified. > From my count, this method only requires one dependency (git) whereas the curl + unzip method requires, well, both curl and unzip. You’re forgetting that git itself depends on curl.
- Karellen 5y ago
  The first time, sure. But the second time, a `git pull` should normally be significantly quicker. Heck, you could check if the tag you want already exists in the repo you have and skip the `git pull` a lot of the time.