7 ms·

Hey folks. I'm the product manager for Git at GitHub. We're sorry for the breakage, we're reverting the change, and we'll communicate better about such changes

Hey folks. I'm the product manager for Git at GitHub. We're sorry for the breakage, we're reverting the change, and we'll communicate better about such changes in the future (including timelines). Also posted here: https://github.com/bazel-contrib/SIG-rules-authors/issues/11#issuecomment-1409438954 https://github.com/bazel-contrib/SIG-rules-authors/issues/11...

deleted 4y ago
[deleted]
vtbassmatt 4y ago
We updated our Git version which made this change for the reasons explained. At the time we didn't foresee the impact. We're quickly rolling back the change now, as it's clear we need to look at this more closely to see if we can make the changes in a less disruptive way. Thanks for letting us know.
- phphphphp 4y ago
  Consumers often mistake hasn’t changed for a commitment to never change: any sufficiently large product will be littered with these kind of implicit commitments made by the product to consumers that nobody has visibility into. You’re unfortunate that we were all relying on this commitment you’ve never made, but the quick reversion is the best we can hope for. People will theorise how this could have been avoided but c’est la vie — easy mistake that you’ve responded well to.
  - VWWHFSfQ 4y ago
    At this point they'll be stuck on old git for all of eternity unless they just roll their own archive/compress step out of band so the old hashes still work. Yikes.
    - groestl 4y ago
      They could also brownout the implied contract over a longer timespan.
      - jeffbee 4y ago
        They could use the old behavior for archives in which all the inputs predate the changeover.
    - rfoo 4y ago
      New git has a flag for keeping the old behavior, so it's not as bad.
  - deleted 4y ago
    [deleted]
  - dharmab 4y ago
    Hyrum's Law: With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.
    - deleted 4y ago
      [deleted]
    - modderation 4y ago
      Incidentally, I'm still waiting for GitHub to implement Spacebar Heating: https://m.xkcd.com/1172/ https://m.xkcd.com/1172/
  - nickitolas 4y ago
    FWIW according to https://github.com/bazel-contrib/SIG-rules-authors/issues/11#issuecomment-1409433084 https://github.com/bazel-contrib/SIG-rules-authors/issues/11... a commitment was made, although in an exchange in some support ticket, and not in documentation.
- johnklos 4y ago
  [flagged]
  - VWWHFSfQ 4y ago
    Ironic that "open source packaging systems" rely on proprietary Microsoft hosting and distribution to function. I think you meant _poorly implemented_ open source packaging systems.
    - bentley 4y ago
      Of the 11,656 packages in OpenBSD’s package repos, 2,984 are built from source originally hosted on GitHub or Sourceforge. That’s a full 25%. Moralize all you want about where these upstreams should host their software, but why claim that the downstream package manager is “poorly implemented” to fetch source code from those hosts? Your complaint was not technical—you imply the proprietariness of Microsoft servers is the problem (although open source servers like GitLab also have the problem of unstable checksums)—but HTTPS is HTTPS.
      - VWWHFSfQ 4y ago
        I'm fairly certain that Homebrew doesn't function without GitHub. The index (and Cargo's index, and probably others) is hard-coded to be hosted on GitHub.
        denom 4y ago
        Exactly this. Tbh, i would expect some developer working on this feature to have an ‘a-ha’ moment: ‘I’m a homebrew user … hey wait a minute’
      - ilyt 4y ago
        > Moralize all you want about where these upstreams should host their software, but why claim that the downstream package manager is “poorly implemented” to fetch source code from those hosts? Because it should validate checksums of content of the tarball, instead of just the outside blob. Then: * you don't care about compression method or implementation * you don't care about archive method or implementation * your system works just as well for "download a tarball" as for "shallow copy the remote repo
denom 4y ago
In my particular use-case, I'm using a set of local dev tools hosted as a homebrew tap. The build looks up the github tar.gz release for each tag and commits the sha256sum of that file to the formula What's odd is that all the _historical_ tags have broken release shasums. Does this mean the entire set of zip/tar.gz archives has been rebuilt? That could be a problem, as perhaps you cannot easily back out of this change...
- crote 4y ago
  The trick here is that a Github release is in essence simply a tag of a specific commit. There is no need to build archives in advance, as they can be dynamically generated from the git repo. However, if you change the compression algorithm used to generate the archive, it'll result in a different checksum! The content is the same, but the archive is not.
- Denvercoder9 4y ago
  > Does this mean the entire set of zip/tar.gz archives has been rebuilt? They are probably generated on-demand (and cached) from the Git repository, not prebuilt.
- scyrybdis 4y ago
  I think the zip/tar.gz archives are being created on the fly when you download them, probably with a caching layer in front.
  - tinus_hn 4y ago
    Pretty bizarre this ever was stable in the first place. Unfortunately for this kind of service you need to actively fiddle with the bytes to prevent people from relying on an implementation detail like this and prevent them from digging you into a too big to fail api stability hole.
    - denom 4y ago
      Alternatively, they could extract the compression code and maintain it for repo tags created before the git algo update release date. Isn’t that the only humane course given all that depends on this?
kris-nova 4y ago
Thanks for the update! There is only 1 internet to watch and learn from. We are all in this together. <3
- deleted 4y ago
  [deleted]
mdouglass 4y ago
We are seeing an npm install failure inside our docker builds pointing at a github URL with a SHA change. Is this possibly related? #15 [dev-builder 4/7] RUN --mount=type=secret,id=npm,dst=/root/.npmrc npm ci #0 4.743 npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead. #0 8.119 npm WARN tarball tarball data for http2@https://github.com/node-apn/node-http2/archive/apn-2.1.4.tar.gz (sha512-ad4u4I88X9AcUgxCRW3RLnbh7xHWQ1f5HbrXa7gEy2x4Xgq+rq+auGx5I+nUDE2YYuqteGIlbxrwQXkIaYTfnQ==) seems to be corrupted. Trying again. #0 8.164 npm ERR! code EINTEGRITY #0 8.169 npm ERR! sha512-ad4u4I88X9AcUgxCRW3RLnbh7xHWQ1f5HbrXa7gEy2x4Xgq+rq+auGx5I+nUDE2YYuqteGIlbxrwQXkIaYTfnQ== integrity checksum failed when using sha512: wanted sha512-ad4u4I88X9AcUgxCRW3RLnbh7xHWQ1f5HbrXa7gEy2x4Xgq+rq+auGx5I+nUDE2YYuqteGIlbxrwQXkIaYTfnQ== but got sha512-GWBlkDNYgpkQElS+zGyIe1CN/XJxdEFuguLHOEGLZOIoDiH4cC9chggBwZsPK/Ls9nPikTzMuRDWfLzoGlKiRw==. (72989 bytes) #0 8.176 #0 8.177 npm ERR! A complete log of this run can be found in: #0 8.177 npm ERR! /root/.npm/_logs/2023-01-30T23_19_36_986Z-debug-0.log #15 ERROR: process "/bin/sh -c npm ci" did not complete successfully: exit code: 1 This was working earlier today and the docker build/package.json haven't changed.
- andrewguenther 4y ago
  Yes, this is the exact issue being described
  - mdouglass 4y ago
    That's what I thought, but I assumed with the rollback an hour plus ago, it wouldn't still be happening. That was off a build just a few minutes ago (actually repeated it in between the time I posted my original message and this reply and it happened again).
    - VWWHFSfQ 4y ago
      Most likely a caching layer at GitHub still has the pre-rollback archive.
- voidbip 4y ago
rsc 4y ago
Thanks for the quick rollback. I want to encourage you to think about locking in the current archive details, at least for archives that have already been served. Verifying that downloaded archives have the expected checksum is a critical best practice for software supply chain security. Training people to ignore checksum changes is training them to ignore attacks. GitHub is a strong leader in other parts of supply chain security, and it can lead here too. Once GitHub has served an archive with a given checksum, it should guarantee that the archive has that checksum forever.
- bentley 4y ago
  I would also appreciate stronger advertising of the ability to turn a Git tag into a GitHub release and upload stable source code files to it. Maybe even a button in the GitHub releases interface to “generate source tarball and attach as stable tarball to this release.”
  - xwolfi 4y ago
    [flagged]
  - misnome 4y ago
    But this isn’t a great solution, because afterwards there is now three, or four source download links, some of which are stable. Not to mention, forcing people to use GitHub releases instead of just tags (which excludes every mirror of somewhere else)
  - mathstuf 4y ago
    I agree this would be great. However, it should also stop you from providing useless tarballs (as `/archive/` does today) if: - you use autoconf (or any other tool(s) that require generating code into the source archive; or - you have submodules (to which `git archive` is completely blind). Note that `git-archive-all`[1] can help as long as your submodules don't do things like `[attr]custom-attr` in their `.gitattributes` as it is only allowed in the top-level `.gitattributes` file and cannot be added to the tree otherwise. [1]https://github.com/roehling/git-archive-all https://github.com/roehling/git-archive-all

deleted 4y ago

[deleted]

vtbassmatt 4y ago

We updated our Git version which made this change for the reasons explained. At the time we didn't foresee the impact. We're quickly rolling back the change now, as it's clear we need to look at this more closely to see if we can make the changes in a less disruptive way. Thanks for letting us know.

phphphphp 4y ago
Consumers often mistake hasn’t changed for a commitment to never change: any sufficiently large product will be littered with these kind of implicit commitments made by the product to consumers that nobody has visibility into. You’re unfortunate that we were all relying on this commitment you’ve never made, but the quick reversion is the best we can hope for. People will theorise how this could have been avoided but c’est la vie — easy mistake that you’ve responded well to.
- VWWHFSfQ 4y ago
  At this point they'll be stuck on old git for all of eternity unless they just roll their own archive/compress step out of band so the old hashes still work. Yikes.
  - groestl 4y ago
    They could also brownout the implied contract over a longer timespan.
    - jeffbee 4y ago
      They could use the old behavior for archives in which all the inputs predate the changeover.
  - rfoo 4y ago
    New git has a flag for keeping the old behavior, so it's not as bad.
- deleted 4y ago
  [deleted]
- dharmab 4y ago
  Hyrum's Law: With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.
  - deleted 4y ago
    [deleted]
  - modderation 4y ago
    Incidentally, I'm still waiting for GitHub to implement Spacebar Heating: https://m.xkcd.com/1172/ https://m.xkcd.com/1172/
- nickitolas 4y ago
  FWIW according to https://github.com/bazel-contrib/SIG-rules-authors/issues/11#issuecomment-1409433084 https://github.com/bazel-contrib/SIG-rules-authors/issues/11... a commitment was made, although in an exchange in some support ticket, and not in documentation.
johnklos 4y ago
[flagged]
- VWWHFSfQ 4y ago
  Ironic that "open source packaging systems" rely on proprietary Microsoft hosting and distribution to function. I think you meant _poorly implemented_ open source packaging systems.
  - bentley 4y ago
    Of the 11,656 packages in OpenBSD’s package repos, 2,984 are built from source originally hosted on GitHub or Sourceforge. That’s a full 25%. Moralize all you want about where these upstreams should host their software, but why claim that the downstream package manager is “poorly implemented” to fetch source code from those hosts? Your complaint was not technical—you imply the proprietariness of Microsoft servers is the problem (although open source servers like GitLab also have the problem of unstable checksums)—but HTTPS is HTTPS.
    - VWWHFSfQ 4y ago
      I'm fairly certain that Homebrew doesn't function without GitHub. The index (and Cargo's index, and probably others) is hard-coded to be hosted on GitHub.
      - denom 4y ago
        Exactly this. Tbh, i would expect some developer working on this feature to have an ‘a-ha’ moment: ‘I’m a homebrew user … hey wait a minute’
    - ilyt 4y ago
      > Moralize all you want about where these upstreams should host their software, but why claim that the downstream package manager is “poorly implemented” to fetch source code from those hosts? Because it should validate checksums of content of the tarball, instead of just the outside blob. Then: * you don't care about compression method or implementation * you don't care about archive method or implementation * your system works just as well for "download a tarball" as for "shallow copy the remote repo

denom 4y ago

In my particular use-case, I'm using a set of local dev tools hosted as a homebrew tap. The build looks up the github tar.gz release for each tag and commits the sha256sum of that file to the formula What's odd is that all the _historical_ tags have broken release shasums. Does this mean the entire set of zip/tar.gz archives has been rebuilt? That could be a problem, as perhaps you cannot easily back out of this change...

crote 4y ago
The trick here is that a Github release is in essence simply a tag of a specific commit. There is no need to build archives in advance, as they can be dynamically generated from the git repo. However, if you change the compression algorithm used to generate the archive, it'll result in a different checksum! The content is the same, but the archive is not.
Denvercoder9 4y ago
> Does this mean the entire set of zip/tar.gz archives has been rebuilt? They are probably generated on-demand (and cached) from the Git repository, not prebuilt.
scyrybdis 4y ago
I think the zip/tar.gz archives are being created on the fly when you download them, probably with a caching layer in front.
- tinus_hn 4y ago
  Pretty bizarre this ever was stable in the first place. Unfortunately for this kind of service you need to actively fiddle with the bytes to prevent people from relying on an implementation detail like this and prevent them from digging you into a too big to fail api stability hole.
  - denom 4y ago
    Alternatively, they could extract the compression code and maintain it for repo tags created before the git algo update release date. Isn’t that the only humane course given all that depends on this?

kris-nova 4y ago

Thanks for the update! There is only 1 internet to watch and learn from. We are all in this together. <3

deleted 4y ago
[deleted]

mdouglass 4y ago

We are seeing an npm install failure inside our docker builds pointing at a github URL with a SHA change. Is this possibly related? #15 [dev-builder 4/7] RUN --mount=type=secret,id=npm,dst=/root/.npmrc npm ci #0 4.743 npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead. #0 8.119 npm WARN tarball tarball data for http2@https://github.com/node-apn/node-http2/archive/apn-2.1.4.tar.gz (sha512-ad4u4I88X9AcUgxCRW3RLnbh7xHWQ1f5HbrXa7gEy2x4Xgq+rq+auGx5I+nUDE2YYuqteGIlbxrwQXkIaYTfnQ==) seems to be corrupted. Trying again. #0 8.164 npm ERR! code EINTEGRITY #0 8.169 npm ERR! sha512-ad4u4I88X9AcUgxCRW3RLnbh7xHWQ1f5HbrXa7gEy2x4Xgq+rq+auGx5I+nUDE2YYuqteGIlbxrwQXkIaYTfnQ== integrity checksum failed when using sha512: wanted sha512-ad4u4I88X9AcUgxCRW3RLnbh7xHWQ1f5HbrXa7gEy2x4Xgq+rq+auGx5I+nUDE2YYuqteGIlbxrwQXkIaYTfnQ== but got sha512-GWBlkDNYgpkQElS+zGyIe1CN/XJxdEFuguLHOEGLZOIoDiH4cC9chggBwZsPK/Ls9nPikTzMuRDWfLzoGlKiRw==. (72989 bytes) #0 8.176 #0 8.177 npm ERR! A complete log of this run can be found in: #0 8.177 npm ERR! /root/.npm/_logs/2023-01-30T23_19_36_986Z-debug-0.log #15 ERROR: process "/bin/sh -c npm ci" did not complete successfully: exit code: 1 This was working earlier today and the docker build/package.json haven't changed.

andrewguenther 4y ago
Yes, this is the exact issue being described
- mdouglass 4y ago
  That's what I thought, but I assumed with the rollback an hour plus ago, it wouldn't still be happening. That was off a build just a few minutes ago (actually repeated it in between the time I posted my original message and this reply and it happened again).
  - VWWHFSfQ 4y ago
    Most likely a caching layer at GitHub still has the pre-rollback archive.
voidbip 4y ago

rsc 4y ago

Thanks for the quick rollback. I want to encourage you to think about locking in the current archive details, at least for archives that have already been served. Verifying that downloaded archives have the expected checksum is a critical best practice for software supply chain security. Training people to ignore checksum changes is training them to ignore attacks. GitHub is a strong leader in other parts of supply chain security, and it can lead here too. Once GitHub has served an archive with a given checksum, it should guarantee that the archive has that checksum forever.

bentley 4y ago
I would also appreciate stronger advertising of the ability to turn a Git tag into a GitHub release and upload stable source code files to it. Maybe even a button in the GitHub releases interface to “generate source tarball and attach as stable tarball to this release.”
- xwolfi 4y ago
  [flagged]
- misnome 4y ago
  But this isn’t a great solution, because afterwards there is now three, or four source download links, some of which are stable. Not to mention, forcing people to use GitHub releases instead of just tags (which excludes every mirror of somewhere else)
- mathstuf 4y ago
  I agree this would be great. However, it should also stop you from providing useless tarballs (as `/archive/` does today) if: - you use autoconf (or any other tool(s) that require generating code into the source archive; or - you have submodules (to which `git archive` is completely blind). Note that `git-archive-all`[1] can help as long as your submodules don't do things like `[attr]custom-attr` in their `.gitattributes` as it is only allowed in the top-level `.gitattributes` file and cannot be added to the tree otherwise. [1]https://github.com/roehling/git-archive-all https://github.com/roehling/git-archive-all