5 ms·

Why is this a thing? Can't packages use specific tags from the git repo? It seems so incredibly stupid to allow this, throwing out all of the "oh but it's open

Why is this a thing? Can't packages use specific tags from the git repo? It seems so incredibly stupid to allow this, throwing out all of the "oh but it's open source you can review it" arguments in one go if the source displayed on GitHub is not what ends up used...

atlasduo 3y ago
Tags can be changed. To actually pin the source code revision, they should pin to a specific commit hash.
- viraptor 3y ago
  Sometimes they get changed for a silly reason (someone just didn't think of the consequences). Sometimes they get changed because they have to - if I remember correctly it was Asterisk that needed to drop some copyrighted music samples and retagged old versions.
  - gunapologist99 3y ago
    That shouldn't matter, then; it should just result in a new release.
    - viraptor 3y ago
      You can't distribute the old versions with copyright violation. The problem doesn't go away just because you released a new version.
      - nilamo 3y ago
        But then you have two different things (with and without copyrighted data) both with the same version. Wouldn't it be better to just remove the bad release, and publish a new one? They are, after all, not the same things, and shouldn't have the same release.
- andersa 3y ago
  Good point, that would make sense. But same difference. Why is that not how it works?
senectus1 3y ago
my understanding is for speed reasons most distro's dont build from the source they build from tarballs. If they built from the source there wouldn't be this issue.
- martijnvds 3y ago
  The tarballs also contain source. Just not the source control/revision history.
- b112 3y ago
  If by "source" you mean the project's version controlled repo, it is for a variety of reasons. One primary one is, sustainability. The build tarballs are kept forever. Try that with a remote repo that could vanish in a mere 10 or 20 years.. or tomorrow! And tar is the most stable archive format in existence, with at least half a century of use. note: Companies that care, should ask themselves what do you do, if you have a build system which relies upon externals, and a part of that build goes down? And you have an urgent fix to PROD required? Hope you can find all the bits, unvarnished, scattered on dev boxes? Cobble them together and hope they build, while your PROD is currently borked? Or do you hotpatch PROD? If your build process breaks due to an external repo going MIA, then you're doing it wrong.
- ZiiS 3y ago
  Do you mean if they built from a git hash? The tarball contains "Source code" which is what they are building.
mjochim 3y ago
The path from source code to distributed binary file is known to be a blind spot. Removing that blind spot is either Harder Than You'd Think or Easier Than You'd Think, depending on your perspective and expectations. You can find some issues listed here: https://reproducible-builds.org/docs/ https://reproducible-builds.org/docs/ Or the homepage of reproducible-builds.org for a general take on the subject. (I am not associated with that website.)
- ncruces 3y ago
  Definitely Harder Than You'd Think and we've known this for a very long time. https://research.swtch.com/nih https://research.swtch.com/nih
segfaultbuserr 3y ago
Historically, it was done for end-user convenience. Traditionally, each project has two separate sources, one is the actual development repository, strictly for use by developers. The next is the source tarball for end-user installation, pre-generated by developers by preprocessing the source repo - such as autotools script generation, gettext translation file generation, etc. The idea was that the end users were running on many flavor of incompatible Unix systems, so installing the full development tools such as autotools or CVS/SVN could be inconvenient. To avoid those troubles, developers pre-generate tarballs with necessary installation scripts such as ./configure for user-friendliness - so source tarballs are installers in a sense, at the middle way between actual source and pre-complied binary. On the other hand, because these scripts are machine-generated and are extremely sensitive to small chances to Makefiles, they should not pollute the canonical source tree, so they're not included in the source repository. Occasionally, under the strict Cathedral style of software development, the canonical source tree may even be private under exclusive access by the core team, end users only have access to release tarballs. Nowadays, the repo-tarball split is largely unnecessary, but the practice remains. The xz incident became a wake-up call that the Reproducible Build movement focused on binary reproducibility, but have so far ignored tarball reproducibility. Hopefully, this problem will be addressed by the community in the future.
- tutfbhuf 3y ago
  [flagged]
  - loloquwowndueo 3y ago
    We don’t understand how AI works. I would not trust an AI to not hallucinate unsafely in this context.
  - zopa 3y ago
    You wouldn’t want it in the CI pipeline, because any model clever enough to find real issues is also going to find plenty of false positives. That seems like too much friction for most open source projects. I’m not one of the downvoters, but you’ve linked to a list of forty or fifty different projects, many of which don’t seem relevant to this use-case. It’s not too surprising people have nothing to say besides “ugh, more AI hype.”
gwd 3y ago
For Xen, it's historical reasons funneled into "don't break things for your users". In the olden times, Xen had our own fork of QEMU, as well as our own fork of Linux, and some other useful tools like pvgrub. These were developed in a separate repository, but it was imported into the main Xen release tarball, so that you could just download the main tarball, do "./configure && make && make install" and have a reasonably complete Xen system. These days most Linux kernels can do everything we need, and our release tag of QEMU might only be one or two patches not yet in the upstream branch; and in any case, you can always use the most upstream release. So this isn't necessary anymore (and indeed we only include QEMU in the release tarball, not Linux or pvgrub). And you can still "./configure && make && make install" to get a fairly complete system, it will just clone other repositories on your behalf. On the other hand, I actually checked just last month, and the tarball for Xen 4.18.0, released back in November, was getting 700 downloads a week. Who are all these people downloading the release tarball, rather than using the distro version of Xen? Do they want and need these extra bits inside? We don't know and were hesitant to make any breaking changes. I think with the xz fiasco, we now have justification to switch entirely to a `git archive` tarball of a specific tag, whether it's inconvenient for people or not.
- brendank310 3y ago
  It's been probably 2 years since I've messed with it, but I was still using the tarball download for Xen in OpenEmbedded builds.
mkj 3y ago
Software projects will outlive git. Mine has gone CVS -> Monotone -> Mercurial -> Git over 20 years, it'll probably move again. But you can still download the tarballs from any of the releases made with those various systems, and they still have the same sha1sum (or now sha256sum). Making reproducible tarballs from VCS isn't hard - with the advice from https://reproducible-builds.org/docs/archives/ https://reproducible-builds.org/docs/archives/ (and requiring gnu tar) it's possible to get byte exact output from both Mercurial or git mirrors, on both Linux and MacOS. Put that in the github CI and it's reasonably difficult to subvert (compare against a local build too at release time). It would be nice if "git archive" had guarantees about archive format, and if the github "download a tar.gz" matched that, but it doesn't seem to be the case at present.

atlasduo 3y ago

Tags can be changed. To actually pin the source code revision, they should pin to a specific commit hash.

viraptor 3y ago
Sometimes they get changed for a silly reason (someone just didn't think of the consequences). Sometimes they get changed because they have to - if I remember correctly it was Asterisk that needed to drop some copyrighted music samples and retagged old versions.
- gunapologist99 3y ago
  That shouldn't matter, then; it should just result in a new release.
  - viraptor 3y ago
    You can't distribute the old versions with copyright violation. The problem doesn't go away just because you released a new version.
    - nilamo 3y ago
      But then you have two different things (with and without copyrighted data) both with the same version. Wouldn't it be better to just remove the bad release, and publish a new one? They are, after all, not the same things, and shouldn't have the same release.
andersa 3y ago
Good point, that would make sense. But same difference. Why is that not how it works?

senectus1 3y ago

my understanding is for speed reasons most distro's dont build from the source they build from tarballs. If they built from the source there wouldn't be this issue.

martijnvds 3y ago
The tarballs also contain source. Just not the source control/revision history.
b112 3y ago
If by "source" you mean the project's version controlled repo, it is for a variety of reasons. One primary one is, sustainability. The build tarballs are kept forever. Try that with a remote repo that could vanish in a mere 10 or 20 years.. or tomorrow! And tar is the most stable archive format in existence, with at least half a century of use. note: Companies that care, should ask themselves what do you do, if you have a build system which relies upon externals, and a part of that build goes down? And you have an urgent fix to PROD required? Hope you can find all the bits, unvarnished, scattered on dev boxes? Cobble them together and hope they build, while your PROD is currently borked? Or do you hotpatch PROD? If your build process breaks due to an external repo going MIA, then you're doing it wrong.
ZiiS 3y ago
Do you mean if they built from a git hash? The tarball contains "Source code" which is what they are building.

mjochim 3y ago

The path from source code to distributed binary file is known to be a blind spot. Removing that blind spot is either Harder Than You'd Think or Easier Than You'd Think, depending on your perspective and expectations. You can find some issues listed here: https://reproducible-builds.org/docs/ https://reproducible-builds.org/docs/ Or the homepage of reproducible-builds.org for a general take on the subject. (I am not associated with that website.)

ncruces 3y ago
Definitely Harder Than You'd Think and we've known this for a very long time. https://research.swtch.com/nih https://research.swtch.com/nih

segfaultbuserr 3y ago

Historically, it was done for end-user convenience. Traditionally, each project has two separate sources, one is the actual development repository, strictly for use by developers. The next is the source tarball for end-user installation, pre-generated by developers by preprocessing the source repo - such as autotools script generation, gettext translation file generation, etc. The idea was that the end users were running on many flavor of incompatible Unix systems, so installing the full development tools such as autotools or CVS/SVN could be inconvenient. To avoid those troubles, developers pre-generate tarballs with necessary installation scripts such as ./configure for user-friendliness - so source tarballs are installers in a sense, at the middle way between actual source and pre-complied binary. On the other hand, because these scripts are machine-generated and are extremely sensitive to small chances to Makefiles, they should not pollute the canonical source tree, so they're not included in the source repository. Occasionally, under the strict Cathedral style of software development, the canonical source tree may even be private under exclusive access by the core team, end users only have access to release tarballs. Nowadays, the repo-tarball split is largely unnecessary, but the practice remains. The xz incident became a wake-up call that the Reproducible Build movement focused on binary reproducibility, but have so far ignored tarball reproducibility. Hopefully, this problem will be addressed by the community in the future.

tutfbhuf 3y ago
[flagged]
- loloquwowndueo 3y ago
  We don’t understand how AI works. I would not trust an AI to not hallucinate unsafely in this context.
- zopa 3y ago
  You wouldn’t want it in the CI pipeline, because any model clever enough to find real issues is also going to find plenty of false positives. That seems like too much friction for most open source projects. I’m not one of the downvoters, but you’ve linked to a list of forty or fifty different projects, many of which don’t seem relevant to this use-case. It’s not too surprising people have nothing to say besides “ugh, more AI hype.”

gwd 3y ago

For Xen, it's historical reasons funneled into "don't break things for your users". In the olden times, Xen had our own fork of QEMU, as well as our own fork of Linux, and some other useful tools like pvgrub. These were developed in a separate repository, but it was imported into the main Xen release tarball, so that you could just download the main tarball, do "./configure && make && make install" and have a reasonably complete Xen system. These days most Linux kernels can do everything we need, and our release tag of QEMU might only be one or two patches not yet in the upstream branch; and in any case, you can always use the most upstream release. So this isn't necessary anymore (and indeed we only include QEMU in the release tarball, not Linux or pvgrub). And you can still "./configure && make && make install" to get a fairly complete system, it will just clone other repositories on your behalf. On the other hand, I actually checked just last month, and the tarball for Xen 4.18.0, released back in November, was getting 700 downloads a week. Who are all these people downloading the release tarball, rather than using the distro version of Xen? Do they want and need these extra bits inside? We don't know and were hesitant to make any breaking changes. I think with the xz fiasco, we now have justification to switch entirely to a `git archive` tarball of a specific tag, whether it's inconvenient for people or not.

brendank310 3y ago
It's been probably 2 years since I've messed with it, but I was still using the tarball download for Xen in OpenEmbedded builds.

mkj 3y ago

Software projects will outlive git. Mine has gone CVS -> Monotone -> Mercurial -> Git over 20 years, it'll probably move again. But you can still download the tarballs from any of the releases made with those various systems, and they still have the same sha1sum (or now sha256sum). Making reproducible tarballs from VCS isn't hard - with the advice from https://reproducible-builds.org/docs/archives/ https://reproducible-builds.org/docs/archives/ (and requiring gnu tar) it's possible to get byte exact output from both Mercurial or git mirrors, on both Linux and MacOS. Put that in the github CI and it's reasonably difficult to subvert (compare against a local build too at release time). It would be nice if "git archive" had guarantees about archive format, and if the github "download a tar.gz" matched that, but it doesn't seem to be the case at present.