4 ms·

Historically, it was done for end-user convenience. Traditionally, each project has two separate sources, one is the actual development repository, strictly for

Historically, it was done for end-user convenience. Traditionally, each project has two separate sources, one is the actual development repository, strictly for use by developers. The next is the source tarball for end-user installation, pre-generated by developers by preprocessing the source repo - such as autotools script generation, gettext translation file generation, etc. The idea was that the end users were running on many flavor of incompatible Unix systems, so installing the full development tools such as autotools or CVS/SVN could be inconvenient. To avoid those troubles, developers pre-generate tarballs with necessary installation scripts such as ./configure for user-friendliness - so source tarballs are installers in a sense, at the middle way between actual source and pre-complied binary. On the other hand, because these scripts are machine-generated and are extremely sensitive to small chances to Makefiles, they should not pollute the canonical source tree, so they're not included in the source repository. Occasionally, under the strict Cathedral style of software development, the canonical source tree may even be private under exclusive access by the core team, end users only have access to release tarballs. Nowadays, the repo-tarball split is largely unnecessary, but the practice remains. The xz incident became a wake-up call that the Reproducible Build movement focused on binary reproducibility, but have so far ignored tarball reproducibility. Hopefully, this problem will be addressed by the community in the future.

tutfbhuf 3y ago
[flagged]
- loloquwowndueo 3y ago
  We don’t understand how AI works. I would not trust an AI to not hallucinate unsafely in this context.
- zopa 3y ago
  You wouldn’t want it in the CI pipeline, because any model clever enough to find real issues is also going to find plenty of false positives. That seems like too much friction for most open source projects. I’m not one of the downvoters, but you’ve linked to a list of forty or fifty different projects, many of which don’t seem relevant to this use-case. It’s not too surprising people have nothing to say besides “ugh, more AI hype.”
- VMG 3y ago
  You are wrong but there is no reason to downvote you. The idea might work in the future.
  - Kwpolska 3y ago
    It might work if AI stops producing random output and bullshit (which proponents call hallucinations to make it sound nice), and produces correct responses deterministically. Which may take forever.
weinzierl 3y ago
"The xz incident became a wake-up call that the Reproducible Build movement focused on binary reproducibility, but have so far ignored tarball reproducibility." The reproducibility must stretch over all the outputs. A related idea is the Hermetic Build", where the sameness of all the input is ensured, even including the build tools.
- yencabulator 3y ago
  In this case, inputs, or more like the whole chain toward "real inputs". A project can be reproducible when built from the release tarball, without being able to reproduce the release tarball from a git tag.
mid-kid 3y ago
> Nowadays, the repo-tarball split is largely unnecessary I think locking the source distribution mechanism to only git would be detrimental in the grand scheme. A source tarball is universal, independent of preferred tooling, easy to hash, sign and verify, and archive. Even git has loopholes in that a tag does not necessarily have to be a part of the master branch, and not everyone uses (or wants to use) github, or a similar online interface for git... It's also worth considering that source generation tools are often keen to changing their API frequently, while their output is more weaponized against the passage of time. Autotools is significantly better about that these days, but many other tools aren't...
- kevincox 3y ago
  I think most packagers that can pull from Git can also pull from Mercurial, SVN or whatever else. Pulling from Git doesn't really require using GitHub other than connecting to their IP. You don't need an account or any tool other than Git to clone a public repo. It seems to me that pulling directly from the primary source of truth is a good idea these days where possible.
- kruador 3y ago
  The Git protocol is also substantially more heavyweight than just downloading a tarball. 'Dumb' HTTP involves requesting `info/refs` which lists all the references (branches and tags) available. The client has to find the tag you want, then request the corresponding commit object. That lists the hash of the tree object, which has to be requested, and recursively the tree object references other trees and finally the file blobs. Any object request could fail because the object isn't stored 'loose' on disk, but is instead in a pack. So the client first downloads `objects/info/http-alternates` to see if it's in a different location, then if that doesn't list anything, it asks for the list of packs `objects/info/packs`. Then it asks for the index of each pack to determine which pack contains the object, then finally downloads that pack. Because that's very slow, Git also supports 'Smart' HTTP. That moves all the lookups server-side, with a back-and-forth between client and server about what the client has and what it wants, ultimately building a custom pack containing what you asked for. This obviously can't be cached. In contrast a tarball is one request for one file on disk which is fully cacheable, both by the server hosting the file and any proxies on the path. This is why things like `bower` and `DefinitelyTyped` got deprecated. Some of them were even hosting the registry itself as a Git repository on GitHub.
lamontcg 3y ago
> Nowadays, the repo-tarball split is largely unnecessary, but the practice remains. I keep seeing this argument, but I don't understand why CI runners can't generate the configure script rather than offloading that onto the end user requiring them to have autotools/automake/autoconf/m4/etc If you've ever tried to fight with that on an decade old enterprise system, those tarballs with configure scripts in them are useful. They get generated with new tooling on modern distros, but run on ancient systems. Getting the right versions installed can be an absolute pain in the ass.

tutfbhuf 3y ago

[flagged]

loloquwowndueo 3y ago
We don’t understand how AI works. I would not trust an AI to not hallucinate unsafely in this context.
zopa 3y ago
You wouldn’t want it in the CI pipeline, because any model clever enough to find real issues is also going to find plenty of false positives. That seems like too much friction for most open source projects. I’m not one of the downvoters, but you’ve linked to a list of forty or fifty different projects, many of which don’t seem relevant to this use-case. It’s not too surprising people have nothing to say besides “ugh, more AI hype.”
VMG 3y ago
You are wrong but there is no reason to downvote you. The idea might work in the future.
- Kwpolska 3y ago
  It might work if AI stops producing random output and bullshit (which proponents call hallucinations to make it sound nice), and produces correct responses deterministically. Which may take forever.

weinzierl 3y ago

"The xz incident became a wake-up call that the Reproducible Build movement focused on binary reproducibility, but have so far ignored tarball reproducibility." The reproducibility must stretch over all the outputs. A related idea is the Hermetic Build", where the sameness of all the input is ensured, even including the build tools.

yencabulator 3y ago
In this case, inputs, or more like the whole chain toward "real inputs". A project can be reproducible when built from the release tarball, without being able to reproduce the release tarball from a git tag.

mid-kid 3y ago

> Nowadays, the repo-tarball split is largely unnecessary I think locking the source distribution mechanism to only git would be detrimental in the grand scheme. A source tarball is universal, independent of preferred tooling, easy to hash, sign and verify, and archive. Even git has loopholes in that a tag does not necessarily have to be a part of the master branch, and not everyone uses (or wants to use) github, or a similar online interface for git... It's also worth considering that source generation tools are often keen to changing their API frequently, while their output is more weaponized against the passage of time. Autotools is significantly better about that these days, but many other tools aren't...

kevincox 3y ago
I think most packagers that can pull from Git can also pull from Mercurial, SVN or whatever else. Pulling from Git doesn't really require using GitHub other than connecting to their IP. You don't need an account or any tool other than Git to clone a public repo. It seems to me that pulling directly from the primary source of truth is a good idea these days where possible.
kruador 3y ago
The Git protocol is also substantially more heavyweight than just downloading a tarball. 'Dumb' HTTP involves requesting `info/refs` which lists all the references (branches and tags) available. The client has to find the tag you want, then request the corresponding commit object. That lists the hash of the tree object, which has to be requested, and recursively the tree object references other trees and finally the file blobs. Any object request could fail because the object isn't stored 'loose' on disk, but is instead in a pack. So the client first downloads `objects/info/http-alternates` to see if it's in a different location, then if that doesn't list anything, it asks for the list of packs `objects/info/packs`. Then it asks for the index of each pack to determine which pack contains the object, then finally downloads that pack. Because that's very slow, Git also supports 'Smart' HTTP. That moves all the lookups server-side, with a back-and-forth between client and server about what the client has and what it wants, ultimately building a custom pack containing what you asked for. This obviously can't be cached. In contrast a tarball is one request for one file on disk which is fully cacheable, both by the server hosting the file and any proxies on the path. This is why things like `bower` and `DefinitelyTyped` got deprecated. Some of them were even hosting the registry itself as a Git repository on GitHub.

lamontcg 3y ago

> Nowadays, the repo-tarball split is largely unnecessary, but the practice remains. I keep seeing this argument, but I don't understand why CI runners can't generate the configure script rather than offloading that onto the end user requiring them to have autotools/automake/autoconf/m4/etc If you've ever tried to fight with that on an decade old enterprise system, those tarballs with configure scripts in them are useful. They get generated with new tooling on modern distros, but run on ancient systems. Getting the right versions installed can be an absolute pain in the ass.