11 ms·

What makes maintainers of major distros still rely on questionable tarballs to build packages? It's not as if these essential programs don't have a public, auth

by kijin 3y ago

What makes maintainers of major distros still rely on questionable tarballs to build packages? It's not as if these essential programs don't have a public, authoritative git repository. Is it because of inertia, because we've been using tarballs since before VCS was a thing? Is it to reduce the burden of package maintainership, by letting upstream do most of the transpiling and autotools plumbing work? Is it because some assets and test data are only included in the tarballs? Why are they not committed, either to the same repo or (if upstream wishes to keep the main repo small) some other repo? People have been calling for reproducible builds for years now, but reproducible builds don't mean anything if they are not built from authoritative sources with immutable commit IDs.

Kluggy 3y ago
It's absolutely because of inertia. Cutting a tarball is a very public indicator that the software is ready to be used. A git tag or branch doesn't have the clarity of "use this" as a release tarball does. That said, this is absolutely going to be changing now. We obviously can't keep relying on tarballs anymore. We'll find a new normal that will work for a very long time until some other critical issue arises and the cycle repeats.
- kzrdude 3y ago
  We can use tarballs - they are useful as signed artifacts, but only if we verify that they are reproducible.
  - lupusreal 3y ago
    If you're going to download both the tarball and the got repo and verify the reproducibility of the tarball, then why bother with the tarball at all? You already have the got repo.
    - b112 3y ago
      A git repo can vanish overnight. Git is often used to snag source, but tarballs are still crafted after even then: https://news.ycombinator.com/item?id=39903813 https://news.ycombinator.com/item?id=39903813
      - kijin 3y ago
        Git repos only vanish if they have no clones. For the purpose of accountability, the "official" repo is not more privileged than any other clone that retains the same history, and can be verifiably recreated from any clone if it is ever lost or tampered with. (Assuming SHA-1 isn't too broken, that is.) For archival purposes, nothing prevents people from creating a tarball that contains the .git directory as well, which would preserve not only the current state of the project but its entire history.
        b112 3y ago
        Git repos only vanish if they have no clones. And? You're speaking probabilities, not certainties. It's not relevant in terms of archiving. You don't guess, you don't hope, you simply 100% ensure that 10 days, 19 years, or a century from now you can build the same thing. I agree that adding "extra stuff" to a tarball isn't a bad idea, and in fact, many already do!
        lupusreal 3y ago
        If you can archive the tarball, you can archive the git repos. If for some reason you can't, you can cut your own tarballs from the git repo and then you don't have to worry about them because you made them yourself.
        b112 3y ago
        Archiving an entire git repo is serious overkill, and untenable realistically. Debian has 30k packages, if not more, and some of them are the linux kernel. My responses in these threads has been to the "why not build from git" logic championed here. You want to build from a reliable, repeatable source. And as I mentioned, that can be git clone -> tarball, sure.
badsectoracula 3y ago
Why are tarballs any less authoritative than a Git repository? Both come from the same authors. If you can't trust the tarball released by some author why would you trust what the repository contains?
- senectus1 3y ago
  as i understand it, the tooling would "see" some of the questionable configs that were in the tarballs. Because they're packaged into tarballs they cant be checked. or or to the point they use the tarballs because its much faster...
  - badsectoracula 3y ago
    What tooling? Git is just a filesystem with file versions, anything you can check in a tarball can also be checked in Git. The only thing Git adds is a change history for the contents in the filesystem (a history that can be forged too - Git even has a bunch of commands for that too). If you cannot trust the authors' tarballs how can you trust their Git repository?
- Terr_ 3y ago
  > Why are tarballs any less authoritative than a Git repository? Both come from the same authors. Traceability. There's a difference in how easily someone can insert nefarious code without it going through processes that might reveal or stop it. Compare: "Why is a paper sack of pills any different than a factory-sealed anti-tamper bottle? You're buying them from the same grocery store..."
  - badsectoracula 3y ago
    > Traceability. There's a difference in how easily someone can insert nefarious code without it going through processes that might reveal or stop it. But both come from the same authors. If you do not trust the tarball because it may have nefarious how would you trust Git to not have nefarious code? The authors have complete control over what goes in Git and what processes might be there. > Compare... The grocery store comparison doesn't fit here because grocery stores do not make the things they sell. The equivalent to a grocery store would be the download site (BTW lets not continue with metaphors because IMO instead of making things more clear you are introducing TWO parallel channels to argue about).
pm215 3y ago
There's a lot of inertia here, but also the tarball is or was the one point of commonality where you could say "this is the standard thing every upstream provides". Today almost everything has a git repo, but it still isn't 100% universal, and in the past even less so -- some projects had git, some svn, some cvs, some mercurial, some didn't use source control at all(!), but everybody provided their releases as source tarballs. So when you're designing a distro packaging workflow 20 years ago, "treat the source tarball as the authoritative upstream output" made sense, because it was the authoritative upstream release output.
shp0ngle 3y ago
There are still nowadays projects without a proper source control. For example, xpdfreader, source of xpdf, is distributed just as source tarballs. The maintainer just publishes them every time there is a new version. https://www.xpdfreader.com/download.html https://www.xpdfreader.com/download.html https://www.xpdfreader.com/old-versions.html https://www.xpdfreader.com/old-versions.html https://packages.debian.org/search?keywords=xpdf https://packages.debian.org/search?keywords=xpdf There is nothing in Debian or FOSS in general that mandates having source control. (I remember that was a point of contention with WebKit/KHTML... in the olden days of 2000-something, Apple forked KHTML to make WebKit, and in order to comply with GPL, they just published source tarballs, which were basically impossible to merge back into KHTML.) edit: ahh I see that the xpdf that is in debian is actually different than this one; someone forked it some time ago and it is in in git here (but no github/other forge, just .git) https://offog.org/git/xpopple.git/ https://offog.org/git/xpopple.git/ well ok, maybe I was wrong. but still, a .git folder on someone webpage is not that much more reliable than a tarball on someone's webpage
- blueflow 3y ago
  No need to look that far, ncurses is also tarball/patch-based.
  - shp0ngle 3y ago
    Interesting. Well, xpdf I knew :) but yeah xpdf can be nuked, ncurses cannot.
- hyperpape 3y ago
  Certainly there's nothing in FOSS that mandates source control, but source control is one of the most important technologies discovered for developing software, and I'd be hard pressed to imagine a justification for relying on software that didn't use it.
  - Kwpolska 3y ago
    It is likely that the `xpdf` project is developed under source control, but they don’t make the repository public. SQLite makes their repository public, but they do not accept contributions.
bregma 3y ago
The tarball, with its signed hash, is the authoritative source with immutable ID. Not every project uses git. Not every project uses a public VCS. Git history can be changed with a simple command and is unreliable. The point of a tarball is that it is the distribution medium of a source release. The developer generates the generated code (where required), runs all tests (eg. `make distcheck`), then when satisfied it's good, signs it and releases it. If you can't trust the developer's signed tarball you can't trust their git repo either, since it's the developer you can't trust. A downstream, such as a Linux distro, uses the release tarball because it's the single, verifiable, published source of truth. They should be able to do reproduceable builds from the tarball (not being able to do so should be filed upstream as a bug), and for licensing reasons (eg. GPL) need to keep the tarball and make it available to anyone to whom the software is distributed.
- gunapologist99 3y ago
  The parent referred to "questionable" (i.e., unsigned) tarballs. Your answer, while excellent, is answering a question that wasn't asked.
  - AshamedCaptain 3y ago
    No distribution I know of uses "unsigned" tarballs. Most of them hardcode the hash of the expected tarball anyway.
    - gunapologist99 3y ago
      Hashes and cryptographic signatures are not the same thing.
- kijin 3y ago
  > Git history can be changed with a simple command and is unreliable. Not if other people have cloned your repo. That's the whole point of having a distributed VCS. Your repo is no longer the single source of truth because other repos remember what truth looked like 3 days, 3 months, 3 years ago. You renounce the right to dictate truth as you wish, and gain trust in exchange. I don't understand why this idea comes up again and again: "You trust the developer, you might as well trust the code they distribute." That makes no sense. Why do you trust them in the first place? Do you know them? Have you met them in person, had a few beers with them? Or is your trust merely based on their past performance as an open-source developer? If it's the latter, whether they have been willing to work in a way that facilitates independent inspection and verification of their output should be a very important factor in whether you should trust them at all. Hardware vendors who dump opaque binary blobs at the kernel's doorstep tend not to receive much respect, after all.
arp242 3y ago
Because it's often easier, and because they're not "questionable tarballs". People seem to have forgotten that all of this has been working well without all that many incidents for 35 years. And these archives are published by the same people you trust to run code on your system. It's not like you can't hide this kind of stuff in the autotools soup committed to git. I doubt this would have been caught it – the "sneak it in the .tar.gz only" was just a bonus defence against detection. I'm not saying using git isn't better, but we also need to retain some perspective on all of this. This ibus thing for example seems entirely innocent. The core of the issue is "who do we trust to write code for us?" Maintainers protect us from randos submitting malicious code, but who maintains the maintainers?

by kijin 3y ago

Kluggy 3y ago

It's absolutely because of inertia. Cutting a tarball is a very public indicator that the software is ready to be used. A git tag or branch doesn't have the clarity of "use this" as a release tarball does. That said, this is absolutely going to be changing now. We obviously can't keep relying on tarballs anymore. We'll find a new normal that will work for a very long time until some other critical issue arises and the cycle repeats.

kzrdude 3y ago
We can use tarballs - they are useful as signed artifacts, but only if we verify that they are reproducible.
- lupusreal 3y ago
  If you're going to download both the tarball and the got repo and verify the reproducibility of the tarball, then why bother with the tarball at all? You already have the got repo.
  - b112 3y ago
    A git repo can vanish overnight. Git is often used to snag source, but tarballs are still crafted after even then: https://news.ycombinator.com/item?id=39903813 https://news.ycombinator.com/item?id=39903813
    - kijin 3y ago
      Git repos only vanish if they have no clones. For the purpose of accountability, the "official" repo is not more privileged than any other clone that retains the same history, and can be verifiably recreated from any clone if it is ever lost or tampered with. (Assuming SHA-1 isn't too broken, that is.) For archival purposes, nothing prevents people from creating a tarball that contains the .git directory as well, which would preserve not only the current state of the project but its entire history.
      - b112 3y ago
        Git repos only vanish if they have no clones. And? You're speaking probabilities, not certainties. It's not relevant in terms of archiving. You don't guess, you don't hope, you simply 100% ensure that 10 days, 19 years, or a century from now you can build the same thing. I agree that adding "extra stuff" to a tarball isn't a bad idea, and in fact, many already do!
        lupusreal 3y ago
        If you can archive the tarball, you can archive the git repos. If for some reason you can't, you can cut your own tarballs from the git repo and then you don't have to worry about them because you made them yourself.
        b112 3y ago
        Archiving an entire git repo is serious overkill, and untenable realistically. Debian has 30k packages, if not more, and some of them are the linux kernel. My responses in these threads has been to the "why not build from git" logic championed here. You want to build from a reliable, repeatable source. And as I mentioned, that can be git clone -> tarball, sure.

badsectoracula 3y ago

Why are tarballs any less authoritative than a Git repository? Both come from the same authors. If you can't trust the tarball released by some author why would you trust what the repository contains?

senectus1 3y ago
as i understand it, the tooling would "see" some of the questionable configs that were in the tarballs. Because they're packaged into tarballs they cant be checked. or or to the point they use the tarballs because its much faster...
- badsectoracula 3y ago
  What tooling? Git is just a filesystem with file versions, anything you can check in a tarball can also be checked in Git. The only thing Git adds is a change history for the contents in the filesystem (a history that can be forged too - Git even has a bunch of commands for that too). If you cannot trust the authors' tarballs how can you trust their Git repository?
Terr_ 3y ago
> Why are tarballs any less authoritative than a Git repository? Both come from the same authors. Traceability. There's a difference in how easily someone can insert nefarious code without it going through processes that might reveal or stop it. Compare: "Why is a paper sack of pills any different than a factory-sealed anti-tamper bottle? You're buying them from the same grocery store..."
- badsectoracula 3y ago
  > Traceability. There's a difference in how easily someone can insert nefarious code without it going through processes that might reveal or stop it. But both come from the same authors. If you do not trust the tarball because it may have nefarious how would you trust Git to not have nefarious code? The authors have complete control over what goes in Git and what processes might be there. > Compare... The grocery store comparison doesn't fit here because grocery stores do not make the things they sell. The equivalent to a grocery store would be the download site (BTW lets not continue with metaphors because IMO instead of making things more clear you are introducing TWO parallel channels to argue about).

pm215 3y ago

There's a lot of inertia here, but also the tarball is or was the one point of commonality where you could say "this is the standard thing every upstream provides". Today almost everything has a git repo, but it still isn't 100% universal, and in the past even less so -- some projects had git, some svn, some cvs, some mercurial, some didn't use source control at all(!), but everybody provided their releases as source tarballs. So when you're designing a distro packaging workflow 20 years ago, "treat the source tarball as the authoritative upstream output" made sense, because it was the authoritative upstream release output.

shp0ngle 3y ago

There are still nowadays projects without a proper source control. For example, xpdfreader, source of xpdf, is distributed just as source tarballs. The maintainer just publishes them every time there is a new version. https://www.xpdfreader.com/download.html https://www.xpdfreader.com/download.html https://www.xpdfreader.com/old-versions.html https://www.xpdfreader.com/old-versions.html https://packages.debian.org/search?keywords=xpdf https://packages.debian.org/search?keywords=xpdf There is nothing in Debian or FOSS in general that mandates having source control. (I remember that was a point of contention with WebKit/KHTML... in the olden days of 2000-something, Apple forked KHTML to make WebKit, and in order to comply with GPL, they just published source tarballs, which were basically impossible to merge back into KHTML.) edit: ahh I see that the xpdf that is in debian is actually different than this one; someone forked it some time ago and it is in in git here (but no github/other forge, just .git) https://offog.org/git/xpopple.git/ https://offog.org/git/xpopple.git/ well ok, maybe I was wrong. but still, a .git folder on someone webpage is not that much more reliable than a tarball on someone's webpage

blueflow 3y ago
No need to look that far, ncurses is also tarball/patch-based.
- shp0ngle 3y ago
  Interesting. Well, xpdf I knew :) but yeah xpdf can be nuked, ncurses cannot.
hyperpape 3y ago
Certainly there's nothing in FOSS that mandates source control, but source control is one of the most important technologies discovered for developing software, and I'd be hard pressed to imagine a justification for relying on software that didn't use it.
- Kwpolska 3y ago
  It is likely that the `xpdf` project is developed under source control, but they don’t make the repository public. SQLite makes their repository public, but they do not accept contributions.

bregma 3y ago

The tarball, with its signed hash, is the authoritative source with immutable ID. Not every project uses git. Not every project uses a public VCS. Git history can be changed with a simple command and is unreliable. The point of a tarball is that it is the distribution medium of a source release. The developer generates the generated code (where required), runs all tests (eg. `make distcheck`), then when satisfied it's good, signs it and releases it. If you can't trust the developer's signed tarball you can't trust their git repo either, since it's the developer you can't trust. A downstream, such as a Linux distro, uses the release tarball because it's the single, verifiable, published source of truth. They should be able to do reproduceable builds from the tarball (not being able to do so should be filed upstream as a bug), and for licensing reasons (eg. GPL) need to keep the tarball and make it available to anyone to whom the software is distributed.

gunapologist99 3y ago
The parent referred to "questionable" (i.e., unsigned) tarballs. Your answer, while excellent, is answering a question that wasn't asked.
- AshamedCaptain 3y ago
  No distribution I know of uses "unsigned" tarballs. Most of them hardcode the hash of the expected tarball anyway.
  - gunapologist99 3y ago
    Hashes and cryptographic signatures are not the same thing.
kijin 3y ago
> Git history can be changed with a simple command and is unreliable. Not if other people have cloned your repo. That's the whole point of having a distributed VCS. Your repo is no longer the single source of truth because other repos remember what truth looked like 3 days, 3 months, 3 years ago. You renounce the right to dictate truth as you wish, and gain trust in exchange. I don't understand why this idea comes up again and again: "You trust the developer, you might as well trust the code they distribute." That makes no sense. Why do you trust them in the first place? Do you know them? Have you met them in person, had a few beers with them? Or is your trust merely based on their past performance as an open-source developer? If it's the latter, whether they have been willing to work in a way that facilitates independent inspection and verification of their output should be a very important factor in whether you should trust them at all. Hardware vendors who dump opaque binary blobs at the kernel's doorstep tend not to receive much respect, after all.

arp242 3y ago

Because it's often easier, and because they're not "questionable tarballs". People seem to have forgotten that all of this has been working well without all that many incidents for 35 years. And these archives are published by the same people you trust to run code on your system. It's not like you can't hide this kind of stuff in the autotools soup committed to git. I doubt this would have been caught it – the "sneak it in the .tar.gz only" was just a bonus defence against detection. I'm not saying using git isn't better, but we also need to retain some perspective on all of this. This ibus thing for example seems entirely innocent. The core of the issue is "who do we trust to write code for us?" Maintainers protect us from randos submitting malicious code, but who maintains the maintainers?