5 ms·

The graph insinuates that before etch (2007) close to 100% of debian's upstream packages didn't use any source control at all. Is that right? It seems like it

The graph insinuates that before etch (2007) close to 100% of debian's upstream packages didn't use any source control at all. Is that right? It seems like it couldn't possibly be true.

carreau 5y ago
IIRC before 2007 debian policy was that packages had to be installed from tarball even if a repo existed. Thus it is likely that the info about the repository was simply not tracked. It might still be the case for many packages. It stemmed from the fact that often releases tars might not be exactly the same as asking git to do a tarfile. For example releasing Python packages may require a build step and installing from PyPI or a git repo is sometime not the same.
- geofft 5y ago
  That's still the norm! You're supposed to use the upstream tarball if one exists, and all of the tooling expects a tarball of some sort. In fact there's an active discussion right now on debian-devel about maybe changing the norms to prefer VCS archives to upstream release tarballs (and rerunning any of the code generation steps like autoconf as part of the Debian build): https://lists.debian.org/debian-devel/2021/08/msg00133.html https://lists.debian.org/debian-devel/2021/08/msg00133.html ... Also, the tooling not only expects a tarball, it expects every tarball with the same name to be bit-identical. So if foo 1.0 is already in Debian, you'd better not generate a slightly different foo-1.0.orig.tar.gz when you do a packaging update but you're still packaging version 1.0 of the upstream sources. In 2007 a tool called "pristine-tar" showed up to allow you to compute just enough information to regenerate the actual foo-1.0.orig.tar.gz from the foo 1.0 VCS tag and commit that delta onto a separate VCS branch. It's a pretty crucial but extremely finicky part of most workflows involving maintaining Debian packages via a VCS. And then it turns out that the author of pristine-tar actually was somewhat trying to "point out absurdities in the way things are done" (https://joeyh.name/blog/entry/twenty_years_of_free_software_--_part_5_pristine-tar/ https://joeyh.name/blog/entry/twenty_years_of_free_software_...) and people just happily adopted the tool and made things more absurd.
  - carreau 5y ago
    I'm not using debian that much anymore, but I'd be happy to make any changes upstream to make it easier on debian. When possible I try to have reproducible builds (https://reproducible-builds.org/ https://reproducible-builds.org/) that respect SOURCE_DATE_EPOCH extracted from the commit datetime. At least for IPython I alway build twice to make sure the sha is identical. I'd also be happy to have a way to signal to debian that there is a new release instead of having QA watch have to scan things regularly.
    - southerntofu 5y ago
      > I'd also be happy to have a way to signal to debian that there is a new release instead of having QA watch have to scan things regularly. That would be great! Unfortunately the current forging model is very centralized, centered around the UX of private companies with their internal teams and little (if any) outside cooperation. That's why we need decentralized forging systems: https://staticadventures.netlib.re/blog/decentralized-forge/ https://staticadventures.netlib.re/blog/decentralized-forge/ There's an ongoing 5000€ bounty for implementing ForgeFed federation (based on ActivityPub) on top of Gitea, in case you know people familiar with golang/ActivityPub who'd like to help. Also worth noting, i feel like there is room for experimenting with anti-authoritarian forging where you don't need upstream permission to receive webhooks (or other update notification). I've started experimenting with this but still pre-alpha quality: https://thunix.net/~southerntofu/forge/ https://thunix.net/~southerntofu/forge/ > All of these CI/CD plateforms consider the repository itself should contain the tasks to be run, for example in a .gitlab-ci.yml file. This top-down deployment model is well suited to an organization controling the whole of its software supply chain, but is a severe restriction to 3rd party involvement, which mostly hinders volunteer-run projects. > The forge suite adopts an opposite approach, where anyone can receive updates from remote repositories, and run the tasks they wish. This allows anyone within or without your projects to setup new test suites, benchmarks, and integrations. The applications are endless and should benefit your projects in many ways.
    - grlass 5y ago
      For those interested in reproducible builds, the gitian [1] project is a fairly simple VM which sets the up the necessary environment for doing this sort of thing. The tooling and community around reproducible builds is growing all the time, and imo we should be insisting on it for things such as government apps. [1] https://github.com/devrandom/gitian-builder https://github.com/devrandom/gitian-builder
duskwuff 5y ago
It's plausible. A lot of Debian packages come from small projects, often ones with a single developer. Before 2007, there weren't a lot of options for public source control -- GitHub didn't exist yet, SourceForge's CVS/SVN hosting was a pain to work with, and hosting it yourself was even more of a pain. Many of these small projects were probably only available as release tarballs, so that's what Debian built from.
- blihp 5y ago
  I guess it depended on which part of the open source crowd you were/are involved with. Pretty much every project I cared about back then, as far back as the mid- to late-90's, had propped up a self-hosted vcs server. I used to run a personal svn server which wasn't that big a deal to set up and nearly zero effort to maintain. Where svn fell apart was the distributed part of the equation: it worked fine for the core developers who had commit access, not so great for anyone else who wanted to contribute back. Even today there are number of projects that are still resistant to git (i.e. git, not github) and clinging to svn.
  - oblio 5y ago
    If even FreeBSD switched to Git I don't think there's any reason for other projects to not do the same, other than inertia/lack of resources. And I'm saying this as someone who's not super fond of Git (especially the UI).
    - zinekeller 5y ago
      > If FreeBSD switched to Git If? It already happened: https://forums.freebsd.org/threads/heads-up-freebsd-changing-from-subversion-to-git-this-weekend.78087/ https://forums.freebsd.org/threads/heads-up-freebsd-changing...
      - oblio 5y ago
        Rephrased.
  - 5y ago
deleted 5y ago
[deleted]
colechristensen 5y ago
Upstream projects likely used source control, but the packages used published artifacts (tarballs) which were released by the upstream project. It's a statement that gives false impressions, the reality isn't very interesting.
geofft 5y ago
Debian packaging is its own form of source control: there's a debian/changelog file with versions, and every time you submit something new to the Debian archive, it has to be a new version with a new changelog entry. Debian also has a strong individual ownership culture of packages, and it was even stronger then. Each package has a listed maintainer (nowadays it's often a team, but there isn't an "everyone" team short of the Debian QA Group, which is who orphaned packages get reassigned to). While every Debian developer has technical access to upload every package, it's strongly socially frowned upon to upload someone else's package. So under that model, you don't really need the collaboration or concurrent-editing features of source control systems. And yes, I remember that in about 2007 very few Debian packages had their packaging in any sort of standard version control tool (CVS, SVN, etc.) You'd get the sources via apt-get source, and then if you wanted to contribute, you could send in a patch by email to the bug tracker and the maintainer would incorporate it. Of course there are a whole lot of other quality-of-life features that real version control systems get you, which is why people ended up adopting them. But it's not like there was a shared directory on a server somewhere that every Debian developer just edited in place or whatever.
- da39a3ee 5y ago
  As a developer of an application, I have no special relationship with Debian. It’s nice they exist, and I do recognize the important role they’ve played in Linux, but still as a maintainer & author I don’t care about them and their practices and requirements any more than Homebrew, or a bunch of other Linux distros/packaging projects. Is what you wrote consistent with what I just wrote?
  - kzrdude 5y ago
    Debian is almost like a trusted retail shop. They mediate between you (the producer of the software) and me (the consumer of the software). The quality control they are responsible for, does make them a trusted "shop" for the users - for example, one believes that using debian packages protects from malware. It's natural the user cares more about this system than the producer - for the producer it looks like hassle and a middleman. (Sidenote: Sometimes I'm the producer side of debian packages too.)
jwilk 5y ago
I think what this graph shows is, contrary to what the text alludes, the VCS use for Debian packaging. There was no standardized way to declare that until late 2006, and it was offically documented only in 2008: https://bugs.debian.org/391023 https://bugs.debian.org/391023

The graph insinuates that before etch (2007) close to 100% of debian's upstream packages didn't use any source control at all. Is that right? It seems like it couldn't possibly be true.

carreau 5y ago

IIRC before 2007 debian policy was that packages had to be installed from tarball even if a repo existed. Thus it is likely that the info about the repository was simply not tracked. It might still be the case for many packages. It stemmed from the fact that often releases tars might not be exactly the same as asking git to do a tarfile. For example releasing Python packages may require a build step and installing from PyPI or a git repo is sometime not the same.

geofft 5y ago
That's still the norm! You're supposed to use the upstream tarball if one exists, and all of the tooling expects a tarball of some sort. In fact there's an active discussion right now on debian-devel about maybe changing the norms to prefer VCS archives to upstream release tarballs (and rerunning any of the code generation steps like autoconf as part of the Debian build): https://lists.debian.org/debian-devel/2021/08/msg00133.html https://lists.debian.org/debian-devel/2021/08/msg00133.html ... Also, the tooling not only expects a tarball, it expects every tarball with the same name to be bit-identical. So if foo 1.0 is already in Debian, you'd better not generate a slightly different foo-1.0.orig.tar.gz when you do a packaging update but you're still packaging version 1.0 of the upstream sources. In 2007 a tool called "pristine-tar" showed up to allow you to compute just enough information to regenerate the actual foo-1.0.orig.tar.gz from the foo 1.0 VCS tag and commit that delta onto a separate VCS branch. It's a pretty crucial but extremely finicky part of most workflows involving maintaining Debian packages via a VCS. And then it turns out that the author of pristine-tar actually was somewhat trying to "point out absurdities in the way things are done" (https://joeyh.name/blog/entry/twenty_years_of_free_software_--_part_5_pristine-tar/ https://joeyh.name/blog/entry/twenty_years_of_free_software_...) and people just happily adopted the tool and made things more absurd.
- carreau 5y ago
  I'm not using debian that much anymore, but I'd be happy to make any changes upstream to make it easier on debian. When possible I try to have reproducible builds (https://reproducible-builds.org/ https://reproducible-builds.org/) that respect SOURCE_DATE_EPOCH extracted from the commit datetime. At least for IPython I alway build twice to make sure the sha is identical. I'd also be happy to have a way to signal to debian that there is a new release instead of having QA watch have to scan things regularly.
  - southerntofu 5y ago
    > I'd also be happy to have a way to signal to debian that there is a new release instead of having QA watch have to scan things regularly. That would be great! Unfortunately the current forging model is very centralized, centered around the UX of private companies with their internal teams and little (if any) outside cooperation. That's why we need decentralized forging systems: https://staticadventures.netlib.re/blog/decentralized-forge/ https://staticadventures.netlib.re/blog/decentralized-forge/ There's an ongoing 5000€ bounty for implementing ForgeFed federation (based on ActivityPub) on top of Gitea, in case you know people familiar with golang/ActivityPub who'd like to help. Also worth noting, i feel like there is room for experimenting with anti-authoritarian forging where you don't need upstream permission to receive webhooks (or other update notification). I've started experimenting with this but still pre-alpha quality: https://thunix.net/~southerntofu/forge/ https://thunix.net/~southerntofu/forge/ > All of these CI/CD plateforms consider the repository itself should contain the tasks to be run, for example in a .gitlab-ci.yml file. This top-down deployment model is well suited to an organization controling the whole of its software supply chain, but is a severe restriction to 3rd party involvement, which mostly hinders volunteer-run projects. > The forge suite adopts an opposite approach, where anyone can receive updates from remote repositories, and run the tasks they wish. This allows anyone within or without your projects to setup new test suites, benchmarks, and integrations. The applications are endless and should benefit your projects in many ways.
  - grlass 5y ago
    For those interested in reproducible builds, the gitian [1] project is a fairly simple VM which sets the up the necessary environment for doing this sort of thing. The tooling and community around reproducible builds is growing all the time, and imo we should be insisting on it for things such as government apps. [1] https://github.com/devrandom/gitian-builder https://github.com/devrandom/gitian-builder

duskwuff 5y ago

It's plausible. A lot of Debian packages come from small projects, often ones with a single developer. Before 2007, there weren't a lot of options for public source control -- GitHub didn't exist yet, SourceForge's CVS/SVN hosting was a pain to work with, and hosting it yourself was even more of a pain. Many of these small projects were probably only available as release tarballs, so that's what Debian built from.

blihp 5y ago
I guess it depended on which part of the open source crowd you were/are involved with. Pretty much every project I cared about back then, as far back as the mid- to late-90's, had propped up a self-hosted vcs server. I used to run a personal svn server which wasn't that big a deal to set up and nearly zero effort to maintain. Where svn fell apart was the distributed part of the equation: it worked fine for the core developers who had commit access, not so great for anyone else who wanted to contribute back. Even today there are number of projects that are still resistant to git (i.e. git, not github) and clinging to svn.
- oblio 5y ago
  If even FreeBSD switched to Git I don't think there's any reason for other projects to not do the same, other than inertia/lack of resources. And I'm saying this as someone who's not super fond of Git (especially the UI).
  - zinekeller 5y ago
    > If FreeBSD switched to Git If? It already happened: https://forums.freebsd.org/threads/heads-up-freebsd-changing-from-subversion-to-git-this-weekend.78087/ https://forums.freebsd.org/threads/heads-up-freebsd-changing...
    - oblio 5y ago
      Rephrased.
- 5y ago

deleted 5y ago

[deleted]

colechristensen 5y ago

Upstream projects likely used source control, but the packages used published artifacts (tarballs) which were released by the upstream project. It's a statement that gives false impressions, the reality isn't very interesting.

geofft 5y ago

Debian packaging is its own form of source control: there's a debian/changelog file with versions, and every time you submit something new to the Debian archive, it has to be a new version with a new changelog entry. Debian also has a strong individual ownership culture of packages, and it was even stronger then. Each package has a listed maintainer (nowadays it's often a team, but there isn't an "everyone" team short of the Debian QA Group, which is who orphaned packages get reassigned to). While every Debian developer has technical access to upload every package, it's strongly socially frowned upon to upload someone else's package. So under that model, you don't really need the collaboration or concurrent-editing features of source control systems. And yes, I remember that in about 2007 very few Debian packages had their packaging in any sort of standard version control tool (CVS, SVN, etc.) You'd get the sources via apt-get source, and then if you wanted to contribute, you could send in a patch by email to the bug tracker and the maintainer would incorporate it. Of course there are a whole lot of other quality-of-life features that real version control systems get you, which is why people ended up adopting them. But it's not like there was a shared directory on a server somewhere that every Debian developer just edited in place or whatever.

da39a3ee 5y ago
As a developer of an application, I have no special relationship with Debian. It’s nice they exist, and I do recognize the important role they’ve played in Linux, but still as a maintainer & author I don’t care about them and their practices and requirements any more than Homebrew, or a bunch of other Linux distros/packaging projects. Is what you wrote consistent with what I just wrote?
- kzrdude 5y ago
  Debian is almost like a trusted retail shop. They mediate between you (the producer of the software) and me (the consumer of the software). The quality control they are responsible for, does make them a trusted "shop" for the users - for example, one believes that using debian packages protects from malware. It's natural the user cares more about this system than the producer - for the producer it looks like hassle and a middleman. (Sidenote: Sometimes I'm the producer side of debian packages too.)

jwilk 5y ago

I think what this graph shows is, contrary to what the text alludes, the VCS use for Debian packaging. There was no standardized way to declare that until late 2006, and it was offically documented only in 2008: https://bugs.debian.org/391023 https://bugs.debian.org/391023