6 ms·

I didn't lift an eyebrow at the headline which is apparently an april fools joke. But what is it that makes it so absurd that it works as a joke, which I'm mis

I didn't lift an eyebrow at the headline which is apparently an april fools joke. But what is it that makes it so absurd that it works as a joke, which I'm missing? The size seems absurd (500GB) but why would the Debian source be 500GB? To me it's in no way obvious that "Debian" would include any third party packages repackaged by the distro maintainers. Are Debian people maintaining N source repositories for third party code today, which would be the code merged into this hypothetical monorepo? Why?

richardwhiuk 3y ago
Debian does include the third party packages. This is either a mirror of the git repository, or a tarball of the upstream code.
- eichin 3y ago
  it's simpler than that: it's just running `dpkg-source -x` on every `*.dsc` file it can find. So that's "all of the source used to build all of the debs" (using dists/stable/main/source/Sources.xz to get the list.) (I think then just a single commit of all of it; the fun bit would be doing this for each release and looking at the size of the diffs...)
  - richardwhiuk 3y ago
    I meant what upstream Debian includes.
tazjin 3y ago
This includes the sources of all the Debian packages. In general this is a good idea, and it is close to how nixpkgs works (which is, in some sense, a "distributed monorepo"), but git, the Debian tooling and so on wouldn't actually be able to support a good workflow here.
- Zambyte 3y ago
  > it is close to how nixpkgs works Not really...? Nix distributes package definitions that describe how to build and install the package. This is close to just about every other package manager, besides the fact that most don't have / use the description on how to build the package, and instead they just download a binary based on the package definition. The Nix equivalent to this joke would be storing all of the package code directly in the nixpkgs repo, instead of storing links to the source code. Debian would ba able to support the same flow as Nix just fine. You can just use source packages[0] [0] https://wiki.debian.org/Packaging/SourcePackage https://wiki.debian.org/Packaging/SourcePackage
  - mananaysiempre 3y ago
    I’d say it is close. A purer example would be gittup[1], which has its (very few) upstreams as submodules. But ultimately, Git submodules are just URLs and commit hashes stored in .gitmodules in the repository root, which doesn’t seem all that different from the URLs and hashes of contents passed to fetchgit et al. in Nixpkgs. The capability of being able to patch an arbitrary package locally and have every transitive dependency rebuilt automatically is also the same. On the other hand, Debian and Fedora maintain full collections of sources used in their releases, while, AFAIU, if you need an old version of something from Nixpkgs, it has fallen out of the official binary cache, and the upstream is gone, you’re on your own. (There have been some efforts to fix that[2], but the first attempt was very limited in scope, and there doesn’t seem to have been a second one.) [1] https://gittup.org/gittup/ https://gittup.org/gittup/ [2] https://www.tweag.io/blog/2020-06-18-software-heritage/ https://www.tweag.io/blog/2020-06-18-software-heritage/
    - tazjin 3y ago
      Things don't "fall out of" the binary cache. Garbage collecting it is an active research area.
benrutter 3y ago
I think its less maintenance and more for pinning versions of software to ensure stability. I guess that comes with maintaining a package manager like apt. Definitely with you on not realizing it was a joke at first. Don't google have a company wide mono repo, that I'd guess includes all search and chrome? Or is that just an urban legend?
- samus 3y ago
  The stuff in Google's monorepo is mostly their own code, not snapshots from upstream.
  - bluGill 3y ago
    Chrome has brought in third party code. Which makes Debian mad and they do a lot of work to strip that out to use the upstream libraries. (Or at least that is how I understand it - I don't follow Debian closely)
    - samus 3y ago
      Yeah, that's why I wrote "mostly" :)
  - tetromino_ 3y ago
    Google's monorepo includes snapshots of everything upstream that Google's proprietary code depends on - starting from the kernel and glibc and core language toolchains, and all the way up to various open-source libraries and modules in c/python/java/whatever.
    - riku_iki 3y ago
      But 3p sources likely are for specific stable version? Linus can't break Google by submitting bug into linux kernel?
      - tetromino_ 3y ago

richardwhiuk 3y ago

Debian does include the third party packages. This is either a mirror of the git repository, or a tarball of the upstream code.

eichin 3y ago
it's simpler than that: it's just running `dpkg-source -x` on every `*.dsc` file it can find. So that's "all of the source used to build all of the debs" (using dists/stable/main/source/Sources.xz to get the list.) (I think then just a single commit of all of it; the fun bit would be doing this for each release and looking at the size of the diffs...)
- richardwhiuk 3y ago
  I meant what upstream Debian includes.

tazjin 3y ago

This includes the sources of all the Debian packages. In general this is a good idea, and it is close to how nixpkgs works (which is, in some sense, a "distributed monorepo"), but git, the Debian tooling and so on wouldn't actually be able to support a good workflow here.

Zambyte 3y ago
> it is close to how nixpkgs works Not really...? Nix distributes package definitions that describe how to build and install the package. This is close to just about every other package manager, besides the fact that most don't have / use the description on how to build the package, and instead they just download a binary based on the package definition. The Nix equivalent to this joke would be storing all of the package code directly in the nixpkgs repo, instead of storing links to the source code. Debian would ba able to support the same flow as Nix just fine. You can just use source packages[0] [0] https://wiki.debian.org/Packaging/SourcePackage https://wiki.debian.org/Packaging/SourcePackage
- mananaysiempre 3y ago
  I’d say it is close. A purer example would be gittup[1], which has its (very few) upstreams as submodules. But ultimately, Git submodules are just URLs and commit hashes stored in .gitmodules in the repository root, which doesn’t seem all that different from the URLs and hashes of contents passed to fetchgit et al. in Nixpkgs. The capability of being able to patch an arbitrary package locally and have every transitive dependency rebuilt automatically is also the same. On the other hand, Debian and Fedora maintain full collections of sources used in their releases, while, AFAIU, if you need an old version of something from Nixpkgs, it has fallen out of the official binary cache, and the upstream is gone, you’re on your own. (There have been some efforts to fix that[2], but the first attempt was very limited in scope, and there doesn’t seem to have been a second one.) [1] https://gittup.org/gittup/ https://gittup.org/gittup/ [2] https://www.tweag.io/blog/2020-06-18-software-heritage/ https://www.tweag.io/blog/2020-06-18-software-heritage/
  - tazjin 3y ago
    Things don't "fall out of" the binary cache. Garbage collecting it is an active research area.

benrutter 3y ago

I think its less maintenance and more for pinning versions of software to ensure stability. I guess that comes with maintaining a package manager like apt. Definitely with you on not realizing it was a joke at first. Don't google have a company wide mono repo, that I'd guess includes all search and chrome? Or is that just an urban legend?

samus 3y ago
The stuff in Google's monorepo is mostly their own code, not snapshots from upstream.
- bluGill 3y ago
  Chrome has brought in third party code. Which makes Debian mad and they do a lot of work to strip that out to use the upstream libraries. (Or at least that is how I understand it - I don't follow Debian closely)
  - samus 3y ago
    Yeah, that's why I wrote "mostly" :)
- tetromino_ 3y ago
  Google's monorepo includes snapshots of everything upstream that Google's proprietary code depends on - starting from the kernel and glibc and core language toolchains, and all the way up to various open-source libraries and modules in c/python/java/whatever.
  - riku_iki 3y ago
    But 3p sources likely are for specific stable version? Linus can't break Google by submitting bug into linux kernel?
    - tetromino_ 3y ago