5 ms·

Does anyone know if they've made the Packages file (repository metadata file, listing the packages in the repository) build reproducibly yet? I tripped over th

Does anyone know if they've made the Packages file (repository metadata file, listing the packages in the repository) build reproducibly yet? I tripped over this a couple weeks ago and was both amused and annoyed, since it seemed that packages were being listed in the file in a random order. I'm asking here because it might already be fixed; we're using a slightly old version of the package/repository tools.

lamby 9y ago
Not yet. Could you file a bug report?
- cperciva 9y ago
  I'll ask my Debian expert to do that. I don't know which package is responsible here, and that seems to be the first step towards filing a bug report.
  - lamby 9y ago
    File against the "ftp.debian.org" pseudopackage. :)
    - cperciva 9y ago
      Really? The problem arose when we were building packages for pkg.tarsnap.com, so it's a tools problem rather than an infrastructure problem...
      - lamby 9y ago
        Oh, you are using dpkg-scanpackages?
        cperciva 9y ago
        Looks like that file is being created by apt-ftparchive.
        lamby 9y ago
        So many tools to do the same thing, huh? :) I'll look into this...
    - deleted 9y ago
      [deleted]
jwilk 9y ago
Why do you care about the order of packages in Packages? What does "build reproducibly" even mean in this context?
- viraptor 9y ago
  Two repositories with same content should have same metadata. For the same reasons we want reproducible binaries - to spot differences. (Also prevents useless re-download in some cases, I guess)
- cperciva 9y ago
  Why do you care about the order of packages in Packages? In my personal case, so that when I build a repository which has some new packages and some old packages, when I look at the resulting pull request in github I can see that the packages which haven't changed have indeed not changed. What does "build reproducibly" even mean in this context? Two repositories with the same packages have identical Packages files. Or for me, slightly more generally, when the Packages file changes, it changes as minimally as possible.
deleted 9y ago
[deleted]
lamby 9y ago
I've filed this as https://bugs.debian.org/869557 https://bugs.debian.org/869557 and will work on this soon.
- cperciva 9y ago
  Thanks! FWIW, I don't think there's any alternative to throwing a sort into here somewhere, since ftw doesn't have any reproducible-order flag. But whether you want to sort the list of packages and then generate the output for each package, or generate the output as you traverse the tree and then sort that output, I don't know.
  - lamby 9y ago
    Yeah it will just be more involved that way :)
  - lamby 9y ago
    … added a patch.
    - cperciva 9y ago
      Looks plausible to me, but I don't speak C++. You're sorting file names, right? Is this guaranteed to DTRT when packages are updated? (i.e., a version number changing can't result in two packages switching order?)
      - lamby 9y ago
        Sorting the filenames, yep. Well I hope; I don't speak C++ either...

lamby 9y ago

Not yet. Could you file a bug report?

cperciva 9y ago
I'll ask my Debian expert to do that. I don't know which package is responsible here, and that seems to be the first step towards filing a bug report.
- lamby 9y ago
  File against the "ftp.debian.org" pseudopackage. :)
  - cperciva 9y ago
    Really? The problem arose when we were building packages for pkg.tarsnap.com, so it's a tools problem rather than an infrastructure problem...
    - lamby 9y ago
      Oh, you are using dpkg-scanpackages?
      - cperciva 9y ago
        Looks like that file is being created by apt-ftparchive.
        lamby 9y ago
        So many tools to do the same thing, huh? :) I'll look into this...
  - deleted 9y ago
    [deleted]

jwilk 9y ago

Why do you care about the order of packages in Packages? What does "build reproducibly" even mean in this context?

viraptor 9y ago
Two repositories with same content should have same metadata. For the same reasons we want reproducible binaries - to spot differences. (Also prevents useless re-download in some cases, I guess)
cperciva 9y ago
Why do you care about the order of packages in Packages? In my personal case, so that when I build a repository which has some new packages and some old packages, when I look at the resulting pull request in github I can see that the packages which haven't changed have indeed not changed. What does "build reproducibly" even mean in this context? Two repositories with the same packages have identical Packages files. Or for me, slightly more generally, when the Packages file changes, it changes as minimally as possible.

deleted 9y ago

[deleted]

lamby 9y ago

I've filed this as https://bugs.debian.org/869557 https://bugs.debian.org/869557 and will work on this soon.

cperciva 9y ago
Thanks! FWIW, I don't think there's any alternative to throwing a sort into here somewhere, since ftw doesn't have any reproducible-order flag. But whether you want to sort the list of packages and then generate the output for each package, or generate the output as you traverse the tree and then sort that output, I don't know.
- lamby 9y ago
  Yeah it will just be more involved that way :)
- lamby 9y ago
  … added a patch.
  - cperciva 9y ago
    Looks plausible to me, but I don't speak C++. You're sorting file names, right? Is this guaranteed to DTRT when packages are updated? (i.e., a version number changing can't result in two packages switching order?)
    - lamby 9y ago
      Sorting the filenames, yep. Well I hope; I don't speak C++ either...