12 ms·

On building 30k Debian packages

139 points by zdw 5y ago

jwilk 5y ago
> dpkg-buildpackage -rfakeroot -rfakeroot has been the default since dpkg 1.14.7, released in 2007: https://lists.debian.org/E1IekzH-0006ul-Cy@ries.debian.org https://lists.debian.org/E1IekzH-0006ul-Cy@ries.debian.org
- moyix 5y ago
  Thanks! I think I started using debian about 7 years before that, so I guess I never caught on :)
- deleted 5y ago
  [deleted]
throw0101a 5y ago
I wonder if also doing this for FreeBSD's 46792 Ports would give any extra 'perspective': * https://en.wikipedia.org/wiki/FreeBSD_Ports https://en.wikipedia.org/wiki/FreeBSD_Ports * https://www.freebsd.org/ports/ https://www.freebsd.org/ports/ There's already a 'pre-canned' bulk build infrastructure: * https://docs.freebsd.org/en/books/handbook/ports/#ports-poudriere https://docs.freebsd.org/en/books/handbook/ports/#ports-poud... * https://wiki.freebsd.org/VladimirKrstulja/Guides/Poudriere https://wiki.freebsd.org/VladimirKrstulja/Guides/Poudriere
- moyix 5y ago
  Nice idea! The more data the better, from my perspective :) Presumably a lot of the code overlaps, but there's also probably plenty that's only in FreeBSD. I would love to extend this even further to include projects on GitHub, but once you step outside the distro build systems it gets impossible to actually compile them in any standardized way. I might try to do some kind of best-effort (try ./configure && make, cmake, etc.) though.
  - boris 5y ago
    Or you could try a package repository for one of the C/C++ package managers. For example, for build2[1] we currently have ~300 packages[2] which you can all build in a uniform way. [1] https://build2.org https://build2.org [2] https://cppget.org https://cppget.org
    - moyix 5y ago
      Neat, I will give those a shot!
  - throw0101a 5y ago
    > […] but once you step outside the distro build systems it gets impossible to actually compile them in any standardized way. FreeBSD Ports allows you to specify GH information to pull down code/tarballs in a standardized fashion: * https://wiki.freebsd.org/Ports/SimpleGithub https://wiki.freebsd.org/Ports/SimpleGithub * https://docs.freebsd.org/en/books/porters-handbook/makefiles/ https://docs.freebsd.org/en/books/porters-handbook/makefiles... Then you'd list build- and run-time dependencies.
    - moyix 5y ago
      Right, and I definitely want to try building existing ports. But unless I'm missing something, this still requires you to write down by hand the dependencies and steps for actually building each project? That's not really feasible at the scale of GitHub.
      - throw0101a 5y ago
        True: there is no embedded metadata (à la Dublin Core) in the source code or in/on Github that tells you about how to compile the source code. There is still a human-involved step in looking at the source code, even if it is in a build infrastructure (like Ports or Homebrew) it still had to be put there by a human. The closest thing for a source hub and automated build system that is probably CPAN and the like (PIP, CRAN). Another build infrastructure, focused more in the HPC niche, is Spack: * https://spack.io https://spack.io * https://spack-tutorial.readthedocs.io/en/latest/index.html https://spack-tutorial.readthedocs.io/en/latest/index.html
frozenport 5y ago
Why not use nix-os? Which is at least guaranteed to have all packages building?
- brendyn 5y ago
  It very much doesn't work like that. I work on Guix a little, and I remember someone saying ~5% of packages were broken at any given time. there is something of an equilibrium between packages breaking due to changes and someone fixing them. Packages are also not truly functional in the functional programming sense. For practical reasons, Nix/Guix just try their best to remove as much state as possible to make things behave consistently. But for example, the number of cores available is different between different machines and race conditions can occur.
- jonringer117 5y ago
  Builds break over time[0]. Generally this is from big updates like gcc, glibc, llvm, openssl, etc. Most software which is maintained upstream and within nixpkgs is fine. But there's a bunch of "cruft" that you accumulate over time. [0]: https://hydra.nixos.org/jobset/nixpkgs/trunk https://hydra.nixos.org/jobset/nixpkgs/trunk P.S. The link above builds for x86_64-linux, aarch64-linux, aarch64-darwin, x86_64-darwin, which also contributes to the failure rate.
- tempay 5y ago
  This was my first thought as well. Their tool for managing builds (Hydra [1]) was pretty simple to set up last time I tried. In practice a few percent of packages will fail to build (packaging is hard!) but it’d be good enough for getting large quantities of compiled code. [1] https://nixos.wiki/wiki/Hydra https://nixos.wiki/wiki/Hydra
yjftsjthsd-h 5y ago
This sounds like 1. fantastic fun, and 2. an effective way to find a number of bugs; remarks like "Unfortunately, not every package respects the build options" and "things that expect dbus to be present" (to build!?) sound like there are clear improvements begging to be worked out. (Not that I'm implying that the author, or anyone else, is obligated to do so; chasing bugs and deficiencies in packaging is easily a full-time job if allowed to be!)
breadchris 5y ago
what a g, maybe you can teach a class on operating systems now
edwcross 5y ago
It would be interesting to know if at least part of the work could somehow be integrated into Software Heritage (https://www.softwareheritage.org/ https://www.softwareheritage.org/) or CodeMeta (https://codemeta.github.io https://codemeta.github.io).
harvie 5y ago
What is the point of compile_commands.json database? I've tried to run bear on my makefile and then i've read the resulting json. It seemed to me like bash script with extra steps. Is there some added value?
- touisteur 5y ago
  I think the idea is to avoid doing the whole checking dependencies, and fast-build your whole system. Suppose you build a new compiler plugin/pass, you just want to force-recompile most stuff without going through all the build and autostuff. It helps building dependency graphs, so that you can run static analysis tools. Coverity IIRC does something like that to get the list of units, and build (preprocessor directives, etc) options.
  - moyix 5y ago
    Yup, pretty much. If you want a static analysis tool to be able to actually parse your C code and see it like the compiler would, it needs to have all the info that the compiler did – header files, defines (including ones passed in with -D options), what the working directory was, what environment variables were in effect, etc. Otherwise there are all sorts of problems like: - Is this code even used? (Maybe it's only included for some specific configurations, or #ifdef'd out) - What does MAGIC_MACRO(x,y) do? Maybe this is only in a header file that's generated at build time, and you can't understand/analyze code that uses it without knowing its definition. - What compiler version was used? Code may be interpreted differently depending on the compiler (particularly if gcc or clang-specific extensions are used, but also just due to things like compiler bugs or idiosyncrasies) Coverity wrote a nice article about some of these issues back in 2010: https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-li... > Law: You can't check code you don't see. It seems too trite to note that checking code requires first finding it... until you try to do so consistently on many large code bases. Probably the most reliable way to check a system is to grab its code during the build process; the build system knows exactly which files are included in the system and how to compile them. This seems like a simple task. Unfortunately, it's often difficult to understand what an ad hoc, homegrown build system is doing well enough to extract this information [...] > The right approach, which we have used for the past seven years, kicks off the build process and intercepts every system call it invokes. As a result, we can see everything needed for checking, including the exact executables invoked, their command lines, the directory they run in, and the version of the compiler (needed for compiler-bug workarounds). This control makes it easy to grab and precisely check all source code, to the extent of automatically changing the language dialect on a per-file basis. [...] > Law: You can't check code you can't parse. Checking code deeply requires understanding the code's semantics. The most basic requirement is that you parse it. Parsing is considered a solved problem. Unfortunately, this view is naïve, rooted in the widely believed myth that programming languages exist. > The C language does not exist; neither does Java, C++, and C#. While a language may exist as an abstract idea, and even have a pile of paper (a standard) purporting to define it, a standard is not a compiler. What language do people write code in? The character strings accepted by their compiler. Further, they equate compilation with certification. A file their compiler does not reject has been certified as "C code" no matter how blatantly illegal its contents may be to a language scholar. Fed this illegal not-C code, a tool's C front-end will reject it. This problem is the tool's problem.
iam-TJ 5y ago
Have you considered talking to the Debian infrastructure and build teams about the possibility of augmenting the Debian package builders to generate, collect, and publish this data as a standard part of the build process ? It would provide an ongoing and always up-to-date data set rather than relying on external entities interest, good-will and resources to do it (such as yourself) and could provide significant insights into the entire Debian project.
- moyix 5y ago
  I will be chatting with some folks who work on reproducible builds (turns out some of them are my colleagues here at NYU! :)) to see if I can use their infrastructure to extract some of this information. That would no doubt be a big boon for the reliability of the build process too – they've been at this for years whereas my own infrastructure was put together over the past couple weeks.
0xbadcafebee 5y ago
> Use an SSD: If you can get your hands on a buttload of memory, try using a tmpfs mount instead. For i/o heavy builds it's like turning on ludicrous speed. > trying to automate everything in a language where failures are silent set -u -e -o pipefail > and can do exciting things like call "rm -rf /" when you meant "rm -rf ${foo}/${bar}" I mean, if you're calling any recursive-file-unlinking thing from Python you face the same problem. Same solution applies: normalize the path (readlink, realpath) and test if it's "/" before adding an argument to the command. (Also don't use -f unless you need to, it ignores errors) set -e -u -o pipefail _rm_r () { declare -a rmargs for arg in "$@" ; do fullpath="$(readlink -f "$arg")" if [ ! "$fullpath" = "/" ] ; then rmargs+=("$fullpath") fi done rm -r "${rmargs[@]}" } _rm_r "${foo}/${bar}" But I agree with OP that you should use any language you're most familiar with to produce the best results possible in the least amount of time.
- gravypod 5y ago
  In addition: you can use overlayfs + tmpfs to run the same build with multiple build args without using any extra memory. I had an experiment I was working on to run tests for mutation testing that I want to open source at some point that used this same principle.
- ezekiel68 5y ago
  Yeah, the rancor for cheap laughs schtick fell flat for me >Avoid shell hackery. This is probably controversial You bet. It's less than 5 minutes to learn how to make shell _not_ fail silently and an additional 10 minutes (I'm overstating the time) to figure out how to check variables for whether they will evaluate to empty strings -- to avoid deletion catastropes. Not saying anyone should be "forced" to learn this. It's simply more accurate and helpful to say, "I don't know and I don't want to learn" instead of blaming a language.
  - moyix 5y ago
    Sure, sorry if I was too harsh on bash here :) It really wasn't the right tool for the job in some parts of my use case though – parsing info out of Sources.gz using awk and sed and friends is just plain bad compared to using python-debian to get the info directly from the apt database.

139 points by zdw 5y ago

> dpkg-buildpackage -rfakeroot -rfakeroot has been the default since dpkg 1.14.7, released in 2007: https://lists.debian.org/E1IekzH-0006ul-Cy@ries.debian.org https://lists.debian.org/E1IekzH-0006ul-Cy@ries.debian.org

moyix 5y ago
Thanks! I think I started using debian about 7 years before that, so I guess I never caught on :)
deleted 5y ago
[deleted]

throw0101a 5y ago

I wonder if also doing this for FreeBSD's 46792 Ports would give any extra 'perspective': * https://en.wikipedia.org/wiki/FreeBSD_Ports https://en.wikipedia.org/wiki/FreeBSD_Ports * https://www.freebsd.org/ports/ https://www.freebsd.org/ports/ There's already a 'pre-canned' bulk build infrastructure: * https://docs.freebsd.org/en/books/handbook/ports/#ports-poudriere https://docs.freebsd.org/en/books/handbook/ports/#ports-poud... * https://wiki.freebsd.org/VladimirKrstulja/Guides/Poudriere https://wiki.freebsd.org/VladimirKrstulja/Guides/Poudriere

moyix 5y ago
Nice idea! The more data the better, from my perspective :) Presumably a lot of the code overlaps, but there's also probably plenty that's only in FreeBSD. I would love to extend this even further to include projects on GitHub, but once you step outside the distro build systems it gets impossible to actually compile them in any standardized way. I might try to do some kind of best-effort (try ./configure && make, cmake, etc.) though.
- boris 5y ago
  Or you could try a package repository for one of the C/C++ package managers. For example, for build2[1] we currently have ~300 packages[2] which you can all build in a uniform way. [1] https://build2.org https://build2.org [2] https://cppget.org https://cppget.org
  - moyix 5y ago
    Neat, I will give those a shot!
- throw0101a 5y ago
  > […] but once you step outside the distro build systems it gets impossible to actually compile them in any standardized way. FreeBSD Ports allows you to specify GH information to pull down code/tarballs in a standardized fashion: * https://wiki.freebsd.org/Ports/SimpleGithub https://wiki.freebsd.org/Ports/SimpleGithub * https://docs.freebsd.org/en/books/porters-handbook/makefiles/ https://docs.freebsd.org/en/books/porters-handbook/makefiles... Then you'd list build- and run-time dependencies.
  - moyix 5y ago
    Right, and I definitely want to try building existing ports. But unless I'm missing something, this still requires you to write down by hand the dependencies and steps for actually building each project? That's not really feasible at the scale of GitHub.
    - throw0101a 5y ago
      True: there is no embedded metadata (à la Dublin Core) in the source code or in/on Github that tells you about how to compile the source code. There is still a human-involved step in looking at the source code, even if it is in a build infrastructure (like Ports or Homebrew) it still had to be put there by a human. The closest thing for a source hub and automated build system that is probably CPAN and the like (PIP, CRAN). Another build infrastructure, focused more in the HPC niche, is Spack: * https://spack.io https://spack.io * https://spack-tutorial.readthedocs.io/en/latest/index.html https://spack-tutorial.readthedocs.io/en/latest/index.html

frozenport 5y ago

Why not use nix-os? Which is at least guaranteed to have all packages building?

brendyn 5y ago
It very much doesn't work like that. I work on Guix a little, and I remember someone saying ~5% of packages were broken at any given time. there is something of an equilibrium between packages breaking due to changes and someone fixing them. Packages are also not truly functional in the functional programming sense. For practical reasons, Nix/Guix just try their best to remove as much state as possible to make things behave consistently. But for example, the number of cores available is different between different machines and race conditions can occur.
jonringer117 5y ago
Builds break over time[0]. Generally this is from big updates like gcc, glibc, llvm, openssl, etc. Most software which is maintained upstream and within nixpkgs is fine. But there's a bunch of "cruft" that you accumulate over time. [0]: https://hydra.nixos.org/jobset/nixpkgs/trunk https://hydra.nixos.org/jobset/nixpkgs/trunk P.S. The link above builds for x86_64-linux, aarch64-linux, aarch64-darwin, x86_64-darwin, which also contributes to the failure rate.
tempay 5y ago
This was my first thought as well. Their tool for managing builds (Hydra [1]) was pretty simple to set up last time I tried. In practice a few percent of packages will fail to build (packaging is hard!) but it’d be good enough for getting large quantities of compiled code. [1] https://nixos.wiki/wiki/Hydra https://nixos.wiki/wiki/Hydra

yjftsjthsd-h 5y ago

This sounds like 1. fantastic fun, and 2. an effective way to find a number of bugs; remarks like "Unfortunately, not every package respects the build options" and "things that expect dbus to be present" (to build!?) sound like there are clear improvements begging to be worked out. (Not that I'm implying that the author, or anyone else, is obligated to do so; chasing bugs and deficiencies in packaging is easily a full-time job if allowed to be!)

breadchris 5y ago

what a g, maybe you can teach a class on operating systems now

edwcross 5y ago

It would be interesting to know if at least part of the work could somehow be integrated into Software Heritage (https://www.softwareheritage.org/ https://www.softwareheritage.org/) or CodeMeta (https://codemeta.github.io https://codemeta.github.io).

harvie 5y ago

What is the point of compile_commands.json database? I've tried to run bear on my makefile and then i've read the resulting json. It seemed to me like bash script with extra steps. Is there some added value?

touisteur 5y ago
I think the idea is to avoid doing the whole checking dependencies, and fast-build your whole system. Suppose you build a new compiler plugin/pass, you just want to force-recompile most stuff without going through all the build and autostuff. It helps building dependency graphs, so that you can run static analysis tools. Coverity IIRC does something like that to get the list of units, and build (preprocessor directives, etc) options.
- moyix 5y ago
  Yup, pretty much. If you want a static analysis tool to be able to actually parse your C code and see it like the compiler would, it needs to have all the info that the compiler did – header files, defines (including ones passed in with -D options), what the working directory was, what environment variables were in effect, etc. Otherwise there are all sorts of problems like: - Is this code even used? (Maybe it's only included for some specific configurations, or #ifdef'd out) - What does MAGIC_MACRO(x,y) do? Maybe this is only in a header file that's generated at build time, and you can't understand/analyze code that uses it without knowing its definition. - What compiler version was used? Code may be interpreted differently depending on the compiler (particularly if gcc or clang-specific extensions are used, but also just due to things like compiler bugs or idiosyncrasies) Coverity wrote a nice article about some of these issues back in 2010: https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-li... > Law: You can't check code you don't see. It seems too trite to note that checking code requires first finding it... until you try to do so consistently on many large code bases. Probably the most reliable way to check a system is to grab its code during the build process; the build system knows exactly which files are included in the system and how to compile them. This seems like a simple task. Unfortunately, it's often difficult to understand what an ad hoc, homegrown build system is doing well enough to extract this information [...] > The right approach, which we have used for the past seven years, kicks off the build process and intercepts every system call it invokes. As a result, we can see everything needed for checking, including the exact executables invoked, their command lines, the directory they run in, and the version of the compiler (needed for compiler-bug workarounds). This control makes it easy to grab and precisely check all source code, to the extent of automatically changing the language dialect on a per-file basis. [...] > Law: You can't check code you can't parse. Checking code deeply requires understanding the code's semantics. The most basic requirement is that you parse it. Parsing is considered a solved problem. Unfortunately, this view is naïve, rooted in the widely believed myth that programming languages exist. > The C language does not exist; neither does Java, C++, and C#. While a language may exist as an abstract idea, and even have a pile of paper (a standard) purporting to define it, a standard is not a compiler. What language do people write code in? The character strings accepted by their compiler. Further, they equate compilation with certification. A file their compiler does not reject has been certified as "C code" no matter how blatantly illegal its contents may be to a language scholar. Fed this illegal not-C code, a tool's C front-end will reject it. This problem is the tool's problem.

iam-TJ 5y ago

Have you considered talking to the Debian infrastructure and build teams about the possibility of augmenting the Debian package builders to generate, collect, and publish this data as a standard part of the build process ? It would provide an ongoing and always up-to-date data set rather than relying on external entities interest, good-will and resources to do it (such as yourself) and could provide significant insights into the entire Debian project.

moyix 5y ago
I will be chatting with some folks who work on reproducible builds (turns out some of them are my colleagues here at NYU! :)) to see if I can use their infrastructure to extract some of this information. That would no doubt be a big boon for the reliability of the build process too – they've been at this for years whereas my own infrastructure was put together over the past couple weeks.

0xbadcafebee 5y ago

> Use an SSD: If you can get your hands on a buttload of memory, try using a tmpfs mount instead. For i/o heavy builds it's like turning on ludicrous speed. > trying to automate everything in a language where failures are silent set -u -e -o pipefail > and can do exciting things like call "rm -rf /" when you meant "rm -rf ${foo}/${bar}" I mean, if you're calling any recursive-file-unlinking thing from Python you face the same problem. Same solution applies: normalize the path (readlink, realpath) and test if it's "/" before adding an argument to the command. (Also don't use -f unless you need to, it ignores errors) set -e -u -o pipefail _rm_r () { declare -a rmargs for arg in "$@" ; do fullpath="$(readlink -f "$arg")" if [ ! "$fullpath" = "/" ] ; then rmargs+=("$fullpath") fi done rm -r "${rmargs[@]}" } _rm_r "${foo}/${bar}" But I agree with OP that you should use any language you're most familiar with to produce the best results possible in the least amount of time.

gravypod 5y ago
In addition: you can use overlayfs + tmpfs to run the same build with multiple build args without using any extra memory. I had an experiment I was working on to run tests for mutation testing that I want to open source at some point that used this same principle.
ezekiel68 5y ago
Yeah, the rancor for cheap laughs schtick fell flat for me >Avoid shell hackery. This is probably controversial You bet. It's less than 5 minutes to learn how to make shell _not_ fail silently and an additional 10 minutes (I'm overstating the time) to figure out how to check variables for whether they will evaluate to empty strings -- to avoid deletion catastropes. Not saying anyone should be "forced" to learn this. It's simply more accurate and helpful to say, "I don't know and I don't want to learn" instead of blaming a language.
- moyix 5y ago
  Sure, sorry if I was too harsh on bash here :) It really wasn't the right tool for the job in some parts of my use case though – parsing info out of Sources.gz using awk and sed and friends is just plain bad compared to using python-debian to get the info directly from the apt database.