6 ms·

Fast Debian package builds using sbuild, mmdebstrap and apt-cacher-ng

60 points by JNRowe 4y ago

synergy20 4y ago
A few years ago I needed to add a new arch into Debian, it's a painful process, there are lots of arcane packages used to truly build and bootstrap the debian and they're all cross-dependent on each other. I wish there is a way like "linuxfromscratch" with clear steps to build Debian from source from scratch. Yes you can bootstrap a Debian root filesystem for a different arch, but that's after someone already did the work before you can do that, which again, is really complex and I have there are better ways to do it now(e.g. adding a new CPU family to Debian)
- IntelMiner 4y ago
  A lot of older Debian documentation has rotted away which is very sad for retro hobby stuff I wanted to rebuild Debian 2.2 against GCC 3.3 with its better support for SMP and SSE support for an old Pentium 3 workstation I've been playing with. Back then that kind of optimization could yield impressive performance benefits Unfortunately there's just no documentation for how to rebuild those ancient versions of Debian. The oldest I could find targeted the much newer Debian 4.0 "Etch" which came out in the mid 2000's
  - djbusby 4y ago
    Gentoo still supports i486 architecture and it's totally ready to tweak build flags.
    - IntelMiner 4y ago
      Gentoo also has a ton of historical rot, something users have been trying to solve for a while https://bugs.gentoo.org/834712 https://bugs.gentoo.org/834712
- thepostman0 4y ago
  Might not be exactly what you're after: https://github.com/scottwilliambeasley/debian-from-scratch https://github.com/scottwilliambeasley/debian-from-scratch
- pabs3 4y ago
  Myself and some other folks recently wrote some new docs on how to add new arches to Debian. With multi-arch, cross-compilers in Debian and rebootstrap, it has gotten a lot less painful. There are still a ridiculous number of packages that need porting to every new arches though. Some of the needed changes should be moved into toolchains in my opinion. https://wiki.debian.org/PortsDocs/New https://wiki.debian.org/PortsDocs/New
- pabs3 4y ago
  Which architecture were you working on adding?
shmerl 4y ago
When is Debian going to enable zstd for packages already? It takes forever.
- thepostman0 4y ago
  The bug looks closed recently: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664#49 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664#49
  - shmerl 4y ago
    Oh, that's good. Is it actually used now in packages in Debian repos though?
secondcoming 4y ago
It's easier to compile the linux kernel from scratch than it is to build a debian package from scratch. It's mind-blowing how complicated it is; at the end of the day it's just an archive. I hate having to do it.
- vaporup 4y ago
  Actually it is not that hard: - default: https://vaporup.github.io/books/the-pragmatic-sysadmin/debian-packaging/ - quick: https://nfpm.goreleaser.com/usage/
  - goodpoint 4y ago
    Read the official documentation instead of random blogs.
    - kasabali 4y ago
      Official documentation is a problem. It is aimed at Debian maintainers building a fully policy compliant package that could be accepted to the official repository. You don't need any of that crap for making a .deb package. It puts off people from creating Debian packages for their own programs or private use.
  - secondcoming 4y ago
    Interesting, thanks. I'll give it a read.
- iso1631 4y ago
  I store files in an appropiate strcture in my repo in a "src" folder, eg "src/usr/bin/myprog", "src/etc/cron.daily/myprog.cron" etc Then I have a "src/DEBIAN" folder with a control file and maybe things like conffiles, postinst, etc. Control file looks like Package: mypackage Version: 1.0.__datestamp__ Architecture: all Maintainer: My Name <my.name@email.com> Depends: perl, libcgi-pm-perl, libtimedate-perl, apache2 Section: web Priority: optional Installed-Size: 1000 Description: My fun package which does good stuff Do wonderful things like existing sed replaces the __datestamp__ line with yyyy.mm.dd.hh.mm.ss, then fakeroot dpkg-deb --build ./src builds a deb. (I have a larger script which does things like checking out from git, tagging, etc)
TwoNineFive 4y ago
I've actually used apt-cacher-ng, or at least I spent two years trying. Sadly, it's a buggy pile of poop that constantly segfaults and has serious data corruption bugs. It corrupts packages. Like, all you gotta do is download a file from the internet, and it messes that up by corrupting it. All of these bugs are super well-known, written about, and documented. I never did find a good Deb package cache manager. I simply wrote a script that distributes common packages between my LAN systems as I update them in a serial matter. It works but I wish there was something better. Everything I've tried is either sub-Alpha bugware and incomprehensibly complicated.
- e12e 4y ago
  How about squid or another straight up http cache?

60 points by JNRowe 4y ago

A few years ago I needed to add a new arch into Debian, it's a painful process, there are lots of arcane packages used to truly build and bootstrap the debian and they're all cross-dependent on each other. I wish there is a way like "linuxfromscratch" with clear steps to build Debian from source from scratch. Yes you can bootstrap a Debian root filesystem for a different arch, but that's after someone already did the work before you can do that, which again, is really complex and I have there are better ways to do it now(e.g. adding a new CPU family to Debian)

IntelMiner 4y ago
A lot of older Debian documentation has rotted away which is very sad for retro hobby stuff I wanted to rebuild Debian 2.2 against GCC 3.3 with its better support for SMP and SSE support for an old Pentium 3 workstation I've been playing with. Back then that kind of optimization could yield impressive performance benefits Unfortunately there's just no documentation for how to rebuild those ancient versions of Debian. The oldest I could find targeted the much newer Debian 4.0 "Etch" which came out in the mid 2000's
- djbusby 4y ago
  Gentoo still supports i486 architecture and it's totally ready to tweak build flags.
  - IntelMiner 4y ago
    Gentoo also has a ton of historical rot, something users have been trying to solve for a while https://bugs.gentoo.org/834712 https://bugs.gentoo.org/834712
thepostman0 4y ago
Might not be exactly what you're after: https://github.com/scottwilliambeasley/debian-from-scratch https://github.com/scottwilliambeasley/debian-from-scratch
pabs3 4y ago
Myself and some other folks recently wrote some new docs on how to add new arches to Debian. With multi-arch, cross-compilers in Debian and rebootstrap, it has gotten a lot less painful. There are still a ridiculous number of packages that need porting to every new arches though. Some of the needed changes should be moved into toolchains in my opinion. https://wiki.debian.org/PortsDocs/New https://wiki.debian.org/PortsDocs/New
pabs3 4y ago
Which architecture were you working on adding?

shmerl 4y ago

When is Debian going to enable zstd for packages already? It takes forever.

thepostman0 4y ago
The bug looks closed recently: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664#49 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664#49
- shmerl 4y ago
  Oh, that's good. Is it actually used now in packages in Debian repos though?

secondcoming 4y ago

It's easier to compile the linux kernel from scratch than it is to build a debian package from scratch. It's mind-blowing how complicated it is; at the end of the day it's just an archive. I hate having to do it.

vaporup 4y ago
Actually it is not that hard: - default: https://vaporup.github.io/books/the-pragmatic-sysadmin/debian-packaging/ - quick: https://nfpm.goreleaser.com/usage/
- goodpoint 4y ago
  Read the official documentation instead of random blogs.
  - kasabali 4y ago
    Official documentation is a problem. It is aimed at Debian maintainers building a fully policy compliant package that could be accepted to the official repository. You don't need any of that crap for making a .deb package. It puts off people from creating Debian packages for their own programs or private use.
- secondcoming 4y ago
  Interesting, thanks. I'll give it a read.
iso1631 4y ago
I store files in an appropiate strcture in my repo in a "src" folder, eg "src/usr/bin/myprog", "src/etc/cron.daily/myprog.cron" etc Then I have a "src/DEBIAN" folder with a control file and maybe things like conffiles, postinst, etc. Control file looks like Package: mypackage Version: 1.0.__datestamp__ Architecture: all Maintainer: My Name <my.name@email.com> Depends: perl, libcgi-pm-perl, libtimedate-perl, apache2 Section: web Priority: optional Installed-Size: 1000 Description: My fun package which does good stuff Do wonderful things like existing sed replaces the __datestamp__ line with yyyy.mm.dd.hh.mm.ss, then fakeroot dpkg-deb --build ./src builds a deb. (I have a larger script which does things like checking out from git, tagging, etc)

TwoNineFive 4y ago

I've actually used apt-cacher-ng, or at least I spent two years trying. Sadly, it's a buggy pile of poop that constantly segfaults and has serious data corruption bugs. It corrupts packages. Like, all you gotta do is download a file from the internet, and it messes that up by corrupting it. All of these bugs are super well-known, written about, and documented. I never did find a good Deb package cache manager. I simply wrote a script that distributes common packages between my LAN systems as I update them in a serial matter. It works but I wish there was something better. Everything I've tried is either sub-Alpha bugware and incomprehensibly complicated.

e12e 4y ago
How about squid or another straight up http cache?