6 ms·

Migration of the build system to autosetup

95 points by sgbeal 2y ago

sgbeal 2y ago
OP here! Richard Hipp, of sqlite fame, has announced that the sqlite project plans to reimplement its canonical build process in the 3.48 release cycle. The project will, if all goes as planned, be moving away from the GNU Autotools and towards Steve Bennett's Autosetup, which we've used to good effect since 2011 in SQLite's sibling project, the Fossil SCM. The primary motivation behind this HN post is to draw eyes to what Richard points out in the linked forum post: this port _will_ break some folks' build processes and we'd like to know, in advance, what we'll be breaking so that we can make that transition as painless as we can. If you do "unusual" things with the SQLite build, please let us know about it (preferably in the linked-to forum, as that's less likely to be overlooked than comments on HN are, but we'll be following this thread for at least a few days for those of you who would prefer to voice your thoughts here).
- ChoHag 2y ago
  [dead]
- westurner 2y ago
  Conda-forge/sqlite-feedstock; recipe/meta.yml, recipe/build.sh: https://github.com/conda-forge/sqlite-feedstock/blob/main/recipe/build.sh https://github.com/conda-forge/sqlite-feedstock/blob/main/re... recipe/build.bat: https://github.com/conda-forge/sqlite-feedstock/blob/main/recipe/bld.bat https://github.com/conda-forge/sqlite-feedstock/blob/main/re... Is there a [multi-stage] Dockerfile to build, install and test SQLite with the new build system? Is there SLSA signing for the SQLite build artifacts? There should be a way to sign OBS Open Build Service packages with sigstore.dev's Cosign to Rekor, too IIUC. slsa-framework/slsa-github-generator: https://github.com/slsa-framework/slsa-github-generator https://github.com/slsa-framework/slsa-github-generator
  - sgbeal 2y ago
    > Conda-forge/sqlite-feedstock; recipe/meta.yml, Aside from the cross-compilation, which is most definitely part of the process but has not yet been explored in depth[^1], nothing in your build immediately sticks out at as problematic, but i've bookmarked it for future reference during the port. The .bat build (Windows) will not change via this port. Only Unix-like environments will potentially be affected. > Is there a [multi-stage] Dockerfile to build, install and test SQLite with the new build system? Not that the sqlite project maintains, no. None of us use docker in any capacity. We work only from the canonical source tree and we like to think that it's easy enough to do that other folks can too. > Is there SLSA signing for the SQLite build artifacts? That's out of scope for, and unrelated to, this particular sub-project. > slsa-framework/slsa-github-generator: https://github.com/slsa-framework/slsa-github-generator https://github.com/slsa-framework/slsa-github-generator says: > This repository contains free tools to generate and verify SLSA Build Level 3 provenance for native GitHub projects using GitHub Actions. We don't use github except to post a read-only mirror of the canonical source tree. [^1]: Autosetup has strong support for cross-compilation, it just hasn't yet been tested in the being-ported build process.
    - westurner 2y ago
      You can sign built artifacts with cosign without actions or podman[-desktop]/docker. It's probably possible to trigger gh actions or gl pipelines builds (container, command) with git repo mirroring. datasette-lite depends on a WASM build of SQLite; what about WASM and e.g. emscripten-forge and micromamba? Edit emscripten-forge/recipes/recipes_emscripten/SQLite/build.sh: https://github.com/emscripten-forge/recipes/blob/main/recipes/recipes_emscripten/sqlite/build.sh https://github.com/emscripten-forge/recipes/blob/main/recipe... ... recipe.yml says 2022: https://github.com/emscripten-forge/recipes/blob/main/recipes/recipes_emscripten/sqlite/recipe.yaml https://github.com/emscripten-forge/recipes/blob/main/recipe...
      - sgbeal 2y ago
        > You can sign built artifacts with cosign without actions or podman[-desktop]/docker. That's all out of scope for, and unrelated to, this build port. > It's probably possible to trigger gh actions or gl pipelines builds (container, command) with git repo mirroring. We maintain our own SCM, which keeps us far removed from the github ways of doing things. None of the SQLite project members actively use github beyond posting the read-only project mirror and occasionally cloning other projects and/or participating in tickets, and that's unlikely to change in any foreseeable future. > emscripten-forge/recipes/recipes_emscripten/SQLite/build.sh: Thank you - bookmarked for later perusal and testing.
- fhdsgbbcaA 2y ago
  .
  - sgbeal 2y ago
    > ... you may want to repost again in 12 hours for more eyes. Duplicate threads are a plague ;). Posting a second time will only lead to chaos as i try to juggle multiple threads while continuing work on the posted-about build port.
  - billfruit 2y ago
    Isn't Bangalore as important as California.
markasoftware 2y ago
Why autosetup? Because it'll hopefully keep the build commands the same? Or are there more fundamental reasons to prefer it over cmake?
- sgbeal 2y ago
  > Why autosetup? A link in the original post leads to a description of the motivation for the port. TL;DR: we've used Autosetup since 2011 in SQLite's sibling, the Fossil SCM, and we quite like it. > Or are there more fundamental reasons to prefer it over cmake? CMake is an out-of-tree tool, which is something the project tries to avoid at all costs for portability and maintainability reasons. GNU Autotools lives halfway in-tree and halfway out-of-tree. It's ubiquitous but also often problematic to upgrade, and is subject to breakage when project developers have different versions of the Autotools installed on their systems. Richard very specifically keeps an ancient version of Autotools on his system to reduce that pain, but that's not a satisfying situation to be in. Autosetup, on the other hand, lives entirely in the project's source tree, meaning we have not only consistency across all developers' machines, but also complete control over when to upgrade (or not) the tools. Over the past 13ish years of using Autosetup in the Fossil SCM we have never had any significant upgrade-related pains.
  - chipdart 2y ago
    > CMake is an out-of-tree tool, which is something the project tries to avoid at all costs for portability and maintainability reasons. Is this relevant though? Portability is not dictated by whether you vend a tool within your project tree, and maintainability is only remotely relevant if you're planning on customizing the tool instead of the scripts executed by the tool, which would be silly. I mean, there is no wrong in just coming forward and stating that you're already using a build system in another project and you're hoping to lower maintenance costs by standardizing on a single build system that you're already using. All other arguments sound dubious at best.
    - sgbeal 2y ago
      > > CMake is an out-of-tree tool, which is something the project tries to avoid at all costs for portability and maintainability reasons. > Is this relevant though? For this project and its day-to-day maintenance, unequivocally. One of Richard's philosophies is "freedom means being able to take care of yourself," and that philosophy permeates all of his software projects, which eschew external dependencies because every single one reduces the project's level of freedom. Every time a third-party tool breaks, or introduces new, incompatible behavior as part of an upgrade, dependents of that tool suffer[^1]. All of Richard's projects have a strong culture of avoiding that suffering by avoiding third-party dependencies where at all possible, to the point of sometimes reinventing the wheel to extreme degrees (like implementing his own SCM). That aspect aside, nobody in the SQLite project uses CMake, so it would never occur to us to migrate the build process to it. We write our own makefiles by hand _and we like it that way_. (Sidebar: Autosetup neither writes our makefiles nor introduces new syntax for doing so, in contrast to CMake.) > I mean, there is no wrong in just coming forward and stating that you're already using a build system in another project and you're hoping to lower maintenance costs by standardizing on a single build system that you're already using. All other arguments sound dubious at best. This isn't about standardizing on one build process, but about simplifying the lives of the developers and future maintenance of the project. We can't do that by porting to tools we don't otherwise use. If we weren't using Autosetup in other trees, and weren't happy with it, we wouldn't have even considered undertaking this effort and facing the downstream upheaval it will undoubtedly cause. We know, however, from 13 years of experience with it, that this tool does what we want, is low friction, and causes us an amount of grief which closely approaches zero[^2]. Without our collective background with Autosetup, we wouldn't have even considered porting away from the Autotools because that would be a case of "out of the frying pan and into the oven." [^1]: As a concrete example: the SQLite project's WASM build is 100% dependent on the Emscripten toolchain for the simple reason that there is no feasible alternative. A handful of times over the past two years we have had to accommodate incompatible changes when upgrading Emscripten. That's hassle nobody wants. Obviously, software has to evolve, but being able to embed our build tools (Autosetup) directly in the source tree gives us control over the pace of that evolution. No system-wide upgrade is going to pull it out from under us, nor will it behave differently on downstream folks' machines. [^2]: You won't ever hear me say that _any_ software is 100% grief-free, but autosetup is as close to it as, e.g., the bash shell.
hranmax 2y ago
Thanks for posting this, autosetup looks like a really cool project. Personally I'm fine with autoconf (not with automake and libtool!), but it does require some maintenance. I've never made the jump to CMake, but autosetup looks like a viable alternative. It is nice to see a Tcl application with an included interpreter!
deleted 2y ago
[deleted]
sgbeal 2y ago
As of moments ago, linked-to announcement became a reality and the current SQLite trunk now uses autosetup instead of the autotools. We still have some cleanups to do before the 3.48 release but the majority of the build functionality is working fine.

95 points by sgbeal 2y ago

OP here! Richard Hipp, of sqlite fame, has announced that the sqlite project plans to reimplement its canonical build process in the 3.48 release cycle. The project will, if all goes as planned, be moving away from the GNU Autotools and towards Steve Bennett's Autosetup, which we've used to good effect since 2011 in SQLite's sibling project, the Fossil SCM. The primary motivation behind this HN post is to draw eyes to what Richard points out in the linked forum post: this port _will_ break some folks' build processes and we'd like to know, in advance, what we'll be breaking so that we can make that transition as painless as we can. If you do "unusual" things with the SQLite build, please let us know about it (preferably in the linked-to forum, as that's less likely to be overlooked than comments on HN are, but we'll be following this thread for at least a few days for those of you who would prefer to voice your thoughts here).

ChoHag 2y ago
[dead]
westurner 2y ago
Conda-forge/sqlite-feedstock; recipe/meta.yml, recipe/build.sh: https://github.com/conda-forge/sqlite-feedstock/blob/main/recipe/build.sh https://github.com/conda-forge/sqlite-feedstock/blob/main/re... recipe/build.bat: https://github.com/conda-forge/sqlite-feedstock/blob/main/recipe/bld.bat https://github.com/conda-forge/sqlite-feedstock/blob/main/re... Is there a [multi-stage] Dockerfile to build, install and test SQLite with the new build system? Is there SLSA signing for the SQLite build artifacts? There should be a way to sign OBS Open Build Service packages with sigstore.dev's Cosign to Rekor, too IIUC. slsa-framework/slsa-github-generator: https://github.com/slsa-framework/slsa-github-generator https://github.com/slsa-framework/slsa-github-generator
- sgbeal 2y ago
  > Conda-forge/sqlite-feedstock; recipe/meta.yml, Aside from the cross-compilation, which is most definitely part of the process but has not yet been explored in depth[^1], nothing in your build immediately sticks out at as problematic, but i've bookmarked it for future reference during the port. The .bat build (Windows) will not change via this port. Only Unix-like environments will potentially be affected. > Is there a [multi-stage] Dockerfile to build, install and test SQLite with the new build system? Not that the sqlite project maintains, no. None of us use docker in any capacity. We work only from the canonical source tree and we like to think that it's easy enough to do that other folks can too. > Is there SLSA signing for the SQLite build artifacts? That's out of scope for, and unrelated to, this particular sub-project. > slsa-framework/slsa-github-generator: https://github.com/slsa-framework/slsa-github-generator https://github.com/slsa-framework/slsa-github-generator says: > This repository contains free tools to generate and verify SLSA Build Level 3 provenance for native GitHub projects using GitHub Actions. We don't use github except to post a read-only mirror of the canonical source tree. [^1]: Autosetup has strong support for cross-compilation, it just hasn't yet been tested in the being-ported build process.
  - westurner 2y ago
    You can sign built artifacts with cosign without actions or podman[-desktop]/docker. It's probably possible to trigger gh actions or gl pipelines builds (container, command) with git repo mirroring. datasette-lite depends on a WASM build of SQLite; what about WASM and e.g. emscripten-forge and micromamba? Edit emscripten-forge/recipes/recipes_emscripten/SQLite/build.sh: https://github.com/emscripten-forge/recipes/blob/main/recipes/recipes_emscripten/sqlite/build.sh https://github.com/emscripten-forge/recipes/blob/main/recipe... ... recipe.yml says 2022: https://github.com/emscripten-forge/recipes/blob/main/recipes/recipes_emscripten/sqlite/recipe.yaml https://github.com/emscripten-forge/recipes/blob/main/recipe...
    - sgbeal 2y ago
      > You can sign built artifacts with cosign without actions or podman[-desktop]/docker. That's all out of scope for, and unrelated to, this build port. > It's probably possible to trigger gh actions or gl pipelines builds (container, command) with git repo mirroring. We maintain our own SCM, which keeps us far removed from the github ways of doing things. None of the SQLite project members actively use github beyond posting the read-only project mirror and occasionally cloning other projects and/or participating in tickets, and that's unlikely to change in any foreseeable future. > emscripten-forge/recipes/recipes_emscripten/SQLite/build.sh: Thank you - bookmarked for later perusal and testing.
fhdsgbbcaA 2y ago
.
- sgbeal 2y ago
  > ... you may want to repost again in 12 hours for more eyes. Duplicate threads are a plague ;). Posting a second time will only lead to chaos as i try to juggle multiple threads while continuing work on the posted-about build port.
- billfruit 2y ago
  Isn't Bangalore as important as California.

markasoftware 2y ago

Why autosetup? Because it'll hopefully keep the build commands the same? Or are there more fundamental reasons to prefer it over cmake?

sgbeal 2y ago
> Why autosetup? A link in the original post leads to a description of the motivation for the port. TL;DR: we've used Autosetup since 2011 in SQLite's sibling, the Fossil SCM, and we quite like it. > Or are there more fundamental reasons to prefer it over cmake? CMake is an out-of-tree tool, which is something the project tries to avoid at all costs for portability and maintainability reasons. GNU Autotools lives halfway in-tree and halfway out-of-tree. It's ubiquitous but also often problematic to upgrade, and is subject to breakage when project developers have different versions of the Autotools installed on their systems. Richard very specifically keeps an ancient version of Autotools on his system to reduce that pain, but that's not a satisfying situation to be in. Autosetup, on the other hand, lives entirely in the project's source tree, meaning we have not only consistency across all developers' machines, but also complete control over when to upgrade (or not) the tools. Over the past 13ish years of using Autosetup in the Fossil SCM we have never had any significant upgrade-related pains.
- chipdart 2y ago
  > CMake is an out-of-tree tool, which is something the project tries to avoid at all costs for portability and maintainability reasons. Is this relevant though? Portability is not dictated by whether you vend a tool within your project tree, and maintainability is only remotely relevant if you're planning on customizing the tool instead of the scripts executed by the tool, which would be silly. I mean, there is no wrong in just coming forward and stating that you're already using a build system in another project and you're hoping to lower maintenance costs by standardizing on a single build system that you're already using. All other arguments sound dubious at best.
  - sgbeal 2y ago
    > > CMake is an out-of-tree tool, which is something the project tries to avoid at all costs for portability and maintainability reasons. > Is this relevant though? For this project and its day-to-day maintenance, unequivocally. One of Richard's philosophies is "freedom means being able to take care of yourself," and that philosophy permeates all of his software projects, which eschew external dependencies because every single one reduces the project's level of freedom. Every time a third-party tool breaks, or introduces new, incompatible behavior as part of an upgrade, dependents of that tool suffer[^1]. All of Richard's projects have a strong culture of avoiding that suffering by avoiding third-party dependencies where at all possible, to the point of sometimes reinventing the wheel to extreme degrees (like implementing his own SCM). That aspect aside, nobody in the SQLite project uses CMake, so it would never occur to us to migrate the build process to it. We write our own makefiles by hand _and we like it that way_. (Sidebar: Autosetup neither writes our makefiles nor introduces new syntax for doing so, in contrast to CMake.) > I mean, there is no wrong in just coming forward and stating that you're already using a build system in another project and you're hoping to lower maintenance costs by standardizing on a single build system that you're already using. All other arguments sound dubious at best. This isn't about standardizing on one build process, but about simplifying the lives of the developers and future maintenance of the project. We can't do that by porting to tools we don't otherwise use. If we weren't using Autosetup in other trees, and weren't happy with it, we wouldn't have even considered undertaking this effort and facing the downstream upheaval it will undoubtedly cause. We know, however, from 13 years of experience with it, that this tool does what we want, is low friction, and causes us an amount of grief which closely approaches zero[^2]. Without our collective background with Autosetup, we wouldn't have even considered porting away from the Autotools because that would be a case of "out of the frying pan and into the oven." [^1]: As a concrete example: the SQLite project's WASM build is 100% dependent on the Emscripten toolchain for the simple reason that there is no feasible alternative. A handful of times over the past two years we have had to accommodate incompatible changes when upgrading Emscripten. That's hassle nobody wants. Obviously, software has to evolve, but being able to embed our build tools (Autosetup) directly in the source tree gives us control over the pace of that evolution. No system-wide upgrade is going to pull it out from under us, nor will it behave differently on downstream folks' machines. [^2]: You won't ever hear me say that _any_ software is 100% grief-free, but autosetup is as close to it as, e.g., the bash shell.

hranmax 2y ago

Thanks for posting this, autosetup looks like a really cool project. Personally I'm fine with autoconf (not with automake and libtool!), but it does require some maintenance. I've never made the jump to CMake, but autosetup looks like a viable alternative. It is nice to see a Tcl application with an included interpreter!

deleted 2y ago

[deleted]

sgbeal 2y ago

As of moments ago, linked-to announcement became a reality and the current SQLite trunk now uses autosetup instead of the autotools. We still have some cleanups to do before the 3.48 release but the majority of the build functionality is working fine.