10 ms·

Moving SciPy to the Meson Build System

94 points by stockholmsin 5y ago

tr33house 5y ago
How does Meson compare to other build systems like bagel?
- qznc 5y ago
  It is Cmake with Python syntax.
  - tomn 5y ago
    Meson internals have nothing to do with cmake, and the syntax is not python. If you want to make that comparison, meson is more restrictive than cmake, but tends to have more functionality built-in to make up for it. bazel is more all-encompasing than either, as it's more about reproducible and distributed builds than a "better makefile" type system.
  - kortex 5y ago
    That doesn't strike me as a good characterization. Especially because both Python and Cmake are quite imperative. Meson is declarative with an emphasis on immutability. It's "cmake in role only" - both are build systems, that's where the similarity falls apart. The syntax is very pythonic, however, I think that's true of many ergonomic centric formats.
    - SAI_Peregrinus 5y ago
      CMake is two smaller systems in a trenchcoat, with a creepy stalker following along. There's the 3.0+ CMakeLists.txt syntax, which is a nice declarative target-based system. There's the *.cmake language syntax for finding dependencies and other scripting, which is a C'thonian horror mix of imperative and declarative bits that the trench coat tries to hide. Then there's the creepy pre-3.0 stalker, the old syntax for both. Non-target based, imperative bits sneaking in everywhere, makes Autotools look sane.
ho_schi 5y ago
I'm using Meson for a project using C++ with Gtkmm, Libmicrohttpd, catch2 and other libraries. My experience is good neglecting some minor issues. Meson is fast, well readable if written well, uses system libraries by default and allows flexible usage. On the other side I'm using Java with Maven and it is - a big burden. It's build in dependency retrieval system also isn't helping. Maybe I just don't like XML - because it is not human-readable ;)
- hawski 5y ago
  I'm sorry to pick a nit, but could you expand on "well readable if written well"? I would say that a Makefile or even a CMakeLists.txt are readable if written well.
  - mumblemumble 5y ago
    For my part, I find that being written well is necessary but not sufficient for readability in a Makefile. The other big one is that you need to have a strong familiarity with Make, and use it often. Familiar because there's no way a person who hasn't actually been through some form of Make documentation in detail can even guess at what things like $@, $?, and $^ do, or accurately decipher its macro replacement syntax, or any of that. And use it often because, even if you were deeply familiar with Make in the past, if you haven't touched it in a few years, you're unlikely to reliably remember it without help.
- 0x138d5 5y ago
  Any particular reason you use Maven instead of Gradle?
  - zaphirplane 5y ago
    Not op Each gradle task is artistically handcrafted written to capture the life experience and pain the author has experienced in their life :) it devolves into a custom shell script
- vips7L 5y ago
jkbbwr 5y ago
Meson is okay, it solved a few problems for us on our deployment but the creator has a few hard decisions and style choices that don't really gel well with other projects. We are tempted to go back to cmake.
rkrzr 5y ago
> Really there were only two viable candidates: CMake and Meson. Would be great to hear why those were the only two candidates that they considered (and not e.g. Bazel or Nix).
- chromatin 5y ago
  Yes, perhaps it is true, and it is obvious to build experts, but as a novice in this area (build systems), I would love for some more exposition.
  - flohofwoe 5y ago
    One typical problem for C/C++ build systems is lack of proper Windows and MSVC toolchain support. Most C/C++ build systems originated in the UNIX world, and either only support GCC-compatible toolchains or if they support MSVC then only as an afterthought. Trying to get this stuff running in a Windows environment is often a mix of frustation and plain hilarity. CMake is one of the few build tools that gets Windows and MSVC support right.
- rjsw 5y ago
  Haven't read the article but meson will run on a lot more systems than bazel.
- ihnorton 5y ago
  Nix is likely a nonstarter because as far as I can tell it does not natively support Visual Studio and MSVC. I suspect Bazel was ruled out because it requires the JVM and it has limited open source uptake relative to CMake (huge open source userbase), and Meson (limited presence in open source scientific software, but adopted by GNOME and systemd).
  - mumblemumble 5y ago
    I can't speak for scipy, but I ruled Bazel out for a project that included Python components a few years ago. It was great for C, C++, and Java projects, but, at least at the time, there was no official Python support. I looked into third-party options and the feasibility of building it myself. Most of what I found was a few different conference talks and blog posts where the presenter was enthusiastically talking about how they'd spent a year trying to get it to work and it's not quite there yet but they're feeling really really confident that they'll turn a corner sometime soon. But I could never find any subsequent evidence that the presenter's team actually had turned that corner. Based on that, I just sort of assumed that, in addition to all the fairly well-documented up-front challenges that these folks had identified and were talking about, there must also be some impassable barrier lurking around in there that nobody finds until they've already sunk a lot of time and money into trying to get it working. I don't know what that is, and I'm not curious enough to spend the better part of a year trying to find it for myself. I ended up choosing Gradle.
xvilka 5y ago
It would be great to port Meson from Python to pure C for portability. There is Boson[1] attempt to do exactly this but it's just a first step towards the goal. [1] https://sr.ht/~bl4ckb0ne/boson/ https://sr.ht/~bl4ckb0ne/boson/
- geofft 5y ago
  Why is Python not portable, as in, on which systems is "build Python and then use that to run Meson" not a reasonable option? The CI for boson seems like it runs on platforms where Python definitely is available, but also I notice the CI uses samurai, a reimplementation of ninja with a similar motivation: https://github.com/michaelforney/samurai https://github.com/michaelforney/samurai Ninja is in C++ so I am even more confused at Sanurai. Is this just an implementation-diversity thing? (which is great!)
ihnorton 5y ago
This is interesting. It will certainly give a boost to Meson, which is in part a better CMake (the compilation model is very similar). It probably makes a lot of sense for SciPy given the alternatives, but from an ecosystem perspective it's not clear whether it is better enough to justify the churn. For example, caching seems like a secondary concern and outsourced to Linux-specific technologies. This is unfortunate given the implications of build time for individual and team (CI) productivity, as well as the environmental considerations of redundant compilation at scale. I haven't followed Meson closely in about 3 years, but I also got the sense that Windows support sometimes lagged. If that's true, it's going to be a tough sell for the many large C++ projects who adopted CMake almost exclusively due to its support for Windows and Visual Studio.
nrclark 5y ago
This is great. I used to support an line of Linux-based realtime audio products, and our engineers wanted me to include Scipy in the internal developer builds. It turned out to be a really tough ask, because of Scipy's build system. The result was a tech-debt blemish on an otherwise excellent codebase. Scipy is very, very hard to cross-compile. Like, almost impossible. I eventually had to compile it on-target and store the outputs for inclusion in later images. Distutils in general is wretched at cross-compilation tasks, should be retired.
misnome 5y ago
This is interesting to read. I've used CMake on-and-off for mixed python/C++ code bases, and in particular https://scikit-build.readthedocs.io/en/latest/ https://scikit-build.readthedocs.io/en/latest/ which aims to integrate with cmake (lightly mentioned here but not evaluated as "getting away from setuptools". But I have never been satisfied with the solutions for out-of-tree builds with python - being able to debug-in-place extensions _and_ eventually install in a single package structure seems to be somewhat in conflict with the way that Python allows you to structure packaging, without lots of environment variable hacks.
kvnhn 5y ago
> Cross-compiling will become possible. For years we've told people "sorry, distutils wasn't really made for cross-compiling, let us know if you have any luck". As a result we've completely ignored users on some exotic platforms, and also spent a lot of time fighting with different CI systems to do native builds. As a user (not a developer) of SciPy, this is the big win. My "exotic platform" is embedded Linux distributions via Buildroot[1]. This opens the door to many downstream libraries becoming available as well, such as pandas and scikit-learn. [1]: https://buildroot.org/ https://buildroot.org/

94 points by stockholmsin 5y ago

tr33house 5y ago

How does Meson compare to other build systems like bagel?

qznc 5y ago
It is Cmake with Python syntax.
- tomn 5y ago
  Meson internals have nothing to do with cmake, and the syntax is not python. If you want to make that comparison, meson is more restrictive than cmake, but tends to have more functionality built-in to make up for it. bazel is more all-encompasing than either, as it's more about reproducible and distributed builds than a "better makefile" type system.
- kortex 5y ago
  That doesn't strike me as a good characterization. Especially because both Python and Cmake are quite imperative. Meson is declarative with an emphasis on immutability. It's "cmake in role only" - both are build systems, that's where the similarity falls apart. The syntax is very pythonic, however, I think that's true of many ergonomic centric formats.
  - SAI_Peregrinus 5y ago
    CMake is two smaller systems in a trenchcoat, with a creepy stalker following along. There's the 3.0+ CMakeLists.txt syntax, which is a nice declarative target-based system. There's the *.cmake language syntax for finding dependencies and other scripting, which is a C'thonian horror mix of imperative and declarative bits that the trench coat tries to hide. Then there's the creepy pre-3.0 stalker, the old syntax for both. Non-target based, imperative bits sneaking in everywhere, makes Autotools look sane.

ho_schi 5y ago

I'm using Meson for a project using C++ with Gtkmm, Libmicrohttpd, catch2 and other libraries. My experience is good neglecting some minor issues. Meson is fast, well readable if written well, uses system libraries by default and allows flexible usage. On the other side I'm using Java with Maven and it is - a big burden. It's build in dependency retrieval system also isn't helping. Maybe I just don't like XML - because it is not human-readable ;)

hawski 5y ago
I'm sorry to pick a nit, but could you expand on "well readable if written well"? I would say that a Makefile or even a CMakeLists.txt are readable if written well.
- mumblemumble 5y ago
  For my part, I find that being written well is necessary but not sufficient for readability in a Makefile. The other big one is that you need to have a strong familiarity with Make, and use it often. Familiar because there's no way a person who hasn't actually been through some form of Make documentation in detail can even guess at what things like $@, $?, and $^ do, or accurately decipher its macro replacement syntax, or any of that. And use it often because, even if you were deeply familiar with Make in the past, if you haven't touched it in a few years, you're unlikely to reliably remember it without help.
0x138d5 5y ago
Any particular reason you use Maven instead of Gradle?
- zaphirplane 5y ago
  Not op Each gradle task is artistically handcrafted written to capture the life experience and pain the author has experienced in their life :) it devolves into a custom shell script
vips7L 5y ago

jkbbwr 5y ago

Meson is okay, it solved a few problems for us on our deployment but the creator has a few hard decisions and style choices that don't really gel well with other projects. We are tempted to go back to cmake.

rkrzr 5y ago

> Really there were only two viable candidates: CMake and Meson. Would be great to hear why those were the only two candidates that they considered (and not e.g. Bazel or Nix).

chromatin 5y ago
Yes, perhaps it is true, and it is obvious to build experts, but as a novice in this area (build systems), I would love for some more exposition.
- flohofwoe 5y ago
  One typical problem for C/C++ build systems is lack of proper Windows and MSVC toolchain support. Most C/C++ build systems originated in the UNIX world, and either only support GCC-compatible toolchains or if they support MSVC then only as an afterthought. Trying to get this stuff running in a Windows environment is often a mix of frustation and plain hilarity. CMake is one of the few build tools that gets Windows and MSVC support right.
rjsw 5y ago
Haven't read the article but meson will run on a lot more systems than bazel.
ihnorton 5y ago
Nix is likely a nonstarter because as far as I can tell it does not natively support Visual Studio and MSVC. I suspect Bazel was ruled out because it requires the JVM and it has limited open source uptake relative to CMake (huge open source userbase), and Meson (limited presence in open source scientific software, but adopted by GNOME and systemd).
- mumblemumble 5y ago
  I can't speak for scipy, but I ruled Bazel out for a project that included Python components a few years ago. It was great for C, C++, and Java projects, but, at least at the time, there was no official Python support. I looked into third-party options and the feasibility of building it myself. Most of what I found was a few different conference talks and blog posts where the presenter was enthusiastically talking about how they'd spent a year trying to get it to work and it's not quite there yet but they're feeling really really confident that they'll turn a corner sometime soon. But I could never find any subsequent evidence that the presenter's team actually had turned that corner. Based on that, I just sort of assumed that, in addition to all the fairly well-documented up-front challenges that these folks had identified and were talking about, there must also be some impassable barrier lurking around in there that nobody finds until they've already sunk a lot of time and money into trying to get it working. I don't know what that is, and I'm not curious enough to spend the better part of a year trying to find it for myself. I ended up choosing Gradle.

xvilka 5y ago

It would be great to port Meson from Python to pure C for portability. There is Boson[1] attempt to do exactly this but it's just a first step towards the goal. [1] https://sr.ht/~bl4ckb0ne/boson/ https://sr.ht/~bl4ckb0ne/boson/

geofft 5y ago
Why is Python not portable, as in, on which systems is "build Python and then use that to run Meson" not a reasonable option? The CI for boson seems like it runs on platforms where Python definitely is available, but also I notice the CI uses samurai, a reimplementation of ninja with a similar motivation: https://github.com/michaelforney/samurai https://github.com/michaelforney/samurai Ninja is in C++ so I am even more confused at Sanurai. Is this just an implementation-diversity thing? (which is great!)

ihnorton 5y ago

This is interesting. It will certainly give a boost to Meson, which is in part a better CMake (the compilation model is very similar). It probably makes a lot of sense for SciPy given the alternatives, but from an ecosystem perspective it's not clear whether it is better enough to justify the churn. For example, caching seems like a secondary concern and outsourced to Linux-specific technologies. This is unfortunate given the implications of build time for individual and team (CI) productivity, as well as the environmental considerations of redundant compilation at scale. I haven't followed Meson closely in about 3 years, but I also got the sense that Windows support sometimes lagged. If that's true, it's going to be a tough sell for the many large C++ projects who adopted CMake almost exclusively due to its support for Windows and Visual Studio.

nrclark 5y ago

This is great. I used to support an line of Linux-based realtime audio products, and our engineers wanted me to include Scipy in the internal developer builds. It turned out to be a really tough ask, because of Scipy's build system. The result was a tech-debt blemish on an otherwise excellent codebase. Scipy is very, very hard to cross-compile. Like, almost impossible. I eventually had to compile it on-target and store the outputs for inclusion in later images. Distutils in general is wretched at cross-compilation tasks, should be retired.

misnome 5y ago

This is interesting to read. I've used CMake on-and-off for mixed python/C++ code bases, and in particular https://scikit-build.readthedocs.io/en/latest/ https://scikit-build.readthedocs.io/en/latest/ which aims to integrate with cmake (lightly mentioned here but not evaluated as "getting away from setuptools". But I have never been satisfied with the solutions for out-of-tree builds with python - being able to debug-in-place extensions _and_ eventually install in a single package structure seems to be somewhat in conflict with the way that Python allows you to structure packaging, without lots of environment variable hacks.

kvnhn 5y ago

> Cross-compiling will become possible. For years we've told people "sorry, distutils wasn't really made for cross-compiling, let us know if you have any luck". As a result we've completely ignored users on some exotic platforms, and also spent a lot of time fighting with different CI systems to do native builds. As a user (not a developer) of SciPy, this is the big win. My "exotic platform" is embedded Linux distributions via Buildroot[1]. This opens the door to many downstream libraries becoming available as well, such as pandas and scikit-learn. [1]: https://buildroot.org/ https://buildroot.org/