5 ms·

Nix has a way of exposing complex and low-quality build systems. Many pieces of the Python ecosystem fall into this category. For example, it wasn't Nix's fault

by beefee 6y ago

Nix has a way of exposing complex and low-quality build systems. Many pieces of the Python ecosystem fall into this category. For example, it wasn't Nix's fault that psycopg2 had a dependency on postgresql C header files, nor was it Nix's fault that the postgresql headers had to be rearranged for psycopg2's build to work. Nix basically reveals the full complexity of the package's actual build environment, and for some packages the work required to make a reliable build is overwhelming. Some easier to use systems have people doing this work for you, like the Debian project. Others rely on luck to supply dependencies. The thousands of Dockerfiles that run "apt-get update" are a great example of this. After using Nix for a while you start to reflexively shy away from software with low-quality build systems. Nix is not easy to use, but it's best of breed in terms of software supply chain auditability and malleability (anything in the system can be trivially modified or patched). If you can move to an "easy" system, that may be the appropriate trade-off for you, but it means you may be in a domain where these properties are unimportant.

throwaway894345 6y ago
It's not Nix's fault, but that doesn't make it cheaper to deal with. Whatever you want to say, `pip install psycopg2` just works because someone else already dealt with the packaging problem.
- takeda 6y ago
  That's the cost of full reproducibility though. If Nix would not expect the postgresql package listed in dependencies and instead relied on what's currently installed on your system we would get to the starting point. Where something works on one person's computer but doesn't on another. Yes it is harder, but if you incorporate nix definitions in your source code (you need to pin nixpkgs though) then everyone will get exact same versions of the packages and your code works on everyone's computer.
  - throwaway894345 6y ago
    There are other ways to get enough reproducibility without the headaches of Nix, however, so that’s what we ended up doing. Reproducibility is nice, but if we need to be able to develop software quickly and for the time being, Nix is an impediment. As previously mentioned, this doesn’t have to be the case; it’s mostly an artifact of lack of documentation and a strong preference for the novel and unfamiliar over the familiar, but also for lack of escape hatches.
    - takeda 6y ago
      What other ways? If you don't specify essential dependency and rely on the dependency to be installed that by definition is not reproducible. The fix in Nix for psycopg2 is specify that it also depends on postgresql, and that's all what's needed. Here's definition of psycopg2 from NixPkgs: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/psycopg2/default.nix#L9-L15 https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen... (the highlighted part is the only thing that's needed everything else is just informational, doCheck is to control whether unit tests should be run during build and disabled = isPyPy tells that psycopg2 doesn't work with PyPy, which it doesn't work with, because PyPy doesn't support C extensions)
berti 6y ago
Read your way down this thread and note that every defence of Nix is merely shifting the blame in it's entirety to some other part of the ecosystem. Rightly or wrongly that's a big problem for Nix if it intends to gain widespread acceptance.
- takeda 6y ago
  It's true somewhat though, but it's a problem is not easy to solve for Python. The problem is that Python supports C extensions, and for example the psycopg2 package has an extra dependency on a libpq which it expects to have installed on the system i.e. if you were using a docker you would do something like yum install postgresql-devel otherwise it will fail. Python won't install it for you and it doesn't even have a way to relay (except for error message) that such package is needed. Many people actually get stuck with this and won't know what to do, but quick google show that package is needed they install and everything works. Nix is functional and doesn't have state so you can't just invoke nix-env -i postgresql and now everything will start to work, that behavior would actually ruin reproducibility. Nix instead expects that the C dependency is also provided otherwise it will refuse the build. That's what OP meant where he said Nix exposes weak build systems. Anyway, there was significant progress in making things better in python, it is still not awesome, but it is very close. Actually the biggest pain point right now is to setup a build environment that's convenient to use. I recently created a template that shows how to do it: https://github.com/takeda/example_python_project https://github.com/takeda/example_python_project It uses pip2nix and it can automatically figure out all python dependencies, the packages like psycopg2 need an additional entry like this: https://github.com/takeda/example_python_project/blob/master/nix/python-packages-overrides.nix#L17-L21 https://github.com/takeda/example_python_project/blob/master... because PyPI packages don't specify any system dependencies.
- Ericson2314 6y ago
  Almost every language build system ecosystem does someting quite wrong---it's irked me for a while. I hope to within the next year or two find the time/budget to pick one language and make it work properly, and integrate with Nix perfectly. I think not everything knows what they are missing, and by doing one complete demo we'll be able to raise the bar and give the other languages a good jolt.

by beefee 6y ago

throwaway894345 6y ago

It's not Nix's fault, but that doesn't make it cheaper to deal with. Whatever you want to say, `pip install psycopg2` just works because someone else already dealt with the packaging problem.

takeda 6y ago
That's the cost of full reproducibility though. If Nix would not expect the postgresql package listed in dependencies and instead relied on what's currently installed on your system we would get to the starting point. Where something works on one person's computer but doesn't on another. Yes it is harder, but if you incorporate nix definitions in your source code (you need to pin nixpkgs though) then everyone will get exact same versions of the packages and your code works on everyone's computer.
- throwaway894345 6y ago
  There are other ways to get enough reproducibility without the headaches of Nix, however, so that’s what we ended up doing. Reproducibility is nice, but if we need to be able to develop software quickly and for the time being, Nix is an impediment. As previously mentioned, this doesn’t have to be the case; it’s mostly an artifact of lack of documentation and a strong preference for the novel and unfamiliar over the familiar, but also for lack of escape hatches.
  - takeda 6y ago
    What other ways? If you don't specify essential dependency and rely on the dependency to be installed that by definition is not reproducible. The fix in Nix for psycopg2 is specify that it also depends on postgresql, and that's all what's needed. Here's definition of psycopg2 from NixPkgs: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/psycopg2/default.nix#L9-L15 https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen... (the highlighted part is the only thing that's needed everything else is just informational, doCheck is to control whether unit tests should be run during build and disabled = isPyPy tells that psycopg2 doesn't work with PyPy, which it doesn't work with, because PyPy doesn't support C extensions)

berti 6y ago

Read your way down this thread and note that every defence of Nix is merely shifting the blame in it's entirety to some other part of the ecosystem. Rightly or wrongly that's a big problem for Nix if it intends to gain widespread acceptance.

takeda 6y ago
It's true somewhat though, but it's a problem is not easy to solve for Python. The problem is that Python supports C extensions, and for example the psycopg2 package has an extra dependency on a libpq which it expects to have installed on the system i.e. if you were using a docker you would do something like yum install postgresql-devel otherwise it will fail. Python won't install it for you and it doesn't even have a way to relay (except for error message) that such package is needed. Many people actually get stuck with this and won't know what to do, but quick google show that package is needed they install and everything works. Nix is functional and doesn't have state so you can't just invoke nix-env -i postgresql and now everything will start to work, that behavior would actually ruin reproducibility. Nix instead expects that the C dependency is also provided otherwise it will refuse the build. That's what OP meant where he said Nix exposes weak build systems. Anyway, there was significant progress in making things better in python, it is still not awesome, but it is very close. Actually the biggest pain point right now is to setup a build environment that's convenient to use. I recently created a template that shows how to do it: https://github.com/takeda/example_python_project https://github.com/takeda/example_python_project It uses pip2nix and it can automatically figure out all python dependencies, the packages like psycopg2 need an additional entry like this: https://github.com/takeda/example_python_project/blob/master/nix/python-packages-overrides.nix#L17-L21 https://github.com/takeda/example_python_project/blob/master... because PyPI packages don't specify any system dependencies.
Ericson2314 6y ago
Almost every language build system ecosystem does someting quite wrong---it's irked me for a while. I hope to within the next year or two find the time/budget to pick one language and make it work properly, and integrate with Nix perfectly. I think not everything knows what they are missing, and by doing one complete demo we'll be able to raise the bar and give the other languages a good jolt.