4 ms·

Good points, but it's amusing that his solution to #1 didn't lock down the patch version, nor the distro around it. I think that also makes a decent point for N

Good points, but it's amusing that his solution to #1 didn't lock down the patch version, nor the distro around it. I think that also makes a decent point for Nix[0], which solves #1-#3 by default (since choosing a particular version of Nixpkgs locks down the whole environment, and considers the build as a DAG of dependencies rather than a linear history). It also supports exporting Docker images, while preserving Nix's richer build caching.[1] [0]: https://nixos.org/nix/ https://nixos.org/nix/ [1]: https://grahamc.com/blog/nix-and-layered-docker-images https://grahamc.com/blog/nix-and-layered-docker-images

itamarst 7y ago
Good point, will go fix that. My soon-to-be-ready attempt at a production-ready template (https://pythonspeed.com/products/pythoncontainer/ https://pythonspeed.com/products/pythoncontainer/) covers the tradeoff between point releases vs. not-point-releases, and it does pin the OS. And yes, Nix fixes some of the problems of building a production-ready image, but only a subset.
- j88439h84 7y ago
  Could you elaborate on the remaining problems with Nix for building Python images?
  - itamarst 7y ago
    Not an expert on Nix, but it's not so much that Nix has problems (though I'm sure it does, my initial research suggested it's not quite there yet for Python packages) but that there other things you need to get right. For example: 1. Signal handling (only one bit of https://hynek.me/articles/docker-signals/ https://hynek.me/articles/docker-signals/ is Dockerfile specific, the rest still applies.) 2. Configuring servers to run correctly in Docker environments (e.g. Gunicorn is broken by default, and some of these issues go beyond Gunicorn: https://pythonspeed.com/articles/gunicorn-in-docker/ https://pythonspeed.com/articles/gunicorn-in-docker/). 3. Not running as root, and dropping capabilities. 4. Building pinned dependencies for Python that you can feed to Nix. 5. Having processes (human and automated) in place to ensure security updates happen. 6. Knowing how to write shell scripts that aren't completely broken (either by not writing them at all and using better language, or by using bash strict mode: http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/) etc.
    - tathougies 7y ago
      > though I'm sure it does, my initial research suggested it's not quite there yet for Python packages) Can you expand on what's missing? I've successfully used nix to cross-compile a pretty substantial python application (+ native extensions, hence the cross compilation), for embedded purposes, and it pretty much worked out of the box. Adding extra dependencies was straightforwards. I think you can use pypi2nix for pinned dependencies, and you can run it periodically for security updates.
      - itamarst 7y ago
        Like I said, it was very preliminary research... I reached the bit where pypi2nix did "nix-env -if https://github.com/garbas/pypi2nix/tarball/master" https://github.com/garbas/pypi2nix/tarball/master" and wasn't super happy about the implications of "just use master" for production readiness. If it works, though, that's great! The more general point though is that in my experience no tool is perfect, or completely done, or without problems. E.g. the cited https://grahamc.com/blog/nix-and-layered-docker-images https://grahamc.com/blog/nix-and-layered-docker-images suggests you need to spend some time manually thinking about how to create layers for caching? Again, very preliminary research—I know people are using it, I'm just skeptical it's a magic bullet because nothing tends to be a magic bullet.
        Nullabillity 7y ago
        Regarding layering, it used to be a completely manual process (just like with Dockerfiles), but the point of the blog post was that you can now use `buildLayeredImage` and correct layering will Just Happen.
        itamarst 7y ago
        Ah, neat, hadn't realized that was an actual Nix feature now. The post made it sound like this was just something they were writing for themselves.
        tathougies 7y ago
X6S1x6Okd1st 7y ago
Nix does a pretty poor job of being able to specify the exact version that you need of something.
- Nullabillity 7y ago
  It's not great at mix-and-match pinning, but if you pin a version of Nixpkgs then you will always build from the same environment (down to the libc). There's some boilerplate, sadly, but it's still fairly simple to understand.[0] [0]: https://github.com/Etimo/photo-garden/blob/f597b95c0c488abad77fc6cb72e4b3c34a8e50b2/default.nix#L7-L19 https://github.com/Etimo/photo-garden/blob/f597b95c0c488abad...
  - asymmetric 7y ago
    You don’t need pkgsBootstrap - you can use builtins.fetchGit instead of fetchFromGitHun
    - Nullabillity 7y ago
      True, but that will do a real git clone (which is generally much slower than just downloading a static zip).

itamarst 7y ago

Good point, will go fix that. My soon-to-be-ready attempt at a production-ready template (https://pythonspeed.com/products/pythoncontainer/ https://pythonspeed.com/products/pythoncontainer/) covers the tradeoff between point releases vs. not-point-releases, and it does pin the OS. And yes, Nix fixes some of the problems of building a production-ready image, but only a subset.

j88439h84 7y ago
Could you elaborate on the remaining problems with Nix for building Python images?
- itamarst 7y ago
  Not an expert on Nix, but it's not so much that Nix has problems (though I'm sure it does, my initial research suggested it's not quite there yet for Python packages) but that there other things you need to get right. For example: 1. Signal handling (only one bit of https://hynek.me/articles/docker-signals/ https://hynek.me/articles/docker-signals/ is Dockerfile specific, the rest still applies.) 2. Configuring servers to run correctly in Docker environments (e.g. Gunicorn is broken by default, and some of these issues go beyond Gunicorn: https://pythonspeed.com/articles/gunicorn-in-docker/ https://pythonspeed.com/articles/gunicorn-in-docker/). 3. Not running as root, and dropping capabilities. 4. Building pinned dependencies for Python that you can feed to Nix. 5. Having processes (human and automated) in place to ensure security updates happen. 6. Knowing how to write shell scripts that aren't completely broken (either by not writing them at all and using better language, or by using bash strict mode: http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/) etc.
  - tathougies 7y ago
    > though I'm sure it does, my initial research suggested it's not quite there yet for Python packages) Can you expand on what's missing? I've successfully used nix to cross-compile a pretty substantial python application (+ native extensions, hence the cross compilation), for embedded purposes, and it pretty much worked out of the box. Adding extra dependencies was straightforwards. I think you can use pypi2nix for pinned dependencies, and you can run it periodically for security updates.
    - itamarst 7y ago
      Like I said, it was very preliminary research... I reached the bit where pypi2nix did "nix-env -if https://github.com/garbas/pypi2nix/tarball/master" https://github.com/garbas/pypi2nix/tarball/master" and wasn't super happy about the implications of "just use master" for production readiness. If it works, though, that's great! The more general point though is that in my experience no tool is perfect, or completely done, or without problems. E.g. the cited https://grahamc.com/blog/nix-and-layered-docker-images https://grahamc.com/blog/nix-and-layered-docker-images suggests you need to spend some time manually thinking about how to create layers for caching? Again, very preliminary research—I know people are using it, I'm just skeptical it's a magic bullet because nothing tends to be a magic bullet.
      - Nullabillity 7y ago
        Regarding layering, it used to be a completely manual process (just like with Dockerfiles), but the point of the blog post was that you can now use `buildLayeredImage` and correct layering will Just Happen.
        itamarst 7y ago
        Ah, neat, hadn't realized that was an actual Nix feature now. The post made it sound like this was just something they were writing for themselves.
      - tathougies 7y ago

X6S1x6Okd1st 7y ago

Nix does a pretty poor job of being able to specify the exact version that you need of something.

Nullabillity 7y ago
It's not great at mix-and-match pinning, but if you pin a version of Nixpkgs then you will always build from the same environment (down to the libc). There's some boilerplate, sadly, but it's still fairly simple to understand.[0] [0]: https://github.com/Etimo/photo-garden/blob/f597b95c0c488abad77fc6cb72e4b3c34a8e50b2/default.nix#L7-L19 https://github.com/Etimo/photo-garden/blob/f597b95c0c488abad...
- asymmetric 7y ago
  You don’t need pkgsBootstrap - you can use builtins.fetchGit instead of fetchFromGitHun
  - Nullabillity 7y ago
    True, but that will do a real git clone (which is generally much slower than just downloading a static zip).