6 ms·

Show HN: Jupyter kernel using Poetry for reproducible Python package management

95 points by travisd 5y ago

Kalanos 5y ago
Are you going to present at jupyter community day?https://discourse.jupyter.org/t/jupyter-community-calendar/2485 https://discourse.jupyter.org/t/jupyter-community-calendar/2...
travisd 5y ago
Hi HN! Author/OP here. Wrote this small but mighty package to make it easy to create, run, and share Jupyter notebooks with reproducible environments. Was definitely inspired by Julia's Project.toml/Manifest.toml package management. Ended up building it to support my startup Pathbird which lets instructors build engaging, interactive courses using computational lessons. A big value proposition there is that every student gets exactly the same environment (no dependency version issues!) so we want to make sure that the environment running on Pathbird exactly matches the environment on the instructor's computer.
l0b0 5y ago
Cool stuff! Have you considered using Nix for this? That way you get even more reproducibility, because all the OS packages (including Python and Poetry) can be easily pinned. If you only use packages which are already in nixpkgs you won't even need Poetry for easy repro. Example[1] [1] https://github.com/linz/stac/blob/6a82e92432945777fbd49631b48de5d381e1a4c3/shell.nix https://github.com/linz/stac/blob/6a82e92432945777fbd49631b4...
- lvass 5y ago
  This example is too impractical if you need arbitrary, pinned PyPI packages. Trying to have Poetry work with Nix has been so painful to me that I'd just rather use a FHS shim.
  - l0b0 5y ago
    Poetry2nix[1] is good for that. A lot of packages unfortunately require overrides to work, but poetry2nix ships with a bunch of these by default. [1] https://github.com/nix-community/poetry2nix https://github.com/nix-community/poetry2nix
- rch 5y ago
  I switched to Nixpkgs earlier this year, after getting annoyed with other options for working with Java and Python (in particular). NixOS on servers, Nixpkgs-darwin instead of homebrew (mostly), and Nixpkgs on Fedora for a laptop. Solid results so far.
jmakov 5y ago
How does it compare to conda/mamba + env.yaml?
- prpl 5y ago
  Likely inferior since it’s python only. You can get pretty far with python first but in most sciences that’s not going to get you all sorts of community standard things that are C/C++/fortran
  - mwt 5y ago
    Likely even a little bit more narrow than Python-only since it seems to be Poetry-only. Most Python projects probably work okay with Poetry, but it's still a relatively new tool compared to the other options.
    - travisd 5y ago
      I don’t think this is much of a concern here. Poetry is still not super compatible with a bunch of tooling, but that only matters when consuming Poetry projects. When you just install a dependency into a poetry project, it gets installed into a venv - nothing particularly exotic about it.
      - JPKab 5y ago
        Correct. They are commenting on Poetry without knowing anything about it, other than it being new and unfamiliar. Always amazed at the arrogance behind that behavior.
      - physicsguy 5y ago
        It matters a lot, because C, C++ and Fortran dependencies make up a large number of Python package dependencies. It’s not really a Poetry problem though, but a PyPi problem. Wheels improved the situation but aren’t a panacea.
    - JPKab 5y ago
      "Likely": Translation: I've never used poetry, don't know anything about it, and I'm going to talk out of my rear end and assume that it doesn't use conventional, existing venv/pip under the hood. There isn't a single python project out there that can't be worked on with poetry. It creates a venv. Poetry is doing a lot of things to manage automatically activating and deactivating it, etc, but it's just a venv.
JPKab 5y ago
This is excellent work. I worked on a similar project for my last company. However the goal was portable notebooks that could be executed anywhere, and rapidly deployed to various environments with minimal installation. I ended up using a combination of LinkedIn's shiv library with Netflix's papermill library. The point was to turn a notebook into an executable runtime with all the dependencies embedded. I don't want to get into the specifics of why we were doing this but it had to do with my previous employers product which is targeted at no code and low code folks, and integrated Jupyter notebooks. I think the use of poetry is very elegant here. And the fact that you can reuse the same kernel easily is a huge plus.
akdor1154 5y ago
Ah noice. I have been doing this with a hand edited kernel.json for a year or two and it works perfectly.. hadn't been looking forward to demoing it to coworkers due to hacky setup. You've solved the only problem then perfectly!
- travisd 5y ago
  Yep, previous version was exactly that! I decided to package it up to make it easy to give to instructors who wanted to use Pathbird. Way more than half of the effort on this project when to looking in to how to distribute the kernel.json file (I ended up copying from ipython). The actual code that runs is little more than a Popen.

95 points by travisd 5y ago

Are you going to present at jupyter community day?https://discourse.jupyter.org/t/jupyter-community-calendar/2485 https://discourse.jupyter.org/t/jupyter-community-calendar/2...

travisd 5y ago

Hi HN! Author/OP here. Wrote this small but mighty package to make it easy to create, run, and share Jupyter notebooks with reproducible environments. Was definitely inspired by Julia's Project.toml/Manifest.toml package management. Ended up building it to support my startup Pathbird which lets instructors build engaging, interactive courses using computational lessons. A big value proposition there is that every student gets exactly the same environment (no dependency version issues!) so we want to make sure that the environment running on Pathbird exactly matches the environment on the instructor's computer.

l0b0 5y ago

Cool stuff! Have you considered using Nix for this? That way you get even more reproducibility, because all the OS packages (including Python and Poetry) can be easily pinned. If you only use packages which are already in nixpkgs you won't even need Poetry for easy repro. Example[1] [1] https://github.com/linz/stac/blob/6a82e92432945777fbd49631b48de5d381e1a4c3/shell.nix https://github.com/linz/stac/blob/6a82e92432945777fbd49631b4...

lvass 5y ago
This example is too impractical if you need arbitrary, pinned PyPI packages. Trying to have Poetry work with Nix has been so painful to me that I'd just rather use a FHS shim.
- l0b0 5y ago
  Poetry2nix[1] is good for that. A lot of packages unfortunately require overrides to work, but poetry2nix ships with a bunch of these by default. [1] https://github.com/nix-community/poetry2nix https://github.com/nix-community/poetry2nix
rch 5y ago
I switched to Nixpkgs earlier this year, after getting annoyed with other options for working with Java and Python (in particular). NixOS on servers, Nixpkgs-darwin instead of homebrew (mostly), and Nixpkgs on Fedora for a laptop. Solid results so far.

jmakov 5y ago

How does it compare to conda/mamba + env.yaml?

prpl 5y ago
Likely inferior since it’s python only. You can get pretty far with python first but in most sciences that’s not going to get you all sorts of community standard things that are C/C++/fortran
- mwt 5y ago
  Likely even a little bit more narrow than Python-only since it seems to be Poetry-only. Most Python projects probably work okay with Poetry, but it's still a relatively new tool compared to the other options.
  - travisd 5y ago
    I don’t think this is much of a concern here. Poetry is still not super compatible with a bunch of tooling, but that only matters when consuming Poetry projects. When you just install a dependency into a poetry project, it gets installed into a venv - nothing particularly exotic about it.
    - JPKab 5y ago
      Correct. They are commenting on Poetry without knowing anything about it, other than it being new and unfamiliar. Always amazed at the arrogance behind that behavior.
    - physicsguy 5y ago
      It matters a lot, because C, C++ and Fortran dependencies make up a large number of Python package dependencies. It’s not really a Poetry problem though, but a PyPi problem. Wheels improved the situation but aren’t a panacea.
  - JPKab 5y ago
    "Likely": Translation: I've never used poetry, don't know anything about it, and I'm going to talk out of my rear end and assume that it doesn't use conventional, existing venv/pip under the hood. There isn't a single python project out there that can't be worked on with poetry. It creates a venv. Poetry is doing a lot of things to manage automatically activating and deactivating it, etc, but it's just a venv.

JPKab 5y ago

This is excellent work. I worked on a similar project for my last company. However the goal was portable notebooks that could be executed anywhere, and rapidly deployed to various environments with minimal installation. I ended up using a combination of LinkedIn's shiv library with Netflix's papermill library. The point was to turn a notebook into an executable runtime with all the dependencies embedded. I don't want to get into the specifics of why we were doing this but it had to do with my previous employers product which is targeted at no code and low code folks, and integrated Jupyter notebooks. I think the use of poetry is very elegant here. And the fact that you can reuse the same kernel easily is a huge plus.

akdor1154 5y ago

Ah noice. I have been doing this with a hand edited kernel.json for a year or two and it works perfectly.. hadn't been looking forward to demoing it to coworkers due to hacky setup. You've solved the only problem then perfectly!

travisd 5y ago
Yep, previous version was exactly that! I decided to package it up to make it easy to give to instructors who wanted to use Pathbird. Way more than half of the effort on this project when to looking in to how to distribute the kernel.json file (I ended up copying from ipython). The actual code that runs is little more than a Popen.