4 ms·

> There aren't good boundaries between Jupyter's own Python environment, and that of your notebooks— if you have a dependency which conflicts with one of Jupyte

> There aren't good boundaries between Jupyter's own Python environment, and that of your notebooks— if you have a dependency which conflicts with one of Jupyter's dependencies, then good luck. It's cumbersome, and I'm not totally sure it's the correct way, but I remember getting around this by creating a virtualenv for my projects and then using that virtualenv's python as Jupyter's "kernel".

anvuong 3y ago
This is the way. I use miniconda to create env for Jupyter and install only Jupyter and its dependency on that, then I just config it to point to all the other Python envs I use for my different projects. This is only one-time setup and it is absolutely worth it given how messy the ecosystem is.
- tetris11 3y ago
  > miniconda Try micromamba, it will shave years off dependency resolution
  - henrydark 3y ago
    I used to use micromamba, recently moved to mach-nix with nix. It's also really fast, but with the reproducibility of nix.
modriano 3y ago
Yeah, that's essentially the right way (I use conda envs, but venvs are probably fine too). Have one env for running the Jupyter Lab/Notebook server (install jupyter, jupyter widgets, nbextensions, etc into this env) and then create other envs for your projects and register that kernel with jupyter [0]. You can also just install all of the jupyter ide cruft in each env if you want, but dependencies can get annoying. [0] https://stackoverflow.com/questions/39604271/conda-environments-not-showing-up-in-jupyter-notebook#:~:text=Just%20run%20conda%20install%20ipykernel,to%20see%20newly%20added%20kernels https://stackoverflow.com/questions/39604271/conda-environme....
garbagecoder 3y ago
I just do a venv inside the venv so I can target several Python versions. I also use Jupiter for C#, Clojure, and Julia. I wish the Clojure integration was as good as the others I listed. I’m to the point now where if anything other than venv/pip is required I won’t use it. Unfortunately there are many things that insist on conda.
- Y_Y 3y ago
  Conda is a fucking nightmare. Luckily you can usually just use pip to get the same packages.
  - jt_b 3y ago
    [flagged]
  - sterlinm 3y ago
    Why does “pip install gcc” keep failing then? ;) Conda isn’t perfect but takes on a lot of problems that pip doesn’t deal with at all. Regular conda is really slow these days but you can use mamba instead or just configure conda to use the libmamba solver and it’s much nicer. The folks at prefix.dev seem to be building some pretty cool drop in replacements for conda too.
  - garbagecoder 3y ago
    One that you can’t is the Clojure kernel for Jupiter. I installed it from source. That’s no big deal, but they aren’t 1-to-1 with packages.
awinter-py 3y ago
had a ton of trouble getting jupyterlab to target an arbitrary virtualenv
- nl 3y ago
  The trick is that you have to deactivate the virtual environment and then resource it after adding Jupyter to that virtual environment. Most shells cache executable paths, so the path for jupyter will be the global path, not the one for your virtual environment. This is unfortunately not at all obvious and leads to very hard to track down bugs that seem to disappear and reappear if you aren't familiar with the issue. I have a recipe here which always works: https://github.com/nlothian/m1_huggingface_diffusers_demo#setup https://github.com/nlothian/m1_huggingface_diffusers_demo#se... If you don't have requirements.txt then do this: `pip3 install jupyter` for that line, then `deactivate` and `source ./venv/bin/activate`.
  - nerdponx 3y ago
    I almost never rely on activating environments anymore. If it's in my project, I refer to it by its path relative to the project top level. If it's meant to be "global", I use Pipx, and if for some reason I can't, I use a script in ~/bin that uses the absolute path.
mauforonda 3y ago
Right. I usually `pipx install jupyterlab` and register each project's env with `python -m ipykernel install --user --name MyProject --display-name "My Project"` as per the ipython docs: https://ipython.readthedocs.io/en/stable/install/kernel_install.html#kernels-for-different-environments https://ipython.readthedocs.io/en/stable/install/kernel_inst...
rahimnathwani 3y ago
Yeah this is the right way. But I don't think it's obvious from the getting started docs. I bet a lot of people end up installing jupyter into each virtual environment instead. This sort of works, as (I think) most people are only working on one or two notebooks at a time, and aren't using notebooks that relate to more than one virtual environment.
nerdponx 3y ago
I do this, but it isn't well supported. Most Jupyterlab extensions expect you to `pip install` and that will handle both the frontend and backend components. The docs make basically no mention of the frontend/backend split. I end up having to install a bunch of frontend stuff in my project envs just to make sure my notebooks work.

anvuong 3y ago

This is the way. I use miniconda to create env for Jupyter and install only Jupyter and its dependency on that, then I just config it to point to all the other Python envs I use for my different projects. This is only one-time setup and it is absolutely worth it given how messy the ecosystem is.

tetris11 3y ago
> miniconda Try micromamba, it will shave years off dependency resolution
- henrydark 3y ago
  I used to use micromamba, recently moved to mach-nix with nix. It's also really fast, but with the reproducibility of nix.

modriano 3y ago

Yeah, that's essentially the right way (I use conda envs, but venvs are probably fine too). Have one env for running the Jupyter Lab/Notebook server (install jupyter, jupyter widgets, nbextensions, etc into this env) and then create other envs for your projects and register that kernel with jupyter [0]. You can also just install all of the jupyter ide cruft in each env if you want, but dependencies can get annoying. [0] https://stackoverflow.com/questions/39604271/conda-environments-not-showing-up-in-jupyter-notebook#:~:text=Just%20run%20conda%20install%20ipykernel,to%20see%20newly%20added%20kernels https://stackoverflow.com/questions/39604271/conda-environme....

garbagecoder 3y ago

I just do a venv inside the venv so I can target several Python versions. I also use Jupiter for C#, Clojure, and Julia. I wish the Clojure integration was as good as the others I listed. I’m to the point now where if anything other than venv/pip is required I won’t use it. Unfortunately there are many things that insist on conda.

Y_Y 3y ago
Conda is a fucking nightmare. Luckily you can usually just use pip to get the same packages.
- jt_b 3y ago
  [flagged]
- sterlinm 3y ago
  Why does “pip install gcc” keep failing then? ;) Conda isn’t perfect but takes on a lot of problems that pip doesn’t deal with at all. Regular conda is really slow these days but you can use mamba instead or just configure conda to use the libmamba solver and it’s much nicer. The folks at prefix.dev seem to be building some pretty cool drop in replacements for conda too.
- garbagecoder 3y ago
  One that you can’t is the Clojure kernel for Jupiter. I installed it from source. That’s no big deal, but they aren’t 1-to-1 with packages.

awinter-py 3y ago

had a ton of trouble getting jupyterlab to target an arbitrary virtualenv

nl 3y ago
The trick is that you have to deactivate the virtual environment and then resource it after adding Jupyter to that virtual environment. Most shells cache executable paths, so the path for jupyter will be the global path, not the one for your virtual environment. This is unfortunately not at all obvious and leads to very hard to track down bugs that seem to disappear and reappear if you aren't familiar with the issue. I have a recipe here which always works: https://github.com/nlothian/m1_huggingface_diffusers_demo#setup https://github.com/nlothian/m1_huggingface_diffusers_demo#se... If you don't have requirements.txt then do this: `pip3 install jupyter` for that line, then `deactivate` and `source ./venv/bin/activate`.
- nerdponx 3y ago
  I almost never rely on activating environments anymore. If it's in my project, I refer to it by its path relative to the project top level. If it's meant to be "global", I use Pipx, and if for some reason I can't, I use a script in ~/bin that uses the absolute path.

mauforonda 3y ago

Right. I usually `pipx install jupyterlab` and register each project's env with `python -m ipykernel install --user --name MyProject --display-name "My Project"` as per the ipython docs: https://ipython.readthedocs.io/en/stable/install/kernel_install.html#kernels-for-different-environments https://ipython.readthedocs.io/en/stable/install/kernel_inst...

rahimnathwani 3y ago

Yeah this is the right way. But I don't think it's obvious from the getting started docs. I bet a lot of people end up installing jupyter into each virtual environment instead. This sort of works, as (I think) most people are only working on one or two notebooks at a time, and aren't using notebooks that relate to more than one virtual environment.

nerdponx 3y ago

I do this, but it isn't well supported. Most Jupyterlab extensions expect you to `pip install` and that will handle both the frontend and backend components. The docs make basically no mention of the frontend/backend split. I end up having to install a bunch of frontend stuff in my project envs just to make sure my notebooks work.