5 ms·

Hmm, what is mamba? I thought I was at least sort of up to date with the various python dependency managers, but that one is new to me. Relatedly, I have a gra

Hmm, what is mamba? I thought I was at least sort of up to date with the various python dependency managers, but that one is new to me. Relatedly, I have a grand vision of having as part of my development environment a jupyter notebook always at hand, in which to explore data as necessary, whip up scripts, accumulate little helper functions, etc. Basically, any time I have that "hm, I wonder..." itch, I'd like to be able to quickly whip up a solution in my notebook. I'm a skilled developer in terms of larger systems, but have a weakness when it comes to very early stage "throwaway" scripts to answer ad-hoc questions. I spent a few days trying to set up jupyterlab in an "ideal" way, so that I could have easy access to python libraries within my notebook, and also a reproducible environment since I plan to push my working directory to GitHub and would like to sync across different machines. I got confused by virtualenv vs venv, and tried conda to install libraries, but ran into various problems. Oh, and part of it was trying to have nbdev (from the fastai folks) as part of this toolkit. Anyone have a setup like this that they use and want to share? Part of me wants to just throw in the towel and learn R and RStudio, since I've heard its ggplot is the best plotting library anyway.

nxpnsv 6y ago
Mamba is a reimplementation of the conda package manager in C++. (quote from README at https://github.com/mamba-org/mamba https://github.com/mamba-org/mamba, linked to in original post)
- vlgnat 6y ago
  Funny, Python is too slow for a package manager and most people who can write fast C extensions have left the scene. So C++ is the natural choice.
  - roseway4 6y ago
    > most people who can write fast C extensions have left the scene Care to elaborate?
  - remram 6y ago
    Not sure if I agree with you but I was very surprised to find that their Python kernel is also implemented in C++: https://github.com/jupyter-xeus/xeus https://github.com/jupyter-xeus/xeus
klelatti 6y ago
Have you looked at any of the Jupyter Docker Stacks images? [1] I've built some personal Docker images based on these with quite a lot of additional features installed including Ruby, OpenCL and lots of additional Python and R packages and JupyterLab extensions. It's been a bit hit and miss - with extension incompatibilities being a bit of an issue. I sync directories on the host machine with the Docker image and so have the benefit of editing in a full text editor (for non-notebook scripts). I'm not a Python env expert so I've probably missed a few tricks but all seems to be working well now. Happy to help if you want to try this route and have any issues. Btw after using JupyterLab for over a year now I'm a huge fan. [1] https://jupyter-docker-stacks.readthedocs.io/en/latest/ https://jupyter-docker-stacks.readthedocs.io/en/latest/
iamlucaswolf 6y ago
I use a setup like this. Essentially, I use pyenv [1] to manage Python versions and Poetry [2] [2](https://python-poetry.org/ https://python-poetry.org/) for virtualenvs/dependencies. The workflow for creating a new project looks like this: 1. Create a project directory (e.g. 'myproject') and `cd` into it. 2. `git init` 3. Fixate the Python version for that project with the `pyenv local` command (e.g. `pyenv local 3.8.6`). This creates a `.python-version` file that you can put under source control. Within the `myproject` directory tree, `python` will now be automatically resolved to the specified version. Your system Python (in fact, any other Python versions you might have installed) remain untouched. 4. Create a new poetry project (`poetry init`). This creates a `pyproject.toml` which contains project metadata + dependencies and can also be checked into git. 5. Add dependencies with `poetry add`. Here, you could for instance add Jupyter Lab (`poetry add jupyterlab`). To access installed dependencies, such as the `jupyter lab` command, you can either execute one command in the virtualenv directly (`poetry run jupyter lab`) or spawn a shell (`poetry shell`). If you open a Jupyter Notebook that way, the packages installed in the virtualenv are directly available from within Jupyter Notebooks, without having to mess around with installing IPython kernels. I like this approach, because it gives you full flexibility, while being portable and easy to use. It gets you around having to deal with conda (which I found to be frustrating at times). Also, you're not tied to the Jupyter frontends, but could e.g. just install `ipykernel` and open notebooks in VSCode. [1](https://github.com/pyenv/pyenv/ https://github.com/pyenv/pyenv/) [2](https://python-poetry.org/ https://python-poetry.org/) Edit: Moved the links
- claytonjy 6y ago
  I do very similarly, except I avoid installing jupyter lab for each project, instead installing `ipykernel` as a dev dependency. I install jupyter lab system-wide with pipx [1], and for each project issue a command like pipenv run python -m ipykernel install --user --name=this_directory Then if I open Jupyter Lab I see "this_directory" as a listed kernel to create a notebook from. This allows me to manage Jupyter settings and plugins in one place rather than in each env, have multiple project's notebooks open in the same Jupyter Lab instance, etc. [1]https://pipxproject.github.io/pipx/ https://pipxproject.github.io/pipx/
liuliu 6y ago
I have an unconventional setup of Jupyterlab, Python dependencies, Swift (through PythonKit package) with Bazel. Surprisingly, the new `rules_python` and `pip_install` support works great at installing packages through pip. This ensures on any machine, I will have a consistent Python runtime (either downloaded or build from the source), as well as the pinned python dependencies when the repo checked out. It also helps because I have a Swift kernel that packaged inside the repo as well. One thing I haven't figured out is about Jupyterlab's extension system, which previously requires node.js for delivery. It seems 3.0 removed that requirement, so I am hopeful with some tuning I can deliver the plugins in consistent way as well.
- zhengyi13 6y ago
  Neat! I don't suppose you've got your BUILD and WORKSPACE up somewhere you wouldn't mind sharing?
abdullahkhalids 6y ago
On linux, I install Anaconda and just launch JupyterLab from Anaconda Navigator. But you could just install conda and just `conda env export -n base` to export your environment.
BadInformatics 6y ago
mamba is (nominally) a drop-in replacement for Conda's CLI and dependency resolver. I've been using it for a couple of months and it really does run an order of magnitude faster for anything that requires dependency resolution. You'll still want to keep ana/miniconda around, but it goes to show how some of Conda's performance woes are entirely self-inflicted.

nxpnsv 6y ago

Mamba is a reimplementation of the conda package manager in C++. (quote from README at https://github.com/mamba-org/mamba https://github.com/mamba-org/mamba, linked to in original post)

vlgnat 6y ago
Funny, Python is too slow for a package manager and most people who can write fast C extensions have left the scene. So C++ is the natural choice.
- roseway4 6y ago
  > most people who can write fast C extensions have left the scene Care to elaborate?
- remram 6y ago
  Not sure if I agree with you but I was very surprised to find that their Python kernel is also implemented in C++: https://github.com/jupyter-xeus/xeus https://github.com/jupyter-xeus/xeus

klelatti 6y ago

Have you looked at any of the Jupyter Docker Stacks images? [1] I've built some personal Docker images based on these with quite a lot of additional features installed including Ruby, OpenCL and lots of additional Python and R packages and JupyterLab extensions. It's been a bit hit and miss - with extension incompatibilities being a bit of an issue. I sync directories on the host machine with the Docker image and so have the benefit of editing in a full text editor (for non-notebook scripts). I'm not a Python env expert so I've probably missed a few tricks but all seems to be working well now. Happy to help if you want to try this route and have any issues. Btw after using JupyterLab for over a year now I'm a huge fan. [1] https://jupyter-docker-stacks.readthedocs.io/en/latest/ https://jupyter-docker-stacks.readthedocs.io/en/latest/

iamlucaswolf 6y ago

I use a setup like this. Essentially, I use pyenv [1] to manage Python versions and Poetry [2] [2](https://python-poetry.org/ https://python-poetry.org/) for virtualenvs/dependencies. The workflow for creating a new project looks like this: 1. Create a project directory (e.g. 'myproject') and `cd` into it. 2. `git init` 3. Fixate the Python version for that project with the `pyenv local` command (e.g. `pyenv local 3.8.6`). This creates a `.python-version` file that you can put under source control. Within the `myproject` directory tree, `python` will now be automatically resolved to the specified version. Your system Python (in fact, any other Python versions you might have installed) remain untouched. 4. Create a new poetry project (`poetry init`). This creates a `pyproject.toml` which contains project metadata + dependencies and can also be checked into git. 5. Add dependencies with `poetry add`. Here, you could for instance add Jupyter Lab (`poetry add jupyterlab`). To access installed dependencies, such as the `jupyter lab` command, you can either execute one command in the virtualenv directly (`poetry run jupyter lab`) or spawn a shell (`poetry shell`). If you open a Jupyter Notebook that way, the packages installed in the virtualenv are directly available from within Jupyter Notebooks, without having to mess around with installing IPython kernels. I like this approach, because it gives you full flexibility, while being portable and easy to use. It gets you around having to deal with conda (which I found to be frustrating at times). Also, you're not tied to the Jupyter frontends, but could e.g. just install `ipykernel` and open notebooks in VSCode. [1](https://github.com/pyenv/pyenv/ https://github.com/pyenv/pyenv/) [2](https://python-poetry.org/ https://python-poetry.org/) Edit: Moved the links

claytonjy 6y ago
I do very similarly, except I avoid installing jupyter lab for each project, instead installing `ipykernel` as a dev dependency. I install jupyter lab system-wide with pipx [1], and for each project issue a command like pipenv run python -m ipykernel install --user --name=this_directory Then if I open Jupyter Lab I see "this_directory" as a listed kernel to create a notebook from. This allows me to manage Jupyter settings and plugins in one place rather than in each env, have multiple project's notebooks open in the same Jupyter Lab instance, etc. [1]https://pipxproject.github.io/pipx/ https://pipxproject.github.io/pipx/

liuliu 6y ago

I have an unconventional setup of Jupyterlab, Python dependencies, Swift (through PythonKit package) with Bazel. Surprisingly, the new `rules_python` and `pip_install` support works great at installing packages through pip. This ensures on any machine, I will have a consistent Python runtime (either downloaded or build from the source), as well as the pinned python dependencies when the repo checked out. It also helps because I have a Swift kernel that packaged inside the repo as well. One thing I haven't figured out is about Jupyterlab's extension system, which previously requires node.js for delivery. It seems 3.0 removed that requirement, so I am hopeful with some tuning I can deliver the plugins in consistent way as well.

zhengyi13 6y ago
Neat! I don't suppose you've got your BUILD and WORKSPACE up somewhere you wouldn't mind sharing?

abdullahkhalids 6y ago

On linux, I install Anaconda and just launch JupyterLab from Anaconda Navigator. But you could just install conda and just `conda env export -n base` to export your environment.

BadInformatics 6y ago

mamba is (nominally) a drop-in replacement for Conda's CLI and dependency resolver. I've been using it for a couple of months and it really does run an order of magnitude faster for anything that requires dependency resolution. You'll still want to keep ana/miniconda around, but it goes to show how some of Conda's performance woes are entirely self-inflicted.