14 ms·

I don't write Python code for my work. Last weekend I came across an interesting Jupiter notebook and figured I'd give it a try on my work laptop. "It's probabl

by pavlov 6y ago

I don't write Python code for my work. Last weekend I came across an interesting Jupiter notebook and figured I'd give it a try on my work laptop. "It's probably as easy as brew install pip and then use that to load the other dependencies," I assumed. Over an hour later I had to give up. There was initially some kind of Python version conflict on my Mac. Eventually some version of JupyterLab was installed somewhere, but it couldn't find any dependencies for the notebook. As a complete newbie to the Python ecosystem, I googled for instructions and found various environment managers, which then failed possibly because something is incompatible with Big Sur. More googling revealed instructions with complex CFLAGS environment setups to fix it, which didn't — and at that point it was very far from the supposed convenience of scripting languages anyway. I don't think it's Python's fault. Probably all the programming toolchains are this hard to a newbie! But it was a humbling experience after 34 years of programming, not being able to load a piece of sample code in a Sunday afternoon.

yshvrdhn 6y ago
can look into google colab to directly load the notebook. Link : https://colab.research.google.com/ https://colab.research.google.com/
turtlebits 6y ago
Yeah I've had this issue in the past with Jupyter. Now I generally look for docker images when I want to check out something new.
paozac 6y ago
True. I like Python, but the ecosystem can be very confusing for a quick dive. It's easy to get lost between pyenv, virtualenv, pipenv, pip, pip3, easy_install and friends.
- fwip 6y ago
  Very true - further evidenced by the five or six different approaches your sibling comments are recommending.
antpls 6y ago
Not sure about Mac, but on Linux, you install some docker images and it just works (modulo some docker flags to learn if you don't already know docker)
typomatic 6y ago
I have used Python on OSX for years and it is and always will be a horrorshow. Using the system Python installation is a nonstarter for many reasons, chief among them is that I don't have any interest in using py2. So then you're using pyenv or homebrew, but your vim install still thinks that it should be using the system python. And whoops, you fixed that and now virtualenv is not finding your interpreter. And etc., etc. OSX and its tooling are just ridiculous. I have no idea to this day how macs became the premier development environment. As always xkcd has a comic for it: https://xkcd.com/1987/ https://xkcd.com/1987/
- jackson1442 6y ago
  > I have no idea to this day how macs became the premier development environment. I've been editing a tutorial one of my coworkers wrote that targets new Python users on Windows. From my findings, the grass is not greener. Granted, geospatial Python is somewhat of a mess, but a lot of tools I have to use are somewhat messy forks of Unix tools (looking at you, pyenv-win) with tons of incompatible extensions. For development, Windows is the exception because you can transfer just about anything from Linux to macOS. On my Mac, I can easily install all of the Python packages I need without needing to install Visual Studio, pipwin, anaconda, etc. I have bash/zsh as my default system shell. Maybe it's easier to native Windows users, but bash/brew is a much better combination than anything I've found in Windows. WSL is a step in the right direction, but it still feels secondary. If a first-class terminal experience existed on Windows, I have a strong feeling that it could be the premier development environment, or at least closer to Mac. Maybe I'm using it wrong, but it definitely hasn't been made clear on how to use it right.
  - analog31 6y ago
    >>>> I've been editing a tutorial one of my coworkers wrote that targets new Python users on Windows. From my findings, the grass is not greener. I've been using WinPython for a few years. It's the closest thing to "just works" that I've found, and non-programming colleagues have been able to install it successfully. Because it works almost like an isolated "container," it's also possible to test your code on a clean install of WinPython to make sure it will run on someone else's computer before you share it. I don't know the technical difference between WinPython and a true container, but you can have multiple WinPython installs on one machine, and they don't interfere with one another, or with a pre-existing mainstream Python installation on the same PC. So you can share your stuff without worrying about screwing up someone else's stuff.
fastball 6y ago
Relevant XKCD. https://xkcd.com/1987/ https://xkcd.com/1987/
bitanarch 6y ago
You should use Anaconda instead of pip to install data science related tools in Python. ``` $ conda create -n {new_environment_name} python $ conda activate {new environment name} $ conda install jupyterlab ```
- modriano 6y ago
  You also have to manually add the kernel for that environment via: ({new_environment_name}) ...$ python -m ipykernel install --user --name {new_environment_name} --display-name "Python ({new environment display name})" The env won't be accessible in jupyter (lab or notebook) until you do this step. [0] https://stackoverflow.com/questions/39604271/conda-environments-not-showing-up-in-jupyter-notebook https://stackoverflow.com/questions/39604271/conda-environme...
  - tmvphil 6y ago
    Nope, what he wrote works just fine at least for conda. That advice is only if you want to be able to choose environments as kernels in the frontend. If you just run jupyter lab/notebook in a conda env which has it installed, the default kernel will be in the activated environment.
- davidf18 6y ago
  You can download anaconda in either the command line or graphical user interface versions selecting the versions of Python you wish to install. Then according to the blog posting for jupyterlab 3.0 execute the following command: conda install -c conda-forge jupyterlab=3
- p1esk 6y ago
  conda install -c conda-forge jupyterlab=3
- eggie5 6y ago
  why?
- auxym 6y ago
  Note that you now have to pay for a commercial anaconda license if you're using it for anything that isn't education or hobbyist stuff. https://www.anaconda.com/blog/anaconda-commercial-edition-faq https://www.anaconda.com/blog/anaconda-commercial-edition-fa...
syntaxing 6y ago
Always try using a virtual env. Also try using pip directly through ‘python3 -m pip install jupyter’ . Same to run it to bypass path issues ‘python3 -m jupyter notebook’. You should be up and running in a couple mins!
- pavlov 6y ago
  These were among the suggestions that I tried. I think the virtual env failed because of Big Sur, but not sure.
  - syntaxing 6y ago
    Hmm that’s super weird, M1 or Intel CPU? Also where does ‘which python3’ point to?
WrtCdEvrydy 6y ago
Yeah, I use portainer and then load a docker image of Jupiter and import the notebook there. Dependency management hell isn't just a Javascript problem these days.
unixhero 6y ago
Just install this: https://www.anaconda.com/products/individual https://www.anaconda.com/products/individual
flaviocopes 6y ago
Not sure about the Jupyter-specific issues but macOS comes with Python 2 installed, so you have to install Python separately and use the `python3` binary, or alias `python` to the `python3` binary. Then remember to run pip using `python3 -m pip <command>` and use virtual environments (venv) I stumbled on this a couple weeks ago and documented the process here https://flaviocopes.com/python-installation-macos/ https://flaviocopes.com/python-installation-macos/ I found the same problem with Ruby (old version preinstalled on macOS)
vmchale 6y ago
> I don't think it's Python's fault. Probably all the programming toolchains are this hard to a newbie! Python is a little worse in that in pushes its dirtiness downstream to your users too.
deleted 6y ago
[deleted]
baldfat 6y ago
Python is AMAZING and Awesome TILL You want to spread what you are doing to more then one computer. Though I have to say I love domain specific languages and personal enjoy Racket as my most fun language to use. It is sad that this has gotten so much worse over the years.
- philsnow 6y ago
  re:re:re: A: it works on my computer B: then we'll ship your computer and that's how docker was born
- auchtopus 6y ago
  What do you enjoy about racket? I used it for a semester in college, and it was next to unbearable. I can appreciate it as a fun little functional programming gimmick, but nothing more than that.
  - baldfat 6y ago
    I just love Lisp and you pretty much have a fun small language you can fit in my small brain with Racket. The biggest issue is the libraries. Everyone writes their own specific libraries for their problems, because its easy and fun. I could have done it in Python but portability was always the issue. At my former job I had to make charts from pictures with English and Spanish lettering. Took me 30 minutes to have a working client that got installed in a dozen computers that afternoon. I needed to also re-structure the companies file system for every computer in our company. Personally I worked in audio video contract work from time to time and I was able to make a dozen short cut programs that actually saved my biscuit several times. I could have done it in python but the portability kicked my butt. Racket made executables super easy. Also I went through How to Design Programs and I learned a ton.
zelphirkalt 6y ago
In a better world, the notebook you downloaded should have come with a list of dependencies and have reproducible code in it. That said, Python environment tools and package managers are still in a state of flux. Some tools make it easier, but are not helping much with reproducibility (plain venv + pip, but only if you got all required system packages installed!), other tools enable reproducibility, but have growing pains and are more difficult to use or get working (pipenv, poetry). If you are in luck, your required libraries and appropriate versions are available as GNU Guix packages and you can use that.
- philsnow 6y ago
  > Python environment tools and package managers are still in a state of flux They've pretty much always been a moving target, haven't they? Nothing has ever been good enough that the community agrees "okay, this is it, we can stop churning on this tooling for a while".
  - zelphirkalt 6y ago
    Probably because the typical culprits are really not good enough. I mean, follow the package manager repos on Github (pipenv for example) for a while. They have loads of issues, which one would think are unthinkable for a package manager to have. There are regressions and sometimes environments that were created just fine in an older version, is not creatable any longer, even though all the specification is, is a bunch of package names and version numbers, which do actually exist. Probably testing package managers is not a simple thing to do.
- tgbugs 6y ago
  I talked to Fernando about this at Neuroinformatics 2018. In principle generating dependencies from a notebook should be as easy as parsing the ast, finding all the import statements, and then using sys to lookup what package provided that module. In practice I stopped using notebooks entirely due to many of the issues described by pavlov above, though in my case it was mostly because it was at a point in time where the Gentoo python ecosystem was having a rough time dealing with the 2/3 transition and there was no minimal amount of work that could be done to get the jupyter server up and running. As for the reason why this still hasn't happened? I'm guessing because it is because the people who care about reproducibility have found other ways to achieve it, and as a result have no need nor incentive to implement it as part of the notebook format. Also because, if someone is running notebooks at all they know enough to bootstrap the jupyter server, which is non-trivial, or they are using a managed environment (be it remote, conda, etc.) where they are blissfully unaware that all their code has massive implicit and undocumented context. In the mean time I have implemented the start of an equivalent system for org files because the implicit environment is much worse given normal Emacs workflows. I have encountered some significant issues and considerations which the notebook ecosystem would have to deal with, and which the default python packaging ecosystem (or any ghettoized language specific packaging solution for that matter) is not equipped to handle. The primary issues is that only language agnostic package managers can actually bootstrap the environment that is required. What do you do if someone doesn't have a toolchain to build numpy and they are on some strange arch where there are no wheels? Do you just fail? The 80% solution here is the one we have with conda or managed solutions. The 90% solution requires that all the language communities stop pretending that they are the only language in the world and stop wasting time on their bespoke package manager.
deepsun 6y ago
No problems on Linux. I use pyenv for managing python versions and virtualenvs.
- cwyers 6y ago
  That doesn't help you install the dependencies for a notebook, if you don't know what they are.
kristjansson 6y ago
At the risk of contributing my own probably-subtly-broken advice to the collection already present in this thread, I would note that there is a relatively sane system python3 on Catalina/Big Sur, and most projects are building and distributing compatible wheels [1] (binary packages). So something like: /usr/bin/python3 -m venv ./venv # create a virtual environment in ./venv ./venv/bin/pip install wheel # optional ./venv/bin/pip install dep1 dep2 dep3 ... ./venv/bin/python ... # up and running is enough to get started in most cases. Of course, all bets are off for GPU or Apple Silicon support... [1]: https://pythonwheels.com/ https://pythonwheels.com/
- deleted 6y ago
  [deleted]
coldtea 6y ago
>I don't think it's Python's fault. Actually it is: https://xkcd.com/1987/ https://xkcd.com/1987/ That said, you would be up and running if you used conda. And this is not the usual "just try distro X" Linux advocate BS when somebody has an issue with distro Y. Conda does makes thing more user friendly by focusing on whole environments. Actually that's why it was built for. Note that this applies for this situation. For other uses cases, e.g. deploying something to production, having used conda would have made your life more difficult.
knuthsat 6y ago
virtualenv makes things a breeze
pm90 6y ago
Funny. I wanted to try this so I just ran the docker image and it worked. Time spent: 2 mins. docker run -p 8888:8888 jupyter/scipy-notebook
getpost 6y ago
I'm in the same boat. That's why I use repl.it[0] for nearly all my programming now. Haven't tried jupyter on repl.it, but here's[1] the first DDG result, which suggests jupyter is supported. [0] https://repl.it/ https://repl.it/ [1] https://repl.it/talk/templates/Jupyter-Notebook/49993 https://repl.it/talk/templates/Jupyter-Notebook/49993
jonathanyc 6y ago
I find that everything in Jupyter-land is as inconvenient to reproduce as it is convenient to experiment with. Case in point: last I checked, you still had to do gymnastics in order to even store your notebooks in Git in a meaningful way (e.g. allowing merges). It really throws me off the whole thing (Julia + Jupyter is even worse...) because I’m not keen on putting effort into stuff that nobody, myself included, will be able to use in a year or two.
- ChrisRackauckas 6y ago
  For Julia you might as well use Pluto (https://github.com/fonsp/Pluto.jl https://github.com/fonsp/Pluto.jl) or an IDE.
  - jonathanyc 6y ago
    Thanks, I’ll check those out! It’s hard to know what’s recommended by the community sometimes. I was using Julia + Jupyter through CoCalc which is great for collaborating but the non reproducibility aspect was starting to raise alarm bells for me.
ben509 6y ago
> I don't think it's Python's fault. Well, there's a long legacy of half-baked solutions and kludges. And, especially, install instructions consistently suggest you do the worst thing: install stuff using pip. Everyone knows that's bad advice, because it always fails in mysterious ways, and there's no good way to roll everything back. I recommend: python3 -m pip install pipx # Don't rely on the pip3 bin. pipx install jupyterlab # Now jupyter has an independent virtualenv. # To install dependencies: pipx inject jupyterlab numpy # Where is everything? pipx list venvs are in /Users/ben/.local/pipx/venvs apps are exposed on your $PATH at /Users/ben/.local/bin ... Once you've done that, never install anything directly with pip3 again. If you're working on a project, use pipx to install poetry / dephell / pipenv and use them to manage the virtualenv. And you can get rid of it all: pipx uninstall-all python3 -m pip uninstall pipx
- ImaCake 6y ago
  I use conda to do effectively the same thing. If my conda environment breaks and I can't roll it back, at least I can destroy it and start again. As an aside; it is really hard to explain why you should use anything but pip3 to people who only rarely need to interact with python.
  - ben509 6y ago
    Conda is pretty nice, and unless someone needs pypi, that's fair advice. And I agree, it's hard to explain not using pip3, so I just wouldn't. "For the best experience, pipx install neato-commando." And maybe link to the pipx website, in case someone actually clicks a link.
    - xapata 6y ago
      Conda works fine with Pip/PyPI.
- atg_abhishek 6y ago
  Yup this is a good way to go around it, similar to another comment below, I use conda to manage all the stuff
mynameisash 6y ago
My most recent professional project involves - among other things - a Python API deployed to AKS. I'd used Python here and there for fun and on the periphery of work, but this was the first time I had to dive into it full-on. While I know it has earned its praise for good reason -- as I'm sure all the data scientists I work with would attest to -- as an engineer, Python has been really frustrating to work with.
zsellera 6y ago
Don't even try. Use docker, there are official pre-built images. It's easy: docker run -p 8888:8888 -v ~/Code/actual-project/:/home/jovyan jupyter/datascience-notebook Note ~/Code/actual-project/ is a git repo or a mounted Google Drive folder. Moving from one machine to an other is pain-free. [1] https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html https://jupyter-docker-stacks.readthedocs.io/en/latest/using...
- fulafel 6y ago
  PSA: Don't use docker run -p xxx:xxx lightly. Even (or especially!) not in examples and documentation. It will listen to connections from anywhere in the internet and go out of its way to manipulate firewall rules on your machine to make a hole for it. Instead, use: docker run -p 127.0.0.1:8888:8888
loopz 6y ago
When following the docs, Go just works. It compiles fast, though not so dynamic as Ruby and Python. Tooling and runtime also matters in one's work.
specialist 6y ago
Had same experience. Hoping to help clear some backlog, I tried to help with some ML & data pipeline house cleaning. Had never used Python before, but how hard could it be? OMG. One really doesn't appreciate ease and simplicity of multiple simultaneous JDK installs, all peacefully coexisting, until working on some python and nodejs projects. Just running user space dev env via Docker was recently front paged here on HN. I believe that will become the norm. Being a macOS user, I looked to see if there was something more lightweight. Like maybe a port of podman. Lighter than running a virtual Linux instance. Again as a macOS user, I feel like I'm overdue to embrace the remote dev desktop set up. I've read that Googlers don't do any work locally; they all remote shell into servers running the tool stack. So very X Windows; what is old is new again.

by pavlov 6y ago

yshvrdhn 6y ago

can look into google colab to directly load the notebook. Link : https://colab.research.google.com/ https://colab.research.google.com/

turtlebits 6y ago

Yeah I've had this issue in the past with Jupyter. Now I generally look for docker images when I want to check out something new.

paozac 6y ago

True. I like Python, but the ecosystem can be very confusing for a quick dive. It's easy to get lost between pyenv, virtualenv, pipenv, pip, pip3, easy_install and friends.

fwip 6y ago
Very true - further evidenced by the five or six different approaches your sibling comments are recommending.

antpls 6y ago

Not sure about Mac, but on Linux, you install some docker images and it just works (modulo some docker flags to learn if you don't already know docker)

typomatic 6y ago

I have used Python on OSX for years and it is and always will be a horrorshow. Using the system Python installation is a nonstarter for many reasons, chief among them is that I don't have any interest in using py2. So then you're using pyenv or homebrew, but your vim install still thinks that it should be using the system python. And whoops, you fixed that and now virtualenv is not finding your interpreter. And etc., etc. OSX and its tooling are just ridiculous. I have no idea to this day how macs became the premier development environment. As always xkcd has a comic for it: https://xkcd.com/1987/ https://xkcd.com/1987/

jackson1442 6y ago
> I have no idea to this day how macs became the premier development environment. I've been editing a tutorial one of my coworkers wrote that targets new Python users on Windows. From my findings, the grass is not greener. Granted, geospatial Python is somewhat of a mess, but a lot of tools I have to use are somewhat messy forks of Unix tools (looking at you, pyenv-win) with tons of incompatible extensions. For development, Windows is the exception because you can transfer just about anything from Linux to macOS. On my Mac, I can easily install all of the Python packages I need without needing to install Visual Studio, pipwin, anaconda, etc. I have bash/zsh as my default system shell. Maybe it's easier to native Windows users, but bash/brew is a much better combination than anything I've found in Windows. WSL is a step in the right direction, but it still feels secondary. If a first-class terminal experience existed on Windows, I have a strong feeling that it could be the premier development environment, or at least closer to Mac. Maybe I'm using it wrong, but it definitely hasn't been made clear on how to use it right.
- analog31 6y ago
  >>>> I've been editing a tutorial one of my coworkers wrote that targets new Python users on Windows. From my findings, the grass is not greener. I've been using WinPython for a few years. It's the closest thing to "just works" that I've found, and non-programming colleagues have been able to install it successfully. Because it works almost like an isolated "container," it's also possible to test your code on a clean install of WinPython to make sure it will run on someone else's computer before you share it. I don't know the technical difference between WinPython and a true container, but you can have multiple WinPython installs on one machine, and they don't interfere with one another, or with a pre-existing mainstream Python installation on the same PC. So you can share your stuff without worrying about screwing up someone else's stuff.

fastball 6y ago

Relevant XKCD. https://xkcd.com/1987/ https://xkcd.com/1987/

bitanarch 6y ago

You should use Anaconda instead of pip to install data science related tools in Python. ``` $ conda create -n {new_environment_name} python $ conda activate {new environment name} $ conda install jupyterlab ```

modriano 6y ago
You also have to manually add the kernel for that environment via: ({new_environment_name}) ...$ python -m ipykernel install --user --name {new_environment_name} --display-name "Python ({new environment display name})" The env won't be accessible in jupyter (lab or notebook) until you do this step. [0] https://stackoverflow.com/questions/39604271/conda-environments-not-showing-up-in-jupyter-notebook https://stackoverflow.com/questions/39604271/conda-environme...
- tmvphil 6y ago
  Nope, what he wrote works just fine at least for conda. That advice is only if you want to be able to choose environments as kernels in the frontend. If you just run jupyter lab/notebook in a conda env which has it installed, the default kernel will be in the activated environment.
davidf18 6y ago
You can download anaconda in either the command line or graphical user interface versions selecting the versions of Python you wish to install. Then according to the blog posting for jupyterlab 3.0 execute the following command: conda install -c conda-forge jupyterlab=3
p1esk 6y ago
conda install -c conda-forge jupyterlab=3
eggie5 6y ago
why?
auxym 6y ago
Note that you now have to pay for a commercial anaconda license if you're using it for anything that isn't education or hobbyist stuff. https://www.anaconda.com/blog/anaconda-commercial-edition-faq https://www.anaconda.com/blog/anaconda-commercial-edition-fa...

syntaxing 6y ago

Always try using a virtual env. Also try using pip directly through ‘python3 -m pip install jupyter’ . Same to run it to bypass path issues ‘python3 -m jupyter notebook’. You should be up and running in a couple mins!

pavlov 6y ago
These were among the suggestions that I tried. I think the virtual env failed because of Big Sur, but not sure.
- syntaxing 6y ago
  Hmm that’s super weird, M1 or Intel CPU? Also where does ‘which python3’ point to?

WrtCdEvrydy 6y ago

Yeah, I use portainer and then load a docker image of Jupiter and import the notebook there. Dependency management hell isn't just a Javascript problem these days.

unixhero 6y ago

Just install this: https://www.anaconda.com/products/individual https://www.anaconda.com/products/individual

flaviocopes 6y ago

Not sure about the Jupyter-specific issues but macOS comes with Python 2 installed, so you have to install Python separately and use the `python3` binary, or alias `python` to the `python3` binary. Then remember to run pip using `python3 -m pip <command>` and use virtual environments (venv) I stumbled on this a couple weeks ago and documented the process here https://flaviocopes.com/python-installation-macos/ https://flaviocopes.com/python-installation-macos/ I found the same problem with Ruby (old version preinstalled on macOS)

vmchale 6y ago

> I don't think it's Python's fault. Probably all the programming toolchains are this hard to a newbie! Python is a little worse in that in pushes its dirtiness downstream to your users too.

deleted 6y ago

[deleted]

baldfat 6y ago

Python is AMAZING and Awesome TILL You want to spread what you are doing to more then one computer. Though I have to say I love domain specific languages and personal enjoy Racket as my most fun language to use. It is sad that this has gotten so much worse over the years.

philsnow 6y ago
re:re:re: A: it works on my computer B: then we'll ship your computer and that's how docker was born
auchtopus 6y ago
What do you enjoy about racket? I used it for a semester in college, and it was next to unbearable. I can appreciate it as a fun little functional programming gimmick, but nothing more than that.
- baldfat 6y ago
  I just love Lisp and you pretty much have a fun small language you can fit in my small brain with Racket. The biggest issue is the libraries. Everyone writes their own specific libraries for their problems, because its easy and fun. I could have done it in Python but portability was always the issue. At my former job I had to make charts from pictures with English and Spanish lettering. Took me 30 minutes to have a working client that got installed in a dozen computers that afternoon. I needed to also re-structure the companies file system for every computer in our company. Personally I worked in audio video contract work from time to time and I was able to make a dozen short cut programs that actually saved my biscuit several times. I could have done it in python but the portability kicked my butt. Racket made executables super easy. Also I went through How to Design Programs and I learned a ton.

zelphirkalt 6y ago

In a better world, the notebook you downloaded should have come with a list of dependencies and have reproducible code in it. That said, Python environment tools and package managers are still in a state of flux. Some tools make it easier, but are not helping much with reproducibility (plain venv + pip, but only if you got all required system packages installed!), other tools enable reproducibility, but have growing pains and are more difficult to use or get working (pipenv, poetry). If you are in luck, your required libraries and appropriate versions are available as GNU Guix packages and you can use that.

philsnow 6y ago
> Python environment tools and package managers are still in a state of flux They've pretty much always been a moving target, haven't they? Nothing has ever been good enough that the community agrees "okay, this is it, we can stop churning on this tooling for a while".
- zelphirkalt 6y ago
  Probably because the typical culprits are really not good enough. I mean, follow the package manager repos on Github (pipenv for example) for a while. They have loads of issues, which one would think are unthinkable for a package manager to have. There are regressions and sometimes environments that were created just fine in an older version, is not creatable any longer, even though all the specification is, is a bunch of package names and version numbers, which do actually exist. Probably testing package managers is not a simple thing to do.
tgbugs 6y ago
I talked to Fernando about this at Neuroinformatics 2018. In principle generating dependencies from a notebook should be as easy as parsing the ast, finding all the import statements, and then using sys to lookup what package provided that module. In practice I stopped using notebooks entirely due to many of the issues described by pavlov above, though in my case it was mostly because it was at a point in time where the Gentoo python ecosystem was having a rough time dealing with the 2/3 transition and there was no minimal amount of work that could be done to get the jupyter server up and running. As for the reason why this still hasn't happened? I'm guessing because it is because the people who care about reproducibility have found other ways to achieve it, and as a result have no need nor incentive to implement it as part of the notebook format. Also because, if someone is running notebooks at all they know enough to bootstrap the jupyter server, which is non-trivial, or they are using a managed environment (be it remote, conda, etc.) where they are blissfully unaware that all their code has massive implicit and undocumented context. In the mean time I have implemented the start of an equivalent system for org files because the implicit environment is much worse given normal Emacs workflows. I have encountered some significant issues and considerations which the notebook ecosystem would have to deal with, and which the default python packaging ecosystem (or any ghettoized language specific packaging solution for that matter) is not equipped to handle. The primary issues is that only language agnostic package managers can actually bootstrap the environment that is required. What do you do if someone doesn't have a toolchain to build numpy and they are on some strange arch where there are no wheels? Do you just fail? The 80% solution here is the one we have with conda or managed solutions. The 90% solution requires that all the language communities stop pretending that they are the only language in the world and stop wasting time on their bespoke package manager.

deepsun 6y ago

No problems on Linux. I use pyenv for managing python versions and virtualenvs.

cwyers 6y ago
That doesn't help you install the dependencies for a notebook, if you don't know what they are.

kristjansson 6y ago

At the risk of contributing my own probably-subtly-broken advice to the collection already present in this thread, I would note that there is a relatively sane system python3 on Catalina/Big Sur, and most projects are building and distributing compatible wheels [1] (binary packages). So something like: /usr/bin/python3 -m venv ./venv # create a virtual environment in ./venv ./venv/bin/pip install wheel # optional ./venv/bin/pip install dep1 dep2 dep3 ... ./venv/bin/python ... # up and running is enough to get started in most cases. Of course, all bets are off for GPU or Apple Silicon support... [1]: https://pythonwheels.com/ https://pythonwheels.com/

deleted 6y ago
[deleted]

coldtea 6y ago

>I don't think it's Python's fault. Actually it is: https://xkcd.com/1987/ https://xkcd.com/1987/ That said, you would be up and running if you used conda. And this is not the usual "just try distro X" Linux advocate BS when somebody has an issue with distro Y. Conda does makes thing more user friendly by focusing on whole environments. Actually that's why it was built for. Note that this applies for this situation. For other uses cases, e.g. deploying something to production, having used conda would have made your life more difficult.

knuthsat 6y ago

virtualenv makes things a breeze

pm90 6y ago

Funny. I wanted to try this so I just ran the docker image and it worked. Time spent: 2 mins. docker run -p 8888:8888 jupyter/scipy-notebook

getpost 6y ago

I'm in the same boat. That's why I use repl.it[0] for nearly all my programming now. Haven't tried jupyter on repl.it, but here's[1] the first DDG result, which suggests jupyter is supported. [0] https://repl.it/ https://repl.it/ [1] https://repl.it/talk/templates/Jupyter-Notebook/49993 https://repl.it/talk/templates/Jupyter-Notebook/49993

jonathanyc 6y ago

I find that everything in Jupyter-land is as inconvenient to reproduce as it is convenient to experiment with. Case in point: last I checked, you still had to do gymnastics in order to even store your notebooks in Git in a meaningful way (e.g. allowing merges). It really throws me off the whole thing (Julia + Jupyter is even worse...) because I’m not keen on putting effort into stuff that nobody, myself included, will be able to use in a year or two.

ChrisRackauckas 6y ago
For Julia you might as well use Pluto (https://github.com/fonsp/Pluto.jl https://github.com/fonsp/Pluto.jl) or an IDE.
- jonathanyc 6y ago
  Thanks, I’ll check those out! It’s hard to know what’s recommended by the community sometimes. I was using Julia + Jupyter through CoCalc which is great for collaborating but the non reproducibility aspect was starting to raise alarm bells for me.

ben509 6y ago

> I don't think it's Python's fault. Well, there's a long legacy of half-baked solutions and kludges. And, especially, install instructions consistently suggest you do the worst thing: install stuff using pip. Everyone knows that's bad advice, because it always fails in mysterious ways, and there's no good way to roll everything back. I recommend: python3 -m pip install pipx # Don't rely on the pip3 bin. pipx install jupyterlab # Now jupyter has an independent virtualenv. # To install dependencies: pipx inject jupyterlab numpy # Where is everything? pipx list venvs are in /Users/ben/.local/pipx/venvs apps are exposed on your $PATH at /Users/ben/.local/bin ... Once you've done that, never install anything directly with pip3 again. If you're working on a project, use pipx to install poetry / dephell / pipenv and use them to manage the virtualenv. And you can get rid of it all: pipx uninstall-all python3 -m pip uninstall pipx

ImaCake 6y ago
I use conda to do effectively the same thing. If my conda environment breaks and I can't roll it back, at least I can destroy it and start again. As an aside; it is really hard to explain why you should use anything but pip3 to people who only rarely need to interact with python.
- ben509 6y ago
  Conda is pretty nice, and unless someone needs pypi, that's fair advice. And I agree, it's hard to explain not using pip3, so I just wouldn't. "For the best experience, pipx install neato-commando." And maybe link to the pipx website, in case someone actually clicks a link.
  - xapata 6y ago
    Conda works fine with Pip/PyPI.
atg_abhishek 6y ago
Yup this is a good way to go around it, similar to another comment below, I use conda to manage all the stuff

mynameisash 6y ago

My most recent professional project involves - among other things - a Python API deployed to AKS. I'd used Python here and there for fun and on the periphery of work, but this was the first time I had to dive into it full-on. While I know it has earned its praise for good reason -- as I'm sure all the data scientists I work with would attest to -- as an engineer, Python has been really frustrating to work with.

zsellera 6y ago

Don't even try. Use docker, there are official pre-built images. It's easy: docker run -p 8888:8888 -v ~/Code/actual-project/:/home/jovyan jupyter/datascience-notebook Note ~/Code/actual-project/ is a git repo or a mounted Google Drive folder. Moving from one machine to an other is pain-free. [1] https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html https://jupyter-docker-stacks.readthedocs.io/en/latest/using...

fulafel 6y ago
PSA: Don't use docker run -p xxx:xxx lightly. Even (or especially!) not in examples and documentation. It will listen to connections from anywhere in the internet and go out of its way to manipulate firewall rules on your machine to make a hole for it. Instead, use: docker run -p 127.0.0.1:8888:8888

loopz 6y ago

When following the docs, Go just works. It compiles fast, though not so dynamic as Ruby and Python. Tooling and runtime also matters in one's work.

specialist 6y ago

Had same experience. Hoping to help clear some backlog, I tried to help with some ML & data pipeline house cleaning. Had never used Python before, but how hard could it be? OMG. One really doesn't appreciate ease and simplicity of multiple simultaneous JDK installs, all peacefully coexisting, until working on some python and nodejs projects. Just running user space dev env via Docker was recently front paged here on HN. I believe that will become the norm. Being a macOS user, I looked to see if there was something more lightweight. Like maybe a port of podman. Lighter than running a virtual Linux instance. Again as a macOS user, I feel like I'm overdue to embrace the remote dev desktop set up. I've read that Googlers don't do any work locally; they all remote shell into servers running the tool stack. So very X Windows; what is old is new again.