8 ms·

I keep experimenting with Jupyter in the context of telemetry/fault analysis and then hitting a wall with it where: - I get an analysis that I like, but there

I keep experimenting with Jupyter in the context of telemetry/fault analysis and then hitting a wall with it where: - I get an analysis that I like, but there isn't a good way to share it with others, so I end up just taking screenshots. - There isn't a good way to take the same analysis and plug new data into it, other than to copy-paste the entire notebook. - The process to "promote" fragments of a notebook into being reusable functions seemed very high-friction: basically you're rewriting it as a normal Python package and then adding that to Jupyter's environment. - There aren't good boundaries between Jupyter's own Python environment, and that of your notebooks— if you have a dependency which conflicts with one of Jupyter's dependencies, then good luck. Some or all of these may be wrong or out of date— Jupyter definitely passes the "oooh nifty" smell test, and I'd love to figure out how to make it usable for these longer-term workflows rather than just banging out one-off toys.

proto-n 3y ago
At least for the last point I use conda with the nb_conda/nb_conda_kernels packages, which makes you able to switch kernels to different environments.
bityard 3y ago
> There aren't good boundaries between Jupyter's own Python environment, and that of your notebooks— if you have a dependency which conflicts with one of Jupyter's dependencies, then good luck. It's cumbersome, and I'm not totally sure it's the correct way, but I remember getting around this by creating a virtualenv for my projects and then using that virtualenv's python as Jupyter's "kernel".
- anvuong 3y ago
  This is the way. I use miniconda to create env for Jupyter and install only Jupyter and its dependency on that, then I just config it to point to all the other Python envs I use for my different projects. This is only one-time setup and it is absolutely worth it given how messy the ecosystem is.
  - tetris11 3y ago
    > miniconda Try micromamba, it will shave years off dependency resolution
    - henrydark 3y ago
      I used to use micromamba, recently moved to mach-nix with nix. It's also really fast, but with the reproducibility of nix.
- modriano 3y ago
  Yeah, that's essentially the right way (I use conda envs, but venvs are probably fine too). Have one env for running the Jupyter Lab/Notebook server (install jupyter, jupyter widgets, nbextensions, etc into this env) and then create other envs for your projects and register that kernel with jupyter [0]. You can also just install all of the jupyter ide cruft in each env if you want, but dependencies can get annoying. [0] https://stackoverflow.com/questions/39604271/conda-environments-not-showing-up-in-jupyter-notebook#:~:text=Just%20run%20conda%20install%20ipykernel,to%20see%20newly%20added%20kernels https://stackoverflow.com/questions/39604271/conda-environme....
- garbagecoder 3y ago
  I just do a venv inside the venv so I can target several Python versions. I also use Jupiter for C#, Clojure, and Julia. I wish the Clojure integration was as good as the others I listed. I’m to the point now where if anything other than venv/pip is required I won’t use it. Unfortunately there are many things that insist on conda.
  - Y_Y 3y ago
    Conda is a fucking nightmare. Luckily you can usually just use pip to get the same packages.
    - jt_b 3y ago
      [flagged]
    - sterlinm 3y ago
      Why does “pip install gcc” keep failing then? ;) Conda isn’t perfect but takes on a lot of problems that pip doesn’t deal with at all. Regular conda is really slow these days but you can use mamba instead or just configure conda to use the libmamba solver and it’s much nicer. The folks at prefix.dev seem to be building some pretty cool drop in replacements for conda too.
    - garbagecoder 3y ago
      One that you can’t is the Clojure kernel for Jupiter. I installed it from source. That’s no big deal, but they aren’t 1-to-1 with packages.
- awinter-py 3y ago
  had a ton of trouble getting jupyterlab to target an arbitrary virtualenv
  - nl 3y ago
    The trick is that you have to deactivate the virtual environment and then resource it after adding Jupyter to that virtual environment. Most shells cache executable paths, so the path for jupyter will be the global path, not the one for your virtual environment. This is unfortunately not at all obvious and leads to very hard to track down bugs that seem to disappear and reappear if you aren't familiar with the issue. I have a recipe here which always works: https://github.com/nlothian/m1_huggingface_diffusers_demo#setup https://github.com/nlothian/m1_huggingface_diffusers_demo#se... If you don't have requirements.txt then do this: `pip3 install jupyter` for that line, then `deactivate` and `source ./venv/bin/activate`.
    - nerdponx 3y ago
      I almost never rely on activating environments anymore. If it's in my project, I refer to it by its path relative to the project top level. If it's meant to be "global", I use Pipx, and if for some reason I can't, I use a script in ~/bin that uses the absolute path.
- mauforonda 3y ago
  Right. I usually `pipx install jupyterlab` and register each project's env with `python -m ipykernel install --user --name MyProject --display-name "My Project"` as per the ipython docs: https://ipython.readthedocs.io/en/stable/install/kernel_install.html#kernels-for-different-environments https://ipython.readthedocs.io/en/stable/install/kernel_inst...
- rahimnathwani 3y ago
  Yeah this is the right way. But I don't think it's obvious from the getting started docs. I bet a lot of people end up installing jupyter into each virtual environment instead. This sort of works, as (I think) most people are only working on one or two notebooks at a time, and aren't using notebooks that relate to more than one virtual environment.
- nerdponx 3y ago
  I do this, but it isn't well supported. Most Jupyterlab extensions expect you to `pip install` and that will handle both the frontend and backend components. The docs make basically no mention of the frontend/backend split. I end up having to install a bunch of frontend stuff in my project envs just to make sure my notebooks work.
smith-kyle 3y ago
What do you use to collect telemetry data?
- mikepurvis 3y ago
  It's a mix of things, but the big one was ROS bag files for robots. So a lot of the conventional up-to-the-minute metrics workflows oriented around grafana/kibana don't really pan out when you're very interested in a five minute window that occurred three days ago, buried in a 1gb datafile.
nigma 3y ago
I’m using Jupyter Book (https://jupyterbook.org/ https://jupyterbook.org/) to build all kinds of daily reports and it works really well. Individual notebooks can also be converted to HTML or pdf with nbconvert using command line or “download as” menu item.
- cadr 3y ago
  +1 for Jupyter Book. I had to fight with it a bit for some things around LaTeX/PDF generation, but was very pleased with the result.
spywaregorilla 3y ago
> The process to "promote" fragments of a notebook into being reusable functions seemed very high-friction: basically you're rewriting it as a normal Python package and then adding that to Jupyter's environment. Don't get in that situation to begin with. Pop an `%load_ext` `%autoreload 2` at the top and just write functions in an imported .py file from the get go.
- jey 3y ago
  Ok, but let's also acknowledge that the mental model and levels of abstraction increased significantly, which presents a barrier to people without any prior software engineering experience, who are just getting into Python for "data science".
  - mstipetic 3y ago
    Well maybe we should expect people to learn the tools they use properly instead of constantly dumbing things down for the sake of convenience. I’m not sure you should be able to just pop in and do “data science”
  - spywaregorilla 3y ago
    I don't think it's a huge leap to say your functions are stored in this .py file vs. in a cell above. the magic commands simply allow you to reload the functions automatically without needing to reimport / reload the kernel.
    - fluidcruft 3y ago
      It is a huge gap when the notebook is documentation. Moving functions out either duplicates documentation or separates them. Notebooks are vastly superior for documentation. Extracting functions from notebooks into a py file isn't difficult (see nbdev for example)
      - spywaregorilla 3y ago
        Notebooks are good for demonstration, not documentation. They are bad for documentation because you need to put docstrings and comments in the code while desiring to put the same content in the markdown.
Twirrim 3y ago
> - I get an analysis that I like, but there isn't a good way to share it with others, so I end up just taking screenshots. Jupyter supports printing out as a PDF, an all-in-one-webpage and all sorts of other formats. I've used this and the support for Markdown to create a number of documents that I've presented up the leadership chain, or to other engineers across the org "Here's the data, here's the narrative, and here's the code that shows you how I got to it so you can reproduce". Can even share the Jupyter notebook itself.
- fluidcruft 3y ago
  Do you have a way to store data directly into the notebook? I do a bunch of device testing and I have master notebooks setup to analyze and condense raw data. I use papermill to evaluate the master notebook to generate a report. But I want to also store/attach intermediates (say a named numpy array) into the notebook for further analysis.
  - fn-mote 3y ago
    > I want to also store/attach intermediates (say a named numpy array) into the notebook for further analysis. Is the issue that you do not want to save the data and report into a folder and distribute that? That is, you want an entirely self-contained notebook? Or is there something else going on here? I'm sure that's possible but it seems kind of wrong to put your binary "data" in with your analysis and presentation-making code.
    - fluidcruft 3y ago
      Somewhat. I want it to be difficult to separate the data from the report. Basically I want the report itself to be ingestible as input to other steps, so it's more of a "documented data" with the analysis results available. The inputs are documented but the final results can be restored without reevaluating the entire notebook. I don't want the entire workspace saved, just the final results. I hoped there was some magic to inject and restore a python object from a cell using some form of introspection. The testing I do is annual equipment performance evaluation. I'd like to be able to process each test and then feed the results into longitudinal monitoring. One thing I am considering is adding a library or extension to papermill that automatically creates a workspace hdf5 or dill or whatever that I can store individual variables into. After studying the ipynb JSON it just seems odd that you can't just store blobs as attachments. But what I understand is it has to do with the kernels and notebooks running as separate processes and passing things around as notifications. So basically the kernels don't have any access to the cells or any sorts of introspection. With papermill you have parameters, there's just not any "return values" in the processed notebooks. If it existed you could treat "reports" more easily as cached function evaluations.
izzymiller 3y ago
I'm always hesitant to self-promote in an hn comment (actually I have never done it before!), but your problems w/ Jupyter are just too closely mapped to what Hex (https://hex.tech/ https://hex.tech/) solves to not plug it here! - I get an analysis that I like, but there isn't a good way to share it with others, so I end up just taking screenshots. You can publish any Hex notebook with literally just a few clicks, and anyone you share it with can access it, or edit it, or fork it, without installing anything— or you can even make it public. You can easily turn a notebook into an "app" or interactive report if you want, hiding/showing certain cells or choosing cells to show only code/only output. You can just share the raw notebook though too. - There isn't a good way to take the same analysis and plug new data into it, other than to copy-paste the entire notebook. Super easy to duplicate a Hex project and hit a different table or data source, or you can use input parameters (like ipywidgets) to make one notebook parameterized and work on a bunch of different data sources. - The process to "promote" fragments of a notebook into being reusable functions seemed very high-friction: basically you're rewriting it as a normal Python package and then adding that to Jupyter's environment. You can promote any part of a project to a "Component" (docs: https://learn.hex.tech/docs/develop-logic/components https://learn.hex.tech/docs/develop-logic/components) that you can import into other projects. They can be data sources, function definitions, anything. If you make upstream changes to the component, you can sync them down into projects that import it. - There aren't good boundaries between Jupyter's own Python environment, and that of your notebooks— if you have a dependency which conflicts with one of Jupyter's dependencies, then good luck. Hex has a ton of default packages in its already installed standard library, and all the dependencies are ironed out— if you have packages you want to use that aren't there, you can pip install them, pull them in from a private github repo, or ask us to add them to the base image. You can also run Hex projects using a custom-provided docker image if you have super custom needs. You should *definitely* check it out if you have these pain points. Here's an example of a pretty complicated public Hex project: https://app.hex.tech/hex-public/app/9b882bc1-ead3-4f0b-87d1-87763750cd2b/21/9e6dc5a8-abcc-451c-88b6-e66365384815 https://app.hex.tech/hex-public/app/9b882bc1-ead3-4f0b-87d1-... And here's a simpler one I just made the other day on a cool Silk Road dataset https://app.hex.tech/hex-public/app/cdc1b8fe-144b-4a74-a5ef-34c9f5418ae4/latest https://app.hex.tech/hex-public/app/cdc1b8fe-144b-4a74-a5ef-.... There's a bunch more examples at https://hex.tech/use-cases https://hex.tech/use-cases. Happy to answer any questions!
ptero 3y ago
For a while I viewed Jupyter as a toy that is neither here nor there (sitting between the chairs of development and explanation, briefing or visualization and not doing either job great). But about 2 years ago when I changed jobs into a "Jupyter heavy" environment I was forced to learn it and have grown to really like it. I primarily use Jupyter for prototyping: trying ideas, plotting results and sharing notebooks for others to improve on (or punch holes in). Once a piece clearly shows promise I move it into a module. While this usually means a significant rewrite I actually see it as a benefit: I can be messy in original prototyping plus a rewrite after experimentation often leads to a better code with a small time investment. Beyond the minimal self-discipline of actually moving code into modules I still have two three-character friction points: vim and git. Pointers on addressing those appreciated! I would love, love, love a tight vim and Jupyter integration to be able to switch, easily and frequently, between editing a set of cells in vim and in Jupyter with solid sync between them. I am perfectly OK with vim ignoring the output. And I would love to have a git mode that only checks in the changes that cause actual differences in the python code; not timestamps or the output.
- leksak 3y ago
  jupytext will solve the git problems for you. Pair with a percent script, ignore the ipynb files.
  - kzrdude 3y ago
    jupytext is great. It even allows you to use only .py files directly as notebooks, but I recommend "pair with ipynb" and version controlling the .py file. The ipynb acts like a cache of the cell outputs between invocations of jupyterlab, which is handy too.
    - slt2021 3y ago
      I always wanted a Jupyter like environment, but the one that would natively support and output .py files jupyter notebook as IDE, but with .py files instead of .ipynb
fluidcruft 3y ago
One thing that I keep running into myself is I want to include data in a notebook as a sort of report or record of an analysis. I really like papermill for creating notebooks that execute and then store results. But you can only include text output or plots. If I wanted to store a numpy array within the notebook for inspection or input into.a next step, there doesn't seem to be a way. I understand it could be difficult to figure out how to make the all of that work. But I don't know. Maybe pickling or something should work?
- kzrdude 3y ago
  Ploomber does this kind of proper jupyter notebook pipeline. I've used it, but not stuck with it (yet)
  - fluidcruft 3y ago
    I'll have to check it out. I have looked at it previously but it's somewhat overwhelming to figure out what it actually does. My sense previously was that is it was more like a dependency graph thing similar to a makefile but I'll give it a closer look.
Equiet 3y ago
We looked into many of these issues with Deepnote (YC S19) [https://deepnote.com/ https://deepnote.com/]. What we found is that these are not necessarily problems of the underlying medium (a notebook), but more of the specific implementation (Jupyter). We've seen a lot of progress in the Jupyter ecosystem, but unfortunately almost none in the areas you mentioned.
bjornasm 3y ago
>- There isn't a good way to take the same analysis and plug new data into it, other than to copy-paste the entire notebook. Cant you just set the parameters in the first cell? >- I get an analysis that I like, but there isn't a good way to share it with others, so I end up just taking screenshots. Export to md, pdf or html?
singhrac 3y ago
I manage a Jupyter instance where we do ~75% of all of our work, and before that used Jupyter daily as a quant. In response to your comments + sibling's: Sharing: there hasn't been a good solution, so you have to hack something together. For us (small team) symlinking a "shared_notebooks" network path works fine (but looking forward to the RTC mentioned in the post). It's very important to restart your notebook before sharing. New data: I've never really run into this in a form that "restart + run all cells" didn't really work. Are you trying to keep older versions? Reusability: I highly recommend installing a personal Python package in development mode to your kernel (i.e. pip install -e .). Then just move functions from the notebook to the package and reload. Boundaries: As others have said, I would just make a "user" kernel right away and never touch the Jupyter environment (as TLJH does by default). I am kind of hopeful that Armin Ronacher's rye fixes Python environment hell and we end up with better solutions for many of these things, but there are definitely some issues with version management in Jupyter.
snicker7 3y ago
Have you considered using Pluto notebooks instead? They are designed to be reproducible (including version references to transitive dependencies). They are also reactive, so it is trivial to change the dataset.
kzrdude 3y ago
Export to HTML is usually a decent report. If you want to refine it a bit, you can export and show only outputs, or (with some code) only certain tagged outputs. To plug in new data, I experiment with papermill. There are solutions that can put jupyter and the kernel in different environments. It's even not too hard to setup, but it astounds me it's not a common way to set it up. It's natural to have kernel + the analysis's deps in its own environment.
patrick451 3y ago
You may be interested in papermill to address the parametrized analysis problem [1]. I think (but I'm not positive) this is what the data team at a previous job used to automate running notebooks for all sorts nightly reports. [1] https://papermill.readthedocs.io/en/latest/# https://papermill.readthedocs.io/en/latest/#
agrama 3y ago
If you're looking for a way for others to reproduce your environments, you might find this useful: https://jupyenv.io/ https://jupyenv.io/
jph00 3y ago
To promote functions to a module use nbdev; execnb or papermill to parameterise your notebook; and nbconda to have environments for your notebooks.
dweinus 3y ago
https://github.com/Anaconda-Platform/nb_conda_kernels https://github.com/Anaconda-Platform/nb_conda_kernels
irjustin 3y ago
I've thought about this a lot in the context of our org. What you're describing is taking modeling -> production (minus the screen shot one). Typically a data engineer will be the one who helps bridge that gap, but that has a problem of does data engineer's output == scientist output, which can be time consuming to handle. To shrink the gap from dev->prod - we have 2 notebooks, one for model development and one model deployment in production. We use papermill[0] to execute directly notebooks in production. Shared functions between the dev/prod that are built by the scientist are put into a separate notebook and then imported via `run`. If I'm honest, our scientist don't do this and simply copy/paste the functions if we don't yell at them to fix it. This basically allows us to stay within the jupyter environment entirely so that dependancies are isolated. So, it's far from perfect, but it's allowed us to shrink the dev->prod life cycle time. Love to hear what others have done towards the same end.
sterlinm 3y ago
If you use conda there are extensions that can help with this by automatically registering any available conda environments that include ipykernel in your Jupyter Lab environment. nb_conda_kernels is pretty reliable but not actively maintained. Gator from the mamba folks is new and still a bit rough around the edges but looks like it will be pretty slick eventually. https://github.com/Anaconda-Platform/nb_conda_kernels https://github.com/Anaconda-Platform/nb_conda_kernels https://github.com/mamba-org/gator https://github.com/mamba-org/gator
__MatrixMan__ 3y ago
> There aren't good boundaries between Jupyter's own Python environment, and that of your notebooks— if you have a dependency which conflicts with one of Jupyter's dependencies, then good luck. I believe that you can use https://github.com/tweag/jupyenv https://github.com/tweag/jupyenv for this.
zelphirkalt 3y ago
Why not write the experimental code in the notebook, then write a function containing that code in the notebook and try it out, and only then move your function to a py file, if at all?
prometheon1 3y ago
> I get an analysis that I like, but there isn't a good way to share it with others, so I end up just taking screenshots. What would be your preferable way of sharing your analysis with others? - You can turn jupyter notebooks into pdfs directly in the jupyter UI. - You can upload them to Gitlab/Github and share the link to the rendered result. - You can upload them to Colab/Binder/Kaggle and let people play with the code themselves. - You can turn jupyter notebooks into beautiful websites/documents with: https://quarto.org/docs/tools/jupyter-lab.html https://quarto.org/docs/tools/jupyter-lab.html - You can add jupyter notebooks to you docs using nbsphinx: https://nbsphinx.readthedocs.io/en/0.9.1/ https://nbsphinx.readthedocs.io/en/0.9.1/ - You can turn jupyter notebooks into interactive web apps with voila: https://github.com/voila-dashboards/voila https://github.com/voila-dashboards/voila - You can turn jupyter notebooks into presentations with rise: https://github.com/voila-dashboards/voila https://github.com/voila-dashboards/voila
- henrikgs 3y ago
  I'm guessing the las one should link here https://rise.readthedocs.io/en/stable/ https://rise.readthedocs.io/en/stable/ and not to voila two times?
  - prometheon1 3y ago
    You're right, thanks!
- Desustorm 3y ago
  To add to this, shamelessly self-promoting, Notebooker (https://github.com/man-group/notebooker https://github.com/man-group/notebooker) is a neat way of scheduling your Jupyter notebooks as parametrisable reports whose results are presented in a little web GUI (either as static HTML, PDF, or as reveal.js slideshow renders)
- bippingchip 3y ago
  While I have no need for its online functionality and the SAAS part of plotly, I really do like plotly python + cufflinks [1]. It lets you make interactive plots in html/js format. Which means you can save the notebook as html, and while people won't be able to rerun the code, they can still zoom in on graphs, hover to see annotations etc, which is a really nice way to share the outcome of your work in a more accessible way. [1] https://github.com/santosjorge/cufflinks https://github.com/santosjorge/cufflinks
Camillo 3y ago
> - The process to "promote" fragments of a notebook into being reusable functions seemed very high-friction: basically you're rewriting it as a normal Python package and then adding that to Jupyter's environment. > - There aren't good boundaries between Jupyter's own Python environment, and that of your notebooks— if you have a dependency which conflicts with one of Jupyter's dependencies, then good luck. The best Jupyter UX for me now is VSCode. Just put an .ipynb file in your workspace and you get the notebook interface inside VSCode. Put `%load_ext autoreload` and `%autoreload 2` in the first cell, and use the same python environment you're using in your workspace for the Jupyter kernel. Then you can import libraries from your project, use them, and it's very easy to promote code from the notebook into a library. You can just cut a function from the notebook, paste it into a library, add an import, and rerun the subsequent cells to verify it still works as expected.

proto-n 3y ago

At least for the last point I use conda with the nb_conda/nb_conda_kernels packages, which makes you able to switch kernels to different environments.

bityard 3y ago

> There aren't good boundaries between Jupyter's own Python environment, and that of your notebooks— if you have a dependency which conflicts with one of Jupyter's dependencies, then good luck. It's cumbersome, and I'm not totally sure it's the correct way, but I remember getting around this by creating a virtualenv for my projects and then using that virtualenv's python as Jupyter's "kernel".

anvuong 3y ago
This is the way. I use miniconda to create env for Jupyter and install only Jupyter and its dependency on that, then I just config it to point to all the other Python envs I use for my different projects. This is only one-time setup and it is absolutely worth it given how messy the ecosystem is.
- tetris11 3y ago
  > miniconda Try micromamba, it will shave years off dependency resolution
  - henrydark 3y ago
    I used to use micromamba, recently moved to mach-nix with nix. It's also really fast, but with the reproducibility of nix.
modriano 3y ago
Yeah, that's essentially the right way (I use conda envs, but venvs are probably fine too). Have one env for running the Jupyter Lab/Notebook server (install jupyter, jupyter widgets, nbextensions, etc into this env) and then create other envs for your projects and register that kernel with jupyter [0]. You can also just install all of the jupyter ide cruft in each env if you want, but dependencies can get annoying. [0] https://stackoverflow.com/questions/39604271/conda-environments-not-showing-up-in-jupyter-notebook#:~:text=Just%20run%20conda%20install%20ipykernel,to%20see%20newly%20added%20kernels https://stackoverflow.com/questions/39604271/conda-environme....
garbagecoder 3y ago
I just do a venv inside the venv so I can target several Python versions. I also use Jupiter for C#, Clojure, and Julia. I wish the Clojure integration was as good as the others I listed. I’m to the point now where if anything other than venv/pip is required I won’t use it. Unfortunately there are many things that insist on conda.
- Y_Y 3y ago
  Conda is a fucking nightmare. Luckily you can usually just use pip to get the same packages.
  - jt_b 3y ago
    [flagged]
  - sterlinm 3y ago
    Why does “pip install gcc” keep failing then? ;) Conda isn’t perfect but takes on a lot of problems that pip doesn’t deal with at all. Regular conda is really slow these days but you can use mamba instead or just configure conda to use the libmamba solver and it’s much nicer. The folks at prefix.dev seem to be building some pretty cool drop in replacements for conda too.
  - garbagecoder 3y ago
    One that you can’t is the Clojure kernel for Jupiter. I installed it from source. That’s no big deal, but they aren’t 1-to-1 with packages.
awinter-py 3y ago
had a ton of trouble getting jupyterlab to target an arbitrary virtualenv
- nl 3y ago
  The trick is that you have to deactivate the virtual environment and then resource it after adding Jupyter to that virtual environment. Most shells cache executable paths, so the path for jupyter will be the global path, not the one for your virtual environment. This is unfortunately not at all obvious and leads to very hard to track down bugs that seem to disappear and reappear if you aren't familiar with the issue. I have a recipe here which always works: https://github.com/nlothian/m1_huggingface_diffusers_demo#setup https://github.com/nlothian/m1_huggingface_diffusers_demo#se... If you don't have requirements.txt then do this: `pip3 install jupyter` for that line, then `deactivate` and `source ./venv/bin/activate`.
  - nerdponx 3y ago
    I almost never rely on activating environments anymore. If it's in my project, I refer to it by its path relative to the project top level. If it's meant to be "global", I use Pipx, and if for some reason I can't, I use a script in ~/bin that uses the absolute path.
mauforonda 3y ago
Right. I usually `pipx install jupyterlab` and register each project's env with `python -m ipykernel install --user --name MyProject --display-name "My Project"` as per the ipython docs: https://ipython.readthedocs.io/en/stable/install/kernel_install.html#kernels-for-different-environments https://ipython.readthedocs.io/en/stable/install/kernel_inst...
rahimnathwani 3y ago
Yeah this is the right way. But I don't think it's obvious from the getting started docs. I bet a lot of people end up installing jupyter into each virtual environment instead. This sort of works, as (I think) most people are only working on one or two notebooks at a time, and aren't using notebooks that relate to more than one virtual environment.
nerdponx 3y ago
I do this, but it isn't well supported. Most Jupyterlab extensions expect you to `pip install` and that will handle both the frontend and backend components. The docs make basically no mention of the frontend/backend split. I end up having to install a bunch of frontend stuff in my project envs just to make sure my notebooks work.

smith-kyle 3y ago

What do you use to collect telemetry data?

mikepurvis 3y ago
It's a mix of things, but the big one was ROS bag files for robots. So a lot of the conventional up-to-the-minute metrics workflows oriented around grafana/kibana don't really pan out when you're very interested in a five minute window that occurred three days ago, buried in a 1gb datafile.

nigma 3y ago

I’m using Jupyter Book (https://jupyterbook.org/ https://jupyterbook.org/) to build all kinds of daily reports and it works really well. Individual notebooks can also be converted to HTML or pdf with nbconvert using command line or “download as” menu item.

cadr 3y ago
+1 for Jupyter Book. I had to fight with it a bit for some things around LaTeX/PDF generation, but was very pleased with the result.

spywaregorilla 3y ago

> The process to "promote" fragments of a notebook into being reusable functions seemed very high-friction: basically you're rewriting it as a normal Python package and then adding that to Jupyter's environment. Don't get in that situation to begin with. Pop an `%load_ext` `%autoreload 2` at the top and just write functions in an imported .py file from the get go.

jey 3y ago
Ok, but let's also acknowledge that the mental model and levels of abstraction increased significantly, which presents a barrier to people without any prior software engineering experience, who are just getting into Python for "data science".
- mstipetic 3y ago
  Well maybe we should expect people to learn the tools they use properly instead of constantly dumbing things down for the sake of convenience. I’m not sure you should be able to just pop in and do “data science”
- spywaregorilla 3y ago
  I don't think it's a huge leap to say your functions are stored in this .py file vs. in a cell above. the magic commands simply allow you to reload the functions automatically without needing to reimport / reload the kernel.
  - fluidcruft 3y ago
    It is a huge gap when the notebook is documentation. Moving functions out either duplicates documentation or separates them. Notebooks are vastly superior for documentation. Extracting functions from notebooks into a py file isn't difficult (see nbdev for example)
    - spywaregorilla 3y ago
      Notebooks are good for demonstration, not documentation. They are bad for documentation because you need to put docstrings and comments in the code while desiring to put the same content in the markdown.

Twirrim 3y ago

> - I get an analysis that I like, but there isn't a good way to share it with others, so I end up just taking screenshots. Jupyter supports printing out as a PDF, an all-in-one-webpage and all sorts of other formats. I've used this and the support for Markdown to create a number of documents that I've presented up the leadership chain, or to other engineers across the org "Here's the data, here's the narrative, and here's the code that shows you how I got to it so you can reproduce". Can even share the Jupyter notebook itself.

fluidcruft 3y ago
Do you have a way to store data directly into the notebook? I do a bunch of device testing and I have master notebooks setup to analyze and condense raw data. I use papermill to evaluate the master notebook to generate a report. But I want to also store/attach intermediates (say a named numpy array) into the notebook for further analysis.
- fn-mote 3y ago
  > I want to also store/attach intermediates (say a named numpy array) into the notebook for further analysis. Is the issue that you do not want to save the data and report into a folder and distribute that? That is, you want an entirely self-contained notebook? Or is there something else going on here? I'm sure that's possible but it seems kind of wrong to put your binary "data" in with your analysis and presentation-making code.
  - fluidcruft 3y ago
    Somewhat. I want it to be difficult to separate the data from the report. Basically I want the report itself to be ingestible as input to other steps, so it's more of a "documented data" with the analysis results available. The inputs are documented but the final results can be restored without reevaluating the entire notebook. I don't want the entire workspace saved, just the final results. I hoped there was some magic to inject and restore a python object from a cell using some form of introspection. The testing I do is annual equipment performance evaluation. I'd like to be able to process each test and then feed the results into longitudinal monitoring. One thing I am considering is adding a library or extension to papermill that automatically creates a workspace hdf5 or dill or whatever that I can store individual variables into. After studying the ipynb JSON it just seems odd that you can't just store blobs as attachments. But what I understand is it has to do with the kernels and notebooks running as separate processes and passing things around as notifications. So basically the kernels don't have any access to the cells or any sorts of introspection. With papermill you have parameters, there's just not any "return values" in the processed notebooks. If it existed you could treat "reports" more easily as cached function evaluations.

izzymiller 3y ago

I'm always hesitant to self-promote in an hn comment (actually I have never done it before!), but your problems w/ Jupyter are just too closely mapped to what Hex (https://hex.tech/ https://hex.tech/) solves to not plug it here! - I get an analysis that I like, but there isn't a good way to share it with others, so I end up just taking screenshots. You can publish any Hex notebook with literally just a few clicks, and anyone you share it with can access it, or edit it, or fork it, without installing anything— or you can even make it public. You can easily turn a notebook into an "app" or interactive report if you want, hiding/showing certain cells or choosing cells to show only code/only output. You can just share the raw notebook though too. - There isn't a good way to take the same analysis and plug new data into it, other than to copy-paste the entire notebook. Super easy to duplicate a Hex project and hit a different table or data source, or you can use input parameters (like ipywidgets) to make one notebook parameterized and work on a bunch of different data sources. - The process to "promote" fragments of a notebook into being reusable functions seemed very high-friction: basically you're rewriting it as a normal Python package and then adding that to Jupyter's environment. You can promote any part of a project to a "Component" (docs: https://learn.hex.tech/docs/develop-logic/components https://learn.hex.tech/docs/develop-logic/components) that you can import into other projects. They can be data sources, function definitions, anything. If you make upstream changes to the component, you can sync them down into projects that import it. - There aren't good boundaries between Jupyter's own Python environment, and that of your notebooks— if you have a dependency which conflicts with one of Jupyter's dependencies, then good luck. Hex has a ton of default packages in its already installed standard library, and all the dependencies are ironed out— if you have packages you want to use that aren't there, you can pip install them, pull them in from a private github repo, or ask us to add them to the base image. You can also run Hex projects using a custom-provided docker image if you have super custom needs. You should *definitely* check it out if you have these pain points. Here's an example of a pretty complicated public Hex project: https://app.hex.tech/hex-public/app/9b882bc1-ead3-4f0b-87d1-87763750cd2b/21/9e6dc5a8-abcc-451c-88b6-e66365384815 https://app.hex.tech/hex-public/app/9b882bc1-ead3-4f0b-87d1-... And here's a simpler one I just made the other day on a cool Silk Road dataset https://app.hex.tech/hex-public/app/cdc1b8fe-144b-4a74-a5ef-34c9f5418ae4/latest https://app.hex.tech/hex-public/app/cdc1b8fe-144b-4a74-a5ef-.... There's a bunch more examples at https://hex.tech/use-cases https://hex.tech/use-cases. Happy to answer any questions!

ptero 3y ago

For a while I viewed Jupyter as a toy that is neither here nor there (sitting between the chairs of development and explanation, briefing or visualization and not doing either job great). But about 2 years ago when I changed jobs into a "Jupyter heavy" environment I was forced to learn it and have grown to really like it. I primarily use Jupyter for prototyping: trying ideas, plotting results and sharing notebooks for others to improve on (or punch holes in). Once a piece clearly shows promise I move it into a module. While this usually means a significant rewrite I actually see it as a benefit: I can be messy in original prototyping plus a rewrite after experimentation often leads to a better code with a small time investment. Beyond the minimal self-discipline of actually moving code into modules I still have two three-character friction points: vim and git. Pointers on addressing those appreciated! I would love, love, love a tight vim and Jupyter integration to be able to switch, easily and frequently, between editing a set of cells in vim and in Jupyter with solid sync between them. I am perfectly OK with vim ignoring the output. And I would love to have a git mode that only checks in the changes that cause actual differences in the python code; not timestamps or the output.

leksak 3y ago
jupytext will solve the git problems for you. Pair with a percent script, ignore the ipynb files.
- kzrdude 3y ago
  jupytext is great. It even allows you to use only .py files directly as notebooks, but I recommend "pair with ipynb" and version controlling the .py file. The ipynb acts like a cache of the cell outputs between invocations of jupyterlab, which is handy too.
  - slt2021 3y ago
    I always wanted a Jupyter like environment, but the one that would natively support and output .py files jupyter notebook as IDE, but with .py files instead of .ipynb

fluidcruft 3y ago

One thing that I keep running into myself is I want to include data in a notebook as a sort of report or record of an analysis. I really like papermill for creating notebooks that execute and then store results. But you can only include text output or plots. If I wanted to store a numpy array within the notebook for inspection or input into.a next step, there doesn't seem to be a way. I understand it could be difficult to figure out how to make the all of that work. But I don't know. Maybe pickling or something should work?

kzrdude 3y ago
Ploomber does this kind of proper jupyter notebook pipeline. I've used it, but not stuck with it (yet)
- fluidcruft 3y ago
  I'll have to check it out. I have looked at it previously but it's somewhat overwhelming to figure out what it actually does. My sense previously was that is it was more like a dependency graph thing similar to a makefile but I'll give it a closer look.

Equiet 3y ago

We looked into many of these issues with Deepnote (YC S19) [https://deepnote.com/ https://deepnote.com/]. What we found is that these are not necessarily problems of the underlying medium (a notebook), but more of the specific implementation (Jupyter). We've seen a lot of progress in the Jupyter ecosystem, but unfortunately almost none in the areas you mentioned.

bjornasm 3y ago

>- There isn't a good way to take the same analysis and plug new data into it, other than to copy-paste the entire notebook. Cant you just set the parameters in the first cell? >- I get an analysis that I like, but there isn't a good way to share it with others, so I end up just taking screenshots. Export to md, pdf or html?

singhrac 3y ago

I manage a Jupyter instance where we do ~75% of all of our work, and before that used Jupyter daily as a quant. In response to your comments + sibling's: Sharing: there hasn't been a good solution, so you have to hack something together. For us (small team) symlinking a "shared_notebooks" network path works fine (but looking forward to the RTC mentioned in the post). It's very important to restart your notebook before sharing. New data: I've never really run into this in a form that "restart + run all cells" didn't really work. Are you trying to keep older versions? Reusability: I highly recommend installing a personal Python package in development mode to your kernel (i.e. pip install -e .). Then just move functions from the notebook to the package and reload. Boundaries: As others have said, I would just make a "user" kernel right away and never touch the Jupyter environment (as TLJH does by default). I am kind of hopeful that Armin Ronacher's rye fixes Python environment hell and we end up with better solutions for many of these things, but there are definitely some issues with version management in Jupyter.

snicker7 3y ago

Have you considered using Pluto notebooks instead? They are designed to be reproducible (including version references to transitive dependencies). They are also reactive, so it is trivial to change the dataset.

kzrdude 3y ago

Export to HTML is usually a decent report. If you want to refine it a bit, you can export and show only outputs, or (with some code) only certain tagged outputs. To plug in new data, I experiment with papermill. There are solutions that can put jupyter and the kernel in different environments. It's even not too hard to setup, but it astounds me it's not a common way to set it up. It's natural to have kernel + the analysis's deps in its own environment.

patrick451 3y ago

You may be interested in papermill to address the parametrized analysis problem [1]. I think (but I'm not positive) this is what the data team at a previous job used to automate running notebooks for all sorts nightly reports. [1] https://papermill.readthedocs.io/en/latest/# https://papermill.readthedocs.io/en/latest/#

agrama 3y ago

If you're looking for a way for others to reproduce your environments, you might find this useful: https://jupyenv.io/ https://jupyenv.io/

jph00 3y ago

To promote functions to a module use nbdev; execnb or papermill to parameterise your notebook; and nbconda to have environments for your notebooks.

dweinus 3y ago

https://github.com/Anaconda-Platform/nb_conda_kernels https://github.com/Anaconda-Platform/nb_conda_kernels

irjustin 3y ago

I've thought about this a lot in the context of our org. What you're describing is taking modeling -> production (minus the screen shot one). Typically a data engineer will be the one who helps bridge that gap, but that has a problem of does data engineer's output == scientist output, which can be time consuming to handle. To shrink the gap from dev->prod - we have 2 notebooks, one for model development and one model deployment in production. We use papermill[0] to execute directly notebooks in production. Shared functions between the dev/prod that are built by the scientist are put into a separate notebook and then imported via `run`. If I'm honest, our scientist don't do this and simply copy/paste the functions if we don't yell at them to fix it. This basically allows us to stay within the jupyter environment entirely so that dependancies are isolated. So, it's far from perfect, but it's allowed us to shrink the dev->prod life cycle time. Love to hear what others have done towards the same end.

sterlinm 3y ago

If you use conda there are extensions that can help with this by automatically registering any available conda environments that include ipykernel in your Jupyter Lab environment. nb_conda_kernels is pretty reliable but not actively maintained. Gator from the mamba folks is new and still a bit rough around the edges but looks like it will be pretty slick eventually. https://github.com/Anaconda-Platform/nb_conda_kernels https://github.com/Anaconda-Platform/nb_conda_kernels https://github.com/mamba-org/gator https://github.com/mamba-org/gator

__MatrixMan__ 3y ago

> There aren't good boundaries between Jupyter's own Python environment, and that of your notebooks— if you have a dependency which conflicts with one of Jupyter's dependencies, then good luck. I believe that you can use https://github.com/tweag/jupyenv https://github.com/tweag/jupyenv for this.

zelphirkalt 3y ago

Why not write the experimental code in the notebook, then write a function containing that code in the notebook and try it out, and only then move your function to a py file, if at all?

prometheon1 3y ago

> I get an analysis that I like, but there isn't a good way to share it with others, so I end up just taking screenshots. What would be your preferable way of sharing your analysis with others? - You can turn jupyter notebooks into pdfs directly in the jupyter UI. - You can upload them to Gitlab/Github and share the link to the rendered result. - You can upload them to Colab/Binder/Kaggle and let people play with the code themselves. - You can turn jupyter notebooks into beautiful websites/documents with: https://quarto.org/docs/tools/jupyter-lab.html https://quarto.org/docs/tools/jupyter-lab.html - You can add jupyter notebooks to you docs using nbsphinx: https://nbsphinx.readthedocs.io/en/0.9.1/ https://nbsphinx.readthedocs.io/en/0.9.1/ - You can turn jupyter notebooks into interactive web apps with voila: https://github.com/voila-dashboards/voila https://github.com/voila-dashboards/voila - You can turn jupyter notebooks into presentations with rise: https://github.com/voila-dashboards/voila https://github.com/voila-dashboards/voila

henrikgs 3y ago
I'm guessing the las one should link here https://rise.readthedocs.io/en/stable/ https://rise.readthedocs.io/en/stable/ and not to voila two times?
- prometheon1 3y ago
  You're right, thanks!
Desustorm 3y ago
To add to this, shamelessly self-promoting, Notebooker (https://github.com/man-group/notebooker https://github.com/man-group/notebooker) is a neat way of scheduling your Jupyter notebooks as parametrisable reports whose results are presented in a little web GUI (either as static HTML, PDF, or as reveal.js slideshow renders)
bippingchip 3y ago
While I have no need for its online functionality and the SAAS part of plotly, I really do like plotly python + cufflinks [1]. It lets you make interactive plots in html/js format. Which means you can save the notebook as html, and while people won't be able to rerun the code, they can still zoom in on graphs, hover to see annotations etc, which is a really nice way to share the outcome of your work in a more accessible way. [1] https://github.com/santosjorge/cufflinks https://github.com/santosjorge/cufflinks

Camillo 3y ago

> - The process to "promote" fragments of a notebook into being reusable functions seemed very high-friction: basically you're rewriting it as a normal Python package and then adding that to Jupyter's environment. > - There aren't good boundaries between Jupyter's own Python environment, and that of your notebooks— if you have a dependency which conflicts with one of Jupyter's dependencies, then good luck. The best Jupyter UX for me now is VSCode. Just put an .ipynb file in your workspace and you get the notebook interface inside VSCode. Put `%load_ext autoreload` and `%autoreload 2` in the first cell, and use the same python environment you're using in your workspace for the Jupyter kernel. Then you can import libraries from your project, use them, and it's very easy to promote code from the notebook into a library. You can just cut a function from the notebook, paste it into a library, add an import, and rerun the subsequent cells to verify it still works as expected.