4 ms·

> - I get an analysis that I like, but there isn't a good way to share it with others, so I end up just taking screenshots. Jupyter supports printing out as a

> - I get an analysis that I like, but there isn't a good way to share it with others, so I end up just taking screenshots. Jupyter supports printing out as a PDF, an all-in-one-webpage and all sorts of other formats. I've used this and the support for Markdown to create a number of documents that I've presented up the leadership chain, or to other engineers across the org "Here's the data, here's the narrative, and here's the code that shows you how I got to it so you can reproduce". Can even share the Jupyter notebook itself.

fluidcruft 3y ago
Do you have a way to store data directly into the notebook? I do a bunch of device testing and I have master notebooks setup to analyze and condense raw data. I use papermill to evaluate the master notebook to generate a report. But I want to also store/attach intermediates (say a named numpy array) into the notebook for further analysis.
- fn-mote 3y ago
  > I want to also store/attach intermediates (say a named numpy array) into the notebook for further analysis. Is the issue that you do not want to save the data and report into a folder and distribute that? That is, you want an entirely self-contained notebook? Or is there something else going on here? I'm sure that's possible but it seems kind of wrong to put your binary "data" in with your analysis and presentation-making code.
  - fluidcruft 3y ago
    Somewhat. I want it to be difficult to separate the data from the report. Basically I want the report itself to be ingestible as input to other steps, so it's more of a "documented data" with the analysis results available. The inputs are documented but the final results can be restored without reevaluating the entire notebook. I don't want the entire workspace saved, just the final results. I hoped there was some magic to inject and restore a python object from a cell using some form of introspection. The testing I do is annual equipment performance evaluation. I'd like to be able to process each test and then feed the results into longitudinal monitoring. One thing I am considering is adding a library or extension to papermill that automatically creates a workspace hdf5 or dill or whatever that I can store individual variables into. After studying the ipynb JSON it just seems odd that you can't just store blobs as attachments. But what I understand is it has to do with the kernels and notebooks running as separate processes and passing things around as notifications. So basically the kernels don't have any access to the cells or any sorts of introspection. With papermill you have parameters, there's just not any "return values" in the processed notebooks. If it existed you could treat "reports" more easily as cached function evaluations.
    - bee_rider 3y ago
      It looks like there’s a %store command in Jupyter. I haven’t tried it out, but is this what you are looking for? https://stackoverflow.com/questions/34342155/how-to-pickle-or-store-jupyter-ipython-notebook-session-for-later https://stackoverflow.com/questions/34342155/how-to-pickle-o... (I just got it by googling “pickle an object in jupyter,” so sorry if this is something obvious that you’ve already seen and doesn’t quite solve your problem).
      - fluidcruft 3y ago
        Yeah it got my hopes up when I found it. But when I was testing it, I didn't find the data actually made it into the .ipynb. It turns out that's actually a global storage in your home directory and doesn't go into the notebook at all. So different notebooks each overwrite the value if they use the same variable name.
        bee_rider 3y ago
        Oh wow, that’s a really awful design. Sorry.
- agons 3y ago
  The problem with that is that Jupyter requires a back-end to do any of the actual processing that runs the code in the cells. Only way I can think to make the html file truly portable while fully functional would be to embed a python interpreter and all required libraries as wasm. Could be possible with pyodide? I haven't used it.
  - fluidcruft 3y ago
    I just want selected/declared "final results" to be restorable in a new kernel. Papermill let's you easily parameterize notebooks and execute them. So the notebook feels like a well documented functions that behave like executibles with really fancy debug print logs that can be reviewed for correctness and reproducibility. Papermill does a great job of recording the input parameters and execution history. But there isn't anything equivalent to the "return foo, bar" part of a function which makes it difficult to build up modules. You don't want to have to digitize a plot to carry on to the next step is what I'm saying.

fluidcruft 3y ago

Do you have a way to store data directly into the notebook? I do a bunch of device testing and I have master notebooks setup to analyze and condense raw data. I use papermill to evaluate the master notebook to generate a report. But I want to also store/attach intermediates (say a named numpy array) into the notebook for further analysis.

fn-mote 3y ago
> I want to also store/attach intermediates (say a named numpy array) into the notebook for further analysis. Is the issue that you do not want to save the data and report into a folder and distribute that? That is, you want an entirely self-contained notebook? Or is there something else going on here? I'm sure that's possible but it seems kind of wrong to put your binary "data" in with your analysis and presentation-making code.
- fluidcruft 3y ago
  Somewhat. I want it to be difficult to separate the data from the report. Basically I want the report itself to be ingestible as input to other steps, so it's more of a "documented data" with the analysis results available. The inputs are documented but the final results can be restored without reevaluating the entire notebook. I don't want the entire workspace saved, just the final results. I hoped there was some magic to inject and restore a python object from a cell using some form of introspection. The testing I do is annual equipment performance evaluation. I'd like to be able to process each test and then feed the results into longitudinal monitoring. One thing I am considering is adding a library or extension to papermill that automatically creates a workspace hdf5 or dill or whatever that I can store individual variables into. After studying the ipynb JSON it just seems odd that you can't just store blobs as attachments. But what I understand is it has to do with the kernels and notebooks running as separate processes and passing things around as notifications. So basically the kernels don't have any access to the cells or any sorts of introspection. With papermill you have parameters, there's just not any "return values" in the processed notebooks. If it existed you could treat "reports" more easily as cached function evaluations.
  - bee_rider 3y ago
    It looks like there’s a %store command in Jupyter. I haven’t tried it out, but is this what you are looking for? https://stackoverflow.com/questions/34342155/how-to-pickle-or-store-jupyter-ipython-notebook-session-for-later https://stackoverflow.com/questions/34342155/how-to-pickle-o... (I just got it by googling “pickle an object in jupyter,” so sorry if this is something obvious that you’ve already seen and doesn’t quite solve your problem).
    - fluidcruft 3y ago
      Yeah it got my hopes up when I found it. But when I was testing it, I didn't find the data actually made it into the .ipynb. It turns out that's actually a global storage in your home directory and doesn't go into the notebook at all. So different notebooks each overwrite the value if they use the same variable name.
      - bee_rider 3y ago
        Oh wow, that’s a really awful design. Sorry.
agons 3y ago
The problem with that is that Jupyter requires a back-end to do any of the actual processing that runs the code in the cells. Only way I can think to make the html file truly portable while fully functional would be to embed a python interpreter and all required libraries as wasm. Could be possible with pyodide? I haven't used it.
- fluidcruft 3y ago
  I just want selected/declared "final results" to be restorable in a new kernel. Papermill let's you easily parameterize notebooks and execute them. So the notebook feels like a well documented functions that behave like executibles with really fancy debug print logs that can be reviewed for correctness and reproducibility. Papermill does a great job of recording the input parameters and execution history. But there isn't anything equivalent to the "return foo, bar" part of a function which makes it difficult to build up modules. You don't want to have to digitize a plot to carry on to the next step is what I'm saying.