3 ms·

I have been heads down in jupyter for the past couple of weeks and I finally realized I just DO NOT LIKE IT AT ALL! Cracks started appearing and then suddenly t

I have been heads down in jupyter for the past couple of weeks and I finally realized I just DO NOT LIKE IT AT ALL! Cracks started appearing and then suddenly there was an avalanche of disappointment. The first crack -- it's almost impossible to build a nice presentation in Jupyter, because you always have to show your code and its stderr. I imported all the TeX goodness, and it looked pretty nice, but I couldn't show the output without showing the TeX code. Importing the TeX interpreter is quite non-standard and means that my notebook doesn't play well with the public servers. I also got burned by some kind of permissions issue, so that all my charts ended up being invisible to read-only users. The second crack -- I can only look at the code from within my own jupyter server. The source is buried in a very noisy json format. The third crack -- Who wants to write code in the impoverished browser based editor provided? How many times have I deleted a closing brace that was automatically inserted incorrectly? How can I do a global search and replace? The fourth crack -- I can't test my code unless I include all the tests in the notebook! I'm complaining. I realize that I don't have anything constructive to offer, and I'm really a beginner. However, I think some of my disappointment is justified, as I think it was reasonable to assume that I could build my notebooks to be next level presentations.

mikepurvis 7y ago
This all sounds very familiar to me. I'm at a robotics company; we had some experimental infrastructure built up around processing ROS bag files via notebooks, and it just eventually became like pulling teeth. Stuff would get cut and pasted between notebooks, or moved out to helper modules which then had versioning and permissions chaos. Each bag needed its own notebook/interpreter instance because there's no way to rerun a notebook on new data, but then the server would explode because of these massive Python processes hanging around with half-processed data state still in them. In the end we dumped it all and turned the good parts into a sane CLI tool which ingests data and dumps out Bokeh plots. At some point we'll throw a Jenkins front end on it, but the current approach seems to be working fine.
bb88 7y ago
So once your code gets large enough that it doesn't fit neatly within a jupyter notebook, it's time to split the code out into another package, and then import it into your notebook. The benefit here is that now your code and be used inside the jupyter notebook, and also inside a webserver say.
amiga_500 7y ago
I've started to use jupyter with kdb on the back end for analysis. For me I have some hope it will hit the sweet spot because: 1. kdb is "too obtuse" for many and python glue makes it more amenable 2. I can still have kdb functions in source code and call them from jupyter with pyq 3. I can do most of my "editing" in emacs to the kdb back-end, write python "libs" for parsing results, and just use jupyter as a fairly thin presentation layer 4. I can share notebooks with analysts who run the same jupyter server virtual env, so finally we can share notes Will this add more value than just using kdb? Time will tell, hard to know right now. I agree that the "ide" experience absolutely sucks.
rcar 7y ago
To offer some help for 2 and 4, you can get a script out of a notebook with jupyter nbconvert <notebook.ipynb> --to python, which you can even include as a cell that starts with ! since that will run shell commands. For part of 3, there is global search and replace accessible either via Edit -> Find and Replace or Esc-F, and it includes case sensitivity toggles, regex search, and the option to change the current cell or globally. For 1, I think there are some plugins that can make that tidier, but I've generally just accepted that if the people I'm presenting to don't want to see code then I'll just need to make a set of slides out of the whole thing.
swebs 7y ago
>it's almost impossible to build a nice presentation in Jupyter, because you always have to show your code and its stderr. You should be able to see a blue vertical bar to the left of every code cell. Click that bar to collapse the cell. You can do this for both the code and the output. I know this works for Jupyter Lab, but I don't know about legacy notebooks.

mikepurvis 7y ago

This all sounds very familiar to me. I'm at a robotics company; we had some experimental infrastructure built up around processing ROS bag files via notebooks, and it just eventually became like pulling teeth. Stuff would get cut and pasted between notebooks, or moved out to helper modules which then had versioning and permissions chaos. Each bag needed its own notebook/interpreter instance because there's no way to rerun a notebook on new data, but then the server would explode because of these massive Python processes hanging around with half-processed data state still in them. In the end we dumped it all and turned the good parts into a sane CLI tool which ingests data and dumps out Bokeh plots. At some point we'll throw a Jenkins front end on it, but the current approach seems to be working fine.

bb88 7y ago

So once your code gets large enough that it doesn't fit neatly within a jupyter notebook, it's time to split the code out into another package, and then import it into your notebook. The benefit here is that now your code and be used inside the jupyter notebook, and also inside a webserver say.

amiga_500 7y ago

I've started to use jupyter with kdb on the back end for analysis. For me I have some hope it will hit the sweet spot because: 1. kdb is "too obtuse" for many and python glue makes it more amenable 2. I can still have kdb functions in source code and call them from jupyter with pyq 3. I can do most of my "editing" in emacs to the kdb back-end, write python "libs" for parsing results, and just use jupyter as a fairly thin presentation layer 4. I can share notebooks with analysts who run the same jupyter server virtual env, so finally we can share notes Will this add more value than just using kdb? Time will tell, hard to know right now. I agree that the "ide" experience absolutely sucks.

rcar 7y ago

To offer some help for 2 and 4, you can get a script out of a notebook with jupyter nbconvert <notebook.ipynb> --to python, which you can even include as a cell that starts with ! since that will run shell commands. For part of 3, there is global search and replace accessible either via Edit -> Find and Replace or Esc-F, and it includes case sensitivity toggles, regex search, and the option to change the current cell or globally. For 1, I think there are some plugins that can make that tidier, but I've generally just accepted that if the people I'm presenting to don't want to see code then I'll just need to make a set of slides out of the whole thing.

swebs 7y ago

>it's almost impossible to build a nice presentation in Jupyter, because you always have to show your code and its stderr. You should be able to see a blue vertical bar to the left of every code cell. Click that bar to collapse the cell. You can do this for both the code and the output. I know this works for Jupyter Lab, but I don't know about legacy notebooks.