3 ms·
I work with R and not python, so some things might not apply, but this: > [...] it fixes the problem at the time, then it gets thrown away with the notebook on
by kkoncevicius 2y ago
I work with R and not python, so some things might not apply, but this:
> [...] it fixes the problem at the time, then it gets thrown away with the notebook only to be written again soon.
Is one of the reasons I stopped using notebooks.
One solution to your problem might be to create a simple executable script that, when called on the file of your dataset in a shell, would produce the visualisation you need. If it's an interactive visualisation then I would create a library or otherwise a re-usable piece of code that can be sourced. It takes some time but ends up saving more time in the end.
If you have custom-made things you have to check on your data tables, then likely no library will solve your problem without you doing some additional the work on top.
And for these:
> Or, especially for pandas, you want to separate functions to depend on the same expensive pre-calc. [...] Now you have to worry about the ordering of functions.
I save expensive outputs to intermediate files, and manage dependencies with a very simple build-system called redo [1][2].
[1]: http://www.goredo.cypherpunks.su http://www.goredo.cypherpunks.su
[2]: http://karolis.koncevicius.lt/posts/using_redo_to_manage_r_data_analysis_workflow/ http://karolis.koncevicius.lt/posts/using_redo_to_manage_r_d...
- paddy_m 2y agoThanks. I see how redo works. For larger datasets, real scripts are a better idea. I expect my stuff to work with datasets up to about 1Gb, caching is easy to layer on and would speed up work for larger datsets, but my code assumes the data fits in memory. It would be easier to add caching, the make sure I don't load an entire dataset into memory. (I don't serialize the entire dataframe to the browser though).