5 ms·

Writing and editting code is a lot more flexible, but it gets repetitive, and I have written the same stuff so many times. It's all adhoc, and it fixes the pro

Writing and editting code is a lot more flexible, but it gets repetitive, and I have written the same stuff so many times. It's all adhoc, and it fixes the problem at the time, then it gets thrown away with the notebook only to be written again soon. As an example, I frequently want to run analytics on a dataframe. More complex summary stats. So you write a couple of functions, and have two for loops, iterating over columns and functions. This works for a bit. It's easy to add functions to the list. Then a function throws an error, and you're trying to figure out where you are in two nested for loops. Or, especially for pandas, you want to separate functions to depend on the same expensive pre-calc. You could pass the existing dict of computed measures so you can reuse that expensive calculation... Now you have to worry about the ordering of functions. So you could put all of your measures into one big function, but that isn't reusable. So you write your big function over and over. I built a small dag library that handles this, and lets you specify that your analysis requires keys and provides keys, then the DAG of functions is ordered for you. How do other people approach these issues?

kkoncevicius 2y ago
I work with R and not python, so some things might not apply, but this: > [...] it fixes the problem at the time, then it gets thrown away with the notebook only to be written again soon. Is one of the reasons I stopped using notebooks. One solution to your problem might be to create a simple executable script that, when called on the file of your dataset in a shell, would produce the visualisation you need. If it's an interactive visualisation then I would create a library or otherwise a re-usable piece of code that can be sourced. It takes some time but ends up saving more time in the end. If you have custom-made things you have to check on your data tables, then likely no library will solve your problem without you doing some additional the work on top. And for these: > Or, especially for pandas, you want to separate functions to depend on the same expensive pre-calc. [...] Now you have to worry about the ordering of functions. I save expensive outputs to intermediate files, and manage dependencies with a very simple build-system called redo [1][2]. [1]: http://www.goredo.cypherpunks.su http://www.goredo.cypherpunks.su [2]: http://karolis.koncevicius.lt/posts/using_redo_to_manage_r_data_analysis_workflow/ http://karolis.koncevicius.lt/posts/using_redo_to_manage_r_d...
- paddy_m 2y ago
  Thanks. I see how redo works. For larger datasets, real scripts are a better idea. I expect my stuff to work with datasets up to about 1Gb, caching is easy to layer on and would speed up work for larger datsets, but my code assumes the data fits in memory. It would be easier to add caching, the make sure I don't load an entire dataset into memory. (I don't serialize the entire dataframe to the browser though).
jampekka 2y ago
Usually I write scripts that use function memoization cache (to disk) for expensive operations. Recently I've also used Marimo sometimes, which has great support for modules (no reloading hacks), can memoize to disk and has deterministic state.

kkoncevicius 2y ago

I work with R and not python, so some things might not apply, but this: > [...] it fixes the problem at the time, then it gets thrown away with the notebook only to be written again soon. Is one of the reasons I stopped using notebooks. One solution to your problem might be to create a simple executable script that, when called on the file of your dataset in a shell, would produce the visualisation you need. If it's an interactive visualisation then I would create a library or otherwise a re-usable piece of code that can be sourced. It takes some time but ends up saving more time in the end. If you have custom-made things you have to check on your data tables, then likely no library will solve your problem without you doing some additional the work on top. And for these: > Or, especially for pandas, you want to separate functions to depend on the same expensive pre-calc. [...] Now you have to worry about the ordering of functions. I save expensive outputs to intermediate files, and manage dependencies with a very simple build-system called redo [1][2]. [1]: http://www.goredo.cypherpunks.su http://www.goredo.cypherpunks.su [2]: http://karolis.koncevicius.lt/posts/using_redo_to_manage_r_data_analysis_workflow/ http://karolis.koncevicius.lt/posts/using_redo_to_manage_r_d...

paddy_m 2y ago
Thanks. I see how redo works. For larger datasets, real scripts are a better idea. I expect my stuff to work with datasets up to about 1Gb, caching is easy to layer on and would speed up work for larger datsets, but my code assumes the data fits in memory. It would be easier to add caching, the make sure I don't load an entire dataset into memory. (I don't serialize the entire dataframe to the browser though).

jampekka 2y ago

Usually I write scripts that use function memoization cache (to disk) for expensive operations. Recently I've also used Marimo sometimes, which has great support for modules (no reloading hacks), can memoize to disk and has deterministic state.