4 ms·
>> marimo notebooks are pure Python and stored as .py files That sounds like a solid improvement. I’m going to give this a test drive. I feel like modularity i
by CraigJPerry 2y ago
>> marimo notebooks are pure Python and stored as .py files
That sounds like a solid improvement. I’m going to give this a test drive. I feel like modularity is one of the hardest aspects of Jupyter notebooks in a team environment.
I’d be interested to hear if anyone has cracked a workflow with notebooks for larger teams. Notebooks are easy for solo or very small teams, and the literate programming style benefits still apply in larger teams but there’s a lot of friction: “hey just %run this shared notebook with a bunch of useful utilities in it - oops yeah it tries to write some files because of some stuff unrelated to your use case in there (that’s essential to my use case)”
My current best that I know of is to keep “calculation” (pure) code in a .py and just the “action“ (side-effectful) code in the notebook. Then as far as physically possible, keep the data outside of notebook (usually a database or csv’s). That helps avoid the main time sink pitfalls (resolving git conflicts, versioning, testing etc) but it doesn’t solve for example tooling you might want to run - maybe mypy against that action code - sure you can use nbqa but… interested to learn better approaches.
- szvsw 2y agoI actually think the problem you are describing is actually sometimes helpful from a design perspective, if you can be conscientious enough to periodically review your notebooks and figure out what is the actual useful code which should be properly integrated into the codebase vs what is the “one-off” / non-modular code. Like you mentioned, calculation vs side-effects is one way to help you decide but not the only. There’s definitely no single answer. The key is to just periodically figure out what ought to be refactored into library code, which notebooks should just be straight up deleted (hopefully as many as possible - you can always get them back in your git history if needed!), and so on.
- qsort 2y agoI'm mostly in the camp that notebooks aren't that great for software development, they thrive as an "excel for coders" of sorts, but take a look at nbdev from fast.ai. The literate programming aspect is very nice and I wish it was explored more.