6 ms·

Notebooks as reusable Python programs

117 points by akshayka 2y ago

dchuk 2y ago
I’ve been tinkering with Marimo, it’s pretty sweet (and you can use cursor or other AI IDEs pretty easily with it). On running notebooks as scripts: I can’t find in the docs what happens if you have plotting and other notebook oriented code? Like I’m using pygwalker to explore data through transformation steps, and end with saving to csv. If I just run the notebook as a script, is all of the plotting automatically skipped?
- cantdutchthis 2y ago
  (someone from the marimo team here) It depends a bit on how the notebook is written. There is `mo.app_meta()` that allows you to detect how the notebook is running. This can be in "app mode", "edit mode" or in "script mode". https://docs.marimo.io/api/app/?h=meta#marimo.app_meta https://docs.marimo.io/api/app/?h=meta#marimo.app_meta Effectively this could allow you to do things like "only run this bit when not in script mode" if you want to skip things. Alternatively you can also run the notebook via the `marimo export` command if you care about the charts and want to have a rendered notebook as an artifact. Gotta ask out of curiosity, anything you can share about your cursor/marimo workflow? Are you using the llm tools from within marimo or outside of it?
floathub 2y ago
One approach to this is org-mode with babel. You can have a plaintext file which is also the program which is also the documentation/notebook/website/etc. It's extremely powerful, and is a compelling example of literate programming. Decent overview here: https://www.johndcook.com/blog/2022/08/02/org-babel-vs-jupyter/ https://www.johndcook.com/blog/2022/08/02/org-babel-vs-jupyt... [edit: better link]
- addisonbeck 2y ago
  I'm writing this comment as I write a literate README.org that documents and tangles a PoC program I'm writing for an upcoming big initiative at my job. You can do most work in org instead of editing programs directly, and it'll save time and produce good documentation. Can't recommend it enough. This approach has the recently developed benefit of being really great as context stores for interacting with LLMs.
florbnit 2y ago
> When working with Jupyter, too often you end up with directories strewn with spaghetti-code notebooks, counting up to Untitled12.ipynb or higher. You the notebook author don’t know what’s in these notebooks This is such a small UX thing but it’s so damn important. The simple fix is to not auto-name notebooks untitled-# when the user clicks new notebook just ask the name straight away, if they can’t name it don’t create it. It might add the smallest amount of friction to the UX, but it’s so damn important. Also the choice of json as the file format is just plain wrong. Why the project hasn’t just abandoned that entirely and done a json-#python back and forth when writing to file is beyond me. There are extensions that do this, but that’s a really clunky interface, and while I can set it up for myself it’s difficult to force upon others in a corporate environment. Great to see someone is taking the seemingly small things up, because they mean a world of difference to the overall ecosystem.
- cantdutchthis 2y ago
  (someone from the marimo team here) How you start the marimo notebook, via `marimo edit must-give-name-to-this-file.py` is indeed one of my teeny by favourite features of it. When you start a new notebook you're kind of forced to name it immediately.
- BrenBarn 2y ago
  > The simple fix is to not auto-name notebooks untitled-# when the user clicks new notebook just ask the name straight away, if they can’t name it don’t create it. The even simpler fix is to just not name them until the user does. That's the way other programs work. If you create a new document in a word processor, it will say "Untitled" at the top of the window, but it doesn't create a file called untitled.doc on disk until you do "Save as" and choose a filename. It has always irritated me that Jupyter insists on having an on-disk file right from the beginning.
- janalsncm 2y ago
  It would be neat if it worked similar to ChatGPT sessions which give themself a name based on the conversation. You could have a small local model that gives a default name automatically. It would also be great to have a decent search across notebooks in a project so I could quickly find an old function.
paddy_m 2y ago
I develop an open source notebook widget. Working with marimo has been a joy compared to developing on top of any other notebook environment. The team is responsive and they care about getting it right. Having a sane file format for serializing notebooks is an example of this. They are thinking about core problems. They are also building in the open. The core jupyter team is very unresponsive and unfocused. When you have a bug, you need to figure out which one of many many interelated projects caused the bug, issues go weeks without a response. It's a mess. Then there are the proprietary notebook like environments. VSCode notebooks, and google colab in particular. They frequently rely on opaque undocumented APIs and are also very unresponsive.
- cantdutchthis 2y ago
  (someone from the marimo team here) Happy to hear it! Got a link to your widget? I am assuming it's an anywidget?
jdaw0 2y ago
i wanted to like marimo, but the best notebook interface i've tried so far is vscode's interactive window [0]. the important thing is that it's a python file first, but you can divide up the code into cells to run in the jupyter kernel either all at once or interactively. 0: https://code.visualstudio.com/docs/python/jupyter-support-py https://code.visualstudio.com/docs/python/jupyter-support-py
- cantdutchthis 2y ago
  Out of curiosity, does this approach also allow for interactive widgets?
  - luke-stanley 2y ago
    Yes. Though it is split into a code section and an interactive section, like with Markdown previews. It really is driven by the code cells though.
- 0cf8612b2e1e 2y ago
  This is also where I have landed. Gives you all of your nice IDE tooling alongside the REPL environment. No need for separate notebook aware code formatters/linters/etc. That they version cleanly is just the cherry on top.
- aaplok 2y ago
  Spyder also has these, possibly for longer than vscode [0]. I don't know who had this idea first but I remember some vim plugins doing that long ago, so maybe the vim community? [0] https://docs.spyder-ide.org/current/panes/editor.html#code-cells https://docs.spyder-ide.org/current/panes/editor.html#code-c...
  - westurner 2y ago
    Jupytext docs > The percent format: https://github.com/mwouts/jupytext/blob/main/docs/formats-scripts.md#the-percent-format https://github.com/mwouts/jupytext/blob/main/docs/formats-sc... : # %% [markdown] # Another Markdown cell # %% # This is a code cell class A(): def one(): return 1 # %% Optional title [cell type] key="value" MyST Markdown has: https://mystmd.org/guide/notebooks-with-markdown https://mystmd.org/guide/notebooks-with-markdown : ```{code-cell} LANGUAGE :key: value CODE TO BE EXECUTED ``` And : --- kernelspec: name: javascript display_name: JavaScript --- # Another markdown cell ```{code-cell} javascript // This is a code cell console.log("hello javascript kernel"); ``` But also does not store the outputs in the markdown.
abdullahkhalids 2y ago
One design decision they made is that outputs are not stored. This means these notebooks are not suitable replacement for heavy computation routines, where the notebook is a record of the final results. Other people are not expected to run minutes/hours long computation to see what the author intended. You can work your way around it by storing the results in a separate file(s), and writing the boiler plate to let the reader load the results. Or they let you export to ipynb - which is still sharing two files. Presumably the reason for this decision is making git diffs short. But to me the solution is to fix git diff to operate on JSON nicely, rather than changing the entire notebook format.
- duped 2y ago
  > Other people are not expected to run minutes/hours long computation to see what the author intended Arguably this is a good thing. You shouldn't distribute things you can't prove have the same results and one way to do that is to require others to run the same computations.
  - 0cf8612b2e1e 2y ago
    You lose out on so many use cases without the stored output. The first that comes to mind is all of the learning resources that are now presented in notebooks. Most learners do not need to fact check the instructor, but do want to see the operations which were run. Those that are curious can run/edit the notebook themselves. Edit: The JupyterBook ecosystem (https://executablebooks.org/en/latest/gallery/ https://executablebooks.org/en/latest/gallery/) as an example of what is possible with stored plots/calculations. Most learners are just going to follow along with the material, but being able to optionally play with the data is the super power of the platform with minimal friction.
    - mscolnick 2y ago
      The parent comment wasn't fully correct. marimo doesn't store outputs in the notebook file, but it does have many ways outputs are stored alongside the notebook, or remotely if you'd like: HTML, ipynb, pickle
dmadisetti 2y ago
Very excited for the top level functions change coming up !!
TheAlchemist 2y ago
This looks really very very neat. One (not a great) workflow I have, is that I use notebooks as quicks UIs to visualize some results. 1. Run a simulation that outputs results to some file 2. Load results in a notebook and do some quick processing + visualization Very often, I want to compare quickly between 2 different runs and end up copying down the cell with visualization, then just re-run the data load + processing + visualization and compare them. My understanding is that this would not be possible with marimo, since it will re-run automatically the cell with my previous data right ?
- mscolnick 2y ago
  It may be preferable to create a variable tied to a UI element that can be used as a toggle to view each analysis. choice = mo.ui.dropdown(['train', 'split') data = load(choice.value) processed = process(data) visualize(processed) This way, you can toggle between just more than two if needed. If you need to see both at once, you'd want to refactor the processing and visualizing step into functions, and then just duplicate the finals cell(s). marimo has a multi-column mode, so you can view them side-by-side
  - blooalien 2y ago
    Marimo also has a nice Tabs widget that can contain whatever else Marimo can display in cleanly separated tabs.
- cantdutchthis 2y ago
  (marimo team-member here) I have had a similar situation and my "hack" for this was to start the same notebook twice and have two tabs open. This worked for something things ... Other times I just bit the bullet and made two forms on two variables so that everything would fit in a single notebook. By having two variables that contain all the inputs you can make sure that only the cells update that need to update. It takes a bit more effort but makes a lot of sense for some apps that you want to share with a colleague. If you share more details about your setup I might be able to give better advice/think along more.
randomNumber7 2y ago
I recently started in data science. I like to just refactor out my stuff into normal phyton files that I import in notebooks
- darkteflon 2y ago
  I think most of the comments in here are concerned with the experience during the period _before_ you’re ready to refactor stuff out - ie the exploratory phase. Most people refactor stuff out as soon as it makes sense to do so.
lostdog 2y ago
They kinda skip over it, but jupytext is underrated. You store the notebook primarily as text, and your version control handles it seamlessly.
idanp 2y ago
For lightweight calculations re-execute everything while typing cells https://github.com/idanpa/jupad https://github.com/idanpa/jupad
epistasis 2y ago
I have looked at Marimo in the past, and read this blog post with great interest, but I still don't "get" Marimo. What it does well: have a sane way to create and interact with widgets. Lots of widget authors and tooling authors, people I respect a lot, admire Marimo and like how it does stuff. However, I'm not sure what the use case is for Marimo. I see Jupyter notebooks being used in two primary use cases: 1) prototyping new code and interactions with services and databases and datasets, as a record of the REPL used to understand something, with interactive notes and plots and pasted in images from docs, etc. 2) a record of how a calculation was performed, experimental data analyzed, and in a permanent artifact that others can look up later. For both of these, outputs and markdown/image cells are just as important as the code cells. These are both "write once" types of things where changes in git are rare, and ideally would never happen. With Marimo, can I check the outputs directory into version control in a reasonable way and have it stored for posterity? Is that .ipynb? Is there a way to convert a stored .ipynb checkpoint back into the marimo format? And why does a small .ipynb change lead to many lines of change in the git diff? It's because the outputs changed. Deciding to not store outputs in version control and counting it as a win for pretty git diffs is saying "this core feature of .ipynb should be ignored because it's inconvenient". I'd much rather educate people about turning on GitHub's visual Jupyter diff rather than switch to an environment where I can no longer store outputs inline. Similarly, being able to import one cell into a different notebook seems like the wrong direction to solve the problem of "it's time to turn the prototype notebook into a reusable module." If it's time to reuse a cell, it's time to make a cleaned-up Python code module file, not have the code interspersed with all the rest of the stuff. I'd like to learn more about the use cases where Marimo is useful. As a scientist, it's not useful to me. I don't care about smaller git diffs on a notebook, in fact if a notebook is getting changed and re-checked into version control then a big awkward diff is not a problem and probably a feature, because notebooks should not be getting changed. They are a notebook something that you write in once and it's done!
- dmadisetti 2y ago
  I'm from a computational mechanics background, and my biggest issue with jupyter notebooks was doing analysis, playing around with my models, getting results I was happy with; then having my kernel break on a fresh session whenever I wanted to revisit the result. You could argue this is a user issue, and being sloppy with cell state is a me problem- not a notebook problem- but marimo prevents this in the first place. Just once is already a wasted day's work. Even if one is especially diligent in Jupyter, they do not have the guarantees marimo provides. Unless you are pressing "restart and run all" in a Jupyter kernel every time - you are accumulating state. I got started with marimo and helped build out the caching because I wanted to open up a notebook from scratch and automatically restart at the point I was at- without having to worry whether my cell order is correct or not. I wanted my code to automatically rerun update outputs if I changed something like a filtering or transform step to my data. I also wanted to go back to previous states easily without having to rerun long computations or deal with data dump versioning. marimo's caching makes this all automatic. The plan is to bring caching to an opt-in cell basis so this happens behind the scenes, with some ideas for handling side-effects like randomness and other "non-pure" cases. But to answer your points, my use case has been specifically scientific computing. Yes for 1) prototyping- but also knowing my prototype does not hold some hidden state or bug. I have shown my advisor results, gone to deploy and been unable to replicate them at scale because of some hidden bug in my notebook execution. To me this is unacceptable. but also 2) documentation, notes, and report sharing. Not saving the outputs directly with the code isn't a blocker here, because caching automatically restores my state, and I can export to a more suitable format like a PDF or HTML file if I need to share it. Saving generated binaries in source for any other setup would be considered poor form- I don't understand why Jupyter gets a pass here. "Write once" is a google doc, notebooks should be "run many times", i.e. reproducible and sharable. Hopefully, we get to address some of the other pain points with existing infrastructure; like auto-scaling on HPCs (easy for marimo, it's already a DAG); integrating code more directly into papers (we have mkdocs, and experimental quarto extension exists); and built-in environments (no more `pip install -R requirements.txt` on a paper project and hoping the authors remembered to freeze and pinned versions) --- Hope my tone doesn't come off too contentious- I've just been burned by Jupyter and I'm excited because I feel like marimo addresses its major issues.
cjohnson318 2y ago
I sometimes use notebooks mostly for taking notes, with a few code samples. In these cases, dealing with ipykernel and firing up a notebook is kind of a pain. Being able to open a "notebook" and make changes in vim sounds great.
- mscolnick 2y ago
  It wasn't included in the blog, but marimo also supports a markdown format, in case you're interested: marimo edit notebook.md
Kydlaw 2y ago
I discovered Marimo a couple weeks/months ago here iirc. This really lands on a sweet spot for me for data exploration. For me the features that really nails it are the easy imports from other modules, the integrated UI components, and the app mode. Being able to build model/simulations easily and being able to share them with others, who can then even interact with the results, as truly motivated me to try more stuff and build more. I've been deploying more and more of these apps as PoCs to prospects and people really like them as well. Big thanks to the team!
ayhanfuat 2y ago
Unfortunately they don’t have Jupyter’s command mode. I wanted to switch a few times but not being able to create/delete/copy/move cells as easily is a big issue for me.
stared 2y ago
I am surprised they didn't mention RMarkdown (https://rmarkdown.rstudio.com/ https://rmarkdown.rstudio.com/), which was developed in parallel to Jupyter Notebooks, with lots of convergent evolution. RMarkdown is essentially Markdown with executable code blocks. While it comes from an R background, code blocks can be written in any language (and you can mix multiple languages). The biggest difference (and, I would say, advantage) is that it separates code from output, making it work well with version control.
blooalien 2y ago
My own personal experience of Marimo vs JupyterLab is that they're both really useful tools, but with somewhat differing (but overlapping) purposes. Marimo has become my go-to for when I want something "notebook-like" but which operates more like a fully-fledged application than a Jupyter notebook would, where JupyterLab still (at least for me) is the better choice for random explorations, learning, and note-taking/documentation of my learnings. At this point I'm really feeling like it's a good thing to have both handy and available, since neither one is really a complete replacement for the other.
albinn 2y ago
Recently I had to use Julia and their Pluto notebooks. While I really did not enjoy Julia, I loved Pluto notebooks, and Marimo is basically this, but for python. The reactive component is great for exploring data in an easy way, with sliders and buttons. Also for simulations when you want to try a bunch of different parameters.
nemoniac 2y ago
"until recently, Jupyter notebooks were the only programming environment that let you see your data while you worked on it." This is false. Org-mode has had this functionality for over two decades. https://orgmode.org/ https://orgmode.org/
- jarpineh 2y ago
  And since in Lisp code is data and data is code you could go even farther back. A tad sensationalist claim from the article authors.
jarpineh 2y ago
I find it a baffling that the popularity of Jupyter and successes of notebook analysis in science hasn’t brought a change in Python to better support this user base. Packaging has (slowly) progressed and Uv nicely made the experience smooth, fast and above all coherent. Yet the Python runtime and parser are the same as ever. The ipynb notebook format and now Marimo’s decorator approach had to be invented on top. Python might never attain the heights of Lisp’s REPL driven development, yet I wonder if it couldn’t be better. As much I enjoy using Jupyter it’s always been something tacked on top of infrastructure that doesn’t want to accommodate it. Thus you need to take care of cell order yourself or learn to use a helper tool (Jupytext, Nbdev). Me, I’d have support in the parser for a language structure or magic comment that points the cell boundaries. I would make dynamic execution of code a first party feature with history tracking and export of all the code sent into the runtime through this. Thus what the runtime saw happen could be committed over what user thought they did. Also, a better, notebook aware evaluation with extension hooks for different usage methods (interactive, scripting, testing). I have no solution to ipynb JSON problem. I do think it is bad that our seemingly only solution for version control can manage only simple text, and users of all the other format have to adapt or suffer.

117 points by akshayka 2y ago

I’ve been tinkering with Marimo, it’s pretty sweet (and you can use cursor or other AI IDEs pretty easily with it). On running notebooks as scripts: I can’t find in the docs what happens if you have plotting and other notebook oriented code? Like I’m using pygwalker to explore data through transformation steps, and end with saving to csv. If I just run the notebook as a script, is all of the plotting automatically skipped?

cantdutchthis 2y ago
(someone from the marimo team here) It depends a bit on how the notebook is written. There is `mo.app_meta()` that allows you to detect how the notebook is running. This can be in "app mode", "edit mode" or in "script mode". https://docs.marimo.io/api/app/?h=meta#marimo.app_meta https://docs.marimo.io/api/app/?h=meta#marimo.app_meta Effectively this could allow you to do things like "only run this bit when not in script mode" if you want to skip things. Alternatively you can also run the notebook via the `marimo export` command if you care about the charts and want to have a rendered notebook as an artifact. Gotta ask out of curiosity, anything you can share about your cursor/marimo workflow? Are you using the llm tools from within marimo or outside of it?

floathub 2y ago

One approach to this is org-mode with babel. You can have a plaintext file which is also the program which is also the documentation/notebook/website/etc. It's extremely powerful, and is a compelling example of literate programming. Decent overview here: https://www.johndcook.com/blog/2022/08/02/org-babel-vs-jupyter/ https://www.johndcook.com/blog/2022/08/02/org-babel-vs-jupyt... [edit: better link]

addisonbeck 2y ago
I'm writing this comment as I write a literate README.org that documents and tangles a PoC program I'm writing for an upcoming big initiative at my job. You can do most work in org instead of editing programs directly, and it'll save time and produce good documentation. Can't recommend it enough. This approach has the recently developed benefit of being really great as context stores for interacting with LLMs.

florbnit 2y ago

> When working with Jupyter, too often you end up with directories strewn with spaghetti-code notebooks, counting up to Untitled12.ipynb or higher. You the notebook author don’t know what’s in these notebooks This is such a small UX thing but it’s so damn important. The simple fix is to not auto-name notebooks untitled-# when the user clicks new notebook just ask the name straight away, if they can’t name it don’t create it. It might add the smallest amount of friction to the UX, but it’s so damn important. Also the choice of json as the file format is just plain wrong. Why the project hasn’t just abandoned that entirely and done a json-#python back and forth when writing to file is beyond me. There are extensions that do this, but that’s a really clunky interface, and while I can set it up for myself it’s difficult to force upon others in a corporate environment. Great to see someone is taking the seemingly small things up, because they mean a world of difference to the overall ecosystem.

cantdutchthis 2y ago
(someone from the marimo team here) How you start the marimo notebook, via `marimo edit must-give-name-to-this-file.py` is indeed one of my teeny by favourite features of it. When you start a new notebook you're kind of forced to name it immediately.
BrenBarn 2y ago
> The simple fix is to not auto-name notebooks untitled-# when the user clicks new notebook just ask the name straight away, if they can’t name it don’t create it. The even simpler fix is to just not name them until the user does. That's the way other programs work. If you create a new document in a word processor, it will say "Untitled" at the top of the window, but it doesn't create a file called untitled.doc on disk until you do "Save as" and choose a filename. It has always irritated me that Jupyter insists on having an on-disk file right from the beginning.
janalsncm 2y ago
It would be neat if it worked similar to ChatGPT sessions which give themself a name based on the conversation. You could have a small local model that gives a default name automatically. It would also be great to have a decent search across notebooks in a project so I could quickly find an old function.

paddy_m 2y ago

I develop an open source notebook widget. Working with marimo has been a joy compared to developing on top of any other notebook environment. The team is responsive and they care about getting it right. Having a sane file format for serializing notebooks is an example of this. They are thinking about core problems. They are also building in the open. The core jupyter team is very unresponsive and unfocused. When you have a bug, you need to figure out which one of many many interelated projects caused the bug, issues go weeks without a response. It's a mess. Then there are the proprietary notebook like environments. VSCode notebooks, and google colab in particular. They frequently rely on opaque undocumented APIs and are also very unresponsive.

cantdutchthis 2y ago
(someone from the marimo team here) Happy to hear it! Got a link to your widget? I am assuming it's an anywidget?

jdaw0 2y ago

i wanted to like marimo, but the best notebook interface i've tried so far is vscode's interactive window [0]. the important thing is that it's a python file first, but you can divide up the code into cells to run in the jupyter kernel either all at once or interactively. 0: https://code.visualstudio.com/docs/python/jupyter-support-py https://code.visualstudio.com/docs/python/jupyter-support-py

cantdutchthis 2y ago
Out of curiosity, does this approach also allow for interactive widgets?
- luke-stanley 2y ago
  Yes. Though it is split into a code section and an interactive section, like with Markdown previews. It really is driven by the code cells though.
0cf8612b2e1e 2y ago
This is also where I have landed. Gives you all of your nice IDE tooling alongside the REPL environment. No need for separate notebook aware code formatters/linters/etc. That they version cleanly is just the cherry on top.
aaplok 2y ago
Spyder also has these, possibly for longer than vscode [0]. I don't know who had this idea first but I remember some vim plugins doing that long ago, so maybe the vim community? [0] https://docs.spyder-ide.org/current/panes/editor.html#code-cells https://docs.spyder-ide.org/current/panes/editor.html#code-c...
- westurner 2y ago
  Jupytext docs > The percent format: https://github.com/mwouts/jupytext/blob/main/docs/formats-scripts.md#the-percent-format https://github.com/mwouts/jupytext/blob/main/docs/formats-sc... : # %% [markdown] # Another Markdown cell # %% # This is a code cell class A(): def one(): return 1 # %% Optional title [cell type] key="value" MyST Markdown has: https://mystmd.org/guide/notebooks-with-markdown https://mystmd.org/guide/notebooks-with-markdown : ```{code-cell} LANGUAGE :key: value CODE TO BE EXECUTED ``` And : --- kernelspec: name: javascript display_name: JavaScript --- # Another markdown cell ```{code-cell} javascript // This is a code cell console.log("hello javascript kernel"); ``` But also does not store the outputs in the markdown.

abdullahkhalids 2y ago

One design decision they made is that outputs are not stored. This means these notebooks are not suitable replacement for heavy computation routines, where the notebook is a record of the final results. Other people are not expected to run minutes/hours long computation to see what the author intended. You can work your way around it by storing the results in a separate file(s), and writing the boiler plate to let the reader load the results. Or they let you export to ipynb - which is still sharing two files. Presumably the reason for this decision is making git diffs short. But to me the solution is to fix git diff to operate on JSON nicely, rather than changing the entire notebook format.

duped 2y ago
> Other people are not expected to run minutes/hours long computation to see what the author intended Arguably this is a good thing. You shouldn't distribute things you can't prove have the same results and one way to do that is to require others to run the same computations.
- 0cf8612b2e1e 2y ago
  You lose out on so many use cases without the stored output. The first that comes to mind is all of the learning resources that are now presented in notebooks. Most learners do not need to fact check the instructor, but do want to see the operations which were run. Those that are curious can run/edit the notebook themselves. Edit: The JupyterBook ecosystem (https://executablebooks.org/en/latest/gallery/ https://executablebooks.org/en/latest/gallery/) as an example of what is possible with stored plots/calculations. Most learners are just going to follow along with the material, but being able to optionally play with the data is the super power of the platform with minimal friction.
  - mscolnick 2y ago
    The parent comment wasn't fully correct. marimo doesn't store outputs in the notebook file, but it does have many ways outputs are stored alongside the notebook, or remotely if you'd like: HTML, ipynb, pickle

dmadisetti 2y ago

Very excited for the top level functions change coming up !!

TheAlchemist 2y ago

This looks really very very neat. One (not a great) workflow I have, is that I use notebooks as quicks UIs to visualize some results. 1. Run a simulation that outputs results to some file 2. Load results in a notebook and do some quick processing + visualization Very often, I want to compare quickly between 2 different runs and end up copying down the cell with visualization, then just re-run the data load + processing + visualization and compare them. My understanding is that this would not be possible with marimo, since it will re-run automatically the cell with my previous data right ?

mscolnick 2y ago
It may be preferable to create a variable tied to a UI element that can be used as a toggle to view each analysis. choice = mo.ui.dropdown(['train', 'split') data = load(choice.value) processed = process(data) visualize(processed) This way, you can toggle between just more than two if needed. If you need to see both at once, you'd want to refactor the processing and visualizing step into functions, and then just duplicate the finals cell(s). marimo has a multi-column mode, so you can view them side-by-side
- blooalien 2y ago
  Marimo also has a nice Tabs widget that can contain whatever else Marimo can display in cleanly separated tabs.
cantdutchthis 2y ago
(marimo team-member here) I have had a similar situation and my "hack" for this was to start the same notebook twice and have two tabs open. This worked for something things ... Other times I just bit the bullet and made two forms on two variables so that everything would fit in a single notebook. By having two variables that contain all the inputs you can make sure that only the cells update that need to update. It takes a bit more effort but makes a lot of sense for some apps that you want to share with a colleague. If you share more details about your setup I might be able to give better advice/think along more.

randomNumber7 2y ago

I recently started in data science. I like to just refactor out my stuff into normal phyton files that I import in notebooks

darkteflon 2y ago
I think most of the comments in here are concerned with the experience during the period _before_ you’re ready to refactor stuff out - ie the exploratory phase. Most people refactor stuff out as soon as it makes sense to do so.

lostdog 2y ago

They kinda skip over it, but jupytext is underrated. You store the notebook primarily as text, and your version control handles it seamlessly.

idanp 2y ago

For lightweight calculations re-execute everything while typing cells https://github.com/idanpa/jupad https://github.com/idanpa/jupad

epistasis 2y ago

I have looked at Marimo in the past, and read this blog post with great interest, but I still don't "get" Marimo. What it does well: have a sane way to create and interact with widgets. Lots of widget authors and tooling authors, people I respect a lot, admire Marimo and like how it does stuff. However, I'm not sure what the use case is for Marimo. I see Jupyter notebooks being used in two primary use cases: 1) prototyping new code and interactions with services and databases and datasets, as a record of the REPL used to understand something, with interactive notes and plots and pasted in images from docs, etc. 2) a record of how a calculation was performed, experimental data analyzed, and in a permanent artifact that others can look up later. For both of these, outputs and markdown/image cells are just as important as the code cells. These are both "write once" types of things where changes in git are rare, and ideally would never happen. With Marimo, can I check the outputs directory into version control in a reasonable way and have it stored for posterity? Is that .ipynb? Is there a way to convert a stored .ipynb checkpoint back into the marimo format? And why does a small .ipynb change lead to many lines of change in the git diff? It's because the outputs changed. Deciding to not store outputs in version control and counting it as a win for pretty git diffs is saying "this core feature of .ipynb should be ignored because it's inconvenient". I'd much rather educate people about turning on GitHub's visual Jupyter diff rather than switch to an environment where I can no longer store outputs inline. Similarly, being able to import one cell into a different notebook seems like the wrong direction to solve the problem of "it's time to turn the prototype notebook into a reusable module." If it's time to reuse a cell, it's time to make a cleaned-up Python code module file, not have the code interspersed with all the rest of the stuff. I'd like to learn more about the use cases where Marimo is useful. As a scientist, it's not useful to me. I don't care about smaller git diffs on a notebook, in fact if a notebook is getting changed and re-checked into version control then a big awkward diff is not a problem and probably a feature, because notebooks should not be getting changed. They are a notebook something that you write in once and it's done!

dmadisetti 2y ago
I'm from a computational mechanics background, and my biggest issue with jupyter notebooks was doing analysis, playing around with my models, getting results I was happy with; then having my kernel break on a fresh session whenever I wanted to revisit the result. You could argue this is a user issue, and being sloppy with cell state is a me problem- not a notebook problem- but marimo prevents this in the first place. Just once is already a wasted day's work. Even if one is especially diligent in Jupyter, they do not have the guarantees marimo provides. Unless you are pressing "restart and run all" in a Jupyter kernel every time - you are accumulating state. I got started with marimo and helped build out the caching because I wanted to open up a notebook from scratch and automatically restart at the point I was at- without having to worry whether my cell order is correct or not. I wanted my code to automatically rerun update outputs if I changed something like a filtering or transform step to my data. I also wanted to go back to previous states easily without having to rerun long computations or deal with data dump versioning. marimo's caching makes this all automatic. The plan is to bring caching to an opt-in cell basis so this happens behind the scenes, with some ideas for handling side-effects like randomness and other "non-pure" cases. But to answer your points, my use case has been specifically scientific computing. Yes for 1) prototyping- but also knowing my prototype does not hold some hidden state or bug. I have shown my advisor results, gone to deploy and been unable to replicate them at scale because of some hidden bug in my notebook execution. To me this is unacceptable. but also 2) documentation, notes, and report sharing. Not saving the outputs directly with the code isn't a blocker here, because caching automatically restores my state, and I can export to a more suitable format like a PDF or HTML file if I need to share it. Saving generated binaries in source for any other setup would be considered poor form- I don't understand why Jupyter gets a pass here. "Write once" is a google doc, notebooks should be "run many times", i.e. reproducible and sharable. Hopefully, we get to address some of the other pain points with existing infrastructure; like auto-scaling on HPCs (easy for marimo, it's already a DAG); integrating code more directly into papers (we have mkdocs, and experimental quarto extension exists); and built-in environments (no more `pip install -R requirements.txt` on a paper project and hoping the authors remembered to freeze and pinned versions) --- Hope my tone doesn't come off too contentious- I've just been burned by Jupyter and I'm excited because I feel like marimo addresses its major issues.

cjohnson318 2y ago

I sometimes use notebooks mostly for taking notes, with a few code samples. In these cases, dealing with ipykernel and firing up a notebook is kind of a pain. Being able to open a "notebook" and make changes in vim sounds great.

mscolnick 2y ago
It wasn't included in the blog, but marimo also supports a markdown format, in case you're interested: marimo edit notebook.md

Kydlaw 2y ago

I discovered Marimo a couple weeks/months ago here iirc. This really lands on a sweet spot for me for data exploration. For me the features that really nails it are the easy imports from other modules, the integrated UI components, and the app mode. Being able to build model/simulations easily and being able to share them with others, who can then even interact with the results, as truly motivated me to try more stuff and build more. I've been deploying more and more of these apps as PoCs to prospects and people really like them as well. Big thanks to the team!

ayhanfuat 2y ago

Unfortunately they don’t have Jupyter’s command mode. I wanted to switch a few times but not being able to create/delete/copy/move cells as easily is a big issue for me.

stared 2y ago

I am surprised they didn't mention RMarkdown (https://rmarkdown.rstudio.com/ https://rmarkdown.rstudio.com/), which was developed in parallel to Jupyter Notebooks, with lots of convergent evolution. RMarkdown is essentially Markdown with executable code blocks. While it comes from an R background, code blocks can be written in any language (and you can mix multiple languages). The biggest difference (and, I would say, advantage) is that it separates code from output, making it work well with version control.

blooalien 2y ago

My own personal experience of Marimo vs JupyterLab is that they're both really useful tools, but with somewhat differing (but overlapping) purposes. Marimo has become my go-to for when I want something "notebook-like" but which operates more like a fully-fledged application than a Jupyter notebook would, where JupyterLab still (at least for me) is the better choice for random explorations, learning, and note-taking/documentation of my learnings. At this point I'm really feeling like it's a good thing to have both handy and available, since neither one is really a complete replacement for the other.

albinn 2y ago

Recently I had to use Julia and their Pluto notebooks. While I really did not enjoy Julia, I loved Pluto notebooks, and Marimo is basically this, but for python. The reactive component is great for exploring data in an easy way, with sliders and buttons. Also for simulations when you want to try a bunch of different parameters.

nemoniac 2y ago

"until recently, Jupyter notebooks were the only programming environment that let you see your data while you worked on it." This is false. Org-mode has had this functionality for over two decades. https://orgmode.org/ https://orgmode.org/

jarpineh 2y ago
And since in Lisp code is data and data is code you could go even farther back. A tad sensationalist claim from the article authors.

jarpineh 2y ago

I find it a baffling that the popularity of Jupyter and successes of notebook analysis in science hasn’t brought a change in Python to better support this user base. Packaging has (slowly) progressed and Uv nicely made the experience smooth, fast and above all coherent. Yet the Python runtime and parser are the same as ever. The ipynb notebook format and now Marimo’s decorator approach had to be invented on top. Python might never attain the heights of Lisp’s REPL driven development, yet I wonder if it couldn’t be better. As much I enjoy using Jupyter it’s always been something tacked on top of infrastructure that doesn’t want to accommodate it. Thus you need to take care of cell order yourself or learn to use a helper tool (Jupytext, Nbdev). Me, I’d have support in the parser for a language structure or magic comment that points the cell boundaries. I would make dynamic execution of code a first party feature with history tracking and export of all the code sent into the runtime through this. Thus what the runtime saw happen could be committed over what user thought they did. Also, a better, notebook aware evaluation with extension hooks for different usage methods (interactive, scripting, testing). I have no solution to ipynb JSON problem. I do think it is bad that our seemingly only solution for version control can manage only simple text, and users of all the other format have to adapt or suffer.