4 ms·

Closer to production quality Python notebooks with `marimo check`

31 points by dmadisetti 1y ago

dmadisetti 1y ago
marimo notebooks double as python Python programs, which means they can be lint-ed like any other code. `marimo check` catches notebook-specific issues (variable redefinition across cells, circular dependencies) with Rust-inspired (and uv; uv is amazing) error messages. It outputs JSON for AI agents to self-correct, integrates into CI pipelines, and includes `--fix` flags for automatic repairs. The team is already using it in their own CI and seeing Claude Code iterate on notebooks without human intervention.
- aitchnyu 1y ago
  IME Agents work fine on human readable error messages.
- sixo 1y ago
  Unfortunately the code in Marimo notebooks implements cross-variable references as untyped function arguments, meaning that nothing passed between cells can be type-safe. This makes it very hard to use tooling directly on the Python representation of notebooks.
  - dmadisetti 1y ago
    Arguments are typed if explicitly specified in definition, e.g. ```python @app.cell def my_cell(): x : int = 1 return (x,) @app.cell def consumer(x : int): y = x return ``` We've talked about building out implicit typing into our serialization- but that may be a larger can of worms.
    - sixo 1y ago
      That would be wonderful. A lot of the arguments are just imports, and these would have probably have the largest upside. It would be great if there was a construct like import numpy @app.cell def my_cell(np: types.ModuleType[numpy]): ... editor treats np like `import numpy as np` ... I do not use Python enough to know if something like this can be hacked together. But if not, I imagine could be PIPed into the standard library if a convincing case were made for it.
      - dmadisetti 1y ago
        We've addressed this too by allowing a setup cell: ... with app.setup: import numpy as np @app.cell def cell_uses_np(): # no np needed in signature result = np.... The best part about this pattern is that it enables "pure" (only dependent on setup cell) functions to be addressable: with app.setup: from typing import Optional, Any @app.function def my_fn(arg: Optional[Any]) -> Any: ... but nice convergent design :) glad to see to see we're addressing pain points
        sixo 1y ago
        ah, I recall seeing that feature but had not taken a look at it for imports. Imports ARE the case where untyped cell arguments are most annoying, but of course it would be nice to get it for free in all cases.
wodenokoto 1y ago
I’m very interested in moving beyond Jupyter notebooks so I have my eye on marimo. My understanding is it’s just python code with a bit of notebook hints and an idea that gives the notebook experience. So I didn’t quite catch from the article why ruff and pylons aren’t enough.
- akshayka 1y ago
  Hi! Thanks for your interest. marimo is much more than that — unlike traditional notebooks, marimo is "reactive", meaning it models notebooks as dataflow graphs and keeps code an outputs in sync. Moreover, marimo notebooks are not "just notebooks". They can be seamlessly run as interactive web apps or as Python scripts. Here is our original ShowHN post that explains what marimo is all about: https://news.ycombinator.com/item?id=38971966 https://news.ycombinator.com/item?id=38971966 Blog that goes deeper: https://marimo.io/blog/lessons-learned https://marimo.io/blog/lessons-learned
  - daxfohl 1y ago
    I was just thinking about something like this a couple hours ago. Jeremy Howard (fast.ai, etc) posted a few weeks ago about his new AI class [1] and I was pumped. I went through the fast.ai course a couple years ago and had been wondering if another would ever come out, and the overview of this sounded really interesting. But the first thing the announcement went into was Jupyter Notebooks, and it kept on and on about them and I rolled my eyes. I remember they were my least favorite part of the fast.ai class. They've always felt awkward to me, being used to a code editor and all the functionality it provides, and I'm sure I'm not alone. I'm sure Jupyter is great for fast iteration if that's what you do all day, but if you spend most of your life in a standard code editor, you really want something that starts from there and adds a bit of interactivity on top, rather than jumping to a completely different concept. I haven't tried marimo yet, but I might give it a shot for my next project. [1] https://news.ycombinator.com/item?id=45455810 https://news.ycombinator.com/item?id=45455810
    - spwa4 1y ago
      Just use VSCode, which will give you full language server editing in ipynb files?
loveparade 1y ago
I wish this didn't have AI in it. I've been looking for a Jupyter alternative that is pure python and can be modified from a regular text editor. Jupytext works okay, but I miss the advanced Jupyter features. But I really don't want to deal with yet another AI assistant, especially not a custom one when I'm already using Claude/etc from the CLI and I want those agents to help me edit the notebooks. Take out all the AI stuff and I'd give it a try. I use AI coding agents as my daily driver, but I really don't need this AI enshittification in every tool/library I'm using.
- 3eb7988a1663 1y ago
  I hate how much I lean into VSCode, but the Python interactive mode gets you a really good live coding environment. Instead of Jupyter cells, you have a regular .py file with chunks of code prefixed with a `# %%`. VSCode gives you a similar experience to a notebook, with the same controls (Run Above Cells, Restart and Run All, etc). So something like # %% import polars as pl # %% df = pl.DataFrame() df.shape # %% def foobar(): return 1 Since it is a regular .py file all of your existing tooling will work with it. The one thing you lose vs a Jupyter notebook is saved output. I mostly use these .py files, but have a few .ipynb notebook files for when I want to commit the output from some important task.
- akshayka 1y ago
  You can not use the AI features, nothing is enabled by default (you have to bring your own keys)
- daxfohl 1y ago
  Reading the article, I don't think it has AI. They've just made the tools in a way that AI assistants can also use them, and so fix linting errors without anyone needing to fine-tune the LLM on the syntax. That's actually pretty slick. I've been wondering how we could avoid blocking innovation in programming languages because of the death cycle of "no training data on language -> LLM can't learn language -> Assistant can't code language -> nobody uses language -> no training data on language".
  - 1y ago
ZeroCool2u 1y ago
Any updates on the revamped VS Code extension? That's kind of a deal breaker for me.
- dmadisetti 1y ago
  Soft launched in pre-release, but a full version is coming!
- Haleshot 11mo ago
  v.07 is live now on both VS Code and Open VSX marketplaces. https://github.com/marimo-team/marimo-lsp/ https://github.com/marimo-team/marimo-lsp/

31 points by dmadisetti 1y ago

marimo notebooks double as python Python programs, which means they can be lint-ed like any other code. `marimo check` catches notebook-specific issues (variable redefinition across cells, circular dependencies) with Rust-inspired (and uv; uv is amazing) error messages. It outputs JSON for AI agents to self-correct, integrates into CI pipelines, and includes `--fix` flags for automatic repairs. The team is already using it in their own CI and seeing Claude Code iterate on notebooks without human intervention.

aitchnyu 1y ago
IME Agents work fine on human readable error messages.
sixo 1y ago
Unfortunately the code in Marimo notebooks implements cross-variable references as untyped function arguments, meaning that nothing passed between cells can be type-safe. This makes it very hard to use tooling directly on the Python representation of notebooks.
- dmadisetti 1y ago
  Arguments are typed if explicitly specified in definition, e.g. ```python @app.cell def my_cell(): x : int = 1 return (x,) @app.cell def consumer(x : int): y = x return ``` We've talked about building out implicit typing into our serialization- but that may be a larger can of worms.
  - sixo 1y ago
    That would be wonderful. A lot of the arguments are just imports, and these would have probably have the largest upside. It would be great if there was a construct like import numpy @app.cell def my_cell(np: types.ModuleType[numpy]): ... editor treats np like `import numpy as np` ... I do not use Python enough to know if something like this can be hacked together. But if not, I imagine could be PIPed into the standard library if a convincing case were made for it.
    - dmadisetti 1y ago
      We've addressed this too by allowing a setup cell: ... with app.setup: import numpy as np @app.cell def cell_uses_np(): # no np needed in signature result = np.... The best part about this pattern is that it enables "pure" (only dependent on setup cell) functions to be addressable: with app.setup: from typing import Optional, Any @app.function def my_fn(arg: Optional[Any]) -> Any: ... but nice convergent design :) glad to see to see we're addressing pain points
      - sixo 1y ago
        ah, I recall seeing that feature but had not taken a look at it for imports. Imports ARE the case where untyped cell arguments are most annoying, but of course it would be nice to get it for free in all cases.

wodenokoto 1y ago

I’m very interested in moving beyond Jupyter notebooks so I have my eye on marimo. My understanding is it’s just python code with a bit of notebook hints and an idea that gives the notebook experience. So I didn’t quite catch from the article why ruff and pylons aren’t enough.

akshayka 1y ago
Hi! Thanks for your interest. marimo is much more than that — unlike traditional notebooks, marimo is "reactive", meaning it models notebooks as dataflow graphs and keeps code an outputs in sync. Moreover, marimo notebooks are not "just notebooks". They can be seamlessly run as interactive web apps or as Python scripts. Here is our original ShowHN post that explains what marimo is all about: https://news.ycombinator.com/item?id=38971966 https://news.ycombinator.com/item?id=38971966 Blog that goes deeper: https://marimo.io/blog/lessons-learned https://marimo.io/blog/lessons-learned
- daxfohl 1y ago
  I was just thinking about something like this a couple hours ago. Jeremy Howard (fast.ai, etc) posted a few weeks ago about his new AI class [1] and I was pumped. I went through the fast.ai course a couple years ago and had been wondering if another would ever come out, and the overview of this sounded really interesting. But the first thing the announcement went into was Jupyter Notebooks, and it kept on and on about them and I rolled my eyes. I remember they were my least favorite part of the fast.ai class. They've always felt awkward to me, being used to a code editor and all the functionality it provides, and I'm sure I'm not alone. I'm sure Jupyter is great for fast iteration if that's what you do all day, but if you spend most of your life in a standard code editor, you really want something that starts from there and adds a bit of interactivity on top, rather than jumping to a completely different concept. I haven't tried marimo yet, but I might give it a shot for my next project. [1] https://news.ycombinator.com/item?id=45455810 https://news.ycombinator.com/item?id=45455810
  - spwa4 1y ago
    Just use VSCode, which will give you full language server editing in ipynb files?

loveparade 1y ago

I wish this didn't have AI in it. I've been looking for a Jupyter alternative that is pure python and can be modified from a regular text editor. Jupytext works okay, but I miss the advanced Jupyter features. But I really don't want to deal with yet another AI assistant, especially not a custom one when I'm already using Claude/etc from the CLI and I want those agents to help me edit the notebooks. Take out all the AI stuff and I'd give it a try. I use AI coding agents as my daily driver, but I really don't need this AI enshittification in every tool/library I'm using.

3eb7988a1663 1y ago
I hate how much I lean into VSCode, but the Python interactive mode gets you a really good live coding environment. Instead of Jupyter cells, you have a regular .py file with chunks of code prefixed with a `# %%`. VSCode gives you a similar experience to a notebook, with the same controls (Run Above Cells, Restart and Run All, etc). So something like # %% import polars as pl # %% df = pl.DataFrame() df.shape # %% def foobar(): return 1 Since it is a regular .py file all of your existing tooling will work with it. The one thing you lose vs a Jupyter notebook is saved output. I mostly use these .py files, but have a few .ipynb notebook files for when I want to commit the output from some important task.
akshayka 1y ago
You can not use the AI features, nothing is enabled by default (you have to bring your own keys)
daxfohl 1y ago
Reading the article, I don't think it has AI. They've just made the tools in a way that AI assistants can also use them, and so fix linting errors without anyone needing to fine-tune the LLM on the syntax. That's actually pretty slick. I've been wondering how we could avoid blocking innovation in programming languages because of the death cycle of "no training data on language -> LLM can't learn language -> Assistant can't code language -> nobody uses language -> no training data on language".
- 1y ago

ZeroCool2u 1y ago

Any updates on the revamped VS Code extension? That's kind of a deal breaker for me.

dmadisetti 1y ago
Soft launched in pre-release, but a full version is coming!
Haleshot 11mo ago
v.07 is live now on both VS Code and Open VSX marketplaces. https://github.com/marimo-team/marimo-lsp/ https://github.com/marimo-team/marimo-lsp/