7 ms·

I decided not to commercialize nbdev

99 points by tim_sw 3y ago

enriquto 3y ago
> For practical purposes, you cannot perform code reviews of notebooks on GitHub without purchasing a tool called ReviewNB. This is false. If you save your notebooks with jupytext, they are just plain python files that can be reviewed normally.
- crimsoneer 3y ago
  ... so you're not doing reviews on your notebooks. Your downloading a plugin that converts your notebooks, then doing reviews there.
  - qbasic_forever 3y ago
    No, jupytext converts the binary .ipynb format into a text format that can be diffed and reasoned about like any other code in a git repo. It hooks into jupyter lab such that it automatically keeps both the .ipynb and text source version of a document in sync. How you review the text version is up to whatever source control system and other tools you use.
    - protoplancton 3y ago
      Note: ipynb is not binary, it's plain text containing a json object
- nl 3y ago
  Heavy notebook user here. This isn't the same at all. Often in a well done notebook you'll have data display etc which doesn't translate well to a plain python file.
- hoosieree 3y ago
  Alternatively, use org-mode and save your file with the save button.
  - mloncode 3y ago
    I think a notebook is VERY different than an ephemeral REPL. I think that is worth considering when thinking about generating docs, tests and source code from a single source of truth (as opposed to separate files).
    - hoosieree 3y ago
      Do you know what org-mode is? [editing to be less snarky and more helpful] I agree with your comment, but I do in fact use org-mode as a single source of truth document that can export documentation, tests, source code, and all that jazz. From what I can tell nbdev and org-mode are about the same in terms of features. It doesn't need any special post-processing to make git-diff useful, because org-mode's file format is plain text.
      - mloncode 3y ago
        No snark interpreted at all. Yes I am well aware of org mode and love it! I think it is fantastic. There are some key differences that I think make notebooks a bit more interesting - however it sounds like you've made a system you are happy with? I would love to see what you have! Mind sharing it?
        hoosieree 3y ago
        Cool, I also wanted to say that I think nbdev looks great, and while I think it and org-mode get similar results, the user experience is surely quite different compared to emacs. My org-mode workflows start with doom-emacs[1] because otherwise Emacs customization is too much of a time sink. I've written a couple peer-reviewed papers in org-mode (exporting to pdf), my blog is written org-mode (exported to html), and I've done the "literate" thing witih code+docs+tests in one file. For me, org-mode is better as a file format because it's plain text. I don't mind making quick edits in GitHub or vim if I'm away from Emacs for some reason. But it's definitely best to work with org-mode from within Emacs. Notebooks are more enjoyable to look at while you're editing them, whereas org-mode generally needs a separate "export" step to be pretty. Here's a representative example from a blog post: https://gitlab.com/hoosieree/hoosieree.gitlab.io/-/blob/master/content/posts/2023-05-21-Stack-VM-Language.org https://gitlab.com/hoosieree/hoosieree.gitlab.io/-/blob/mast... (result is here: https://alexshroyer.com/posts/2023-05-21-Stack-VM-Language.html https://alexshroyer.com/posts/2023-05-21-Stack-VM-Language.h...) The svg images (which gitlab renders) are actually generated by a code block in the org-mode file: #+CAPTION: N times M compilers #+headers: :file ../images/n_times_m.svg :cmdline -Tsvg #+begin_src dot :exports results digraph { node [fontname="Comic Sans MS" shape=record]; L1[label="Language1"]; L2[label="Language2"]; node [shape=oval]; A1[label="Arch1"]; A2[label="Arch2"]; A3[label="Arch3"]; L1->A1; L1->A2; L1->A3; L2->A1; L2->A2; L2->A3; } #+end_src I put my cursor in there and hit Enter, and it generates the svg.
marktangotango 3y ago
What is nbdev? > nbdev is an innovative software development framework for Python that embraces literate and exploratory programming. I worked on nbdev from 2020-2023 with Jeremy Howard and, later, Wasim Lorgat. I had the privilege and excitement of exploring the boundaries of developer tools and exploratory programming while working with very talented software engineers. In addition to creating a tool many people enjoyed, I enjoyed using nbdev for personal and professional projects. So I still wasn't sure and navigated to nbdev.fast.ai: > Create delightful software with Jupyter Notebooks Ah "nb" is notebook and we're talking about jupyter notebooks here!
- mloncode 3y ago
  Yes, I encourage you to look at some example projects made with nbdev incase that is helpful: - fastcore: https://github.com/fastai/fastcore https://github.com/fastai/fastcore - the nbs/ folder contains the source files that generate source code, docs and tests. - ghapi: https://github.com/fastai/ghapi https://github.com/fastai/ghapi - the notebooks that create source code, docs and tests are located in the root.
- WatchDog 3y ago
  I still don't really understand what nbdev is, I understand the idea of using Jupyter Notebooks to run code, but what does nbdev itself actually do?
  - hamelsmu 3y ago
    Here is a 15 minute explanation (you can watch it on 2x speed in even less time) https://youtu.be/C8kDPmb_IKU https://youtu.be/C8kDPmb_IKU
  - inshadows 3y ago
    [dead]
mloncode 3y ago
Hello! Hamel here, author of the original post. I'm surprised to see this on the front page of HN! Anyways, feel free to AMA!
- greazy 3y ago
  Have you considered approaching Anaconda? This is right up their alley.
  - hamelsmu 3y ago
    Good question! Yes we did talk to Anaconda (we talked to Peter Wang), and they seemed interested in related but slightly different approaches. Some of their forays into exploratory programming are pyscript https://pyscript.com/ https://pyscript.com/, the also acquired python anywhere https://www.pythonanywhere.com/ https://www.pythonanywhere.com/, which is sort of like Wordpress for Python developers! We ended up partnering with Posit (formerly RStudio) a bit more on this at this was a bit more natural in terms of alignment of product and vision. You should definitely keep an eye on Posit (https://posit.co/ https://posit.co/) to see what they come up with in this space. I don't think there was an opportunity to "be acquired" (not sure if that was what you were asking) because we didn't get far enough down the commercialization path to justify that.
- malshe 3y ago
  It's great to learn that you are collaborating with Posit (previously RStudio)! I love where Quarto is going. I use it even in VSCode.
  - mloncode 3y ago
    Yes Quarto is one of my favorite tools. My blog (the topic of this thread) is made with Quarto!
- KerrickStaley 3y ago
  Jupytext provides a system for version controlling notebooks, code reviewing them, and using them in the context of a larger Python project, without imposing any constraints on the larger project, because Jupytext saves notebooks as plain .py files. I ended up choosing Jupytext for a personal project because it seemed less complex than nbdev. (I never made it through the 90 minute nbdev introductory video: https://www.youtube.com/watch?v=l7zS8Ld4_iA https://www.youtube.com/watch?v=l7zS8Ld4_iA). Are there advantages of nbdev over Jupytext that I'm missing out on?
crimsoneer 3y ago
I really like nbdev, and the entire paradigm it encourages. I wish the onboarding was slightly smoother (a slick interactive tutorial type thing in vscode would do it a world of good) but it's now my default project base when I'm working on personal projects.
- mloncode 3y ago
  That's great to hear!
da39a3ee 3y ago
> it was (and still is) a pain to version control notebooks in a way that’s conducive to collaboration. For practical purposes, you cannot perform code reviews of notebooks on GitHub without purchasing a tool called ReviewNB. I have always been really surprised that the IPython developers made the choice to use JSON despite it being unusable with git (and despite them being excellent software engineers). This is one reason I never use notebooks (though not as big as the difficulty of developing with all that long-lived mutable state). Have they ever commented on this?

99 points by tim_sw 3y ago

> For practical purposes, you cannot perform code reviews of notebooks on GitHub without purchasing a tool called ReviewNB. This is false. If you save your notebooks with jupytext, they are just plain python files that can be reviewed normally.

crimsoneer 3y ago
... so you're not doing reviews on your notebooks. Your downloading a plugin that converts your notebooks, then doing reviews there.
- qbasic_forever 3y ago
  No, jupytext converts the binary .ipynb format into a text format that can be diffed and reasoned about like any other code in a git repo. It hooks into jupyter lab such that it automatically keeps both the .ipynb and text source version of a document in sync. How you review the text version is up to whatever source control system and other tools you use.
  - protoplancton 3y ago
    Note: ipynb is not binary, it's plain text containing a json object
nl 3y ago
Heavy notebook user here. This isn't the same at all. Often in a well done notebook you'll have data display etc which doesn't translate well to a plain python file.
hoosieree 3y ago
Alternatively, use org-mode and save your file with the save button.
- mloncode 3y ago
  I think a notebook is VERY different than an ephemeral REPL. I think that is worth considering when thinking about generating docs, tests and source code from a single source of truth (as opposed to separate files).
  - hoosieree 3y ago
    Do you know what org-mode is? [editing to be less snarky and more helpful] I agree with your comment, but I do in fact use org-mode as a single source of truth document that can export documentation, tests, source code, and all that jazz. From what I can tell nbdev and org-mode are about the same in terms of features. It doesn't need any special post-processing to make git-diff useful, because org-mode's file format is plain text.
    - mloncode 3y ago
      No snark interpreted at all. Yes I am well aware of org mode and love it! I think it is fantastic. There are some key differences that I think make notebooks a bit more interesting - however it sounds like you've made a system you are happy with? I would love to see what you have! Mind sharing it?
      - hoosieree 3y ago
        Cool, I also wanted to say that I think nbdev looks great, and while I think it and org-mode get similar results, the user experience is surely quite different compared to emacs. My org-mode workflows start with doom-emacs[1] because otherwise Emacs customization is too much of a time sink. I've written a couple peer-reviewed papers in org-mode (exporting to pdf), my blog is written org-mode (exported to html), and I've done the "literate" thing witih code+docs+tests in one file. For me, org-mode is better as a file format because it's plain text. I don't mind making quick edits in GitHub or vim if I'm away from Emacs for some reason. But it's definitely best to work with org-mode from within Emacs. Notebooks are more enjoyable to look at while you're editing them, whereas org-mode generally needs a separate "export" step to be pretty. Here's a representative example from a blog post: https://gitlab.com/hoosieree/hoosieree.gitlab.io/-/blob/master/content/posts/2023-05-21-Stack-VM-Language.org https://gitlab.com/hoosieree/hoosieree.gitlab.io/-/blob/mast... (result is here: https://alexshroyer.com/posts/2023-05-21-Stack-VM-Language.html https://alexshroyer.com/posts/2023-05-21-Stack-VM-Language.h...) The svg images (which gitlab renders) are actually generated by a code block in the org-mode file: #+CAPTION: N times M compilers #+headers: :file ../images/n_times_m.svg :cmdline -Tsvg #+begin_src dot :exports results digraph { node [fontname="Comic Sans MS" shape=record]; L1[label="Language1"]; L2[label="Language2"]; node [shape=oval]; A1[label="Arch1"]; A2[label="Arch2"]; A3[label="Arch3"]; L1->A1; L1->A2; L1->A3; L2->A1; L2->A2; L2->A3; } #+end_src I put my cursor in there and hit Enter, and it generates the svg.

marktangotango 3y ago

What is nbdev? > nbdev is an innovative software development framework for Python that embraces literate and exploratory programming. I worked on nbdev from 2020-2023 with Jeremy Howard and, later, Wasim Lorgat. I had the privilege and excitement of exploring the boundaries of developer tools and exploratory programming while working with very talented software engineers. In addition to creating a tool many people enjoyed, I enjoyed using nbdev for personal and professional projects. So I still wasn't sure and navigated to nbdev.fast.ai: > Create delightful software with Jupyter Notebooks Ah "nb" is notebook and we're talking about jupyter notebooks here!

mloncode 3y ago
Yes, I encourage you to look at some example projects made with nbdev incase that is helpful: - fastcore: https://github.com/fastai/fastcore https://github.com/fastai/fastcore - the nbs/ folder contains the source files that generate source code, docs and tests. - ghapi: https://github.com/fastai/ghapi https://github.com/fastai/ghapi - the notebooks that create source code, docs and tests are located in the root.
WatchDog 3y ago
I still don't really understand what nbdev is, I understand the idea of using Jupyter Notebooks to run code, but what does nbdev itself actually do?
- hamelsmu 3y ago
  Here is a 15 minute explanation (you can watch it on 2x speed in even less time) https://youtu.be/C8kDPmb_IKU https://youtu.be/C8kDPmb_IKU
- inshadows 3y ago
  [dead]

mloncode 3y ago

Hello! Hamel here, author of the original post. I'm surprised to see this on the front page of HN! Anyways, feel free to AMA!

greazy 3y ago
Have you considered approaching Anaconda? This is right up their alley.
- hamelsmu 3y ago
  Good question! Yes we did talk to Anaconda (we talked to Peter Wang), and they seemed interested in related but slightly different approaches. Some of their forays into exploratory programming are pyscript https://pyscript.com/ https://pyscript.com/, the also acquired python anywhere https://www.pythonanywhere.com/ https://www.pythonanywhere.com/, which is sort of like Wordpress for Python developers! We ended up partnering with Posit (formerly RStudio) a bit more on this at this was a bit more natural in terms of alignment of product and vision. You should definitely keep an eye on Posit (https://posit.co/ https://posit.co/) to see what they come up with in this space. I don't think there was an opportunity to "be acquired" (not sure if that was what you were asking) because we didn't get far enough down the commercialization path to justify that.
malshe 3y ago
It's great to learn that you are collaborating with Posit (previously RStudio)! I love where Quarto is going. I use it even in VSCode.
- mloncode 3y ago
  Yes Quarto is one of my favorite tools. My blog (the topic of this thread) is made with Quarto!
KerrickStaley 3y ago
Jupytext provides a system for version controlling notebooks, code reviewing them, and using them in the context of a larger Python project, without imposing any constraints on the larger project, because Jupytext saves notebooks as plain .py files. I ended up choosing Jupytext for a personal project because it seemed less complex than nbdev. (I never made it through the 90 minute nbdev introductory video: https://www.youtube.com/watch?v=l7zS8Ld4_iA https://www.youtube.com/watch?v=l7zS8Ld4_iA). Are there advantages of nbdev over Jupytext that I'm missing out on?

crimsoneer 3y ago

I really like nbdev, and the entire paradigm it encourages. I wish the onboarding was slightly smoother (a slick interactive tutorial type thing in vscode would do it a world of good) but it's now my default project base when I'm working on personal projects.

mloncode 3y ago
That's great to hear!

da39a3ee 3y ago

> it was (and still is) a pain to version control notebooks in a way that’s conducive to collaboration. For practical purposes, you cannot perform code reviews of notebooks on GitHub without purchasing a tool called ReviewNB. I have always been really surprised that the IPython developers made the choice to use JSON despite it being unusable with git (and despite them being excellent software engineers). This is one reason I never use notebooks (though not as big as the difficulty of developing with all that long-lived mutable state). Have they ever commented on this?