5 ms·

Nbdev: Use Notebooks for Literate Programming

74 points by pcr910303 7y ago

spv 7y ago
Can nbdev be used with other Jupyter kernels, for eg like Julia. My main issue with notebooks is putting them under version control in git. This seems like it could help with that.
- nerdponx 7y ago
  I use NBDime to make readable diffs of Jupyter notebooks https://nbdime.readthedocs.io/en/latest/ https://nbdime.readthedocs.io/en/latest/
  - spv 7y ago
    this is exactly what I was looking for. thanks.
- amirathi 7y ago
  Yes, nbdev can help with merge conflicts for notebooks [1]. Also checkout, - GitPlus[2] - JupyterLab extension for git version control - ReviewNB[3] - For notebooks diffs & commenting Disclaimer: I built GitPlus & ReviewNB [1] https://nbdev.fast.ai/#Avoiding-and-handling-git-conflicts https://nbdev.fast.ai/#Avoiding-and-handling-git-conflicts [2] https://github.com/ReviewNB/jupyterlab-gitplus/ https://github.com/ReviewNB/jupyterlab-gitplus/ [3] https://www.reviewnb.com/ https://www.reviewnb.com/
  - spv 7y ago
    reviewnb looks interesting. cool project.
- mafm 7y ago
  Jupytext (https://github.com/mwouts/jupytext https://github.com/mwouts/jupytext) solves most of the problems with managing notebooks in git.
- JPKab 7y ago
  nbdev works with any kernel, because it only cares about the notebook files, which as you know are just JSON. I've been using it with xeus-python kernel lately. Edit: Forgot the Git part. It was designed to handle the git limitations in notebooks as well. In fact, one of the commands in the tutorials is to set up it's built-in git hooks, which will run special clean commands on commits.
erikgaas 7y ago
Been using this for work projects. A lot of raised eyebrows when people hear jupyter first development, but the automated docs, flexibility with prose, inline testing, out of the box pip packaging, and git integration make it well worth it. A bit of a learning curve, but very rewarding.
- taeric 7y ago
  Didn't pydoc already have inline testing?
  - CGamesPlay 7y ago
    Sure, but git already had the ability to commit files to it and pip already had the ability to create packages.
- smabie 7y ago
  I've written a lot of jupyter notebooks and honestly, emacs+org-mode is way better. Maybe if jupyter wasn't in a webbrowser, had vim or emacs keybindings, it would be better. Also, I'm not sure the idea of notebooks is even a good idea: it's very easy to get into a inconsistent state and with no textual source of truth it can be very difficult getting back. Though, thinking about it, the real problem is that when I'm using a real editor (emacs), I feel like a wizard, I know it like the back of my hand and have any number of extensions and libraries I can use. With jupyter, I'm always fighting something and there's no meaningful way to configure it to do what you want. Also, the intellisense sucks. In addition, and maybe this is silly, but I find using a webbrowser to write code to be distasteful.
  - ginko 7y ago
    I'm feeling the same way. I really like the idea of jupyter, but it should be a native application, not something running in a browser. Maybe it's also me not being used to working with notebooks, but I find it strange that I manually have to reevaluate all following cells when I change an earlier one. Shouldn't that just happen automatically? I dabbled a bit with EIN[1], an emacs client for Jupyter, but it didn't work all that well for me. In particular it didn't work well with my dark color scheme and you still needed to run a jupyter server to connect to. [1] http://millejoh.github.io/emacs-ipython-notebook/ http://millejoh.github.io/emacs-ipython-notebook/
bpesquet 7y ago
Are there some example projects available somewhere? I cannot find any (apart from nbdev itself).
foxes 7y ago
I'm sorry, slight rant, but I do not think notebooks are good for a completed piece of software. Maybe it's good for development/brainstorming but once you are done the code needs to be extracted into a more traditional format. In research I have seen countless mathematica/python notebooks which have been released into the wild, and they are all over the place. Maybe software developers will structure it better, but I'm kind of sick of seeing an incoherent spaghetti state. Especially since there is no strict ordering of the evaluation of cells. Maybe it would be nice if notebooks could break from the traditional "page of working" format. It could display as some tree/graph which orders the state/dependency in evaluation. Otherwise it needs an "export" button.
- conradludgate 7y ago
  nbdev has exactly that export functionality. https://nbdev.fast.ai/export/ https://nbdev.fast.ai/export/
- dr_zoidberg 7y ago
  What you complain about is more about the author of the notebook being disorganized rather than the format. The format also allows that to happen, but it's a lack of discipline on the author that creates that mess. Many people don't care about reproducibility, or having a paper that can be simply read and understood. They just care about getting their paper out. Part to blame is also on the reviewers who don't really evaluate this as part of the paper (assuming they're given access and the opportunity to evaluate the code, that is!). I recently reviewed a paper for a conference and it was a mess in every aspect. I wish they had included the .ipynb file. Instead, they provided mangled (both by cropping and jpeg compression) screenshots of the notebook. The code was an utter mess (instead of doing X_prime = X[10:100], they did X_prime = [X[10], X[11], X[12], ..., X[98], X[99]]). I gave that mess the strong reject it deserved (the rest of the paper was on par with the code, if not worse). I'm still waiting to see if the other reviewes will do the right thing, or they'll jump on the fad that """deep learning""" (they used 90's machine learning methods at best) has become.
  - dr_zoidberg 7y ago

74 points by pcr910303 7y ago

spv 7y ago

Can nbdev be used with other Jupyter kernels, for eg like Julia. My main issue with notebooks is putting them under version control in git. This seems like it could help with that.

nerdponx 7y ago
I use NBDime to make readable diffs of Jupyter notebooks https://nbdime.readthedocs.io/en/latest/ https://nbdime.readthedocs.io/en/latest/
- spv 7y ago
  this is exactly what I was looking for. thanks.
amirathi 7y ago
Yes, nbdev can help with merge conflicts for notebooks [1]. Also checkout, - GitPlus[2] - JupyterLab extension for git version control - ReviewNB[3] - For notebooks diffs & commenting Disclaimer: I built GitPlus & ReviewNB [1] https://nbdev.fast.ai/#Avoiding-and-handling-git-conflicts https://nbdev.fast.ai/#Avoiding-and-handling-git-conflicts [2] https://github.com/ReviewNB/jupyterlab-gitplus/ https://github.com/ReviewNB/jupyterlab-gitplus/ [3] https://www.reviewnb.com/ https://www.reviewnb.com/
- spv 7y ago
  reviewnb looks interesting. cool project.
mafm 7y ago
Jupytext (https://github.com/mwouts/jupytext https://github.com/mwouts/jupytext) solves most of the problems with managing notebooks in git.
JPKab 7y ago
nbdev works with any kernel, because it only cares about the notebook files, which as you know are just JSON. I've been using it with xeus-python kernel lately. Edit: Forgot the Git part. It was designed to handle the git limitations in notebooks as well. In fact, one of the commands in the tutorials is to set up it's built-in git hooks, which will run special clean commands on commits.

erikgaas 7y ago

Been using this for work projects. A lot of raised eyebrows when people hear jupyter first development, but the automated docs, flexibility with prose, inline testing, out of the box pip packaging, and git integration make it well worth it. A bit of a learning curve, but very rewarding.

taeric 7y ago
Didn't pydoc already have inline testing?
- CGamesPlay 7y ago
  Sure, but git already had the ability to commit files to it and pip already had the ability to create packages.
smabie 7y ago
I've written a lot of jupyter notebooks and honestly, emacs+org-mode is way better. Maybe if jupyter wasn't in a webbrowser, had vim or emacs keybindings, it would be better. Also, I'm not sure the idea of notebooks is even a good idea: it's very easy to get into a inconsistent state and with no textual source of truth it can be very difficult getting back. Though, thinking about it, the real problem is that when I'm using a real editor (emacs), I feel like a wizard, I know it like the back of my hand and have any number of extensions and libraries I can use. With jupyter, I'm always fighting something and there's no meaningful way to configure it to do what you want. Also, the intellisense sucks. In addition, and maybe this is silly, but I find using a webbrowser to write code to be distasteful.
- ginko 7y ago
  I'm feeling the same way. I really like the idea of jupyter, but it should be a native application, not something running in a browser. Maybe it's also me not being used to working with notebooks, but I find it strange that I manually have to reevaluate all following cells when I change an earlier one. Shouldn't that just happen automatically? I dabbled a bit with EIN[1], an emacs client for Jupyter, but it didn't work all that well for me. In particular it didn't work well with my dark color scheme and you still needed to run a jupyter server to connect to. [1] http://millejoh.github.io/emacs-ipython-notebook/ http://millejoh.github.io/emacs-ipython-notebook/

bpesquet 7y ago

Are there some example projects available somewhere? I cannot find any (apart from nbdev itself).

foxes 7y ago

I'm sorry, slight rant, but I do not think notebooks are good for a completed piece of software. Maybe it's good for development/brainstorming but once you are done the code needs to be extracted into a more traditional format. In research I have seen countless mathematica/python notebooks which have been released into the wild, and they are all over the place. Maybe software developers will structure it better, but I'm kind of sick of seeing an incoherent spaghetti state. Especially since there is no strict ordering of the evaluation of cells. Maybe it would be nice if notebooks could break from the traditional "page of working" format. It could display as some tree/graph which orders the state/dependency in evaluation. Otherwise it needs an "export" button.

conradludgate 7y ago
nbdev has exactly that export functionality. https://nbdev.fast.ai/export/ https://nbdev.fast.ai/export/
dr_zoidberg 7y ago
What you complain about is more about the author of the notebook being disorganized rather than the format. The format also allows that to happen, but it's a lack of discipline on the author that creates that mess. Many people don't care about reproducibility, or having a paper that can be simply read and understood. They just care about getting their paper out. Part to blame is also on the reviewers who don't really evaluate this as part of the paper (assuming they're given access and the opportunity to evaluate the code, that is!). I recently reviewed a paper for a conference and it was a mess in every aspect. I wish they had included the .ipynb file. Instead, they provided mangled (both by cropping and jpeg compression) screenshots of the notebook. The code was an utter mess (instead of doing X_prime = X[10:100], they did X_prime = [X[10], X[11], X[12], ..., X[98], X[99]]). I gave that mess the strong reject it deserved (the rest of the paper was on par with the code, if not worse). I'm still waiting to see if the other reviewes will do the right thing, or they'll jump on the fad that """deep learning""" (they used 90's machine learning methods at best) has become.
- dr_zoidberg 7y ago