5 ms·

This is definitely one of my concerns too. Ad hoc code inside these notebooks is almost completely unmanageable from any reasonable software maintenance perspec

by zmmmmm 8y ago

This is definitely one of my concerns too. Ad hoc code inside these notebooks is almost completely unmanageable from any reasonable software maintenance perspective, and refactoring code out of them is prohibitively difficult as well. I really want something to emerge that combines the best of both worlds of an IDE and notebook development, but there isn't anything close currently.

staticassertion 8y ago
What? What are you using Jupyter notebooks for where you want maintenance? They should be records of data analysis/ procedures, not code that runs in production or something.
- ekianjo 8y ago
  exactly. I use it for quick data exploration and experimentation but it remains at that level.
  - gaius 8y ago
    People do that in Excel too and the next thing you know a spreadsheet is managing a portfolio or being used as the basis for published science!
    - ekianjo 8y ago
      Its not the fault of the tools if users dont know any better.
      - TeMPOraL 8y ago
        It's also not the fault of the tools if there are no better alternatives.
      - fsloth 8y ago
        So what would you suggest a financial modeler, who has no experience of any programming environments, to use instead? The value of Excel is that is a zero-config tool, available everywhere as 'standard' business installation, allows very quick iteration with visual output in certain range of tasks, is battle tested in millions of computers... etc. And everyone else is using it too. For a programmer it's easy to suggest to just pick some good language. Once you know two programming languages you can navigate pretty soon fluently in third. But picking up the first language fluency? That's hard. You can't just suggest to use python or such. You need to provide a tool that holds hands, has lots tutorials available etc. Visual programming is not necessary the answer either. I've understood that without discipline LabView programs become actual visual spaghetti pretty quickly.
        ekianjo 8y ago
        of course its hard. Just like its hard a new language thats is far from your native tongue. But we should not be scared of learning something difficult, because the returns are well worth the investment. There is only so much you can do in Excel and if you want to advance your career you need to move past it even if you have no programming background. Additionally, its never been easier than in 2018 to find free resources to learn. The only things you need are awareness, time, and will.
    - kgwgk 8y ago
      I can’t tell if this is sarcasm or not... I’ll assume it is and upvote :-)
    - balaba 8y ago
      everyone knows that excel is a drawing program https://www.thisiscolossal.com/2017/12/tatsuo-horiuchi-excel-artist/ https://www.thisiscolossal.com/2017/12/tatsuo-horiuchi-excel...
- laichzeit0 8y ago
  Maybe read the article. They are experimenting with putting the notebooks directly into production ala bash script. I don’t think this is a great idea either.
  - staticassertion 8y ago
    Yeah, I read it after, silly of me to comment first.
- brianberns 8y ago
  Did you read the article? "A Netflix engineer described how they have replaced Bash scripts with Jupyter notebooks for ETL pipelines and cron jobs."
stewbrew 8y ago
Some simpler notebook-like environments stay closer to source code in that they basically are source code with interleaved results (e.g. as comments). IMHO they hit a sweet spot between REPLs and those notebook environments inspired by mathematica, maple and similar more mathematically oriented software products.
- TeMPOraL 8y ago
  That's just a REPL with basic editor integration (eval at point, paste result). Surprisingly unpopular outside Emacs/Lisp land.
  - stewbrew 8y ago
    Well yes, but these notebooks are little more, aren't they.
dangirsh 8y ago
ob-ipython enables IDE-like editing features within the code cells. It's embedded in a polyglot, git-friendly, literate programming environment called Org-mode. I use it every day and love it. Other goodies: - easily manage multiple kernels (in different languages / machines) in one file - tree-based organization manages complexity better than linear notebooks - no browser in sight (unless you need interactive widgets) - highly exportable, including to ipynb via ox-ipynb Downsides: - Emacs-only - small user base / limited docs - can't easily import from ipynb - async cell execution support is early-stage See Scimax ipython for examples
- chimprich 8y ago
  Interesting. I use emacs for hacking Python but I've never heard of this tool. Do you actually use this for software development or is it more of a data-science-type exploration tool?
  - d0mine 8y ago
    related "literate devops in emacs" https://youtu.be/dljNabciEGg https://youtu.be/dljNabciEGg
  - dangirsh 8y ago
    I use Org for software development whenever I can, which is currently everyday. Usually, new code starts in cells with some Org-managed context (e.g. a Jupyter kernel in a remote container with some DB/service access). This is done using the :session code cell keyword, which works per subtree. Managing remote sessions like this generally keeps me away from terminals. Surrounding the cell are various mini-dashboards with useful docs / links / commands for that part of the project. Since Org supports embedding elisp and shell commands in clickable links [1], these mini-dashboards can be made very quickly. Org lets me edit the code using the proper Emacs mode for its language, while pulling dynamic completion / docs from the Jupyter kernel. Just like Jupyter notebooks, I can view rich outputs from the cells in-line. I can then name the outputs and make them inputs to other cells, including ones in different languages / kernels. AFAIK that's an Org-only trick. Most code eventually finds it's way to normal source files (see Org's "tangle" feature). This feels more natural than moving code from notebooks since, again, the cell editing mode is the same as the one for source files. Org's tree-manipulation capabilities + support for multiple sessions means that (so far) I've only ever needed 1 Org file per project. I track this in git, which is simple since Org is just plain-text. To share with non-Org users, I usually export to ipynb [2] or, for static docs, HTML [3]. [1]: https://orgmode.org/manual/External-links.html https://orgmode.org/manual/External-links.html [2]: https://github.com/jkitchin/ox-ipynb https://github.com/jkitchin/ox-ipynb [3]: https://github.com/fniessen/org-html-themes https://github.com/fniessen/org-html-themes

by zmmmmm 8y ago

staticassertion 8y ago

What? What are you using Jupyter notebooks for where you want maintenance? They should be records of data analysis/ procedures, not code that runs in production or something.

ekianjo 8y ago
exactly. I use it for quick data exploration and experimentation but it remains at that level.
- gaius 8y ago
  People do that in Excel too and the next thing you know a spreadsheet is managing a portfolio or being used as the basis for published science!
  - ekianjo 8y ago
    Its not the fault of the tools if users dont know any better.
    - TeMPOraL 8y ago
      It's also not the fault of the tools if there are no better alternatives.
    - fsloth 8y ago
      So what would you suggest a financial modeler, who has no experience of any programming environments, to use instead? The value of Excel is that is a zero-config tool, available everywhere as 'standard' business installation, allows very quick iteration with visual output in certain range of tasks, is battle tested in millions of computers... etc. And everyone else is using it too. For a programmer it's easy to suggest to just pick some good language. Once you know two programming languages you can navigate pretty soon fluently in third. But picking up the first language fluency? That's hard. You can't just suggest to use python or such. You need to provide a tool that holds hands, has lots tutorials available etc. Visual programming is not necessary the answer either. I've understood that without discipline LabView programs become actual visual spaghetti pretty quickly.
      - ekianjo 8y ago
        of course its hard. Just like its hard a new language thats is far from your native tongue. But we should not be scared of learning something difficult, because the returns are well worth the investment. There is only so much you can do in Excel and if you want to advance your career you need to move past it even if you have no programming background. Additionally, its never been easier than in 2018 to find free resources to learn. The only things you need are awareness, time, and will.
  - kgwgk 8y ago
    I can’t tell if this is sarcasm or not... I’ll assume it is and upvote :-)
  - balaba 8y ago
    everyone knows that excel is a drawing program https://www.thisiscolossal.com/2017/12/tatsuo-horiuchi-excel-artist/ https://www.thisiscolossal.com/2017/12/tatsuo-horiuchi-excel...
laichzeit0 8y ago
Maybe read the article. They are experimenting with putting the notebooks directly into production ala bash script. I don’t think this is a great idea either.
- staticassertion 8y ago
  Yeah, I read it after, silly of me to comment first.
brianberns 8y ago
Did you read the article? "A Netflix engineer described how they have replaced Bash scripts with Jupyter notebooks for ETL pipelines and cron jobs."

stewbrew 8y ago

Some simpler notebook-like environments stay closer to source code in that they basically are source code with interleaved results (e.g. as comments). IMHO they hit a sweet spot between REPLs and those notebook environments inspired by mathematica, maple and similar more mathematically oriented software products.

TeMPOraL 8y ago
That's just a REPL with basic editor integration (eval at point, paste result). Surprisingly unpopular outside Emacs/Lisp land.
- stewbrew 8y ago
  Well yes, but these notebooks are little more, aren't they.

dangirsh 8y ago

ob-ipython enables IDE-like editing features within the code cells. It's embedded in a polyglot, git-friendly, literate programming environment called Org-mode. I use it every day and love it. Other goodies: - easily manage multiple kernels (in different languages / machines) in one file - tree-based organization manages complexity better than linear notebooks - no browser in sight (unless you need interactive widgets) - highly exportable, including to ipynb via ox-ipynb Downsides: - Emacs-only - small user base / limited docs - can't easily import from ipynb - async cell execution support is early-stage See Scimax ipython for examples

chimprich 8y ago
Interesting. I use emacs for hacking Python but I've never heard of this tool. Do you actually use this for software development or is it more of a data-science-type exploration tool?
- d0mine 8y ago
  related "literate devops in emacs" https://youtu.be/dljNabciEGg https://youtu.be/dljNabciEGg
- dangirsh 8y ago
  I use Org for software development whenever I can, which is currently everyday. Usually, new code starts in cells with some Org-managed context (e.g. a Jupyter kernel in a remote container with some DB/service access). This is done using the :session code cell keyword, which works per subtree. Managing remote sessions like this generally keeps me away from terminals. Surrounding the cell are various mini-dashboards with useful docs / links / commands for that part of the project. Since Org supports embedding elisp and shell commands in clickable links [1], these mini-dashboards can be made very quickly. Org lets me edit the code using the proper Emacs mode for its language, while pulling dynamic completion / docs from the Jupyter kernel. Just like Jupyter notebooks, I can view rich outputs from the cells in-line. I can then name the outputs and make them inputs to other cells, including ones in different languages / kernels. AFAIK that's an Org-only trick. Most code eventually finds it's way to normal source files (see Org's "tangle" feature). This feels more natural than moving code from notebooks since, again, the cell editing mode is the same as the one for source files. Org's tree-manipulation capabilities + support for multiple sessions means that (so far) I've only ever needed 1 Org file per project. I track this in git, which is simple since Org is just plain-text. To share with non-Org users, I usually export to ipynb [2] or, for static docs, HTML [3]. [1]: https://orgmode.org/manual/External-links.html https://orgmode.org/manual/External-links.html [2]: https://github.com/jkitchin/ox-ipynb https://github.com/jkitchin/ox-ipynb [3]: https://github.com/fniessen/org-html-themes https://github.com/fniessen/org-html-themes