6 ms·

I understand why they became popular, but as a software engineer considering how they work, I am just full of disappointment. we're going to spend the next t

I understand why they became popular, but as a software engineer considering how they work, I am just full of disappointment. we're going to spend the next ten years re-inventing every single software engineering best practice for jupyter's weirdo environment.

sipCom19 8y ago
Testing, style checking, code coverage...
gaius 8y ago
In the mid-90s I was a Mech Eng undergrad using a program called MathCAD which provided a "notebook" interactive computation mixed with text environment by running as a Word plugin. In 2018 I use Jupyter and it's not clear where the progress has been. There are too many compromises trying to make it work in a web browser. For what Jupyter is for I find that RStudio or Spyder are infinitely superior. I do interactive exploring and transforming that into both documents and reusable and deployable code all in the one tool
- semi-extrinsic 8y ago
  I too prefer RStudio/RMarkdown. Now we have the Reticulate package that allows running Python inside RMarkdown, for my purposes it's way better than Jupyter. It's just plain text so my editor and VCS and everything plays nicely. But perhaps the biggest strength is that you go through Pandoc, so you can just click a button and get the output as a Word file for sending to someone who's not a software dev, or you can get it as a LaTeX source file and extend it into a proper formal document like a journal paper, or you can do it as HTML where embedding Bokeh scripts and other interactive things Just Works. I've even made presentations with it going through the reveal.js framework, where you can put an interactive plot on one of your slides, and show people live "what happens when we change this parameter". That's still semi-witchcraft in 2018, but it's going to become a common thing (hopefully).
- kgwgk 8y ago
  Unfortunately RStudio also makes compromises to make it work in a web browser. A native tool could be even better.
  - gaius 8y ago
    RStudio is mainly a native tool, the web version is quite inferior. Linux, Windows and OSX native versions
    - kgwgk 8y ago
      One has to strecth the term quite a lot to say that RStudio is a "native" GUI. The interface is just a browser window, the only native part is the menu bar. https://imgur.com/yszRlMk https://imgur.com/yszRlMk
zmmmmm 8y ago
This is definitely one of my concerns too. Ad hoc code inside these notebooks is almost completely unmanageable from any reasonable software maintenance perspective, and refactoring code out of them is prohibitively difficult as well. I really want something to emerge that combines the best of both worlds of an IDE and notebook development, but there isn't anything close currently.
- staticassertion 8y ago
  What? What are you using Jupyter notebooks for where you want maintenance? They should be records of data analysis/ procedures, not code that runs in production or something.
  - ekianjo 8y ago
    exactly. I use it for quick data exploration and experimentation but it remains at that level.
    - gaius 8y ago
      People do that in Excel too and the next thing you know a spreadsheet is managing a portfolio or being used as the basis for published science!
      - ekianjo 8y ago
        Its not the fault of the tools if users dont know any better.
        TeMPOraL 8y ago
        It's also not the fault of the tools if there are no better alternatives.
        fsloth 8y ago
        So what would you suggest a financial modeler, who has no experience of any programming environments, to use instead? The value of Excel is that is a zero-config tool, available everywhere as 'standard' business installation, allows very quick iteration with visual output in certain range of tasks, is battle tested in millions of computers... etc. And everyone else is using it too. For a programmer it's easy to suggest to just pick some good language. Once you know two programming languages you can navigate pretty soon fluently in third. But picking up the first language fluency? That's hard. You can't just suggest to use python or such. You need to provide a tool that holds hands, has lots tutorials available etc. Visual programming is not necessary the answer either. I've understood that without discipline LabView programs become actual visual spaghetti pretty quickly.
wjnc 8y ago
You reason from a downside (more layman programmers without proper workflow). I reason from an upside: much better flow than Excel or Excel/VBA and hence less errors, better accountability, etc.
billfruit 8y ago
People are already doing bad programming practices in large scale with MATLAB, I guess jupyter is a step up from that.
guy98238710 8y ago
On the other hand, notebooks encourage many good practices: literate programming, purely functional code with immutable outputs, visibility of state, and reproducibility.

sipCom19 8y ago

Testing, style checking, code coverage...

gaius 8y ago

In the mid-90s I was a Mech Eng undergrad using a program called MathCAD which provided a "notebook" interactive computation mixed with text environment by running as a Word plugin. In 2018 I use Jupyter and it's not clear where the progress has been. There are too many compromises trying to make it work in a web browser. For what Jupyter is for I find that RStudio or Spyder are infinitely superior. I do interactive exploring and transforming that into both documents and reusable and deployable code all in the one tool

semi-extrinsic 8y ago
I too prefer RStudio/RMarkdown. Now we have the Reticulate package that allows running Python inside RMarkdown, for my purposes it's way better than Jupyter. It's just plain text so my editor and VCS and everything plays nicely. But perhaps the biggest strength is that you go through Pandoc, so you can just click a button and get the output as a Word file for sending to someone who's not a software dev, or you can get it as a LaTeX source file and extend it into a proper formal document like a journal paper, or you can do it as HTML where embedding Bokeh scripts and other interactive things Just Works. I've even made presentations with it going through the reveal.js framework, where you can put an interactive plot on one of your slides, and show people live "what happens when we change this parameter". That's still semi-witchcraft in 2018, but it's going to become a common thing (hopefully).
kgwgk 8y ago
Unfortunately RStudio also makes compromises to make it work in a web browser. A native tool could be even better.
- gaius 8y ago
  RStudio is mainly a native tool, the web version is quite inferior. Linux, Windows and OSX native versions
  - kgwgk 8y ago
    One has to strecth the term quite a lot to say that RStudio is a "native" GUI. The interface is just a browser window, the only native part is the menu bar. https://imgur.com/yszRlMk https://imgur.com/yszRlMk

zmmmmm 8y ago

This is definitely one of my concerns too. Ad hoc code inside these notebooks is almost completely unmanageable from any reasonable software maintenance perspective, and refactoring code out of them is prohibitively difficult as well. I really want something to emerge that combines the best of both worlds of an IDE and notebook development, but there isn't anything close currently.

staticassertion 8y ago
What? What are you using Jupyter notebooks for where you want maintenance? They should be records of data analysis/ procedures, not code that runs in production or something.
- ekianjo 8y ago
  exactly. I use it for quick data exploration and experimentation but it remains at that level.
  - gaius 8y ago
    People do that in Excel too and the next thing you know a spreadsheet is managing a portfolio or being used as the basis for published science!
    - ekianjo 8y ago
      Its not the fault of the tools if users dont know any better.
      - TeMPOraL 8y ago
        It's also not the fault of the tools if there are no better alternatives.
      - fsloth 8y ago
        So what would you suggest a financial modeler, who has no experience of any programming environments, to use instead? The value of Excel is that is a zero-config tool, available everywhere as 'standard' business installation, allows very quick iteration with visual output in certain range of tasks, is battle tested in millions of computers... etc. And everyone else is using it too. For a programmer it's easy to suggest to just pick some good language. Once you know two programming languages you can navigate pretty soon fluently in third. But picking up the first language fluency? That's hard. You can't just suggest to use python or such. You need to provide a tool that holds hands, has lots tutorials available etc. Visual programming is not necessary the answer either. I've understood that without discipline LabView programs become actual visual spaghetti pretty quickly.

wjnc 8y ago

You reason from a downside (more layman programmers without proper workflow). I reason from an upside: much better flow than Excel or Excel/VBA and hence less errors, better accountability, etc.

billfruit 8y ago

People are already doing bad programming practices in large scale with MATLAB, I guess jupyter is a step up from that.

guy98238710 8y ago

On the other hand, notebooks encourage many good practices: literate programming, purely functional code with immutable outputs, visibility of state, and reproducibility.