5 ms·

I seriously cannot agree more. Jupyter notebooks were fun to use for a bit, then I hit the inevitable wall of "ok, now let's turn this into a real, properly bu

I seriously cannot agree more. Jupyter notebooks were fun to use for a bit, then I hit the inevitable wall of "ok, now let's turn this into a real, properly built script, but now everything is breaking for inexplicable reasons". Notebooks are fine for early stage experimenting, but if you've got to the point where you start up you are relying on a notebook for anything, or your workflow consists of "start up notebook, run cells until I get to the one I'm working on" then you need to stop, and build it into something proper. Experimentation is fine, but it should not come at the cost of writing things properly when the time comes, and they are not an excuse for not knowing some good software engineering. Serious alarm bells go off in my head when I read tweets like "Data science code doesn't need to follow the rules of good software engineering". Edit: that remote notebook - is that honestly not one of the most terrifying things you've seen? That runs so counter to almost every good bit of software design and engineering practice about clarity, maintainability, good practice, security, etc that I can think of. It's the very definition of indecipherable, inscrutable hidden state and unknown side effects.

megamindbrian2 8y ago
I wrote a loader for this. And there are tools to split cells for compiling. I wrote a tool to split up cells too and compile in order. Pretty easy, all the code is in notebooks.
- megamindbrian2 8y ago
  A few quick searches, maybe something along these lines? https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Importing%20Notebooks.html https://jupyter-notebook.readthedocs.io/en/stable/examples/N... My JavaScript version I just put a simple wrapper around my print-results call to check if it is in module mode or notebook mode.
daveguy 8y ago
I don't even like it for initial experimentation and exploration. Much prefer the command line repl (the jupyter/ipython one). Faster feedback and easier navigation then you can just export the history when you're ready to capture and make permanent some workflow. It still requires going through and picking out the important bits, but reading the history like a story of commands is generally enough to pick out the important parts. The only thing I would use notebooks for is demonstration / teaching.
- cdancette 8y ago
  If you work with image processing, notebooks are very handy as they can display images directly. Also to display tables nicely with pandas, or for any data visualization actually.
  - bunderbunder 8y ago
    That's my problem. I agree that notebooks create a troublesome dev situation, but "%matplotlib inline" and stuff like the pivot table plugin keep me crawling back.
  - srean 8y ago
    Doesn't IPython repl running in a QTconsole terminal do that ?
    - cdancette 8y ago
      Yeah but it's less nice, and if you're running on a distant server you have to deal with X forwarding
- kovek 8y ago
  Will the REPL allow you to easily edit function definitions and then reinterpret them?
  - rspeer 8y ago
    I mean Jupyter doesn't really do that either, right? You can't edit most function definitions. Only ones that you've defined right there in the notebook, and only because you can run the cell that defines them again, overwriting the function. But if you do use libraries instead of just a huge mess of notebooks, you're stuck if you want to change the code in a well-supported way. You can ask your notebook to monkey-patch the code (which is an even bigger mess), or you can use an unreliable magic extension, "%autoreload".
  - sstanie 8y ago
    Yep, in the ipython REPL if you do %load_ext autoreload autoreload 2 it'll reload functions from any module you edit each time you run a new command.
commandlinefan 8y ago
Wow, I thought I was the only one. I guess the biggest lesson I've ever learned from the internet is that I'm never the only one.
Dowwie 8y ago
One-off "experiments" become production technology all the time. They address an immediate need at the cost of tomorrow. This hasn't ever changed. Hyperbolic discounting is real. When you're in a job that requires running experiments, analyzing results, and summarizing findings, all within an hour, shortcuts are required. I think that analysts/scientists using python and Jupyter notebooks are doing IT a favor. If your job is to refactor/re-engineer python notebooks and corresponding workflows, you've either forgotten how much worse it could be with excel+vba or you fortunately arrived after trench warfare.

megamindbrian2 8y ago

I wrote a loader for this. And there are tools to split cells for compiling. I wrote a tool to split up cells too and compile in order. Pretty easy, all the code is in notebooks.

megamindbrian2 8y ago
A few quick searches, maybe something along these lines? https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Importing%20Notebooks.html https://jupyter-notebook.readthedocs.io/en/stable/examples/N... My JavaScript version I just put a simple wrapper around my print-results call to check if it is in module mode or notebook mode.

daveguy 8y ago

I don't even like it for initial experimentation and exploration. Much prefer the command line repl (the jupyter/ipython one). Faster feedback and easier navigation then you can just export the history when you're ready to capture and make permanent some workflow. It still requires going through and picking out the important bits, but reading the history like a story of commands is generally enough to pick out the important parts. The only thing I would use notebooks for is demonstration / teaching.

cdancette 8y ago
If you work with image processing, notebooks are very handy as they can display images directly. Also to display tables nicely with pandas, or for any data visualization actually.
- bunderbunder 8y ago
  That's my problem. I agree that notebooks create a troublesome dev situation, but "%matplotlib inline" and stuff like the pivot table plugin keep me crawling back.
- srean 8y ago
  Doesn't IPython repl running in a QTconsole terminal do that ?
  - cdancette 8y ago
    Yeah but it's less nice, and if you're running on a distant server you have to deal with X forwarding
kovek 8y ago
Will the REPL allow you to easily edit function definitions and then reinterpret them?
- rspeer 8y ago
  I mean Jupyter doesn't really do that either, right? You can't edit most function definitions. Only ones that you've defined right there in the notebook, and only because you can run the cell that defines them again, overwriting the function. But if you do use libraries instead of just a huge mess of notebooks, you're stuck if you want to change the code in a well-supported way. You can ask your notebook to monkey-patch the code (which is an even bigger mess), or you can use an unreliable magic extension, "%autoreload".
- sstanie 8y ago
  Yep, in the ipython REPL if you do %load_ext autoreload autoreload 2 it'll reload functions from any module you edit each time you run a new command.

commandlinefan 8y ago

Wow, I thought I was the only one. I guess the biggest lesson I've ever learned from the internet is that I'm never the only one.

Dowwie 8y ago

One-off "experiments" become production technology all the time. They address an immediate need at the cost of tomorrow. This hasn't ever changed. Hyperbolic discounting is real. When you're in a job that requires running experiments, analyzing results, and summarizing findings, all within an hour, shortcuts are required. I think that analysts/scientists using python and Jupyter notebooks are doing IT a favor. If your job is to refactor/re-engineer python notebooks and corresponding workflows, you've either forgotten how much worse it could be with excel+vba or you fortunately arrived after trench warfare.