3 ms·

As a computer scientist/software engineer, please allow me the question: Why would I prefer a notebook over e.g. equivalent python script(s) in a git? I first

As a computer scientist/software engineer, please allow me the question: Why would I prefer a notebook over e.g. equivalent python script(s) in a git? I first saw jupyter notebooks when my sister (physicist, non-programmer) used it for analyzing economical data with pandas. Run-time for the full data set was half a day (and IMHO for that analysis SQL would have been better suited). I understand that as a non-programmer it looks alluring, but once the language proficiency is build up, why not use an IDE and run the code on a shell?

fsh 7y ago
The main reason is that one has to fiddle with the code a lot and re-running the whole thing is much too slow. A common example is that a huge text file containing experimental data gets parsed in the beginning. Then you have to explore the data step-by-step using all kinds of visualization and analysis such as Fourier transforms, curve fitting, etc. If you simply put everything into one giant python script, for every step you have to re-run the entire thing which takes forever. Of course you can speed things up by writing intermediate results to disk, but this adds tons of boilerplate code and is quite error prone. One alternative would be to write individual scripts for each step and read them into an interactive REPL shell. However, then you still have to somehow record the proper execution order if you ever want to repeat the analysis.
PeterisP 7y ago
The key factor is iteration speed. If step A takes 5 minutes (and 5 minutes is a very short time) and I want to experiment on step B, then I don't want to rerun step A each time while I'm writing and running code that helps me understand what step B is going to be; I'd want that to be interactive and immediate, not have each rerun take 5 minutes. Storing/loading to disk is not a good option because all the data that needs to be stored is not yet determined until the exploration is finished; If I write code to save/load A, then I need to change (and test) it after I'm done with B and now want to experiment with C, and it all becomes even more complicated when I need to add an extra step and data field to step A and rerun everything. Deciding what data should be stored in what format is something that you can do in 'productionizing' the code after you've done the exploratory analysis. REPL is not a good option because it's not convenient to save and replicate the code that got you to the current REPL state. The other aspect is that visually 'debugging' intermediary data through various plots is not conveniently possible in IDEs. I could generate some picture files in a folder or possibly an HTML 'dashboard' to see the results of my most recent run but that takes extra code and effort, and the results aren't immediately in my face like in a notebook.
- archi42 7y ago
  Ah, I think I have a hugely different approach to data processing: For my work I often have a very good idea what the output should look like, and what transformations are required on the input to get there. E.g. when processing log files to generate an overview page, or (as I'm doing right now) adding a target to binutils (assembler, linker,...). (Obviously I'm not a data scientist ;-) With what you describe, intuitively I would use a library that allows me to store&load data per step (with verifying the structure matches), or pass it in-memory. Think JSON (yeah, slow) or something like protobuffers. That way I could do both > store(A(read(input)) -> file); store(B(load(file)) -> file2) during development (or in case B is in another language as A), and in production just > B(A(read(some_other_input))) But yeah, that's just my intuition of course. Maybe I'd be a bad data scientist. However, can't you just experiment with smaller data sets? That's what I usually do if processing is slow (e.g. instead of parsing 10GB of log files, I'll just do 50MB to verify the processing pipeline works, and once that's it, run it on the full 10GB and grab a coffee while it runs). Not an option for data science?
  - PeterisP 7y ago
    Sure, if you know what the output should look like and if it's possible to e.g. write tests to verify that it's correct, then jupyter notebooks would not be the proper tool to use. The intended usecase of these notebooks is in scenarios where the main output is not the code and not a particular set transformed data, but knowledge gained during a 'computational exploration' of that data. With that knowledge in hand, you can then build 'productionized' code with different methodologies (possibly but not necessarily using or adapting large parts of the code in your notebook), if that's needed - and in such data analysis scenarios it often happens that it's not ever needed. Sampling a subset of the data sometimes works. Sometimes it would alter the results substantionally and drive the exploration in a wrong direction; questioning and verifying assumptions is important, and it can be a big difference if all A's are also B or only 99% of them are.
  - fock 7y ago
    https://joblib.readthedocs.io/en/latest/ https://joblib.readthedocs.io/en/latest/ (though it has a large overhead)

fsh 7y ago

The main reason is that one has to fiddle with the code a lot and re-running the whole thing is much too slow. A common example is that a huge text file containing experimental data gets parsed in the beginning. Then you have to explore the data step-by-step using all kinds of visualization and analysis such as Fourier transforms, curve fitting, etc. If you simply put everything into one giant python script, for every step you have to re-run the entire thing which takes forever. Of course you can speed things up by writing intermediate results to disk, but this adds tons of boilerplate code and is quite error prone. One alternative would be to write individual scripts for each step and read them into an interactive REPL shell. However, then you still have to somehow record the proper execution order if you ever want to repeat the analysis.

PeterisP 7y ago

The key factor is iteration speed. If step A takes 5 minutes (and 5 minutes is a very short time) and I want to experiment on step B, then I don't want to rerun step A each time while I'm writing and running code that helps me understand what step B is going to be; I'd want that to be interactive and immediate, not have each rerun take 5 minutes. Storing/loading to disk is not a good option because all the data that needs to be stored is not yet determined until the exploration is finished; If I write code to save/load A, then I need to change (and test) it after I'm done with B and now want to experiment with C, and it all becomes even more complicated when I need to add an extra step and data field to step A and rerun everything. Deciding what data should be stored in what format is something that you can do in 'productionizing' the code after you've done the exploratory analysis. REPL is not a good option because it's not convenient to save and replicate the code that got you to the current REPL state. The other aspect is that visually 'debugging' intermediary data through various plots is not conveniently possible in IDEs. I could generate some picture files in a folder or possibly an HTML 'dashboard' to see the results of my most recent run but that takes extra code and effort, and the results aren't immediately in my face like in a notebook.

archi42 7y ago
Ah, I think I have a hugely different approach to data processing: For my work I often have a very good idea what the output should look like, and what transformations are required on the input to get there. E.g. when processing log files to generate an overview page, or (as I'm doing right now) adding a target to binutils (assembler, linker,...). (Obviously I'm not a data scientist ;-) With what you describe, intuitively I would use a library that allows me to store&load data per step (with verifying the structure matches), or pass it in-memory. Think JSON (yeah, slow) or something like protobuffers. That way I could do both > store(A(read(input)) -> file); store(B(load(file)) -> file2) during development (or in case B is in another language as A), and in production just > B(A(read(some_other_input))) But yeah, that's just my intuition of course. Maybe I'd be a bad data scientist. However, can't you just experiment with smaller data sets? That's what I usually do if processing is slow (e.g. instead of parsing 10GB of log files, I'll just do 50MB to verify the processing pipeline works, and once that's it, run it on the full 10GB and grab a coffee while it runs). Not an option for data science?
- PeterisP 7y ago
  Sure, if you know what the output should look like and if it's possible to e.g. write tests to verify that it's correct, then jupyter notebooks would not be the proper tool to use. The intended usecase of these notebooks is in scenarios where the main output is not the code and not a particular set transformed data, but knowledge gained during a 'computational exploration' of that data. With that knowledge in hand, you can then build 'productionized' code with different methodologies (possibly but not necessarily using or adapting large parts of the code in your notebook), if that's needed - and in such data analysis scenarios it often happens that it's not ever needed. Sampling a subset of the data sometimes works. Sometimes it would alter the results substantionally and drive the exploration in a wrong direction; questioning and verifying assumptions is important, and it can be a big difference if all A's are also B or only 99% of them are.
- fock 7y ago
  https://joblib.readthedocs.io/en/latest/ https://joblib.readthedocs.io/en/latest/ (though it has a large overhead)