3 ms·

Ah, I think I have a hugely different approach to data processing: For my work I often have a very good idea what the output should look like, and what transfor

Ah, I think I have a hugely different approach to data processing: For my work I often have a very good idea what the output should look like, and what transformations are required on the input to get there. E.g. when processing log files to generate an overview page, or (as I'm doing right now) adding a target to binutils (assembler, linker,...). (Obviously I'm not a data scientist ;-) With what you describe, intuitively I would use a library that allows me to store&load data per step (with verifying the structure matches), or pass it in-memory. Think JSON (yeah, slow) or something like protobuffers. That way I could do both > store(A(read(input)) -> file); store(B(load(file)) -> file2) during development (or in case B is in another language as A), and in production just > B(A(read(some_other_input))) But yeah, that's just my intuition of course. Maybe I'd be a bad data scientist. However, can't you just experiment with smaller data sets? That's what I usually do if processing is slow (e.g. instead of parsing 10GB of log files, I'll just do 50MB to verify the processing pipeline works, and once that's it, run it on the full 10GB and grab a coffee while it runs). Not an option for data science?

PeterisP 7y ago
Sure, if you know what the output should look like and if it's possible to e.g. write tests to verify that it's correct, then jupyter notebooks would not be the proper tool to use. The intended usecase of these notebooks is in scenarios where the main output is not the code and not a particular set transformed data, but knowledge gained during a 'computational exploration' of that data. With that knowledge in hand, you can then build 'productionized' code with different methodologies (possibly but not necessarily using or adapting large parts of the code in your notebook), if that's needed - and in such data analysis scenarios it often happens that it's not ever needed. Sampling a subset of the data sometimes works. Sometimes it would alter the results substantionally and drive the exploration in a wrong direction; questioning and verifying assumptions is important, and it can be a big difference if all A's are also B or only 99% of them are.
fock 7y ago
https://joblib.readthedocs.io/en/latest/ https://joblib.readthedocs.io/en/latest/ (though it has a large overhead)

PeterisP 7y ago

Sure, if you know what the output should look like and if it's possible to e.g. write tests to verify that it's correct, then jupyter notebooks would not be the proper tool to use. The intended usecase of these notebooks is in scenarios where the main output is not the code and not a particular set transformed data, but knowledge gained during a 'computational exploration' of that data. With that knowledge in hand, you can then build 'productionized' code with different methodologies (possibly but not necessarily using or adapting large parts of the code in your notebook), if that's needed - and in such data analysis scenarios it often happens that it's not ever needed. Sampling a subset of the data sometimes works. Sometimes it would alter the results substantionally and drive the exploration in a wrong direction; questioning and verifying assumptions is important, and it can be a big difference if all A's are also B or only 99% of them are.

fock 7y ago

https://joblib.readthedocs.io/en/latest/ https://joblib.readthedocs.io/en/latest/ (though it has a large overhead)