6 ms·
Ok, but let's also acknowledge that the mental model and levels of abstraction increased significantly, which presents a barrier to people without any prior sof
by jey 3y ago
Ok, but let's also acknowledge that the mental model and levels of abstraction increased significantly, which presents a barrier to people without any prior software engineering experience, who are just getting into Python for "data science".
- mstipetic 3y agoWell maybe we should expect people to learn the tools they use properly instead of constantly dumbing things down for the sake of convenience. I’m not sure you should be able to just pop in and do “data science”
- spywaregorilla 3y agoI don't think it's a huge leap to say your functions are stored in this .py file vs. in a cell above. the magic commands simply allow you to reload the functions automatically without needing to reimport / reload the kernel.
- fluidcruft 3y agoIt is a huge gap when the notebook is documentation. Moving functions out either duplicates documentation or separates them. Notebooks are vastly superior for documentation. Extracting functions from notebooks into a py file isn't difficult (see nbdev for example)
- spywaregorilla 3y agoNotebooks are good for demonstration, not documentation. They are bad for documentation because you need to put docstrings and comments in the code while desiring to put the same content in the markdown.
- fluidcruft 3y agoI typically don't find docstrings are particularly useful for documenting an analysis or understanding what a function is doing. Even analyses that are repeated with different data. When you are debugging or reviewing an analysis, the docstrings are the least of your concerns. Validity is far more important. I would take a notebook that demonstrates what's happening to my data any day over a docstring that may or may not be correct. Particularly if I have to render it in yet a third system to juggle and manually keep in sync. So we've gone from notebooks to... notebook + py + docstring... I'm sure we can think of another useless layer of indirection to bolt on.
- spywaregorilla 3y ago> Particularly if I have to render it in yet a third system to juggle and manually keep in sync. I'm not sure you know what a docstring is if this is your response.
- fluidcruft 3y agoIf you are putting your docstrings in the py and they contain things like equations or figures or images, you have to render them to get the same effect that you would have just reviewing cells in the notebook. And as far as I know you can't even spread docstrings out over the body of a functions, the way you could if you broke the function up into cells. Docstrings are not literate programming.
- spywaregorilla 3y ago> And as far as I know you can't even spread docstrings out over the body of a functions, I really feel like you do not understand what a docstring is. This does not make any sense. > the way you could if you broke the function up into cells If your logic is broken up across cells then you cannot use it anywhere but in that notebook...
- fluidcruft 3y agoI realize you think the problem is that I don't understand what docstrings are, but it's more that you don't understand the sorts of documentation I'm talking about. docstrings are fine for a certain type of limited documentation that is formatted to prevent syntax errors. But they are extremely limited. Reading docstring/doxygen style documentation is the slowest and most error prone in my experience. Particularly when you are describing a complex calculation. > If your logic is broken up across cells then you cannot use it anywhere but in that notebook... You don't seem to understand how notebooks can be used and processed themselves. They are just data and there are libraries for loading and transforming them. You might check out pydev and papermill to get an idea about this (and maybe also literate programming in general).