4 ms·

As someone who flips back and forward b/w data science and software engineering quite a lot, I think they are highly complementary. The benefit of things like J

by zmmmmm 7y ago

As someone who flips back and forward b/w data science and software engineering quite a lot, I think they are highly complementary. The benefit of things like Jupyter is the ability to thoroughly explore a problem space without requiring the overhead of things like version control. The way I see it, every piece of code written actually has such an exploratory phase where you try out a few different approaches, test your assumptions, often realise a mistake half way through and rework your algorithm etc. The danger when that is done in an IDE setting is that because you build up commitment in a solution as you go, a reluctance sets in to rework it. So the final output is nothing like what you would write if you did it from scratch - it's littered with historical quirks of how you arrived at that implementation. So I actually think that breaking out of the IDE and doing an exploratory phase in something like Jupyter is a really useful way to get your ideas into reasonable shape before you write your "real" code.

Myrmornis 7y ago
With the greatest possible respect, I suspect that you are still on the path towards getting really comfortable with git. I hope this doesn't come across as arrogant or presumptuous. What I suspect, is that you are at a stage which everyone passes through, where the act of "making a commit" feels permanent. You know that there are ways to change history, but it feels like they are going to be a huge distraction from getting work done. It really doesn't help that git uses the word "commit"! I sort of wish they'd used the word "snapshot" or something. In any case, if I am right, what you'll come to see soon is that it really isn't painful to rework commits and there is nothing constraining about git. Just create a copy of your current branch first if you're at all worried about messing something up (git checkout -b mybranch-snapshot-1), and then `git reset $commit_before_your_experimental_throwaway_commits`. There's no need to do anything more complicated than that when it's just your private work on your laptop, that you haven't pushed to a shared remote. There were a couple of things you wrote that make me think that. Firstly: "without requiring the overhead of things like version control". There is a miniscule overhead to using git in the way I describe: `git init`, `git add`, `git commit`, `git reset` are the only commands you need and they take a second to invoke. Secondly "you build up commitment in a solution as you go": as i said above, I believe that as one gets more comfortable with git, it no longer feels a constraint -- quite the opposite, you feel liberated to experiment because you always know you can get back to any state you wish.
- zmmmmm 7y ago
  Thanks ... I take your point .... but it's not really about being comfortable with git. It's more about what kind of activity you are doing. When I'm an exploratory process with Jupyter it's iterative with a feedback cycle that is almost subsecond. I often have 3 - 4 versions of the algorithm I am exploring visible at the same time in different cells. I'm using autocomplete and interactive evaluation continuously to understand what state the algorithm is in and what attributes and methods are available to me and how they behave. No amount of git or anything else an IDE can do achieves those things.
  - Myrmornis 7y ago
    Right, fair enough. Especially helpful to have that iterative feedback with plots. I guess I'm just saying, have a workflow for moving that code into a version-controlled python package as you get happy with it. (I've given some instructions on how to work with your own python packages in a notebook in anothetr comment in this subthread). Now that I've fully embraced my notebookless workflow I've been using the following for plots in a traditional ipython shell: import matplotlib matplotlib.use("Qt5Agg") But yes, even I might start up a notebook to iteratively refine a plot! And the HTML table output for dataframes is perfect also.

by zmmmmm 7y ago

Myrmornis 7y ago

With the greatest possible respect, I suspect that you are still on the path towards getting really comfortable with git. I hope this doesn't come across as arrogant or presumptuous. What I suspect, is that you are at a stage which everyone passes through, where the act of "making a commit" feels permanent. You know that there are ways to change history, but it feels like they are going to be a huge distraction from getting work done. It really doesn't help that git uses the word "commit"! I sort of wish they'd used the word "snapshot" or something. In any case, if I am right, what you'll come to see soon is that it really isn't painful to rework commits and there is nothing constraining about git. Just create a copy of your current branch first if you're at all worried about messing something up (git checkout -b mybranch-snapshot-1), and then `git reset $commit_before_your_experimental_throwaway_commits`. There's no need to do anything more complicated than that when it's just your private work on your laptop, that you haven't pushed to a shared remote. There were a couple of things you wrote that make me think that. Firstly: "without requiring the overhead of things like version control". There is a miniscule overhead to using git in the way I describe: `git init`, `git add`, `git commit`, `git reset` are the only commands you need and they take a second to invoke. Secondly "you build up commitment in a solution as you go": as i said above, I believe that as one gets more comfortable with git, it no longer feels a constraint -- quite the opposite, you feel liberated to experiment because you always know you can get back to any state you wish.

zmmmmm 7y ago
Thanks ... I take your point .... but it's not really about being comfortable with git. It's more about what kind of activity you are doing. When I'm an exploratory process with Jupyter it's iterative with a feedback cycle that is almost subsecond. I often have 3 - 4 versions of the algorithm I am exploring visible at the same time in different cells. I'm using autocomplete and interactive evaluation continuously to understand what state the algorithm is in and what attributes and methods are available to me and how they behave. No amount of git or anything else an IDE can do achieves those things.
- Myrmornis 7y ago
  Right, fair enough. Especially helpful to have that iterative feedback with plots. I guess I'm just saying, have a workflow for moving that code into a version-controlled python package as you get happy with it. (I've given some instructions on how to work with your own python packages in a notebook in anothetr comment in this subthread). Now that I've fully embraced my notebookless workflow I've been using the following for plots in a traditional ipython shell: import matplotlib matplotlib.use("Qt5Agg") But yes, even I might start up a notebook to iteratively refine a plot! And the HTML table output for dataframes is perfect also.