I find that IPython NB is a great tool for initial exploration -- to sketch out the ETL steps that are needed, what types of computations I'm going to be doing,

I find that IPython NB is a great tool for initial exploration -- to sketch out the ETL steps that are needed, what types of computations I'm going to be doing, etc. As soon as those ideas are roughed out, I switch to writing Python scripts that I can run through completely, write tests for, git commit and get reasonable diffs, and more. I'll generally keep an ipynb open in a tab for experimentation, but I'm doing all of the work in PyCharm or Spyder. This also forces me to think about the engineering implications of the analysis a bit sooner. Inevitably these projects are not one-offs, and will need to be at least repeated regularly, if not outright productionalized -- and ipynb files do not lead to production-ready code. I also agree that using a real database is often a better option than pandas. Many folks avoid it because it's less comfortable to spin up postgres + a new DB instance than it is to sit in the comfort of IPython, but it's totally worth it. Exploring and previewing the data via SQL is just so much faster and more intuitive than spinning it around with pandas.