4 ms·

Pandas is the library I wish I'd had in the late '00s when my employer decided our site license for JMP was too costly. Well, really pandas plus matplotlib plu

Pandas is the library I wish I'd had in the late '00s when my employer decided our site license for JMP was too costly. Well, really pandas plus matplotlib plus Jupyter notebooks. My job frequently involved creating plots and putting them in Powerpoint. Often the same plot, day in and day out, with new data from the production line. An interactive tool that can automate this, with a low barrier to entry, can save an incredible amount of time. Since I discovered pandas, I've been recommending it to anybody who works in a putting-plots-in-powerpoint job. And there are a lot of people who have jobs like that.

taeric 9y ago
That sounds more amenable to an Excel sheet, honestly. Which I suppose is not that surprising, since spreadsheets were the original freeform notebook style program.
- sevensor 9y ago
  Excel is what we had to fall back on when they pulled our JMP license. There's a lot you can do with Excel, but automating Excel is incredibly error-prone. I came away from that experience with the conclusion that Excel is great as long as you stick to writing formulas, but as soon as you start writing macros things go bad in a hurry. And if you're not using macros for automation, that means pasting data in by hand every day, which is quite possibly worse.
  - taeric 9y ago
    I feel like I could replace "Excel" with "Jupyter" and not much changes there, honestly. Having a data ingest process in a notebook worries me because it seems to take a ton of the engineering practices we have managed to get in software, and completely ignore them. And I get that I am being a little harsher than reality dictates. However, the testing and "build" process that surrounds most of "notebooks" is laughably like what we specifically avoided in software when we said your build should be standardized in an external file. And not scripting the main IDE that you happen to be using. Indeed, I am perplexed by folks that don't know how to move between IDEs or who won't bother to understand how they are pulling dependencies into their system. Notebooks, though, seem to embrace that. Which, as I've indicated elsewhere, is great for interactive use, but seems a major step backwards for serious solutions.
    - sevensor 9y ago
      The advantage I see with Jupyter is that at least there's a path from the notebook to a proper programming language. Excel macros live in Excel and are tied to the sheets in a particular workbook. Pandas / Matplotlib / Jupyter lets you turn your exploratory analysis history into a script that runs outside of Jupyter. That's a huge advantage -- schedule it to run at 5am for your 6am meeting, and you can come in to work half an hour later! Excel macros can do this, but because they rely on the interaction between formula evaluation and procedural code, it's such a headache in comparison that it's much less likely to be worth it. Overwrite the wrong cell and the whole thing falls apart. I'll take the Python world any day.
      - taeric 9y ago
        Sadly, I'm cynical enough to think that just because there is a path, doesn't mean it is encouraged or used. In fact, most of the excitement seems to be about doubling down on the Jupyter infrastructure so that you can have "executable notebooks." I don't know why that bothers me, but it definitely does.
        sevensor 9y ago
        > I don't know why that bothers me, but it definitely does. It bothers me, and I can tell you why! Between pandas and matplotlib, the royal road to liberation from routine analysis tasks is paved with Python scripts. Jupyter has an important ancillary role in aiding discovery. But this whole notion of "executable notebooks" seems designed to keep people in bondage to fragile workflows based on capturing and replaying user input. It caters to the least common denominator, to the one person on the team who can't be trusted to read things. I'm infuriated on behalf of anybody subjected to such foolishness.
        taeric 9y ago
        I was really hoping someone would give a counter argument to this. Do you know of any common "devil's advocates" in this vein?
teekert 9y ago
I'd add seaborn to that, it work perfectly with pandas.DataFrames, often it creates exactly what you want with minimal input (just a dataframe), ie: import seaborn as sns sns.violinplot(data=dataframe) Set %matplotlib inline and you don't need more commands in your notebook.
kortex 9y ago
Look into Jupyter dashboard
- sevensor 9y ago
  Yikes, I didn't know about this. I would strongly discourage this sort of thing. There's a world of difference between exploratory data analysis and BI platforms. Just because they both produce scatter plots, doesn't mean they should work the same way. Confusion about this, in my opinion, is how TIBCO basically destroyed Spotfire. Fortunately, the dashboards project seems not to be moving forward. https://github.com/jupyter/enhancement-proposals/blob/master/jupyter-dashboards-deployment-attic/jupyter-dashboards-deployment-attic.md https://github.com/jupyter/enhancement-proposals/blob/master... Also, there's a world of difference between this and the way I'd recommend people use Jupyter. Jupyter is great for exploratory data analysis. It caputres every step you take along the way, especially if you're disciplined about not reusing cells. At the end, you have something you can paste into Powerpoint. If the boss asks you for that same plot the next day, you don't reuse the notebook -- that means repeating all of your mistakes. You pull out the parts of the analysis you want to keep into a Python script, and you run that in the future. In no way is it a good idea to try to use the notebook operationally.

taeric 9y ago

That sounds more amenable to an Excel sheet, honestly. Which I suppose is not that surprising, since spreadsheets were the original freeform notebook style program.

sevensor 9y ago
Excel is what we had to fall back on when they pulled our JMP license. There's a lot you can do with Excel, but automating Excel is incredibly error-prone. I came away from that experience with the conclusion that Excel is great as long as you stick to writing formulas, but as soon as you start writing macros things go bad in a hurry. And if you're not using macros for automation, that means pasting data in by hand every day, which is quite possibly worse.
- taeric 9y ago
  I feel like I could replace "Excel" with "Jupyter" and not much changes there, honestly. Having a data ingest process in a notebook worries me because it seems to take a ton of the engineering practices we have managed to get in software, and completely ignore them. And I get that I am being a little harsher than reality dictates. However, the testing and "build" process that surrounds most of "notebooks" is laughably like what we specifically avoided in software when we said your build should be standardized in an external file. And not scripting the main IDE that you happen to be using. Indeed, I am perplexed by folks that don't know how to move between IDEs or who won't bother to understand how they are pulling dependencies into their system. Notebooks, though, seem to embrace that. Which, as I've indicated elsewhere, is great for interactive use, but seems a major step backwards for serious solutions.
  - sevensor 9y ago
    The advantage I see with Jupyter is that at least there's a path from the notebook to a proper programming language. Excel macros live in Excel and are tied to the sheets in a particular workbook. Pandas / Matplotlib / Jupyter lets you turn your exploratory analysis history into a script that runs outside of Jupyter. That's a huge advantage -- schedule it to run at 5am for your 6am meeting, and you can come in to work half an hour later! Excel macros can do this, but because they rely on the interaction between formula evaluation and procedural code, it's such a headache in comparison that it's much less likely to be worth it. Overwrite the wrong cell and the whole thing falls apart. I'll take the Python world any day.
    - taeric 9y ago
      Sadly, I'm cynical enough to think that just because there is a path, doesn't mean it is encouraged or used. In fact, most of the excitement seems to be about doubling down on the Jupyter infrastructure so that you can have "executable notebooks." I don't know why that bothers me, but it definitely does.
      - sevensor 9y ago
        > I don't know why that bothers me, but it definitely does. It bothers me, and I can tell you why! Between pandas and matplotlib, the royal road to liberation from routine analysis tasks is paved with Python scripts. Jupyter has an important ancillary role in aiding discovery. But this whole notion of "executable notebooks" seems designed to keep people in bondage to fragile workflows based on capturing and replaying user input. It caters to the least common denominator, to the one person on the team who can't be trusted to read things. I'm infuriated on behalf of anybody subjected to such foolishness.
        taeric 9y ago
        I was really hoping someone would give a counter argument to this. Do you know of any common "devil's advocates" in this vein?

teekert 9y ago

I'd add seaborn to that, it work perfectly with pandas.DataFrames, often it creates exactly what you want with minimal input (just a dataframe), ie: import seaborn as sns sns.violinplot(data=dataframe) Set %matplotlib inline and you don't need more commands in your notebook.

kortex 9y ago

Look into Jupyter dashboard

sevensor 9y ago
Yikes, I didn't know about this. I would strongly discourage this sort of thing. There's a world of difference between exploratory data analysis and BI platforms. Just because they both produce scatter plots, doesn't mean they should work the same way. Confusion about this, in my opinion, is how TIBCO basically destroyed Spotfire. Fortunately, the dashboards project seems not to be moving forward. https://github.com/jupyter/enhancement-proposals/blob/master/jupyter-dashboards-deployment-attic/jupyter-dashboards-deployment-attic.md https://github.com/jupyter/enhancement-proposals/blob/master... Also, there's a world of difference between this and the way I'd recommend people use Jupyter. Jupyter is great for exploratory data analysis. It caputres every step you take along the way, especially if you're disciplined about not reusing cells. At the end, you have something you can paste into Powerpoint. If the boss asks you for that same plot the next day, you don't reuse the notebook -- that means repeating all of your mistakes. You pull out the parts of the analysis you want to keep into a Python script, and you run that in the future. In no way is it a good idea to try to use the notebook operationally.