3 ms·

I'm a huge fan of pandas (http://pandas.pydata.org/ http://pandas.pydata.org/) for data analysis. It offers a lot of the basic functionality of R, but everythi

by birken 14y ago

I'm a huge fan of pandas (http://pandas.pydata.org/ http://pandas.pydata.org/) for data analysis. It offers a lot of the basic functionality of R, but everything is in python. The original author of pandas, Wes McKinney, even wrote a book about it: Python for Data Analysis (http://shop.oreilly.com/product/0636920023784.do http://shop.oreilly.com/product/0636920023784.do). One caveat I would mention about data analysis would be that statistics is not just number crunching. It is really a bit of an art to making sure you are looking at the right sample of data in the right way, and ensure you are accounting for all potential biases. Surprisingly, I have noticed as I've gotten more experience doing data analysis, it takes me longer to do and I make less confident assertions. But on the other hand, I now very rarely make assertions which were incorrect, which is extremely important. I believe that incorrect data analysis is significantly worse than no data analysis. So, the advice I would give to people getting started is whenever you come to a conclusion by analyzing a particular piece of data, ask your "if I look at the data differently, can I come to the opposite conclusion?". You would be surprised how often the answer to this question is yes, and that is a good indicator that you a) need more data or b) cannot make a significant conclusion. This can be especially difficult when you are already sure you know the answer to a question even before you do the data analysis, but you really have to be disciplined about it.

ttunguz 14y ago
Thanks. I'll check out Pandas. R syntax was a bit of a steep learning curve. This would be a nice bridge. I'm curious about the depth of statistical libraries and graphing libraries. ggplot is pretty great. My goal in writing the article is to address the concern that you raise about finding the right conclusion. Only if most people are on the same page about analysis can the team see through the kind of twisting you mention.
Wilduck 14y ago
> One caveat I would mention about data analysis would be that statistics is not just number crunching Agreed. Analysis that is well presented tells a story. The story begins with where the data comes from, then describes how the data was analyzed, and finally how the results relate back to the real world. If any of these pieces are missing, or unconvincing, it's a good sign that something is off, like you say, with either a) The data, or b) the significance of the conclusion. I emphasize the story aspect because the act of laying out all of the assumptions in a semi-narrative form goes a long way towards deobfuscating the potentially confusing statistics. It is also the sort of discipline that can help to lay bare all of the assumptions that could make you sure that you know the answer.

by birken 14y ago

ttunguz 14y ago

Thanks. I'll check out Pandas. R syntax was a bit of a steep learning curve. This would be a nice bridge. I'm curious about the depth of statistical libraries and graphing libraries. ggplot is pretty great. My goal in writing the article is to address the concern that you raise about finding the right conclusion. Only if most people are on the same page about analysis can the team see through the kind of twisting you mention.

Wilduck 14y ago

> One caveat I would mention about data analysis would be that statistics is not just number crunching Agreed. Analysis that is well presented tells a story. The story begins with where the data comes from, then describes how the data was analyzed, and finally how the results relate back to the real world. If any of these pieces are missing, or unconvincing, it's a good sign that something is off, like you say, with either a) The data, or b) the significance of the conclusion. I emphasize the story aspect because the act of laying out all of the assumptions in a semi-narrative form goes a long way towards deobfuscating the potentially confusing statistics. It is also the sort of discipline that can help to lay bare all of the assumptions that could make you sure that you know the answer.