8 ms·

Python for Data Analysis (new O'Reilly book from creator of Pandas)

186 points by plessthanpt05 14y ago

th0ma5 14y ago
Python is great for doing weird transformations of data that ideally you shouldn't have to do, or normalizing legacy things into modern standards. I like developing in several languages, but I often pop open Python to really "look" at something. I sadly haven't had much of a reason (yet!) to use Pandas too much, but anything we can do to get the word out about these kinds of techniques will hopefully increase procedural literacy in the world, and help ease the flow information even.
deleted 14y ago
[deleted]
tshauck 14y ago
I purchased this early release, and like it so far. Sadly, I keep finding myself going back to R for small data analysis instead of Pandas. Pandas may make the data transformation on par with, and computationally quicker than, R. But the lack of libraries in Python makes it difficult to work with for long periods of time.
monatron 14y ago
Pandas is an extremely solid, well-tested and super-powerful library. Congrats to Wes on the release of this book.
wesm 14y ago
Note, this is the "Early Release" version, code for "not done yet". I'll be putting the wraps on the full draft with luck by end of June.
- aklein 14y ago
  Congrats Wes on this Early Release milestone! As a very early reader of some of Wes's first chapters, I'm eagerly awaiting the final product. It's some fine work, and Wes is not one to cut corners in his exposition or skim over important things to know. I hope this book will advance recognition of a great set of standard tools that will make working with imperfect data as painless as possible, in a tech stack that is very amenable to 'productionization'. Kudos, sir.
- SatvikBeri 14y ago
  Hey Wes, is there a table of contents available yet?
  - wesm 14y ago
    I'll see if I can get one posted; should be soon.
    - 0ren 14y ago
      It seems that if you follow the link to Safari Books Online, you get a table of content. Here's a direct link: http://my.safaribooksonline.com/9781449323592?portal=oreilly&cid=orm-cat-readnow-9781449323592 http://my.safaribooksonline.com/9781449323592?portal=oreilly...
      - obtu 14y ago
        It seems far from final: the outline for Python basics is way more detailed than anything about Pandas.
- TheCowboy 14y ago
  Consider browsing the criticism of Data Analysis with Open Source Tools(O'Reilly) for suggestions. Many people, including myself, expected a book that would have been more in line with what your book is probably trying to accomplish.
  - wesm 14y ago
    I was also very disapointed with DAwOST; it's a book about data analysis techniques with a bit of code mixed in. This is a book about _programming tools for working with data_, pretty much the exact opposite.
- obtu 14y ago
  I was going to ask, but the early release does come with (free!) upgrades towards the final product: http://shop.oreilly.com/category/early-release.do http://shop.oreilly.com/category/early-release.do
  - tatsuke95 14y ago
    Stupid publishers. I'm buying the hard copy, because I love books, but they don't allow me to get in on the early release? They need to offer an "all editions" package.
brianobush 14y ago
does this include numpy, or is it just pandas?
- wesm 14y ago
  NumPy, IPython, matplotlib, pandas, a little bit of SciPy, scikit-learn, statsmodels, and plenty other fun stuff
  - neovive 14y ago
    Sold! Any suggested prerequisites?
    - wesm 14y ago
      Some working general purpose Python knowledge would be helpful but not required
apechai 14y ago
Ayone have experience as to how Python compare to R, Matlab (Octave) and other tools for data analysis? R has great libraries but I would prefer to use Python.
- aggie 14y ago
  You might want to look into RPy for using R libraries with python.
  - dbecker 14y ago
    I've very briefly experimented with rpy2. It got the job done, but I thought it was tricky enough that I'd want a good reason to combine R and python. Otherwise, I'd try and do the whole project in one or the other (And I haven't used rpy2 since I first tried it out.)
- dbecker 14y ago
  I've done a lot of work using both R and Python, and a little bit using Matlab (and SAS and Stata). Matlab is behind R and Python for both data cleaning and analysis. This is especially true if you have string variables and factor variables. Matlab's great if you are doing matrix operations of clean data (and you don't need to do anything fancy in how you report them). But, I don't find it worth using for real data analysis. Python and R are both great, though they feel pretty similar to me. Python obviously has the big advantage if you want to do general purpose computing too. Python has been faster than R in most cases I've compared them. For my work, development speed is more important than execution speed... so this hasn't been a huge factor for me. R has a couple of big advantages. First, the libraries. This is a big deal for me. I saw that there is a python equivalent to ggplot2 in the works. This will definitely strengthen the case for python, but the availability of libraries in R is awesome. Second, the community and help resources in R are amazing. I rarely run into a problems in R that haven't already been addressed on stackexchange.com. Perhaps I should be more proactive about asking python questions when I run into them, but I usually just work it out myself (which is more time consuming than looking the answer up online.) Lastly, I'm not an expert on big data. But, spending relatively little time with both R's bigmemory and Python's PyTables, it seems easier to get up to speed on big data with R at the moment. Though I haven't met them, my sense is that Wes, Travis Oliphant and the other relevant python developers are putting in a heroic effort to get Python up to speed. I have every expectation that Python will be my choice of the future.
joshu 14y ago
I'm thrilled that Pandas is named Pandas and not Pyndas.
hkmurakami 14y ago
As I get my feet wet again in programming, I've gone through phases in dabbling with web, app, and game programming. But about a month ago I realized, "You know, what I really loved about my studies back in academic years was numbers. That's what I want to do with code; I want to crunch numbers" This realization made me switch over from learning Ruby to learning Python. This book is going to be perfect for my interests. Wes, thank you for your efforts in putting this book together.
- Cieplak 14y ago
  If you really want to crunch numbers, I highly recommend Haskell. It falls short of Python in terms of libraries and by some accounts, readability, but it is substantially faster and excels at number crunching. I find it easier to express number crunching in functional rather than object-oriented style programming. Python has lambdas, but usually theres a more pythonic way to express something in Python. Although I must say, I find Haskell's list comprehension syntax slightly more elegant. http://tryhaskell.org/ http://tryhaskell.org/ edit: I take it back about Haskell's list comprehension syntax. At the end of the day they're both great. Haskell's syntax just feels like reading set theory notation.
  - hkmurakami 14y ago
    Thanks for the advice :) I haven't programmed in any substantial quantity or quality since 2004, so I'll start with a more approachable language (Python), but I'll definitely keep an open mind.
    - alexchamberlain 14y ago
      Do keep an open mind, but bear in mind that Haskell is a different style of programming all together and there are libraries to speed number crunching in Python (NumPy).
  - carterschonwald 14y ago
    And there's also a great and growing set of tools for numerical and data related computation. Scratch that, the current ongoing dev and research work that is making its way onto the haskell hackage is both astonishing and often superlative code. The libs released in the past month and the ones I know will be made available over the coming months... It's just great work! (I'm biased owing To using some of these tools for fun and profit :-). ) That aside, Pandas is a really nice piece of engineering that really works. It's a good role model for other libs, even when restricted to just its data frame part
look_lookatme 14y ago
Don't forget to brush up on the basics (with Python!): http://greenteapress.com/thinkstats/ http://greenteapress.com/thinkstats/
- neves 14y ago
  And don't forget to say that this guy publishes a free pdf version in the website above. I'm reading it now in my Kindle.
wgrover 14y ago
wesm, do you cover using cloud-based computing resources for tackling computationally intensive data analysis problems? I use Python and PiCloud to turn what is an hours-long data analysis problem on my Mac into a minutes-long job on Amazon's hardware. I don't know MATLAB, R, etc. well enough to say whether they can use the cloud as easily as Python, but it sure was easy with Python.
- wesm 14y ago
  The focus is primarily on data manipulation / preparation / cleaning / processing tools, data algorithms, visualization, and practical analysis case studies. The platform where you use them is a separate matter-- I will have some material on distributed computing tools, however.
  - wgrover 14y ago
    Sounds good. I can't tell you how thrilled I am to have a book like this coming out. I'm very interested in using it as a text for an instrumental analysis class I'm putting together. Thanks so much!
- msellout 14y ago
  For R, MATLAB, or Python in a cloud Map/Reduce framework, try out Opani. http://opani.com/help/wiki http://opani.com/help/wiki
  - wgrover 14y ago
    Didn't know about that one - thanks for the tip!
npcomplete 14y ago
Can we get the table of contents?
- wesm 14y ago
  I'll blog/tweet/G+ a ToC pretty soon, doing a bit of reorganization before the next Early Release update
- pwang 14y ago
  If you're in the LinkedIn "Python Data" group, I posted a comment with a list of the ToC (hope you don't mind, Wes!) http://www.linkedin.com/groupItem?view=&gid=4388870&type=member&item=115808571&commentID=81103690&qid=043286c4-a34b-4614-8cde-95b8a79b8087&trk=group_most_popular-0-b-cmn&goback=%2Egmp_4388870#commentID_81103690 http://www.linkedin.com/groupItem?view=&gid=4388870&...
sodelate 14y ago
can we use it for trade analysis?
- wesm 14y ago
  Without a doubt (how I got my start in Python)
ruffyen 14y ago
Looking at the table of contents, this seems like a great reference book for several technologies. I think I am sold, especially with the free updates.
salimmadjd 14y ago
I recommend reading this blog,for those interested in crunching numbers on python http://slendrmeans.wordpress.com/ http://slendrmeans.wordpress.com/
neves 14y ago
Where's the table of contents?

186 points by plessthanpt05 14y ago

Python is great for doing weird transformations of data that ideally you shouldn't have to do, or normalizing legacy things into modern standards. I like developing in several languages, but I often pop open Python to really "look" at something. I sadly haven't had much of a reason (yet!) to use Pandas too much, but anything we can do to get the word out about these kinds of techniques will hopefully increase procedural literacy in the world, and help ease the flow information even.

deleted 14y ago

[deleted]

tshauck 14y ago

I purchased this early release, and like it so far. Sadly, I keep finding myself going back to R for small data analysis instead of Pandas. Pandas may make the data transformation on par with, and computationally quicker than, R. But the lack of libraries in Python makes it difficult to work with for long periods of time.

monatron 14y ago

Pandas is an extremely solid, well-tested and super-powerful library. Congrats to Wes on the release of this book.

wesm 14y ago

Note, this is the "Early Release" version, code for "not done yet". I'll be putting the wraps on the full draft with luck by end of June.

aklein 14y ago
Congrats Wes on this Early Release milestone! As a very early reader of some of Wes's first chapters, I'm eagerly awaiting the final product. It's some fine work, and Wes is not one to cut corners in his exposition or skim over important things to know. I hope this book will advance recognition of a great set of standard tools that will make working with imperfect data as painless as possible, in a tech stack that is very amenable to 'productionization'. Kudos, sir.
SatvikBeri 14y ago
Hey Wes, is there a table of contents available yet?
- wesm 14y ago
  I'll see if I can get one posted; should be soon.
  - 0ren 14y ago
    It seems that if you follow the link to Safari Books Online, you get a table of content. Here's a direct link: http://my.safaribooksonline.com/9781449323592?portal=oreilly&cid=orm-cat-readnow-9781449323592 http://my.safaribooksonline.com/9781449323592?portal=oreilly...
    - obtu 14y ago
      It seems far from final: the outline for Python basics is way more detailed than anything about Pandas.
TheCowboy 14y ago
Consider browsing the criticism of Data Analysis with Open Source Tools(O'Reilly) for suggestions. Many people, including myself, expected a book that would have been more in line with what your book is probably trying to accomplish.
- wesm 14y ago
  I was also very disapointed with DAwOST; it's a book about data analysis techniques with a bit of code mixed in. This is a book about _programming tools for working with data_, pretty much the exact opposite.
obtu 14y ago
I was going to ask, but the early release does come with (free!) upgrades towards the final product: http://shop.oreilly.com/category/early-release.do http://shop.oreilly.com/category/early-release.do
- tatsuke95 14y ago
  Stupid publishers. I'm buying the hard copy, because I love books, but they don't allow me to get in on the early release? They need to offer an "all editions" package.

brianobush 14y ago

does this include numpy, or is it just pandas?

wesm 14y ago
NumPy, IPython, matplotlib, pandas, a little bit of SciPy, scikit-learn, statsmodels, and plenty other fun stuff
- neovive 14y ago
  Sold! Any suggested prerequisites?
  - wesm 14y ago
    Some working general purpose Python knowledge would be helpful but not required

apechai 14y ago

Ayone have experience as to how Python compare to R, Matlab (Octave) and other tools for data analysis? R has great libraries but I would prefer to use Python.

aggie 14y ago
You might want to look into RPy for using R libraries with python.
- dbecker 14y ago
  I've very briefly experimented with rpy2. It got the job done, but I thought it was tricky enough that I'd want a good reason to combine R and python. Otherwise, I'd try and do the whole project in one or the other (And I haven't used rpy2 since I first tried it out.)
dbecker 14y ago
I've done a lot of work using both R and Python, and a little bit using Matlab (and SAS and Stata). Matlab is behind R and Python for both data cleaning and analysis. This is especially true if you have string variables and factor variables. Matlab's great if you are doing matrix operations of clean data (and you don't need to do anything fancy in how you report them). But, I don't find it worth using for real data analysis. Python and R are both great, though they feel pretty similar to me. Python obviously has the big advantage if you want to do general purpose computing too. Python has been faster than R in most cases I've compared them. For my work, development speed is more important than execution speed... so this hasn't been a huge factor for me. R has a couple of big advantages. First, the libraries. This is a big deal for me. I saw that there is a python equivalent to ggplot2 in the works. This will definitely strengthen the case for python, but the availability of libraries in R is awesome. Second, the community and help resources in R are amazing. I rarely run into a problems in R that haven't already been addressed on stackexchange.com. Perhaps I should be more proactive about asking python questions when I run into them, but I usually just work it out myself (which is more time consuming than looking the answer up online.) Lastly, I'm not an expert on big data. But, spending relatively little time with both R's bigmemory and Python's PyTables, it seems easier to get up to speed on big data with R at the moment. Though I haven't met them, my sense is that Wes, Travis Oliphant and the other relevant python developers are putting in a heroic effort to get Python up to speed. I have every expectation that Python will be my choice of the future.

joshu 14y ago

I'm thrilled that Pandas is named Pandas and not Pyndas.

hkmurakami 14y ago

As I get my feet wet again in programming, I've gone through phases in dabbling with web, app, and game programming. But about a month ago I realized, "You know, what I really loved about my studies back in academic years was numbers. That's what I want to do with code; I want to crunch numbers" This realization made me switch over from learning Ruby to learning Python. This book is going to be perfect for my interests. Wes, thank you for your efforts in putting this book together.

Cieplak 14y ago
If you really want to crunch numbers, I highly recommend Haskell. It falls short of Python in terms of libraries and by some accounts, readability, but it is substantially faster and excels at number crunching. I find it easier to express number crunching in functional rather than object-oriented style programming. Python has lambdas, but usually theres a more pythonic way to express something in Python. Although I must say, I find Haskell's list comprehension syntax slightly more elegant. http://tryhaskell.org/ http://tryhaskell.org/ edit: I take it back about Haskell's list comprehension syntax. At the end of the day they're both great. Haskell's syntax just feels like reading set theory notation.
- hkmurakami 14y ago
  Thanks for the advice :) I haven't programmed in any substantial quantity or quality since 2004, so I'll start with a more approachable language (Python), but I'll definitely keep an open mind.
  - alexchamberlain 14y ago
    Do keep an open mind, but bear in mind that Haskell is a different style of programming all together and there are libraries to speed number crunching in Python (NumPy).
- carterschonwald 14y ago
  And there's also a great and growing set of tools for numerical and data related computation. Scratch that, the current ongoing dev and research work that is making its way onto the haskell hackage is both astonishing and often superlative code. The libs released in the past month and the ones I know will be made available over the coming months... It's just great work! (I'm biased owing To using some of these tools for fun and profit :-). ) That aside, Pandas is a really nice piece of engineering that really works. It's a good role model for other libs, even when restricted to just its data frame part

look_lookatme 14y ago

Don't forget to brush up on the basics (with Python!): http://greenteapress.com/thinkstats/ http://greenteapress.com/thinkstats/

neves 14y ago
And don't forget to say that this guy publishes a free pdf version in the website above. I'm reading it now in my Kindle.

wgrover 14y ago

wesm, do you cover using cloud-based computing resources for tackling computationally intensive data analysis problems? I use Python and PiCloud to turn what is an hours-long data analysis problem on my Mac into a minutes-long job on Amazon's hardware. I don't know MATLAB, R, etc. well enough to say whether they can use the cloud as easily as Python, but it sure was easy with Python.

wesm 14y ago
The focus is primarily on data manipulation / preparation / cleaning / processing tools, data algorithms, visualization, and practical analysis case studies. The platform where you use them is a separate matter-- I will have some material on distributed computing tools, however.
- wgrover 14y ago
  Sounds good. I can't tell you how thrilled I am to have a book like this coming out. I'm very interested in using it as a text for an instrumental analysis class I'm putting together. Thanks so much!
msellout 14y ago
For R, MATLAB, or Python in a cloud Map/Reduce framework, try out Opani. http://opani.com/help/wiki http://opani.com/help/wiki
- wgrover 14y ago
  Didn't know about that one - thanks for the tip!

npcomplete 14y ago

Can we get the table of contents?

wesm 14y ago
I'll blog/tweet/G+ a ToC pretty soon, doing a bit of reorganization before the next Early Release update
pwang 14y ago
If you're in the LinkedIn "Python Data" group, I posted a comment with a list of the ToC (hope you don't mind, Wes!) http://www.linkedin.com/groupItem?view=&gid=4388870&type=member&item=115808571&commentID=81103690&qid=043286c4-a34b-4614-8cde-95b8a79b8087&trk=group_most_popular-0-b-cmn&goback=%2Egmp_4388870#commentID_81103690 http://www.linkedin.com/groupItem?view=&gid=4388870&...

sodelate 14y ago

can we use it for trade analysis?

wesm 14y ago
Without a doubt (how I got my start in Python)

ruffyen 14y ago

Looking at the table of contents, this seems like a great reference book for several technologies. I think I am sold, especially with the free updates.

salimmadjd 14y ago

I recommend reading this blog,for those interested in crunching numbers on python http://slendrmeans.wordpress.com/ http://slendrmeans.wordpress.com/

neves 14y ago

Where's the table of contents?