7 ms·

Self-Directed Pandas Crash Course

109 points by luu 6y ago

nolroz 6y ago
I like the outline you have here for a crash course on Pandas. I've been tinkering with it myself off and on for a while but have been wanting to to really dig in lately for the same reasons you mention. Just a couple of nitpicks: 1. The style of your website makes it pretty much impossible to tell if a bit of text is linked anywhere. I only figured it out after clicking on the word "here" in the first paragraph and coming back and clicking on essentially everything. None of this happened until after I visited the site on my desktop instead of my mobile and slowed down to read things carefully. 2. It would be great to get a link to the denvergov.org data set, or corresponding area of the site.
- kellyfoulk 6y ago
  Thanks! I enjoyed working through the 100 Pandas Puzzles repo and would recommend it - I've found myself going back to reference my answers many times. Thanks for the critique on the links, I hadn't realized how hard they were to see until now. I'm in the middle of refactoring the site and I'll add that to the to-do list.
- yardshop 6y ago
  He links it in the first sentence under Resources. As you mention, it's a "here" link with no styling. One shouldn't have to, but FWIW the HTML of his page is very clean so you can see all of the links by viewing the source.
  - gk1 6y ago
    *She
systemvoltage 6y ago
Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but its one of those pieces of software that has a very non-intuitive API. Does any one find it difficult to use it or its just me? In particular, I find this answer infuriating [1]. I've come across it so many times. Look, I have a CSV with 200 rows and I need to loop through them in the most intuitive way. Sure, its not optimal but I don't want fast code. I have a mental model of how to modify this dataframe. Let me do it, please. [1] https://stackoverflow.com/a/55557758 https://stackoverflow.com/a/55557758
- jzwinck 6y ago
  Here are two options which various people might find intuitive: 1. Make a good old numeric for loop like "for ii in range(len(df))" with df.iloc or df.ix, etc. 2. Use df.apply() to create a new DataFrame with your changes. Both of these are mentioned in brief answers to that SO post. But not in the accepted answer. A lot of the answers focus on the most efficient ways to do things, even though the question was very basic and did not imply the data were large.
  - smabie 6y ago
    It's not even about performance, writing vectorized code is just superior. The array language model is very powerful and you can literally write a couple lines of code that would take hundreds of lines of normal Python. That being said, Pandas isn't a very good array language. I really like kdb+/q and find it better and more expressive than almost any other language I've used.
- qwantim1 6y ago
  I just read the GitHub repo mentioned in the article and it looks like gibberish to me. Pandas to me are still big lazy black and white bears eating bamboo, until someone can point me to something more intelligible.
- chaps 6y ago
  100% my experience whenever I work with it, and I've been working with it on and off for about five years. I get its appeal -- it fits that fuzzy place where a database is too heavy, but vanilla python is cumbersome... but damn, is it tough to work with. With almost no fail, I always seem to just scrap the pandas code in favor of vanilla python, either from usability issues, or from yet another a library's false promise that it works well with dataframes. So many hours lost in using pandas. That said, one of its best features, and probably the only thing I use it for these days is, `pd.read_sql(sqlstr, conn).to_csv(fp)`. This is far less cumbersome than using psycopg2. Edit: for charting these days, similar to OP's visualiztions, I highly recommend vega-lite.
  - 6y ago
asicsp 6y ago
Here's another recent resource: https://www.gormanalysis.com/blog/python-pandas-for-your-grandpa/ https://www.gormanalysis.com/blog/python-pandas-for-your-gra...
- PlugTunin 6y ago
  Subscribed. Thanks for putting together this quality content. I was checking out your merge dataframes video the other day, that I was led to from a Stack Overflow post. I'm not a novice w/Pandas but I find that things stick better after repeated exposure, from different sources at that. The various ways of using merge/join/concat can be tricky to keep straight at first. I will be sure to dig deeper into your content in the near future, including the series of videos on Numpy.
  - asicsp 6y ago
    Not mine, I'd have mentioned if it was my resource. I saw it on: https://www.reddit.com/r/Python/comments/lain0r/hey_reddit_heres_my_comprehensive_course_on/ https://www.reddit.com/r/Python/comments/lain0r/hey_reddit_h...
DoofusOfDeath 6y ago
I'm delighted at the multiple ways one can parse the headline!
- ayewo 6y ago
  Yea :) * Self-Directed, Pandas Crash Course * Self-Directed Pandas, Crash Course Rewriting as Self-Directed Crash Course on Pandas would eliminate the ambiguity.
  - kellyfoulk 6y ago
    Good point, I'll add that to my to-do list
- StringyBob 6y ago
  Well that’s apt for Pandas: Eats, Shoots & Leaves https://en.wikipedia.org/wiki/Eats,_Shoots_%26_Leaves https://en.wikipedia.org/wiki/Eats,_Shoots_%26_Leaves
GreenWatermelon 6y ago
I think Corey Shafer's tutorial[1] is a great place to get started. His voice is clear even on 2x and he doesn't bore me [1] https://youtube.com/playlist?list=PL-osiE80TeTsWmV9i9c58mdDCSskIFdDS https://youtube.com/playlist?list=PL-osiE80TeTsWmV9i9c58mdDC...
- kellyfoulk 6y ago
  I love all of his tutorials, good recommendation
ivan_ah 6y ago
Khuyen Tran has a nice collection of bite-sized tricks and tips for learning Pandas on her blog: https://mathdatasimplified.com/page/1/?s=pandas https://mathdatasimplified.com/page/1/?s=pandas https://mathdatasimplified.com/page/2/?s=pandas https://mathdatasimplified.com/page/2/?s=pandas https://mathdatasimplified.com/page/3/?s=pandas https://mathdatasimplified.com/page/3/?s=pandas It's not like I'll remember all the syntax, but good to know what tools/tricks exist.

109 points by luu 6y ago

I like the outline you have here for a crash course on Pandas. I've been tinkering with it myself off and on for a while but have been wanting to to really dig in lately for the same reasons you mention. Just a couple of nitpicks: 1. The style of your website makes it pretty much impossible to tell if a bit of text is linked anywhere. I only figured it out after clicking on the word "here" in the first paragraph and coming back and clicking on essentially everything. None of this happened until after I visited the site on my desktop instead of my mobile and slowed down to read things carefully. 2. It would be great to get a link to the denvergov.org data set, or corresponding area of the site.

kellyfoulk 6y ago
Thanks! I enjoyed working through the 100 Pandas Puzzles repo and would recommend it - I've found myself going back to reference my answers many times. Thanks for the critique on the links, I hadn't realized how hard they were to see until now. I'm in the middle of refactoring the site and I'll add that to the to-do list.
yardshop 6y ago
He links it in the first sentence under Resources. As you mention, it's a "here" link with no styling. One shouldn't have to, but FWIW the HTML of his page is very clean so you can see all of the links by viewing the source.
- gk1 6y ago
  *She

systemvoltage 6y ago

Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but its one of those pieces of software that has a very non-intuitive API. Does any one find it difficult to use it or its just me? In particular, I find this answer infuriating [1]. I've come across it so many times. Look, I have a CSV with 200 rows and I need to loop through them in the most intuitive way. Sure, its not optimal but I don't want fast code. I have a mental model of how to modify this dataframe. Let me do it, please. [1] https://stackoverflow.com/a/55557758 https://stackoverflow.com/a/55557758

jzwinck 6y ago
Here are two options which various people might find intuitive: 1. Make a good old numeric for loop like "for ii in range(len(df))" with df.iloc or df.ix, etc. 2. Use df.apply() to create a new DataFrame with your changes. Both of these are mentioned in brief answers to that SO post. But not in the accepted answer. A lot of the answers focus on the most efficient ways to do things, even though the question was very basic and did not imply the data were large.
- smabie 6y ago
  It's not even about performance, writing vectorized code is just superior. The array language model is very powerful and you can literally write a couple lines of code that would take hundreds of lines of normal Python. That being said, Pandas isn't a very good array language. I really like kdb+/q and find it better and more expressive than almost any other language I've used.
qwantim1 6y ago
I just read the GitHub repo mentioned in the article and it looks like gibberish to me. Pandas to me are still big lazy black and white bears eating bamboo, until someone can point me to something more intelligible.
chaps 6y ago
100% my experience whenever I work with it, and I've been working with it on and off for about five years. I get its appeal -- it fits that fuzzy place where a database is too heavy, but vanilla python is cumbersome... but damn, is it tough to work with. With almost no fail, I always seem to just scrap the pandas code in favor of vanilla python, either from usability issues, or from yet another a library's false promise that it works well with dataframes. So many hours lost in using pandas. That said, one of its best features, and probably the only thing I use it for these days is, `pd.read_sql(sqlstr, conn).to_csv(fp)`. This is far less cumbersome than using psycopg2. Edit: for charting these days, similar to OP's visualiztions, I highly recommend vega-lite.
- 6y ago

asicsp 6y ago

Here's another recent resource: https://www.gormanalysis.com/blog/python-pandas-for-your-grandpa/ https://www.gormanalysis.com/blog/python-pandas-for-your-gra...

PlugTunin 6y ago
Subscribed. Thanks for putting together this quality content. I was checking out your merge dataframes video the other day, that I was led to from a Stack Overflow post. I'm not a novice w/Pandas but I find that things stick better after repeated exposure, from different sources at that. The various ways of using merge/join/concat can be tricky to keep straight at first. I will be sure to dig deeper into your content in the near future, including the series of videos on Numpy.
- asicsp 6y ago
  Not mine, I'd have mentioned if it was my resource. I saw it on: https://www.reddit.com/r/Python/comments/lain0r/hey_reddit_heres_my_comprehensive_course_on/ https://www.reddit.com/r/Python/comments/lain0r/hey_reddit_h...

DoofusOfDeath 6y ago

I'm delighted at the multiple ways one can parse the headline!

ayewo 6y ago
Yea :) * Self-Directed, Pandas Crash Course * Self-Directed Pandas, Crash Course Rewriting as Self-Directed Crash Course on Pandas would eliminate the ambiguity.
- kellyfoulk 6y ago
  Good point, I'll add that to my to-do list
StringyBob 6y ago
Well that’s apt for Pandas: Eats, Shoots & Leaves https://en.wikipedia.org/wiki/Eats,_Shoots_%26_Leaves https://en.wikipedia.org/wiki/Eats,_Shoots_%26_Leaves

GreenWatermelon 6y ago

I think Corey Shafer's tutorial[1] is a great place to get started. His voice is clear even on 2x and he doesn't bore me [1] https://youtube.com/playlist?list=PL-osiE80TeTsWmV9i9c58mdDCSskIFdDS https://youtube.com/playlist?list=PL-osiE80TeTsWmV9i9c58mdDC...

kellyfoulk 6y ago
I love all of his tutorials, good recommendation

ivan_ah 6y ago

Khuyen Tran has a nice collection of bite-sized tricks and tips for learning Pandas on her blog: https://mathdatasimplified.com/page/1/?s=pandas https://mathdatasimplified.com/page/1/?s=pandas https://mathdatasimplified.com/page/2/?s=pandas https://mathdatasimplified.com/page/2/?s=pandas https://mathdatasimplified.com/page/3/?s=pandas https://mathdatasimplified.com/page/3/?s=pandas It's not like I'll remember all the syntax, but good to know what tools/tricks exist.