6 ms·

Python for Data Analysis, 3rd Edition

271 points by mariuz 4y ago

johnpublic 4y ago
You can save so much time and effort using the Google colab notebooks rather than setting up python on your own machine (as is recommended in this guide)
- savant_penguin 4y ago
  Although this is true I find these online notebooks awfully slow
- TimSchumann 4y ago
  I think there may be many a use case where the data being operated on cannot be shared with third parties.
- mkl 4y ago
  You can save a little bit of copy-paste that will take a few minutes at most. If you do it, you can work directly with your own files, work offline, control the hardware, etc. I think it's simpler than this guide makes out too, as it tries to minimise the amount of space used, which is often not needed. Installing Anaconda instead of Miniconda would get you pretty much set up in one step, plus a single copy-paste step if you want all the packages the book uses.
- atty 4y ago
  Downloading an installing Anaconda is pretty much painless and gives you more flexibility (and better responsiveness) than collab.
  - auxym 4y ago
    Not sure about colab, but its important to note that Anaconda is not free for commercial use.
    - prepend 4y ago
      Miniconda is free and is great for running jupyter.
    - Tarq0n 4y ago
      They keep making confusing statements about this, I thought you only needed a license for CI/CD style usage of their repository?
  - deleted 4y ago
    [deleted]
- analog31 4y ago
  Worth mentioning Jupyter Lite in this context too. Warning: This link will open a Jupyter notebook in your browser: https://jupyter.org/try-jupyter/lab/ It's worked pretty smoothly for me so far. I can't vouch for how it handles big data sets or obscure libraries, but seems like a pretty good starting point for those who are learning Python. It has become how I prefer to share simple notebooks with colleagues too. But either of these options is nice for dealing with the situation of getting a beginner through the Python installation process. Another is WinPython, which is my preferred environment for local installation.
- tmaly 4y ago
  Are there anything like this online that could run stuff written for pygame? I know lots of beginners start with Scratch, but having some type of gaming in the browser for Python would be nice.
- anigbrowl 4y ago
  Absolutely true, but learning things that hard way was worth it to me. Plus I am old-fashioned enough to like doing things on my own hardware and not necessarily wanting to share my data/code every time for reasons of security or modesty (as in embarrassingly basic). I do like what Colab offers and appreciate having all that processing power/infrastructure available.
wirthjason 4y ago
The first edition of this book was a game changer. I still have my hardcopy paperback but haven’t turned to it in years. It’s probably largely out of date but I look at it fondly. Glad to see the material updated with new APIs and Python 3.10. The open access is very nice because it can easily be used for teaching and reference. I would like to see some of Wes’ cautionary tales included into the book. Pandas feels so magical at times that you can uses it to solve anything. That can create problems. Ideas like this should be some appendix to the book. https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/ Lots of tools in the data space. It feels like choosing JS web front ends. I use pandas quite a lot but less frequently, as my data has gotten larger I turn to spark more. I’m also impressed by the work DuckDB is doing.
- wdroz 4y ago
  I switched from pandas to polars[0] . I'm happy with the performance gain. You still need to use Spark/dask on "big" data, but polars give you more room before reaching to these solutions. [0] -- https://www.pola.rs/ https://www.pola.rs/
  - mistrial9 4y ago
    glad it works for you but this sounds like bad advice for most people.. why change to an outlier toolchain for some percentage increase in performance and raw data capacity, when you admit yourself that many Big Data sets are not fitting there? No computer on my desk has less than 32GB RAM now -- pandas works well with that.
    - Dzugaru 4y ago
      Some percentage? I haven't used polars, but I ditched pandas after observing x20 speedup by just rewriting a piece of code to plain csv.reader. Pandas is unacceptably slow for medium data, period.
      - curiousgal 4y ago
neves 4y ago
Very nice to have an open access version. The paper version is really expensive if your salary isn't in dollars. I've regretted to by the print Brazilian edition. O'Reilly is my favorite technical publisher, but the non English editions are crap. My book didn't have an index, making it useless as a reference.
TrackerFF 4y ago
I use Python extensively for my analysis projects - and while Pandas is my go-to library for many things, I feel it's just very slow. I know that just doing stuff in numpy instead speeds things up considerably, and that's way before doing other optimization, but are there any other libraries out there similar to Pandas, but made for higher performance?
- HFguy 4y ago
  Polars and Modin support multiple cores and have optimizations. Modin has most of the Pandas api supported. Pyspark api for pandas supports multiple cores and clusters.
- kzrdude 4y ago
  polars, but it's still developing.
- sireat 4y ago
  Curious where do you experience slowness with Pandas? Usually slowness in Pandas will be if you start doing non vector things. Whenever you catch yourself writing a loop in Pandas you know you've gone the wrong way. I too use Python extensively (with Pandas among other things) and usually Pandas is perfectly fine. (havent gone over 64GB memory usage yet) I consider Pandas to be Numpy with benefits (methods) - since Pandas dataframe is basically just a collection of Numpy arrays. Those are about as close to C type arrays you can get in Python. The only practical problems have been with regressions among supporting libraries such as pyarrow not playing nice with numpy when working with parquet files.
  - anothernewdude 4y ago
    Slowness from pandas mostly comes when people forget to use df.values before running their algorithm.
civilized 4y ago
Still waiting for Wes to admit that the pandas API is a mess and support the development and adoption of a more dplyr-like Python library. Pandas was a great step on the path to making Python a decent data analysis language, Wes is smarter than me, I never could have built it, but it's time to move on.
- closed 4y ago
  Hey--I maintain a port of dplyr to python, called siuba[1]! Right now it supports a decent number of verbs + SQL generation. I tried to break down why R users find pandas difficult in an RStudioConf talk last year[2]. Between siuba and tools like polars and duckdb, I'm hopeful that someone hits the data analysis sweet spot for python in the next couple years. [1]: http://github.com/machow/siuba http://github.com/machow/siuba [2]: https://youtu.be/w4Mi0u4urbQ https://youtu.be/w4Mi0u4urbQ
  - sebastiansm 4y ago
    I learned data analysis in the Hadley/Tidyverse way and constantly struggle when working with pandas. I'll try siuba this week at work. Just one question, this runs over pandas? It's possible to get the pandas syntax like dbplyr gives the SQL query syntax?
    - closed 4y ago
      Yeah, it runs over pandas or sql databases! Here's an example querying sql: https://siuba.readthedocs.io/en/latest/intro.html#Working-with-SQL https://siuba.readthedocs.io/en/latest/intro.html#Working-wi... You can use the verbs collect() and show_query() like in dbplyr. If you DM me on twitter, would love to set up time to hear about your work / walk through siuba! https://twitter.com/chowthedog https://twitter.com/chowthedog
- RobinL 4y ago
  I think that's part of the point of his current project: Arrow. Specifically, one of its goals is to implement the low level computations, whilst enabling more competition in the space of data analysis API design.
argella 4y ago
anyone have a suggestion for a best online course on Python programming? Wanting to focus on Python the language and getting solid at programming, not necessarily a particular library. I’m pretty rusty at programming having last studied formally algorithms, data structures, oop in c++ 15 years ago in an undergrad compsci program. I’ve done mostly sql coding and database work ever since but need to level up my skill set.
dang 4y ago
Related: Python for Data Analysis – A Critical Line-By-Line Reivew - https://news.ycombinator.com/item?id=15786977 https://news.ycombinator.com/item?id=15786977 - Nov 2017 (3 comments) Ask HN: What are the best resources to learn Python for Data Analysis - https://news.ycombinator.com/item?id=12775775 https://news.ycombinator.com/item?id=12775775 - Oct 2016 (15 comments) Python for Data Analysis - https://news.ycombinator.com/item?id=4697665 https://news.ycombinator.com/item?id=4697665 - Oct 2012 (28 comments) Escaping The Walled Garden of Enterprise Analytics Using R and Python - https://news.ycombinator.com/item?id=4624186 https://news.ycombinator.com/item?id=4624186 - Oct 2012 (1 comment) Python for Data Analysis (new O'Reilly book from creator of Pandas) - https://news.ycombinator.com/item?id=4020187 https://news.ycombinator.com/item?id=4020187 - May 2012 (57 comments) Also: Wes McKinney, the developer of Pandas - https://news.ycombinator.com/item?id=15878616 https://news.ycombinator.com/item?id=15878616 - Dec 2017 (50 comments)
timcavel 4y ago

271 points by mariuz 4y ago

You can save so much time and effort using the Google colab notebooks rather than setting up python on your own machine (as is recommended in this guide)

savant_penguin 4y ago
Although this is true I find these online notebooks awfully slow
TimSchumann 4y ago
I think there may be many a use case where the data being operated on cannot be shared with third parties.
mkl 4y ago
You can save a little bit of copy-paste that will take a few minutes at most. If you do it, you can work directly with your own files, work offline, control the hardware, etc. I think it's simpler than this guide makes out too, as it tries to minimise the amount of space used, which is often not needed. Installing Anaconda instead of Miniconda would get you pretty much set up in one step, plus a single copy-paste step if you want all the packages the book uses.
atty 4y ago
Downloading an installing Anaconda is pretty much painless and gives you more flexibility (and better responsiveness) than collab.
- auxym 4y ago
  Not sure about colab, but its important to note that Anaconda is not free for commercial use.
  - prepend 4y ago
    Miniconda is free and is great for running jupyter.
  - Tarq0n 4y ago
    They keep making confusing statements about this, I thought you only needed a license for CI/CD style usage of their repository?
- deleted 4y ago
  [deleted]
analog31 4y ago
Worth mentioning Jupyter Lite in this context too. Warning: This link will open a Jupyter notebook in your browser: https://jupyter.org/try-jupyter/lab/ It's worked pretty smoothly for me so far. I can't vouch for how it handles big data sets or obscure libraries, but seems like a pretty good starting point for those who are learning Python. It has become how I prefer to share simple notebooks with colleagues too. But either of these options is nice for dealing with the situation of getting a beginner through the Python installation process. Another is WinPython, which is my preferred environment for local installation.
tmaly 4y ago
Are there anything like this online that could run stuff written for pygame? I know lots of beginners start with Scratch, but having some type of gaming in the browser for Python would be nice.
anigbrowl 4y ago
Absolutely true, but learning things that hard way was worth it to me. Plus I am old-fashioned enough to like doing things on my own hardware and not necessarily wanting to share my data/code every time for reasons of security or modesty (as in embarrassingly basic). I do like what Colab offers and appreciate having all that processing power/infrastructure available.

wirthjason 4y ago

The first edition of this book was a game changer. I still have my hardcopy paperback but haven’t turned to it in years. It’s probably largely out of date but I look at it fondly. Glad to see the material updated with new APIs and Python 3.10. The open access is very nice because it can easily be used for teaching and reference. I would like to see some of Wes’ cautionary tales included into the book. Pandas feels so magical at times that you can uses it to solve anything. That can create problems. Ideas like this should be some appendix to the book. https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/ Lots of tools in the data space. It feels like choosing JS web front ends. I use pandas quite a lot but less frequently, as my data has gotten larger I turn to spark more. I’m also impressed by the work DuckDB is doing.

wdroz 4y ago
I switched from pandas to polars[0] . I'm happy with the performance gain. You still need to use Spark/dask on "big" data, but polars give you more room before reaching to these solutions. [0] -- https://www.pola.rs/ https://www.pola.rs/
- mistrial9 4y ago
  glad it works for you but this sounds like bad advice for most people.. why change to an outlier toolchain for some percentage increase in performance and raw data capacity, when you admit yourself that many Big Data sets are not fitting there? No computer on my desk has less than 32GB RAM now -- pandas works well with that.
  - Dzugaru 4y ago
    Some percentage? I haven't used polars, but I ditched pandas after observing x20 speedup by just rewriting a piece of code to plain csv.reader. Pandas is unacceptably slow for medium data, period.
    - curiousgal 4y ago

neves 4y ago

Very nice to have an open access version. The paper version is really expensive if your salary isn't in dollars. I've regretted to by the print Brazilian edition. O'Reilly is my favorite technical publisher, but the non English editions are crap. My book didn't have an index, making it useless as a reference.

TrackerFF 4y ago

I use Python extensively for my analysis projects - and while Pandas is my go-to library for many things, I feel it's just very slow. I know that just doing stuff in numpy instead speeds things up considerably, and that's way before doing other optimization, but are there any other libraries out there similar to Pandas, but made for higher performance?

HFguy 4y ago
Polars and Modin support multiple cores and have optimizations. Modin has most of the Pandas api supported. Pyspark api for pandas supports multiple cores and clusters.
kzrdude 4y ago
polars, but it's still developing.
sireat 4y ago
Curious where do you experience slowness with Pandas? Usually slowness in Pandas will be if you start doing non vector things. Whenever you catch yourself writing a loop in Pandas you know you've gone the wrong way. I too use Python extensively (with Pandas among other things) and usually Pandas is perfectly fine. (havent gone over 64GB memory usage yet) I consider Pandas to be Numpy with benefits (methods) - since Pandas dataframe is basically just a collection of Numpy arrays. Those are about as close to C type arrays you can get in Python. The only practical problems have been with regressions among supporting libraries such as pyarrow not playing nice with numpy when working with parquet files.
- anothernewdude 4y ago
  Slowness from pandas mostly comes when people forget to use df.values before running their algorithm.

civilized 4y ago

Still waiting for Wes to admit that the pandas API is a mess and support the development and adoption of a more dplyr-like Python library. Pandas was a great step on the path to making Python a decent data analysis language, Wes is smarter than me, I never could have built it, but it's time to move on.

closed 4y ago
Hey--I maintain a port of dplyr to python, called siuba[1]! Right now it supports a decent number of verbs + SQL generation. I tried to break down why R users find pandas difficult in an RStudioConf talk last year[2]. Between siuba and tools like polars and duckdb, I'm hopeful that someone hits the data analysis sweet spot for python in the next couple years. [1]: http://github.com/machow/siuba http://github.com/machow/siuba [2]: https://youtu.be/w4Mi0u4urbQ https://youtu.be/w4Mi0u4urbQ
- sebastiansm 4y ago
  I learned data analysis in the Hadley/Tidyverse way and constantly struggle when working with pandas. I'll try siuba this week at work. Just one question, this runs over pandas? It's possible to get the pandas syntax like dbplyr gives the SQL query syntax?
  - closed 4y ago
    Yeah, it runs over pandas or sql databases! Here's an example querying sql: https://siuba.readthedocs.io/en/latest/intro.html#Working-with-SQL https://siuba.readthedocs.io/en/latest/intro.html#Working-wi... You can use the verbs collect() and show_query() like in dbplyr. If you DM me on twitter, would love to set up time to hear about your work / walk through siuba! https://twitter.com/chowthedog https://twitter.com/chowthedog
RobinL 4y ago
I think that's part of the point of his current project: Arrow. Specifically, one of its goals is to implement the low level computations, whilst enabling more competition in the space of data analysis API design.

argella 4y ago

anyone have a suggestion for a best online course on Python programming? Wanting to focus on Python the language and getting solid at programming, not necessarily a particular library. I’m pretty rusty at programming having last studied formally algorithms, data structures, oop in c++ 15 years ago in an undergrad compsci program. I’ve done mostly sql coding and database work ever since but need to level up my skill set.

dang 4y ago

Related: Python for Data Analysis – A Critical Line-By-Line Reivew - https://news.ycombinator.com/item?id=15786977 https://news.ycombinator.com/item?id=15786977 - Nov 2017 (3 comments) Ask HN: What are the best resources to learn Python for Data Analysis - https://news.ycombinator.com/item?id=12775775 https://news.ycombinator.com/item?id=12775775 - Oct 2016 (15 comments) Python for Data Analysis - https://news.ycombinator.com/item?id=4697665 https://news.ycombinator.com/item?id=4697665 - Oct 2012 (28 comments) Escaping The Walled Garden of Enterprise Analytics Using R and Python - https://news.ycombinator.com/item?id=4624186 https://news.ycombinator.com/item?id=4624186 - Oct 2012 (1 comment) Python for Data Analysis (new O'Reilly book from creator of Pandas) - https://news.ycombinator.com/item?id=4020187 https://news.ycombinator.com/item?id=4020187 - May 2012 (57 comments) Also: Wes McKinney, the developer of Pandas - https://news.ycombinator.com/item?id=15878616 https://news.ycombinator.com/item?id=15878616 - Dec 2017 (50 comments)

timcavel 4y ago