5 ms·

Python Data Science Handbook

312 points by cl3misch 10mo ago

BenGosub 10mo ago
He's a great writer and I miss his blog. He had an awesome post on pivot table that I think is now a part of this book.
- ayhanfuat 10mo ago
  He is also the creator of the Altair visualization library (Vega-Lite in Python https://altair-viz.github.io/ https://altair-viz.github.io/). I really like using it.
  - AI-NoGuardrails 10mo ago
    [flagged]
  - linhns 10mo ago
    Thanks for the fact, I used Altair sometimes and really admire the simplicity, not knowing it was written by Jake.
ellisv 10mo ago
These types of books are always interesting to me because they tackle so many different things. They cover a range of topics at a high level (data manipulation, visualization, machine learning) and each could have its own book. They balance teaching programming while introducing concepts (and sometimes theory). In short I think it's hard to strike an appropriate balance between these but this seems to be a good intro level book.
sschnei8 10mo ago
Interesting choice of Pandas in this day and age. Maybe he’s after imparting general concepts that you could apply to any tabular data manipulator rather than selecting for the latest shiny tool.
- msto 10mo ago
  It was originally published in 2016, and I think this is still the first edition.
  - mkl 10mo ago
    Looks like it. From https://jakevdp.github.io/PythonDataScienceHandbook/00.00-preface.html https://jakevdp.github.io/PythonDataScienceHandbook/00.00-pr...: > Copyright 2016
- xenophonf 10mo ago
  What's wrong with Pandas?
  - clickety_clack 10mo ago
    I probably wouldn’t rewrite an entire data science stack that used pandas, but most people would use polars if starting a new project today.
    - deleted 10mo ago
      [deleted]
    - biofox 10mo ago
      R and Matlab workflows have been fairly stable for the past decade. Why is the Python ecosystem so... unstable? It puts me off investing any time in it.
      - clickety_clack 10mo ago
        The R ecosystem has had a similar evolution with the tidyverse, it was just a little further ago. As for Matlab, I initially learned statistical programming with it a long time ago, but I’m not sure I’ve ever seen it in the wild. I don’t know what’s going on there. I’m actually quite partial to R myself, and I used to use it extensively back when quick analysis was more valuable to my career. Things have probably progressed, but I dropped it in favor of python because python can integrate into production systems whereas R was (and maybe still is) geared towards writing reports. One of the best things to happen recently in data science is the plotnine library, bringing the grammar of graphics to python imho. The fact is that today, if you want career opportunities as a data scientist, you need to be fluent in python.
wiz21c 10mo ago
I wouldn't say it's a handbook because it's more like an introduction. But it's pretty well written.
farhanhubble 10mo ago
I loved his Statistics for Hackers talk: https://speakerdeck.com/pycon2016/jake-vanderplas-statistics-for-hackers https://speakerdeck.com/pycon2016/jake-vanderplas-statistics...
- yboris 10mo ago
  Amazing Thank you for sharing. Reminds me of how thinking using frequencies rather than computing probabilities is easier and can avoid errors (e.g. a 99% accurate test being positive does not mean 99% likelihood of having disease for a disease with a 1/10,000 prevalence in population).
synergy20 10mo ago
it's written 8 years ago though, there is a 2ed of the book by the same author.
- phone_book 10mo ago
  The linked Github seems to have the 2nd edition in the form of notebooks, https://github.com/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/00.00-Preface.ipynb https://github.com/jakevdp/PythonDataScienceHandbook/blob/ma..., under the Using Code Examples section, "attribution usually includes the title, author, publisher, and ISBN. For example: "Python Data Science Handbook, 2nd edition, by Jake VanderPlas (O’Reilly). Copyright 2023..." compared to the OP's link which has "The Python Data Science Handbook by Jake VanderPlas (O’Reilly). Copyright 2016..."
- mayankkaizen 10mo ago
  There is a second edition?
__rito__ 10mo ago
This is one of the few books that I read cover-to-cover when I was starting out learning Data Science in 2020/21. Will recommend.
trio8453 10mo ago
This book was absolute fire for getting started with data science in 2017-2018, Jake is a great teacher.
- mayankkaizen 10mo ago
  This was the solid book to get into data science and ML scene. Covers everything. Jake is a fantastic teacher. I really wish he comes up with updated second edition.
badmonster 10mo ago
[flagged]
- deleted 10mo ago
  [deleted]
pantsforbirds 10mo ago
I used the Kernel Density Estimation (KDE) page/blog at my very first job. It was immensely useful and I've loved his work ever since.
pid-1 10mo ago
Pandas is cancer. Please stop teaching it to people. Everything it does can be done reasonable well with list comprehensions and objects that support type annotations and runtime type checking (if needed). Pandas code is untestable, unreadable, hard to refactor and impossible to reuse. Trillions of dollars are wasted every year by people having to rewrite pandas code.
- mttpgn 10mo ago
  > Pandas code is untestable The thousand-plus data integrity tests I've written in pandas tell a different story...
- physicsguy 10mo ago
  I found Pandera quite good for wrapping input/output expectations over Pandas. At the end of the day the vectorisation of operations in it and other table based formats mean they’re not easy to replace performantly.
- mulmboy 10mo ago
  > Everything it does can be done reasonable well with list comprehensions and objects that support type annotations and runtime type checking (if needed). I see this take somewhat often, and usually with similar lack of nuance. How do you come to this? In other cases where I've seen this it's from people who haven't worked in any context where performance or scientific computing ecosystem interoperability matters - missing a massive part of the picture. I've struggled to get through to them before. Genuine question.
- isolatedsystem 10mo ago
  I've recently had to migrate over to Python from Matlab. Pandas has been doing my head in. The syntax is so unintuitive. In Matlab, everything begins with a `for` loop. Inelegant and slow, yes, but easy to reason about. Easy to see the scope and domain of the problem, to visualise the data wrangling. Pandas insist you never use a for loop. So, I feel guilty if I ever need a throwaway variable on the way to creating a new column. Sometimes methods are attached to objects, other times they aren't. And if you need to use a function that isn't vectorised, you've got to do df.apply anyway. You have to remember to change the 'axis' too. Plotting is another thing that I can't get my head around. Am I supposed to use Pandas' helpers like df.plot() all the time? Or ditch it and use the low level matplotlib directly? What is idiomatic? I cannot find answers to much of it, even with ChatGPT. Worse, I can't seem to create a mental model of what Pandas expects me to do in a given situation. Pandas has disabused me of the notion that Python syntax is self-explanatory and executable-pseudocode. I find it terrible to look at. Matlab was infinitely more enjoyable.
refactor_master 10mo ago
I honestly don't get why you'd hate pandas more than anything else in the Python ecosystem. It's probably not the best tool in the world, and sure, like everybody else I'd rewrite the universe in Rust if I could start over, and had infinite time to catch up. But the code base I work on has thousands and THOUSANDS of lines of Pandas churning through big data, and I can't remember the last time it lead to a bug or error in production. We use pandas + static schema wrapper + type checker, so you'll have to get exotic to break things.
- bormaj 10mo ago
  Custom schema wrapper or some package you'd recommend from pypi?
  - refactor_master 10mo ago
    Originally I used Pandera, but it had several issues last * Mypy dependency and really bad PEP compliance * Sub-optimal runtime check decorators * Subclasses pd.DataFrame, so using e.g. .assign(...) makes the type checker think it's still the same type, but now you just violated your own schema So I wrote my own library that solves all these issues, but it's currently company-internal. I've been meaning to push for open-sourcing it, but just haven't had the time.
runningmike 10mo ago
https://learningds.org/intro.html https://learningds.org/intro.html Cc-by-nc-nd

312 points by cl3misch 10mo ago

BenGosub 10mo ago

He's a great writer and I miss his blog. He had an awesome post on pivot table that I think is now a part of this book.

ayhanfuat 10mo ago
He is also the creator of the Altair visualization library (Vega-Lite in Python https://altair-viz.github.io/ https://altair-viz.github.io/). I really like using it.
- AI-NoGuardrails 10mo ago
  [flagged]
- linhns 10mo ago
  Thanks for the fact, I used Altair sometimes and really admire the simplicity, not knowing it was written by Jake.

ellisv 10mo ago

These types of books are always interesting to me because they tackle so many different things. They cover a range of topics at a high level (data manipulation, visualization, machine learning) and each could have its own book. They balance teaching programming while introducing concepts (and sometimes theory). In short I think it's hard to strike an appropriate balance between these but this seems to be a good intro level book.

sschnei8 10mo ago

Interesting choice of Pandas in this day and age. Maybe he’s after imparting general concepts that you could apply to any tabular data manipulator rather than selecting for the latest shiny tool.

msto 10mo ago
It was originally published in 2016, and I think this is still the first edition.
- mkl 10mo ago
  Looks like it. From https://jakevdp.github.io/PythonDataScienceHandbook/00.00-preface.html https://jakevdp.github.io/PythonDataScienceHandbook/00.00-pr...: > Copyright 2016
xenophonf 10mo ago
What's wrong with Pandas?
- clickety_clack 10mo ago
  I probably wouldn’t rewrite an entire data science stack that used pandas, but most people would use polars if starting a new project today.
  - deleted 10mo ago
    [deleted]
  - biofox 10mo ago
    R and Matlab workflows have been fairly stable for the past decade. Why is the Python ecosystem so... unstable? It puts me off investing any time in it.
    - clickety_clack 10mo ago
      The R ecosystem has had a similar evolution with the tidyverse, it was just a little further ago. As for Matlab, I initially learned statistical programming with it a long time ago, but I’m not sure I’ve ever seen it in the wild. I don’t know what’s going on there. I’m actually quite partial to R myself, and I used to use it extensively back when quick analysis was more valuable to my career. Things have probably progressed, but I dropped it in favor of python because python can integrate into production systems whereas R was (and maybe still is) geared towards writing reports. One of the best things to happen recently in data science is the plotnine library, bringing the grammar of graphics to python imho. The fact is that today, if you want career opportunities as a data scientist, you need to be fluent in python.

wiz21c 10mo ago

I wouldn't say it's a handbook because it's more like an introduction. But it's pretty well written.

farhanhubble 10mo ago

I loved his Statistics for Hackers talk: https://speakerdeck.com/pycon2016/jake-vanderplas-statistics-for-hackers https://speakerdeck.com/pycon2016/jake-vanderplas-statistics...

yboris 10mo ago
Amazing Thank you for sharing. Reminds me of how thinking using frequencies rather than computing probabilities is easier and can avoid errors (e.g. a 99% accurate test being positive does not mean 99% likelihood of having disease for a disease with a 1/10,000 prevalence in population).

synergy20 10mo ago

it's written 8 years ago though, there is a 2ed of the book by the same author.

phone_book 10mo ago
The linked Github seems to have the 2nd edition in the form of notebooks, https://github.com/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/00.00-Preface.ipynb https://github.com/jakevdp/PythonDataScienceHandbook/blob/ma..., under the Using Code Examples section, "attribution usually includes the title, author, publisher, and ISBN. For example: "Python Data Science Handbook, 2nd edition, by Jake VanderPlas (O’Reilly). Copyright 2023..." compared to the OP's link which has "The Python Data Science Handbook by Jake VanderPlas (O’Reilly). Copyright 2016..."
mayankkaizen 10mo ago
There is a second edition?

__rito__ 10mo ago

This is one of the few books that I read cover-to-cover when I was starting out learning Data Science in 2020/21. Will recommend.

trio8453 10mo ago

This book was absolute fire for getting started with data science in 2017-2018, Jake is a great teacher.

mayankkaizen 10mo ago
This was the solid book to get into data science and ML scene. Covers everything. Jake is a fantastic teacher. I really wish he comes up with updated second edition.

badmonster 10mo ago

[flagged]

deleted 10mo ago
[deleted]

pantsforbirds 10mo ago

I used the Kernel Density Estimation (KDE) page/blog at my very first job. It was immensely useful and I've loved his work ever since.

pid-1 10mo ago

Pandas is cancer. Please stop teaching it to people. Everything it does can be done reasonable well with list comprehensions and objects that support type annotations and runtime type checking (if needed). Pandas code is untestable, unreadable, hard to refactor and impossible to reuse. Trillions of dollars are wasted every year by people having to rewrite pandas code.

mttpgn 10mo ago
> Pandas code is untestable The thousand-plus data integrity tests I've written in pandas tell a different story...
physicsguy 10mo ago
I found Pandera quite good for wrapping input/output expectations over Pandas. At the end of the day the vectorisation of operations in it and other table based formats mean they’re not easy to replace performantly.
mulmboy 10mo ago
> Everything it does can be done reasonable well with list comprehensions and objects that support type annotations and runtime type checking (if needed). I see this take somewhat often, and usually with similar lack of nuance. How do you come to this? In other cases where I've seen this it's from people who haven't worked in any context where performance or scientific computing ecosystem interoperability matters - missing a massive part of the picture. I've struggled to get through to them before. Genuine question.
isolatedsystem 10mo ago
I've recently had to migrate over to Python from Matlab. Pandas has been doing my head in. The syntax is so unintuitive. In Matlab, everything begins with a `for` loop. Inelegant and slow, yes, but easy to reason about. Easy to see the scope and domain of the problem, to visualise the data wrangling. Pandas insist you never use a for loop. So, I feel guilty if I ever need a throwaway variable on the way to creating a new column. Sometimes methods are attached to objects, other times they aren't. And if you need to use a function that isn't vectorised, you've got to do df.apply anyway. You have to remember to change the 'axis' too. Plotting is another thing that I can't get my head around. Am I supposed to use Pandas' helpers like df.plot() all the time? Or ditch it and use the low level matplotlib directly? What is idiomatic? I cannot find answers to much of it, even with ChatGPT. Worse, I can't seem to create a mental model of what Pandas expects me to do in a given situation. Pandas has disabused me of the notion that Python syntax is self-explanatory and executable-pseudocode. I find it terrible to look at. Matlab was infinitely more enjoyable.

refactor_master 10mo ago

I honestly don't get why you'd hate pandas more than anything else in the Python ecosystem. It's probably not the best tool in the world, and sure, like everybody else I'd rewrite the universe in Rust if I could start over, and had infinite time to catch up. But the code base I work on has thousands and THOUSANDS of lines of Pandas churning through big data, and I can't remember the last time it lead to a bug or error in production. We use pandas + static schema wrapper + type checker, so you'll have to get exotic to break things.

bormaj 10mo ago
Custom schema wrapper or some package you'd recommend from pypi?
- refactor_master 10mo ago
  Originally I used Pandera, but it had several issues last * Mypy dependency and really bad PEP compliance * Sub-optimal runtime check decorators * Subclasses pd.DataFrame, so using e.g. .assign(...) makes the type checker think it's still the same type, but now you just violated your own schema So I wrote my own library that solves all these issues, but it's currently company-internal. I've been meaning to push for open-sourcing it, but just haven't had the time.

runningmike 10mo ago

https://learningds.org/intro.html https://learningds.org/intro.html Cc-by-nc-nd