17 ms·

Pandas Manual [pdf]

56 points by garren 5y ago

curiousgal 5y ago
Yeah I can see why 3500 pages are needed to document this inconsistent piece of software. Can never tell whether the method I have in mind is static or not, or how to access elements, (is it loc, iloc, query, [] directly, etc), will the results be a dataframe or a series or straight up numpy array/lists, index issues, etc etc. If you think that these complaints are not valid or stem from a lack of experience, I urge you to use R first.
- bradrn 5y ago
  Huh, so it wasn’t just me who was confused when I tried to use pandas. I found pretty much anything non-trivial to be difficult at best and impossible at worst. (IIRC I needed to sum rows grouped by time period or something like that.) Then again, I haven’t used pandas much at all. (I’ve heard tell that R is worse, but I wouldn’t have any opinion on that since I’ve never used the language.)
  - dwater 5y ago
    Yeah, I'm not sure I understand the "I urge you to use R first" comment in the grandparent post, because I've used both and R's tidyverse was at least as difficult to approach as pandas for me. And what's the alternative, Excel? So it stands to reason there must be a common explanation whey they are all so obtuse, which I would venture is something like: they solve a set of problems so broad and complex that the tools themselves must also be massively broad and complex, and so the only way to avoid API inconsistencies is to have the system designed whole-cloth from the very beginning, but doing so would make the development process so huge that it would take a decade before you had something ready for users and so it would never get off the ground. The tools that succeed in reality are the ones that solve a common subset of problems, and then grow organically as they are extended to solve other subsets, and the varying subsets have logical inconsistencies and conflicts, thus the state of tools available today having APIs with inconsistencies and conflicts. That said, pandas is really good at solving problems with tabular data, and even though it's not easy to do complex work with I still like and appreciate it very much.
    - nn3 5y ago
      A lot of learning curve come from them essentially being array programming languages (like APL). They are powerful, but just very different if you aren't used to them before. The alternative really is to write simple imperative style scripts with for loops in your favorite scripting language. Script will be much longer and likely more prone to errors, but the learning curve for simple tasks is just a lot less steep. Eventually learning the array oriented approach is worth it though. It's the usual trade off in how much time is it worth learning a new tool vs kludging some old tool that doesn't quite fit for the task.
    - lf-non 5y ago
      > And what's the alternative, Excel? Take this with a grain of salt from someone who needs data manipulation occasionally every now and then (as opposed to being a full time number-cruncher, data-scientist, statistician etc.), using krangl[1] for Kotlin has been a great experience. I was drawn to this library because I use Kotlin in my dayjob for backend development, but I love how well Kotlin's succinct syntax & features like extension functions lends itself to data transformation & ETL kind of use cases. Also it doesn't hurt that JVM has a plethora of libraries available for things like DB access, plotting, etc. I am sure that Pandas has many features I am unaware of, and for a lot of people the high-ish startup time can be a deterrant, but for most of my day to day data munging the combination of jbang, krangl & kravis has been a pretty good fit. [1] https://github.com/holgerbrandl/krangl https://github.com/holgerbrandl/krangl
    - mejutoco 5y ago
      I have used pandas a lot and by now I am used to its quirks, but I still find R to be much more consistent. Just another data point.
    - hermitcrab 5y ago
      >And what's the alternative, Excel? Lots of GUI based alternatives for data munging such as: Easy Data Transform, Alteryx, Knime. Whether they are better than text based tools, such as R or Python+Pandas, depends a lot on your use case and experience.
- mjburgess 5y ago
  Python wasn't designed with vectorised data structures in mind, `R` was -- and that's the origin of a vast number of issues. Data Science happened to python largely against its will (ie., that of GvR) -- and language-level support simply doesn't exist.
  - wodenokoto 5y ago
    Pandas is an incredible piece of software, but it is also a confusing, hard to use and inconsistent api. It can be both, but it doesn’t have to.
    - mjburgess 5y ago
      Data analysis isn't object-oriented nor imperative. Transformations are functions on datasets, not a sequence of stateful actions. Python is an imperative, object-oriented language largely concerned with side-effecting programs (web severs, os automation, etc.). It is about as hostile as it is possible to be, from "ideas" pov, to data science. However those ideas are just a syntax skin over something much more flexible: pointers and dictionaries (/hashmaps). These are like the "Python VM Language" on which "Data Science" sits. Data Science libraries essentially use the "actual" semantics of python and ignore the "official" syntax, as that's the only way of providing an API that makes any sense. (Ie., isnt just reams of for-loops with 5 sequential obj.method() calls). I don't believe there is a "non-confusing" way of ignore python's syntax, *within python*; and that's just what is required.
      - lostmsu 5y ago
        What a weird notion. Why do you think building transformations is not a good use for an (optionally) imperative OOP language?
        aaron-santos 5y ago
        When working with data, I like to avoid stateful data mutations at all cost. OOP's focus is mutating state. These are at odds. The identity semantics I hold for data are not invariant under transformation.
- _Wintermute 5y ago
  If anyone thinks pandas has a lot of inconsistencies and unexpected behaviour, you're in for a wild ride with R.
  - CornCobs 5y ago
    I beg to differ. If you are talking about dplyr, it has a much smaller and consistent API. And I learned python first. It is certainly interesting though, how there are so many people firmly in opposite camps. I really cannot imagine how pandas could be considered well put together; I'm always searching stackoverflow for data transformations I assume should be common and simple. In dplyr, on the other hand, each function does one and only one thing (the touted Unix principle) and you just pass the data along via the pipe %>% operator
    - _Wintermute 5y ago
      dplyr is nice, but the fact there is an entire company writing packages to get around R's inconsistent standard library might be a bit of a hint. Also, as soon as you stray away from dataframes R is completely lost.
- BeetleB 5y ago
  I recently inherited an R codebase at work. It was not fun understanding it, and trying to hunt down docs to explain some of the functions was a much worse experience than it is with Pandas[1]. So I did the sane thing: I figured it all out and rewrote it in Pandas. [1] I have never failed to find an answer for Pandas on SO for even the most convoluted manipulations I'm attempting. The first hit I look at usually has my answer.
- mejutoco 5y ago
  Cannot agree more. Even using R just to read csvs, transform the data, and output another csv, can be an improvement, and it lets you still use python for other things.
- mands 5y ago
  How is Julia in this regard?
hereforphone 5y ago
Pandas is marvelous. As is numpy, scipy, and all of that family. And they run essentially as C libraries, and approximate that level of performance. Someday I hope there will be golang analogues of these
- deleted 5y ago
  [deleted]
- yashap 5y ago
  Once generics ship, there could be progress in this direction. Until then, though, very high level data structure type libraries, along the lines of pandas, are always going to be terrible in Go.
- matsemann 5y ago
  As someone new to python trying to use these, I find them often too magic. Soo much overloading, magic, parameters etc, with virtually no help from a type system. Like this standard usage which misuses all the operators: a = b[c == 1] One would think that would be equivalent to b[true] or false, and what's that supoosed to mean? But if c is a numpy array it's instead doing element wise comparison on c, and the new array of booleans will be used to index into b. If you don't know this, it's impossible to figure out looking at the code. Too many footguns imo. Same with pytorch, broadcasting is nice, but makes it hard to figure out what's going to happen in practice with the shapes being calculated upon.
  - superbatfish 5y ago
    Your complaint applies to array processing libraries in general. It applies to numpy. Yet those syntactic niceties are exactly what make them so convenient. Sure, it takes time to get comfortable writing array-oriented code. But it isn’t a good idea to optimize for complete beginners. Some familiarity with the general domain is a prerequisite to using array (and dataframe) libraries.
    - matsemann 5y ago
queuebert 5y ago
Pandas is a great CSV reader. The rest of it is probably ok too.
- mmiliauskas 5y ago
  Honestly "a great CSV reader" is the only thing it has going from my experience. Other then that, it is not intuitive, a kitchen sink of features, memory greedy.
  - MrPowers 5y ago
    I'm relatively new to Pandas after years of working with Apache Spark. I've found it to be relatively intuitive. From what I understand, Pandas has gotten much less memory greedy over time as PyArrow has been integrated. They just released a new type to make strings much more efficient for example: https://youtu.be/_zoPmQ6J1aE https://youtu.be/_zoPmQ6J1aE
  - kzrdude 5y ago
    But, what can replace it in the python ecosystem? Apart from using xarray whenever feasible, but that's just one small part (and always in conjunction with pandas).
  - ziml77 5y ago
    It's probably good for someone who hasn't really done programming before, but for me I can certainly agree with it being unintuitive. Like putting filters within square brackets. It felt so unnatural to put something that was evaluated across all rows into what should just take an index or key. And there's gotchas like join() not actually working like you'd expect a join to coming from SQL. IIRC a SQL-like join is done via merge(). And then I still don't understand when I need to differentiate between indexes and columns. I don't even know why they need to be differentiated at all. Would it really have been that hard for them to design indexes to always project out to a column so you can treat them as such?
forgingahead 5y ago
For dataframes libraries in Ruby, there are Rover[0] and Daru[1]: [0]:https://github.com/ankane/rover https://github.com/ankane/rover [1]:https://github.com/SciRuby/daru https://github.com/SciRuby/daru
anonu 5y ago
So this thing is 3500 pages. Either pandas is overly complex and over engineered or it's a meticulously planned library that thoughtfully provides tools and use case coverage for most data analysis jobs. Spoiler alert: it's the former.
- gravypod 5y ago
  What's worse is the docs often don't document all values for 'enums' or other types of parameters. Most is copy and paste from someone else. Very painful to debug and upgrade.
- superbatfish 5y ago
  Honestly, I thought you were going to say it’s the latter. It’s great. It has a huge API because it solves a lot of problems. Does some stuff get complicated? Sure. Sometimes there are edge cases that get messy. The other day I was surprised to see my categorical labels implicitly converted to strings if they were used in a multi-level index. But for me, at least, that’s a fairly esoteric use-case. It stands out to me now precisely because pandas so rarely does something I didn’t expect.
rahimnathwani 5y ago
This manual is pretty nice. I usually just google when I don't know (or just forgot) how something works. Over half the time, I end up on the right page in the official docs. The rest of the time, on SO. But I could imagine that, if I were about to do something quite different from things I've done before, that reading the relevant chapter in the book. For example, of course I've dealt with dates and times using Pandas. But I haven't used time spans. Instead of googling when I get stuck, I could read section 2.20.
- avnigo 5y ago
  That's what I do when the language server fails me, but it's always good to have an offline reference to search in.
  - rahimnathwani 5y ago
    Yeah I think the language server doesn't help if you don't know (or can't guess) about the existence of something. e.g. if you don't know about pandas.Series.str then you probably wouldn't find df['colname'].str.upper just by hitting tab. So, yes good to have Google and also an offline manual that has approximately everything.
MrPowers 5y ago
I recently switched from Scala/Spark to the PyData stack and have been loving it. It's been so refreshing to stop having to think about the JVM, SBT, Scala types, and IntelliJ. PyData libraries let you focus on your data. Wes wrote a 10 things I hate about Pandas article back in 2017 that said "pandas rule of thumb: have 5 to 10 times as much RAM as the size of your dataset". Source: https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/ I'm not sure that rule of thumb is up-to-date anymore. Modern versions of Pandas use PyArrow, another project Wes started, and should be more memory efficient. Wes just started a new company that'll continue building Arrow: https://medium.com/@patterson.joshua/joining-forces-for-an-arrow-native-future-cbafe27b6eb7 https://medium.com/@patterson.joshua/joining-forces-for-an-a... Dask makes it easy to scale Pandas, a la Spark. Each partition in a Dask DataFrame is a Pandas DataFrame, so you can indirectly use Pandas on terabytes of data. Disclosure: I work at Coiled, the Dask company founded by Matt Rocklin. The PyData stack makes machine learning / advanced analytics so much easier than Spark in my opinion. I still love Spark for ETL, but think modern PyData is a better stack for ML (for most companies).
- leetrout 5y ago
  Did you evaluate Julia (or already have experience with it)?
  - marcinzm 5y ago
    The issue is that 95% of the ML ecosystem is in Python so if you're doing ML your best options are Python and things that integrated with Python (ie: PySpark).
    - MrPowers 5y ago
      I agree that most of the ML ecosystem libs are in Python, but don't think they're easily accessible via PySpark. PySpark is a thin wrapper on Scala Spark (via py4j). PySpark has better interop with Scala/Java libs. Here's an example PySpark function: def input_file_name(): sc = SparkContext._active_spark_context return Column(sc._jvm.functions.input_file_name()) You can see it's just using py4j to invoke a JVM function.
iheartpandas 5y ago
Pandas is not a Python library, really, it is a DSL for vectorized ETL work. Like any language you have to learn the idioms. Pandas is the swiss army knife of data science prep work. There are very few things it can't do. But to achieve that flexibility, it has a very complex API. That can be confusing when starting out. A lot of beginner work feels like just memorizing arbitrary recipes. Once you understand it, it becomes much more powerful. When I first started, the gratuitous use of nested square brackets drove me crazy. Now I understand what's happening beneath the hood, and I use that expressive power on a daily basis. The critiques of memory usage are well founded and hopefully the underlying tech implementations continue to improve over time.
yongjik 5y ago
pandas is a very powerful library, but it has the highest stackoverflow-search-per-lines-of-code-written ratio I've ever experienced. Regularly exceeds one. It's funny, I have no problem understanding basic syntax of numpy, but when I meet a DataFrame, my brain just can't.
- stevesimmons 5y ago
  The best way to remember Pandas idioms is write your own cheat sheet, updating it each time you look something up.
- amitport 5y ago
  To be fair, numpy's use case, is just a small portion of what pandas tries to solve.
Pinus 5y ago
I'm not sure where I'm going with this, and I may be completely off the mark but: Pandas, like SQL, feels like it doesn't compose very well. If you know some basic programming language — including Basic! — you've got a set of building blocks that let you, at least in principle, implement any algorithm. But in Pandas (and SQL) everything needs a new keyword or method or something. We need a better way of expressing high-level concepts, but if I knew what that was, I'd be a high-flying professor somewhere, not an anonymous commenter on HN!
- mirker 5y ago
  SQL expresses programs via the relational model. Pandas is similar. I’m not an expert on the areas, but it seems that relational modeling is the best tradeoff between performance and abstraction. To your point, people did have a “programming language” for databases and it was called CODASYL. It died in the 1970s in favor to the relational model. NoSQL and other recent trends have similarly been criticized pretty heavily compared to relational databases. I guess the point is that SQL and friends may feel restrictive but alternatives sacrifice on performance or safety.
tpoacher 5y ago
Shame pandas overshadowed numpy structured arrays though. I like those. Nice, simple, work well with the rest of numpy. Pandas is powerful, but I can never get a command to work on the first try.

56 points by garren 5y ago

Yeah I can see why 3500 pages are needed to document this inconsistent piece of software. Can never tell whether the method I have in mind is static or not, or how to access elements, (is it loc, iloc, query, [] directly, etc), will the results be a dataframe or a series or straight up numpy array/lists, index issues, etc etc. If you think that these complaints are not valid or stem from a lack of experience, I urge you to use R first.

bradrn 5y ago
Huh, so it wasn’t just me who was confused when I tried to use pandas. I found pretty much anything non-trivial to be difficult at best and impossible at worst. (IIRC I needed to sum rows grouped by time period or something like that.) Then again, I haven’t used pandas much at all. (I’ve heard tell that R is worse, but I wouldn’t have any opinion on that since I’ve never used the language.)
- dwater 5y ago
  Yeah, I'm not sure I understand the "I urge you to use R first" comment in the grandparent post, because I've used both and R's tidyverse was at least as difficult to approach as pandas for me. And what's the alternative, Excel? So it stands to reason there must be a common explanation whey they are all so obtuse, which I would venture is something like: they solve a set of problems so broad and complex that the tools themselves must also be massively broad and complex, and so the only way to avoid API inconsistencies is to have the system designed whole-cloth from the very beginning, but doing so would make the development process so huge that it would take a decade before you had something ready for users and so it would never get off the ground. The tools that succeed in reality are the ones that solve a common subset of problems, and then grow organically as they are extended to solve other subsets, and the varying subsets have logical inconsistencies and conflicts, thus the state of tools available today having APIs with inconsistencies and conflicts. That said, pandas is really good at solving problems with tabular data, and even though it's not easy to do complex work with I still like and appreciate it very much.
  - nn3 5y ago
    A lot of learning curve come from them essentially being array programming languages (like APL). They are powerful, but just very different if you aren't used to them before. The alternative really is to write simple imperative style scripts with for loops in your favorite scripting language. Script will be much longer and likely more prone to errors, but the learning curve for simple tasks is just a lot less steep. Eventually learning the array oriented approach is worth it though. It's the usual trade off in how much time is it worth learning a new tool vs kludging some old tool that doesn't quite fit for the task.
  - lf-non 5y ago
    > And what's the alternative, Excel? Take this with a grain of salt from someone who needs data manipulation occasionally every now and then (as opposed to being a full time number-cruncher, data-scientist, statistician etc.), using krangl[1] for Kotlin has been a great experience. I was drawn to this library because I use Kotlin in my dayjob for backend development, but I love how well Kotlin's succinct syntax & features like extension functions lends itself to data transformation & ETL kind of use cases. Also it doesn't hurt that JVM has a plethora of libraries available for things like DB access, plotting, etc. I am sure that Pandas has many features I am unaware of, and for a lot of people the high-ish startup time can be a deterrant, but for most of my day to day data munging the combination of jbang, krangl & kravis has been a pretty good fit. [1] https://github.com/holgerbrandl/krangl https://github.com/holgerbrandl/krangl
  - mejutoco 5y ago
    I have used pandas a lot and by now I am used to its quirks, but I still find R to be much more consistent. Just another data point.
  - hermitcrab 5y ago
    >And what's the alternative, Excel? Lots of GUI based alternatives for data munging such as: Easy Data Transform, Alteryx, Knime. Whether they are better than text based tools, such as R or Python+Pandas, depends a lot on your use case and experience.
mjburgess 5y ago
Python wasn't designed with vectorised data structures in mind, `R` was -- and that's the origin of a vast number of issues. Data Science happened to python largely against its will (ie., that of GvR) -- and language-level support simply doesn't exist.
- wodenokoto 5y ago
  Pandas is an incredible piece of software, but it is also a confusing, hard to use and inconsistent api. It can be both, but it doesn’t have to.
  - mjburgess 5y ago
    Data analysis isn't object-oriented nor imperative. Transformations are functions on datasets, not a sequence of stateful actions. Python is an imperative, object-oriented language largely concerned with side-effecting programs (web severs, os automation, etc.). It is about as hostile as it is possible to be, from "ideas" pov, to data science. However those ideas are just a syntax skin over something much more flexible: pointers and dictionaries (/hashmaps). These are like the "Python VM Language" on which "Data Science" sits. Data Science libraries essentially use the "actual" semantics of python and ignore the "official" syntax, as that's the only way of providing an API that makes any sense. (Ie., isnt just reams of for-loops with 5 sequential obj.method() calls). I don't believe there is a "non-confusing" way of ignore python's syntax, *within python*; and that's just what is required.
    - lostmsu 5y ago
      What a weird notion. Why do you think building transformations is not a good use for an (optionally) imperative OOP language?
      - aaron-santos 5y ago
        When working with data, I like to avoid stateful data mutations at all cost. OOP's focus is mutating state. These are at odds. The identity semantics I hold for data are not invariant under transformation.
_Wintermute 5y ago
If anyone thinks pandas has a lot of inconsistencies and unexpected behaviour, you're in for a wild ride with R.
- CornCobs 5y ago
  I beg to differ. If you are talking about dplyr, it has a much smaller and consistent API. And I learned python first. It is certainly interesting though, how there are so many people firmly in opposite camps. I really cannot imagine how pandas could be considered well put together; I'm always searching stackoverflow for data transformations I assume should be common and simple. In dplyr, on the other hand, each function does one and only one thing (the touted Unix principle) and you just pass the data along via the pipe %>% operator
  - _Wintermute 5y ago
    dplyr is nice, but the fact there is an entire company writing packages to get around R's inconsistent standard library might be a bit of a hint. Also, as soon as you stray away from dataframes R is completely lost.
BeetleB 5y ago
I recently inherited an R codebase at work. It was not fun understanding it, and trying to hunt down docs to explain some of the functions was a much worse experience than it is with Pandas[1]. So I did the sane thing: I figured it all out and rewrote it in Pandas. [1] I have never failed to find an answer for Pandas on SO for even the most convoluted manipulations I'm attempting. The first hit I look at usually has my answer.
mejutoco 5y ago
Cannot agree more. Even using R just to read csvs, transform the data, and output another csv, can be an improvement, and it lets you still use python for other things.
mands 5y ago
How is Julia in this regard?

hereforphone 5y ago

Pandas is marvelous. As is numpy, scipy, and all of that family. And they run essentially as C libraries, and approximate that level of performance. Someday I hope there will be golang analogues of these

deleted 5y ago
[deleted]
yashap 5y ago
Once generics ship, there could be progress in this direction. Until then, though, very high level data structure type libraries, along the lines of pandas, are always going to be terrible in Go.
matsemann 5y ago
As someone new to python trying to use these, I find them often too magic. Soo much overloading, magic, parameters etc, with virtually no help from a type system. Like this standard usage which misuses all the operators: a = b[c == 1] One would think that would be equivalent to b[true] or false, and what's that supoosed to mean? But if c is a numpy array it's instead doing element wise comparison on c, and the new array of booleans will be used to index into b. If you don't know this, it's impossible to figure out looking at the code. Too many footguns imo. Same with pytorch, broadcasting is nice, but makes it hard to figure out what's going to happen in practice with the shapes being calculated upon.
- superbatfish 5y ago
  Your complaint applies to array processing libraries in general. It applies to numpy. Yet those syntactic niceties are exactly what make them so convenient. Sure, it takes time to get comfortable writing array-oriented code. But it isn’t a good idea to optimize for complete beginners. Some familiarity with the general domain is a prerequisite to using array (and dataframe) libraries.
  - matsemann 5y ago

queuebert 5y ago

Pandas is a great CSV reader. The rest of it is probably ok too.

mmiliauskas 5y ago
Honestly "a great CSV reader" is the only thing it has going from my experience. Other then that, it is not intuitive, a kitchen sink of features, memory greedy.
- MrPowers 5y ago
  I'm relatively new to Pandas after years of working with Apache Spark. I've found it to be relatively intuitive. From what I understand, Pandas has gotten much less memory greedy over time as PyArrow has been integrated. They just released a new type to make strings much more efficient for example: https://youtu.be/_zoPmQ6J1aE https://youtu.be/_zoPmQ6J1aE
- kzrdude 5y ago
  But, what can replace it in the python ecosystem? Apart from using xarray whenever feasible, but that's just one small part (and always in conjunction with pandas).
- ziml77 5y ago
  It's probably good for someone who hasn't really done programming before, but for me I can certainly agree with it being unintuitive. Like putting filters within square brackets. It felt so unnatural to put something that was evaluated across all rows into what should just take an index or key. And there's gotchas like join() not actually working like you'd expect a join to coming from SQL. IIRC a SQL-like join is done via merge(). And then I still don't understand when I need to differentiate between indexes and columns. I don't even know why they need to be differentiated at all. Would it really have been that hard for them to design indexes to always project out to a column so you can treat them as such?

forgingahead 5y ago

For dataframes libraries in Ruby, there are Rover[0] and Daru[1]: [0]:https://github.com/ankane/rover https://github.com/ankane/rover [1]:https://github.com/SciRuby/daru https://github.com/SciRuby/daru

anonu 5y ago

So this thing is 3500 pages. Either pandas is overly complex and over engineered or it's a meticulously planned library that thoughtfully provides tools and use case coverage for most data analysis jobs. Spoiler alert: it's the former.

gravypod 5y ago
What's worse is the docs often don't document all values for 'enums' or other types of parameters. Most is copy and paste from someone else. Very painful to debug and upgrade.
superbatfish 5y ago
Honestly, I thought you were going to say it’s the latter. It’s great. It has a huge API because it solves a lot of problems. Does some stuff get complicated? Sure. Sometimes there are edge cases that get messy. The other day I was surprised to see my categorical labels implicitly converted to strings if they were used in a multi-level index. But for me, at least, that’s a fairly esoteric use-case. It stands out to me now precisely because pandas so rarely does something I didn’t expect.

rahimnathwani 5y ago

This manual is pretty nice. I usually just google when I don't know (or just forgot) how something works. Over half the time, I end up on the right page in the official docs. The rest of the time, on SO. But I could imagine that, if I were about to do something quite different from things I've done before, that reading the relevant chapter in the book. For example, of course I've dealt with dates and times using Pandas. But I haven't used time spans. Instead of googling when I get stuck, I could read section 2.20.

avnigo 5y ago
That's what I do when the language server fails me, but it's always good to have an offline reference to search in.
- rahimnathwani 5y ago
  Yeah I think the language server doesn't help if you don't know (or can't guess) about the existence of something. e.g. if you don't know about pandas.Series.str then you probably wouldn't find df['colname'].str.upper just by hitting tab. So, yes good to have Google and also an offline manual that has approximately everything.

MrPowers 5y ago

I recently switched from Scala/Spark to the PyData stack and have been loving it. It's been so refreshing to stop having to think about the JVM, SBT, Scala types, and IntelliJ. PyData libraries let you focus on your data. Wes wrote a 10 things I hate about Pandas article back in 2017 that said "pandas rule of thumb: have 5 to 10 times as much RAM as the size of your dataset". Source: https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/ I'm not sure that rule of thumb is up-to-date anymore. Modern versions of Pandas use PyArrow, another project Wes started, and should be more memory efficient. Wes just started a new company that'll continue building Arrow: https://medium.com/@patterson.joshua/joining-forces-for-an-arrow-native-future-cbafe27b6eb7 https://medium.com/@patterson.joshua/joining-forces-for-an-a... Dask makes it easy to scale Pandas, a la Spark. Each partition in a Dask DataFrame is a Pandas DataFrame, so you can indirectly use Pandas on terabytes of data. Disclosure: I work at Coiled, the Dask company founded by Matt Rocklin. The PyData stack makes machine learning / advanced analytics so much easier than Spark in my opinion. I still love Spark for ETL, but think modern PyData is a better stack for ML (for most companies).

leetrout 5y ago
Did you evaluate Julia (or already have experience with it)?
- marcinzm 5y ago
  The issue is that 95% of the ML ecosystem is in Python so if you're doing ML your best options are Python and things that integrated with Python (ie: PySpark).
  - MrPowers 5y ago
    I agree that most of the ML ecosystem libs are in Python, but don't think they're easily accessible via PySpark. PySpark is a thin wrapper on Scala Spark (via py4j). PySpark has better interop with Scala/Java libs. Here's an example PySpark function: def input_file_name(): sc = SparkContext._active_spark_context return Column(sc._jvm.functions.input_file_name()) You can see it's just using py4j to invoke a JVM function.

iheartpandas 5y ago

Pandas is not a Python library, really, it is a DSL for vectorized ETL work. Like any language you have to learn the idioms. Pandas is the swiss army knife of data science prep work. There are very few things it can't do. But to achieve that flexibility, it has a very complex API. That can be confusing when starting out. A lot of beginner work feels like just memorizing arbitrary recipes. Once you understand it, it becomes much more powerful. When I first started, the gratuitous use of nested square brackets drove me crazy. Now I understand what's happening beneath the hood, and I use that expressive power on a daily basis. The critiques of memory usage are well founded and hopefully the underlying tech implementations continue to improve over time.

yongjik 5y ago

pandas is a very powerful library, but it has the highest stackoverflow-search-per-lines-of-code-written ratio I've ever experienced. Regularly exceeds one. It's funny, I have no problem understanding basic syntax of numpy, but when I meet a DataFrame, my brain just can't.

stevesimmons 5y ago
The best way to remember Pandas idioms is write your own cheat sheet, updating it each time you look something up.
amitport 5y ago
To be fair, numpy's use case, is just a small portion of what pandas tries to solve.

Pinus 5y ago

I'm not sure where I'm going with this, and I may be completely off the mark but: Pandas, like SQL, feels like it doesn't compose very well. If you know some basic programming language — including Basic! — you've got a set of building blocks that let you, at least in principle, implement any algorithm. But in Pandas (and SQL) everything needs a new keyword or method or something. We need a better way of expressing high-level concepts, but if I knew what that was, I'd be a high-flying professor somewhere, not an anonymous commenter on HN!

mirker 5y ago
SQL expresses programs via the relational model. Pandas is similar. I’m not an expert on the areas, but it seems that relational modeling is the best tradeoff between performance and abstraction. To your point, people did have a “programming language” for databases and it was called CODASYL. It died in the 1970s in favor to the relational model. NoSQL and other recent trends have similarly been criticized pretty heavily compared to relational databases. I guess the point is that SQL and friends may feel restrictive but alternatives sacrifice on performance or safety.

tpoacher 5y ago

Shame pandas overshadowed numpy structured arrays though. I like those. Nice, simple, work well with the rest of numpy. Pandas is powerful, but I can never get a command to work on the first try.