3 ms·

I've been trying to learn python for a while now, and I have a grasp of it in a similar way that I have a grasp of golf- I know what to do, but I'm still missin

I've been trying to learn python for a while now, and I have a grasp of it in a similar way that I have a grasp of golf- I know what to do, but I'm still missing a few fundamentals. All these intro to X things seem to gloss over the high level stuff, but I've hit a few walls in certain concepts that I just can't possibly Google. Maybe HN could help me out: take a pandas dataframe. There's a .describe() method that I can chain to get the mean of the columns in the data frame grouped by var1: df.groupby(df.var1).describe().mean But then, there's also: df.describe(df.var1).mean() They largely do the same thing, but one gives me the mean across all variables, and the other one doesn't. Now, how do I possibly Google this question? One method has parentheses, the other doesn't.... but It's clear I'm missing something fundamental here that I thought an intro online resource should address... but none do.

llao 8y ago
Ok, this was more complex than I thought. I hope I got it right and this gives you some pointers how I investigated. :) Use type() to see what you are getting: >>> type(df.describe().mean()) <class 'pandas.core.series.Series'> >>> type(df.describe().mean) <class 'method'> Something without parenthesis cannot be a method call, at least not from a normal viewpoint. Instead you are trying to access the attribute "mean" of the result of describe() if you run df.groupby(df.var1).describe().mean. This attribute might very well be a method itself (a method object). Last in that chain was describe(), so let's look at that: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html https://pandas.pydata.org/pandas-docs/stable/generated/panda... It returns a "summary: Series/DataFrame of summary statistics", so calling .mean() on that, will call https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.mean.html#pandas.DataFrame.mean https://pandas.pydata.org/pandas-docs/stable/generated/panda... Accessing .mean will instead return that method object itself. By default pandas will return a string describing the data itself when printing a method: >>> df.describe() numeric n2 count 3.0 3.0 mean 2.0 3.0 std 1.0 1.0 min 1.0 2.0 25% 1.5 2.5 50% 2.0 3.0 75% 2.5 3.5 max 3.0 4.0 >>> df.describe().mean <bound method DataFrame.mean of numeric n2 count 3.0 3.0 mean 2.0 3.0 std 1.0 1.0 min 1.0 2.0 25% 1.5 2.5 50% 2.0 3.0 75% 2.5 3.5 max 3.0 4.0> >>> df.describe().mean() numeric 2.00 n2 2.75 dtype: float64 I assume this is what threw you off? You are not using the "results" of .mean in further code but were just looking at them, right?
- treyfitty 8y ago
  That’s right, I was using notebooks for a project and luckily, didn’t have to actually extract the mean. Thanks for such an insightful answer
  - llao 8y ago
    Glad to, as I need to learn Pandas in more depth than I currently know. Also, I noticed something very very dangerous you did: .describe() will return a DataFrame with all those statistics. One column per numeric column of the input. One row per statistical value (like mean, std, etc). If you call .mean() on THAT DataFrame, you will get the mean of the statistical values per column, so e.g. the mean of columnA's count, mean, std, etc values. Definitely not what you want! Instead, use either .mean() directly on your "data DataFrame" (not the one returned by .describe()). Or access the specific value from the result via .describe()['columnname']['mean'].
ForHackernews 8y ago
The question you're asking is really about the Pandas library and its API, not so much about Python the language as such. Python objects can have both callable methods (e.g. mean()) and properties/attributes that don't need an explicit call (e.g. mean). For whatever reason, it looks like Pandas' describe() method returns a result with a different interface when called with an argument than without one.
- danso 8y ago
  My first thought exactly. Pandas API is complicated enough that it requires a good grasp of Python in general to be able to tell what is regular Python, and what is the convoluted convention that Pandas uses to achieve its ends. This is not a criticism of pandas -- I think it extremely very well in being an R-like data analysis framework given the limitations/features of the Python syntax. But learning it before being confident with Python is like learning Ruby at the same time as learning Ruby on Rails.
bobwaycott 8y ago
This is very Pandas specific, but I’m going to give you my intuition with a bit of Python knowledge (10yrs): 1. Anything that doesn’t have parens is either an attribute or a property. A property is basically an attribute, except that a computation of some sort is usually executed to set/get its value, whereas an attribute has its value set explicitly. Look at the source to identify if the result of the first code block sets an attribute named `mean` or if it has an `@property` declaration named `mean`. Either way, they’re roughly the same when using them in code. 2. The fact that both `describe()` method examples can have another function and attribute/property chained to their result means `describe()` is returning an object of some type that is not a stdlib data type. 3. The `df.groupby().describe()` is likely returning a different type of object than the second `df.describe()`, as evidenced by the first result having a `mean` attr/property, while the second has a `mean()` function. From here, I’d start by inspecting the object type returned by `df.groupby()`, `df.groupby().describe()`, and the `df.describe()` to discover the differences.

llao 8y ago

Ok, this was more complex than I thought. I hope I got it right and this gives you some pointers how I investigated. :) Use type() to see what you are getting: >>> type(df.describe().mean()) <class 'pandas.core.series.Series'> >>> type(df.describe().mean) <class 'method'> Something without parenthesis cannot be a method call, at least not from a normal viewpoint. Instead you are trying to access the attribute "mean" of the result of describe() if you run df.groupby(df.var1).describe().mean. This attribute might very well be a method itself (a method object). Last in that chain was describe(), so let's look at that: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html https://pandas.pydata.org/pandas-docs/stable/generated/panda... It returns a "summary: Series/DataFrame of summary statistics", so calling .mean() on that, will call https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.mean.html#pandas.DataFrame.mean https://pandas.pydata.org/pandas-docs/stable/generated/panda... Accessing .mean will instead return that method object itself. By default pandas will return a string describing the data itself when printing a method: >>> df.describe() numeric n2 count 3.0 3.0 mean 2.0 3.0 std 1.0 1.0 min 1.0 2.0 25% 1.5 2.5 50% 2.0 3.0 75% 2.5 3.5 max 3.0 4.0 >>> df.describe().mean <bound method DataFrame.mean of numeric n2 count 3.0 3.0 mean 2.0 3.0 std 1.0 1.0 min 1.0 2.0 25% 1.5 2.5 50% 2.0 3.0 75% 2.5 3.5 max 3.0 4.0> >>> df.describe().mean() numeric 2.00 n2 2.75 dtype: float64 I assume this is what threw you off? You are not using the "results" of .mean in further code but were just looking at them, right?

treyfitty 8y ago
That’s right, I was using notebooks for a project and luckily, didn’t have to actually extract the mean. Thanks for such an insightful answer
- llao 8y ago
  Glad to, as I need to learn Pandas in more depth than I currently know. Also, I noticed something very very dangerous you did: .describe() will return a DataFrame with all those statistics. One column per numeric column of the input. One row per statistical value (like mean, std, etc). If you call .mean() on THAT DataFrame, you will get the mean of the statistical values per column, so e.g. the mean of columnA's count, mean, std, etc values. Definitely not what you want! Instead, use either .mean() directly on your "data DataFrame" (not the one returned by .describe()). Or access the specific value from the result via .describe()['columnname']['mean'].

ForHackernews 8y ago

The question you're asking is really about the Pandas library and its API, not so much about Python the language as such. Python objects can have both callable methods (e.g. mean()) and properties/attributes that don't need an explicit call (e.g. mean). For whatever reason, it looks like Pandas' describe() method returns a result with a different interface when called with an argument than without one.

danso 8y ago
My first thought exactly. Pandas API is complicated enough that it requires a good grasp of Python in general to be able to tell what is regular Python, and what is the convoluted convention that Pandas uses to achieve its ends. This is not a criticism of pandas -- I think it extremely very well in being an R-like data analysis framework given the limitations/features of the Python syntax. But learning it before being confident with Python is like learning Ruby at the same time as learning Ruby on Rails.

bobwaycott 8y ago

This is very Pandas specific, but I’m going to give you my intuition with a bit of Python knowledge (10yrs): 1. Anything that doesn’t have parens is either an attribute or a property. A property is basically an attribute, except that a computation of some sort is usually executed to set/get its value, whereas an attribute has its value set explicitly. Look at the source to identify if the result of the first code block sets an attribute named `mean` or if it has an `@property` declaration named `mean`. Either way, they’re roughly the same when using them in code. 2. The fact that both `describe()` method examples can have another function and attribute/property chained to their result means `describe()` is returning an object of some type that is not a stdlib data type. 3. The `df.groupby().describe()` is likely returning a different type of object than the second `df.describe()`, as evidenced by the first result having a `mean` attr/property, while the second has a `mean()` function. From here, I’d start by inspecting the object type returned by `df.groupby()`, `df.groupby().describe()`, and the `df.describe()` to discover the differences.