4 ms·

[Sobs in hours spent in pandas documentation] Or, just use a relational database. Alternatively, airtable and similar are basically relational databases that

by chaps 4y ago

[Sobs in hours spent in pandas documentation] Or, just use a relational database. Alternatively, airtable and similar are basically relational databases that have some really neat features that let you create relationships by just copying/pasting data, or importing CSVs. They're limited in a lot of ways, but it solves a certain set of problems that can't be solved with code or excel.

anigbrowl 4y ago
I went through that as well; Wes McKinney is super-smart, but not the best teacher, and unfortunately his style of example has become a norm ion Stack Overflow etc. import pandas as pd df = pd.whaaargarrrrbllllll[(['what']['the']['fuck'), is.this['shit'], I, mean, seriously] (outputs) df[.astype('int64').fillna('spork') df.groupby[['uppers']['downers']['all arounders']].join(inner, child, trauma, (yes && no)) df['confused'].very(simple['example']) # the thing being explained I'm exaggerating, but not by much. Most examples in documentation or McKinney's tutorial work is presented as a complete small program in a REPL, and while that does make it easy to follow along by imitation, learning pandas feels like a painfully fragmented process at first. Also, tehre's a widespread assumption among pandas experts that people coming to pandas are already familiar with SQL, even more than Python in fact. I'm sure this reflects the initial user base and to bfair it's probably a true assumption for a lot of folk. But if you came from a more CS or scientific context rather than a database one, it's anotehr avoidable layer of confusion. I can't recommend a book unfortunately - I just worked with McKinney's own materials and suffered for a while until things started to click. Once I realized what I found frustrating about the tutorial materials I began to realize that I could read it more selectively - and also that the code base is in constant flux. There are often 2 or 3 different ways to do the same thing, with different approaches being deprecated or promoted over time.
- deleted 4y ago
  [deleted]
- chaps 4y ago
  The year is 2025. You wake up after what feels like a long nap and you haven't touched pandas since you wrote this comment. In front of you is a pen and piece of paper with instructions to hand-write some pandas code to transform a collection of XLSX files of census data and municipal data into a choropleth representing deltas between the datasets. The pandas documentation is printed in-full on a bookshelf to your left. You have 20 minutes.
  - anigbrowl 4y ago
    import pandas as pd import geopandas as gpd census_df = pd.read_excel('census_data_yuge.xlsx').fillna(0) muni_df = pd.read_excel('muni_data_yuge.xlsx.').fillna(0) # assumes same criteria & column names in both census_df.drop(['address 1', 'address 2', 'zip'], inplace=True) muni_df.drop(['address 1', 'address 2', 'zip'], inplace=True) cities = gpd.DataFrame(muni_df.groupby(['city']).mean() - census_df.groupby(['city']).dropna().mean()) gpd.plot(column='num_residents', cmap='bwr') A fun exercise. I stuck to the time but score it as a C- because I cheated with geopandas (which I've never used but seems to do a lot of this out of the box) and gave myself unrealistically clean imaginary data. I have parsed census data once and remember it being a lot of work to tidy up before I even tried to answer my question.
    - anigbrowl 4y ago
      mfw realizing the last line should be cities.plot... ToT
karmakaze 4y ago
I was wondering the same, why not SQL? > This leads to a lot of logical conditions, since to prevent double-counting each aggregate tuple’s conditions must assert both that the matching tuples matched and that the non-matching tuples didn’t. I understand and know to handle edge cases with SQL, I'd have to learn that all over again with a custom language and its own unique quirks. If I was putting such an investment, I'd want it to be better established. Unless of course it's to scratch an itch and this is a perfect way to start out and share. Nice work and good luck.

by chaps 4y ago

anigbrowl 4y ago

I went through that as well; Wes McKinney is super-smart, but not the best teacher, and unfortunately his style of example has become a norm ion Stack Overflow etc. import pandas as pd df = pd.whaaargarrrrbllllll[(['what']['the']['fuck'), is.this['shit'], I, mean, seriously] (outputs) df[.astype('int64').fillna('spork') df.groupby[['uppers']['downers']['all arounders']].join(inner, child, trauma, (yes && no)) df['confused'].very(simple['example']) # the thing being explained I'm exaggerating, but not by much. Most examples in documentation or McKinney's tutorial work is presented as a complete small program in a REPL, and while that does make it easy to follow along by imitation, learning pandas feels like a painfully fragmented process at first. Also, tehre's a widespread assumption among pandas experts that people coming to pandas are already familiar with SQL, even more than Python in fact. I'm sure this reflects the initial user base and to bfair it's probably a true assumption for a lot of folk. But if you came from a more CS or scientific context rather than a database one, it's anotehr avoidable layer of confusion. I can't recommend a book unfortunately - I just worked with McKinney's own materials and suffered for a while until things started to click. Once I realized what I found frustrating about the tutorial materials I began to realize that I could read it more selectively - and also that the code base is in constant flux. There are often 2 or 3 different ways to do the same thing, with different approaches being deprecated or promoted over time.

deleted 4y ago
[deleted]
chaps 4y ago
The year is 2025. You wake up after what feels like a long nap and you haven't touched pandas since you wrote this comment. In front of you is a pen and piece of paper with instructions to hand-write some pandas code to transform a collection of XLSX files of census data and municipal data into a choropleth representing deltas between the datasets. The pandas documentation is printed in-full on a bookshelf to your left. You have 20 minutes.
- anigbrowl 4y ago
  import pandas as pd import geopandas as gpd census_df = pd.read_excel('census_data_yuge.xlsx').fillna(0) muni_df = pd.read_excel('muni_data_yuge.xlsx.').fillna(0) # assumes same criteria & column names in both census_df.drop(['address 1', 'address 2', 'zip'], inplace=True) muni_df.drop(['address 1', 'address 2', 'zip'], inplace=True) cities = gpd.DataFrame(muni_df.groupby(['city']).mean() - census_df.groupby(['city']).dropna().mean()) gpd.plot(column='num_residents', cmap='bwr') A fun exercise. I stuck to the time but score it as a C- because I cheated with geopandas (which I've never used but seems to do a lot of this out of the box) and gave myself unrealistically clean imaginary data. I have parsed census data once and remember it being a lot of work to tidy up before I even tried to answer my question.
  - anigbrowl 4y ago
    mfw realizing the last line should be cities.plot... ToT

karmakaze 4y ago

I was wondering the same, why not SQL? > This leads to a lot of logical conditions, since to prevent double-counting each aggregate tuple’s conditions must assert both that the matching tuples matched and that the non-matching tuples didn’t. I understand and know to handle edge cases with SQL, I'd have to learn that all over again with a custom language and its own unique quirks. If I was putting such an investment, I'd want it to be better established. Unless of course it's to scratch an itch and this is a perfect way to start out and share. Nice work and good luck.