3 ms·
[Laughs in pandas] I don't want to be dismissive, this is nice work and it's clean and lightweight. But it might be good to look at existing solutions in this
by anigbrowl 4y ago
[Laughs in pandas]
I don't want to be dismissive, this is nice work and it's clean and lightweight. But it might be good to look at existing solutions in this area - pandas was developed within the financial industry to solve exactly this sort of issue. If you need more topological flexibility there is xarray, and if you need spreadsheet type immediacy it's worth looking into Mito.
Rustaceans should look into pola.rs: https://github.com/pola-rs/polars https://github.com/pola-rs/polars
- chaps 4y ago[Sobs in hours spent in pandas documentation] Or, just use a relational database. Alternatively, airtable and similar are basically relational databases that have some really neat features that let you create relationships by just copying/pasting data, or importing CSVs. They're limited in a lot of ways, but it solves a certain set of problems that can't be solved with code or excel.
- anigbrowl 4y agoI went through that as well; Wes McKinney is super-smart, but not the best teacher, and unfortunately his style of example has become a norm ion Stack Overflow etc. import pandas as pd df = pd.whaaargarrrrbllllll[(['what']['the']['fuck'), is.this['shit'], I, mean, seriously] (outputs) df[.astype('int64').fillna('spork') df.groupby[['uppers']['downers']['all arounders']].join(inner, child, trauma, (yes && no)) df['confused'].very(simple['example']) # the thing being explained I'm exaggerating, but not by much. Most examples in documentation or McKinney's tutorial work is presented as a complete small program in a REPL, and while that does make it easy to follow along by imitation, learning pandas feels like a painfully fragmented process at first. Also, tehre's a widespread assumption among pandas experts that people coming to pandas are already familiar with SQL, even more than Python in fact. I'm sure this reflects the initial user base and to bfair it's probably a true assumption for a lot of folk. But if you came from a more CS or scientific context rather than a database one, it's anotehr avoidable layer of confusion. I can't recommend a book unfortunately - I just worked with McKinney's own materials and suffered for a while until things started to click. Once I realized what I found frustrating about the tutorial materials I began to realize that I could read it more selectively - and also that the code base is in constant flux. There are often 2 or 3 different ways to do the same thing, with different approaches being deprecated or promoted over time.
- deleted 4y ago[deleted]
- chaps 4y agoThe year is 2025. You wake up after what feels like a long nap and you haven't touched pandas since you wrote this comment. In front of you is a pen and piece of paper with instructions to hand-write some pandas code to transform a collection of XLSX files of census data and municipal data into a choropleth representing deltas between the datasets. The pandas documentation is printed in-full on a bookshelf to your left. You have 20 minutes.
- anigbrowl 4y agoimport pandas as pd import geopandas as gpd census_df = pd.read_excel('census_data_yuge.xlsx').fillna(0) muni_df = pd.read_excel('muni_data_yuge.xlsx.').fillna(0) # assumes same criteria & column names in both census_df.drop(['address 1', 'address 2', 'zip'], inplace=True) muni_df.drop(['address 1', 'address 2', 'zip'], inplace=True) cities = gpd.DataFrame(muni_df.groupby(['city']).mean() - census_df.groupby(['city']).dropna().mean()) gpd.plot(column='num_residents', cmap='bwr') A fun exercise. I stuck to the time but score it as a C- because I cheated with geopandas (which I've never used but seems to do a lot of this out of the box) and gave myself unrealistically clean imaginary data. I have parsed census data once and remember it being a lot of work to tidy up before I even tried to answer my question.
- anigbrowl 4y agomfw realizing the last line should be cities.plot... ToT
- karmakaze 4y agoI was wondering the same, why not SQL? > This leads to a lot of logical conditions, since to prevent double-counting each aggregate tuple’s conditions must assert both that the matching tuples matched and that the non-matching tuples didn’t. I understand and know to handle edge cases with SQL, I'd have to learn that all over again with a custom language and its own unique quirks. If I was putting such an investment, I'd want it to be better established. Unless of course it's to scratch an itch and this is a perfect way to start out and share. Nice work and good luck.
- curiousgal 4y agoI work in finance and I hate pandas! Also worth pointing out that almost all big banks have their own functional languages baked into excel that allow not only relational operations and extend excel functions but also tapping into their inhouse analytics libraries written in C++.
- gigatexal 4y agoThe polars syntax is hella nice. It’s also much faster to boot.
- gigatexal 4y agoI see your pandas (even the arrow based pandas 2.0) and raise you polars. ;)
- brahbrah 4y agoI see your: ( polars_df1 .join(polars_df2, on=['state', 'county', 'timestamp'], suffix='_r') .with_column( ( pl.col('val') + pl.col('val_r')).alias('val') ) .select(['state', 'county', 'timestamp', 'val']) ) and raise you: pandas_df1 + pandas_df2
- gigatexal 4y agoHmmm touché
- brahbrah 4y agoFwiw I actually do prefer polars for standard long format relational operations. But sometimes it’s just more convenient to work with data in other ways. Another example: Polars: polars_df.with_column( pl.when(pl.col('timestamp').is_between( datetime('2023-03-01'), datetime('2023-03-31'), include_bounds=True )).then(pl.col('val') * 1.1) .otherwise(pl.col('val')) .alias('val') ) Pandas: pandas_df.loc['2023-03'] *= 1.1
- narush 4y agoMito co-founder here - thanks for the mention! OP you can check out Mito here [1]. Tldr: Mito is a spreadsheet that you can use to edit pandas dataframes, from directly within a Jupyter notebook. For every edit you make, Mito generates the equivalent python code, allowing you to perform some tricky pandas operations with the ease of a spreadsheet. In practice, most of our most active users _are_ in the financial industry. They turn to Mito because they’re transitioning from spreadsheets to a programming language (for one of many reasonsg, but doing so while not knowing how to program is challenging. For most people, the hard part of writing code is the actually process of writing code - so we give these folks a spreadsheet interface they already know. Happy to answer any questions / hear any feedback! P.S. We’re open core + source available. Check it out if that’s your thing [2] [1] https://trymito.io https://trymito.io [2] https://GitHub.com/mito-ds/monorepo https://GitHub.com/mito-ds/monorepo
- anigbrowl 4y agoI tell everyone to start with Mito now. If I had had it around when I learned pandas I think I would have got comfortable in 6 weeks rather than 6 months.
- goosedragons 4y agoIsn't Pandas just a Python clone of R dataframes?
- anigbrowl 4y agoI believe so, yes. I just came to it through Python first, I hate R and try never to think about it.