10 ms·

Polars is missing a crucial feature for replacing pandas in Finance: first class timeseries handling. Pandas allows me to easily do algebra on timeseries. I can

Polars is missing a crucial feature for replacing pandas in Finance: first class timeseries handling. Pandas allows me to easily do algebra on timeseries. I can easily resample data with the resample(...) method, I can reason about the index frequency, I can do algebra between timeseries, etc. You can do the same with Polars, but you have to start messing about with datetimes and convert the simple problem "I want to calculate a monthly sum anchored on the last business day of the month" to SQL-like operations. Pandas grew a large and obtuse API because it provides specialized functions for 99% of the tasks one needs to do on timeseries. If I want to calculate an exponential weighted covariance between two time series, I can trivially do this with pandas: series1.ewm(...).cov(series2). I welcome people to try and do this with Polars. It'll be a horrible and barely readable contraption. YC is mostly populated by technologists, and technologists are often completely ignorant about what makes pandas useful and popular. It was built by quants/scientists, for doing (interactive) research. In this respect it is similar to R, which is not a language well liked by technologists, but it is (surprise) deeply loved by many scientists.

dkga 2y ago
Exactly the single reason why I use pandas when I need to use python. But coming from R, it still feels like “second best”.
n8henrie 2y ago
I don't know what exponential weighted covariance is, but I've had pretty good luck converting time series-based analyses from pandas to polars (for patient presentations to my emergency department -- patients per hour, per day, per shift, etc.). Resample has a direct (and easier IMO) replacement in polars, and there is group_by_dynamic. I've had trouble determining whether one timestamp falls between two others across tens of thousands of rows (with the polars team suggesting I use a massive cross product and filter -- which worked but excludes the memory requirement), whereas in pandas I was able to sort the timestamps and thereby only need to compare against the preceding / following few based on the index of the last match. The other issue I've had with resampling is with polars automatically dropping time periods with zero events, giving me a null instead of zero for the count of events in certain time periods (which then gets dropped from aggregations). This has caught me a few times. But other than that I've had good luck.
- short_sells_poo 2y ago
  I'm curious how is polars group_by_dynamic easier than resample in pandas. In pandas if I want to resample to a monthly frequency anchored to the last business day of the month, I'd write: > my_df.resample("BME").apply(...) Done. I don't think it gets any easier than this. Every time I tried something similar with polars, I got bogged down in calendar treatment hell and large and obscure SQL like contraptions. Edit: original tone was unintentionally combative - apologies.
  - n8henrie 2y ago
    Totally fair. And thank you for the rewording (sincerely). I haven't used polars for anything business or finance related, so this is likely one of many blind spots for me. Reviewing my work, only needed an hourly aggregation, which was similarly easy in polars and pandas (I misspoke about being easier) -- what I found easier was grouping by time data that wasn't amenable to `resample`. In polars I had no problems using a regular group_by with a pl.col.dt object, whereas in pandas I remember struggling to do so, even though it seemed straightforward. Sorry, I wish I could remember more details; this was probably 5 years ago that I was writing the pandas code and just converted it to polars about a year ago, so it's possible that I just got better at python in the meantime (though I was writing much more python back then). And of course a rewrite is likely to feel easier the second time. The other confounding issue is that the eager pandas code crashed with OOM regularly and took several minutes to run, whereas polars handles it very well (which I'm sure to some degree is it optimizing things that I could have done manually), but this made iterating on this codebase feel much less onerous.
- cmdlineluser 2y ago
  > cross product and filter `.join_where()`[1] was also added recently. [1]: https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.join_where.html https://docs.pola.rs/api/python/stable/reference/dataframe/a...
marcogorelli 2y ago
Could you show how you write "calculate a monthly sum anchored on the last business day of the month" in pandas please?
- sebg 2y ago
  Not OP. But I'm guessing it's something like this: import pandas as pd def calculate_monthly_business_sum(df, date_column, value_column): """ Calculate monthly sums anchored to the last business day of each month Parameters: df: DataFrame with dates and values date_column: name of date column value_column: name of value column to sum Returns: DataFrame with sums anchored to last business day """ # Ensure date column is datetime df[date_column] = pd.to_datetime(df[date_column]) # Group by end of business month and sum monthly_sum = df.groupby(pd.Grouper( key=date_column, freq='BME' # Business Month End frequency ))[value_column].sum().reset_index() return monthly_sum # Example usage: df = pd.DataFrame({ 'date': ['2024-01-01', '2024-01-31', '2024-02-29'], 'amount': [100, 200, 300] }) result = calculate_monthly_business_sum(df, 'date', 'amount') print(result) Which you can run here => https://python-fiddle.com/examples/pandas?checkpoint=1732114144 https://python-fiddle.com/examples/pandas?checkpoint=1732114...
  - short_sells_poo 2y ago
    It's actually much simpler than that. Assuming the index of the dataframe DF is composed of timestamps (which is normal for timeseries): df.resample("BME").sum() Done. One line of code and it is quite obvious what it is doing - with perhaps the small exception of BME, but if you want max readability you could do: df.resample(pd.offsets.BusinessMonthEnd()).sum() This is why people use pandas.
- short_sells_poo 2y ago
  Answered the child comment but let me copy paste here too. It's literally one (short) line: > df.resample("BME").sum() Assuming `df` is a dataframe (ie table) indexed by a timestamp index, which is usual for timeseries analysis. "BME" stands for BusinessMonthEnd, which you can type out if you want the code to be easier to read by someone not familiar with pandas.
bobbylarrybobby 2y ago
Is LazyFrame.group_by_dynamic not basically the same thing?

dkga 2y ago

Exactly the single reason why I use pandas when I need to use python. But coming from R, it still feels like “second best”.

n8henrie 2y ago

I don't know what exponential weighted covariance is, but I've had pretty good luck converting time series-based analyses from pandas to polars (for patient presentations to my emergency department -- patients per hour, per day, per shift, etc.). Resample has a direct (and easier IMO) replacement in polars, and there is group_by_dynamic. I've had trouble determining whether one timestamp falls between two others across tens of thousands of rows (with the polars team suggesting I use a massive cross product and filter -- which worked but excludes the memory requirement), whereas in pandas I was able to sort the timestamps and thereby only need to compare against the preceding / following few based on the index of the last match. The other issue I've had with resampling is with polars automatically dropping time periods with zero events, giving me a null instead of zero for the count of events in certain time periods (which then gets dropped from aggregations). This has caught me a few times. But other than that I've had good luck.

short_sells_poo 2y ago
I'm curious how is polars group_by_dynamic easier than resample in pandas. In pandas if I want to resample to a monthly frequency anchored to the last business day of the month, I'd write: > my_df.resample("BME").apply(...) Done. I don't think it gets any easier than this. Every time I tried something similar with polars, I got bogged down in calendar treatment hell and large and obscure SQL like contraptions. Edit: original tone was unintentionally combative - apologies.
- n8henrie 2y ago
  Totally fair. And thank you for the rewording (sincerely). I haven't used polars for anything business or finance related, so this is likely one of many blind spots for me. Reviewing my work, only needed an hourly aggregation, which was similarly easy in polars and pandas (I misspoke about being easier) -- what I found easier was grouping by time data that wasn't amenable to `resample`. In polars I had no problems using a regular group_by with a pl.col.dt object, whereas in pandas I remember struggling to do so, even though it seemed straightforward. Sorry, I wish I could remember more details; this was probably 5 years ago that I was writing the pandas code and just converted it to polars about a year ago, so it's possible that I just got better at python in the meantime (though I was writing much more python back then). And of course a rewrite is likely to feel easier the second time. The other confounding issue is that the eager pandas code crashed with OOM regularly and took several minutes to run, whereas polars handles it very well (which I'm sure to some degree is it optimizing things that I could have done manually), but this made iterating on this codebase feel much less onerous.
cmdlineluser 2y ago
> cross product and filter `.join_where()`[1] was also added recently. [1]: https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.join_where.html https://docs.pola.rs/api/python/stable/reference/dataframe/a...

marcogorelli 2y ago

Could you show how you write "calculate a monthly sum anchored on the last business day of the month" in pandas please?

sebg 2y ago
Not OP. But I'm guessing it's something like this: import pandas as pd def calculate_monthly_business_sum(df, date_column, value_column): """ Calculate monthly sums anchored to the last business day of each month Parameters: df: DataFrame with dates and values date_column: name of date column value_column: name of value column to sum Returns: DataFrame with sums anchored to last business day """ # Ensure date column is datetime df[date_column] = pd.to_datetime(df[date_column]) # Group by end of business month and sum monthly_sum = df.groupby(pd.Grouper( key=date_column, freq='BME' # Business Month End frequency ))[value_column].sum().reset_index() return monthly_sum # Example usage: df = pd.DataFrame({ 'date': ['2024-01-01', '2024-01-31', '2024-02-29'], 'amount': [100, 200, 300] }) result = calculate_monthly_business_sum(df, 'date', 'amount') print(result) Which you can run here => https://python-fiddle.com/examples/pandas?checkpoint=1732114144 https://python-fiddle.com/examples/pandas?checkpoint=1732114...
- short_sells_poo 2y ago
  It's actually much simpler than that. Assuming the index of the dataframe DF is composed of timestamps (which is normal for timeseries): df.resample("BME").sum() Done. One line of code and it is quite obvious what it is doing - with perhaps the small exception of BME, but if you want max readability you could do: df.resample(pd.offsets.BusinessMonthEnd()).sum() This is why people use pandas.
short_sells_poo 2y ago
Answered the child comment but let me copy paste here too. It's literally one (short) line: > df.resample("BME").sum() Assuming `df` is a dataframe (ie table) indexed by a timestamp index, which is usual for timeseries analysis. "BME" stands for BusinessMonthEnd, which you can type out if you want the code to be easier to read by someone not familiar with pandas.

bobbylarrybobby 2y ago

Is LazyFrame.group_by_dynamic not basically the same thing?