3 ms·
How does ibis manage ordering of the data frame for SQL based engine? One big gap that I noticed is that pandas retain ordering between operations, bust most S
by zhangyt26 2y ago
How does ibis manage ordering of the data frame for SQL based engine?
One big gap that I noticed is that pandas retain ordering between operations, bust most SQL based engine doesn’t give you that guarantee. Looking at duckdb, it looks they only do that for csv.
- codyvoda 2y agothis is a huge difference between Ibis (and really, any dataframe library that works with larger-than-RAM or distributed engines) and pandas. PySpark, Polars, and Snowpark all to some extent also do not implicitly order your data so in Ibis, you need to explicitly manage order with an `order_by` call at the end. there are a lot of technical reasons for why maintaining order is inefficient as you scale up