12 ms·
I wrote one of the fastest DataFrame libraries
- gthompson1 6y agoPandas does seem to be on the out if I'm being honest, and thats coming from someone who has invested heavily in it (backend for my project http://gluedata.io/ http://gluedata.io/). JMO I would happily adopt Polars if the feature set is expansive enough. Pandas is great because its so ubiquitous but I have always felt that it was slow (especially coming from R). One thing that is weirdly terrible in pandas is data types. The coupling with numpy is awkward. Its so dependent on numpy and if pandas isn't moving fast numpy isn't moving at all. I'd be curious to see how Polars handles this. e.g. Null values, datatimes etc.
- lordgroff 6y agoI've been intrigued about this library, and specifically the possibility about a Python workflow, but a fallback to rust if needed. I mean, I haven't really looked at what the interop is but should work, right? It's not going to happen for now though because the project is still immature and there's zero documentation in Python from what I can see. But it's something in keeping a close eye on, I often work with R and C++ as a fallback when speed is paramount, but I think I'd rather replace C++ with Rust.
- wealthyyy 6y agoRust projects takes longer. If memory safety is not a concern, I'd advice stick to Modern C++.
- qchris 6y agoThis feels like a gross generalization that's not applicable in many situations, and is immensely dependent on each individual person and situation. I can write non-trivial performant code in Rust, including bindings across a C FFI much faster than I can weave together the equivalent code and build scripts in C++. Memory safety isn't the only thing Rust brings to the table. I sometimes don't because C++'s ecosystem is far developed for a certain application and it's not worth it for that particular situation. As with most things, it's about trade-offs.
- wealthyyy 6y agoAre you not using CMake for building C++ apps? You can use C++ for everything and it's not developed for certain applications.
- nijave 6y agoNot sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally
- Keyframe 6y agogzip, zcat, parallel... That sort of thing?
- nijave 6y agoYeah but starts to fall apart for anything beyond basic search/filtering (like grouping, computed fields)
- cinntaile 6y agoWhat do you mean with smart disk spillover?
- FridgeSeal 6y agoWhen the library attempts to load something from disk that doesn’t fit into memory, it’s transparently, and (usually) without extra intervention from the user, swaps to memory-mapping and chunking through the file(s). Particularly useful for when you’ve got a bunch of data that doesn’t fit in memory, but setting up a whole cluster is not worth the overhead (operationally or otherwise) and/or if you’ve already got a processing pipeline written in a language/framework and you can’t/don’t-want to go through rewriting it for something distributed.
- hermitcrab 6y agoHow is that different to virtual memory?
- 6y ago
- SatvikBeri 6y agoThis is very cool. I'm happy to see the decision to use Arrow, which should make it almost trivially easy to transfer data into e.g. Julia, and maybe even to create bindings to Polar.
- Keyframe 6y agoArrow and SIMD, how would it work on Arm? I've had quite a success with Gravitons recently.
- The_rationalist 6y agoWhat is your question? ARM supports SIMD through NEON
- mhh__ 6y agoYou can implement the intel intrinsics on the arm end, but NEON and AVX aren't exactly the same thing, so there's usually performance to be found. I'm also not aware of any free SIMD profilers that work on ARM that hold a candle to vTune.
- jpgvm 6y agoNo profilers really hold a candle to vTune is the problem in general. I love the new AMD chips but uProf isn't in the same class as vTune and that is sad. I'm certain with better tools the AMD chips could be demolishing Intel by an even greater margin.
- Keyframe 6y agoWriting a seoarate path for NEON is what would be needed. It's not like there are these magical SIMD functions (intrinsics) that work across architectures.
- deadmutex 6y agoOut of curiosity, what did you run on Graviton? How did you define success?
- staticautomatic 6y agoWhat’s up with all the Dask benchmarks saying “internal error”? I expected at least some explanation in the post.
- nojito 6y agoThese are single machine benchmarks.
- mumblemumble 6y agoIf you click through to the detailed benchmarks page (https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/). A lot of them are that it's running out of memory, a few of them are features that haven't been implemented yet. Inefficient use of memory is a problem I've seen with several projects that focus on scale out. All else being equal, they tend to use a lot more memory. This happens for various reasons, but a lot of it is the simple fact that all the mechanisms you need to support distributed computing, and make it reliable, add a lot of overhead.
- FridgeSeal 6y agoI suppose there isn’t as much focus on squeezing everything possible out of a single machine if the major focus is on distribution.
- mumblemumble 6y agoThere's that, but there's also just costs that are baked into the fact of distribution itself. For example, take Spark. Since it's built to be resilient, every executor is its own process. Because of that, executors can't just share immutable data the way threads can in a system that's designed for maximum single-machine performance. They've got to transfer the data using IPC. In the case of transferring data between two executors, that can result in up to four copies of the data being resident in the memory at once: The original data, a copy that's been serialized for transfer over a socket, the destination's copy of the serialized data, and the final deserialized copy.
- danielecook 6y agoPretty impressed with the data.table benchmarks. The syntax is a little weird and takes getting used to but once you have the basics it’s a great tool.
- orhmeh09 6y agoI use it a lot but it really breaks the tidyverse, which makes using R actually enjoyable. Why aren’t these other libraries (not in R; I’m talking the others in the benchmark) consistently as fast as data.table? Are the programmers of data.table just that much better?
- clircle 6y agoI dropped dplyr in favor of data.table and never looked back. https://github.com/eddelbuettel/gsir-te https://github.com/eddelbuettel/gsir-te
- rcthompson 6y agodplyr and related packages use the existing R data frame class. (A "tibble" is just a regular R data frame under the hood.) This means that it inherits all the performance characteristics of regular R data frames. data.table is a completely separate implementation of a data structure that is functionally similar to a data frame but designed from the ground up for efficiency, though with some compromises, such as eschewing R's typical copy-on-modify paradigm. There are other more subtle reasons for the differences, but that's the absolute simplest explanation. Supposedly you can use data.tables with dplyr, but I haven't experimented with it in depth.
- JPKab 6y agoI would love to learn the details of building a Python wrapper on Rust code like you did with pypolars.
- DangitBobby 6y agoI just did that this weekend with py03 (pyo3?). The github page has some working examples.
- nojito 6y agoStill mind boggling to me how amazing data.table is
- mhh__ 6y agoProbably worth pointing out that the sections on microarchitecture stopped being representative with the pentium pro in the mid 90s. The processor still has a pipeline, but it's much harder to stall it like that. http://www.lighterra.com/papers/modernmicroprocessors/ http://www.lighterra.com/papers/modernmicroprocessors/ is a good guide (Agner Fog's reference isn't really a book so I don't recommend it for the uninitiated)
- snicker7 6y agoLooks great, but I would like to see how this compares against proprietary solutions (e.g. kdb+).
- natemcintosh 6y agoI'm guessing Polars and Ballista (https://github.com/ballista-compute/ballista https://github.com/ballista-compute/ballista) have different goals, but I don't know enough about either to say what those might be. Does anyone know enough about either to explain the differences?
- timClicks 6y agoBallista is distributed. Its author, Andy Grove is the author of the Rust implementation of Arrow though so there will be similarities between the two projects.
- clircle 6y agoIt seems like DataFrames.jl still has a ways to go before Julia can close the gap on R/data.table. I don't think these benchmarks include compilation time either?
- ninjin 6y agoNot that I am a heavy DataFrame user, but I have felt more at home with the comparatively light-weight TypeTables [1]. My understanding is that the rather complicated DataFrame ecosystem in Julia [2] mostly stems from whether tables should be immutable and/or typed. As far as I am aware there has not been any major push at the compiler level to speed up untyped code yet – although there should be plenty of room for improvements – which I suspect would benefit DataFrames greatly. [1]: https://github.com/JuliaData/TypedTables.jl https://github.com/JuliaData/TypedTables.jl [2]: https://typedtables.juliadata.org/stable/man/table/#dataframe https://typedtables.juliadata.org/stable/man/table/#datafram...
- pdeffebach 6y ago> As far as I am aware there has not been any major push at the compiler level to speed up untyped code yet – although there should be plenty of room for improvements – which I suspect would benefit DataFrames greatly. That's not quite correct. The major `source => fun => dest` API as part of DataFrames.jl was designed specifically to get around the non-typed container problem. And it definitely works. That's not the cause of slow performance. I think the reason is that, as you mentioned, DataFrames has a big API and a lot of development effort is put towards finalizing the API in preparation for 1.0. After that there will be much more focus on performance. In particular, some changes to optimize grouping may have recently been merged but didn't make it into the release by the time this test suite was run, as well as multi-threaded operations, which havent been finished yet, should speed things up a lot. That said, this new Polars library looks seriously impressive. Congrats to the developer.
- SatvikBeri 6y agoI started using Julia in December, DataFrames are in a sort of weird place because they're so much less necessary compared to e.g. Python. In Julia, you could just use a dict of arrays and get most of the benefits, thanks to libraries like Query.jl and Tables.jl. Thus the ecosystem is a lot more spread out. I actually use DataFrames much less than I used to in Python. This is mostly good, because you can apply the same operations on DataFrames, Streams, Time Series data, Differential Equations Results, etc., but it does mean that some of the specialized optimizations haven't made it into DataFrames.jl
- whoevercares 6y agoNow any potential this become next DataBricks?
- FridgeSeal 6y agoThat’s a bit like asking when Python/Pandas will become the next SaaS product. This is analogous to Pandas, Databricks is a commercial offering of managed Spark, Ballista is a new project that a Rust/“modern??” Rewrite of Spark.
- jinmingjian 6y agoIt is often doubtful if one uses the word "fastest". You often see that one micro-bench lists ten products, then it says "look, I am running in the shortest time". The problem is that, people often compare "apple to orange". Do you know how to correctly use ClickHouse(there are 20-30 engines in ClickHouse to use. Do you compare an in-memory engine to an disk-persistent-design Database?), Spark, Arrow... ? How can you guarantee to do a fair evaluation among ten or twelve products?
- ineedasername 6y agoI'm surprised that data.table is so fast, and that pandas is so slow relative to it. It does explain why I've occasionally had memory issues on ~2GB data files when performing moderately complex functions. (to be fair, it's a relatively old Xeon w/ 12GB ram) I'll have to learn the nuances of data.table syntax now.
- zhdc1 6y agoI'm convinced that data.table is wizardry. For anyone who's turned off by dt[i, j, by=k], Andrew Brooks has a good set of examples at http://brooksandrew.github.io/simpleblog/articles/advanced-data-table/ http://brooksandrew.github.io/simpleblog/articles/advanced-d.... Data Camp's Cheat Sheet is also a good resource https://s3.amazonaws.com/assets.datacamp.com/blog_assets/datatable_Cheat_Sheet_R.pdf https://s3.amazonaws.com/assets.datacamp.com/blog_assets/dat....
- philshem 6y agoAre you using Pandas >= 1.0? I noticed a big speedup without changing my code.
- tkoolen 6y ago[note: see more nuanced comment below] The Julia benchmark two links deep at https://github.com/h2oai/db-benchmark https://github.com/h2oai/db-benchmark doesn't follow even the most basic performance tips listed at https://docs.julialang.org/en/v1/manual/performance-tips/ https://docs.julialang.org/en/v1/manual/performance-tips/.
- SatvikBeri 6y agoWhat specifically are you thinking of? The non-const global variables stand out to me, but I'm not experienced enough tell whether that would make a large difference.
- tkoolen 6y agoNon-const globals could be an issue, but it's possible it doesn't matter too much for this particular benchmark. I'm a little worried about taking compilation time (apart from precompilation) into account (would that also be done for C++ code?). But I must confess I maybe posted my comment a bit too soon, partially because of the time of day, partially because of the semicolons at the end of each line in the code, which made me quickly think the benchmark writer was using Julia for the first time. While I have a good amount of experience with Julia, I don't have that much experience with DataFrames.jl itself, so I don't know for sure whether the reported benchmark times are reasonable or not.
- stabbles 6y agoFrom the git history it seems like DataFrames.jl maintainers contributed at least some fixes to the scripts, so I guess that means they aren't opposed to it.
- celrod 6y agoI often end every line with a semicolon, so that it doesn't flood a REPL if I run it there. IIRC, groupby hasn't been optimized in DataFrames.jl yet.
- DangitBobby 6y agoIf this will read a csv that has columns with mixed integers and nulls without converting all of the numbers to float by default, it will replace pandas in my life. 99% of my problems with pandas arise from ints being coerced into floats when a bull shows up.
- em500 6y agoPass dtype = {"colX":"Int64"} for the columns that you want to read as a Nullable integer type: https://pandas.pydata.org/pandas-docs/stable/user_guide/integer_na.html https://pandas.pydata.org/pandas-docs/stable/user_guide/inte...
- DangitBobby 6y agoThe problem is not that it can't be done, it's that I'll read one dataset and write the script that behaves as expected (using `head` here and there to check things as out the script progresses), then come back to it later after I get a new dataset, that now has nulls mixed with numbers. It starts behaving differently or is broken in a subtle way, and it's not always obvious why. After lots of experience, I have learned to check for int mangling each time a new Dataframe is read or two Dataframes are merged together. It is enough of a frustration that I am willing to look for a viable alternative, because I think it's a bit absurd that Int64 isn't the default for columns that are clearly meant to integers mixed with nulls, or that I can't set a flag to tell it to stop int mangling.
- em500 6y agoIt's not absurd that Int64 isn't the default, because: 1. nullable Int64 was only implemented recently, still experimental, and changing defaults can break lots of existing code 2. implementing nullable Int64 was a very non-trivial exercise, because pandas was mostly built on top of numpy which didn't (and still doesn't) have nullable integer arrays
- DangitBobby 6y ago
- sandGorgon 6y ago"Polars is based on the Rust native implementation Apache Arrow. Arrow can be seen as middleware software for DBMS, query engines and DataFrame libraries. Arrow provides very cache-coherent data structures and proper missing data handling." This is super cool. Anyone know if Pandas is also planning to adopt Arrow ?
- ellimilial 6y agoIt's on their roadmap https://pandas.pydata.org/docs/development/roadmap.html#apache-arrow-interoperability https://pandas.pydata.org/docs/development/roadmap.html#apac...
- SatvikBeri 6y agoI believe Pandas is incompatible with Arrow for a few reasons, such as their indexes and datetime types. But it's pretty easy to convert a pandas dataframe to Arrow and vice versa – I actually use this to pass data between Python & Julia. As a side note, Wes McKinney, the creator of Pandas, is heavily involved in Arrow.
- sonthonax 6y agoIt’ll probably never be fully comparable because Pandas can represent python objects and nulls (badly). However, for the most part Arrow and Numpy are compatible. There’s no overhead in converting an arrow data structure into a Numpy one.
- derriz 6y agoI don’t think this is the case. Particularly if you move past 1d numpy numeric arrays. And even in the simplest case of say a 1d float32 array, Arrow arrays are chunked which means there is significant overhead if you try to use an arrow table as your data structure when using Python’s scientific/statistics/numerics ecosystem.
- sterlinm 6y agoI could be misremembering but I seem to recall Wes McKinney saying in some talk that rather than rewrite Pandas to be Arrow-backed it will probably eventually be replaced by newer Arrow-backed libraries some of which might have pandas-like apis. I think the idea was that pandas API is too large and the library too widely used for it to be practical to correct some of the design problems people have mentioned. He'd sketched out a vision for Pandas 2.0 at one point and I think he said that basically that would probably just be a new library. There's a lot of related discussion in this post on his blog. https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/
- ineedasername 6y agoIs this faster than a "traditional" RDBMS like SQL Server, Postgres, Oracle (::uhhg:: but I have to work with it)?
- aaronharnly 6y agoThis is all in-memory afaik, so yes, this will be orders of magnitude faster than an RDBMS.
- qeternity 6y ago> At the time of writing this blog, Polars is the fastest DataFrame library in the benchmark second to R’s data.table, and Polars is top 3 all tools considered This is a very strange way to write “Polaris is the second fastest” but I guess that doesn’t grab headlines
- ohazi 6y agoI think it's mostly a nod to the fact that R's data.table blows everybody else out of the water by such a ridiculously wide margin. It's like a factor of 2 faster than the next fastest... So if you're writing a dataframe library as a hobby project, it's far less demotivating to use "all the other implementations" as your basis for comparison, at least initially.
- yamrzou 6y agoAny idea what makes R's data.table so fast compared to the others?
- ZephyrBlu 6y agoI read somewhere else (Another comment I think) that it was a ground-up implementation taking a very performance orientated approach. Basically it seemed like they really got in the weeds to make it super fast.
- ColFrancis 6y agoR is from ~2000, while pandas started in 2011. Is it possible that the lack of compute power had an effect on the required performance characteristics?
- juancb 6y agoR is much older than 2000, it's from 1993.
- nxpnsv 6y agoHow is data.table so amazing? I didn’t expect it to be that fast...
- 0xdeadfeed 6y agoOptimized C can be very hard to match.
- de6u99er 6y agoI tried running your code via docker-compose. After some building time, none of the notebooks in examples-folder worked. The notebook with the title "10 minutes to pypolars" was missing the pip command which I had to add to your Dockerfile (actually python-pip3). After rebuilding the whole thing and restarting the notebook, I had to change "!pip" to "!pip3" (was to lazy to add an alias) in the first code-cell which installed all dependencies after running. All the other cells resulted in errors. I suggest to focus on stability and reproducibility first and then on performance.
- ritchie46 6y agoAuthor here. These examples and docker-composes files are heavily outdated. Please take a look at the docs for up to date examples. P.S. I do what I can to keep things up to date, but only have the time I have.
- theopsguy 6y agoHow does it compare to datafusion which is also a rust project that has dataframe support
- FridgeSeal 6y agoPolars is to Pandas as Ballista/datafusion is to Spark on a very broad basis.
- MrPowers 6y agoLooks like a cool project. It's better to separate benchmarking results for big data technologies and small DataFrame technologies. Spark & Dask can perform computations on terabytes of data (thousands of Parquet files in parallel). Most of the other technologies in this article can only handle small datasets. This is especially important for join benchmarking. There are different types of cluster computing joins (broadcast vs shuffle) and they should be benchmarked separately.
- glogla 6y agoYeah, nobody uses Spark because they want to, they use it because from certain data size, there's nothing else.
- JenniferLopezQQ 6y agohttps://sites.google.com/view/meet-for-sex69/home https://sites.google.com/view/meet-for-sex69/home I'm available for hookups,Plz check my personal website
- davnn 6y agoDoes anyone have insights into why data.table is as fast as it is? PS: Great work with polars!
- Hendrikto 6y ago> This directly shows a clear advantage over Pandas for instance, where there is no clear distinction between a float NaN and missing data, where they really should represent different things. Not true anymore: > Starting from pandas 1.0, an experimental pd.NA value (singleton) is available to represent scalar missing values. At this moment, it is used in the nullable integer, boolean and dedicated string data types as the missing value indicator. > The goal of pd.NA is provide a “missing” indicator that can be used consistently across data types (instead of np.nan, None or pd.NaT depending on the data type). (https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#missing-data-na https://pandas.pydata.org/pandas-docs/stable/user_guide/miss...)
- wtallis 6y agoAnd much more recently (December 26, 2020): https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.2.0.html https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.2.0... > Experimental nullable data types for float data > We’ve added Float32Dtype / Float64Dtype and FloatingArray. These are extension data types dedicated to floating point data that can hold the pd.NA missing value indicator (GH32265, GH34307). > While the default float data type already supports missing values using np.nan, these new data types use pd.NA (and its corresponding behavior) as the missing value indicator, in line with the already existing nullable integer and boolean data types.
- kzrdude 6y agoI wonder how pandas can both be at version 1.0 and have a an experimental feature for something so central. Honest question
- Hendrikto 6y agoIt‘s at version 1.0 because it has a mature and stable interface. That does not mean that it cannot have experimental features which are not part of that stable interface.
- otsaloma 6y agoBecause Pandas is built on top of NumPy and NumPy has never had a proper NA value. I would call that a serious design problem in NumPy, but it seems to be difficult to fix. There have been multiple NEPs (NumPy Enhancement Proposals) over the years, but they haven't gone anywhere. Probably since things are not moving along in NumPy, a lot of development that should logically happen at the NumPy level is now happening in Pandas. But, I agree, I find it baffling how Python has gotten so big in data science and been around so long without having proper NA support. https://numpy.org/neps/#deferred-and-superseded-neps https://numpy.org/neps/#deferred-and-superseded-neps
- sradman 6y agoMy naive interpretation - The canonical Apache Arrow implementation is written in C++ with multiple language bindings like PyArrow. The Rust bindings for Apache Arrow re-implemented the Arrow specification so it can be used as a pure Rust implementation. Andy Grove [1] built two projects on top of Rust-Arrow: 1. DataFusion, a query engine for Arrow that can optimize SQL-like JOIN and GROUP BY queries, and 2. Ballista, clustered DataFusion-like queries (vs. Dask and Spark). DataFusion was integrated into the Apache Arrow Rust project. Ritchie Vink has introduced Polars that also builds upon Rust-Arrow. It offers an Eager API that is an alternative to PyArrow and a Lazy API that is a query engine and optimizer like DataFusion. The linked benchmark is focused on JOIN and GROUP BY queries on large datasets executed on a server/workstation-class machine (125 GB memory). This seems like a specialized use case that pushes the limits of a single developer machine and overlaps with the use case for a dedicated column store (like Redshift) or a distributed batch processing system like Spark/MapReduce. Why Polars over DataFusion? Why Python bindings to Rust-Arrow rather than canonical PyArrow/C++? Is there something wrong with PyArrow? [1] https://andygrove.io/projects/ https://andygrove.io/projects/
- ritchie46 6y agoHi Author here, Polars is not an alternative to PyArrow. Polars merely uses arrow as its in-memory representation of data. Similar to how pandas uses numpy. Arrow provides the efficient data structures and some compute kernels, like a SUM, a FILTER, a MAX etc. Arrow is not a query engine. Polars is a DataFrame library on top of arrow that has implemented efficient algorithms for JOINS, GROUPBY, PIVOTs, MELTs, QUERY OPTIMIZATION, etc. (the things you expect from a DF lib). Polars could be best described as an in-memory DataFrame library with a query optimizer. Because it uses Rust Arrow, it can easily swap pointers around to pyarrow and get zero-copy data interop. DataFusion is another query engine on top of arrow. They both use arrow as lower level memory layout, but both have a different implementation of their query engine and their API. I would say that DataFusion is more focused on a Query Engine and Polars is more focused an a DataFrame lib, but this is subjective. Maybe its like comparing Rust Tokio vs Rust async-std. Just different implementations striving the same goal. (Only Polars and DataFusion can easily be mixed as they use the same memory structures).
- sdmac99 6y agoI would like it if Vaex was added to the benchmarks.
- robin21 6y agoIs there a data frame project for Node.js?
- lmeyerov 6y agoWe have been rewriting our stack for multi/many-gpu scale out via python GPU dataframes, but it's clear that smaller workloads and some others would be fine on CPU (and thus free up the GPUs for our other tenants), so having a good CPU impl is exciting, esp if they achieve API compatibility w pandas/dask as RAPIDS and others do. I've been eyeing vauex here (I think the other rust arrow project isn't DF's?), so good to have a contender! I'd love to see a comparison to RAPIDS dataframes for the single GPU case (ex: 2 GB), single GPU bigger-than-memory (ex: 100 GB), and then the same for multi-GPU. We have started to measure as things like "200 GB/s in-memory and 60 GB / s when bigger than memory", to give perspective.
- voceboy521 6y agoam i supposed to know what a dataframes is?