4 ms·
Daft: Distributed DataFrame for Python
- jmakov 3y agoWondering if network becomes a bottleneck. Anybody using this?
- teitoklien 3y agoGreat question! Depends on your workload size, if the compute time is less than time it takes to transfer the data back and forth, then it might be a bad idea to use it indeed. Typically these solutions should only be tested after maxing out vertical scaling, before applying for horizontal scaling. Network is one hell of a destroyer when it comes to advantages gained from distributed computing.
- spratzt 3y agoI would go even further and argue that for the vast majority of organizations vertical scaling is all you are ever going to need.
- sammysidhu 3y agohorizontal scaling however provides you with more aggregate network bandwidth. Most enterprises run workloads that downloads data from a data lake (usually S3), which is usually the bottleneck. Having horizontal scaling here allows the query leverage much higher network than just having a large single machine.
- jaychia 3y agoHello, Daft developer here! The network indeed becomes the bottleneck. In 2 main ways: 1. Reading data from cloud storage is very expensive. Here’s a blogpost where we talk about some of the optimizations we’ve done in that area: https://blog.getdaft.io/p/announcing-daft-02-10x-faster-io https://blog.getdaft.io/p/announcing-daft-02-10x-faster-io 2. During a global shuffle stage (e.g. sorts, joins, aggregations) network transfer of data between nodes becomes the bottleneck. This is why the advice is often to stick with a local solution such as DuckDB, Polars or Pandas if you can keep vertically scaling! However, horizontally scaling does have some advantages: - Higher aggregate network bandwidth for performing I/O with storage - Auto-scaling to your workload’s resource requirements - Scaling to large workloads which may not fit on a single machine. This is more common in Daft usage because we also work with multimodal data such as images, tensors and more for ML data modalities. Hope this helps!
- mborch 3y agoThey mention https://pola.rs/ https://pola.rs/ and highlight how it has the exact same properties (out of core etc) but don't include it in benchmarks and no mention on their detailed comparison page.
- wdroz 3y agoIt's because they are comparing to distributed solutions only. Polars is one of the fastest solution for single computer workflow, but it doesn't support distributed workflow.
- baggiponte 3y agoI was told they use Polars engine; so they "just" build a distributed system around it
- wdroz 3y agoI don't see any direct dependencies to polars. But in the comment, they wrote that some part are "taken from", "influenced by", "adapted from" and "based on" polars. [0] -- https://github.com/search?q=repo%3AEventual-Inc%2FDaft%20polars&type=code https://github.com/search?q=repo%3AEventual-Inc%2FDaft%20pol...
- baggiponte 3y agoyes exactly, they """took""" part of the Polars engine code.
- antman 3y agoMany many dataframes and dialects. Is this dataframe dialect similar to polars or pandas? Porting and testing is a key factor for adoption Regex for strings and more detail on how partitioning works would be interesting
- jaychia 3y agoHello! Daft developer here. We are most similar in API to Polars and PySpark. And thanks for the feedback! We’ll add more capabilities for regex, as well as flesh out our documentation for partitioning. Edit: added a new issue for regex support :) https://github.com/Eventual-Inc/Daft/issues/1962 https://github.com/Eventual-Inc/Daft/issues/1962
- roter 3y ago> To help improve Daft, we collect non-identifiable data. > To disable this behavior, set the following environment variable: DAFT_ANALYTICS_ENABLED=0 > [0] In short, we collect the following: > On import, we track system information such as the runner being used, version of Daft, OS, Python version, etc. > On calls of public methods on the DataFrame object, we track metadata about the execution: the name of the method, the walltime for execution and the class of error raised (if any). Function parameters and stacktraces are not logged, ensuring that user data remains private. Is this telemetry really necessary? [0] https://www.getdaft.io/projects/docs/en/latest/faq/telemetry.html https://www.getdaft.io/projects/docs/en/latest/faq/telemetry...
- buremba 3y agoI don't think they call it "really necessary" but I assume it's handy to understand the user base and focus on the right use cases in the long run considering not all the users who use these libraries are active on Github.
- deleted 3y ago[deleted]
- jonnat 3y agoIt has an Apache 2.0 license. I assume it allows it to be repackaged removing telemetry.
- pletnes 3y agoHow does this compare with dask? (Polars was mentioned but isn’t distributed like dask is)
- hiyer 3y agoThere are benchmarks here - https://github.com/Eventual-Inc/Daft?tab=readme-ov-file#benchmarks https://github.com/Eventual-Inc/Daft?tab=readme-ov-file#benc.... Seems to outperform Dask by a fair bit.
- SiempreViernes 3y agoThese benchmarks seem strange, it looks like some sort of database test suite? And compared to the performance that is listed on the benchmarks own page[1], the Daft results look abysmal? Like Daft takes several hours to complete ten queries and the results the benchmark authors point to can do around a million queries per hour... oh, and the software they use for that is MS-SQL. [1]: https://www.tpc.org/tpch/results/tpch_advanced_sort5.asp?PRINTVER=false&VERSION=3&FLTCOL1=h_sf&FLTCOLOPR1=%3D&FLTCHO1=1000&ADDFILTERROW=&filterRowCount=1&SRTCOL1=h_server_threads&SRTDIR1=ASC&ADDSORTROW=&sortRowCount=1&DISPRES=100+PERCENT&include_withdrawn_results=none&include_historic_results=no https://www.tpc.org/tpch/results/tpch_advanced_sort5.asp?PRI...
- jaychia 3y agoHello! Daft developer here. The benchmarks we performed aren’t directly comparable to the benchmarks on TPC-H’s own page because of differences in hardware, storage etc. For hardware, we were using AWS i3.2xlarge machines in a distributed cluster. And on the storage side we are reading Parquet files over the network from AWS S3. This is most representative of how users run query engines like Daft. The TPC-H benchmarks are usually performed on databases which have pre-ingested the data into a single-node server-grade machine that’s running the database. Note that Daft isn’t really a “database”, because we don’t have proprietary storage. Part of the appeal of using query engines like Daft and Spark is to able to read data “at rest” (as Parquet, CSV, JSON etc). However this will definitely be slower than a database which has pre-ingested the data into indexed storage and proprietary formats! Hope that helps explain the discrepancies!
- chrisaycock 3y agoHere's the comparison chart to other Dataframe libraries: https://www.getdaft.io/projects/docs/en/latest/faq/dataframe_comparison.html https://www.getdaft.io/projects/docs/en/latest/faq/dataframe... Daft is distributed and vectorized with plenty of benefits from lazy evaluation.
- benrutter 3y agoSuper tiny nitpick but dask is down as non-arrow back end. Since it uses pandas as its underlying data structure, fully arrow backends should be fine as long as the installed version of pandas is 2+
- 3abiton 3y agoInteresting, the main selling point is, pySpark but better as it is python native?
- hiyer 3y agoReally impressive benchmarks - outperforms even Spark by a good margin. Great work!
- sammysidhu 3y agoOne of the daft developers here - Thank you!
- nivekney 3y agoI see Rust, I like
- sammysidhu 3y agowe love rust too!
- amamama 3y agoWe did extensive comparison with Spark and a lot of other tools at work. They always came on top. Highly recommend!
- whoevercares 3y agoLooks like iceberg support is coming - dope!
- jaychia 3y agoDaft developer here! We actually already have read support. Check out the pyiceberg docs' Daft section: https://py.iceberg.apache.org/api/#daft https://py.iceberg.apache.org/api/#daft It's also very easy to use from Daft itself: `daft.read_iceberg(pyiceberg_table)`. Give it a shot and let us know how it works for you!
- oblvious-earth 3y agoIs this comparable to Ibis or is it something that would be a backend for it?
- sammysidhu 3y agoThis would be more comparable to being a backend for ibis. We're working on adding the remaining operations (like regex on strings or trigonometry functions) that ibis requires!
- benrutter 3y agoThis looks really nice! It feels like in the last few years (probably with the increase in data science and analytics) dataframe engines have become really exciting, its very cool to see so much innovation in the field. One thing i haven't seen yet and I'd be really interested in knowing if column level type hinting is on the plan at all for this (or any similar tools)? I'm a data engineer and my team use mypy pretty heavily, but that can only really tell us if something is a "dataframe" type. It always strikes me that 90% of data issues during development (incorrect joins, column misspellings or non-type appropriate operations, not factoring in nulls) would be eliminated if mypy had information of the columns and their data types.
- jaychia 3y agoInteresting. Daft currently does validation on types/names only at runtime. The flow looks like: 1. Construct a dataframe (performs schema inference) 2. Access (now well-typed) columns and operations on those columns in the dataframe, with associated validations. Unfortunately step (1) can only happen at runtime and not at type-checking-time since it requires running some schema inference logic, and step (2) relies on step (1) because the expressions of computation are "resolved" against those inferred types. However, if we can fix (1) to happen at type-checking time using user-provided type-hints in place of the schema inference, we can maybe figure out a way to propagate this information through to mypy. Would love to continue the discussion further as an Issue/Discussion on our Github!
- benrutter 3y agoThanks for getting back, thats really interesting, for sure I will do!