4 ms·

There are benchmarks here - https://github.com/Eventual-Inc/Daft?tab=readme-ov-file#benchmarks https://github.com/Eventual-Inc/Daft?tab=readme-ov-file#benc....

by hiyer 3y ago

There are benchmarks here - https://github.com/Eventual-Inc/Daft?tab=readme-ov-file#benchmarks https://github.com/Eventual-Inc/Daft?tab=readme-ov-file#benc.... Seems to outperform Dask by a fair bit.

SiempreViernes 3y ago
These benchmarks seem strange, it looks like some sort of database test suite? And compared to the performance that is listed on the benchmarks own page[1], the Daft results look abysmal? Like Daft takes several hours to complete ten queries and the results the benchmark authors point to can do around a million queries per hour... oh, and the software they use for that is MS-SQL. [1]: https://www.tpc.org/tpch/results/tpch_advanced_sort5.asp?PRINTVER=false&VERSION=3&FLTCOL1=h_sf&FLTCOLOPR1=%3D&FLTCHO1=1000&ADDFILTERROW=&filterRowCount=1&SRTCOL1=h_server_threads&SRTDIR1=ASC&ADDSORTROW=&sortRowCount=1&DISPRES=100+PERCENT&include_withdrawn_results=none&include_historic_results=no https://www.tpc.org/tpch/results/tpch_advanced_sort5.asp?PRI...
- jaychia 3y ago
  Hello! Daft developer here. The benchmarks we performed aren’t directly comparable to the benchmarks on TPC-H’s own page because of differences in hardware, storage etc. For hardware, we were using AWS i3.2xlarge machines in a distributed cluster. And on the storage side we are reading Parquet files over the network from AWS S3. This is most representative of how users run query engines like Daft. The TPC-H benchmarks are usually performed on databases which have pre-ingested the data into a single-node server-grade machine that’s running the database. Note that Daft isn’t really a “database”, because we don’t have proprietary storage. Part of the appeal of using query engines like Daft and Spark is to able to read data “at rest” (as Parquet, CSV, JSON etc). However this will definitely be slower than a database which has pre-ingested the data into indexed storage and proprietary formats! Hope that helps explain the discrepancies!
  - pletnes 3y ago
    I’m sure you mean slow = high latency, but I do hope you’re «high throughput», like spark/…?
- kristjansson 3y ago
  The QphH metric reported for TPC-H[1] is _not_ queries-per-hour, it's a composite metric: > The performance metric reported by TPC-H is called the TPC-H Composite Query-per-Hour Performance Metric (QphH@Size), and reflects multiple aspects of the capability of the system to process queries. so it's not directly comparable to seconds/query reported by Daft. [1]: https://www.tpc.org/tpch/default5.asp https://www.tpc.org/tpch/default5.asp

by hiyer 3y ago

There are benchmarks here - https://github.com/Eventual-Inc/Daft?tab=readme-ov-file#benchmarks https://github.com/Eventual-Inc/Daft?tab=readme-ov-file#benc.... Seems to outperform Dask by a fair bit.

SiempreViernes 3y ago

These benchmarks seem strange, it looks like some sort of database test suite? And compared to the performance that is listed on the benchmarks own page[1], the Daft results look abysmal? Like Daft takes several hours to complete ten queries and the results the benchmark authors point to can do around a million queries per hour... oh, and the software they use for that is MS-SQL. [1]: https://www.tpc.org/tpch/results/tpch_advanced_sort5.asp?PRINTVER=false&VERSION=3&FLTCOL1=h_sf&FLTCOLOPR1=%3D&FLTCHO1=1000&ADDFILTERROW=&filterRowCount=1&SRTCOL1=h_server_threads&SRTDIR1=ASC&ADDSORTROW=&sortRowCount=1&DISPRES=100+PERCENT&include_withdrawn_results=none&include_historic_results=no https://www.tpc.org/tpch/results/tpch_advanced_sort5.asp?PRI...

jaychia 3y ago
Hello! Daft developer here. The benchmarks we performed aren’t directly comparable to the benchmarks on TPC-H’s own page because of differences in hardware, storage etc. For hardware, we were using AWS i3.2xlarge machines in a distributed cluster. And on the storage side we are reading Parquet files over the network from AWS S3. This is most representative of how users run query engines like Daft. The TPC-H benchmarks are usually performed on databases which have pre-ingested the data into a single-node server-grade machine that’s running the database. Note that Daft isn’t really a “database”, because we don’t have proprietary storage. Part of the appeal of using query engines like Daft and Spark is to able to read data “at rest” (as Parquet, CSV, JSON etc). However this will definitely be slower than a database which has pre-ingested the data into indexed storage and proprietary formats! Hope that helps explain the discrepancies!
- pletnes 3y ago
  I’m sure you mean slow = high latency, but I do hope you’re «high throughput», like spark/…?
kristjansson 3y ago
The QphH metric reported for TPC-H[1] is _not_ queries-per-hour, it's a composite metric: > The performance metric reported by TPC-H is called the TPC-H Composite Query-per-Hour Performance Metric (QphH@Size), and reflects multiple aspects of the capability of the system to process queries. so it's not directly comparable to seconds/query reported by Daft. [1]: https://www.tpc.org/tpch/default5.asp https://www.tpc.org/tpch/default5.asp