2 ms·

And if you're looking for a similar experience (very fast analytical SQL queries) but over HTTP, for example, to power a public dashboard or a visualization, yo

And if you're looking for a similar experience (very fast analytical SQL queries) but over HTTP, for example, to power a public dashboard or a visualization, you can try ROAPI [0] or Seafowl [1], also built on top of DataFusion (disclaimer: working on Seafowl): [0]: https://github.com/roapi/roapi https://github.com/roapi/roapi [1]: https://github.com/splitgraph/seafowl https://github.com/splitgraph/seafowl

cube2222 4y ago
That's what I thought about ROAPI as well, until I benchmarked it, and it ended up being very slow[0]. [0]: https://news.ycombinator.com/item?id=32970495 https://news.ycombinator.com/item?id=32970495
- mildbyte 4y ago
  It could be the NDJSON parser (DF source: [0]) or could be a variety of other factors. Looking at the ROAPI release archive [1], it doesn't ship with the definitive `columnq` binary from your comment (EDIT: it does, I was looking in the wrong place! https://github.com/roapi/roapi/releases/tag/columnq-cli-v0.3.0 https://github.com/roapi/roapi/releases/tag/columnq-cli-v0.3...), so it could also have something to do with compilation-time flags. FWIW, we use the Parquet format with DataFusion and get very good speeds similar to DuckDB [2], e.g. 1.5s to run a more complex aggregation query `SELECT date_trunc('month', tpep_pickup_datetime) AS month, COUNT(*) AS total_trips, SUM(total_amount) FROM tripdata GROUP BY 1 ORDER BY 1 ASC)` on a 55M row subset of NY Taxi trip data. [0]: https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/datasource/file_format/json.rs https://github.com/apache/arrow-datafusion/blob/master/dataf... [1]: https://github.com/roapi/roapi/releases/tag/roapi-v0.8.0 https://github.com/roapi/roapi/releases/tag/roapi-v0.8.0 [2]: https://observablehq.com/@seafowl/benchmarks https://observablehq.com/@seafowl/benchmarks
  - cube2222 4y ago
    Yes, DataFusion itself is definitely fast, no denying that.

cube2222 4y ago

That's what I thought about ROAPI as well, until I benchmarked it, and it ended up being very slow[0]. [0]: https://news.ycombinator.com/item?id=32970495 https://news.ycombinator.com/item?id=32970495

mildbyte 4y ago
It could be the NDJSON parser (DF source: [0]) or could be a variety of other factors. Looking at the ROAPI release archive [1], it doesn't ship with the definitive `columnq` binary from your comment (EDIT: it does, I was looking in the wrong place! https://github.com/roapi/roapi/releases/tag/columnq-cli-v0.3.0 https://github.com/roapi/roapi/releases/tag/columnq-cli-v0.3...), so it could also have something to do with compilation-time flags. FWIW, we use the Parquet format with DataFusion and get very good speeds similar to DuckDB [2], e.g. 1.5s to run a more complex aggregation query `SELECT date_trunc('month', tpep_pickup_datetime) AS month, COUNT(*) AS total_trips, SUM(total_amount) FROM tripdata GROUP BY 1 ORDER BY 1 ASC)` on a 55M row subset of NY Taxi trip data. [0]: https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/datasource/file_format/json.rs https://github.com/apache/arrow-datafusion/blob/master/dataf... [1]: https://github.com/roapi/roapi/releases/tag/roapi-v0.8.0 https://github.com/roapi/roapi/releases/tag/roapi-v0.8.0 [2]: https://observablehq.com/@seafowl/benchmarks https://observablehq.com/@seafowl/benchmarks
- cube2222 4y ago
  Yes, DataFusion itself is definitely fast, no denying that.