3 ms·

Take a look at query engines like Trino (formerly PrestoSQL) [https://trino.io/ https://trino.io/]. (Disclaimer: I'm a contributor to Trino). I used it at a pr

Take a look at query engines like Trino (formerly PrestoSQL) [https://trino.io/ https://trino.io/]. (Disclaimer: I'm a contributor to Trino). I used it at a previous job to combine data from MongoDB, Kafka, S3 and Postgres to great effect. It tries to push-down as many operations as possible to the source too to improve performance. Full ANSI SQL support over multiple number of backends (Kafka, Cassandra, Postgres, ClickHouse, S3 and many more). The best part is it has a plugin ecosystem so you can very easily implement your own connectors and all the heavy lifting gets done by the core-engine while your plugin only has to abstract your backend to concepts that the engine can understand.

ckdarby 5y ago
Came here to say the same great things about Trino
flakiness 5y ago
Offtopic, but I didn't realize there is prestosql(now trino)[1] and prestodb. Was only aware of prestodb side. I love the paper [3], which was apparently written before they left FB to form the new company. [1] https://trino.io/blog/2020/12/27/announcing-trino.html [2] https://prestodb.io/ [3] https://trino.io/Presto_SQL_on_Everything.pdf
LaserToy 5y ago
The problem with Trino is that it is not that easy to scale to possible RPS of Clickhouse, it introduces tons a of latency and push downs are faaaar from perfect. Uber has a smart solution for Pinot, when they run it as a single node proxies
- hashhar 5y ago
  I wouldn't use Trino if you are looking to ONLY query a single database like Clickhouse or Postgres etc (unless you want an ANSI SQL abstraction over your choice of database). Obviously ClickHouse and Postgres will have lower latency when hit directly because you can bypass the analysis, planning, optimization and scheudling that Trino does. It does federation better than ClickHouse and that's where it shines. Joins across disparate systems - even between relational and non-relational systems. And obviously for the MPP queries on distributed filesystems.
  - LaserToy 5y ago
    I know. I worked on Presto :)

ckdarby 5y ago

Came here to say the same great things about Trino

flakiness 5y ago

Offtopic, but I didn't realize there is prestosql(now trino)[1] and prestodb. Was only aware of prestodb side. I love the paper [3], which was apparently written before they left FB to form the new company. [1] https://trino.io/blog/2020/12/27/announcing-trino.html [2] https://prestodb.io/ [3] https://trino.io/Presto_SQL_on_Everything.pdf

LaserToy 5y ago

The problem with Trino is that it is not that easy to scale to possible RPS of Clickhouse, it introduces tons a of latency and push downs are faaaar from perfect. Uber has a smart solution for Pinot, when they run it as a single node proxies

hashhar 5y ago
I wouldn't use Trino if you are looking to ONLY query a single database like Clickhouse or Postgres etc (unless you want an ANSI SQL abstraction over your choice of database). Obviously ClickHouse and Postgres will have lower latency when hit directly because you can bypass the analysis, planning, optimization and scheudling that Trino does. It does federation better than ClickHouse and that's where it shines. Joins across disparate systems - even between relational and non-relational systems. And obviously for the MPP queries on distributed filesystems.
- LaserToy 5y ago
  I know. I worked on Presto :)