7 ms·

The SQL query engine Trino (formerly PrestoSQL) recaps a decade of innovation

96 points by simpligility 4y ago

tekkertje 4y ago
One of my favorite OSS projects! Probably the most flexible and fully featured distributed SQL query engine around. Congrats and looking forward to the next decade!
- bitsondatadev 4y ago
  <3
QuotedAtoms 4y ago
Can anyone clarify the differences between Trino and SparkSQL? Our company has used SparkSQL to aggressively replace use-cases that were based on PrestoSQL in the past.
- ergocoder 4y ago
  I can chime in in an okayish useful manner. Apart from implementation details, probably not much different. It is similar to mysql vs postgresql. You are probably okay with either.
  - hashhar 4y ago
    I must disclaim that I contribute to Trino. I agree but it depends a bit on what purpose you are using them for. If you mainly use the tool to JOIN some data in bulk and then write output somewhere else (i.e. ETL) - either will serve you fine. If you write complex queries with multiple filters and want to JOIN across multiple datasets - sure Spark can do that as well but it's not as efficient in pushing down computation to the source. e.g. A query like SELECT c.custkey, sum(totalprice) FROM orders o INNER JOIN customer c ON o.custkey = c.custkey WHERE o.orderstatus = 'O' GROUP BY c.custkey; when ran on Spark will pull both tables into memory and then perform the join + filter for orderstatus = 'O' and then compute the sum. While in case of Trino it'll push down the entire query into the remote database (in this case, in other queries it'll push down some parts of the query) so the source database will not need to return gigabytes of data over the network every time the query runs (and hence finish faster as well). Trino tries to push-down some operations to the remote system which can be done more efficiently there. e.g. filtering on a column that has an index in the remote RDBMS will be faster than pulling all data and then filtering in Trino. Spark doesn't have strong pushdown and has to pull most of the raw data and then apply processing on top of it. That's one of the main differences. Spark is a distributed job execution framework first while Trino is a distributed federated query engine first and it shows in their strengths and weaknesses. If you want to run arbitrary user defined transformations on data then Spark definitely has much more to offer than Trino.
simpligility 4y ago
Its great to see how far the project has come from the humble beginnings to the current, rich open source ecosystem and community.
bitsondatadev 4y ago
btw, if you want to know the backstory on why Presto is now called Trino, here's the article: https://trino.io/blog/2022/08/02/leaving-facebook-meta-best-for-trino.html https://trino.io/blog/2022/08/02/leaving-facebook-meta-best-...
georgewfraser 4y ago
The thing I wonder about with Presto and to a lesser extent Spark is, how many of their users adopted this tool because it was an easy migration path from Hive, and how many of those users will eventually re-platform to something else?
- bitsondatadev 4y ago
  I mean, the hive migration path is one thing. Now that Iceberg is taking over the old Hive model, data lakes are all the rage again. The other thing I would say is that Trino and Presto are not one-trick ponies or just hive replacements. There's also the ability to query across multiple systems that is, to me, the feature that future proofs a lot of architectures. It inherently frees you up to fiddle with your data in different systems but keep the access to that system in one location.
  - georgewfraser 4y ago
    Yeah I think that is the key question: will data lakes become the dominant paradigm? There is certainly a lot of talk around them, though I see a ton of companies are still just going all in on a conventional data warehouse, but they tend not to talk about it because it’s not a new or interesting thing to do.
    - bitsondatadev 4y ago
      Yeah, though a lot of Fivetran customers are likely the type that would go all in on paying for a conventional data warehouse where people using open source stacks may be the ones that are using open ingestion alternatives. We see a pretty even mix from the Trino/Starburst lens. Bigger companies like to mix and match.
gavinray 4y ago
I recently had to write SQL query generation for AWS Athena, which is based off Presto 0.217 It turns out that the dialect doesn't support LATERAL joins with a LIMIT in them. The below query only works if you remove the LIMIT clause. https://i.stack.imgur.com/rdB1s.png https://i.stack.imgur.com/rdB1s.png This makes saying things like "Fetch all artists where ..., for each artist fetch their first 3 albums where ..., and for each album fetch the top 10 tracks where ..." really difficult Does Trino support this out of curiosity?
- karma_fountain 4y ago
  Is it possible to achieve this with a window function?
  - gavinray 4y ago
    I found out it is! Kudos to this kind internet stranger for telling me: https://stackoverflow.com/a/73129836/13485494 https://stackoverflow.com/a/73129836/13485494 But man is it a huge PITA (especially when doing programmatic code generation of the SQL) compared to LATERAL joins Someone familiar with the CockroachDB query planner showed me that a window function like this is what Cockroach turns LATERAL joins into for instance: demo@127.0.0.1:26257/movr> explain select * from abc, lateral (select * from xyz where x = a limit 2); • filter │ estimated row count: 1 │ filter: row_num <= 2 │ └── • window │ estimated row count: 2 │ └── • hash join │ estimated row count: 2 │ equality: (x) = (a) │ ├── • scan │ estimated row count: 6 (100% of the table; stats collected 2 minutes ago) │ table: xyz@xyz_pkey │ spans: FULL SCAN │ └── • scan estimated row count: 1 (100% of the table; stats collected 3 minutes ago) table: abc@abc_pkey spans: FULL SCAN
- bitsondatadev 4y ago
  Check out this PR. I believe we may have tackled this one but you'd need to try it out on Trino: https://github.com/trinodb/trino/pull/1415 https://github.com/trinodb/trino/pull/1415
ck_one 4y ago
Can Trino be used as a Snowflake replacement? How is the query speed compared to Snowflake?
- cpard 4y ago
  Hey ck_one that's a hard question to answer and not get into "benchmarketing" territory. My suggestion is to try both under your own workloads and see the difference. Trino is also used by products like Athena (AWS) and Galaxy (Starburst) so if you want to play around and see how Trino performs without spending too much time on setting up clusters on your own, you can try these great products. Having said that, I'd like to add that building a performant distributed query engine is just hard. Trino has been in development for ten years and used by major companies in very demanding environments, these environments is where the technology has been defined and makes it what it is today and it is a proof of its performance and stability. (edited to add an important disclaimer that I work at Starburst)
- simpligility 4y ago
  Yes ... Starburst Enterprise, which is a commercial distribution of Trino, can in fact also query Snowflake, but also Delta Lake and many many other systems at the same time.
  - dominotw 4y ago
    hey do you work for the company. Prbly add a disclaimer.
    - simpligility 4y ago
      Yes... where would I put the disclaimer?
- chrsig 4y ago
  i can't speak to trino, but with my experience with aws athena and snowflake, they're roughly on par with each other across the board.
- 4y ago
simpligility 4y ago
Also working on a new edition for Trino: The Definitive Guide at the moment.
dmead 4y ago
I do the support for my department's trino cluster. We move ~1tb (and growing) in ETL jobs and support interactive queries for the data scientists/analysts. It would be super good if you guys added big query write support. Its really annoying to have to run a hive cluster in google to act as a proxy for this.
- atwebb 4y ago
  Any chance you have an overview of the architecture and operations support required? How many data sources are you pinging?
  - dmead 4y ago
    like 20 different sources. I nag my service reps about it. I"m sure it's been filed in your jira.
- hashhar 4y ago
  BigQuery very recently announced their Storage Write API which is one of the ways we were looking to implement this but there are some issues with the latency and consistency guarantees that it offers. But, yes, we do plan to add that eventually after ironing out all the kinks. See https://github.com/trinodb/trino/pull/13094 https://github.com/trinodb/trino/pull/13094
  - bitsondatadev 4y ago
    Also, you can keep track of all the BQ progress here: https://github.com/trinodb/trino/issues/6867 https://github.com/trinodb/trino/issues/6867
    - dmead 4y ago
      thanks. If we can eliminate the costs and upkeep of hive in gcp, it would make my life easier for sure.
skadamat 4y ago
Big shout out to Brian Olsen from the Trino community (and Starburst) for helping the Trino community be successful - https://github.com/bitsondatadev https://github.com/bitsondatadev - https://www.linkedin.com/in/bitsondatadev/ https://www.linkedin.com/in/bitsondatadev/ I recommend the Trino Slack for people not already in it: https://trino.io/slack.html https://trino.io/slack.html
- bitsondatadev 4y ago
  Thanks for the shoutout! :) If you want to get started with Trino, here's a repo I created to do so: https://github.com/bitsondatadev/trino-getting-started https://github.com/bitsondatadev/trino-getting-started
simpligility 4y ago
Also just to note.. I am currently working on a refresh of Trino: The Definitive Guide .. and would love to see you all at Trino Summit in November. https://trino.io/blog/2022/06/30/trino-summit-call-for-speakers.html https://trino.io/blog/2022/06/30/trino-summit-call-for-speak...
jerryjerryjerry 4y ago
One of the features I'm interested in (or would like to have) from Trino or Presto is the workload management which can better manage different types of queries and allocate resources accordingly. This becomes important when more applications adopt Trino or Presto as a distributed SQL database/platform, where the impact from different queries or workloads can be mitigated, besides the dedicated resources (CPU, MEM, etc.) can be allocated to high priority workloads. I'm really wondering if/when such capabilities may be provided. BTW, purely curiosity, I compared Trino with Presto from OSS point of view (https://ossinsight.io/analyze/prestodb/presto?vs=trinodb%2Ftrino https://ossinsight.io/analyze/prestodb/presto?vs=trinodb%2Ft...), both communities are still popular but Trino seems more active than Presto now. I also wonder if two communities may reunion someday again to really boost its impact (comparing to Spark community).
- bitsondatadev 4y ago
  For managing difrerent workloads, check out this blogs and this videos from Shopify, Salesforce, Goldman Sachs, and Electronic Arts, respectively: - https://engineering.salesforce.com/how-to-etl-at-petabyte-scale-with-trino-5fe8ac134e36/ https://engineering.salesforce.com/how-to-etl-at-petabyte-sc... - https://shopify.engineering/faster-trino-query-execution-infrastructure https://shopify.engineering/faster-trino-query-execution-inf... - https://trino.io/episodes/33.html https://trino.io/episodes/33.html - https://www.youtube.com/watch?v=-5mlZGjt6H4 https://www.youtube.com/watch?v=-5mlZGjt6H4 All use the Lyft "Presto but really Trino"-Gateway project to run different clusters to handle various workloads. They go into various details for how this is achieved. https://github.com/lyft/presto-gateway https://github.com/lyft/presto-gateway Regarding the Trino/Presto split. I recommend looking at this blog to better understand why these two communities aren't mergeing. TL;DR Presto is a Facebook-driven project that mainly considers running on the Facebook infrastructure. Trino is community-driven that works on running well with all clouds and common infasturcture in the Trino community which is why you see a higher velocity there. https://trino.io/blog/2022/08/02/leaving-facebook-meta-best-for-trino.html https://trino.io/blog/2022/08/02/leaving-facebook-meta-best-... https://trino.io/blog/2020/12/27/announcing-trino.html https://trino.io/blog/2020/12/27/announcing-trino.html Soon we anticipate that Trino will become the common name in the community space but we'll always love the origins of the Trino project being Presto.
tzury 4y ago
Trino vs ClickHouse, can anyone tell from experience how those two compare?
- bitsondatadev 4y ago
  Clickhouse is a realtime system where Trino is a batch-oriented system. There are tradeoffs for doing realtime vs batch. Realtime is generally more expensive to run as you process every individual row as it comes, batch is when you can deal with minute latency and want to handle a lot of data in chunks. Trino is also a query engine rather than a database and it connects to many different systems: https://trino.io/docs/current/connector.html https://trino.io/docs/current/connector.html It also happens to connect to Clickhouse and it's very common that people will use Trino to query clickhouse realtime data and join it with data in big query, an object store data lake, or Snowflake: https://trino.io/docs/current/connector/clickhouse.html https://trino.io/docs/current/connector/clickhouse.html
- deleted 4y ago
  [deleted]
- qoega 4y ago
  You can consider that ClickHouse allows both to query a lot of supported external data sources(s3/hdfs/mysql/postgre/...) and to store data in pretty efficient columnar way with compression, indexes and all the bells and whistles. Native storage allows to use all the information about keys/indices to build query plan faster. With trino you can't store data inside trino. You can't even insert data using trino which allows you to solve scenarios like 'readonly analytics'. Trino allows you to use single query language for all the supported systems. So if you have a zoo of DBMS and object storages that you can just query it can help you to hide this complexity.
mrwnmonm 4y ago
Can you use Trino as a database proxy?
- simpligility 4y ago
  Essentially yes ... it can be the query engine for many databases at the same time.
kache_ 4y ago
trino is awesome can't believe this shit is free as in freedom
mrwnmonm 4y ago
We are building a SaaS BI tool. To enable the users to connect to their databases... we have a form that collects the database credentials from the user, saves it in a secure way, and when the user writes or uses an SQL query, we establish a database connection right away (from our server), execute it, and return the results, and we keep the connection alive for like 15mins. But with serverless architecture, first query could go to instance 1, so instance 1 will establish a db connection, then the second query could go to instance 2, so instance 2 will establish another one. You could end up with a lot of unnecessary connections. If you use AWS RDS (for yourself), beside lambda for example, AWS have RDS Proxy to solve this problem. So I was thinking about using Trino like the RDS Proxy, but for more databases, and for our customers database, not ours. Is that doable with Trino?

96 points by simpligility 4y ago

One of my favorite OSS projects! Probably the most flexible and fully featured distributed SQL query engine around. Congrats and looking forward to the next decade!

bitsondatadev 4y ago
<3

QuotedAtoms 4y ago

Can anyone clarify the differences between Trino and SparkSQL? Our company has used SparkSQL to aggressively replace use-cases that were based on PrestoSQL in the past.

ergocoder 4y ago
I can chime in in an okayish useful manner. Apart from implementation details, probably not much different. It is similar to mysql vs postgresql. You are probably okay with either.
- hashhar 4y ago
  I must disclaim that I contribute to Trino. I agree but it depends a bit on what purpose you are using them for. If you mainly use the tool to JOIN some data in bulk and then write output somewhere else (i.e. ETL) - either will serve you fine. If you write complex queries with multiple filters and want to JOIN across multiple datasets - sure Spark can do that as well but it's not as efficient in pushing down computation to the source. e.g. A query like SELECT c.custkey, sum(totalprice) FROM orders o INNER JOIN customer c ON o.custkey = c.custkey WHERE o.orderstatus = 'O' GROUP BY c.custkey; when ran on Spark will pull both tables into memory and then perform the join + filter for orderstatus = 'O' and then compute the sum. While in case of Trino it'll push down the entire query into the remote database (in this case, in other queries it'll push down some parts of the query) so the source database will not need to return gigabytes of data over the network every time the query runs (and hence finish faster as well). Trino tries to push-down some operations to the remote system which can be done more efficiently there. e.g. filtering on a column that has an index in the remote RDBMS will be faster than pulling all data and then filtering in Trino. Spark doesn't have strong pushdown and has to pull most of the raw data and then apply processing on top of it. That's one of the main differences. Spark is a distributed job execution framework first while Trino is a distributed federated query engine first and it shows in their strengths and weaknesses. If you want to run arbitrary user defined transformations on data then Spark definitely has much more to offer than Trino.

simpligility 4y ago

Its great to see how far the project has come from the humble beginnings to the current, rich open source ecosystem and community.

bitsondatadev 4y ago

btw, if you want to know the backstory on why Presto is now called Trino, here's the article: https://trino.io/blog/2022/08/02/leaving-facebook-meta-best-for-trino.html https://trino.io/blog/2022/08/02/leaving-facebook-meta-best-...

georgewfraser 4y ago

The thing I wonder about with Presto and to a lesser extent Spark is, how many of their users adopted this tool because it was an easy migration path from Hive, and how many of those users will eventually re-platform to something else?

bitsondatadev 4y ago
I mean, the hive migration path is one thing. Now that Iceberg is taking over the old Hive model, data lakes are all the rage again. The other thing I would say is that Trino and Presto are not one-trick ponies or just hive replacements. There's also the ability to query across multiple systems that is, to me, the feature that future proofs a lot of architectures. It inherently frees you up to fiddle with your data in different systems but keep the access to that system in one location.
- georgewfraser 4y ago
  Yeah I think that is the key question: will data lakes become the dominant paradigm? There is certainly a lot of talk around them, though I see a ton of companies are still just going all in on a conventional data warehouse, but they tend not to talk about it because it’s not a new or interesting thing to do.
  - bitsondatadev 4y ago
    Yeah, though a lot of Fivetran customers are likely the type that would go all in on paying for a conventional data warehouse where people using open source stacks may be the ones that are using open ingestion alternatives. We see a pretty even mix from the Trino/Starburst lens. Bigger companies like to mix and match.

gavinray 4y ago

I recently had to write SQL query generation for AWS Athena, which is based off Presto 0.217 It turns out that the dialect doesn't support LATERAL joins with a LIMIT in them. The below query only works if you remove the LIMIT clause. https://i.stack.imgur.com/rdB1s.png https://i.stack.imgur.com/rdB1s.png This makes saying things like "Fetch all artists where ..., for each artist fetch their first 3 albums where ..., and for each album fetch the top 10 tracks where ..." really difficult Does Trino support this out of curiosity?

karma_fountain 4y ago
Is it possible to achieve this with a window function?
- gavinray 4y ago
  I found out it is! Kudos to this kind internet stranger for telling me: https://stackoverflow.com/a/73129836/13485494 https://stackoverflow.com/a/73129836/13485494 But man is it a huge PITA (especially when doing programmatic code generation of the SQL) compared to LATERAL joins Someone familiar with the CockroachDB query planner showed me that a window function like this is what Cockroach turns LATERAL joins into for instance: demo@127.0.0.1:26257/movr> explain select * from abc, lateral (select * from xyz where x = a limit 2); • filter │ estimated row count: 1 │ filter: row_num <= 2 │ └── • window │ estimated row count: 2 │ └── • hash join │ estimated row count: 2 │ equality: (x) = (a) │ ├── • scan │ estimated row count: 6 (100% of the table; stats collected 2 minutes ago) │ table: xyz@xyz_pkey │ spans: FULL SCAN │ └── • scan estimated row count: 1 (100% of the table; stats collected 3 minutes ago) table: abc@abc_pkey spans: FULL SCAN
bitsondatadev 4y ago
Check out this PR. I believe we may have tackled this one but you'd need to try it out on Trino: https://github.com/trinodb/trino/pull/1415 https://github.com/trinodb/trino/pull/1415

ck_one 4y ago

Can Trino be used as a Snowflake replacement? How is the query speed compared to Snowflake?

cpard 4y ago
Hey ck_one that's a hard question to answer and not get into "benchmarketing" territory. My suggestion is to try both under your own workloads and see the difference. Trino is also used by products like Athena (AWS) and Galaxy (Starburst) so if you want to play around and see how Trino performs without spending too much time on setting up clusters on your own, you can try these great products. Having said that, I'd like to add that building a performant distributed query engine is just hard. Trino has been in development for ten years and used by major companies in very demanding environments, these environments is where the technology has been defined and makes it what it is today and it is a proof of its performance and stability. (edited to add an important disclaimer that I work at Starburst)
simpligility 4y ago
Yes ... Starburst Enterprise, which is a commercial distribution of Trino, can in fact also query Snowflake, but also Delta Lake and many many other systems at the same time.
- dominotw 4y ago
  hey do you work for the company. Prbly add a disclaimer.
  - simpligility 4y ago
    Yes... where would I put the disclaimer?
chrsig 4y ago
i can't speak to trino, but with my experience with aws athena and snowflake, they're roughly on par with each other across the board.
4y ago

simpligility 4y ago

Also working on a new edition for Trino: The Definitive Guide at the moment.

dmead 4y ago

I do the support for my department's trino cluster. We move ~1tb (and growing) in ETL jobs and support interactive queries for the data scientists/analysts. It would be super good if you guys added big query write support. Its really annoying to have to run a hive cluster in google to act as a proxy for this.

atwebb 4y ago
Any chance you have an overview of the architecture and operations support required? How many data sources are you pinging?
- dmead 4y ago
  like 20 different sources. I nag my service reps about it. I"m sure it's been filed in your jira.
hashhar 4y ago
BigQuery very recently announced their Storage Write API which is one of the ways we were looking to implement this but there are some issues with the latency and consistency guarantees that it offers. But, yes, we do plan to add that eventually after ironing out all the kinks. See https://github.com/trinodb/trino/pull/13094 https://github.com/trinodb/trino/pull/13094
- bitsondatadev 4y ago
  Also, you can keep track of all the BQ progress here: https://github.com/trinodb/trino/issues/6867 https://github.com/trinodb/trino/issues/6867
  - dmead 4y ago
    thanks. If we can eliminate the costs and upkeep of hive in gcp, it would make my life easier for sure.

skadamat 4y ago

Big shout out to Brian Olsen from the Trino community (and Starburst) for helping the Trino community be successful - https://github.com/bitsondatadev https://github.com/bitsondatadev - https://www.linkedin.com/in/bitsondatadev/ https://www.linkedin.com/in/bitsondatadev/ I recommend the Trino Slack for people not already in it: https://trino.io/slack.html https://trino.io/slack.html

bitsondatadev 4y ago
Thanks for the shoutout! :) If you want to get started with Trino, here's a repo I created to do so: https://github.com/bitsondatadev/trino-getting-started https://github.com/bitsondatadev/trino-getting-started

simpligility 4y ago

Also just to note.. I am currently working on a refresh of Trino: The Definitive Guide .. and would love to see you all at Trino Summit in November. https://trino.io/blog/2022/06/30/trino-summit-call-for-speakers.html https://trino.io/blog/2022/06/30/trino-summit-call-for-speak...

jerryjerryjerry 4y ago

One of the features I'm interested in (or would like to have) from Trino or Presto is the workload management which can better manage different types of queries and allocate resources accordingly. This becomes important when more applications adopt Trino or Presto as a distributed SQL database/platform, where the impact from different queries or workloads can be mitigated, besides the dedicated resources (CPU, MEM, etc.) can be allocated to high priority workloads. I'm really wondering if/when such capabilities may be provided. BTW, purely curiosity, I compared Trino with Presto from OSS point of view (https://ossinsight.io/analyze/prestodb/presto?vs=trinodb%2Ftrino https://ossinsight.io/analyze/prestodb/presto?vs=trinodb%2Ft...), both communities are still popular but Trino seems more active than Presto now. I also wonder if two communities may reunion someday again to really boost its impact (comparing to Spark community).

bitsondatadev 4y ago
For managing difrerent workloads, check out this blogs and this videos from Shopify, Salesforce, Goldman Sachs, and Electronic Arts, respectively: - https://engineering.salesforce.com/how-to-etl-at-petabyte-scale-with-trino-5fe8ac134e36/ https://engineering.salesforce.com/how-to-etl-at-petabyte-sc... - https://shopify.engineering/faster-trino-query-execution-infrastructure https://shopify.engineering/faster-trino-query-execution-inf... - https://trino.io/episodes/33.html https://trino.io/episodes/33.html - https://www.youtube.com/watch?v=-5mlZGjt6H4 https://www.youtube.com/watch?v=-5mlZGjt6H4 All use the Lyft "Presto but really Trino"-Gateway project to run different clusters to handle various workloads. They go into various details for how this is achieved. https://github.com/lyft/presto-gateway https://github.com/lyft/presto-gateway Regarding the Trino/Presto split. I recommend looking at this blog to better understand why these two communities aren't mergeing. TL;DR Presto is a Facebook-driven project that mainly considers running on the Facebook infrastructure. Trino is community-driven that works on running well with all clouds and common infasturcture in the Trino community which is why you see a higher velocity there. https://trino.io/blog/2022/08/02/leaving-facebook-meta-best-for-trino.html https://trino.io/blog/2022/08/02/leaving-facebook-meta-best-... https://trino.io/blog/2020/12/27/announcing-trino.html https://trino.io/blog/2020/12/27/announcing-trino.html Soon we anticipate that Trino will become the common name in the community space but we'll always love the origins of the Trino project being Presto.

tzury 4y ago

Trino vs ClickHouse, can anyone tell from experience how those two compare?

bitsondatadev 4y ago
Clickhouse is a realtime system where Trino is a batch-oriented system. There are tradeoffs for doing realtime vs batch. Realtime is generally more expensive to run as you process every individual row as it comes, batch is when you can deal with minute latency and want to handle a lot of data in chunks. Trino is also a query engine rather than a database and it connects to many different systems: https://trino.io/docs/current/connector.html https://trino.io/docs/current/connector.html It also happens to connect to Clickhouse and it's very common that people will use Trino to query clickhouse realtime data and join it with data in big query, an object store data lake, or Snowflake: https://trino.io/docs/current/connector/clickhouse.html https://trino.io/docs/current/connector/clickhouse.html
deleted 4y ago
[deleted]
qoega 4y ago
You can consider that ClickHouse allows both to query a lot of supported external data sources(s3/hdfs/mysql/postgre/...) and to store data in pretty efficient columnar way with compression, indexes and all the bells and whistles. Native storage allows to use all the information about keys/indices to build query plan faster. With trino you can't store data inside trino. You can't even insert data using trino which allows you to solve scenarios like 'readonly analytics'. Trino allows you to use single query language for all the supported systems. So if you have a zoo of DBMS and object storages that you can just query it can help you to hide this complexity.

mrwnmonm 4y ago

Can you use Trino as a database proxy?

simpligility 4y ago
Essentially yes ... it can be the query engine for many databases at the same time.

kache_ 4y ago

trino is awesome can't believe this shit is free as in freedom

mrwnmonm 4y ago

We are building a SaaS BI tool. To enable the users to connect to their databases... we have a form that collects the database credentials from the user, saves it in a secure way, and when the user writes or uses an SQL query, we establish a database connection right away (from our server), execute it, and return the results, and we keep the connection alive for like 15mins. But with serverless architecture, first query could go to instance 1, so instance 1 will establish a db connection, then the second query could go to instance 2, so instance 2 will establish another one. You could end up with a lot of unnecessary connections. If you use AWS RDS (for yourself), beside lambda for example, AWS have RDS Proxy to solve this problem. So I was thinking about using Trino like the RDS Proxy, but for more databases, and for our customers database, not ours. Is that doable with Trino?