5 ms·

Show HN: Epsio – Incremental views for your existing database

24 points by ntur1337 3y ago

akrymski 3y ago
Fantastic. I've always wondered why Postgres doesn't have this built in. Wish it was open source.
- dkgs1998 3y ago
  Hi, I am one of the authors of the project. Thanks! One thing that became clear to us while working on this project is the significant difference in how incremental views (or stream processing) store and access data, compared to how PostgreSQL looks at and processes data (batch processing). So, it made a lot of sense for us to separate the technologies (i.e., have a transformation layer, and a "serving" layer), but still try to integrate the two as seamlessly as possible.
dkgs1998 3y ago
Hi, one of the project's authors here. Would love to answer any questions you have :)
gkapur 3y ago
Cool product! A few questions mostly out of curiosity: - What underlying data flow technology is this based on? Is it Timely dataflow a la Materialize, something else a la RisingWave, etc.? - What can the materialized views not have? Ie. Does it support window functions? Unions? User-defined functions? - Finally I’m curious about how you think of the landscape versus RisingWave, Readyset, Materialize, etc.!
- ntur1337 3y ago
  Hi! - Actually, Epsio has its own dataflow engine. It's based on much of the same theory as these other technologies, we just found that their implementation doesn't quite fit our use case neatly. Just as an example, Timely/Differential dataflow requires saving state in memory. While this results in extremely low latency (beneficial for its original purpose), the trade-off is the high memory footprint required to save state for complex SQL queries (JOINS, DISTINCT ON, etc.). We found this to be quite costly for the use case of replacing heavy SQL queries in PG/MySQL. We actually recently published a blog post on how we see the streaming landscape -https://www.epsio.io/blog/epsios-diff-in-the-streaming-landscape https://www.epsio.io/blog/epsios-diff-in-the-streaming-lands... , welcome to take a look for our full thoughts :) - Regarding SQL support, we support today DISTINCT [ON], UNIONs, JOINs, CTEs, ORDER BYs, GROUP BYs, and some additional operators/functions (full list is in this link: https://docs.epsio.io/sql-support/queries/ https://docs.epsio.io/sql-support/queries/) Window functions are supposed to be released soon :)
warpspin 3y ago
Product looks very very great, based on browsing the docs. Pricing is unfortunately very opaque, at least if you have not signed up yet, naming some kind of starting price "depending on instance size" and I have no clue depending on what exactly, especially if I plan to deploy on-premises. I so much hate "contact us" quotes. Also a big no-go you want to fix is after clicking around on the page, I still have no clue what or who Epsio is. There's no business address, no info on company etc. Judging by the "website terms of use" I finally found an address, though, but that was really really hard. Also, still no clue if these "epsio website" terms of use are supposed to govern my usage of the product or only me browsing the website or whether the entity named in the "website" terms of use will be the one providing the software to me at all. Seriously. As great as your product is, it basically has access to the most sensitive thing any company has - its databases - so you have to be a tad bit more forthcoming with building trust.

24 points by ntur1337 3y ago

akrymski 3y ago

Fantastic. I've always wondered why Postgres doesn't have this built in. Wish it was open source.

dkgs1998 3y ago
Hi, I am one of the authors of the project. Thanks! One thing that became clear to us while working on this project is the significant difference in how incremental views (or stream processing) store and access data, compared to how PostgreSQL looks at and processes data (batch processing). So, it made a lot of sense for us to separate the technologies (i.e., have a transformation layer, and a "serving" layer), but still try to integrate the two as seamlessly as possible.

dkgs1998 3y ago

Hi, one of the project's authors here. Would love to answer any questions you have :)

gkapur 3y ago

Cool product! A few questions mostly out of curiosity: - What underlying data flow technology is this based on? Is it Timely dataflow a la Materialize, something else a la RisingWave, etc.? - What can the materialized views not have? Ie. Does it support window functions? Unions? User-defined functions? - Finally I’m curious about how you think of the landscape versus RisingWave, Readyset, Materialize, etc.!

ntur1337 3y ago
Hi! - Actually, Epsio has its own dataflow engine. It's based on much of the same theory as these other technologies, we just found that their implementation doesn't quite fit our use case neatly. Just as an example, Timely/Differential dataflow requires saving state in memory. While this results in extremely low latency (beneficial for its original purpose), the trade-off is the high memory footprint required to save state for complex SQL queries (JOINS, DISTINCT ON, etc.). We found this to be quite costly for the use case of replacing heavy SQL queries in PG/MySQL. We actually recently published a blog post on how we see the streaming landscape -https://www.epsio.io/blog/epsios-diff-in-the-streaming-landscape https://www.epsio.io/blog/epsios-diff-in-the-streaming-lands... , welcome to take a look for our full thoughts :) - Regarding SQL support, we support today DISTINCT [ON], UNIONs, JOINs, CTEs, ORDER BYs, GROUP BYs, and some additional operators/functions (full list is in this link: https://docs.epsio.io/sql-support/queries/ https://docs.epsio.io/sql-support/queries/) Window functions are supposed to be released soon :)

warpspin 3y ago

Product looks very very great, based on browsing the docs. Pricing is unfortunately very opaque, at least if you have not signed up yet, naming some kind of starting price "depending on instance size" and I have no clue depending on what exactly, especially if I plan to deploy on-premises. I so much hate "contact us" quotes. Also a big no-go you want to fix is after clicking around on the page, I still have no clue what or who Epsio is. There's no business address, no info on company etc. Judging by the "website terms of use" I finally found an address, though, but that was really really hard. Also, still no clue if these "epsio website" terms of use are supposed to govern my usage of the product or only me browsing the website or whether the entity named in the "website" terms of use will be the one providing the software to me at all. Seriously. As great as your product is, it basically has access to the most sensitive thing any company has - its databases - so you have to be a tad bit more forthcoming with building trust.