5 ms·

I think there’s a huge unacknowledged gap in the database industry for good streaming products. If we are building a report or dashboard that we pull up a few

I think there’s a huge unacknowledged gap in the database industry for good streaming products. If we are building a report or dashboard that we pull up a few times a day then a pull based model where we query the database on page load is fine. For almost anything else such as an app, a microservice, an alerting system, a web page, a dashboard, we want to be able to update it in near real time for the user experience. Receiving a stream of query results is by far the easiest way to do this. Polling is obviously a poor interim solution. I think streaming will be a huge story in data over the next decade. The products are coming through now which is a start.

Already__Taken 4y ago
That's why I was watching rethinkDB but that didn't pan out.
tixocloud 4y ago
Would you not be able to solve this by having a messaging queue and getting the database to publish it’s updates? Curious what products you’ve seen as well.
- dagss 4y ago
  A message queue is a database too. So the question is: Why have two copies of your data, two products to learn and monitor and operate, write boilerplate to move data between the DBs, etc.? A "message queue" comes down to being another index on your table/set of tables ordered by a post-commit sequence number. These are things all SQL DBs have already, it just lacks a bit of exposing/packaging to be as convenient to use as a messaging queue.
lmm 4y ago
Streaming has been the story for the past couple of decades already. But once you accept the value of an event-first architecture, the benefits of traditional RDBMSes become much weaker.
- benjaminwootton 4y ago
  It’s been discussed for a long time, but the ecosystem is still spotty. Many databases have some kind of “live query” feature for instance but with limitations or not intended for production use. A lot of work has taken place in the Kafka, Flink, DataFlow ecosystem but that still leaves a lot of work for the developer over a simple subscribe to query results. I do think a lot of work has been done, but it all needs to move up a few levels of abstraction.
  - lmm 4y ago
    > A lot of work has taken place in the Kafka, Flink, DataFlow ecosystem but that still leaves a lot of work for the developer over a simple subscribe to query results. Personally I find it much easier to write the code explicitly than try to understand what a query planner is doing, especially if performance is relevant. (That's not to say there's not plenty of room for improvement in the streaming world - but I'd rather have a helper library that I can use on top of the low-level API, than have to go through a parser and planner for every query even when I know exactly what I want to do) But I seem to be an outlier in this regard.
  - tango12 4y ago
    Yep. Exactly this. Our focus (Hasura) is on the last mile so that innovations on the data side (eg: materialize, ksql, timescale continuous aggregates) are “just obvious” to start building applications and services against.
- cultofmetatron 4y ago
  maybe if you're a really big company with petabytes of data but I wouldn't' say its worth the operational complexity for 90% of tech companies out there. Seriously, design your schema so you can have tables optimized for aggregate functions. you can use materialized views or have them populate via triggers. Either is still going to be an order of magnitude less work that dealing with all the edge cases that are ging to come up when you try to build a distributed data pipeline. Thats a death by thousand cuts for any small to medium startup that doesn't already have a team of engineers experienced in making such systems work. Plus you can stream db changes from postgres to kafka for those edge cases where you really need it TLDR: if you're in a startup and thinking of building a distributed system.. DONT. stick with a monolith and spin out services as NEEDED.
thom 4y ago
Materialize gives a vaguely PostgreSQLish user land on top of this idea. Obviously Postgres has LISTEN/NOTIFY which are fine for most places you need a queue and push updates but the key is incremental view updates so you get instant results on complex queries.
- insanitybit 4y ago
  Unfortunately, Materialize has a pretty restrictive license. https://github.com/MaterializeInc/materialize/blob/main/LICENSE https://github.com/MaterializeInc/materialize/blob/main/LICE... I'm generally pretty happy to pay for open source software, but licensing like this is just too risky. I need to be able to experiment with something at scale, in production, before I start paying someone.
  - kanwisher 4y ago
    Free for 1 node, paid after. Code reverts to permissive license after 4 years. Think this is fair. you can bootstrap for free and only pay if it works out
    - deleted 4y ago
      [deleted]
    - insanitybit 4y ago
      It's more than fair, of course. It's just a barrier.
      - zinclozenge 4y ago
        There's https://www.risingwave.dev https://www.risingwave.dev which is similar to materialize.
gunnarmorling 4y ago
I wouldn't quite say that this need is unacknowledged, most DBs have some kind of streaming interface for actual data contents (not queries, though), which for instance are leveraged by Debezium to expose change event streams in one uniform event format (disclaimer: I work on Debezium). Going foward, I expect that these capabilities will be substantially built out, also providing streaming query support (incrementally updated materialized views), not being locked in separate licensing layers or a more or less complex to use after-thought, but a primary part of database offerings, just as SELECT today.

Already__Taken 4y ago

That's why I was watching rethinkDB but that didn't pan out.

tixocloud 4y ago

Would you not be able to solve this by having a messaging queue and getting the database to publish it’s updates? Curious what products you’ve seen as well.

dagss 4y ago
A message queue is a database too. So the question is: Why have two copies of your data, two products to learn and monitor and operate, write boilerplate to move data between the DBs, etc.? A "message queue" comes down to being another index on your table/set of tables ordered by a post-commit sequence number. These are things all SQL DBs have already, it just lacks a bit of exposing/packaging to be as convenient to use as a messaging queue.

lmm 4y ago

Streaming has been the story for the past couple of decades already. But once you accept the value of an event-first architecture, the benefits of traditional RDBMSes become much weaker.

benjaminwootton 4y ago
It’s been discussed for a long time, but the ecosystem is still spotty. Many databases have some kind of “live query” feature for instance but with limitations or not intended for production use. A lot of work has taken place in the Kafka, Flink, DataFlow ecosystem but that still leaves a lot of work for the developer over a simple subscribe to query results. I do think a lot of work has been done, but it all needs to move up a few levels of abstraction.
- lmm 4y ago
  > A lot of work has taken place in the Kafka, Flink, DataFlow ecosystem but that still leaves a lot of work for the developer over a simple subscribe to query results. Personally I find it much easier to write the code explicitly than try to understand what a query planner is doing, especially if performance is relevant. (That's not to say there's not plenty of room for improvement in the streaming world - but I'd rather have a helper library that I can use on top of the low-level API, than have to go through a parser and planner for every query even when I know exactly what I want to do) But I seem to be an outlier in this regard.
- tango12 4y ago
  Yep. Exactly this. Our focus (Hasura) is on the last mile so that innovations on the data side (eg: materialize, ksql, timescale continuous aggregates) are “just obvious” to start building applications and services against.
cultofmetatron 4y ago
maybe if you're a really big company with petabytes of data but I wouldn't' say its worth the operational complexity for 90% of tech companies out there. Seriously, design your schema so you can have tables optimized for aggregate functions. you can use materialized views or have them populate via triggers. Either is still going to be an order of magnitude less work that dealing with all the edge cases that are ging to come up when you try to build a distributed data pipeline. Thats a death by thousand cuts for any small to medium startup that doesn't already have a team of engineers experienced in making such systems work. Plus you can stream db changes from postgres to kafka for those edge cases where you really need it TLDR: if you're in a startup and thinking of building a distributed system.. DONT. stick with a monolith and spin out services as NEEDED.

thom 4y ago

Materialize gives a vaguely PostgreSQLish user land on top of this idea. Obviously Postgres has LISTEN/NOTIFY which are fine for most places you need a queue and push updates but the key is incremental view updates so you get instant results on complex queries.

insanitybit 4y ago
Unfortunately, Materialize has a pretty restrictive license. https://github.com/MaterializeInc/materialize/blob/main/LICENSE https://github.com/MaterializeInc/materialize/blob/main/LICE... I'm generally pretty happy to pay for open source software, but licensing like this is just too risky. I need to be able to experiment with something at scale, in production, before I start paying someone.
- kanwisher 4y ago
  Free for 1 node, paid after. Code reverts to permissive license after 4 years. Think this is fair. you can bootstrap for free and only pay if it works out
  - deleted 4y ago
    [deleted]
  - insanitybit 4y ago
    It's more than fair, of course. It's just a barrier.
    - zinclozenge 4y ago
      There's https://www.risingwave.dev https://www.risingwave.dev which is similar to materialize.

gunnarmorling 4y ago

I wouldn't quite say that this need is unacknowledged, most DBs have some kind of streaming interface for actual data contents (not queries, though), which for instance are leveraged by Debezium to expose change event streams in one uniform event format (disclaimer: I work on Debezium). Going foward, I expect that these capabilities will be substantially built out, also providing streaming query support (incrementally updated materialized views), not being locked in separate licensing layers or a more or less complex to use after-thought, but a primary part of database offerings, just as SELECT today.