5 ms·

PipelineDB is great..and the guys are super helpful.. influxdb has something similar, and now so does AWS Kinesis (with their Kinesis Analytics product).

Fergi 9y ago
PipelineDB will become a standard, open-source PostgreSQL extension this year, enabling anybody using PostgreSQL to leverage continuous SQL queries in combination with all other PostgreSQL functionality. https://www.pipelinedb.com/blog/pipelinedb-0-9-8-postgresql-extension-refactor-nearing-completion https://www.pipelinedb.com/blog/pipelinedb-0-9-8-postgresql-...
- _pmf_ 9y ago
  > PipelineDB will become a standard, open-source PostgreSQL extension What does this imply?
cbsmith 9y ago
Are InfluxDB's continuous queries really like this? It's so much less of a streaming solution I hadn't realized it did this.
stuartaxelowen 9y ago
Doesn't PipelinDB limit you to postgres' scaling capabilities?
- Fergi 9y ago
  Not for streaming analytic workloads, because PipelineDB fundamentally adds continuous queries to PostgreSQL, so data is continuously distilled and aggregated as it arrives, before it is stored, which drastically reduces the amount of data stored in PipelineDB (or soon, in PostgreSQL via the extension refactor). PipelineDB also offers a clustering extension for large workloads (see: http://enterprise.pipelinedb.com/docs/ http://enterprise.pipelinedb.com/docs/) But in terms of ad hoc, exploratory analytics workloads, yes - the scaling limitations would be the same, since for ad hoc, exploratory analytics PipelineDB and PostgreSQL are the same. But with that said, the processed, aggregated data that gets stored is generally much smaller than large volumes of granular data, so there is much less data to comb through with PipelineDB.
  - stuartaxelowen 9y ago
    Thanks for explaining!
  - sandGorgon 9y ago
    this is awesome! will it work on Amazon RDS for postgresql ?

Fergi 9y ago

PipelineDB will become a standard, open-source PostgreSQL extension this year, enabling anybody using PostgreSQL to leverage continuous SQL queries in combination with all other PostgreSQL functionality. https://www.pipelinedb.com/blog/pipelinedb-0-9-8-postgresql-extension-refactor-nearing-completion https://www.pipelinedb.com/blog/pipelinedb-0-9-8-postgresql-...

_pmf_ 9y ago
> PipelineDB will become a standard, open-source PostgreSQL extension What does this imply?

cbsmith 9y ago

Are InfluxDB's continuous queries really like this? It's so much less of a streaming solution I hadn't realized it did this.

stuartaxelowen 9y ago

Doesn't PipelinDB limit you to postgres' scaling capabilities?

Fergi 9y ago
Not for streaming analytic workloads, because PipelineDB fundamentally adds continuous queries to PostgreSQL, so data is continuously distilled and aggregated as it arrives, before it is stored, which drastically reduces the amount of data stored in PipelineDB (or soon, in PostgreSQL via the extension refactor). PipelineDB also offers a clustering extension for large workloads (see: http://enterprise.pipelinedb.com/docs/ http://enterprise.pipelinedb.com/docs/) But in terms of ad hoc, exploratory analytics workloads, yes - the scaling limitations would be the same, since for ad hoc, exploratory analytics PipelineDB and PostgreSQL are the same. But with that said, the processed, aggregated data that gets stored is generally much smaller than large volumes of granular data, so there is much less data to comb through with PipelineDB.
- stuartaxelowen 9y ago
  Thanks for explaining!
- sandGorgon 9y ago
  this is awesome! will it work on Amazon RDS for postgresql ?