3 ms·

Out of genuine interest and perhaps even a need for such a solution, is there anything that does what Metabase/Superset/Dash/GDS/ModeAnalytics does but for real

Out of genuine interest and perhaps even a need for such a solution, is there anything that does what Metabase/Superset/Dash/GDS/ModeAnalytics does but for realtime datastreams? For instance, parsing and recording and visualizing the events coming from a websockets or some event queue/bus?

oarabbus_ 7y ago
I believe you'd want a tool such as Grafana (which I believe is free/open source), which I have seen eng teams implement for realtime streaming. There is also Kibana which I am less familiar with.
manigandham 7y ago
Those all work for that already, because they're just frontends to whatever backend system you're using. You a database like BigQuery with real-time streaming inserts and rerun your queries for results. If you want analytics directly on the stream, then there are plugins available to support reading the query results of something like Kinesis Analytics or Confluent's KSQL.
oarabbus_ 7y ago
Ah yes like the other user mentioned AWS Kinesis Firehose and GCP Pub/Sub may fit the bill as well Grafana/Kibana are the actual charting tools
infinite8s 7y ago
Yes perspective is built for streaming data - https://perspective.finos.org https://perspective.finos.org. Open source streaming pivoting engine that operates in the browser (using wasm in a webworker, with integration with Apache arrow for ingest of binary data strrams off websockets)
alexis2b 7y ago
You might want to check Striim - it wraps kafka + streaming transformers with a scripting language, and a visual pipeline design tool. It then offers some really nice dashboards with a real-time “feel”. Not affiliated in any way but I took it for a spin and was happy to see how much could get done with it in just a few hours and without prior knowledge.
muse1489 7y ago
If you familiar with Druid then Check out Metatron Discovery( https://metatron.app https://metatron.app) Druid is strong for data streams.

oarabbus_ 7y ago

I believe you'd want a tool such as Grafana (which I believe is free/open source), which I have seen eng teams implement for realtime streaming. There is also Kibana which I am less familiar with.

manigandham 7y ago

Those all work for that already, because they're just frontends to whatever backend system you're using. You a database like BigQuery with real-time streaming inserts and rerun your queries for results. If you want analytics directly on the stream, then there are plugins available to support reading the query results of something like Kinesis Analytics or Confluent's KSQL.

oarabbus_ 7y ago

Ah yes like the other user mentioned AWS Kinesis Firehose and GCP Pub/Sub may fit the bill as well Grafana/Kibana are the actual charting tools

infinite8s 7y ago

Yes perspective is built for streaming data - https://perspective.finos.org https://perspective.finos.org. Open source streaming pivoting engine that operates in the browser (using wasm in a webworker, with integration with Apache arrow for ingest of binary data strrams off websockets)

alexis2b 7y ago

You might want to check Striim - it wraps kafka + streaming transformers with a scripting language, and a visual pipeline design tool. It then offers some really nice dashboards with a real-time “feel”. Not affiliated in any way but I took it for a spin and was happy to see how much could get done with it in just a few hours and without prior knowledge.

muse1489 7y ago

If you familiar with Druid then Check out Metatron Discovery( https://metatron.app https://metatron.app) Druid is strong for data streams.