3 ms·

This is a special-purpose time-series data warehouse and a UI for querying it. If you are not Facebook, it is almost always better to do projects like this by u

This is a special-purpose time-series data warehouse and a UI for querying it. If you are not Facebook, it is almost always better to do projects like this by using a standard data warehouse like Redshift or BigQuery, a queue like Kinesis, and a BI tool like Looker or Tableau. Your data won't be quite as real-time and your queries won't be quite as fast, but it will take much less engineering effort and you'll be able to use the same tools for other projects.

spicyj 10y ago
I think you could build a Scuba-like UI over BigQuery that is simple to use and works great.
buremba 10y ago
Shameless plug: I'm the maintainer of Rakam project. You don't have to deal with all of these complexities. There are open-source projects that can setup that infrastructure for you. For example, we provide Cloudformation scripts that setup data analytics cluster for you, Kinesis, S3, PrestoDB and RESTFul API for collecting and querying data-sets so that you can use it similar to how you use a SaaS product. We also have an integrated visualization project that can connect your Rakam API and allow you to create dynamic reports and custom dashboards. You can run queries via user interface similar to Scube UI and also complex behavioral queries such as funnel and cohort queries as well as SQL. https://github.com/rakam-io/rakam https://github.com/rakam-io/rakam
_fsjdf_ 10y ago
Wouldn't InfluxDB be a better comparison?
- georgewfraser 10y ago
  The advantage of general-purpose data warehouses is they give you all of SQL, and they are compatible with BI tools. Time-series data is just one of the types of data you will want to analyze. It's best to choose tools that will work for all your data sources, even if these tools are suboptimal for time-series in particular.
siliconc0w 10y ago
Here is a good list of redshift 'gotchas' - https://github.com/open-guides/og-aws#redshift-gotchas-and-limitations https://github.com/open-guides/og-aws#redshift-gotchas-and-l... We ran into several of those. Notably, it's difficult to achieve the 'real-time' promise of redshift because of the huge performance hit while loading data into the DB so you have to do it off-hours. You can update a replica and then 'hot-swap' it in but this gets expensive. For operational analytics it's better to go with one of the purpose-built timeseries databases and dual write to that and your data warehouse.
- georgewfraser 10y ago
  The biggest gotcha listed there is how Redshift gets bogged down if you're loading a lot of tables, frequently. You can't run a production Redshift with lots of tables at <15m latency. But in most cases, Redshift is still an overall better choice than a timeseries database because: * It has all of SQL, including JOINs * You can use it for both timeseries data and all your other data.

spicyj 10y ago

I think you could build a Scuba-like UI over BigQuery that is simple to use and works great.

buremba 10y ago

Shameless plug: I'm the maintainer of Rakam project. You don't have to deal with all of these complexities. There are open-source projects that can setup that infrastructure for you. For example, we provide Cloudformation scripts that setup data analytics cluster for you, Kinesis, S3, PrestoDB and RESTFul API for collecting and querying data-sets so that you can use it similar to how you use a SaaS product. We also have an integrated visualization project that can connect your Rakam API and allow you to create dynamic reports and custom dashboards. You can run queries via user interface similar to Scube UI and also complex behavioral queries such as funnel and cohort queries as well as SQL. https://github.com/rakam-io/rakam https://github.com/rakam-io/rakam

_fsjdf_ 10y ago

Wouldn't InfluxDB be a better comparison?

georgewfraser 10y ago
The advantage of general-purpose data warehouses is they give you all of SQL, and they are compatible with BI tools. Time-series data is just one of the types of data you will want to analyze. It's best to choose tools that will work for all your data sources, even if these tools are suboptimal for time-series in particular.

siliconc0w 10y ago

Here is a good list of redshift 'gotchas' - https://github.com/open-guides/og-aws#redshift-gotchas-and-limitations https://github.com/open-guides/og-aws#redshift-gotchas-and-l... We ran into several of those. Notably, it's difficult to achieve the 'real-time' promise of redshift because of the huge performance hit while loading data into the DB so you have to do it off-hours. You can update a replica and then 'hot-swap' it in but this gets expensive. For operational analytics it's better to go with one of the purpose-built timeseries databases and dual write to that and your data warehouse.

georgewfraser 10y ago
The biggest gotcha listed there is how Redshift gets bogged down if you're loading a lot of tables, frequently. You can't run a production Redshift with lots of tables at <15m latency. But in most cases, Redshift is still an overall better choice than a timeseries database because: * It has all of SQL, including JOINs * You can use it for both timeseries data and all your other data.