10 ms·
This is an overly complex solution that we were able to resolve using a simple VPS running Clickhouse as backend and Grafana for frontend. Our production db is
by pupdogg 6y ago
This is an overly complex solution that we were able to resolve using a simple VPS running Clickhouse as backend and Grafana for frontend. Our production db is an Aurora MySQL instance and we keep it lean by performing daily dumps of reporting related data into a CSV with gzip compression -> push it to S3 -> convert it to parquet file format using AWS glue -> bring it into ClickHouse. Data size for these specific reports is approx 100k rows daily and is partitioned by MONTH/YEAR. Overall cost: $20/month VPS and approx. $15/month in AWS billing.
- ekianjo 6y agoYou do not even need to use AWS. use Minio as S3 compatible system and Nifi to convert files to parquet once they land in Minio... No dependency on AWS.
- antman 6y agoMinio and nifi, require lots of resources for themselves. Better off using pure python and if one wants something lighweight and visually pleasing Mara [0] or Dagster with Dagit [1] will do the job [0] https://github.com/mara/mara-pipelines https://github.com/mara/mara-pipelines [1] https://docs.dagster.io/tutorial/execute https://docs.dagster.io/tutorial/execute
- ekianjo 6y agoYes, Minio and Nifi are not lightweight but if you look for robustness they have been used in large environments and proven to scale.
- twotwotwo 6y agoAppreciate the info on real-world use here. A data warehouse would be sort of interesting at work but is not urgently needed (because reporting from MySQL works, without the nifty speedups data warehouses can achieve), and we're somewhere between the size you're talking about and the really-big-data use cases I tend to see blogged about more often. Am curious about ClickHouse and a lower-cost deployment might make it worthwhile when it wouldn't be otherwise.