4 ms·
TLDR: This depends on your requirements. We find that users with similar requirements follow one of two approaches: (1) You keep raw data for a certain time pe
by ozgune 10y ago
TLDR: This depends on your requirements. We find that users with similar requirements follow one of two approaches:
(1) You keep raw data for a certain time period and regularly roll it up: https://www.citusdata.com/blog/2016/11/29/event-aggregation-at-scale-with-postgresql https://www.citusdata.com/blog/2016/11/29/event-aggregation-...
(2) Citus + cstore_fdw if you're cool with doing some manual work. In particular, our customers who follow this approach shard on one column (automatic) and then manually partition by time. We're looking to leverage upcoming partitioning improvements in Postgres 10 and further automate this setup: https://github.com/citusdata/citus/issues/183 https://github.com/citusdata/citus/issues/183
For more on Citus + cstore, this recent HN comment could also be helpful: https://news.ycombinator.com/item?id=14039001 https://news.ycombinator.com/item?id=14039001
Happy to chat more if you drop us a line. How much data did you have in mind? Are you looking to power a B2B or B2C application?
- lima 10y agoInternal analytics, may accidentally become a B2B at some point (who knows), but for now, non business-critical with no availability requirements. About 30+ GB per day, but that's uncompressed and unprocessed. Thanks for the pointers! Manual sharding sounds like an option.