6 ms·
Building a Scalable Postgres Metrics Backend Using the Citus Extension
- kornish 10y agoFor anyone with questions about how to properly shard a table, how to fit a problem to a distribution strategy, or general questions about deployment, Citus has a Slack channel which is active and extremely welcoming. You can request an invite here: https://slack.citusdata.com/ https://slack.citusdata.com/ Disclaimer: don't work there, but on the Slack and the Citus team is nothing short of awesome.
- doh 10y agoWe[0] just recently moved from Hbase to Postgres w/ Citus. Still not fully settled in, but got incredible support from the Citus team (seriously amazing guys) and things are looking much brighter than before. On a 8 worker nodes (16 cores, 80GB ram, 3TB SSD), we're at around 20k inserts/second, while most of the selects are running at around 100ms (besides some counts and complex queries). If you have a lot of data and know Postgres, give it a chance. [0] https://pexe.so https://pexe.so
- Jake232 10y agoThis is a super cool product. Good job.
- locusm 10y agoInteresting product - good luck :)
- koolba 10y ago> On a 8 worker nodes (16 cores, 80GB ram, 3TB SSD), we're at around 20k inserts/second, while most of the selects are running at around 100ms (besides some counts and complex queries). Is that 20K separate transactions or are some of the 20K inserts grouped together?
- doh 10y agoSeparate (going through the master).
- devbug 10y agoIf you haven't already seen it, Dan from Heap Analytics has a great talk about how they use Postgres and Citus. https://www.youtube.com/watch?v=NVl9_6J1G60 https://www.youtube.com/watch?v=NVl9_6J1G60
- sciurus 10y agoIf you're looking for a metrics backend built on traditional Postgres, check out Tgres. http://grisha.org/blog/2016/07/29/state-of-tgres-2016/ http://grisha.org/blog/2016/07/29/state-of-tgres-2016/
- GrayTShirt 10y ago+1 for tgres https://github.com/tgres/tgres https://github.com/tgres/tgres On an aside, I found his posts on holt-winters very informative.
- dataloopio 10y agoI had a look at Citus when evaluating time series databases and the documents said that work was still under way on masterless setup for faster ingestion (up to 500k metrics /sec). https://docs.citusdata.com/en/v5.2/performance/scaling_data_ingestion.html#masterless-citus-50k-s-500k-s https://docs.citusdata.com/en/v5.2/performance/scaling_data_... If that has changed I'd add it to my table. https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCEWdPtAoccJ4a-IuZv4fXDHxM/edit#gid=0 https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCE...
- dataloopio 10y agoI had a look at Citus when evaluating time series databases and the documents said that work was still under way on masterless setup for faster ingestion (up to 500k metrics /sec). https://docs.citusdata.com/en/v5.2/performance/scaling_data_ingestion.html#masterless-citus-50k-s-500k-s https://docs.citusdata.com/en/v5.2/performance/scaling_data_... If that has changed I'd add it to my table. https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCEWdPtAoccJ4a-IuZv4fXDHxM/edit#gid=0 https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCE...