4 ms·
Ultimately it depends on your use case and how much tuning/schema design/explicit management you are willing to do. My team lead did a talk at NGINX Conf 2017
by gshulegaard 9y ago
Ultimately it depends on your use case and how much tuning/schema design/explicit management you are willing to do.
My team lead did a talk at NGINX Conf 2017 which touched on how we tuned Postgres to handle 1 Billion metrics per day for NGINX Amplify (https://www.nginx.com/products/nginx-amplify/ https://www.nginx.com/products/nginx-amplify/). I would link it, but it seems it hasn't been published by our Conf team yet.
We did this on 9.4 so the changes outlined here regarding native partitioning, parallel queries, replication improvements, etc. would likely make doing a similar scaling structure easier and just straight up more performant.
I wouldn't classify 1 Billion metrics a day as "Big Data". I also would point out that our use case is unfairly biased to "recent" data rather than "all historical" data...which allows us to make some query optimizations that wouldn't be acceptable in the latter use case.
But we are using Postgres to host a constant, write-heavy workload that conventional wisdom dictates it is unsuited for. Moreover, it was/is fairly simple to implement, understand, and scale indefinitely. Personally, I think both Redshift and CitusData are various degrees of proof that scaling Postgres to handle these types of workloads is possible.
For me, the reason you want to consider using a Big Data focused tool (such as Cassandra) is at some point you end up fighting your tool to make a square peg fit in a round hole. At some point it makes more sense to adopt a specialized tool rather than work to specialize your implementation of a specific tool.
In all fairness, we are considering just that. Our road map actually has Scylla in the near future. It's not that we don't think Postgres could handle the load, but rather we think it will be easier to move specific data to Scylla and in the long run will save us resources (primarily in man hours and potentially in compute requirements as well).
Even if we move all our timeseries data to a specialized tool, we won't be getting rid of Postgres. For most/if not all our other persistence requirements Postgres is more than adequate. This is one of the reasons we chose to start with Postgres: even once we hit a scale that specialized tools made sense, Postgres would still be useful for everything else that doesn't require a specialized tool.
Moral of the story: Postgres will get you a lot farther than many give it credit for...and if you are just starting out I would highly encourage you to just pick Postgres and see how far it can take you. If you hit the limit, replace that specific part.
Edit: Also, there appears to be some movement on Columnar indexes which is worth keeping an eye on:
https://blog.2ndquadrant.com/postgresql-10-roadmap/ https://blog.2ndquadrant.com/postgresql-10-roadmap/
- spullara 9y agoCitusData yes but RedShift is the Paraccel backend and just uses the frontend of postgresql.
- gshulegaard 9y agoHmmm...I thought ParAccel was a derivative of Postgres: https://wiki.postgresql.org/wiki/PostgreSQL_derived_databases https://wiki.postgresql.org/wiki/PostgreSQL_derived_database...