6 ms·

Another way of asking: Citus seems amazing. What's the worst part of using it (other than cost)?

by rattray 10y ago

agentgt 10y ago
I only have about a month experience but the real issue is you have to do a lot of things manually that are done automatically in other systems like Cassandra. Particularly rollups and stream like things. That is you have to do the rollups (as well stream like things) either through postgres rules,triggers Listen/Notify, cron etc. With system like Druid (Cassandra), Elastic Search and other time series databases this is not needed (either the query ability is efficient enough because of compressed or it is automatically indexed or both). For the stream like stuff we have been using Pipelinedb. Pipelinedb is annoying though in that it is a fork and not an extension. Pipelinedb has a similar problem. You have to create the streams apriori. So both for Citus and Pipelinedb you have to do some planning of your schema where as the other guys you can do more exploratory/adhoc analysis and queries. The disadvantage to the NoSQL guys (besides maturity and literally no SQL) is that you pay a lot more in memory as most of them (IMO cheat) and rely heavily on available memory. Postgres is very memory friendly.
- rattray 10y ago
  Hmm, interesting. Sounds like a problem with a Postgres, not Citus, but it still sounds like a disadvantage compared to competing products (RethinkDB also comes to mind). I haven't implemented anything for that before personally; would a tool like Bottled Water [0] be helpful for this usecase? [0] https://github.com/confluentinc/bottledwater-pg https://github.com/confluentinc/bottledwater-pg
  - agentgt 10y ago
    Yes Postgres has disadvantages (I accidentally posted my comment too soon so that might have caused some confusion) but like I said it is: 1. Mature, 2. SQL is damn powerful, 3. Memory friendly (aka cost). RethinkDB is cool. I just never got into it to know how well it actually works. As for bottledwater we use Postgres + AMQP aka Rabbit + Listen/Notify. I'm not really a big fan of kafka even though it does have amazing scaling ability.
- geoka9 10y ago
  Have you used PostgreSQL before using Citus? If yes, could you share any pain points compared to plain PG (joins, write performance, lack of full SQL support? anything that comes to mind).
- grammr 10y ago
  > Pipelinedb is annoying though in that it is a fork and not an extension. Hi, I'm one of the PipelineDB co-founders--thanks for using our product! Making PipelineDB an extension is the most consistent piece of feedback we've received from our users, and I promise we're listening: PipelineDB 1.0 will be a standard PostgreSQL extension, incrementally rolled out via versions 0.9.8, 0.9.9, and 1.0.0.
- merb 10y ago
  well sadly Listen/Notify on PG does not work with streaming replication. That would make it really really great tough.
- manigandham 10y ago
  Memory is there to be used. 99% CPU and RAM usage is perfect utilization.
jonathanpdenney 10y ago
We've been running on Citus for about 2 months now, with a saas app that has lots of reporting functionality. It's no different than standard postgres in terms of configuration, other than that you have to define a distribution column for the tables you want to be distributed into tenants (e.g. accounts for saas). That's how you really see a significant benefit when working with larger datasets. That and the extra memory they provide over other managed postgres services. For us, "distributing our tables" meant just adding a rails migration using their gem. We were upgrading from heroku postgres, and chose this over RDS, because they take care of the distribution of data into tenants without us having to change our logic, since everything our users create was already getting associated to their organization / account. Only issue we've had was at one point we hit our plan's database connection limit of 300 for a few minutes, because of a new long running query we added, that started locking up connections and causing timeouts as soon as we went to production. We reached out immediately and their team jumped all over it to help us free up those connections, while we fixed the query.
- rattray 10y ago
  Thanks, that's super helpful!
websymphony 10y ago
They also have a community edition, which also seems pretty powerful. https://www.citusdata.com/product/community https://www.citusdata.com/product/community So cost is maybe not a big drawback either.
doh 10y ago
We're using Citus for almost a year, 8 months in production. Currently running 22 workers [32 core, 3TB space, 120GB RAM] containing almost a trillion (yep) rows in one single table. Citus has a lot of quirks and one could write a book about it. But for shortness here is just couple of the top things you should know. - Citus is not PG. It runs on PG, it looks like PG but it doesn't behave like PG. By using Citus you lose majority of advance strategies, especially ones that manipulate data (UPDATE from SELECT, DELETE, etc.). In very short, you have to forget about everything you know about PG. This means that you have to rethink your table schemas, distribution strategies, etc. This is common in every distributed database (No/SQL). - Citus deployed in a cloud is a bit sensitive. As the master node keeps track of each worker in almost realtime fashion, any breaks in communication results in corrupted shards. If you don't use just one user (postgres) but have different rights for different users, fixing the shards can be a pain (if you don't pay for the enterprise version, which provides you with helpful scripts). - Adding new worker is easy, but rebalancing them to spread the shards evenly is not (again different for enterprise version). - PG's planner can be a bit stubborn while using citus. It sometimes behaves differently from the native PG. The result is that you have to rethink you strategies and rewrite queries up until you don't get the desired result. - Writing functions and automatic triggers can be challenging. The usefulness of them is determined by how well you designed you original tables. Once the tables are colocated and sharded on the same column you get impressive results. Writing the functions can be challenging though as you have to deploy the function to each table. They offer a script to help you with it https://github.com/citusdata/citus/blob/a2f6b29a6d4537b626f8cb8c30525a95c8e7ffe9/src/backend/distributed/master/citus_tools.sql https://github.com/citusdata/citus/blob/a2f6b29a6d4537b626f8... - You do want to consider how many shards you need. More automatically means that Citus gets slower, but you want to be able to grow so chose more than you have workers. There is much more, but those are the things that you should consider. As for us, we learned to live with the quirks and now we're able to run aggregation queries through billions of rows in under 600ms. I would suggest to consider Citus only if you're considering something like Cassandra or Hbase.
- jpgvm 10y ago
  Are you using the commercial distribution or the OSS one? If the OSS release how have you implemented rebalancing? It seems to me the APIs are there in the extension, just user defined functions are missing to actually do the rebalance. I have been looking at integrating Citus into Flynn as an option for more scale-out workloads but needing non-OSS features to recover from workers going missing or adding new workers put me off implementing it for now.
ozgune 10y ago
(Ozgun from Citus Data) I think the answer to your question depends on the workload you're planning to use Citus for. We find that users rely on Postgres for different use cases and Citus' benefits (and worst parts :)) vary across them. Common use cases, ordered by popularity & benefits, include: 1. Multi-tenant database: Most B2B applications already have the notion of a tenant / customer built into their data model. Citus allows you to scale your transactional relational database to 100K+ tenants with minimal changes to your application -- and with all the benefits of Postgres. 2. Operational analytics: Citus enables ingesting large volumes of data and running analytical queries on that data in human real-time. Example applications include analytic dashboards with sub-second response times. 3. NoSQL++: As we increased Citus' feature set, we found that users started relying on it for workloads that have high read / write throughputs. We have a few "hidden" features in v6.1 that make Citus even more powerful here. That said, sometimes the open source community is way ahead of us on this. :) 4. Other: These include anything from distributed transactions to real-time notifications. Since we extend Postgres, if you're a power user and understand Citus' underlying primitives, sky's the limit. (We debated adding a section to our documentation for other workloads. We just couldn't agree if we wanted to name it Top Gun or Highway to the Danger Zone.) Our goal in each release is to add new features that makes Citus stronger in use cases where it's already strong, as well as to listen in to our users & customers and look to expand into areas where we already see good adoption.
- tadruj 10y ago
  Ozgun, any performance benefits for time-series data? +1 for Top Gun

Another way of asking: Citus seems amazing. What's the worst part of using it (other than cost)?

by rattray 10y ago

Another way of asking: Citus seems amazing. What's the worst part of using it (other than cost)?

I only have about a month experience but the real issue is you have to do a lot of things manually that are done automatically in other systems like Cassandra. Particularly rollups and stream like things. That is you have to do the rollups (as well stream like things) either through postgres rules,triggers Listen/Notify, cron etc. With system like Druid (Cassandra), Elastic Search and other time series databases this is not needed (either the query ability is efficient enough because of compressed or it is automatically indexed or both). For the stream like stuff we have been using Pipelinedb. Pipelinedb is annoying though in that it is a fork and not an extension. Pipelinedb has a similar problem. You have to create the streams apriori. So both for Citus and Pipelinedb you have to do some planning of your schema where as the other guys you can do more exploratory/adhoc analysis and queries. The disadvantage to the NoSQL guys (besides maturity and literally no SQL) is that you pay a lot more in memory as most of them (IMO cheat) and rely heavily on available memory. Postgres is very memory friendly.

rattray 10y ago
Hmm, interesting. Sounds like a problem with a Postgres, not Citus, but it still sounds like a disadvantage compared to competing products (RethinkDB also comes to mind). I haven't implemented anything for that before personally; would a tool like Bottled Water [0] be helpful for this usecase? [0] https://github.com/confluentinc/bottledwater-pg https://github.com/confluentinc/bottledwater-pg
- agentgt 10y ago
  Yes Postgres has disadvantages (I accidentally posted my comment too soon so that might have caused some confusion) but like I said it is: 1. Mature, 2. SQL is damn powerful, 3. Memory friendly (aka cost). RethinkDB is cool. I just never got into it to know how well it actually works. As for bottledwater we use Postgres + AMQP aka Rabbit + Listen/Notify. I'm not really a big fan of kafka even though it does have amazing scaling ability.
geoka9 10y ago
Have you used PostgreSQL before using Citus? If yes, could you share any pain points compared to plain PG (joins, write performance, lack of full SQL support? anything that comes to mind).
grammr 10y ago
> Pipelinedb is annoying though in that it is a fork and not an extension. Hi, I'm one of the PipelineDB co-founders--thanks for using our product! Making PipelineDB an extension is the most consistent piece of feedback we've received from our users, and I promise we're listening: PipelineDB 1.0 will be a standard PostgreSQL extension, incrementally rolled out via versions 0.9.8, 0.9.9, and 1.0.0.
merb 10y ago
well sadly Listen/Notify on PG does not work with streaming replication. That would make it really really great tough.
manigandham 10y ago
Memory is there to be used. 99% CPU and RAM usage is perfect utilization.

jonathanpdenney 10y ago

We've been running on Citus for about 2 months now, with a saas app that has lots of reporting functionality. It's no different than standard postgres in terms of configuration, other than that you have to define a distribution column for the tables you want to be distributed into tenants (e.g. accounts for saas). That's how you really see a significant benefit when working with larger datasets. That and the extra memory they provide over other managed postgres services. For us, "distributing our tables" meant just adding a rails migration using their gem. We were upgrading from heroku postgres, and chose this over RDS, because they take care of the distribution of data into tenants without us having to change our logic, since everything our users create was already getting associated to their organization / account. Only issue we've had was at one point we hit our plan's database connection limit of 300 for a few minutes, because of a new long running query we added, that started locking up connections and causing timeouts as soon as we went to production. We reached out immediately and their team jumped all over it to help us free up those connections, while we fixed the query.

rattray 10y ago
Thanks, that's super helpful!

websymphony 10y ago

They also have a community edition, which also seems pretty powerful. https://www.citusdata.com/product/community https://www.citusdata.com/product/community So cost is maybe not a big drawback either.

doh 10y ago

We're using Citus for almost a year, 8 months in production. Currently running 22 workers [32 core, 3TB space, 120GB RAM] containing almost a trillion (yep) rows in one single table. Citus has a lot of quirks and one could write a book about it. But for shortness here is just couple of the top things you should know. - Citus is not PG. It runs on PG, it looks like PG but it doesn't behave like PG. By using Citus you lose majority of advance strategies, especially ones that manipulate data (UPDATE from SELECT, DELETE, etc.). In very short, you have to forget about everything you know about PG. This means that you have to rethink your table schemas, distribution strategies, etc. This is common in every distributed database (No/SQL). - Citus deployed in a cloud is a bit sensitive. As the master node keeps track of each worker in almost realtime fashion, any breaks in communication results in corrupted shards. If you don't use just one user (postgres) but have different rights for different users, fixing the shards can be a pain (if you don't pay for the enterprise version, which provides you with helpful scripts). - Adding new worker is easy, but rebalancing them to spread the shards evenly is not (again different for enterprise version). - PG's planner can be a bit stubborn while using citus. It sometimes behaves differently from the native PG. The result is that you have to rethink you strategies and rewrite queries up until you don't get the desired result. - Writing functions and automatic triggers can be challenging. The usefulness of them is determined by how well you designed you original tables. Once the tables are colocated and sharded on the same column you get impressive results. Writing the functions can be challenging though as you have to deploy the function to each table. They offer a script to help you with it https://github.com/citusdata/citus/blob/a2f6b29a6d4537b626f8cb8c30525a95c8e7ffe9/src/backend/distributed/master/citus_tools.sql https://github.com/citusdata/citus/blob/a2f6b29a6d4537b626f8... - You do want to consider how many shards you need. More automatically means that Citus gets slower, but you want to be able to grow so chose more than you have workers. There is much more, but those are the things that you should consider. As for us, we learned to live with the quirks and now we're able to run aggregation queries through billions of rows in under 600ms. I would suggest to consider Citus only if you're considering something like Cassandra or Hbase.

jpgvm 10y ago
Are you using the commercial distribution or the OSS one? If the OSS release how have you implemented rebalancing? It seems to me the APIs are there in the extension, just user defined functions are missing to actually do the rebalance. I have been looking at integrating Citus into Flynn as an option for more scale-out workloads but needing non-OSS features to recover from workers going missing or adding new workers put me off implementing it for now.

ozgune 10y ago

(Ozgun from Citus Data) I think the answer to your question depends on the workload you're planning to use Citus for. We find that users rely on Postgres for different use cases and Citus' benefits (and worst parts :)) vary across them. Common use cases, ordered by popularity & benefits, include: 1. Multi-tenant database: Most B2B applications already have the notion of a tenant / customer built into their data model. Citus allows you to scale your transactional relational database to 100K+ tenants with minimal changes to your application -- and with all the benefits of Postgres. 2. Operational analytics: Citus enables ingesting large volumes of data and running analytical queries on that data in human real-time. Example applications include analytic dashboards with sub-second response times. 3. NoSQL++: As we increased Citus' feature set, we found that users started relying on it for workloads that have high read / write throughputs. We have a few "hidden" features in v6.1 that make Citus even more powerful here. That said, sometimes the open source community is way ahead of us on this. :) 4. Other: These include anything from distributed transactions to real-time notifications. Since we extend Postgres, if you're a power user and understand Citus' underlying primitives, sky's the limit. (We debated adding a section to our documentation for other workloads. We just couldn't agree if we wanted to name it Top Gun or Highway to the Danger Zone.) Our goal in each release is to add new features that makes Citus stronger in use cases where it's already strong, as well as to listen in to our users & customers and look to expand into areas where we already see good adoption.

tadruj 10y ago
Ozgun, any performance benefits for time-series data? +1 for Top Gun