3 ms·

Logical replication doesn't matter as much to us. We are using CitusDB (in other words sharded Postgres), and we have our own code for handling replication.

qeternity 10y ago
Citus should already be saturating i/o during a scan unless you're only hitting one shard. What am I missing about your use case?
- throwaway91111 10y ago
  Doesn't this inherently depend on the query? Shard-local i/o saturation affects the latency of the entire query if you're not expecting the i/o to the client to saturate.
  - qeternity 10y ago
    This was exactly my point. But usually you're doing a full scan across all shards, so disk i/o is being saturated by virtue of the MPP sharding.
- anarazel 10y ago
  In many cases all your data is actually in memory, especially when you can distribute your workload across many machines. If all your data is in memory, you'll very often hit CPU bottlenecks before memory bandwidth issues. Partially because bandwidth, disregarding latency, is just quite high, partially because expression evaluation in postgres isn't that well optimized (yet).
- malisper 10y ago
  We shard our data by customer and individual queries touch only data for one customer. We do split a customer's data into multiple shards, but the number of shards we would need to split each customer into is not worth the overhead of maintaining that many shards.
  - manigandham 10y ago
    Why? Is this a security issue? If you're sharing the same distributed table then sharding by customer is limiting the number of nodes/scans in parallel that can run.
    - malisper 10y ago
      Having millions of shards is a bit of a pain ops wise. There is also some other costs to having lots of shards. For example, the current Citus distribution planner takes time linear in the number of total shards. This isn't an issue 99% of the time, but since we use a composite type as our distribution key, we have an extremely high constant overhead for planning time. I believe the current median Citus Master planning time is ~350ms. We've talked to Citus about this and have discussed possible solutions. Currently we are trying to purge the composite type from our database.

qeternity 10y ago

Citus should already be saturating i/o during a scan unless you're only hitting one shard. What am I missing about your use case?

throwaway91111 10y ago
Doesn't this inherently depend on the query? Shard-local i/o saturation affects the latency of the entire query if you're not expecting the i/o to the client to saturate.
- qeternity 10y ago
  This was exactly my point. But usually you're doing a full scan across all shards, so disk i/o is being saturated by virtue of the MPP sharding.
anarazel 10y ago
In many cases all your data is actually in memory, especially when you can distribute your workload across many machines. If all your data is in memory, you'll very often hit CPU bottlenecks before memory bandwidth issues. Partially because bandwidth, disregarding latency, is just quite high, partially because expression evaluation in postgres isn't that well optimized (yet).
malisper 10y ago
We shard our data by customer and individual queries touch only data for one customer. We do split a customer's data into multiple shards, but the number of shards we would need to split each customer into is not worth the overhead of maintaining that many shards.
- manigandham 10y ago
  Why? Is this a security issue? If you're sharing the same distributed table then sharding by customer is limiting the number of nodes/scans in parallel that can run.
  - malisper 10y ago
    Having millions of shards is a bit of a pain ops wise. There is also some other costs to having lots of shards. For example, the current Citus distribution planner takes time linear in the number of total shards. This isn't an issue 99% of the time, but since we use a composite type as our distribution key, we have an extremely high constant overhead for planning time. I believe the current median Citus Master planning time is ~350ms. We've talked to Citus about this and have discussed possible solutions. Currently we are trying to purge the composite type from our database.