3 ms·

> Instead of worker processes and low-level row ingression, pgcat uses sql template to apply high-level sql commands, so it could make largest compatibility on

> Instead of worker processes and low-level row ingression, pgcat uses sql template to apply high-level sql commands, so it could make largest compatibility on target table type. It is written in golang and runs in separate process. Doing "pgcat uses sql template to apply high-level sql commands" unconditionally is somewhat sad. It has WAY higher overhead than the referenced "low-level row ingression". It makes sense to so in some cases (into a view, triggers, apply functions, ...), but always doesn't seem great.

jinhualuo 7y ago
Yes, the template generated sql would bring some overhead. But it's the cost for flexibility: The target table would not be limited to base table. It could be a view, a foreign table, partitioned table, or citus distributed table. And, it also support table name mapping.
- anarazel 7y ago
  You can detect whether a target is a plain table or not. And use a different path if not. And independent of using SQL or not: Executing them separately out of process kills you via latency, even if you're on the same machine. The context switches are just too expensive. Which makes single threaded apply untenable for anything with a few more clients, or bulk load.
  - jinhualuo 7y ago
    I would try to do more test for performance, especially in product environment. Detect the target table is easy, but it seems tricky to redirect the flow to pg build-in logical replication (not necessary to re-implement it, especially it was written in C language, so redirection is the right way). In fact, you could use pg logical replication directly for some cases like big and real-time data flows. But currently, it has one big flaw: bi-directional replication or multi-master replication is impossible, where it would cause dead cycle loops, because it do not filter any origin currently, you could check function pgoutput_origin_filter. And, even if the filter is right, it still lacks of conflict solution, but pgcat could help there, and you don't need to use pgcat, because pgcat has trigger way to resolve conflict, you could check pgcat_setup_lww command.

jinhualuo 7y ago

Yes, the template generated sql would bring some overhead. But it's the cost for flexibility: The target table would not be limited to base table. It could be a view, a foreign table, partitioned table, or citus distributed table. And, it also support table name mapping.

anarazel 7y ago
You can detect whether a target is a plain table or not. And use a different path if not. And independent of using SQL or not: Executing them separately out of process kills you via latency, even if you're on the same machine. The context switches are just too expensive. Which makes single threaded apply untenable for anything with a few more clients, or bulk load.
- jinhualuo 7y ago
  I would try to do more test for performance, especially in product environment. Detect the target table is easy, but it seems tricky to redirect the flow to pg build-in logical replication (not necessary to re-implement it, especially it was written in C language, so redirection is the right way). In fact, you could use pg logical replication directly for some cases like big and real-time data flows. But currently, it has one big flaw: bi-directional replication or multi-master replication is impossible, where it would cause dead cycle loops, because it do not filter any origin currently, you could check function pgoutput_origin_filter. And, even if the filter is right, it still lacks of conflict solution, but pgcat could help there, and you don't need to use pgcat, because pgcat has trigger way to resolve conflict, you could check pgcat_setup_lww command.