4 ms·

The feature I'd really love is master selection with Raft or similar and automatic query redirection to the master for all write queries (and maybe for reads wi

by api 10y ago

The feature I'd really love is master selection with Raft or similar and automatic query redirection to the master for all write queries (and maybe for reads with a query keyword). That would make it very easy and robust to cluster pg without requiring a big complicated (a.k.a. high admin overhead and failure prone) stack with lots of secondary tools. This kind of fire and forget cluster is really the killer feature of things like MongoDB and RethinkDB. Yes people with really huge deployments might want something more tunable, but that's only like 1% of the market. Of course those NoSQL databases also offer eventual and other weaker but more scalable consistency modes, but like highly tuned manual deployment these too are features for the 1% of the market that actually needs that kind of scale. A fire and forget cluster-able fully consistent SQL database would be nirvana for most of the market.

mb4nck 10y ago
About redirection of write queries to the master, from 10 on, you will be able to specify all members of the cluster in the connection string and demand to connect to the master (like "postgresql://host1:5432,host2:5432/somedb?target_session_attrs=read-write"); libpq will do this automatically for you then, see the parameters "host" (now plural) and "target_session_attr" in section 33.1.2. here: https://www.postgresql.org/docs/devel/static/libpq-connect.html https://www.postgresql.org/docs/devel/static/libpq-connect.h... About raft-based leader-election, I believe the current recommendation is to look at patroni ( https://github.com/zalando/patroni https://github.com/zalando/patroni), which has been built for docker and is now being integrated with Kubernetes; however, I don't think there is an inherent limitation that it couldn't be run on bare-metal.
- api 10y ago
  Yes on the first one, and a big nope on the second for now. I passionately loathe Rube Goldberg machine deployments and am the kind of engineer who constantly asks "do we really need that?". I love exterminating complexity. But maybe that will change when we get to millions of concurrent users and tens of millions of devices and actually need Kubernetes to scale. Raft is not complex. I doubt leader elect would be terribly hard to implement.
- rachbelaid 10y ago
  With 2ndquadrant working on Postgres-XL (http://www.postgres-xl.org/ http://www.postgres-xl.org/), I think that you can be confident that you will see a lot of the features being proposed to core postgres. It will just take some times to build the building block necessary like: global index, distributed sequence, repartition ... I quite confident that the postgresql from 5y in future will be quite different in term of storage / server topology support. I won't be surprise pg_bouncer capacity to finally make its way to core when we have a coordinator. Postgres has steady progression (even if not fast enough for some people) but they are moving without compromising robustness of their product for the users.
xyzzy_plugh 10y ago
Can't pgbouncer/pgpool2 solve query redirection? I don't understand the desire for all-in-one solutions.
- api 10y ago
  That desire comes from three places: 1. Minimize cognitive load by minimizing the number of things you have to learn. 2. Minimize deployment complexity and dependencies. 3. Complexity is just evil I'm general. Linear increases in complexity result in exponential increases in bugs, vulnerabilities, and failure modes. It's just combinatorics.
  - mb4nck 10y ago
    patroni builds upon etcd and (optionally) haproxy, two rather mature pieces of infrastructure (which can both be made HA on their own if SPOFs are to be avoided). I understand where you are coming from, but having this kind of multi-server-who-is-master knowledge baked into Postgres itself will surely take another couple of releases, if it will be included at all. Probably BDR (bi-directional replication, master-master like logical replication) will be there first, but whether the question is whether it will help a lot for local scale-out workloads (as opposed to glueing two datacenters together and allow transactions on both sides).
  - xyzzy_plugh 10y ago
    Number 2 is the only point I see merit in. With respect to 3, isn't that an argument against increasing the complexity of postgresql?
- deleted 10y ago
  [deleted]
- manigandham 10y ago
  > I don't understand the desire for all-in-one solutions You really don't understand it? It's less moving pieces to think about, worry about, read about, deploy, maintain, fix and almost always leads to better performance and security. Something as simple as connection pooling should've already been part of the database and query redirection is even more important to have included.

by api 10y ago

mb4nck 10y ago

About redirection of write queries to the master, from 10 on, you will be able to specify all members of the cluster in the connection string and demand to connect to the master (like "postgresql://host1:5432,host2:5432/somedb?target_session_attrs=read-write"); libpq will do this automatically for you then, see the parameters "host" (now plural) and "target_session_attr" in section 33.1.2. here: https://www.postgresql.org/docs/devel/static/libpq-connect.html https://www.postgresql.org/docs/devel/static/libpq-connect.h... About raft-based leader-election, I believe the current recommendation is to look at patroni ( https://github.com/zalando/patroni https://github.com/zalando/patroni), which has been built for docker and is now being integrated with Kubernetes; however, I don't think there is an inherent limitation that it couldn't be run on bare-metal.

api 10y ago
Yes on the first one, and a big nope on the second for now. I passionately loathe Rube Goldberg machine deployments and am the kind of engineer who constantly asks "do we really need that?". I love exterminating complexity. But maybe that will change when we get to millions of concurrent users and tens of millions of devices and actually need Kubernetes to scale. Raft is not complex. I doubt leader elect would be terribly hard to implement.
rachbelaid 10y ago
With 2ndquadrant working on Postgres-XL (http://www.postgres-xl.org/ http://www.postgres-xl.org/), I think that you can be confident that you will see a lot of the features being proposed to core postgres. It will just take some times to build the building block necessary like: global index, distributed sequence, repartition ... I quite confident that the postgresql from 5y in future will be quite different in term of storage / server topology support. I won't be surprise pg_bouncer capacity to finally make its way to core when we have a coordinator. Postgres has steady progression (even if not fast enough for some people) but they are moving without compromising robustness of their product for the users.

xyzzy_plugh 10y ago

Can't pgbouncer/pgpool2 solve query redirection? I don't understand the desire for all-in-one solutions.

api 10y ago
That desire comes from three places: 1. Minimize cognitive load by minimizing the number of things you have to learn. 2. Minimize deployment complexity and dependencies. 3. Complexity is just evil I'm general. Linear increases in complexity result in exponential increases in bugs, vulnerabilities, and failure modes. It's just combinatorics.
- mb4nck 10y ago
  patroni builds upon etcd and (optionally) haproxy, two rather mature pieces of infrastructure (which can both be made HA on their own if SPOFs are to be avoided). I understand where you are coming from, but having this kind of multi-server-who-is-master knowledge baked into Postgres itself will surely take another couple of releases, if it will be included at all. Probably BDR (bi-directional replication, master-master like logical replication) will be there first, but whether the question is whether it will help a lot for local scale-out workloads (as opposed to glueing two datacenters together and allow transactions on both sides).
- xyzzy_plugh 10y ago
  Number 2 is the only point I see merit in. With respect to 3, isn't that an argument against increasing the complexity of postgresql?
deleted 10y ago
[deleted]
manigandham 10y ago
> I don't understand the desire for all-in-one solutions You really don't understand it? It's less moving pieces to think about, worry about, read about, deploy, maintain, fix and almost always leads to better performance and security. Something as simple as connection pooling should've already been part of the database and query redirection is even more important to have included.