7 ms·

Good news is that postgres is going to get multi-master replication soon in postgres 10, so a lot of the painpoints when it comes to scaling our writes, are hop

Good news is that postgres is going to get multi-master replication soon in postgres 10, so a lot of the painpoints when it comes to scaling our writes, are hopefully going to be solved.

bradhe 9y ago
> are hopefully going to be solved. Right, because multi-master replication schemes have such a long lineage of being operationally simple. /s
- paladin314159 9y ago
  Yeah, that's the real power of DynamoDB, despite all of its shortcomings. It's operationally simple (given you have the money).
crypto5 9y ago
> Good news is that postgres is going to get multi-master replication soon in postgres 10 Where did you read this?..
- anarazel 9y ago
  I think the poster of that claim is confusing "plain" logical replication and multi-master replication. There's a lot of infrastructure in common between both, but the conflict resolution part of async mm didn't make it it into postgres 10...
b777 9y ago
How does multi-master replication solve write scalability issues? I'm not familiar with the postgres implementation, but don't all writes have to happen on all servers? And if all writes need to happen on all servers, how does that improve write scalability as the load write load on each server is the same as it would be without multi-master replication.
- anarazel 9y ago
  > How does multi-master replication solve write scalability issues? I'm not familiar with the postgres implementation, but don't all writes have to happen on all servers? It doesn't generally. You often can increase the total throughput because independent transactions can be batch committed when replicated, which can increase throughput noticeably, but that's more like a constant factor improvement than a real scalability improvement. What it does however often allow you to do is to: a) Address latency issues with writers distributed across the globe. If you have masters on each continent your total write rate can often be massively higher than if you have application servers from each of those continents write to a centralized database. b) Partially replicate your data globally, partially not. Quite often you'll want to have some data available everywhere (typically metadata, security related data, etc), but a lot of the bulk data only available in certain places.
  - darkr 9y ago
    > a) Address latency issues with writers distributed across the globe How? Surely to provide ACID compliance all writers need to co-ordinate a lock if they want to update a record. Same with incrementing IDs etc.
    - anarazel 9y ago
      > > a) Address latency issues with writers distributed across the globe > How? Surely to provide ACID compliance all writers need to co-ordinate a lock if they want to update a record. Well, there's different systems out there. What you want for something like this is asynchronous multi-master. Which, as you observe, allows for conflicts to occur, which need to be resolved somehow. > Same with incrementing IDs etc. The solution for those is usually to have each node assign sequences with different start values, have a system that distributes "chunks" of values (that was what BDR, an async MM version of postgres used to do), or avoiding sequences in favor of UUIDs or similar.
m00x 9y ago
Wouldn't multi-master replication be bad for write throughput? Depending on the scheme, it will have to be committed to all machines for every write, therefore increasing write operation complexity. If you want to scale up your writes, sharding would be a better solution.
manigandham 9y ago
> postgres is going to get multi-master replication soon in postgres 10 No it's not. Postgres barely has decent replication as is. Logical replication is a good step beyond (while still missing major functionality) but nowhere near what it takes for multi-master replication.
- anarazel 9y ago
  > No it's not. Postgres barely has decent replication as is. You're overly negative whenever postgres comes up on HN. Obviously the v10 stuff is not multi-master and the project definitely is not claiming it has, but saying that Postgres' replication "barely has decent replication as is" is a bit absurd given that it works quite well in a lot of high-throughput scenarios in a lot of companies, and has so for 5+ years. There's a lot of valid complaints about it you can make (especially around flexibility and operational issues), but it's definitely not just "barely decent". I'm not saying that you have to be fan. It's fine to say the replication isn't good enough for everything, or just ignore postgres. > but nowhere near what it takes for multi-master replication. FWIW, there's async MM built on top of the relevant pieces, and it's used in production. It's not ready for wide audiences, but arguably (and I think one can very validly disagree about that) it's closer than "nowhere near".
  - anarazel 9y ago
    https://news.ycombinator.com/item?id=14578213 https://news.ycombinator.com/item?id=14578213 https://news.ycombinator.com/item?id=14373424 https://news.ycombinator.com/item?id=14373424 https://news.ycombinator.com/item?id=13653708 https://news.ycombinator.com/item?id=13653708 https://news.ycombinator.com/item?id=13423951 https://news.ycombinator.com/item?id=13423951 ...
  - manigandham 9y ago
    What is the point of this comment? And then linking to my past comments? Is it wrong to have perspective and criticism or to express it multiple times? I get that you're a committer but "overly negative" is as much your opinion as what I say, and my comments are the truth as I see it. I've even spent an hour talking to Ozgun about databases and moving to Citus before so my statements come from being a user, operator and maintainer of dozens of different database technologies in production for years, including postgres right now, and from all that experience it is definitely the most behind when it comes to replication/HA compared to anything else I use. All of those threads have plenty of upvotes and other user comments so it's clear I don't stand alone here. Perhaps instead of policing my statements, time would be better spent on improving the features and experience of the product.

Good news is that postgres is going to get multi-master replication soon in postgres 10, so a lot of the painpoints when it comes to scaling our writes, are hopefully going to be solved.

bradhe 9y ago

> are hopefully going to be solved. Right, because multi-master replication schemes have such a long lineage of being operationally simple. /s

paladin314159 9y ago
Yeah, that's the real power of DynamoDB, despite all of its shortcomings. It's operationally simple (given you have the money).

crypto5 9y ago

> Good news is that postgres is going to get multi-master replication soon in postgres 10 Where did you read this?..

anarazel 9y ago
I think the poster of that claim is confusing "plain" logical replication and multi-master replication. There's a lot of infrastructure in common between both, but the conflict resolution part of async mm didn't make it it into postgres 10...

b777 9y ago

How does multi-master replication solve write scalability issues? I'm not familiar with the postgres implementation, but don't all writes have to happen on all servers? And if all writes need to happen on all servers, how does that improve write scalability as the load write load on each server is the same as it would be without multi-master replication.

anarazel 9y ago
> How does multi-master replication solve write scalability issues? I'm not familiar with the postgres implementation, but don't all writes have to happen on all servers? It doesn't generally. You often can increase the total throughput because independent transactions can be batch committed when replicated, which can increase throughput noticeably, but that's more like a constant factor improvement than a real scalability improvement. What it does however often allow you to do is to: a) Address latency issues with writers distributed across the globe. If you have masters on each continent your total write rate can often be massively higher than if you have application servers from each of those continents write to a centralized database. b) Partially replicate your data globally, partially not. Quite often you'll want to have some data available everywhere (typically metadata, security related data, etc), but a lot of the bulk data only available in certain places.
- darkr 9y ago
  > a) Address latency issues with writers distributed across the globe How? Surely to provide ACID compliance all writers need to co-ordinate a lock if they want to update a record. Same with incrementing IDs etc.
  - anarazel 9y ago
    > > a) Address latency issues with writers distributed across the globe > How? Surely to provide ACID compliance all writers need to co-ordinate a lock if they want to update a record. Well, there's different systems out there. What you want for something like this is asynchronous multi-master. Which, as you observe, allows for conflicts to occur, which need to be resolved somehow. > Same with incrementing IDs etc. The solution for those is usually to have each node assign sequences with different start values, have a system that distributes "chunks" of values (that was what BDR, an async MM version of postgres used to do), or avoiding sequences in favor of UUIDs or similar.

m00x 9y ago

Wouldn't multi-master replication be bad for write throughput? Depending on the scheme, it will have to be committed to all machines for every write, therefore increasing write operation complexity. If you want to scale up your writes, sharding would be a better solution.

manigandham 9y ago

> postgres is going to get multi-master replication soon in postgres 10 No it's not. Postgres barely has decent replication as is. Logical replication is a good step beyond (while still missing major functionality) but nowhere near what it takes for multi-master replication.

anarazel 9y ago
> No it's not. Postgres barely has decent replication as is. You're overly negative whenever postgres comes up on HN. Obviously the v10 stuff is not multi-master and the project definitely is not claiming it has, but saying that Postgres' replication "barely has decent replication as is" is a bit absurd given that it works quite well in a lot of high-throughput scenarios in a lot of companies, and has so for 5+ years. There's a lot of valid complaints about it you can make (especially around flexibility and operational issues), but it's definitely not just "barely decent". I'm not saying that you have to be fan. It's fine to say the replication isn't good enough for everything, or just ignore postgres. > but nowhere near what it takes for multi-master replication. FWIW, there's async MM built on top of the relevant pieces, and it's used in production. It's not ready for wide audiences, but arguably (and I think one can very validly disagree about that) it's closer than "nowhere near".
- anarazel 9y ago
  https://news.ycombinator.com/item?id=14578213 https://news.ycombinator.com/item?id=14578213 https://news.ycombinator.com/item?id=14373424 https://news.ycombinator.com/item?id=14373424 https://news.ycombinator.com/item?id=13653708 https://news.ycombinator.com/item?id=13653708 https://news.ycombinator.com/item?id=13423951 https://news.ycombinator.com/item?id=13423951 ...
- manigandham 9y ago
  What is the point of this comment? And then linking to my past comments? Is it wrong to have perspective and criticism or to express it multiple times? I get that you're a committer but "overly negative" is as much your opinion as what I say, and my comments are the truth as I see it. I've even spent an hour talking to Ozgun about databases and moving to Citus before so my statements come from being a user, operator and maintainer of dozens of different database technologies in production for years, including postgres right now, and from all that experience it is definitely the most behind when it comes to replication/HA compared to anything else I use. All of those threads have plenty of upvotes and other user comments so it's clear I don't stand alone here. Perhaps instead of policing my statements, time would be better spent on improving the features and experience of the product.