4 ms·

This comes right at a time when I started attempting something similar (though I don't care that much about documenting it) You might want to read some overvie

This comes right at a time when I started attempting something similar (though I don't care that much about documenting it) You might want to read some overviews on: * postgres atomicity: https://brandur.org/postgres-atomicity * postgres disk format: http://rachbelaid.com/introduction-to-postgres-physical-storage/ I have not read much on replication, though. Does anyone have any pointers?

flavio81 9y ago
> Does anyone have any pointers? Sure, here you are: void *x= malloc(16384) (Couldn't resist)
- mcguire 9y ago
  Welcome to systems programming, home of bad jokes and worse problems...
  - flavio81 9y ago
    "I called the janitor the other day to see what he could do about my dingy linoleum floor. He said he would have been happy to loan me a polisher, but that he hadn't the slightest idea what he had done with it. I told him not to worry about it - that as a programmer it wasn't the first time I had experienced a buffer allocation failure due to a memory error." --- "Q: Why did the concurrent chicken cross the road? A: the side other To to get" (These were copied from the S.O. website)
    - mcguire 9y ago
      One piece of advice I'll never forget: Never grep a yacc by the i-node.
- wruza 9y ago
  I thought replication pointer would come from strdup() at least.
- robotresearcher 9y ago
  Memcheck: 16384 bytes probably lost.
vajrabum 9y ago
The most common strategy for database replication is to copy over an initial snapshot then synchronize the replica by replaying transaction logs. So essentially replication is continuous incremental recovery on another server.
dfox 9y ago
Typical approach to replication is to somehow record changes made to database and somehow push these to slave replicas. There are different approaches to both "somehows". For the format of the changes themselves there is continuum of different levels of abstraction that range from propagating the user's commands to slave servers directly (this is essentially what mysql does) to propagating commited writes to on-disk files (which is what default pgsql replication is about). Both extremes have their tradeoffs: in the SQL-level case the user has to make sure that results of transactions are deterministic (which is more complex than just not calling functions like rand() and now()) and in the physical page case all involved servers need to have exactly same on-disk layout (ie. same cpu architecture and you have to copy the actual on-disk files for initial synchronisation, dump and restore usually does not work) For the actual propagation there are two approaches: write the change records somewhere and let the slaves read that at whatever pace they can or push the changes to slaves before the transaction is reported as commited to the client. The full spectrum of PostgreSQL replication solutions in fact completely fills the space of these two choices and you probably should select carefully what matches your usecase best. (The two significant questions to ask is whether you can live with replicating the whole database cluster as one indivisible unit and whether you can live with possibly stale state on readonly replicas. When both answers are yes then streaming replication is what you want. If answer to second one is no, then you should probably redesign your application such that the data that requires this kind of guarantees is as small as possible)

flavio81 9y ago

> Does anyone have any pointers? Sure, here you are: void *x= malloc(16384) (Couldn't resist)

mcguire 9y ago
Welcome to systems programming, home of bad jokes and worse problems...
- flavio81 9y ago
  "I called the janitor the other day to see what he could do about my dingy linoleum floor. He said he would have been happy to loan me a polisher, but that he hadn't the slightest idea what he had done with it. I told him not to worry about it - that as a programmer it wasn't the first time I had experienced a buffer allocation failure due to a memory error." --- "Q: Why did the concurrent chicken cross the road? A: the side other To to get" (These were copied from the S.O. website)
  - mcguire 9y ago
    One piece of advice I'll never forget: Never grep a yacc by the i-node.
wruza 9y ago
I thought replication pointer would come from strdup() at least.
robotresearcher 9y ago
Memcheck: 16384 bytes probably lost.

vajrabum 9y ago

The most common strategy for database replication is to copy over an initial snapshot then synchronize the replica by replaying transaction logs. So essentially replication is continuous incremental recovery on another server.

dfox 9y ago

Typical approach to replication is to somehow record changes made to database and somehow push these to slave replicas. There are different approaches to both "somehows". For the format of the changes themselves there is continuum of different levels of abstraction that range from propagating the user's commands to slave servers directly (this is essentially what mysql does) to propagating commited writes to on-disk files (which is what default pgsql replication is about). Both extremes have their tradeoffs: in the SQL-level case the user has to make sure that results of transactions are deterministic (which is more complex than just not calling functions like rand() and now()) and in the physical page case all involved servers need to have exactly same on-disk layout (ie. same cpu architecture and you have to copy the actual on-disk files for initial synchronisation, dump and restore usually does not work) For the actual propagation there are two approaches: write the change records somewhere and let the slaves read that at whatever pace they can or push the changes to slaves before the transaction is reported as commited to the client. The full spectrum of PostgreSQL replication solutions in fact completely fills the space of these two choices and you probably should select carefully what matches your usecase best. (The two significant questions to ask is whether you can live with replicating the whole database cluster as one indivisible unit and whether you can live with possibly stale state on readonly replicas. When both answers are yes then streaming replication is what you want. If answer to second one is no, then you should probably redesign your application such that the data that requires this kind of guarantees is as small as possible)