4 ms·

I've seen analytics services do this - periodic sync of multiple data sources into their analysis service, which is usually BigQuery or some OLAP system. I don

I've seen analytics services do this - periodic sync of multiple data sources into their analysis service, which is usually BigQuery or some OLAP system. I don't think it would be a good idea for replication though. General syncing is super intensive, you have to diff everything between the two databases. Major DBMSs all have better replication processes that avoid this, in MySQL's case the slaves are synchronized using the master's binary log so they replay transactions that happen on master. It's far faster and allows for near real-time synchronization.

hardwaresofton 10y ago
What I was suggesting was moving the binary log replaying OUT of the database layer, which is where I imagined the efficiency losses might arise. I'm still fleshing this out in my head, but the idea is like: maintain a log of SQL commands that were to be done on the database at a layer right _before_ the database layer. then, as writes come in, they go to the layer-before, get replicated/distributed, and get copied to every node, and you could get easy (?) multi-master support. At the very least, you could do master-slave replication at the same cost that databases are doing it now, minus any inefficiencies from not using the specialized binary log each DB uses today. I guess the pre-database layer would do more gossiping than anything (in a multi-master type setup). The real benefit here is that you would then be able to distribute any database that just supported SQL instantly (and even some that don't, with a properly written adapter), by composing it with another external (rather than internal) management process. There might even be some efficiencies to be gained if the databases are sharded -- localized data might go to the wrong node, but live there long enough to get reads and stuff like that. I guess maybe instead of saying I'm thinking of building a layer on top, I'd rather pull binary log replication out of the databases themselves, and find some close-to-SQL log of commands that all DBs can share.

hardwaresofton 10y ago

What I was suggesting was moving the binary log replaying OUT of the database layer, which is where I imagined the efficiency losses might arise. I'm still fleshing this out in my head, but the idea is like: maintain a log of SQL commands that were to be done on the database at a layer right _before_ the database layer. then, as writes come in, they go to the layer-before, get replicated/distributed, and get copied to every node, and you could get easy (?) multi-master support. At the very least, you could do master-slave replication at the same cost that databases are doing it now, minus any inefficiencies from not using the specialized binary log each DB uses today. I guess the pre-database layer would do more gossiping than anything (in a multi-master type setup). The real benefit here is that you would then be able to distribute any database that just supported SQL instantly (and even some that don't, with a properly written adapter), by composing it with another external (rather than internal) management process. There might even be some efficiencies to be gained if the databases are sharded -- localized data might go to the wrong node, but live there long enough to get reads and stuff like that. I guess maybe instead of saying I'm thinking of building a layer on top, I'd rather pull binary log replication out of the databases themselves, and find some close-to-SQL log of commands that all DBs can share.