5 ms·

This is what I'd like to know too. It's easy to suggest mirroring your infrastructure on multiple providers but in practice it's very difficult. Multi-master SQ

This is what I'd like to know too. It's easy to suggest mirroring your infrastructure on multiple providers but in practice it's very difficult. Multi-master SQL in particular is non-trivial and that's just one piece of the puzzle. We have MongoDBs, CouchDBs, Redis, and Elasticsearch -- each with varying difficulty and costs to cluster.

atom_enger 11y ago
You're definitely not mirroring your infrastructure any time soon :). Not to say that it's impossible because I don't believe in impossible, but that's quite a stack to replicate. We have much the same stack at Reverb.com and I focus more on creating a disaster recovery plan than tackling a mirror of our infra. I'm focusing on bringing up our stack in another region of AWS if we would ever need to rather than mirroring everything. Sometimes a mitigation plan is far more useful than building in excessively complicated redundancy.
- mikegioia 11y ago
  That's interesting and probably more in line with what I need to do. It wouldn't be difficult for me to bring the entire stack online in another datacenter, but if all of your customer data resides on (for simplicity) a master SQL node in Linode-Dallas and that datacenter goes offline, how can you bring that backup node online when you can no longer access the Dallas machines? Are you suggesting to have a slave ready to become master in the 2nd datacenter? If that's the case, my main area of uncertainty is: if you promote a slave to master in a 2nd datacenter and that new master accepts writes, you then have a brutal problem of propagating writes to the old master when he comes back online. The whole thing may exceed my capability right now but I think having a disaster recovery is far better than a mirror now that I think through it.
- nickpsecurity 11y ago
  I'm curious about the stuff that's hard to replicate. Don't most of them store their stuff in files? Could those users perhaps use a distributed filesystem for replication? Clustered filesystems were used in supercomputers, are used by Google, and similar methods go back to VMS systems on VAXen. I'm not saying something off-the-shelf will work or out of the box. Just seems like there would be a clear path for admins or developers of such software to integrate it with... something [1]... that did that. [1] Example: http://sector.sourceforge.net/ http://sector.sourceforge.net/
vox_mollis 11y ago
Nobody said anything about multimaster. You keep a hot standby in a different availability zone and stream your transactions to it. Failover when needed. Unless you're a bank, you run asynchronous with watchdog scripts and accept the very very small risk that you lose a few transactions within the ~100ms window of async latency. It's all risk analysis. For most of us, hours of downtime is riskier than losing a tiny sliver of data for a small number of customers.
- nickpsecurity 11y ago
  That's a proven approach going back decades. A number of banks do it, too, because it often works well enough in practice where data isn't lost. A trick on that is having something on each end that sort of checksums the data(base) for certain time periods to see if anything was lost. Then, resends that stuff. My use of that was inspired by the transport layer of the Tsunami file-transfer protocol with the implementation very similar, too. :)

atom_enger 11y ago

You're definitely not mirroring your infrastructure any time soon :). Not to say that it's impossible because I don't believe in impossible, but that's quite a stack to replicate. We have much the same stack at Reverb.com and I focus more on creating a disaster recovery plan than tackling a mirror of our infra. I'm focusing on bringing up our stack in another region of AWS if we would ever need to rather than mirroring everything. Sometimes a mitigation plan is far more useful than building in excessively complicated redundancy.

mikegioia 11y ago
That's interesting and probably more in line with what I need to do. It wouldn't be difficult for me to bring the entire stack online in another datacenter, but if all of your customer data resides on (for simplicity) a master SQL node in Linode-Dallas and that datacenter goes offline, how can you bring that backup node online when you can no longer access the Dallas machines? Are you suggesting to have a slave ready to become master in the 2nd datacenter? If that's the case, my main area of uncertainty is: if you promote a slave to master in a 2nd datacenter and that new master accepts writes, you then have a brutal problem of propagating writes to the old master when he comes back online. The whole thing may exceed my capability right now but I think having a disaster recovery is far better than a mirror now that I think through it.
nickpsecurity 11y ago
I'm curious about the stuff that's hard to replicate. Don't most of them store their stuff in files? Could those users perhaps use a distributed filesystem for replication? Clustered filesystems were used in supercomputers, are used by Google, and similar methods go back to VMS systems on VAXen. I'm not saying something off-the-shelf will work or out of the box. Just seems like there would be a clear path for admins or developers of such software to integrate it with... something [1]... that did that. [1] Example: http://sector.sourceforge.net/ http://sector.sourceforge.net/

vox_mollis 11y ago

Nobody said anything about multimaster. You keep a hot standby in a different availability zone and stream your transactions to it. Failover when needed. Unless you're a bank, you run asynchronous with watchdog scripts and accept the very very small risk that you lose a few transactions within the ~100ms window of async latency. It's all risk analysis. For most of us, hours of downtime is riskier than losing a tiny sliver of data for a small number of customers.

nickpsecurity 11y ago
That's a proven approach going back decades. A number of banks do it, too, because it often works well enough in practice where data isn't lost. A trick on that is having something on each end that sort of checksums the data(base) for certain time periods to see if anything was lost. Then, resends that stuff. My use of that was inspired by the transport layer of the Tsunami file-transfer protocol with the implementation very similar, too. :)