5 ms·

DRBD works very well for high availability, especially good to provide failover for master database, since that usually requires fast failover like under 10 sec

by ww520 10y ago

DRBD works very well for high availability, especially good to provide failover for master database, since that usually requires fast failover like under 10 seconds. Five or six years ago, did couple HA setup with DRBD along with Linux HA and virtual IP. The failover work great.

kbenson 10y ago
I was under the impression the accepted way to fail-over with mysql and DRBD was to fail out the old server, and then start repairing tables, because you couldn't be sure they were in a valid state. That's about a decade old info though. I always just set up master/master replication with mysql. You can get free distributed reads that way if you architect your application right.
- Rapzid 10y ago
  DRBD supports flushes and barriers. You just switch the primary and startup the database. InnoDB will recover as necessary and you are good to go. This is assuming you are using all InnoDB tables(you likely should be).
  - kbenson 10y ago
    > This is assuming you are using all InnoDB tables(you likely should be). Ah, that's probably it. ~2004/2005 InnoDB wasn't nearly as popular as it is today, IIRC (but it was there, and was used). In any case, multiple master with failover setups always worked well in my experience, as long as you take care to track the replication state.
- ww520 10y ago
  DRBD produces two mirror copies of data. When the primary failed, the standby has whatever data the primary has just before the crash. When the standby starts up, the RDBMS goes through the normal recovery and bootup. It's same as the primary crashed and being started up again. MySQL's master/master has a number of complication and problems: 1. data loss due to async nature of replication, 2. update conflict on same data on multiple masters, 3. two masters mean two IP so all the clients need to know how to fail over to different IP, 4. complication in adding or removing master. With DRBD, the disks are mirrored so that's no chance of data loss once a transaction is committed. There's only one master so no complicate conflict resolution. Linux HA's virtual IP means the standby will take over the primary IP so the clients don't need to know there's a server failover. Adding or removing standby is easy. DRBD will sync the disks automatically, no downtime on the primary.
  - kbenson 10y ago
    > 1. data loss due to async nature of replication That depends on what you mean by async. The replication itself is synchronous (statements cannot happen out of order), it's just not lockstep with disk writes and commits. I think it's more illustrative to say it's delayed. > 2. update conflict on same data on multiple masters, 3. two masters mean two IP so all the clients need to know how to fail over to different IP I'm not referring to multiple live masters, I'm referring to a set of servers where each is both master and slave to the other, and one live master server which gets the HA IP address. In that respect, there is no difference to a DRBD replicated setup. Clients just use the HA IP. > 4. complication in adding or removing master. I never found it that complicated, and I deployed it at least 5-6 times for multiple companies, and even with slightly different topologies (master/master where each master had an additional dedicated slave reserved for intensive read-only queries). What did you find complicated about it?
    - ww520 10y ago
      1. MySQL replication is async by default; see [1]. That means commit returns before replication to peer is complete. Committed data can be lost if master's disk is destroyed before a slave replicates it. NDB has synchronous replication but it has other limitation. 5.7 seems to have a semi-sync mode now. [1] http://dev.mysql.com/doc/refman/5.7/en/replication.html http://dev.mysql.com/doc/refman/5.7/en/replication.html 4. For complication, you need to find out and configure the binary log position. Also because replication is on the binary log, if you ever truncate the log, you can't simply add a brand new master. You have to do a backup on the primary and restore to the new master, and then set up the log position to just prior to backup. Just lots of extra complication.
      - kbenson 10y ago
        > That means commit returns before replication to peer is complete. That's what I was talking about. It's just a matter of what aspect of it you are talking about, but I'll give you that it's in their official documentation, so there's no point in me pressing the issue. > Committed data can be lost if master's disk is destroyed before a slave replicates it. That is true. It's a trade-off you can make for slightly different CAP assurances, or nuances in the failure states at least (mostly in what you might expect to do in a split brain scenario). > For complication, you need to find out and configure the binary log position. Your backups should be logging the binary log position as well (--dump-slave or --maser-data). If they aren't, you aren't doing yourself any favors. > Also because replication is on the binary log, if you ever truncate the log, you can't simply add a brand new master. You have to do a backup on the primary and restore to the new master, and then set up the log position to just prior to backup. Just lots of extra complication. If you have to do another backup because it was truncated recently, you aren't much worse off than doing backups with DRBD replication (which even if you have a slave configured and do backups off that, you can truncate logs and need to backup from the master then as well). The downside is that you may not want to immediately do a backup of the master due to reasons of load, which will leave you without a failover for a short while. Whether an extra queryable resource available is worth that is up the the architect. I remember a Percona training I was at a few years back there were a few more clustering options available I hadn't played with (and still haven't). Percona XtraDB Cluster was one, and it's supposed to support synchronous master/master replication. That might be the best of both worlds, if it lives up to its billing.
Annatar 10y ago
DRBD is failover - there is an interruption while the switch is made. High availability is when you can afford to lose one or more nodes, and there is no service interruption. Moreover, the processing and I/O are symmetric and performance scales linearly by adding more nodes.

by ww520 10y ago

kbenson 10y ago

I was under the impression the accepted way to fail-over with mysql and DRBD was to fail out the old server, and then start repairing tables, because you couldn't be sure they were in a valid state. That's about a decade old info though. I always just set up master/master replication with mysql. You can get free distributed reads that way if you architect your application right.

Rapzid 10y ago
DRBD supports flushes and barriers. You just switch the primary and startup the database. InnoDB will recover as necessary and you are good to go. This is assuming you are using all InnoDB tables(you likely should be).
- kbenson 10y ago
  > This is assuming you are using all InnoDB tables(you likely should be). Ah, that's probably it. ~2004/2005 InnoDB wasn't nearly as popular as it is today, IIRC (but it was there, and was used). In any case, multiple master with failover setups always worked well in my experience, as long as you take care to track the replication state.
ww520 10y ago
DRBD produces two mirror copies of data. When the primary failed, the standby has whatever data the primary has just before the crash. When the standby starts up, the RDBMS goes through the normal recovery and bootup. It's same as the primary crashed and being started up again. MySQL's master/master has a number of complication and problems: 1. data loss due to async nature of replication, 2. update conflict on same data on multiple masters, 3. two masters mean two IP so all the clients need to know how to fail over to different IP, 4. complication in adding or removing master. With DRBD, the disks are mirrored so that's no chance of data loss once a transaction is committed. There's only one master so no complicate conflict resolution. Linux HA's virtual IP means the standby will take over the primary IP so the clients don't need to know there's a server failover. Adding or removing standby is easy. DRBD will sync the disks automatically, no downtime on the primary.
- kbenson 10y ago
  > 1. data loss due to async nature of replication That depends on what you mean by async. The replication itself is synchronous (statements cannot happen out of order), it's just not lockstep with disk writes and commits. I think it's more illustrative to say it's delayed. > 2. update conflict on same data on multiple masters, 3. two masters mean two IP so all the clients need to know how to fail over to different IP I'm not referring to multiple live masters, I'm referring to a set of servers where each is both master and slave to the other, and one live master server which gets the HA IP address. In that respect, there is no difference to a DRBD replicated setup. Clients just use the HA IP. > 4. complication in adding or removing master. I never found it that complicated, and I deployed it at least 5-6 times for multiple companies, and even with slightly different topologies (master/master where each master had an additional dedicated slave reserved for intensive read-only queries). What did you find complicated about it?
  - ww520 10y ago
    1. MySQL replication is async by default; see [1]. That means commit returns before replication to peer is complete. Committed data can be lost if master's disk is destroyed before a slave replicates it. NDB has synchronous replication but it has other limitation. 5.7 seems to have a semi-sync mode now. [1] http://dev.mysql.com/doc/refman/5.7/en/replication.html http://dev.mysql.com/doc/refman/5.7/en/replication.html 4. For complication, you need to find out and configure the binary log position. Also because replication is on the binary log, if you ever truncate the log, you can't simply add a brand new master. You have to do a backup on the primary and restore to the new master, and then set up the log position to just prior to backup. Just lots of extra complication.
    - kbenson 10y ago
      > That means commit returns before replication to peer is complete. That's what I was talking about. It's just a matter of what aspect of it you are talking about, but I'll give you that it's in their official documentation, so there's no point in me pressing the issue. > Committed data can be lost if master's disk is destroyed before a slave replicates it. That is true. It's a trade-off you can make for slightly different CAP assurances, or nuances in the failure states at least (mostly in what you might expect to do in a split brain scenario). > For complication, you need to find out and configure the binary log position. Your backups should be logging the binary log position as well (--dump-slave or --maser-data). If they aren't, you aren't doing yourself any favors. > Also because replication is on the binary log, if you ever truncate the log, you can't simply add a brand new master. You have to do a backup on the primary and restore to the new master, and then set up the log position to just prior to backup. Just lots of extra complication. If you have to do another backup because it was truncated recently, you aren't much worse off than doing backups with DRBD replication (which even if you have a slave configured and do backups off that, you can truncate logs and need to backup from the master then as well). The downside is that you may not want to immediately do a backup of the master due to reasons of load, which will leave you without a failover for a short while. Whether an extra queryable resource available is worth that is up the the architect. I remember a Percona training I was at a few years back there were a few more clustering options available I hadn't played with (and still haven't). Percona XtraDB Cluster was one, and it's supposed to support synchronous master/master replication. That might be the best of both worlds, if it lives up to its billing.

Annatar 10y ago

DRBD is failover - there is an interruption while the switch is made. High availability is when you can afford to lose one or more nodes, and there is no service interruption. Moreover, the processing and I/O are symmetric and performance scales linearly by adding more nodes.