5 ms·

This article is pretty thin. We are currently building out a clustered application on corosync/pacemaker with postgres synchronous replication and tomcat, and

This article is pretty thin. We are currently building out a clustered application on corosync/pacemaker with postgres synchronous replication and tomcat, and I have mixed feelings about Linux HA so far. It isn't too bad to get something basic set up (cluster with virtual IP for instance), but when things don't work it can be difficult to figure out why. If you are looking for a distributed filesytem GFS2 in this stack isn't bad. However, it seems like there are a lot of differences between package versions, and the interactions between versions of heartbeat, corosync, pacemaker, crm, pcs, your resource definition ocf files, the stonith resources, cluster-glue, along with the linux packages makes problems hard to track down and much of the web info you do find out of date. I've often had to resort to irc or the mailing list to try and figure things out, and even then sometimes it seems like nobody knows. The whole thing feels a little bit shaky at first, but it is possible to build a solid cluster on top of it with enough effort.

jvoorhis 13y ago
If you truly care about reliability, finding a reliable and fully tested distribution of these packages is the way to go. RedHat, SUSE and LinBit offer certified, fully tested binary distributions and support, and companies like Percona also provide support for specific HA stacks, and their expertise can save you a lot of guesswork. The available documentation for heartbeat, corosync and pacemaker can be disappointing. I found the best way to learn is by doing–if you can afford it, cobble together a lab environment and see how many different ways you can break a system. With pacemaker, I find this depends highly on the resources you are managing!
contingencies 13y ago
I feel your pain. I have been maintaining a probably larger cluster with a similar stack for a few years. Right now I am working on a simulation system that will enable testing arbtrary failure scenarios for arbitrary services across corosync/pacemaker in a virtualized environment. Things like pulling cables, machine death, random upticks in latency, that sort of thing. This will also help to test new designs for new data centers, and corosync/pacemaker's performance over (potentially) high latency links such as the internet when physical site / legal jurisdiction redundancy is also required. What I've done, since it's unrealistic to expect average service authors to grok n-layers of complexity in the networking/storage/service setup fabric and related security processes, is to reduce the problem to one of rigid developer workflow neatly segregating 'platforms', 'services' and 'cloud providers'. The latter, which are either external commercial black boxes or your own infrastructure, includes one that implements corosync/pacemaker, writes your cluster rules for you, segments things neatly in to VLANs, dynamically reconfigures routers, disklessly provisions physical nodes, maintains a detailed hardware inventory including all serial numbers, etc. While it's impossible to replicate real hardware precisely in simulation, I came to the conclusion that I basically needed this degree of automation to have any confidence that things are working and will continue to work.

jvoorhis 13y ago

If you truly care about reliability, finding a reliable and fully tested distribution of these packages is the way to go. RedHat, SUSE and LinBit offer certified, fully tested binary distributions and support, and companies like Percona also provide support for specific HA stacks, and their expertise can save you a lot of guesswork. The available documentation for heartbeat, corosync and pacemaker can be disappointing. I found the best way to learn is by doing–if you can afford it, cobble together a lab environment and see how many different ways you can break a system. With pacemaker, I find this depends highly on the resources you are managing!

contingencies 13y ago

I feel your pain. I have been maintaining a probably larger cluster with a similar stack for a few years. Right now I am working on a simulation system that will enable testing arbtrary failure scenarios for arbitrary services across corosync/pacemaker in a virtualized environment. Things like pulling cables, machine death, random upticks in latency, that sort of thing. This will also help to test new designs for new data centers, and corosync/pacemaker's performance over (potentially) high latency links such as the internet when physical site / legal jurisdiction redundancy is also required. What I've done, since it's unrealistic to expect average service authors to grok n-layers of complexity in the networking/storage/service setup fabric and related security processes, is to reduce the problem to one of rigid developer workflow neatly segregating 'platforms', 'services' and 'cloud providers'. The latter, which are either external commercial black boxes or your own infrastructure, includes one that implements corosync/pacemaker, writes your cluster rules for you, segments things neatly in to VLANs, dynamically reconfigures routers, disklessly provisions physical nodes, maintains a detailed hardware inventory including all serial numbers, etc. While it's impossible to replicate real hardware precisely in simulation, I came to the conclusion that I basically needed this degree of automation to have any confidence that things are working and will continue to work.