6 ms·

How does Mysos handle the spin-up of new slaves? One of the pain points of our current Chef-driven environment is the provisioning of new slaves. We use Che

How does Mysos handle the spin-up of new slaves? One of the pain points of our current Chef-driven environment is the provisioning of new slaves. We use Chef to install and configure MySQL but to get a slave up and running, we use xtrabackup to dump the master's DBs to disk, then rsync them to the slave, then use xtrabackup to restore the dump. Once that's done, we point the new slave at the master and start it slaving. It's totally manual and takes around 15 minutes each time. How does Mysos automate this?

gpapilion 11y ago
Looking at to code, it pulls from a backup and restores the state. It manages the cluster slave and master states via zookeeper. When the master fails a slave is promoted. So, in order to run this you need zookeeper, hdfs, and mesos. Its a great solution, but not for everyone.
- jpgvm 11y ago
  To be fair, if you have Mesos it's highly likely you already have HDFS, ZK is a dependency of Mesos. Though it's not strictly necessary to run HDFS a lot of Mesos native applications and frameworks already expect it. The thing is you can often run HDFS on your Mesos slaves so it ends up stacking up quite nicely, atleast in small deployments.
  - merb 11y ago
    How is HDFS quite nicely in small environments? I mean a minimal HA deployment of HDFS contains at least 5 nodes. 2x Namenode, 3x Data Nodes. Sorry but for "starting out small" this is definitely too much.
    - jpgvm 11y ago
      I don't see how Mesos is going to be useful to you with less than 5 servers.
xu 11y ago
Mysos follows similar steps, it just automates them for you. Mesos gives the scheduler abstracted resources and finds the box to launch the MySQL task. The task still needs to install MySQL (possibly pull the package from HDFS) and recover the state (also possibly pull from DHFS). These things are specific to the organization's workflow and the MySQL task's behavior can be customized by implementing interfaces for these steps. Mesos bridges the central scheduler and the MySQL instances distributed across the datacenter so it can monitor them, make centralized decisions and communicate with the instances reliably.

gpapilion 11y ago

Looking at to code, it pulls from a backup and restores the state. It manages the cluster slave and master states via zookeeper. When the master fails a slave is promoted. So, in order to run this you need zookeeper, hdfs, and mesos. Its a great solution, but not for everyone.

jpgvm 11y ago
To be fair, if you have Mesos it's highly likely you already have HDFS, ZK is a dependency of Mesos. Though it's not strictly necessary to run HDFS a lot of Mesos native applications and frameworks already expect it. The thing is you can often run HDFS on your Mesos slaves so it ends up stacking up quite nicely, atleast in small deployments.
- merb 11y ago
  How is HDFS quite nicely in small environments? I mean a minimal HA deployment of HDFS contains at least 5 nodes. 2x Namenode, 3x Data Nodes. Sorry but for "starting out small" this is definitely too much.
  - jpgvm 11y ago
    I don't see how Mesos is going to be useful to you with less than 5 servers.

xu 11y ago

Mysos follows similar steps, it just automates them for you. Mesos gives the scheduler abstracted resources and finds the box to launch the MySQL task. The task still needs to install MySQL (possibly pull the package from HDFS) and recover the state (also possibly pull from DHFS). These things are specific to the organization's workflow and the MySQL task's behavior can be customized by implementing interfaces for these steps. Mesos bridges the central scheduler and the MySQL instances distributed across the datacenter so it can monitor them, make centralized decisions and communicate with the instances reliably.