8 ms·

Ask HN: Which VM orchestrator do you use?

47 points by kosolam 5y ago

Looking into hashicorp nomad but I wonder if there is a good non commercial alternative?

mountainriver 5y ago
Kubevirt
zhengiszen 5y ago
Saltstack with saltcloud looks like a perfect fit : https://fr.slideshare.net/MoRawi/salt-cloud-vmwareorchestration https://fr.slideshare.net/MoRawi/salt-cloud-vmwareorchestrat...
bserge 5y ago
bootcat 5y ago
I am looking into something likewise, I liked caprover from this list, https://github.com/guettli/open-source-paas https://github.com/guettli/open-source-paas I feel nomad is a bit resource intensive for my use case,
- proxysna 5y ago
  System requirements for nomad on Hashicopr's assume a relatively high-load. Running a single Rpi 3b+ master with a 10-13 clients at home was not resource intensive at all. Master was effectively running on idle temps and opening something like mc or htop took more resources than the server itself.
- capableweb 5y ago
  > resource intensive Resource intensive? I run Nomad on a bunch of Raspberry PIs at home, not sure what orchestrator is more lightweight than Nomad honestly.
  - bootcat 5y ago
    thank you for sharing this - let me try from my end. My comment and denial towards using nomad was based on their hardware requirements - https://www.nomadproject.io/docs/install/production/requirements https://www.nomadproject.io/docs/install/production/requirem... For raspberry pis - I would go with the following, https://k0sproject.io https://k0sproject.io https://k3s.io https://k3s.io
    - don-code 5y ago
      Having tried Kubernetes on Raspberry Pi's once (my homelab is all SBCs, not all Pis but all ARM), there are two major pitfalls: 1. K3s may not be too heavy, but basically anything that runs on it is. For all but the most basic jobs, I was still looking at one, or maybe two, apps per SBC. Couple with 10/100 networking on many of these, and there's a lot of extra time/latency spent just on chattering across the network. 2. It's relatively rare that you're using _just_ the SBC - more likely is that you have some peripherals, or some local storage. That limits where your pods can schedule, but because of #1, you still end up with dedicated nodes for dedicated functions. I ended up pulling the plug on that project early, and went back to scheduling workloads manually. For instance, I have Transmission and Squid running on the Pi with the big flash drive. InfluxDB and Grafana live side by side, and also live next to the SDR (I take in some metrics by radio). It's all still containerized, just with manual `docker run`'s and versioned config files, instead of with Kubernetes.
    - kakoni 5y ago
      Tested both k3s and nomad on a lightweight cluster, k3s used about 25-30% from one CPU core constantly whereas nomad was running 1-2% tops.
denysvitali 5y ago
Are you sure Nomad orchestrates VMs? Do you have any relevant links? I thought it was mostly used for Docker containers, but of course you can run QEMU in a Docker container à la kubevirt
- dharmab 5y ago
  Nomad can orchestrate any process, though it's commonly used with containers (Docker or otherwise). However using to to orchestrate VMs sounds like a poor fit.
- Queue29 5y ago
  Nomad engineer here; Nomad supports Task Driver[0] plugins enabling you to orchestrate anything you need it to. Nomad ships with a QEMU task driver[1] built in. [0] https://www.nomadproject.io/docs/drivers https://www.nomadproject.io/docs/drivers [1] https://www.nomadproject.io/docs/drivers/qemu https://www.nomadproject.io/docs/drivers/qemu
  - bootcat 5y ago
    hi queue29, what are the minimum production requirements for nomad ? what is the cheapest machine i could use from digitalocean for a production task ?
    - Queue29 5y ago
      Check out the recommended requirements[0] documentation. In practice it depends greatly on how many client agents you want to manage (i.e. how large is your cluster). I personally run Nomad + Consul + Vault on the $6 AMD DO droplet to run a few website, and that node still has room to spare. [0] https://www.nomadproject.io/docs/install/production/requirements https://www.nomadproject.io/docs/install/production/requirem...
  - denysvitali 5y ago
    Uh, awesome! I think I missed this feature then. When was it introduced? I remember using Nomad back when it was still 0.x, and I don't remember ever reading about this feature. Awesome addition btw!
kosolam 5y ago
I wonder if there is anything at all in this space in addition to nomad?
- moondev 5y ago
  https://kubevirt.io https://kubevirt.io - basically a kubernetes operator for kvm virtual machines
  - kosolam 5y ago
    And who will orchestrate kubernetes?
    - moondev 5y ago
      I guess in this case the same thing that orchestrates/bootstraps nomad.
      - kosolam 5y ago
        Keep guessing my friend :-)
mrweasel 5y ago
What are you actually trying to do? I ask because I’m looking for a good tool to provision VMs on VMWare ESXi. That’s different though, Nomad won’t help me with that. In my case we use Ansible and Puppet to orchestrate VMs. Puppet for the stuff all VMs need, and Ansible for deploying custom software. Given that we at the most have 8 VMs that are completely identical, Nomad makes very little sense.
- edmcnulty101 5y ago
  Why use configuration management over dockerfiles and images?
  - nailer 5y ago
    Orchestration is more than just configuring individual instances, but rather being able to deploy your entire app which may include instances, load balancers, lambdas, data stores and/or other infrastructure.
    - edmcnulty101 5y ago
      No I mean why use Ansible and Puppet when you can create clean workloads from a docker image.
      - exdsq 5y ago
        They said they’re using VMs over Docker for better security control
        edmcnulty101 5y ago
        Ah gotcha. Thanks. Missed that.
        deivid 5y ago
        you can still export a container and run it as a vm with a bit of effort: https://blog.davidv.dev/docker-based-images-on-baremetal.html https://blog.davidv.dev/docker-based-images-on-baremetal.htm...
NexRebular 5y ago
We use Joyent's Triton in production. It's not something that can be plugged into existing systems easily as you need at least two dedicated server nodes for it, though.
- kosolam 5y ago
  Looks interesting..
toddnni 5y ago
Using none currently, the following seems to be enough: containers for apps and KVM+vagrant on a laptop for experimenting. However, for VM orchestration. OpenStack has been my choice for long time, but about to check KubeVirt and (old and underappreciated?) Apache CloudStack.
bryogenic 5y ago
qemu-system-x86_64
jambay 5y ago
HashiCorp employee here. Nomad has an open source core which is very capable. There is an Enterprise edition for those that want support and access to several Enterprise features. Does the open source edition address your non commercial criteria?
- encryptluks2 5y ago
  Does Nomad support typical VM orchestration like OpenStack or vSphere? It seems to me that it is more of just a generic driver that will do basic tasks.
  - proxysna 5y ago
    Nomad supports it. I ran Nomad instead of Kubevirt at some point. Functionality was great, but it missed the live migration. My usecase did not require it, but it was later pointed out to me.
- skimdesk 5y ago
  I also found that Nomad looked promising for VM orchestration. In my case I was specifically looking to run Firecracker VMs. Unfortunately I found the Firecracker community driver [1] to be difficult to work with when setting up networking beyond a simple "hello world". It does not play well with Nomad's CNI plugin [2] and I could not get a working setup where Nomad, CNI and the guest would agree on an IP. The firecracker driver is a community one, however, and the offical Qemu driver is most likely more polished. Did not try that one since it did not match my needs. Has anyone had success with the FC driver here? Edit: typos [1] https://www.nomadproject.io/plugins/drivers/community/firecracker-task-driver https://www.nomadproject.io/plugins/drivers/community/firecr... [2] https://github.com/cneira/firecracker-task-driver/issues/14 https://github.com/cneira/firecracker-task-driver/issues/14
rank0 5y ago
RHEL's cockpit project is fantastic.
oneplane 5y ago
"It depends"... as usual. Here comes a long one, but tl;dr: make sure you know your goal before you select your tools. Orchestration is a somewhat broad concept and can be interpreted at many levels. From a management perspective it is about what processes are acting to the benefit of the organisation to fulfil a business need, regardless of the amount of automation. Having a room full of humans vs. a couple of binaries and event logic doesn't matter if the orchestrated result is the same. Now, when it comes to your extra information you seem to be using microservices, but deploying them as individual virtual machines instead of just containers. While I applaud the choice of better isolation, virtual machines isn't the only way to do this. Firecracker does this too but much more efficient with the same protections (https://firecracker-microvm.github.io https://firecracker-microvm.github.io). So before going all-in on virtual machines, think about your goal and check to see if this is really the best way to do it. General business applications really don't have this level of isolation as the main factor in safeguarding applications as there are many other more fragile layers where the actual problems occur. A well-isolated application with no AAA on its interfaces might as well not be isolated at all from a security perspective. The selection of tools is probably going to be influenced by what else you are going to do. If you have automated infrastructure you'll probably want to tie in to that. Some automations have to trigger at deploy-time to (de-)provision at the right moment, while others have to simply react to the fact that something is needed and autonomously scale with the requirements. If your (micro)service needs a database, it probably also needs configuration to access it, and the database needs to be configured as well. Perhaps there are some networking components, and storage if you have object or block persistence needs. If you are running on AWS for example, your 'orchestrator' is EC2, but you still need to 'orchestrate the orchestrator' to start the correct configuration with all its dependencies prepared ahead of time. You'll probably want to prepare the virtual machine image so it actually contains everything it needs during build-time so it can just do the running at run-time. Packer comes to mind for such a job. Most systems that are 'good' got there because they had a lot of usage, because that is what generates data (feedback) on how well it does the job it needs to do. With that larger amount of usage comes a higher load from stakeholders and with that there tends to be a growth of commercial incentive turning a lot of products into some collection of commercial side-options or main-options. To get a bit closer to a possible answer, I'll sketch out two scenarios, a managed one and an unmanaged one (from a Layer 1 and 2 perspective, hardware). Scenario 1: Mostly managed primitives. Depending on what you already have you are likely to have an API that interfaces with the storage, networking and compute resources that are available to you. Depending on the systems, Terraform and Ansible can configure and prepare almost anything to your liking and make sure it stays in the desired state. When deploying an application as a virtual machine image, you'll probably have the following events happen: 1. SCM change triggers application build (Git and some CI solution) 2. Successful build triggers next integration step (CI) 3. Integration takes your build output and wraps it in a virtual machine image (Packer) 4. Successful integration triggers deployment step (CI) 5. Deployment step contacts your API of choice to let it know about the new image (Salt/Ansible/Terraform/FaaS/Webhook) 6. API registers the new image and you can start rolling over instances until all the old ones are gone Scenario 2: You have to DIY everything. Steps in there are essentially things you have to do before you can start thinking about scenario 1 which is what follows after this. 1. Make your hardware boot from the network 2. Setup a network boot server 3. Prepare self-starting hypervisor images of your choice (KVM-based, Xen-based, Hyper-V-Base, vmWare-based) 4. Make sure your DHCP and/or TFTP servers have logs you can access/parse 5. Once a server boots, it loads a self-starting image and depending on how you configured the images they will start out 'empty' with a well-known key or other credential 6. Use something like Ansible or SaltStack to discover any servers and select the 'empty'/'fresh' ones 7. Depending on your selected hypervisor, you'll now have to connect to a management API, control plane or simply write out a list of nodes somewhere (useful for small-scale SaltStack setups!) At this point, the 'compute' is done and ready for deployments. A big issue here is that we have many assumptions and the amount of new variables are so large that it's hard to get some more steps in. Next, you'll need storage which is much too broad to cover here, but it generally ends up with some sort of redundant NFS, iSCSI or more modern setup like Ceph. Those are all configurable via SaltStack or Ansible. You'll want networking after that, something like a vswitch with VLANs maybe. A very simple way to do this is start an OpenWRT or OPNSense VM with a bunch of pre-loaded virtual interfaces, subnets, and firewall rules. How this exactly works depends very much on what hypervisor and control plane you have. VMWare with NSX is very different from Proxmox with openvswitch (but they do the same thing). When you now want to 'deploy' a VM, you use your Ansible/Salt/Terraform-enabled infrastructure where you can 'update' the base configuration of a VM-based application and let Ansible/Salt/Terraform create new VMs and destroy the old ones. Configurations might be flat files written out to the network-booted compute nodes, or some configuration database in a controlplane behind an API. Now, all of this can be partially homelab-automated with things like Proxmox, Xen XCP-ng, VMWare etc. But as soon as you actually need more facilities like actual API-configued load balancers, databases, metrics, discovery you'll start home-cooking a complete OpenStack replacement. At that point, just get OpenStack or realise that the benefit of scale you get with Azure/GCP/AWS lets you do what makes you money instead of doing busywork that doesn't actually matter in the end in many cases.
bithavoc 5y ago
I use XCP-Ng for orchestration[0] and Xen-Orchestra[1] for management, both are open-source. Xen-Orchestra is open-core but if you build from sources or use an unofficial docker image, you can get a lot of the commercial offering back for free at your own risk. With Xen-Orchestra you manage one or more XCP-Ng Servers, combine them in pools, live migrate between hosts, patch your hosts and schedule VM backups to S3. It's a killer combination. [0] https://xcp-ng.org/ https://xcp-ng.org/ [1] https://xen-orchestra.com/ https://xen-orchestra.com/
Notanothertoo 5y ago
On prem Vms setup manually. I cloud provider Vms have their own primitives (aws asg), with something like terraform. Then a kube flavor for all the real orchestration.
AtlasBarfed 5y ago
Orchestration is a tough one because the second you bring it up, it devolves into a bunch of enterprise offerings and datastores and servers and holy wars and evaluations and managed services and horrendously complicated setup. Workflow engines, schedulers, all supporting "standards" but none interop without a million lines of glue code. I've ended up writing basically this: - Groovy for a lot of syntactic power, decent performance, ability to make workflow-ish scripts - GPars for orchestration primitives to do stuff either serially or in parallel. - a configuration service to have configuration scoped at the global/environment/cluster/datacenter/rack/individual node levels - an access layer for executing remote commands and getting stdout/stderr back, and for sending, receiving files. It can be SSH, kubectl, AWS SSM, dockerrun, etc. I've had to rewrite the core framework four times, but it has enabled me to do complicated orchestrations such as spawn clustered databases, spawn fleets of load generators, data loaders, or api servers, tear them down, etc. I can code distributed backups and restores, and other cluster-wide "applications" with this as the basis. I can run things from the IDE. I can run locally or from remote servers. I can CRON them or whatever. I can debug them from IDE. I can do adhoc/arbitrary cluster-wide operations/commands using the Groovy Console. I will probably OSS it at some point, but it's not a finsihed product. But after three years I can't imagine doing what I do with it anywhere else. It could be called by workflow engines, it can call out to terraform and other tools. It can do multicloud (with enough code :-), it can do mixed access / deployments (with enough code :-) I guess the other limitation is the basic problem with a single orchestration node: probably won't easily scale to the 1000s of VM/nodes easily, or as efficiently as some other distributed coordinator could, but those things all come with the one-orchestration/integration/workflow-to-rule-them-all products. I don't want to be that.
- zmmmmm 5y ago
  Would love to see this if you ever get time to open source it!
rank0 5y ago
I’ve been playing with lxd and it’s been very pleasant so far. Anyone else got experience?

47 points by kosolam 5y ago

Looking into hashicorp nomad but I wonder if there is a good non commercial alternative?

mountainriver 5y ago

Kubevirt

zhengiszen 5y ago

Saltstack with saltcloud looks like a perfect fit : https://fr.slideshare.net/MoRawi/salt-cloud-vmwareorchestration https://fr.slideshare.net/MoRawi/salt-cloud-vmwareorchestrat...

bserge 5y ago

bootcat 5y ago

I am looking into something likewise, I liked caprover from this list, https://github.com/guettli/open-source-paas https://github.com/guettli/open-source-paas I feel nomad is a bit resource intensive for my use case,

proxysna 5y ago
System requirements for nomad on Hashicopr's assume a relatively high-load. Running a single Rpi 3b+ master with a 10-13 clients at home was not resource intensive at all. Master was effectively running on idle temps and opening something like mc or htop took more resources than the server itself.
capableweb 5y ago
> resource intensive Resource intensive? I run Nomad on a bunch of Raspberry PIs at home, not sure what orchestrator is more lightweight than Nomad honestly.
- bootcat 5y ago
  thank you for sharing this - let me try from my end. My comment and denial towards using nomad was based on their hardware requirements - https://www.nomadproject.io/docs/install/production/requirements https://www.nomadproject.io/docs/install/production/requirem... For raspberry pis - I would go with the following, https://k0sproject.io https://k0sproject.io https://k3s.io https://k3s.io
  - don-code 5y ago
    Having tried Kubernetes on Raspberry Pi's once (my homelab is all SBCs, not all Pis but all ARM), there are two major pitfalls: 1. K3s may not be too heavy, but basically anything that runs on it is. For all but the most basic jobs, I was still looking at one, or maybe two, apps per SBC. Couple with 10/100 networking on many of these, and there's a lot of extra time/latency spent just on chattering across the network. 2. It's relatively rare that you're using _just_ the SBC - more likely is that you have some peripherals, or some local storage. That limits where your pods can schedule, but because of #1, you still end up with dedicated nodes for dedicated functions. I ended up pulling the plug on that project early, and went back to scheduling workloads manually. For instance, I have Transmission and Squid running on the Pi with the big flash drive. InfluxDB and Grafana live side by side, and also live next to the SDR (I take in some metrics by radio). It's all still containerized, just with manual `docker run`'s and versioned config files, instead of with Kubernetes.
  - kakoni 5y ago
    Tested both k3s and nomad on a lightweight cluster, k3s used about 25-30% from one CPU core constantly whereas nomad was running 1-2% tops.

denysvitali 5y ago

Are you sure Nomad orchestrates VMs? Do you have any relevant links? I thought it was mostly used for Docker containers, but of course you can run QEMU in a Docker container à la kubevirt

dharmab 5y ago
Nomad can orchestrate any process, though it's commonly used with containers (Docker or otherwise). However using to to orchestrate VMs sounds like a poor fit.
Queue29 5y ago
Nomad engineer here; Nomad supports Task Driver[0] plugins enabling you to orchestrate anything you need it to. Nomad ships with a QEMU task driver[1] built in. [0] https://www.nomadproject.io/docs/drivers https://www.nomadproject.io/docs/drivers [1] https://www.nomadproject.io/docs/drivers/qemu https://www.nomadproject.io/docs/drivers/qemu
- bootcat 5y ago
  hi queue29, what are the minimum production requirements for nomad ? what is the cheapest machine i could use from digitalocean for a production task ?
  - Queue29 5y ago
    Check out the recommended requirements[0] documentation. In practice it depends greatly on how many client agents you want to manage (i.e. how large is your cluster). I personally run Nomad + Consul + Vault on the $6 AMD DO droplet to run a few website, and that node still has room to spare. [0] https://www.nomadproject.io/docs/install/production/requirements https://www.nomadproject.io/docs/install/production/requirem...
- denysvitali 5y ago
  Uh, awesome! I think I missed this feature then. When was it introduced? I remember using Nomad back when it was still 0.x, and I don't remember ever reading about this feature. Awesome addition btw!

kosolam 5y ago

I wonder if there is anything at all in this space in addition to nomad?

moondev 5y ago
https://kubevirt.io https://kubevirt.io - basically a kubernetes operator for kvm virtual machines
- kosolam 5y ago
  And who will orchestrate kubernetes?
  - moondev 5y ago
    I guess in this case the same thing that orchestrates/bootstraps nomad.
    - kosolam 5y ago
      Keep guessing my friend :-)

mrweasel 5y ago

What are you actually trying to do? I ask because I’m looking for a good tool to provision VMs on VMWare ESXi. That’s different though, Nomad won’t help me with that. In my case we use Ansible and Puppet to orchestrate VMs. Puppet for the stuff all VMs need, and Ansible for deploying custom software. Given that we at the most have 8 VMs that are completely identical, Nomad makes very little sense.

edmcnulty101 5y ago
Why use configuration management over dockerfiles and images?
- nailer 5y ago
  Orchestration is more than just configuring individual instances, but rather being able to deploy your entire app which may include instances, load balancers, lambdas, data stores and/or other infrastructure.
  - edmcnulty101 5y ago
    No I mean why use Ansible and Puppet when you can create clean workloads from a docker image.
    - exdsq 5y ago
      They said they’re using VMs over Docker for better security control
      - edmcnulty101 5y ago
        Ah gotcha. Thanks. Missed that.
      - deivid 5y ago
        you can still export a container and run it as a vm with a bit of effort: https://blog.davidv.dev/docker-based-images-on-baremetal.html https://blog.davidv.dev/docker-based-images-on-baremetal.htm...

NexRebular 5y ago

We use Joyent's Triton in production. It's not something that can be plugged into existing systems easily as you need at least two dedicated server nodes for it, though.

kosolam 5y ago
Looks interesting..

toddnni 5y ago

Using none currently, the following seems to be enough: containers for apps and KVM+vagrant on a laptop for experimenting. However, for VM orchestration. OpenStack has been my choice for long time, but about to check KubeVirt and (old and underappreciated?) Apache CloudStack.

bryogenic 5y ago

qemu-system-x86_64

jambay 5y ago

HashiCorp employee here. Nomad has an open source core which is very capable. There is an Enterprise edition for those that want support and access to several Enterprise features. Does the open source edition address your non commercial criteria?

encryptluks2 5y ago
Does Nomad support typical VM orchestration like OpenStack or vSphere? It seems to me that it is more of just a generic driver that will do basic tasks.
- proxysna 5y ago
  Nomad supports it. I ran Nomad instead of Kubevirt at some point. Functionality was great, but it missed the live migration. My usecase did not require it, but it was later pointed out to me.
skimdesk 5y ago
I also found that Nomad looked promising for VM orchestration. In my case I was specifically looking to run Firecracker VMs. Unfortunately I found the Firecracker community driver [1] to be difficult to work with when setting up networking beyond a simple "hello world". It does not play well with Nomad's CNI plugin [2] and I could not get a working setup where Nomad, CNI and the guest would agree on an IP. The firecracker driver is a community one, however, and the offical Qemu driver is most likely more polished. Did not try that one since it did not match my needs. Has anyone had success with the FC driver here? Edit: typos [1] https://www.nomadproject.io/plugins/drivers/community/firecracker-task-driver https://www.nomadproject.io/plugins/drivers/community/firecr... [2] https://github.com/cneira/firecracker-task-driver/issues/14 https://github.com/cneira/firecracker-task-driver/issues/14

rank0 5y ago

RHEL's cockpit project is fantastic.

oneplane 5y ago

"It depends"... as usual. Here comes a long one, but tl;dr: make sure you know your goal before you select your tools. Orchestration is a somewhat broad concept and can be interpreted at many levels. From a management perspective it is about what processes are acting to the benefit of the organisation to fulfil a business need, regardless of the amount of automation. Having a room full of humans vs. a couple of binaries and event logic doesn't matter if the orchestrated result is the same. Now, when it comes to your extra information you seem to be using microservices, but deploying them as individual virtual machines instead of just containers. While I applaud the choice of better isolation, virtual machines isn't the only way to do this. Firecracker does this too but much more efficient with the same protections (https://firecracker-microvm.github.io https://firecracker-microvm.github.io). So before going all-in on virtual machines, think about your goal and check to see if this is really the best way to do it. General business applications really don't have this level of isolation as the main factor in safeguarding applications as there are many other more fragile layers where the actual problems occur. A well-isolated application with no AAA on its interfaces might as well not be isolated at all from a security perspective. The selection of tools is probably going to be influenced by what else you are going to do. If you have automated infrastructure you'll probably want to tie in to that. Some automations have to trigger at deploy-time to (de-)provision at the right moment, while others have to simply react to the fact that something is needed and autonomously scale with the requirements. If your (micro)service needs a database, it probably also needs configuration to access it, and the database needs to be configured as well. Perhaps there are some networking components, and storage if you have object or block persistence needs. If you are running on AWS for example, your 'orchestrator' is EC2, but you still need to 'orchestrate the orchestrator' to start the correct configuration with all its dependencies prepared ahead of time. You'll probably want to prepare the virtual machine image so it actually contains everything it needs during build-time so it can just do the running at run-time. Packer comes to mind for such a job. Most systems that are 'good' got there because they had a lot of usage, because that is what generates data (feedback) on how well it does the job it needs to do. With that larger amount of usage comes a higher load from stakeholders and with that there tends to be a growth of commercial incentive turning a lot of products into some collection of commercial side-options or main-options. To get a bit closer to a possible answer, I'll sketch out two scenarios, a managed one and an unmanaged one (from a Layer 1 and 2 perspective, hardware). Scenario 1: Mostly managed primitives. Depending on what you already have you are likely to have an API that interfaces with the storage, networking and compute resources that are available to you. Depending on the systems, Terraform and Ansible can configure and prepare almost anything to your liking and make sure it stays in the desired state. When deploying an application as a virtual machine image, you'll probably have the following events happen: 1. SCM change triggers application build (Git and some CI solution) 2. Successful build triggers next integration step (CI) 3. Integration takes your build output and wraps it in a virtual machine image (Packer) 4. Successful integration triggers deployment step (CI) 5. Deployment step contacts your API of choice to let it know about the new image (Salt/Ansible/Terraform/FaaS/Webhook) 6. API registers the new image and you can start rolling over instances until all the old ones are gone Scenario 2: You have to DIY everything. Steps in there are essentially things you have to do before you can start thinking about scenario 1 which is what follows after this. 1. Make your hardware boot from the network 2. Setup a network boot server 3. Prepare self-starting hypervisor images of your choice (KVM-based, Xen-based, Hyper-V-Base, vmWare-based) 4. Make sure your DHCP and/or TFTP servers have logs you can access/parse 5. Once a server boots, it loads a self-starting image and depending on how you configured the images they will start out 'empty' with a well-known key or other credential 6. Use something like Ansible or SaltStack to discover any servers and select the 'empty'/'fresh' ones 7. Depending on your selected hypervisor, you'll now have to connect to a management API, control plane or simply write out a list of nodes somewhere (useful for small-scale SaltStack setups!) At this point, the 'compute' is done and ready for deployments. A big issue here is that we have many assumptions and the amount of new variables are so large that it's hard to get some more steps in. Next, you'll need storage which is much too broad to cover here, but it generally ends up with some sort of redundant NFS, iSCSI or more modern setup like Ceph. Those are all configurable via SaltStack or Ansible. You'll want networking after that, something like a vswitch with VLANs maybe. A very simple way to do this is start an OpenWRT or OPNSense VM with a bunch of pre-loaded virtual interfaces, subnets, and firewall rules. How this exactly works depends very much on what hypervisor and control plane you have. VMWare with NSX is very different from Proxmox with openvswitch (but they do the same thing). When you now want to 'deploy' a VM, you use your Ansible/Salt/Terraform-enabled infrastructure where you can 'update' the base configuration of a VM-based application and let Ansible/Salt/Terraform create new VMs and destroy the old ones. Configurations might be flat files written out to the network-booted compute nodes, or some configuration database in a controlplane behind an API. Now, all of this can be partially homelab-automated with things like Proxmox, Xen XCP-ng, VMWare etc. But as soon as you actually need more facilities like actual API-configued load balancers, databases, metrics, discovery you'll start home-cooking a complete OpenStack replacement. At that point, just get OpenStack or realise that the benefit of scale you get with Azure/GCP/AWS lets you do what makes you money instead of doing busywork that doesn't actually matter in the end in many cases.

bithavoc 5y ago

I use XCP-Ng for orchestration[0] and Xen-Orchestra[1] for management, both are open-source. Xen-Orchestra is open-core but if you build from sources or use an unofficial docker image, you can get a lot of the commercial offering back for free at your own risk. With Xen-Orchestra you manage one or more XCP-Ng Servers, combine them in pools, live migrate between hosts, patch your hosts and schedule VM backups to S3. It's a killer combination. [0] https://xcp-ng.org/ https://xcp-ng.org/ [1] https://xen-orchestra.com/ https://xen-orchestra.com/

Notanothertoo 5y ago

On prem Vms setup manually. I cloud provider Vms have their own primitives (aws asg), with something like terraform. Then a kube flavor for all the real orchestration.

AtlasBarfed 5y ago

Orchestration is a tough one because the second you bring it up, it devolves into a bunch of enterprise offerings and datastores and servers and holy wars and evaluations and managed services and horrendously complicated setup. Workflow engines, schedulers, all supporting "standards" but none interop without a million lines of glue code. I've ended up writing basically this: - Groovy for a lot of syntactic power, decent performance, ability to make workflow-ish scripts - GPars for orchestration primitives to do stuff either serially or in parallel. - a configuration service to have configuration scoped at the global/environment/cluster/datacenter/rack/individual node levels - an access layer for executing remote commands and getting stdout/stderr back, and for sending, receiving files. It can be SSH, kubectl, AWS SSM, dockerrun, etc. I've had to rewrite the core framework four times, but it has enabled me to do complicated orchestrations such as spawn clustered databases, spawn fleets of load generators, data loaders, or api servers, tear them down, etc. I can code distributed backups and restores, and other cluster-wide "applications" with this as the basis. I can run things from the IDE. I can run locally or from remote servers. I can CRON them or whatever. I can debug them from IDE. I can do adhoc/arbitrary cluster-wide operations/commands using the Groovy Console. I will probably OSS it at some point, but it's not a finsihed product. But after three years I can't imagine doing what I do with it anywhere else. It could be called by workflow engines, it can call out to terraform and other tools. It can do multicloud (with enough code :-), it can do mixed access / deployments (with enough code :-) I guess the other limitation is the basic problem with a single orchestration node: probably won't easily scale to the 1000s of VM/nodes easily, or as efficiently as some other distributed coordinator could, but those things all come with the one-orchestration/integration/workflow-to-rule-them-all products. I don't want to be that.

zmmmmm 5y ago
Would love to see this if you ever get time to open source it!

rank0 5y ago

I’ve been playing with lxd and it’s been very pleasant so far. Anyone else got experience?