If you already have all your virtualisation covered, it becomes much simpler: - Find out what APIs/Configuration stores are available - Put long-lived depl

If you already have all your virtualisation covered, it becomes much simpler: - Find out what APIs/Configuration stores are available - Put long-lived deployment-static things in Terraform/Salt - Put show-lived deployment-bound things in CI (and then Ansible/Salt inside there optionally) The CI-things can trigger OpenStack/XCP/Proxmox/Vmware and tell it to re-deploy your stuff. If you have many 'things', Nomad, Kubevirt (with ArgoCD or Flux), Tekton, Teletraan, Spinnaker are your best options in order of growing magnitude. For me, if I were to run a small VMWare set for example, I would select Terraform as my orchestrator of choice. Then in my CI (GitLab-CI) I would make a step that once Packer has built a new VM image, I would update a file listing my to-be-deployed images and automatically commit it to a state repository. I would then also have Atlantis running which notices the change and runs terraform for me. It would add/remove all the resources needed for my VM to work with the correct network and storage. I would leave discovery and joining/leaving load balancers etc. to be the job of the VM image, perhaps via Cloud-init or Consul or something like that. If I were to run a bit of a bigger setup, I might run Proxmox and Ceph, combine Kubevirt with that and run ArgoCD instead. Alternatively I might check out https://harvesterhci.io https://harvesterhci.io (via https://medium.com/@rmin.aminian/deploy-harvester-in-proxmox-environment-part1-network-and-management-node-c5e4c3f75252 https://medium.com/@rmin.aminian/deploy-harvester-in-proxmox...). The problem with applications-as-virtual-machines is that those are surrounded by tools and ideas that they live reasonably long lives and consume quite some resources. You won't really see someone run 1000 VMs with 0.15 CPU and 64MB RAM as a microservice 'instance' but you'd see that happen a billion times with containers. Managing the lifecycle of a virtual machine means you have to manage its storage and networking too, and responding to outages of any of the primitives (net, cpu, storage) is much more cumbersome than a container since the container doesn't need to know about any of that, it just receives it from the host. A secondary problem is that whenever you end up in orchestration land, people tend to think it's complex tooling, or that containers are complex or that Kubernetes is complex. To a degree that is true, but when you start reinventing those wheels you just end up implementing 50% of those existing solutions with 0% documentation and nobody in the world will be able to help you because it contains so much nonstandard stuff. A lot of CI/CD and management processes are just 'state machines' or control loops. This is also what most services in Kubernetes are, and most services in public clouds. They just respond to the state of the world vs. the desired state. Whenever you're not using what already exists, you're essentially just re-implementing those exact same control loops/state machines in a less-robust and untested way.