6 ms·

On a related note, is anyone aware of any tool(s) to declaratively describe an application architecture and automate its deployment on something like AWS or in

On a related note, is anyone aware of any tool(s) to declaratively describe an application architecture and automate its deployment on something like AWS or in a single virtual machine + docker? For example "I want 10 replicated microservice1 processes and 5 replicated microservice2 processes in front of a nginx load balancer. Also, throw in a Redis and a PostgreSQL database." This would be immensely useful for (one man show) developers who don't necessarily have the time or experience to "properly" setup and fine tune infrastructure. I think what I'm describing is known as "Infrastructure as code" in DevOps parlance. Edit: Since this question is getting many replies, I'll be a bit more specific in what I am looking for. Is there a tool that would let me describe the infrastructure and deploy it on a given cloud provider but also have the ability to deploy the same infrastructure on my local machine (using VMs/docker) for development purposes.

pavlov 12y ago
Sounds like you want AWS OpsWorks? https://aws.amazon.com/opsworks/ https://aws.amazon.com/opsworks/
Denzel 12y ago
Salt/salt-cloud operates as you describe. It allows you to define any number of virtual machines associated to a cloud provider 'profile', i.e., instance type, size, platform, etc.
kikibobo69 12y ago
We're working on a very low-tech version of this at Zalando, here: https://github.com/zalando/aws-minion https://github.com/zalando/aws-minion
ryanbrunner 12y ago
Cloud66 can handle most of what you describe using manifest files. You can have a relatively simple yml file describing what types of servers exist in your stack - something like "I need Postgres servers, Redis, a RabbitMQ server, oh and here's the 3 different types of process servers I have and also how my main web server functions." There's still a point where it's beneficial to split an application into seperate stacks, in which case the manifest concept breaks down and you're stuck doing a lot of work yourself, but they can take you a fairly long way.
- olalonde 12y ago
  > There's still a point where it's beneficial to split an application into seperate stacks Do you mean like having a different "infrastructure manifest" for each micro-service? In that case, couldn't each micro-service specify which other micro-service it depends upon? For example, I could have a central "API gateway" service which would specify all the other micro-services it depends upon. This reminds me a bit of a Gemfile/Package.json but for processes instead of libraries. On the other hand, I can easily see how this could turn into a dependency hell.
detaro 12y ago
For OpenStack clouds, the HEAT project is working on that.
aphexairlines 12y ago
NixOps: https://nixos.org/nixops/ https://nixos.org/nixops/
- iElectric2 12y ago
  Yeah, NixOS + NixOps declaratively define logical and physical part of an machine network.
matthewmacleod 12y ago
Are you looking for something like https://www.tutum.co https://www.tutum.co maybe? Locally, you'd use http://www.fig.sh http://www.fig.sh to describe and run your app. You can then import this description into Tutum and run it on e.g. DO or AWS.
twic 12y ago
We built that at a previous employer: https://github.com/tim-group/stackbuilder https://github.com/tim-group/stackbuilder https://devblog.timgroup.com/2012/12/20/towards-a-rich-domain-model-for-infrastructure/ https://devblog.timgroup.com/2012/12/20/towards-a-rich-domai... The documentation is pretty rubbish, but here's an example config file: https://github.com/tim-group/stackbuilder-config/blob/master/stack.rb https://github.com/tim-group/stackbuilder-config/blob/master... It uses libvirt for provisioning, and Puppet for configuration. The coupling is extensive but not tight. It's also coupled to a specific way of deploying Java apps. There's a sort of backdoor that would let you deploy other kinds of apps by writing some Puppet code. I do think the basic ideas are good, though. Possibly more of an object of study than a tool to actually use!
chrisan 12y ago
> Edit: Since this question is getting many replies, I'll be a bit more specific in what I am looking for. Is there a tool that would let me describe the infrastructure and deploy it on a given cloud provider but also have the ability to deploy the same infrastructure on my local machine (using VMs/docker) for development purposes. Seems like the https://www.hashicorp.com/ https://www.hashicorp.com/ guys would have something, perhaps Terraform/Atlas? I have not done this yet obviously but am in the same boat as you for our company goals this year
- jacques_chester 12y ago
  What I liked most about Terraform was the explicit separation of planning and execution.
jacques_chester 12y ago
> Is there a tool that would let me describe the infrastructure and deploy it on a given cloud provider but also have the ability to deploy the same infrastructure on my local machine (using VMs/docker) for development purposes. Depending on your concept of "deploy", I can point you to two options, both produced by the company I work for. If you think in terms of controlling from the VM up, look at BOSH[1]. It's an IaaS deployment / management / update tool which has been used in production for several years. You create two yaml files: a release manifest (describing the software components of your system) and a deployment manifest (describing the computers and networks of your system and how to map components to them). This separation means that a single release manifest can be applied to many different actual deployments. At my day job I deploy a large PaaS release into a single virtual machine. My colleagues deploy exactly the same release into AWS with thousands of machines. Systems defined using BOSH can be deployed to AWS, vSphere, OpenStack or Warden (a containerisation system). For local development, the Warden backend enables "BOSH lite"[2] -- releasing and deploying into a virtual machine that hosts a local cloud of containers. If you want to go the next level up to a proper PaaS, you can try Cloud Foundry[3]. It's defined as a BOSH release[4], which is how at my day job my co-workers and I are able to work both locally and at scale on identical software. In Cloud Foundry we distinguish between stateless (apps) and stateful (services) software, and we provide a simple way to connect them. The easiest way to play with it is to log into Pivotal Web Services. IBM Bluemix is the same software and is also open to the public. So for example, with Cloud Foundry, if you want ten copies of your app, you do this: $ cf scale my_app -i 10 And it'll be scaled up to 10 instances. Need more RAM? $ cf scale my_app -m 2G Now each node gets 2Gb. And so on. [1] https://github.com/cloudfoundry/bosh https://github.com/cloudfoundry/bosh [2] https://github.com/cloudfoundry/bosh-lite https://github.com/cloudfoundry/bosh-lite [3] http://docs.cloudfoundry.org/ http://docs.cloudfoundry.org/ [4] https://github.com/cloudfoundry/cf-release https://github.com/cloudfoundry/cf-release
illza 12y ago
There are some limitations still, but ubuntu juju [1] was the first thing that came to mind that somewhat fulfills the requirements. [1] https://juju.ubuntu.com/ https://juju.ubuntu.com/
morgante 12y ago
Docker with CoreOS and fleet will get you pretty far on that front. You can just specify that you want 6 of container x running across the cluster, and 3 of container y. Supplementing this with Hashicorp tools like Terraform and Consul can get you to the last step.
ossreality 12y ago
Kubernetes is what you want. Yay for being hellbanned with the right answer. This is one time I wouldn't mind being repeated by someone that isn't hellbanned because Kubernetes is a better choice than the other options suggested here.
parasubvert 12y ago
Cloud Foundry + BOSH, IME, does exactly that. http://bosh.cloudfoundry.org http://bosh.cloudfoundry.org https://github.com/cloudfoundry/bosh-lite https://github.com/cloudfoundry/bosh-lite https://github.com/CloudCredo/dockpush https://github.com/CloudCredo/dockpush You download a BOSH release for your infrastructure, tweak the manifest for your chosen cloud, and push the deployment out. If you make a change, you just push it out. Cloud Foundry itself (the elastic runtime) is a BOSH application that includes an HAPRoxy load balancer, dynamic router, and allows you to easily push/scale the stateless micro services and declaratively wire them up with other services you've deployed (with CF or Bosh). It can run on AWS, vSphere, OpenStack, or on your own Vagrant VM in containers (bosh-lite). Or you can use Pivotal Web Services public cloud: http://run.pivotal.io http://run.pivotal.io BOSH requires some wrestling early on but I find it to be a fascinating project.

pavlov 12y ago

Sounds like you want AWS OpsWorks? https://aws.amazon.com/opsworks/ https://aws.amazon.com/opsworks/

Denzel 12y ago

Salt/salt-cloud operates as you describe. It allows you to define any number of virtual machines associated to a cloud provider 'profile', i.e., instance type, size, platform, etc.

kikibobo69 12y ago

We're working on a very low-tech version of this at Zalando, here: https://github.com/zalando/aws-minion https://github.com/zalando/aws-minion

ryanbrunner 12y ago

Cloud66 can handle most of what you describe using manifest files. You can have a relatively simple yml file describing what types of servers exist in your stack - something like "I need Postgres servers, Redis, a RabbitMQ server, oh and here's the 3 different types of process servers I have and also how my main web server functions." There's still a point where it's beneficial to split an application into seperate stacks, in which case the manifest concept breaks down and you're stuck doing a lot of work yourself, but they can take you a fairly long way.

olalonde 12y ago
> There's still a point where it's beneficial to split an application into seperate stacks Do you mean like having a different "infrastructure manifest" for each micro-service? In that case, couldn't each micro-service specify which other micro-service it depends upon? For example, I could have a central "API gateway" service which would specify all the other micro-services it depends upon. This reminds me a bit of a Gemfile/Package.json but for processes instead of libraries. On the other hand, I can easily see how this could turn into a dependency hell.

detaro 12y ago

For OpenStack clouds, the HEAT project is working on that.

aphexairlines 12y ago

NixOps: https://nixos.org/nixops/ https://nixos.org/nixops/

iElectric2 12y ago
Yeah, NixOS + NixOps declaratively define logical and physical part of an machine network.

matthewmacleod 12y ago

Are you looking for something like https://www.tutum.co https://www.tutum.co maybe? Locally, you'd use http://www.fig.sh http://www.fig.sh to describe and run your app. You can then import this description into Tutum and run it on e.g. DO or AWS.

twic 12y ago

We built that at a previous employer: https://github.com/tim-group/stackbuilder https://github.com/tim-group/stackbuilder https://devblog.timgroup.com/2012/12/20/towards-a-rich-domain-model-for-infrastructure/ https://devblog.timgroup.com/2012/12/20/towards-a-rich-domai... The documentation is pretty rubbish, but here's an example config file: https://github.com/tim-group/stackbuilder-config/blob/master/stack.rb https://github.com/tim-group/stackbuilder-config/blob/master... It uses libvirt for provisioning, and Puppet for configuration. The coupling is extensive but not tight. It's also coupled to a specific way of deploying Java apps. There's a sort of backdoor that would let you deploy other kinds of apps by writing some Puppet code. I do think the basic ideas are good, though. Possibly more of an object of study than a tool to actually use!

chrisan 12y ago

> Edit: Since this question is getting many replies, I'll be a bit more specific in what I am looking for. Is there a tool that would let me describe the infrastructure and deploy it on a given cloud provider but also have the ability to deploy the same infrastructure on my local machine (using VMs/docker) for development purposes. Seems like the https://www.hashicorp.com/ https://www.hashicorp.com/ guys would have something, perhaps Terraform/Atlas? I have not done this yet obviously but am in the same boat as you for our company goals this year

jacques_chester 12y ago
What I liked most about Terraform was the explicit separation of planning and execution.

jacques_chester 12y ago

> Is there a tool that would let me describe the infrastructure and deploy it on a given cloud provider but also have the ability to deploy the same infrastructure on my local machine (using VMs/docker) for development purposes. Depending on your concept of "deploy", I can point you to two options, both produced by the company I work for. If you think in terms of controlling from the VM up, look at BOSH[1]. It's an IaaS deployment / management / update tool which has been used in production for several years. You create two yaml files: a release manifest (describing the software components of your system) and a deployment manifest (describing the computers and networks of your system and how to map components to them). This separation means that a single release manifest can be applied to many different actual deployments. At my day job I deploy a large PaaS release into a single virtual machine. My colleagues deploy exactly the same release into AWS with thousands of machines. Systems defined using BOSH can be deployed to AWS, vSphere, OpenStack or Warden (a containerisation system). For local development, the Warden backend enables "BOSH lite"[2] -- releasing and deploying into a virtual machine that hosts a local cloud of containers. If you want to go the next level up to a proper PaaS, you can try Cloud Foundry[3]. It's defined as a BOSH release[4], which is how at my day job my co-workers and I are able to work both locally and at scale on identical software. In Cloud Foundry we distinguish between stateless (apps) and stateful (services) software, and we provide a simple way to connect them. The easiest way to play with it is to log into Pivotal Web Services. IBM Bluemix is the same software and is also open to the public. So for example, with Cloud Foundry, if you want ten copies of your app, you do this: $ cf scale my_app -i 10 And it'll be scaled up to 10 instances. Need more RAM? $ cf scale my_app -m 2G Now each node gets 2Gb. And so on. [1] https://github.com/cloudfoundry/bosh https://github.com/cloudfoundry/bosh [2] https://github.com/cloudfoundry/bosh-lite https://github.com/cloudfoundry/bosh-lite [3] http://docs.cloudfoundry.org/ http://docs.cloudfoundry.org/ [4] https://github.com/cloudfoundry/cf-release https://github.com/cloudfoundry/cf-release

illza 12y ago

There are some limitations still, but ubuntu juju [1] was the first thing that came to mind that somewhat fulfills the requirements. [1] https://juju.ubuntu.com/ https://juju.ubuntu.com/

morgante 12y ago

Docker with CoreOS and fleet will get you pretty far on that front. You can just specify that you want 6 of container x running across the cluster, and 3 of container y. Supplementing this with Hashicorp tools like Terraform and Consul can get you to the last step.

ossreality 12y ago

Kubernetes is what you want. Yay for being hellbanned with the right answer. This is one time I wouldn't mind being repeated by someone that isn't hellbanned because Kubernetes is a better choice than the other options suggested here.

parasubvert 12y ago

Cloud Foundry + BOSH, IME, does exactly that. http://bosh.cloudfoundry.org http://bosh.cloudfoundry.org https://github.com/cloudfoundry/bosh-lite https://github.com/cloudfoundry/bosh-lite https://github.com/CloudCredo/dockpush https://github.com/CloudCredo/dockpush You download a BOSH release for your infrastructure, tweak the manifest for your chosen cloud, and push the deployment out. If you make a change, you just push it out. Cloud Foundry itself (the elastic runtime) is a BOSH application that includes an HAPRoxy load balancer, dynamic router, and allows you to easily push/scale the stateless micro services and declaratively wire them up with other services you've deployed (with CF or Bosh). It can run on AWS, vSphere, OpenStack, or on your own Vagrant VM in containers (bosh-lite). Or you can use Pivotal Web Services public cloud: http://run.pivotal.io http://run.pivotal.io BOSH requires some wrestling early on but I find it to be a fascinating project.