6 ms·

While this looks nice, part of me can't help but be annoyed by yet another deployment option on AWS. We now have CloudFormation, Elastic Beanstalk (which can ta

by STRML 12y ago

While this looks nice, part of me can't help but be annoyed by yet another deployment option on AWS. We now have CloudFormation, Elastic Beanstalk (which can take many forms, including Docker), CodeDeploy, and Opsworks. I can imagine how, for a new user, it's utterly baffling which of these options is the best for the longterm, with the least friction. I use OpsWorks quite a bit but have found it very challenging, and the feedback cycle when attempting to develop new cookbooks is excruciatingly slow. All I want, personally, is a system that uses a set of interchangeable scripts that represent dependencies, so my server configs can live in version control. It doesn't even need to run on multiple OSs (which seems to be a central tenet of Chef). It just needs to deploy/rollback with zero downtime, and ideally autoscale as quickly as possible. Is this it? Is there any way to know without spending weeks fleshing out how it works?

sciurus 12y ago
CloudFormation is really a different beast. It's focused on creating and managing related collections of AWS resources, not on application deployment. Anyone doing, well, pretty much anything on AWS should probably be using Cloudformation. http://aws.amazon.com/cloudformation/ http://aws.amazon.com/cloudformation/
kxo 12y ago
How about an Amazon-managed Docker (not shit ElasticBeanstalk) infrastructure/config management system? If they handled things like EBS integration, orchestration, ELB integration, and all of the usual stuff, I'd be sold in a second.
- STRML 12y ago
  From my understanding, after talking to them at Web Summit, something like this is in the works. And they're aware of how incredibly complicated the AWS console is becoming. I don't know how or when they plan to address it, however.
  - mentat 12y ago
    They promise lots of things (like cross region VPC peering) to person to person at Summits...
    - STRML 12y ago
      Yes, I am still waiting for something prettier than software VPN or OpenS/WAN. Their docs show 5 ways to connect VPCs, all of them complex, all of them with significant downsides. One would think this sort of thing would be easier.
- tedchs 12y ago
  On Google Cloud Platform, there is a new service called Google Container Engine (currently in alpha). It uses Kubernetes to manage Docker containers, running on Google Compute Engine. You could also run Kubernetes on EC2 if you wanted to.
- vidarh 12y ago
  It takes a couple of days to get Ansible playbooks set up to deploy CoreOS VMs onto EC2 with a decent Docker setup and EBS. I know, I've just done it. And I don't like Ansible, and haven't used it seriously before (I use it for project because it's on a contract and it's easy-ish for them to find people who understands Ansible if/when it's needed) so it took longer than I'd have liked. Frankly, I don't understand why people are so happy to lock themselves tighter into AWS.
zackangelo 12y ago
I've found developing OpsWorks recipes directly on the instance to be the best remedy for the slow feedback cycle. Your cookbook is located /opt/aws/opsworks/current/site-cookbooks. If you run a command from the web console, you can repeat the command with a tailed log output using "opsworks-agent-cli run_command".
softdev12 12y ago
I'm a big fan of Elastic Beanstalk. AWS with Elastic Beanstalk just makes things super easy for my deployment. And today Amazon announced they have over 1 million active users. This just shows how powerful their offerings are.
- sultano 12y ago
  Or it shows that AWS doesn't have any other simple deployment service which doesn't tie you into using particular technologies. Hopefully this is where CodeDeploy will come in.
pjungwir 12y ago
> I use OpsWorks quite a bit but have found it very challenging, and the feedback cycle when attempting to develop new cookbooks is excruciatingly slow. Yes. As a frequent Chef user, I was really excited about OpsWorks, but it feels quite beta, and as you said the build/test cycle is even worse than usual for config management. Have you had any success developing your Chef scripts on Vagrant and then using them on OpsWorks? It seems like the real thing is just too different. The open source OpsWorks cookbooks don't match what's actually running, you've got a VPC environment around you, there are extra tweaks to the AMIs, etc. For new projects I think CloudFormation + regular Chef is a better way to go.
- STRML 12y ago
  I tried getting a Vagrant environment up but I'm not convinced it's worth it. There is at least one repo with what looks like a working config [1]. I was far enough along in development by the time I started messing with Vagrant that I never finished it. I started debugging using the opsworks-agent-cli as mentioned in zackangelo's comment. I'd just been `git push`ing changes, updating cookbooks and testing again - his method of editing cookbooks [2] directly actually sounds rather workable. I imagine in the future I will mostly edit and test cookbooks this way. 1. https://github.com/wwestenbrink/vagrant-opsworks https://github.com/wwestenbrink/vagrant-opsworks 2. https://news.ycombinator.com/item?id=8598255 https://news.ycombinator.com/item?id=8598255
- mentat 12y ago
  I used this development flow regularly. You had to account for opsworks nodes settings not to be there. There's discussion where I currently work that there really should be a plugin to simulate metadata and perhaps the same or another to simulate opsworks.
x0x0 12y ago
also check out http://cloudnative.io http://cloudnative.io instead of using rolling updates, it's a blue-green deploy -- essentially what netflix does. it also automates ami building, triggered by a jenkins plugin after a successful test run.
- pas256 12y ago
  Thanks for the link. Yes I was excited to see AWS moving higher up the stack and offer code pipeline and deployment options, but CodeDeploy is exactly what would have been designed (and in fact was) if EC2, ELBs and AutoScaling did not exist. It was born 18 years ago. It suffers from all the same issues that any rolling update with mutable infrastructure does. Blue/Green, while harder to use without tooling like CloudNative adds, has numerous benefits including near instant rollback to a previously known good state. Netflix figured this out 5 years ago.
pallen 12y ago
Wow STRML - thats exactly how i felt and why i created CloudCoreo.com... you are literally describing what we do (plus we handle services other than ec2 as well). it's not technically open to the public until January, but after seeing what you want i would love your feedback on the system. Let me know if you are interested and i can walk you through how it works.

by STRML 12y ago

sciurus 12y ago

CloudFormation is really a different beast. It's focused on creating and managing related collections of AWS resources, not on application deployment. Anyone doing, well, pretty much anything on AWS should probably be using Cloudformation. http://aws.amazon.com/cloudformation/ http://aws.amazon.com/cloudformation/

kxo 12y ago

How about an Amazon-managed Docker (not shit ElasticBeanstalk) infrastructure/config management system? If they handled things like EBS integration, orchestration, ELB integration, and all of the usual stuff, I'd be sold in a second.

STRML 12y ago
From my understanding, after talking to them at Web Summit, something like this is in the works. And they're aware of how incredibly complicated the AWS console is becoming. I don't know how or when they plan to address it, however.
- mentat 12y ago
  They promise lots of things (like cross region VPC peering) to person to person at Summits...
  - STRML 12y ago
    Yes, I am still waiting for something prettier than software VPN or OpenS/WAN. Their docs show 5 ways to connect VPCs, all of them complex, all of them with significant downsides. One would think this sort of thing would be easier.
tedchs 12y ago
On Google Cloud Platform, there is a new service called Google Container Engine (currently in alpha). It uses Kubernetes to manage Docker containers, running on Google Compute Engine. You could also run Kubernetes on EC2 if you wanted to.
vidarh 12y ago
It takes a couple of days to get Ansible playbooks set up to deploy CoreOS VMs onto EC2 with a decent Docker setup and EBS. I know, I've just done it. And I don't like Ansible, and haven't used it seriously before (I use it for project because it's on a contract and it's easy-ish for them to find people who understands Ansible if/when it's needed) so it took longer than I'd have liked. Frankly, I don't understand why people are so happy to lock themselves tighter into AWS.

zackangelo 12y ago

I've found developing OpsWorks recipes directly on the instance to be the best remedy for the slow feedback cycle. Your cookbook is located /opt/aws/opsworks/current/site-cookbooks. If you run a command from the web console, you can repeat the command with a tailed log output using "opsworks-agent-cli run_command".

softdev12 12y ago

I'm a big fan of Elastic Beanstalk. AWS with Elastic Beanstalk just makes things super easy for my deployment. And today Amazon announced they have over 1 million active users. This just shows how powerful their offerings are.

sultano 12y ago
Or it shows that AWS doesn't have any other simple deployment service which doesn't tie you into using particular technologies. Hopefully this is where CodeDeploy will come in.

pjungwir 12y ago

> I use OpsWorks quite a bit but have found it very challenging, and the feedback cycle when attempting to develop new cookbooks is excruciatingly slow. Yes. As a frequent Chef user, I was really excited about OpsWorks, but it feels quite beta, and as you said the build/test cycle is even worse than usual for config management. Have you had any success developing your Chef scripts on Vagrant and then using them on OpsWorks? It seems like the real thing is just too different. The open source OpsWorks cookbooks don't match what's actually running, you've got a VPC environment around you, there are extra tweaks to the AMIs, etc. For new projects I think CloudFormation + regular Chef is a better way to go.

STRML 12y ago
I tried getting a Vagrant environment up but I'm not convinced it's worth it. There is at least one repo with what looks like a working config [1]. I was far enough along in development by the time I started messing with Vagrant that I never finished it. I started debugging using the opsworks-agent-cli as mentioned in zackangelo's comment. I'd just been `git push`ing changes, updating cookbooks and testing again - his method of editing cookbooks [2] directly actually sounds rather workable. I imagine in the future I will mostly edit and test cookbooks this way. 1. https://github.com/wwestenbrink/vagrant-opsworks https://github.com/wwestenbrink/vagrant-opsworks 2. https://news.ycombinator.com/item?id=8598255 https://news.ycombinator.com/item?id=8598255
mentat 12y ago
I used this development flow regularly. You had to account for opsworks nodes settings not to be there. There's discussion where I currently work that there really should be a plugin to simulate metadata and perhaps the same or another to simulate opsworks.

x0x0 12y ago

also check out http://cloudnative.io http://cloudnative.io instead of using rolling updates, it's a blue-green deploy -- essentially what netflix does. it also automates ami building, triggered by a jenkins plugin after a successful test run.

pas256 12y ago
Thanks for the link. Yes I was excited to see AWS moving higher up the stack and offer code pipeline and deployment options, but CodeDeploy is exactly what would have been designed (and in fact was) if EC2, ELBs and AutoScaling did not exist. It was born 18 years ago. It suffers from all the same issues that any rolling update with mutable infrastructure does. Blue/Green, while harder to use without tooling like CloudNative adds, has numerous benefits including near instant rollback to a previously known good state. Netflix figured this out 5 years ago.

pallen 12y ago

Wow STRML - thats exactly how i felt and why i created CloudCoreo.com... you are literally describing what we do (plus we handle services other than ec2 as well). it's not technically open to the public until January, but after seeing what you want i would love your feedback on the system. Let me know if you are interested and i can walk you through how it works.