4 ms·

Introduction to AWS OpsWorks

91 points by joeya 13y ago

chasb 13y ago
Being able to roll your own level/complexity of devops is fantastic. The next step is to bring this functionality to high-compliance users, like banks and healthcare providers.
mattjaynes 13y ago
OpsWorks is a nice convenience on AWS and is implemented with the Chef configuration management tool (provided by Opscode, the company behind Chef, hence the name). Shameless plug: If you know you should be using a configuration management tool, but haven't picked one yet, I'll be releasing a book on Wednesday Sept 4th just for you. The book is called "Taste Test" and compares Puppet, Chef, Salt, and Ansible. In the book, I implement an identical project with each tool so you can see what each one is like to work with. I definitely had some big surprises when writing the book. Spoiler: Ansible was by far the simplest, easiest to understand, and quickest to get up and going. To get a discount for the book release, just sign up on the mailing list: http://devopsu.com/books/taste-test-puppet-chef-salt-stack-ansible.html http://devopsu.com/books/taste-test-puppet-chef-salt-stack-a...
- vosper 13y ago
  This is awesome, I'm eager to check out your book; in-depth side-by-side comparisons are something that's missing from way too many technology discussions.
- joeya 13y ago
  I've heard a little about Ansible but look forward to learning more. I'm in awe that you took the time to get up to speed on each of those tools. Some of the learning curves are... intimidating. For what it's worth, I actually found OpsWorks' usage of Chef to be more intuitive than Opscode's own. OpsWorks relies on "events" (setup, deploy, shutdown, etc.) that trigger (i.e., push) recipes to be applied to instances. Opscode, at least in its typical architecture, relies on clients pulling configuration from a Chef server at a regular interval.
- gingerlime 13y ago
  sounds interesting. I was reading some great things about ansible and am keen to give it a try. However, reading about AWS Opswork makes me wonder if it's better to stick to chef, even if it's not as easy. If Opsworks picks up, chef's likely to become the gold standard. I'm curious to know your take on this?
  - mattjaynes 13y ago
    I've been recommending and using Puppet and Chef for years. However, the new generation of tools like Ansible and Salt are miles ahead in terms of usability. Ansible in particular makes it easy to be up and going in a matter of minutes where some of the other tools literally take days. So, if you're on the fence, I'd try Ansible first, then if for some reason it's not working for you, then consider Chef.
- ZoFreX 13y ago
  I'm curious - why didn't you include CF-Engine?
timdorr 13y ago
OpsWorks is great. We were also on Heroku and were planning on the DIY route with Puppet. My coworker spent 2-3 weeks working on that setup and was 90% of the way there. Then he found out about OpsWorks. We had our apps deployed in a single day and migrated over by the end of the week. There was a bit of learning to deal with Chef, but it was an overall smooth experience. We've linked it up to Codeship and now we simply push to Github, Codeship runs our test suite, and they run a rake task to initiate a deploy via OpsWorks. Ever since Codeship made some serious optimizations in the past week, our time from git push to deploy is about 5 minutes. Awesome!
- skyebook 13y ago
  Similarly good experiences with setting up custom recipes that don't use the built in 'App' stack (which provides Rails, Node, and some others). I was able to go from the 'tinkering at 2am' level of ruby experience to making Chef do my bidding in setting up alternative apt-repos, deploying config files, etc in a day or two. My one gripe, as with the rest of AWS, is that they seem more interested in MySQL than Postgres. The opsworks cookbook repo has an included mysql cookbook as well as a preconfigured "App" layer, nothing for Postgres besides forum responses to go the DIY route.
  - miles932 13y ago
    On the PostgreSQL note, here's a whitepaper I helped with that enumerates best practices for it on AWS: http://media.amazonwebservices.com/AWS_RDBMS_PostgreSQL.pdf http://media.amazonwebservices.com/AWS_RDBMS_PostgreSQL.pdf
    - skyebook 13y ago
      Please allow me to give an emphatic THANK YOU for writing that... I've gone to it more than once in the short while its been floating around and it has generally been helpful in either giving me some conviction in what I'm doing or righting my ways. Unfortunately, and this drives me nuts about so many PaaS offerings, maintaining a database at scale is a really niche thing that people devote tons of time to.. Arguably many times more difficult than keeping a bunch of app servers up.. I would love to see more hosted offerings along the line of EnterpriseDB and Heroku's Postgres service. Thanks again for the awesome resource :)
redeemedfadi 13y ago
Thanks for this. Can't believe I haven't heard about it before. At least all the time spent trying to learn Chef is not completely wasted.
danso 13y ago
First, some pedantry: I love Artsy's writeups but I'm really surprised at why they use that purple for their body text color. Every time I read their engineering blog, I use the web inspector to change the color to #222...and the purple actually looks great as an accent (for links)...I might have to rip off that color scheme for my own blog. /pedantry I'm kind of surprised I hadn't heard of OpsWorks before, but I guess that's just a testament to how many things AWS is rolling out...this seems like the best fit for my needs, after having struggled with EBS and rolling my own EC2 deploys. But I have to really commend Artsy's team for not just sharing their experiences, but doing it in such a presentable way...The OP links to the AWS documentation, which looks like this: http://docs.aws.amazon.com/opsworks/latest/userguide/workingapps.html http://docs.aws.amazon.com/opsworks/latest/userguide/working... It's standard Amazon documentation style, which for me, has always been too fragmented (look at all those nested menu items you have to click through) with uninspired layout and typography to boot (i.e. 100% width for the body text) Even if I had seen the AWS info on OpsWorks, I probably would've skipped the docs. Artsy's explanation and presentation of the concepts is just fantastic and so readable...hopefully Amazon rips off from Artsy's blog as well.
- joeya 13y ago
  Thanks for the kind words! To be honest, I have to agree with you about the text color. Will fix.
- dblock 13y ago
  If @joeyAghion doesn't get to it first, we will gladly take a PR to change the color! Our blog repo is public here: https://github.com/artsy/artsy.github.com/tree/source https://github.com/artsy/artsy.github.com/tree/source
simonebrunozzi 13y ago
This is a 13-minute demo of AWS Opsworks (I am the author, and I work for AWS): https://www.youtube.com/watch?v=2_TtiSJSbHo&feature=youtu.be https://www.youtube.com/watch?v=2_TtiSJSbHo&feature=youtu.be I hope you'll find it useful.
- milkshakes 13y ago
  edit: thanks for updating to chef 11!
  - bbgm 13y ago
    Chef 11 support was announced recently: http://aws.typepad.com/aws/2013/07/aws-opsworks-update-custom-amis-and-chef-11-support.html http://aws.typepad.com/aws/2013/07/aws-opsworks-update-custo...
milkshakes 13y ago
are they still using an ancient version of chef? trying to backport community cookbooks was an exercise in frustration
- joeya 13y ago
  They were originally using an older version, but Chef 11 is now supported. I believe it's the default for new stacks, and old ones can be updated from the stack settings page. http://aws.amazon.com/about-aws/whats-new/2013/07/24/aws-opsworks-supports-chef-custom-ami/ http://aws.amazon.com/about-aws/whats-new/2013/07/24/aws-ops...
  - milkshakes 13y ago
    thank god. that was the only dealbreaker for me. thanks for the article by the way! and the free wine on fridays
AsymetricCom 13y ago
This sounds very similar to the types of tools Obama used for his campaign. It's nice to see such tools are now available to the everyman w/o having to invest in a team of operational specialists. http://www.williamhertling.com/2013/07/printable-obama-for-america-aws.html http://www.williamhertling.com/2013/07/printable-obama-for-a...
- miles932 13y ago
  see the original here: http://awsofa.info http://awsofa.info The OFA team used Puppet, CloudFormation, furious typing in boto and the CLI, and later added a healthy helping of Netflix's Asgard system for system mgmt.
  - AsymetricCom 13y ago
    There was an article that goes more in depth than this infographic, but I haven't found it. I think it's originally from Wired print.
mnutt 13y ago
Does anyone know if anyone is working on a standalone version of an event-based system for Chef? I'm not on AWS, but I think an event-based system similar to OpsWorks might have a nice fit between chef-solo and chef-server.
- joeya 13y ago
  Hey mnutt, I agree--it's a nice model. Scalarium was building such a product, but Amazon bought them and made it OpsWorks :)
xorgar831 13y ago
After having used Chef on a project, I've found it to be really a clueless train-wreck of a product that they got their buddies in the USA NW to use. The basis of their approach is that they have no clue what you're doing, because every one's environment is different, so they just leave it up to you how you want to do everything. Yet the whole value in such a product is enforcing good factorization, implementation and maintainability. If you let something like this lose on your Fortune 500 you're just going to have a pile of flakey, inconstant and convoluted and unmaintainable DSL crap doing all sorts of random stuff... and that's being kind to the product IMO.
elazdins 13y ago
Last week I wrote a blog post on OpsWorks deployment from the command line and via Codeship. The article is here: http://novoit.eu/blog/02-deploying-code-to-amazon-opsworks-using-codeship http://novoit.eu/blog/02-deploying-code-to-amazon-opsworks-u...

91 points by joeya 13y ago

chasb 13y ago

Being able to roll your own level/complexity of devops is fantastic. The next step is to bring this functionality to high-compliance users, like banks and healthcare providers.

mattjaynes 13y ago

OpsWorks is a nice convenience on AWS and is implemented with the Chef configuration management tool (provided by Opscode, the company behind Chef, hence the name). Shameless plug: If you know you should be using a configuration management tool, but haven't picked one yet, I'll be releasing a book on Wednesday Sept 4th just for you. The book is called "Taste Test" and compares Puppet, Chef, Salt, and Ansible. In the book, I implement an identical project with each tool so you can see what each one is like to work with. I definitely had some big surprises when writing the book. Spoiler: Ansible was by far the simplest, easiest to understand, and quickest to get up and going. To get a discount for the book release, just sign up on the mailing list: http://devopsu.com/books/taste-test-puppet-chef-salt-stack-ansible.html http://devopsu.com/books/taste-test-puppet-chef-salt-stack-a...

vosper 13y ago
This is awesome, I'm eager to check out your book; in-depth side-by-side comparisons are something that's missing from way too many technology discussions.
joeya 13y ago
I've heard a little about Ansible but look forward to learning more. I'm in awe that you took the time to get up to speed on each of those tools. Some of the learning curves are... intimidating. For what it's worth, I actually found OpsWorks' usage of Chef to be more intuitive than Opscode's own. OpsWorks relies on "events" (setup, deploy, shutdown, etc.) that trigger (i.e., push) recipes to be applied to instances. Opscode, at least in its typical architecture, relies on clients pulling configuration from a Chef server at a regular interval.
gingerlime 13y ago
sounds interesting. I was reading some great things about ansible and am keen to give it a try. However, reading about AWS Opswork makes me wonder if it's better to stick to chef, even if it's not as easy. If Opsworks picks up, chef's likely to become the gold standard. I'm curious to know your take on this?
- mattjaynes 13y ago
  I've been recommending and using Puppet and Chef for years. However, the new generation of tools like Ansible and Salt are miles ahead in terms of usability. Ansible in particular makes it easy to be up and going in a matter of minutes where some of the other tools literally take days. So, if you're on the fence, I'd try Ansible first, then if for some reason it's not working for you, then consider Chef.
ZoFreX 13y ago
I'm curious - why didn't you include CF-Engine?

timdorr 13y ago

OpsWorks is great. We were also on Heroku and were planning on the DIY route with Puppet. My coworker spent 2-3 weeks working on that setup and was 90% of the way there. Then he found out about OpsWorks. We had our apps deployed in a single day and migrated over by the end of the week. There was a bit of learning to deal with Chef, but it was an overall smooth experience. We've linked it up to Codeship and now we simply push to Github, Codeship runs our test suite, and they run a rake task to initiate a deploy via OpsWorks. Ever since Codeship made some serious optimizations in the past week, our time from git push to deploy is about 5 minutes. Awesome!

skyebook 13y ago
Similarly good experiences with setting up custom recipes that don't use the built in 'App' stack (which provides Rails, Node, and some others). I was able to go from the 'tinkering at 2am' level of ruby experience to making Chef do my bidding in setting up alternative apt-repos, deploying config files, etc in a day or two. My one gripe, as with the rest of AWS, is that they seem more interested in MySQL than Postgres. The opsworks cookbook repo has an included mysql cookbook as well as a preconfigured "App" layer, nothing for Postgres besides forum responses to go the DIY route.
- miles932 13y ago
  On the PostgreSQL note, here's a whitepaper I helped with that enumerates best practices for it on AWS: http://media.amazonwebservices.com/AWS_RDBMS_PostgreSQL.pdf http://media.amazonwebservices.com/AWS_RDBMS_PostgreSQL.pdf
  - skyebook 13y ago
    Please allow me to give an emphatic THANK YOU for writing that... I've gone to it more than once in the short while its been floating around and it has generally been helpful in either giving me some conviction in what I'm doing or righting my ways. Unfortunately, and this drives me nuts about so many PaaS offerings, maintaining a database at scale is a really niche thing that people devote tons of time to.. Arguably many times more difficult than keeping a bunch of app servers up.. I would love to see more hosted offerings along the line of EnterpriseDB and Heroku's Postgres service. Thanks again for the awesome resource :)

redeemedfadi 13y ago

Thanks for this. Can't believe I haven't heard about it before. At least all the time spent trying to learn Chef is not completely wasted.

danso 13y ago

First, some pedantry: I love Artsy's writeups but I'm really surprised at why they use that purple for their body text color. Every time I read their engineering blog, I use the web inspector to change the color to #222...and the purple actually looks great as an accent (for links)...I might have to rip off that color scheme for my own blog. /pedantry I'm kind of surprised I hadn't heard of OpsWorks before, but I guess that's just a testament to how many things AWS is rolling out...this seems like the best fit for my needs, after having struggled with EBS and rolling my own EC2 deploys. But I have to really commend Artsy's team for not just sharing their experiences, but doing it in such a presentable way...The OP links to the AWS documentation, which looks like this: http://docs.aws.amazon.com/opsworks/latest/userguide/workingapps.html http://docs.aws.amazon.com/opsworks/latest/userguide/working... It's standard Amazon documentation style, which for me, has always been too fragmented (look at all those nested menu items you have to click through) with uninspired layout and typography to boot (i.e. 100% width for the body text) Even if I had seen the AWS info on OpsWorks, I probably would've skipped the docs. Artsy's explanation and presentation of the concepts is just fantastic and so readable...hopefully Amazon rips off from Artsy's blog as well.

joeya 13y ago
Thanks for the kind words! To be honest, I have to agree with you about the text color. Will fix.
dblock 13y ago
If @joeyAghion doesn't get to it first, we will gladly take a PR to change the color! Our blog repo is public here: https://github.com/artsy/artsy.github.com/tree/source https://github.com/artsy/artsy.github.com/tree/source

simonebrunozzi 13y ago

This is a 13-minute demo of AWS Opsworks (I am the author, and I work for AWS): https://www.youtube.com/watch?v=2_TtiSJSbHo&feature=youtu.be https://www.youtube.com/watch?v=2_TtiSJSbHo&feature=youtu.be I hope you'll find it useful.

milkshakes 13y ago
edit: thanks for updating to chef 11!
- bbgm 13y ago
  Chef 11 support was announced recently: http://aws.typepad.com/aws/2013/07/aws-opsworks-update-custom-amis-and-chef-11-support.html http://aws.typepad.com/aws/2013/07/aws-opsworks-update-custo...

milkshakes 13y ago

are they still using an ancient version of chef? trying to backport community cookbooks was an exercise in frustration

joeya 13y ago
They were originally using an older version, but Chef 11 is now supported. I believe it's the default for new stacks, and old ones can be updated from the stack settings page. http://aws.amazon.com/about-aws/whats-new/2013/07/24/aws-opsworks-supports-chef-custom-ami/ http://aws.amazon.com/about-aws/whats-new/2013/07/24/aws-ops...
- milkshakes 13y ago
  thank god. that was the only dealbreaker for me. thanks for the article by the way! and the free wine on fridays

AsymetricCom 13y ago

This sounds very similar to the types of tools Obama used for his campaign. It's nice to see such tools are now available to the everyman w/o having to invest in a team of operational specialists. http://www.williamhertling.com/2013/07/printable-obama-for-america-aws.html http://www.williamhertling.com/2013/07/printable-obama-for-a...

miles932 13y ago
see the original here: http://awsofa.info http://awsofa.info The OFA team used Puppet, CloudFormation, furious typing in boto and the CLI, and later added a healthy helping of Netflix's Asgard system for system mgmt.
- AsymetricCom 13y ago
  There was an article that goes more in depth than this infographic, but I haven't found it. I think it's originally from Wired print.

mnutt 13y ago

Does anyone know if anyone is working on a standalone version of an event-based system for Chef? I'm not on AWS, but I think an event-based system similar to OpsWorks might have a nice fit between chef-solo and chef-server.

joeya 13y ago
Hey mnutt, I agree--it's a nice model. Scalarium was building such a product, but Amazon bought them and made it OpsWorks :)

xorgar831 13y ago

After having used Chef on a project, I've found it to be really a clueless train-wreck of a product that they got their buddies in the USA NW to use. The basis of their approach is that they have no clue what you're doing, because every one's environment is different, so they just leave it up to you how you want to do everything. Yet the whole value in such a product is enforcing good factorization, implementation and maintainability. If you let something like this lose on your Fortune 500 you're just going to have a pile of flakey, inconstant and convoluted and unmaintainable DSL crap doing all sorts of random stuff... and that's being kind to the product IMO.

elazdins 13y ago

Last week I wrote a blog post on OpsWorks deployment from the command line and via Codeship. The article is here: http://novoit.eu/blog/02-deploying-code-to-amazon-opsworks-using-codeship http://novoit.eu/blog/02-deploying-code-to-amazon-opsworks-u...