8 ms·

Why Chef?

55 points by bryanwb 15y ago

apinstein 15y ago
I am a co-founder of a small SaaS company. We recently decided to make the investment of upgrading our infrastructure setup process from "Hey David go do that" to being 100% chef-based. I managed the process, and consider it a success. However, here are some points I would make: 1. It took a long time. Let's be generous and say it took 2-3 man-months of time to set up 4-5 different projects and roles. This was probably 10-20x what it would've taken to set up the servers directly. Why? Learning curve with chef for both our programmer and sysadmin. Figuring out how to make config changes automatable and idempotent. 2. The scale you get from chef is bigger than managing production infrastructure. We now use chef for not only production deployment, but also dev. Once paired with Vagrant, we are able to get new devs up with a complete stack in about 10m of keyboard time. If we need to upgrade to some new version of something, only one person has to deal with the sysadmin; everyone else can just update their box. 3. I think it will save money in the long-term. A good sysadmin is $100/hr+. Unfortunately you have to pay that rate whether they're doing architecture, security review, or just editing text files. With chef, a non-sysadmin resource can generate recipes with just architectural advice and review from a sysadmin. This is much more efficient, especially for small shops where a sysadmin is an expensive and not immediately available resource.
- bryanwb 15y ago
  awesome comments, apinstein btw, I am copying your method of managing dotfiles with a Rakefile as we speak. I counter that while it takes a while to learn chef, it also takes many times more effort to maintain a custom set of shell scripts over the long term
- devmach 15y ago
  Is there a something like Foreman[1] for Chef ? Something able to create and spin up new VM and execute receipts after install ( by using just a "web browser" ). [1]: http://theforeman.org/projects/foreman/wiki/Screencasts http://theforeman.org/projects/foreman/wiki/Screencasts
  - apinstein 15y ago
    Oh nice I hadn't seen Foreman before. Looks like EC2 console but for your own stack. We use XEN, that might be nice. I don't know what you mean by "execute receipts" however in the video they kick off a puppet run, which makes me think you could similarly kick off a chef run instead.
    - manvsmachine 15y ago
      I'm guessing that 'receipts' was meant to be 'recipes'.
    - devmach 15y ago
      Sorry, my bad. I meant to say "recipes".
- tptacek 15y ago
  A good sysadmin costs $200k/year? Or do you have a contract sysadmin? Because that number sounds way, way, way high. (We recently hired a sysadmin "+").
  - apinstein 15y ago
    We've been in business for about 8 years and use about ~100 sysadmin hours a year. I interviewed several sysadmins over the years and no one that I deemed competent charged less than $100/hour. I did find people charging as low as $50/hour but they didn't seem that great, or that reliable, or that available. For me as well, you pretty much need to trust your sysadmin more than almost anyone else in your company (except people that can sign checks). Trust comes at a premium. I will yield that I don't like working with substandard people. I hate having to manage people and am willing to pay a premium for people I trust to work on the right things with the right skills in a timely manner. Besides, it doesn't scale. One of my business goals is to never have middle management.
    - tptacek 15y ago
      You'd be more convincing if you didn't imply that anyone who worked for less than $100/hr was "substandard". For full-time salaried, I assure you, $200k/yr is not the going rate for sysadmin.
      - nestlequ1k 15y ago
        You cant directly equate contract hourly rate with yearly salary. If you don't have the budget to hire a 100k a yr salary, you'll have to pay contract rates which could be over 100/hr
        tptacek 15y ago
        I buy that the valley is so hot right now that a sysadmin commands $100k/yr there, but they don't in Chicago, Seattle, or New York. I'm not looking to argue so much as to inject some more data into the price point that was casually dropped on this thread earlier. I do not think the other guy overpaid for sysadmin; if he's got an amazing admin, great! I can see paying a premium for that. Another point I'd like to raise is, if you're paying $100k/hr for sysadmin, and using them frequently, contracting instead of fulltiming sysadmin might be penny-wise-pound-foolish. But maybe not, if you're only paying $10,000/yr in sysadmin. We've used over 100 hours of admin in just the last couple weeks. A great hire; one we made "37signals-style", after realizing that doing all the sysadmin chores ourselves was subtly making us all miserable and ineffective. Believe me, I know the difference between contracting rates and salary. ;)
- sjwright 15y ago
  I learned a new word today: Idempotence. Thank you.
- Dalves 15y ago
  You've just echoed my experience with chef but I'm still on the fence as to whether I think it was a success or not and whether I'd use it again. We were able to achieve everything we wanted but to say it took a metric fuckton of time would be a grave understatement. Even what are ordinarily trivial things took a lot longer than I expected. Some things I just found to be oddly designed and surprisingly inflexible, especially the recipe construct. I still to this day see no point of having a folder with 2 subfolders with 3 files combined for what amounts to "apt get install somepackage". And having to learn a DSL to replace the already very effective bash is in my opinion not a very enticing trade. Although it's very new, I think I'll try out juju.ubuntu.com for my next large deployment instead of sticking with chef or puppet, if only for the flexibility of the hooks-in-any-language thing and a chance to feed my bash fetish.
- lkanies 15y ago
  (disclaimer - Puppet Labs employee) 2-3 months is a long time. We consistently hear 2-6 weeks to get to full production, with 30-120 minutes for some proof of concept work. And one of the great things about Puppet is you can start very small - automate the parts that are hardest, most critical, etc., and do bits manually as it makes sense for your problem set. And yeah, managing dev and prod the same way is absolutely critical for clean deployments.
leandrod 15y ago
I cringe at CouchDB. Gimme good, old PostgreSQL.
codypo 15y ago
I agree with the author that Chef is an incredibly powerful tool and that it has numerous benefits over plain old bash. However, I'm reluctant to actually use it because of its dependencies. Chef relies on CouchDB, RabbitMQ, and Solr, and all of those have non-trivial dependencies as well. Then, with a stack like that, I worry about the overheard involved. FWIW, Puppet's dependencies are much simpler. I don't know much about Chef vs Puppet, but I can say that from an installation and dependency maintenance POV, Puppet wins.
- atsaloli 15y ago
  Speaking of dependencies, CFEngine 3 is written in C and has 3 dependencies: berkeley db, libcrypo, and PCRE. It compiles into small binaries and is usable anywhere - in the cloud, on supercompute clusters, on the desktop or laptop, on a smartphone, in embedded devices.
- dholmen 15y ago
  You really should check out CFEngine 3. Very few dependencies (pcre,berkeleydb,openssl), and they also provide free packages with all the dependencies included: http://cfengine.com/download http://cfengine.com/download The memory footprint is about 10 MB, install size maybe 30 MB.
- Fluxx 15y ago
  You can use "Hosted Chef," where Opscode hosts all the dependencies for you: http://www.opscode.com/hosted-chef/ http://www.opscode.com/hosted-chef/
  - sjs 15y ago
    If you've already decided on Chef then the Opscode platform provides great value. You get the best and most knowledgable Chef admins handling your Chef server for a few bucks per hour.
    - dlsspy 15y ago
      If you haven't decided, it's a great way to try it out. I'm an opscode customer, but before that, I was playing around with their free plan.
adient 15y ago
I think this post should be titled Why Configuration Management?, with a subtext of using Chef as an example. The main points the author is making are true of CFEngine, Puppet, Chef, and other config management software. The question of Why Chef? can be answered very simply: because the author likes it. Which is a perfectly valid way to choose your tool chain, assuming the technical requirements are met.
- bryanwb 15y ago
  I wrote about that it the previous post, see her http://devopsanywhere.blogspot.com/2011/10/puppet-vs-chef-fight.html http://devopsanywhere.blogspot.com/2011/10/puppet-vs-chef-fi...
grandalf 15y ago
For < 50 servers, can anyone comment on my choice of fabric and cuisine as an alternative to chef?
- thibaut_barrere 15y ago
  I decided to use chef-solo instead, mostly as a way to start learning chef but it ended be good enough for me. This way I'll be able to invest in the server part of chef later on, when I need it.
- JoachimSchipper 15y ago
  The number of different configurations is far more important than the number of boxes, no? If you have 1000 completely identical boxes (e.g. big compute cluster), a shell script that sets up a freshly installed box to your requirements is quite possibly sufficient.
nona 15y ago
Two reasons I'm wary of Chef wrt Puppet 1) the declarative vs imperative aspect 2) Chef's heavy dependencies As a Ruby developer, I like Chef's Ruby DSL; but I somehow feel that its imperative DSL will lead to something similar to Bash Hell. I'd like to read more about the declarative properties of Puppet vs the imperative way of Chef, and why one should prefer one over the other. Secondly, like some commenters before mention, Chef's dependencies seem quite excessive.
- parasubvert 15y ago
  Chef is becoming more declarative as it grows. The main difference with Chef is that it has imperative structure (i.e. ruby-based scripting) to fall back on, whereas Puppet forces you to go out to a script if you want to be imperative. In the long run, it's arguable that a strict declarative structure has more interesting properties when dealing with query of status & handling changes or drifts rather than "stamping out a server". Puppet has been great for me to detect and correct drift, for example, all the while ensuring pre-requisites are executed in the proper order. The tradeoff is that you have to think through the various dependencies at a detailed level, which can be difficult. An extreme analogy would be programming with a logic language vs. an imperative language. My experience has been Puppet is growing in popularity with enterprises, not just web companies, and it seems Puppet Labs is targeting this audience more than Opscode is, comparing customer lists. I'm not quite sure why this is - could be attitude - Chef is more about "get it done now", Puppet more about "get it right for the long haul", but even that is a caricature.
  - lkanies 15y ago
    (disclaimer - founder of Puppet) We at Puppet Labs have always focused on building tools that anyone can use, not just the best hackers in the world. We've got some of the best sysadmins working with us, but we punish them by making them write software that even people who aren't the best sysadmins can use. So yeah, we get a ton of enterprise adoption as a result, but we've also got a ton of web companies using Puppet. It's true that in the Rails web startup, we're not always the number 1 choice, though. :)
gpapilion 15y ago
I have three fundamental issues with Chef; 1. The node object ends up being two large, which leads to memory issues when a search returns more that 200 nodes (800MB of memory). 2. Chef discourages declarative configuration. 3. Chef lacks a remote trigger mechanism. Issue one starts to kill you once you have a large number of nodes. Dedicating a quater of the available memory to configuration management seem like a poor financial choice. We've come up with work arounds at my company(generate files centrally, and distribute with chef remote_file syntax), but I still feel they are hacky. Issue two is more serious. Reindexing on chef only occurs after a node has submitted its node object back to the chef server. This results in incomplete searches until a node successfully completes a run. If you wish to remove a node from a particular role or attribute from a host, you may have a hard time doing so until the next chef run completes. Problem three is really a result of the expense of running chef. If the memory and CPU costs were lower, there wouldn't be any real issues running Chef more frequently. Some changes I need to go out immediately, some don't matter. I end up back in the world of the SSH loop too often with Chef.
- jtimberman 15y ago
  1. We are working on making the search more performance and use less memory. 2. Chef definitely does not discourage declarative configuration. Chef recipes include declarative resource for configuring your infrastructure. Since recipes are an internal ruby dsl, there may be nondeclarative code in them. 3. Chef itself doesn't have a remote trigger mechanism because the. Her run is all about configuring the local node. Nothing prevents you from using the ruby language in a recipe to hook up some kind of remote trigger though. People in the chef community are doing this with projects like Noah and Pylon. Github.com/lusis/Noah Github.com/fujin/pylon
  - gpapilion 15y ago
    1. I understand that, and have heard that from you guys several times. The issue is the deserialization from JSON. The monkey patch solution I've see so far stores less data, essentially white listing the attributes for search. It sort of destroys the value of search. 2. I should be more specific. Generally chef relies on the information that ohai provides, not with information enumerated by the administrator. There is a general assumption that the systems are properly configured, and chef is only furthering that, since the hosts provide most of the configuration details. (Yes, you could do stuff with data bags to address this issue.) 3. Thanks for the links. I've not seen those projects previously. I've solved the issue for myself using a much lighter weight solution.
kim0 15y ago
I've been working with juju https://juju.ubuntu.com/ https://juju.ubuntu.com/ (a new Ubuntu server tool) and been very happy with it. It does not directly compete with puppet/chef for enforcing a server configuration however. Juju operates at the higher level of "services" that can be deployed to a cloud, to hardware, or to local LXC containers! It's basically like "apt-get" except for servers. The dev team hangs out at #juju on freenode irc

55 points by bryanwb 15y ago

I am a co-founder of a small SaaS company. We recently decided to make the investment of upgrading our infrastructure setup process from "Hey David go do that" to being 100% chef-based. I managed the process, and consider it a success. However, here are some points I would make: 1. It took a long time. Let's be generous and say it took 2-3 man-months of time to set up 4-5 different projects and roles. This was probably 10-20x what it would've taken to set up the servers directly. Why? Learning curve with chef for both our programmer and sysadmin. Figuring out how to make config changes automatable and idempotent. 2. The scale you get from chef is bigger than managing production infrastructure. We now use chef for not only production deployment, but also dev. Once paired with Vagrant, we are able to get new devs up with a complete stack in about 10m of keyboard time. If we need to upgrade to some new version of something, only one person has to deal with the sysadmin; everyone else can just update their box. 3. I think it will save money in the long-term. A good sysadmin is $100/hr+. Unfortunately you have to pay that rate whether they're doing architecture, security review, or just editing text files. With chef, a non-sysadmin resource can generate recipes with just architectural advice and review from a sysadmin. This is much more efficient, especially for small shops where a sysadmin is an expensive and not immediately available resource.

bryanwb 15y ago
awesome comments, apinstein btw, I am copying your method of managing dotfiles with a Rakefile as we speak. I counter that while it takes a while to learn chef, it also takes many times more effort to maintain a custom set of shell scripts over the long term
devmach 15y ago
Is there a something like Foreman[1] for Chef ? Something able to create and spin up new VM and execute receipts after install ( by using just a "web browser" ). [1]: http://theforeman.org/projects/foreman/wiki/Screencasts http://theforeman.org/projects/foreman/wiki/Screencasts
- apinstein 15y ago
  Oh nice I hadn't seen Foreman before. Looks like EC2 console but for your own stack. We use XEN, that might be nice. I don't know what you mean by "execute receipts" however in the video they kick off a puppet run, which makes me think you could similarly kick off a chef run instead.
  - manvsmachine 15y ago
    I'm guessing that 'receipts' was meant to be 'recipes'.
  - devmach 15y ago
    Sorry, my bad. I meant to say "recipes".
tptacek 15y ago
A good sysadmin costs $200k/year? Or do you have a contract sysadmin? Because that number sounds way, way, way high. (We recently hired a sysadmin "+").
- apinstein 15y ago
  We've been in business for about 8 years and use about ~100 sysadmin hours a year. I interviewed several sysadmins over the years and no one that I deemed competent charged less than $100/hour. I did find people charging as low as $50/hour but they didn't seem that great, or that reliable, or that available. For me as well, you pretty much need to trust your sysadmin more than almost anyone else in your company (except people that can sign checks). Trust comes at a premium. I will yield that I don't like working with substandard people. I hate having to manage people and am willing to pay a premium for people I trust to work on the right things with the right skills in a timely manner. Besides, it doesn't scale. One of my business goals is to never have middle management.
  - tptacek 15y ago
    You'd be more convincing if you didn't imply that anyone who worked for less than $100/hr was "substandard". For full-time salaried, I assure you, $200k/yr is not the going rate for sysadmin.
    - nestlequ1k 15y ago
      You cant directly equate contract hourly rate with yearly salary. If you don't have the budget to hire a 100k a yr salary, you'll have to pay contract rates which could be over 100/hr
      - tptacek 15y ago
        I buy that the valley is so hot right now that a sysadmin commands $100k/yr there, but they don't in Chicago, Seattle, or New York. I'm not looking to argue so much as to inject some more data into the price point that was casually dropped on this thread earlier. I do not think the other guy overpaid for sysadmin; if he's got an amazing admin, great! I can see paying a premium for that. Another point I'd like to raise is, if you're paying $100k/hr for sysadmin, and using them frequently, contracting instead of fulltiming sysadmin might be penny-wise-pound-foolish. But maybe not, if you're only paying $10,000/yr in sysadmin. We've used over 100 hours of admin in just the last couple weeks. A great hire; one we made "37signals-style", after realizing that doing all the sysadmin chores ourselves was subtly making us all miserable and ineffective. Believe me, I know the difference between contracting rates and salary. ;)
sjwright 15y ago
I learned a new word today: Idempotence. Thank you.
Dalves 15y ago
You've just echoed my experience with chef but I'm still on the fence as to whether I think it was a success or not and whether I'd use it again. We were able to achieve everything we wanted but to say it took a metric fuckton of time would be a grave understatement. Even what are ordinarily trivial things took a lot longer than I expected. Some things I just found to be oddly designed and surprisingly inflexible, especially the recipe construct. I still to this day see no point of having a folder with 2 subfolders with 3 files combined for what amounts to "apt get install somepackage". And having to learn a DSL to replace the already very effective bash is in my opinion not a very enticing trade. Although it's very new, I think I'll try out juju.ubuntu.com for my next large deployment instead of sticking with chef or puppet, if only for the flexibility of the hooks-in-any-language thing and a chance to feed my bash fetish.
lkanies 15y ago
(disclaimer - Puppet Labs employee) 2-3 months is a long time. We consistently hear 2-6 weeks to get to full production, with 30-120 minutes for some proof of concept work. And one of the great things about Puppet is you can start very small - automate the parts that are hardest, most critical, etc., and do bits manually as it makes sense for your problem set. And yeah, managing dev and prod the same way is absolutely critical for clean deployments.

leandrod 15y ago

I cringe at CouchDB. Gimme good, old PostgreSQL.

codypo 15y ago

I agree with the author that Chef is an incredibly powerful tool and that it has numerous benefits over plain old bash. However, I'm reluctant to actually use it because of its dependencies. Chef relies on CouchDB, RabbitMQ, and Solr, and all of those have non-trivial dependencies as well. Then, with a stack like that, I worry about the overheard involved. FWIW, Puppet's dependencies are much simpler. I don't know much about Chef vs Puppet, but I can say that from an installation and dependency maintenance POV, Puppet wins.

atsaloli 15y ago
Speaking of dependencies, CFEngine 3 is written in C and has 3 dependencies: berkeley db, libcrypo, and PCRE. It compiles into small binaries and is usable anywhere - in the cloud, on supercompute clusters, on the desktop or laptop, on a smartphone, in embedded devices.
dholmen 15y ago
You really should check out CFEngine 3. Very few dependencies (pcre,berkeleydb,openssl), and they also provide free packages with all the dependencies included: http://cfengine.com/download http://cfengine.com/download The memory footprint is about 10 MB, install size maybe 30 MB.
Fluxx 15y ago
You can use "Hosted Chef," where Opscode hosts all the dependencies for you: http://www.opscode.com/hosted-chef/ http://www.opscode.com/hosted-chef/
- sjs 15y ago
  If you've already decided on Chef then the Opscode platform provides great value. You get the best and most knowledgable Chef admins handling your Chef server for a few bucks per hour.
  - dlsspy 15y ago
    If you haven't decided, it's a great way to try it out. I'm an opscode customer, but before that, I was playing around with their free plan.

adient 15y ago

I think this post should be titled Why Configuration Management?, with a subtext of using Chef as an example. The main points the author is making are true of CFEngine, Puppet, Chef, and other config management software. The question of Why Chef? can be answered very simply: because the author likes it. Which is a perfectly valid way to choose your tool chain, assuming the technical requirements are met.

bryanwb 15y ago
I wrote about that it the previous post, see her http://devopsanywhere.blogspot.com/2011/10/puppet-vs-chef-fight.html http://devopsanywhere.blogspot.com/2011/10/puppet-vs-chef-fi...

grandalf 15y ago

For < 50 servers, can anyone comment on my choice of fabric and cuisine as an alternative to chef?

thibaut_barrere 15y ago
I decided to use chef-solo instead, mostly as a way to start learning chef but it ended be good enough for me. This way I'll be able to invest in the server part of chef later on, when I need it.
JoachimSchipper 15y ago
The number of different configurations is far more important than the number of boxes, no? If you have 1000 completely identical boxes (e.g. big compute cluster), a shell script that sets up a freshly installed box to your requirements is quite possibly sufficient.

nona 15y ago

Two reasons I'm wary of Chef wrt Puppet 1) the declarative vs imperative aspect 2) Chef's heavy dependencies As a Ruby developer, I like Chef's Ruby DSL; but I somehow feel that its imperative DSL will lead to something similar to Bash Hell. I'd like to read more about the declarative properties of Puppet vs the imperative way of Chef, and why one should prefer one over the other. Secondly, like some commenters before mention, Chef's dependencies seem quite excessive.

parasubvert 15y ago
Chef is becoming more declarative as it grows. The main difference with Chef is that it has imperative structure (i.e. ruby-based scripting) to fall back on, whereas Puppet forces you to go out to a script if you want to be imperative. In the long run, it's arguable that a strict declarative structure has more interesting properties when dealing with query of status & handling changes or drifts rather than "stamping out a server". Puppet has been great for me to detect and correct drift, for example, all the while ensuring pre-requisites are executed in the proper order. The tradeoff is that you have to think through the various dependencies at a detailed level, which can be difficult. An extreme analogy would be programming with a logic language vs. an imperative language. My experience has been Puppet is growing in popularity with enterprises, not just web companies, and it seems Puppet Labs is targeting this audience more than Opscode is, comparing customer lists. I'm not quite sure why this is - could be attitude - Chef is more about "get it done now", Puppet more about "get it right for the long haul", but even that is a caricature.
- lkanies 15y ago
  (disclaimer - founder of Puppet) We at Puppet Labs have always focused on building tools that anyone can use, not just the best hackers in the world. We've got some of the best sysadmins working with us, but we punish them by making them write software that even people who aren't the best sysadmins can use. So yeah, we get a ton of enterprise adoption as a result, but we've also got a ton of web companies using Puppet. It's true that in the Rails web startup, we're not always the number 1 choice, though. :)

gpapilion 15y ago

I have three fundamental issues with Chef; 1. The node object ends up being two large, which leads to memory issues when a search returns more that 200 nodes (800MB of memory). 2. Chef discourages declarative configuration. 3. Chef lacks a remote trigger mechanism. Issue one starts to kill you once you have a large number of nodes. Dedicating a quater of the available memory to configuration management seem like a poor financial choice. We've come up with work arounds at my company(generate files centrally, and distribute with chef remote_file syntax), but I still feel they are hacky. Issue two is more serious. Reindexing on chef only occurs after a node has submitted its node object back to the chef server. This results in incomplete searches until a node successfully completes a run. If you wish to remove a node from a particular role or attribute from a host, you may have a hard time doing so until the next chef run completes. Problem three is really a result of the expense of running chef. If the memory and CPU costs were lower, there wouldn't be any real issues running Chef more frequently. Some changes I need to go out immediately, some don't matter. I end up back in the world of the SSH loop too often with Chef.

jtimberman 15y ago
1. We are working on making the search more performance and use less memory. 2. Chef definitely does not discourage declarative configuration. Chef recipes include declarative resource for configuring your infrastructure. Since recipes are an internal ruby dsl, there may be nondeclarative code in them. 3. Chef itself doesn't have a remote trigger mechanism because the. Her run is all about configuring the local node. Nothing prevents you from using the ruby language in a recipe to hook up some kind of remote trigger though. People in the chef community are doing this with projects like Noah and Pylon. Github.com/lusis/Noah Github.com/fujin/pylon
- gpapilion 15y ago
  1. I understand that, and have heard that from you guys several times. The issue is the deserialization from JSON. The monkey patch solution I've see so far stores less data, essentially white listing the attributes for search. It sort of destroys the value of search. 2. I should be more specific. Generally chef relies on the information that ohai provides, not with information enumerated by the administrator. There is a general assumption that the systems are properly configured, and chef is only furthering that, since the hosts provide most of the configuration details. (Yes, you could do stuff with data bags to address this issue.) 3. Thanks for the links. I've not seen those projects previously. I've solved the issue for myself using a much lighter weight solution.

kim0 15y ago

I've been working with juju https://juju.ubuntu.com/ https://juju.ubuntu.com/ (a new Ubuntu server tool) and been very happy with it. It does not directly compete with puppet/chef for enforcing a server configuration however. Juju operates at the higher level of "services" that can be deployed to a cloud, to hardware, or to local LXC containers! It's basically like "apt-get" except for servers. The dev team hangs out at #juju on freenode irc