5 ms·
DevOps Engineering: Who does your automation?
Here a brief intro for those that aren't familiar with DevOps:
http://www.jedi.be/blog/2010/02/12/what-is-this-devops-thing-anyway/
Before being introduced to the concept of DevOps, we at Twilio struggled with what to call the role. It requires a different kind of thinking. To us, DevOps is about automation to improve productivity, reduce risk, and achieve scale. DevOps engineers should always be striving to deprecate themselves. It has been more then 2 year since we founded Twilio and we've never had a sysadmin or an ops person. This has forced us to write better software automation from day one. However, having someone that can provide leadership on DevOps issues (and fundamentally understands distributed systems/CAP/etc) to the whole team is extremely valuable.
Configuration management tools like Chef and Puppet are the beginning but a more complete view of infrastructure and service automation (e.g., http://redeyemon.sourceforge.net/) seems inevitable in the long term.
We are looking to hire a Lead DevOps engineer. Here is the detailed job description if you know someone that might be interested:
http://twilio.jobscore.com/jobs/twilio/devops-engineer/bSHjYWq2Sr37U6eJe4aGWH
- fragmede 16y agoLinky - http://www.jedi.be/blog/2010/02/12/what-is-this-devops-thing-anyway/ http://www.jedi.be/blog/2010/02/12/what-is-this-devops-thing... Call it what you will, 'DevOps' is just what you'd expect a good sysadmin to do in a sufficiently large org. That there is on going friction between developers and sysadmin ("it works on my box" is a valid excuse?) is perhaps a larger sign of the culture and people involved.
- WestCoastJustin 16y agoWhenever I hear that term I cringe a little. I think this was a word invented by developers (please correct me if I'm wrong) to bring sysadmins into the fold. It is definitely good to have an understanding of the application layer but developers also need to understand the systems running their applications. I work in a large organization (5000+ employees) as a unix/linux sysadmin. You have to automate deployment, monitoring, configuration management because there just isn't time to do anything else. We have a group (about 4) out of 30+ that have an overall view of what you might call devops. The rest specilize in development or sysadmins. We meet once a week and review all progress on the dev/ops issues. I like the term OpSec (http://en.wikipedia.org/wiki/Operations_security http://en.wikipedia.org/wiki/Operations_security) someone who looks after Systems/Security ;) Since a good system admin should also know a great deal about security. Maybe Dev/Op/Sec.. haha, but really ;) p.s. I found that job posting to be full of PR fluff. It would be nice to see job posting actually post what their looking for in a distilled form.
- emcooke 16y agoThanks for the pointer to OpSec term. DevOpsSec... I sense a tshirt in the making?
- WestCoastJustin 16y agoI first heard about OpSec from Roland Dobbins with Arbor. Really smart fellow who has many suggestions on network design / defence. I just think people need to specialize in their field (+ a general knowledge of how the overall systems works). Almost how a carpenter, plumber, and electrician each specialize but probably have a general idea how everything works. Do you really want a plumber building the foundation of your house? Just seems a little silly to me. Each domain is so large that I would rather have deep expert knowledge in a area than a general handyman who might cut corners or waste time with solutions he doesn't know about.
- gaius 16y agoWe have a group (about 4) out of 30+ that have an overall view of what you might call devops This is called "second line support".
- lil_cain 16y agoDevops may be what a good sysadmin does. I'm not sure that job is. Looks far more like they're looking for a developer to write sysadmin tools.
- blueben 16y agoThis is exactly true. Not to mention "DevOps" is not a job position, it's a culture and philosophy. In the past few months as DevOps has taken off, people listing for "DevOps" job titles usually mean highly proficient sysadmins. In this case, Twilio just wants a developer with a cloud management background. There's nothing DevOpsy about it. I'm sad that people are not taking the time to learn exactly what DevOps means and thusly commandeering the name for whatever they think they need.
- emcooke 16y agoCan't someone whose role embodies a certain culture and philosophy not have a title that fits that goal? How would you define DevOps?
- gaius 16y agopeople are not taking the time to learn exactly what DevOps means Now I know that "DevOps" means "those who forget history are doomed to repeat it". Whenever you come across something in IT you think is new, take the time to find out if mainframe shops were doing it in the 1970s... Back then sysadmins wrote assembly language to automate systems tasks and it was just an everyday part of the job!
- blueben 16y agoOuch. See, DevOps is not just "sysadmins writing automation tools". Your post illustrates my point perfectly.
- generalk 16y agoI expect developers and sysadmins to work together on these things. I'm always confused when they don't, and saddened when people think that's okay. For instance: I wanted to do a few projects in Rails. I made sure that my admins had links, guides, and all the info they needed when I started the project. By the time I was ready to launch, they'd already had Ruby, Passenger, and all my dependent gems installed. Deploy was a breeze. Later, a different coworker did a project in Django, but didn't breathe a word to our sysadmins. When he and the client agreed that it was launch time, our admins were scrambling to figure out what version of Python we needed, how to set up the environment, and what the deployment best practices were. Everyone was angry and the deploy was late, but the developer didn't see any consequences. Kills me.
- dedward 16y agoI feel the same way. In the old-world way of things, the Architects of the system would take all of that into account - the sysadmins were just there to keep the systems up and running and install the software per the developers instructions. What's new, or seemingly new, because of the rapid advance of technology and the internet in general, is the rapid develop/deploy cycle, which ends up requiring a much tighter coupling between development and operations... the two almost blend together. The lines are getting fuzzy.
- deleted 16y ago[deleted]
- emcooke 16y agoIn a 100% ideal scenario an organization would only need developers as every possible failure/problem would be covered and handled by automation. Obviously the real world is different. New code is deployed that has bugs, there are unpredicted events, there are complex failure scenarios that are difficult to automate etc. A DevOps engineer may write automation software in conjunction with developers to automate the operational aspects of business logic. Thus, it's a partnership between the DevOps engineer whose metrics are driven by availability/reliability/scalability/security and the developer who is trying to attain some business objective.
- blueben 16y agoYou say "DevOps engineer" when you really mean SysAdmin, but I upvoted you anyway because you get it.
- openfly 16y agoSounds like regular Operations work to me. I don't understand why it needs to be differentiated.
- mmt 16y agoSometimes I wonder if DevOps isn't really just those who would truly prefer to be developers but don't mind sysadmin work and, for whatever reason, started out doing the latter. I, too, cringe a little when I hear the term, perhaps because I've always viewed myself as a "pure" sysadmin. That is, this is the kind of work which continues to bring me joy after 2 decades. Application development, on the other hand, despite an ability to code, does not. CM tools (current and their predecessors like cfengine) as they tend to be implemented strike me as a way of avoiding administering the whole system, by focusing on a single component (servers) and reducing them to a lowest common denominator. It also fits the pattern of trying to solve problems entirely in a custom software solution, a trait I associate with developers, not sysadmins. This is the source of my inference of DevOps' true leanings. The reality is that the system consists of considerably more than fungible servers[1]. There's the underlying hardware, the peripheral hardware like disks, database software[2], network hardware and configuration, and service providers. These are all traditional sysadmin areas, with plenty of room for professional growth and overall benefit. So much so, that I have difficulty imagining why any SA (not a developer in SA's clothing) would wish to do more dev at the expense of increaaed mastery of the infrastructure. By way of example, something I wrote on LinkedIn, in response to why I'm not a fan of puppet: I'm uncomfortable with any system that attempts to maintain state in place on a running system, as well as possibly override the native packaging ethos. That the client takes its own local file inventory on a fresh system, rather than trusting the one the package manager has, is wasteful at best. What's worse, to me, is that it creates yet another system to administer. I haven't yet found anything similarly suitable for the Ubuntu/Debian world, but I very much liked Cobbler a couple years ago, as it was primarily a wrapper for what could otherwise be standard, standalone services (kickstart, package repo). Even without such a wrapper, yum/rpm or apt/dpkg can do nearly all the work" [1] Virtualization and "cloud" notwithstanding, as anyone who's experienced an outage with a hosting provider or an i/o perofrmance issue with a cloud server can attest. [2] Which can be like an encapsulated OS itself
- chuhnk 16y agoYea I think like some the term DevOps makes me cringe as well since people use it as a job title when it should be more thought of like agile development, just another term to describe a process of working. I use chef because I needed a way to consistently deploy software in exactly the same way every time to multiple servers. A repeatable deployment process that scales. Why did I pick chef over puppet or cfengine? Because I also wanted to learn ruby and the entire thing is in ruby, win. Plus we have a ruby on rails team in the office, if ever I needed a hand, they were there or if they wanted to write recipes to deploy software thats not an issue either. DevOps is just a new buzz word. My title is System Administrator. I have a depth of knowledge in the infrastructure of all our technologies and how they are interconnected. I know how to deploy, maintain, scale and do it all over again. My knowledge in programming in second to that but is necessary to automate repetitive processes. DevOps I guess is being classed as a way to bring the IT team together, making sys admins and developers aware of each others "agendas" when it comes to running apps on servers. I as a sys admin have to understand what each of our applications does throughout its life cycle, from the browser right down to the database. Because if something goes wrong, I'll be the first to figure out where that happened and relay this back to the developer. By understanding core functionality of the code I can make it clear to the developer, "look you are executing a query on the database thats scanning millions of rows because its not using an index, its occurring in this controller in your code". We are seeing this term DevOps be classified now as more and more startups are emerging, because its essentially alot easier to execute in a startup environment where everyone is talking to each other to roll things out, everyone is trying to solve problems together. In large companies you'll see a massive separation and segregation along with processes to keep people in their place and slow results. I hope that all makes sense. DevOps as a process is not new, the classification of it as this term is.
- gaius 16y ago40 years ago IBM called these guys "systems programmers". That this is a new concept to some people is symptomatic of a wider problem in the industry: everything needs to be shiny and new so every 10 years we reinvent the wheel with new jargon.
- nordgren 16y agoThere is a bit of the same movement in CCP Games where I work, it's not called DevOps here, but there are groups forming in both the EVE Online software team and the Core technology group with an engineering focus on QA, which in practice translates pretty well to the DevOps style of thinking about the needs for this type of development and quality work. Here's one of the advertised positions for example: http://www.ccpgames.com/en/jobs/job-details.aspx?jobid=101 http://www.ccpgames.com/en/jobs/job-details.aspx?jobid=101