31 ms·
Lessons Learned from Writing Over 300k Lines of Infrastructure Code
- mehh 8y agoThat your writing too much code?
- yjftsjthsd-h 8y agoAre you criticizing the abstractions, or amount of infrastructure?
- mehh 8y agoThats code on top of frameworks, to do roughly similar things. As a software engineer I wouldn't be happy writing, and definitely maintaining so much code. His points are fine, but shocker its the same principals that apply to all code, infra isn't that special, thought we established that years ago. Also starting by boasting about the number of lines of code you have written to achieve something is asking for trouble. It feels like many in the 'devops' community that are from an ops background are rediscovering software principals, this chap isn't alone in that!
- scarface74 8y agoInfrastructure is special - if I make a change to my code, it usually won’t kill a whole database, load balancers, knock out connectivity, etc.
- mehh 8y agoA load balancer is not much use if the application on the other side is borked because of the application code is faulty, I don't really see your point. What I do see is 'devops' typically changing code in critical areas without taking sufficient care, but thats more a cultural thing than infra code being special.
- scarface74 8y agoIf you make a minor mistake in application code, it usually doesn’t affect the whole site. Besides, when you are deploying code on a group of servers, hopefully you have sense enough to at least do a rolling deployment and are using automated health checks to make sure that your whole site isn’t down during a deployment.
- meh2frdf 8y agoNor does my infra code, because I write tests, and have a pre-prod end to test it in a controlled ci approach.
- halbritt 8y agoAgreed and I'm from the "devops" side of the house. I don't see a lot of build pipelines with automated tests and deployment for IaC. Standardized processes like pull requests with code review are also not ubiquitous. With terraform one typically writes a template, does a plan, then apply and hopes for the best. What I've seen and the OP mentions the same in his talk is that things occasionally go terribly awry in ways that common software development practices would prevent.
- yjftsjthsd-h 8y agoDefinitely appreciate the prod-ready checklist; it's super useful to have a sanity check that you didn't miss anything.
- tarp 8y agoHere is their full checklist: https://www.gruntwork.io/devops-checklist/ https://www.gruntwork.io/devops-checklist/
- Boxxed 8y agoI'd like to hear the logical conclusion of "Large Modules Considered Harmful." Is he implying infrastructure code (e.g., terraform or ansible or cloud formation) should be split between the repositories for each component? Or just saying, "be smart about how you layout the infrastructure repo"?
- brikis98 8y agoMore of the latter. Don't put all your eggs in one basket. Don't create a single "module" (i.e., single deployable thing) with 100,000 lines of code and all your infrastructure in it. Break things up into small, reusable, composable pieces. This is what you typically do in any general purpose programming language, and it turns out it's a very good idea with infrastructure-as-code languages too.
- btschaegg 8y agoSo much this. I've gotten so tired of seeing the same story again and again: Someone sees something that's done by multiple pieces of code and goes "I´m gonna write a framework for that!". Two months later you've got an unmaintainable behemoth of a god class that does everything under the sun and drives everyone insane who has to look at it (a Cthulhu class, if you will). So, yes, I got to the same conclusion. Basically, solve every part of the problem: - in isolation/as focused as possible - as a library - with the easiest to use and most composable API you can come up with After that, if you have to, you can still glue them together into a framework-like thing. But at least, anyone can pick any feature out of it without succumbing to madness. Also, this is a great way to avoid the whole "Big Ball of Mud" problem that forces you to pull a whole ecosystem into your project although all you wanted to do was log something.
- idunno246 8y agoI think Segment’s terraform repos and blog show the logical conclusion of this pretty well. I’ve gone through the process of splitting up a single cfn stack for the entire environment, it was less about repo structure and more about deployment units for us https://github.com/segmentio/stack https://github.com/segmentio/stack
- justaaron 8y agonice. it's rather on-point.
- mjevans 8y agoIs there a version of this that is //just text// instead of a whole bunch of images and other resources that don't load without scripts enabled?
- jermaustin1 8y agoThe text loads without problem right?
- brikis98 8y agoThere's a transcript of the talk on the HashiCorp website: https://www.hashicorp.com/resources/lessons-learned-300000-lines-code https://www.hashicorp.com/resources/lessons-learned-300000-l...
- amanzi 8y agoTry using Mercury Reader or Outline if you don't like the way they chose to present the article.
- itronitron 8y agoanyone care to take a stab at defining what the author means by 'infrastructure code' ?
- merlincorey 8y agoMost people probably think of Terraform[0] or Fugue[1] here, but it also includes the more venerable likes of CloudFormation[2] and OpenStack Resources[3], both writeable in Troposphere[4]. It also includes any random mess of shell scripts and the like that manage the lifecycle of infrastructure. [0] https://www.terraform.io/ https://www.terraform.io/ [1] https://www.fugue.co/ https://www.fugue.co/ [2] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui... [3] https://docs.openstack.org/heat/latest/template_guide/openstack.html https://docs.openstack.org/heat/latest/template_guide/openst... [4] https://github.com/cloudtools/troposphere https://github.com/cloudtools/troposphere
- isodude 8y agoCode that defines what your servers/services are, what they should contain, where they should be put, how they should interact. E.g. a simple plan in Terraform gives you the possibility to say how many of what instances should be up in given zones. Edit: every piece of tool used here is used for infrastructure, controlling them is infrastructure code: (it's massive!) https://landscape.cncf.io/format=landscape https://landscape.cncf.io/format=landscape
- superfrank 8y agoInfrastructure as code is starting to be a big thing. Lets say you want to provision a new AWS EC2 instance, you could go into the UI and click around to do it, but at a large company, that's probably not the best idea. That's not really scaleable and leaves a lot of room for human error. You could instead use something like terraform, which allows you to write code specifying your requirements and then run that code. This allows other devs to review your code, takes a lot of human error out of the equation, and is much more sustainable. When he says infrastructure code, I think he means code that keeps the infrastructure running. Some examples that come to m would be terraform files, dockerfiles/docker-compose files, jenkinsfiles, bash scripts, etc. Basically, code that keeps the servers running.
- perfunctory 8y agoI recently replaced ~2000 lines of code written my predecessor with about 200 lines. The resulting code also performed 1000 times faster. After many years of real world experience I developed the following rule of thumb. A competent, experienced developer should produce about 20 lines of code per day, on average, to have any hope for a decent code quality. I am talking about high level languages with big standard libraries and reach ecosystems like python. At that rate 300,000 lines of code is ~41 menyears.
- simplify 8y agoYou should probably scope and clarify your rule of thumb. For example, writing a function that renders readable error messages to a UI will be far more than 20 lines of code.
- Boxxed 8y agoSo? That means it will take more than one day to write, by this metric.
- simplify 8y agoIf you have your list of possible errors, it really shouldn't take that long to write a `errorObj -> string` function.
- Jtsummers 8y agoIt’s an average. Novel problems: low SLOC/day. Common or “solved” (but not implemented) problems: high SLOC. EDIT: “Autocorrect” got me.
- k__ 8y agoI think when writing UI or 3D code 20 LoC/day isn't enough, but yes there are probably parts of the stack where this holds true.
- pmiller2 8y ago
- sulam 8y agoLike: the emphasis on tests, with real pointers to testing Terraform plans. That's legit, and hard advice to come by. Dislike: 300K lines of infrastructure code, but all he talked about was DevOps issues? Surely he learned something about writing good code, too, and I hope he didn't write 300K lines of Terraform code.
- deleted 8y ago[deleted]
- cataflam 8y ago> hope he didn't write 300K lines of Terraform code. If you look at the company, their product is providing infrastructure code, it was not a byproduct.
- auslander 8y ago... , then replace all that Terraform code with Cloudformation templates. You will get 5x less code, native vs third party tool v0.11 and my respect :)
- yjftsjthsd-h 8y agoYou are the first person I've encountered who preferred cloudformation over terraform. CF has far worse ability to cope with out-of-band changes or problems in general (in contrast to TF, which can usually just `apply` things back into compliance), and somehow gets feature support slower than TF in spite of being 1st party.
- auslander 8y ago> just `apply` things back into compliance If your things are wandering out of compliance by themselves, you have bigger problems, imho.
- ucarion 8y agoLikely the commenter is referring to development environments, where it is common for infra to be put in bizarre states by actors performing changes. Terraform alone is does not constitute change control.
- scarface74 8y agoThen don’t do that. But CF at least now has drift detection But what I prefer about CF is the “easy button”. If something weird happens or I can’t figure out something, I can use our business support plan with AWS.
- yjftsjthsd-h 8y ago> drift detection As of 2 weeks ago[0] drift detection exists, but coverage is poor[1]. [0] https://aws.amazon.com/about-aws/whats-new/2018/11/aws-cloudformation-now-supports-drift-detection/ https://aws.amazon.com/about-aws/whats-new/2018/11/aws-cloud... [1] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-stack-drift-resource-list.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
- actionowl 8y ago> Make sure your team has the time to master these tools What a luxury!
- isodude 8y agoReally though, if you can't master the tools, maybe you should avoid them.
- jacobr1 8y agoThere are degrees here. Depends what kind of scotsman we are talking about. Having a general understanding of the uses of the tool, understanding what feasible in general, some best-practice rules of thumb and knowing how to google for details gets you most the way there. You could call that mastery. It certainly requires greater expertise than "I read a blog post and changed a few things until it seemed to work." But the bar is much lower than: is familiar with the source code; or even you can use the tool for common uses cases without recourse to the documentation. I think you do need to invest in getting over that initial learning curve such that your tools aren't "magic." You have a conceptual mental model of what is going on. And I agree that takes much more than a token effort. Yet for me, that also stops short of mastery.
- isodude 8y agoWell, it should at least be possible to become master. "I know how this works but someone else did the grunt work". I always aim for simple to use tools, if I need to I can dig deeper and just-fix-the-problem. Or replace it at a whim. So I agree with you. Infrastructure code is really tough when it's scaling, so keep your modules in shape/order and small/easy.
- opportune 8y agohas the time to master =/= can't master At a lot of companies, asking "can I have a week to play around with these new tools/frameworks before I get started with them" will be met with "no way". Of course, not saying that's good: often the new tools are strong enough that you would still long term get massive development time savings even with the upfront costs. But a lot of managers and PMs are only focused on "shipping" ASAP, never mind the technical debt or maintenance costs (and to be fair, sometimes you need to ship ASAP... but sometimes you don't)
- isodude 8y agoWriting infrastructure code is a pain because it's so much more than everyone think it is. I think this article puts it well and hopefully makes it easier for folks to explain it to their team/boss. There's many single-devops out there facing this challange.
- vinceguidry 8y agoI'd say it's mostly a pain because it interfaces with parts of the system that you only touch a few times a year if it's working correctly, but yet is totally outside of your domain expertise. Nobody works with directly Ansible or EC2 instances enough to be an expert at it. Like how every time I need to renew a SSL cert, I need to reread the man pages for how to make certs and cert requests. You'd think a replicable procedure could be documented, but there's just enough variables so that it's slightly different every time.
- fogetti 8y agoThe vast majority of developers don’t know what those details are Let me fix this for you. Vast majority of developers are actively denied access to the infrastructure by their managers. They pretty much know everything what an infra eng knows (and even more, after all all this catchy cutting-edge apps like Kubernetes/Elasticsearch/etc. was written by them) or even more. Blame the managers and no-one else.
- auslander 8y ago> developers are actively denied access to the infrastructure As it should be. Devs make apps, Infra guys, SRE, make these apps run in production. Different skills, PHP vs sysadmin's work.
- fogetti 8y agoVery outdated thinking. As per Martin Fowler's advice if you write microservices you should organize your code around products and use vertical teams. The guy who writes the frontend should be able to write the deployment code for it too and maintain it for the complete lifecycle. If a devops eng can and want to do only sysadmin work he should travel back maybe 20 years in time. That was common back then.
- auslander 8y agoI mentioned sysadmin skills just for example of what is out of scope of apps or web devs skills. I may add networking, load balancing, monitoring, autoscaling, DB sharding, DR and so on. And all the above requires coding skills to make it work in the cloud. Ask your frontend guy about SR-IOV, just for fun :)
- fogetti 8y agoThese are all integral part of the dev skills. So your point is? I worked in all kinds of companies. The lamest and slowest ones were these outdated thinking ones. The most productive and highest quality products were delivered in those companies where the team was responsible for their apps networking, load balancing, monitoring, autoscaling, DB sharding. You think these are somehow special skills. But you didn't provide any arguments.
- mlacks 8y agoI stopped coding long before I ever got proficient enough to say I was a “coder”, but have spent enough time struggling with “intro to x” beginner code writing courses to get a hint of what goes on when writing programs. I’m presently going through something similar to the authors situation as I’m migrating my leads into Microsoft Dynamics and learning to make the platform for my use case (real estate brokerage). It’s not “infrastructure” but it’s the foundation for my business I can say that even something as simple as correct implementation of software is not something that should be done at speed, or without serious consideration towards portability. For example, because I’m solo and also working another full time job, my emphasis is on completing the migration ASAP energy though I’m aware of small errors in the database that might bite me later. Also, while I’m comfortable with MS’s record of maintaining enterprise software, it’s slightly suffocating to feel that I will effectively loose my time investment should they decide to abandon Microsoft Dynamics
- Jorge1o1 8y agoThey're not gonna abandon Dynamics. If anything, they're doubling down on the whole 365 suite. (And they should, SharePoint and OneDrive for Business are really good.)
- nwmcsween 8y agoThe issue with configuration management is it creates a DAG above a build system that does the same, as in its superfluous.
- ezrast 8y agoConfiguration management provides opinionated abstractions over inconsistent and frequently user-unfriendly systems. You might as well say that C is superfluous because it all maps down to assembly.
- nwmcsween 8y agoI'm saying creating two DAGs is superfluous, fbsd, obsd, nbsd, etc all don't do this. If you want to abstract the CLI tools why do it in a non-portable implementation specific language?
- deboflo 8y agoLOC is a very very bad metric. How exactly is it counted? Include mass refactoring? Is there a better metric? Yes, a long list of testimonials from happy customers.
- pmiller2 8y agoAre there any good "worked examples" of infrastructure as code out there? Something one can learn good practices from?
- dom_hutton 8y agoThe https://github.com/gruntwork-io https://github.com/gruntwork-io & https://github.com/cloudposse https://github.com/cloudposse examples spring to mind. As for example implementations review cloudposse in depth or take a look at the https://github.com/travis-ci/terraform-config https://github.com/travis-ci/terraform-config repo. This answer is skewed towards infrastructure as code. Often conflated are things such as configuration management & provisioning.
- dom_hutton 8y agoLots of people in this thread bashing on LOC as a metric, which is fair. I'd just like to point out that infrastructure code is incredibly verbose as is, so the number is way overstated to start. It's a pretty good example of doctoring headline worthy titles. IIRC the author gave a talk of a similar name at the hashiconf recently.
- peterwwillis 8y agoAll of the points are very valid and valuable. But by reading only this post, you will have basically no idea how to go ahead and do all these things. You have to scour the internet, and collect every blog post and video ever produced on the subject, and intone them all, and start experimenting, and just sort of create your own version of what these things are. I'm going to interviews right now where people are trying to hire a "Senior DevOps Engineer". They seem to think hiring this somewhat low-level employee will magically imbue their organization with DevOps principles, and suddenly everything will be cheaper and faster. I tell them it may take them 2 years to build a solid DevOps practice, assuming they do everything right, and they frown. Why can't we just have the DevOps right now? To that I reply, go hire a company that specializes in DevOps. And again they frown. I don't think it should be this hard. I think we can consolidate all the important knowledge and pre-packaged work, collaboratively, to empower teams to actually get shit done quicker and better. I think we can make it so nobody ever has to go through the painful process of learning by trial and error ever again. And, of course, I think it should all be free. So I've started a wiki, and I hope I can convince a few more people to help contribute content. GruntWork, I love what you're doing, but I'm also hoping to make your business obsolete. :)
- perfunctory 8y ago> I don't think it should be this hard. Totally agree. Unfortunately, you are up against the industry that is not interested in getting shit done quicker and better. "Institutions will try to preserve the problem to which they are the solution" https://en.wikipedia.org/wiki/Clay_Shirky#Shirky_principle https://en.wikipedia.org/wiki/Clay_Shirky#Shirky_principle
- therealdrag0 8y agoCouldn't believe my ears when I heard an Ops guy say "we can't automate stuff, then what would we do?" He came around after he had 3x more work then he could ever do himself.
- auslander 8y ago> we can consolidate all the important knowledge and pre-packaged work Noble intent, but time for that is not yet. The state of cloud infrastructure practices today is a primordial soup of ideas, marketing and the myriad of tools trying to survive on the market, like Terraform vs Ansible vs Cloudformation :) Good infrastructure is hard, there is global networking with CDNs, DNS, CI/CD, Git integration, HA, monitoring, security, you name it. Thats why you will not find the code on Github, only blog posts on Medium, code brings bread and butter to so many people, including me :), why sharing it ? And most important. To create really reusable by many infra code is even harder. You need to feel what features should be there for everyone, and what is bloat. It'll take many times the effort of writing non-reusable architecture for one company. > .. And again they frown Made me smile :)
- w_t_payne 8y agoIs it just me or are we constantly repeating the same lessons, the same messages, over and over and over again?
- jbverschoor 8y agoSame cycles over and over again? Because people in tech don't learn from history.
- sigi45 8y agoWe do and do again and do it again again. And if you finally have it more or less done it in your local environment / company, you switch and suddenly you start again. But you know, my salary still increases anyway so shrug (still looking for a solution)
- w_t_payne 8y agoSo if I made an open source solution that automates a lot of this ... I guess it would help a lot of people.
- mattbillenstein 8y agoSeveral good tips in here, but slides 60-72 are probably the most valuable - don't do anything by hand, find some tools and automate everything using those tools.
- KaiserPro 8y agoOne glaring omission is metrics. In the production grade checklist, there is nothing mentioned about metrics. Infact there is no mention in the post of metrics or graphs. This implies that everything is done via logs, which is just horrific at scale. Everything should emit metrics: o hits per second? metric o Memory use? metric o upstream service response time? metric o That new lib you wrote? metrics This is especially important with microservices. Open trace is grand, but thats for after you've found where the problem is. Your metrics should be your single pain of glass that indicates the health and performance of your system.
- bsaul 8y agoi think it's called monitoring in the checklist.
- kgilpin 8y agoThere is a simple way to make it much easier. Use hosted platforms (eg PaaS) instead of self-maintained infrastructure. Unfortunately, most enterprises are still not willing to outsource their IT platform to the real pros and thereby make it someone else’s problem to keep that infrastructure up and running. They want to build their own in-house competency. But the problem is, if you’re a true DevOps pro, why would you work for some random company in which IT is considered a cost center, rather than working for either a hosted platform provider or a (for lack of a better phrase) “independent provider of DevOps”? IT and security managers are able to convince business executives that IT infrastructure needs to be kept in-house, even though today’s typical enterprise has no chance of actually building and maintaining a infrastructure that is 1% as good as what they could rent from a dedicated provider.
- mrunkel 8y agoThe problem with your theory is that even in hosted platforms, IT is a cost center. "The real pros" you speak of are also under the same cost pressures as everyone else. I have encountered many such services where after the "real pros" have set up a system, they are replaced by low cost staff who can maintain and minimally extend while billing...