23 ms·
The sad state of sysadmin in the age of containers (2015)
- kbrwn 9y agoThere are plenty of mechanisms in the container ecosystem to address each of the problems the author states. Building software isn't all that it is cracked up to be and it isn't very fun. Often times build steps for open source projects are not well documented or the build process is inherently difficult to push a paid product. Example: nginx
- voltagex_ 9y agoThe rantiness and constant NSA references obscure some really good points.
- cwyers 9y ago> And since nobody is still able to compile things from scratch, everybody just downloads precompiled binaries from random websites. Often without any authentication or signature. Apache has official mirrors that host repo files for various package managers so you can install using apt-get or whatever it is that replaced yum (dnf? dnf): https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.2.1/repos/ https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.2.1/re... So precompiled binaries from official sources are certainly available.
- bovermyer 9y agoI think the author's point was this: "Unless you compile it yourself, you can't trust it."
- rdeboo 9y agoHow is it more secure? Do you read the entire source code to search for backdoors?
- tortasaur 9y agoWeb servers hosting binaries seem to be compromised more often than git repos. Transmission comes to mind as a semi-recent example.
- deleted 9y ago[deleted]
- cwyers 9y agoBut this isn't a website hosting a binary. These are binary repos hosted by Apache, who self-hosts their VCS repos as well. The idea that Apache can be trusted to host one safely but not the other is absurd, and the idea that you are more likely to notice malicious tampering via MitM attack on 2.9 million lines of code than you are a binary is laughable.
- deleted 9y ago[deleted]
- bovermyer 9y agoI didn't say I agreed with the statement. Which I don't.
- cwyers 9y agoThere are over 2.9 million lines of code in Apache Hadoop alone, not counting dependencies. If you can't trust Apache, you can't trust Hadoop, regardless of whether or not you can compile it yourself.
- w4tson 9y agoEXACTLY. It’s just software. There’s no easy answers here. There’s vulns in everything from hypervisors to node modules. Building from scratch isn’t going help. Pragmatic solutions where possible. Like scanning containers, using OWASP tools on your repos etc
- elcritch 9y agoI wish I could upvote this twice. Auto scanning containers should become the default process for everyone.
- lbenes 9y agoThere are nearly 10 million lines of code in libreoffice, and yet I can and have built it from source just by typing: 1. $ git clone git://anongit.freedesktop.org/libreoffice/core 2. $ apt-get build-dep libreoffice 3. $ ./autogen.sh && make Just because something has a large code base doesn't mean we shouldn't be able to build it from source ourselves.
- lostcolony 9y agoDid you read all those lines yourself? Did you even confirm checksums matched before running them? I think that's the parent's point. You can build from source, but how do you trust the source? Is it any more egregious to trust a prebuilt binary from a specific website than it is the raw source? If you can't trust the binary being hosted by the author/caretaker, can you really trust the source being hosted or maintained by the author/caretaker?
- pjz 9y agoI don't think his point is so much about the source as it is about updating N containers. For instance, say there's a known libssl bug. Can you tell how many of your containers are running that version of libssl? And how do they get updated?
- deleted 9y ago[deleted]
- leetbulb 9y agoI believe this is true and if so, then the argument should be against package managers, not Docker specifically. Most (if not all) of the official Docker images are built either by compiling the binaries from source, properly installing the binaries from the base distro's package manager, or pulling and verifying a pre-built binary from the vendor's website. For most cases, I don't see anything wrong with any of these. Personally, what I like is no longer having to setup arch specific build machines containing all of the build tools and dependencies for all binaries that I wish to self-compile. Instead, I either use the vendor's Dockerfile which already contains everything it needs to build from source or I simply write my own if there is not one available. Building and distributing these binaries in the form of Docker images is a breeze using Gitlab CI and container registry and is just as easy with a small VPS and Docker Hub.
- Florin_Andrei 9y agoNo, the point was "when you routinely use binaries from a bajillion different sources of varying degrees of trustworthiness, bad stuff is bound to happen".
- mmt 9y agoActually, I think there's a subtle distinction everyone's missing (which the original article may or may not have been making): Unless one can compile it oneself, how can one trust that a particular version of a binary release correspond to a particular version of a source release? If the process is reproduced by another trusted-enough source and is identical to the official release, then I'd say one can go ahead and trust the binary release of either one. Sadly, I don't think this is generally done, though perhaps ones own spot-checking of the official release is enough. That's supposed to be the basis of modern science, too, though, of course, it's not generally done there, either.
- awakeasleep 9y agoI want a prediction market to offer bets on which big company is affected by the breach that starts the reckoning.
- politician 9y agoAfter the reckoning, how do you propose to collect your winnings?
- ddingus 9y agoWhuffie
- auslander 9y agoI bet $100 on Google. It has most user data.
- mmt 9y agoExcept.. isn't that based on the misconception that all of these "cloud" tools that "come from Google" are actually used there? Google isn't using Kubernetes with Docker on AWS (or even GCP). I'm quite confident that they're running their own software on custom bare metal, with custom virtualization/container abstraction layers and custom management software.
- pc86 9y ago> None of these "fancy" tools still builds by a traditional make command. Is there anything more "get-off-my-lawn" than "These tools don't use the thing I like!"
- ProAm 9y agoIn reality it's all peachy until stuff doesn't work and no one knows why, or how to investigate an issue or remotely where to begin to troubleshoot it. Change and evolution is good, but I think there is still a lot to be said about knowing the basics to anything. Levels of abstractions eventually hurt more than it helps.
- icebraining 9y agoHow are these tools any more abstract than make?
- Angostura 9y agoI think that may be hehind some of these tales of 'luddite sysadmins'. Sysadmins need to keep things running and complexity and dependencies, even if they being convenience is something that makes them nervous. It's not about being a luddite, its about being able to hold a mental map of how it all works, so that when it stops working you can dive in.
- kazen44 9y agoThis is very true. Having a mental map of how things work is vital for correct (and good) troubleshooting. Also, a ton of people use abstraction as an excuse to not learn a body of knowledge that contains fundementals (especially on the systems side). In comparison to network engineering for instance, where having (quite) low level knowledge of how protocols work and interact with each other is vital.
- mancerayder 9y agoAnd not to mention getting called at 2 am because something didn't build or the release bombed and having to examine, for the first time, some over-complicated mechanism to build things that goes through some pipeline where you're eyeballing large log files full of long exception chains. Devs see the world as one where velocity and progress is among the most important, while 'sysadmins' (a term no longer used by companies and recruiters, unfortunately) have to worry about keeping the applications actually up so they can be used. DevOps just seemed to have swept the sysadmin under the rug under some pretty words about breaking down barriers. It feels more that devs broke down the wall sometimes. The amount of infrastructure chaos and mess and confusion I see today (as a contractor bouncing around different places) seems higher than traditional infrastructure shops of 10+ years ago.
- bovermyer 9y agoTL;DR version: "No one cares about verifiable security anymore." The author defines a category of people and then attacks it for A) not verifying the integrity of its systems, and B) not doing things with make. A is valid and something that needs attention. B is missing the point.
- gigatexal 9y agoUgh make files. Kill me. Give me a bash script or give me death.
- triztian 9y agoWhats wrong with Makefiles?; they're quick and easy ways to describe how to build simple and _not super_ complicated systems. I've found this to be a great resource, hope you find it useful: * http://gromnitsky.users.sourceforge.net/articles/notes-for-new-make-users/ http://gromnitsky.users.sourceforge.net/articles/notes-for-n...
- gigatexal 9y agoIt’s ironic. I love python but the syntax for make files was a pain. They’re rather cryptic. I guess they’re a staple time to conquer them once and for all.
- thirduncle 9y agoThis 'make' thing is a powerful tool, actually.
- mancerayder 9y agoIt's trivial to call bash syntax from within a Makefile. The Makefile just breaks these things up into rules which define steps and dependencies for these rules to occur.
- myWindoonn 9y agoReproducible build tools are a thing. Try Nix sometime. nixpkgs has, say, Hadoop. nixpkgs can make Docker containers too.
- voltagex_ 9y agoDebian has made great strides towards this too.
- davexunit 9y agonixpkgs isn't rigorous about reproducible builds. Hadoop is actually a great example of this. They do not build it from source, a prerequisite for calling a build reproducible. Instead, they download the binaries that the Apache project has already built and run patchelf on them to make them work. https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/networking/cluster/hadoop/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/applicatio...
- deleted 9y ago[deleted]
- ixxie 9y agoThat is true, but in my experience in small scale personal desktop and cloud computing, NixOS is in practice reasonably reproducible on the system level. For me in terms of interface, the strength of the Nix ecosystem is declarative system, ops and service configs in the same language used for package and build specification. The technical strength is striving towards reproducibe builds by hashing the dependency tree to build an immutable store. Yes sometimes this means getting binaries but you can always pin the package version or even have multiple versions in tandem. The practical upshot of this immutability is system level rollbacks, which are generally reliable although there are ways to break it. Yes there is garbage collection. Nixpkgs has is quite an achievement, and yes it has its warts but we are working hard to make it better. If we manage to shape up the data science I will try it out at work too. Im very curious how it might scale.
- hodgesrm 9y agoI see this as more of an opportunity than a problem. The fact that Hadoop, Kubernetes, and other platform-like systems are complex to manage properly with good attention to security implies they should be delivered as cloud services rather than having everyone run their own. This enables K8s users to focus on apps while offloading management to specialists who can focus on running the services well. If you operating at large enough scale you can bring the "cloud service" in-house.
- scurvy 9y agoOne of the gating factors here is the both the speed of Kubernetes development (move fast and break all the things), and the terrible state of accompanying documentation. If "they should be delivered as cloud services" is some sort of k8s apologist stance for its sorry state of maturity, then we got issues. OTOH, if it's "You shouldn't run it in house unless you have an army of people to read every new commit", that's wrong, too.
- hodgesrm 9y agoFirst of all let me be clear I'm not an expert in K8s and certainly not an apologist for bad software. On the other hand there's a level of complexity in distributed systems that is impossible to avoid even in stable infrastructure. You have a design choice of trying to make the system as easy as possible to operate (at the cost of other features) vs. finding operating models that make it less of an issue. Personally I would rather spend time futzing around with my applications that run on kubernetes vs. trying to run kubernetes itself. It would be sufficient if Kubernetes services were portable across a marketplace of providers so I could pick a place to run my applications.
- hodgesrm 9y agoAlso as far as security is concerned it appears to me that a lot of people are deploying technology that they simply don't understand. This is not just a problem with Docker but with apps from ecosystems running on npm and pip. You can build images securely with Docker but it requires building them yourself, using private registries, checking carefully for vulnerabilities, and testing. If you don't want to do this, pay somebody else to do it right. There's no free lunch.
- arca_vorago 9y agoI was all ready to rant and then read the first part: "I’m not complaining about old-school sysadmins. They know how to keep systems running, manage update and upgrade paths. This rant is about containers, prebuilt VMs, and the incredible mess they cause because their concept lacks notions of “trust” and “upgrades”." Fair enough, rant preemptively avoided.
- Xaena 9y agoI'd love to see a revised version of this for 2018 (with fewer NSA tinfoil references).
- rdsubhas 9y agoAs a "major theme", the author takes: > Consider for example Hadoop. Nobody seems to know how to build Hadoop from scratch. It’s an incredible mess of dependencies, version requirements and build tools. And as the major introduction to the blog post: > I’m not complaining about old-school sysadmins. They know how to keep systems running, manage update and upgrade paths. Huh? Old-school sysadmins know how to keep systems running, manage updates and upgrades. At the same time nobody knows how to build Hadoop from scratch. At the same time, Hadoop build instructions themselves have curl|sh scripts or mirrors and the wiki page is outdated. And it uses Java (and thus maven/ivy). And that downloads the internet. According to the blog, Hadoop, maven/ivy/sbt/any dependency manager, package managers, and everything is broken. But the tagline is: > This rant is about containers, prebuilt VMs What does any of this have to do with the "Age of containers" and pre-built VMs? Is the author just talking about Gentoo/LFS-style "compile the whole system from scratch"? This feels like an incredibly rushed rant. I can only envision the author requiring to setup hadoop for the first time, breaking their head for a few days (it happens), and taking it out on everything.
- carrja99 9y agoIt's a rant, my inclination is that it has to do with a very specific situation the author is facing at work.
- djsumdog 9y agoThe author should have grabbed the .sdeb or the debian build scripts and tore them apart if they really wanted to make a point (if, upon examining the build, there was one to make). I mean there is a lot of cognitive load/disconnect we're talk about. As an ops guy, I can't look into every package. That's why I trust the package manager (apt-get, yum, whatever) and all the build maintainers who either volunteer or work on for Redhat/Canonical/SuSE/IBM/whoever. Things get through. That's why we have all those security people out there who are digging around for bug bounties and find crap like the recent Ubuntu Snap package craziness. Docker containers can be good. You can use an official Ubuntu or Alpine image, build your base, and create scripts to make sure your base containers don't go out of date. Most people don't do that. The official Docker containers are kinda a mess, but at least they're maintained. Grabbing some random container off Dockerhub? Yea that's not going to end well; unless you just use their source to build your own. Or if it's a container continually maintained but the person/company who wrote the service. Docker containers do need better security introspection and that's going to be a big deal going forward. But this article is all rant and some, but not enough, substance.
- loteck 9y agoThe clearest explanation of why this happens is at the end: Before, admins would try hard to prevent security holes, now they call themselves “devops” and happily introduce them to the network themselves! 1) The merging of devs into the sysadmin role was a product of: the work of sysadmins (particularly systems change control and security compliance) not being valued in our culture. 2) Devs delighted to be free of the shackles placed upon them by sysadmins who were encumbered by the concerns expressed in this article. If you were a devop who resolved to fix the problems bemoaned in this article, my guess is you would turn around in 60 days to discover you'd become a sysadmin.
- agentultra 9y agoI recall the idea of "devops" from this book: https://landing.google.com/sre/book.html https://landing.google.com/sre/book.html The stated goal of putting both systems administrators and software engineers on the same team is to reduce friction and increase communication. One of the worst, productivity-killing situations you can find yourself in when developing network software and services is caused by the traditional "old school" mentality of separating the two camps. When your software developers operate independently of your systems engineers and administrators they're forced to make assumptions about infrastructure, operations, and compliance goals. Both teams have the same goals so why are they not on the same team? I think some "old school" system administrators don't realize how costly such communication mistakes are. Getting 6 months into a development project to be told you cannot have a critical piece of infrastructure _for reasons_ is a costly, costly mistake. Containers are a smart solution to the build problem. Don't build your containers from public, un-trusted images! Build your own images. Run your own, protected, registry. You still have all of the compliance and validation necessary and you don't end up debugging failed builds because one machine out of a thousand is running on some minor shared library version not supported by your software.
- notyourday 9y ago> Don't build your containers from public, un-trusted images! Build your own images. Run your own, protected, registry. You still have all of the compliance and validation necessary and you don't end up debugging failed builds because one machine out of a thousand is running on some minor shared library version not supported by your software. You have just lost all the speed to production advantages of containers.
- jchw 9y agoHadoop is a rather extreme example... It's bad, but not everything nowadays is. Many newer pieces of software install entirely from source with one command. Also, this is not at all endemic of containers, there's simply zero connection. Dockerfiles tend to be very simple and easy to reason about. The most complex application I have is around 50 lines of Dockerfile, and that's mostly just made more complicated to arrange things for the best layer caching. I suppose we're supposed to believe that this is somehow worse than the days of debugging m4 macros and autotools just to get a build that doesn't work.
- gaadd33 9y agoDoes that Dockerfile build a container that only has that app in and it's required dependencies? Almost all of the ones I've seen given as examples seem to have an entire copy of the OS in.
- jchw 9y agoA container is basically a glorified chroot, so there is a few things not strictly needed, but I typically use Alpine as a base system, which has a shell, some core utils, and musl libc in just ~5 MB. Since it's on its own layer, it gets deduplicated both in build and at runtime with other Alpine containers (and many Dockerhub images have an Alpine option.) That being said, since Go binaries have no inherent dependencies, I have indeed made Docker images containing exactly one file: the Go binary. These containers are basically the same as fat binaries, with the benefits of Docker scheduling and networking at runtime.
- seabrookmx 9y ago> I have indeed made Docker images containing exactly one file To anyone reading this, you need some magic compiler flags in both Rust and GoLang to make sure it's a statically compiled binary (doesn't dynamically link against GNU lib-c). But yes, this is super neat. I also like how it reads in the docker file: FROM scratch ...
- 9y ago
- rb808 9y agoHe's right that security right now is a bit unreliable but random sysadmins writing scripts and manually configuring things is never a good guarantee either. As a dev I'd like nothing more than k8s or similar to be the standard platform to run applications so that everthing is standardized and I dont ever have to require a sysadmin. I think already this is possible more secure than the "good old days", but in the future I expect it to be more so.
- sonaltr 9y agonot to be pedantic but why should containers be patched for security issues? the entire point of containers (and orchestration systems) is the ability to push updates without downtime. Just update with a newer container....am I missing something? Also about container security - a strict process internally can easily help counter that (I believe Shopify had a nice talk about it at the Google Cloud Platform event in Toronto - everything from using trusted images, running only signed images and going through a security check for each layer). EDIT: To add to it, please don't patch containers - the entire point of an "image" is that if I run it locally and on my datacenter - it should behave the same - live patching them just voids this concept.
- claviola 9y agoHow many people do you see setting up a deploy pipeline that includes pulling security updates into the base image and redeploying as needed? In my experience, it's much more common to see docker images that have been untouched for months with zero accountability of what exactly is running there.
- secabeen 9y agoThat's my real concern: old, out of date images. How will we handle another OpenSSL-level vulnerability in 7 years, with bad code buried in containers that haven't been updated in 4, and for which the build infrastructure is no longer functional?
- Spivak 9y agoThis really isn't that different from having some pre-built statically liked app still kicking on your system with the source and/or build tooling long gone. There aren't really easy answers here. You can't fix bad software with more tooling.
- sonaltr 9y agoI blame that on docker hub. It's the fault of Docker - the company. They have a security scanning software that they decided was an enterprise feature. This sort of issue is to be expected if you claim that security to be an enterprise feature.
- YouAreGreat 9y ago> The first internet worm spreading via flawed docker images? Good question, why don't we see exploits of all that implicit trust to the degree that, eg, the DOS shareware scene gave your PC visible virus infection, or the early internet gave us worms that would bog down the whole net? My attempt at an answer: Because the black hats aren't hobbyists anymore. Visibility is for amateurs.
- icebraining 9y agoAre you saying that every new black hat is immediately a professional? Or that there aren't any new black hats? In every other activity of human life, new amateurs appear as the older ones become professionals. Where are the visible exploits from the new amateurs?
- YouAreGreat 9y ago> Are you saying that every new black hat is immediately a professional To a degree that's what I'm saying. Sentencing for "computer crimes" when perpetrated by non-corporate entities has reached epic levels. I'm sure qualified talent takes that into account.
- icebraining 9y agoHas sentencing significantly increase around the world? The US is not the only source of malware.
- _Kristijan_ 9y ago"Feels like downloading Windows shareware in the 90s to me." Bull's eye!
- sofaofthedamned 9y agoThat really is a superb quote.
- tw1010 9y agoIsn't this just an example of automation? Other than sunk cost, by what argument is it reasonable that the sysadmin job should in any way deserve to be protected or aught to continue to be a thing? To my ears containers sounds like they've kind of solved the problem.
- kazen44 9y agoHow have containers solved maintaining production systems? It's just moving the abstraction layer higher up the stack. Also, who will design and build security for these systems? Or care about low-level performance? A ton of dev's don't care about prod in my experience, and just want to ship shiny features. Containers and VM's are not very different from a deployment or maintance standpoint. The deployment strategies used with containers could be done with (lightweight) VM's a decade ago. (heck, jails have been used for nearly 25 years) Also, sysadmins have been automating stuff since forever. Without automation one inherently has a very unstable system.
- mcguire 9y agohttps://xkcd.com/908/ https://xkcd.com/908/
- johngalt 9y agoNo one is arguing that a sysadmin job should be protected. The article is more of a reminder that the sysadmin responsibilities don't disappear because we paper over them. Containers are a great tool, but they can trick people into thinking that certain things aren't a problem merely because they have been swept under the rug operationally. With that in mind, I don't see sysadmins going anywhere soon. Every layer of abstraction is supposed to 'eliminate sysadmins', but inevitably it becomes yet another system that someone needs to audit and maintain.
- philjohn 9y agoRun your own artifactory, have that as your docker repository too, only allow the use of jars that have been vetted ... it's what we do where I work, it's not hard, and it solves 99% of these gripes.
- ppeetteerr 9y agoRegardless of your take on the article, could we, at least, agree that all build tools suck in their own special way after a certain level of complexity is reached?
- ex_amazon_sde 9y agoEx Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it back to you as a service. Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works. Modern devops, with its million tools that break backward compatibility every month sometimes becomes the running joke at lunch.
- icebraining 9y agoIs puppet one of those tools?
- kazen44 9y agoDepends, heck. I have seem some guys do magic with CFengine and freeBSD. (freebsd is a nice OS from an ops perspective anyhow, and has a lot more common sense and sane defaults in a lot of places).
- fb03 9y agoYou are my hero.
- spiderjerusalem 9y agoWhat fascinates me about this is, and sorry for being morbid, but what happens when y'all die? Does knowledge of the lower levels of the stack go away with your generation, or will there be enough of us young ones picking the important stuff up?
- dsr_ 9y agoThat's part of the reason why my last two hires have been at the beginning of their career. For both of them, it was their first major sysadmin responsibility after having jobs involving tech support and occasional Linux experience. The key is to pick smart people who are good at learning and find complex systems interesting. Then, of course, you need to have interesting projects for them to work on.
- komali2 9y agoI do often think about how, by using large JS packages, it's very feasible that in the daisy chain of NPM dependencies, somebody's managed to slip in malware. I'm not sure what to do about it other than just not using NPM at all!
- draw_down 9y agoIt’s crazy that JS is the only language with security problems. People should stop using it.
- Can_Not 9y agoCheckout from npm retire and auditjs (there's also more non free options). You'll likely get some false positives (such as, who would host jQuery on a public/untrusted cdn? or why would you even pass user input into that?), but if a real malware actually showed up, you can find out as part of your CI process. Using the lockfile provided by yarn/npm is also a good way to reduce accidental unnecessary package updates.
- rambossa 9y ago> Ever tried to security update a container? Wouldnt the approach for security updates be to replace the container?
- icebraining 9y agoDepends on the approach, you can use mutable or immutable containers. In fact, the OpenVZ VPSs that were at one time reasonably popular were just containers.
- nunez 9y agoPretty much. Update the image or do the update in a container based off of the old image and package that
- thepumpkin1979 9y ago(2015)
- sctb 9y agoThanks! We've updated the headline.
- Fritsdehacker 9y agoI work for a hosting company and we host our own Openstack based public cloud. We have a hard demand that for production systems we build all binaries we use from source. We actually build these in docker and use that to deploy to production. What I'm trying to say is that the one doesn't exclude the other. And we actually use make quite extensively. I do however see the ops point, building from source hasn't gotten easier.
- Fritsdehacker 9y agoIn the case of Openstack though, the community made it really easy to build docker containers from source.
- mr_tristan 9y agoThis is definitely a rant that obscures the underlying point: the introduction of _untrusted_ or _unreliable_ network resources, frequently hidden in a string of dependencies. I'm baffled how often I see an someone throw this sort of craziness - "go fetch this thing from some random third party" - into very important places, such as the startup procedures of a container. It's something I see in a culture of the two person startup just trying to get something out the door. It's definitely "technical debt", and frequently, it won't get removed. Thus, you try to scale up to meet load, and all these new instances go time out on the same external resource that's randomly having problems... boom! At the worst possible time. Never mind the potential huge security gaps. But the specific _tools_ aren't the issue here. It's the culture of "ship something now we'll deal with fallout later". A lot of people start using Docker and won't ever look at the Dockerfile, or, will add a Maven dependency and won't even check licenses or security updates for _any_ of the transient dependencies. Cloud technologies and containerization make everyone just think "we can do things so fast now" and never, ever pay attention to details that can come back to bite you. On the flip side, it's a good time to be in cybersecurity; because this cultural problem will never, ever, get solved. :)
- alfalfasprout 9y agoYou know, I used to agree with you. But the reality is you have to weigh the massive productivity boosts that things like docker bring to the table vs. the potential issues it can bring. To a large degree, good perimeter security mitigates a lot of the concerns of containers themselves running slightly out of date software.
- kazen44 9y ago> To a large degree, good perimeter security mitigates a lot of the concerns of containers themselves running slightly out of date software. this is a very naive way of setting up a secure production enviroment. Your perimeter security is worthless if you are loading non public images which have malware or even worse, unknown malicious code in them. having a data breach or hack on your hands is something which could kill the company. That risk is not worth having a slightly faster productivity boost because you or your ops team is not able or willing to build a proper private repository setup.
- pmontra 9y agoIf a company has a threat model and a list of business goals then they would have at least a risk matrix and they might decide what to do: either go the slow way and build software they can trust or accept the risks, backdoors and all the rest. Most companies skip all of that and hope for the best. Not always a conscious decision. Sometimes they get their unpatched servers encrypted by some ransomware, remember they don't have any backup, close shop and move on to the next business idea. I've seen that happen.
- sysbell 9y agoThe fact is, many open source software projects provide their own Dockerfile (and in many cases, images). Using these is akin to downloading and deploying a release tarball.
- nwmcsween 9y agoThis has been my complaint from day one. Instead of docker have something like ports or pkgsrc and simply create tools that simplify sandboxing, like cgexec, a google kafel -> ebpf filter then all the package manager has to do is well package. Docker IMO is a mudball of concerns that need to be separated.
- gm-conspiracy 9y agoBSD jails?
- kazen44 9y agojails are actually very mature if you compare them to docker. They have a well working security system and sane networking. (docker just does endless NAT abstraction, which is terrible for certain use cases. Not to mention it breaks a ton of useful networking features)
- gm-conspiracy 9y agoI was asking a coworker the other day about using containers with a "shared" mapped filesystem subdirectory for UNIX file socket communication versus encapsulating everything in a TCP/IP network stack, but my coworker was concerned about the security risks of a mapped filesystem on the containers. I sometimes feel like a crazy person.
- nwmcsween 9y agoI don't know enough about BSD to really comment but do any of them have resource and syscall limiting (xgroips, seccomp) like functionality? IMO it should be mandatory for distributed application bundles.
- auslander 9y agoOpenBSD pledge
- auslander 9y agoSElinux, kernel namespaces and syscalls for Docker seccomp profiles. Docker is not production ready yet.
- 89vision 9y agoTo bake a pie from scratch, first you must compile Hadoop.
- golergka 9y ago> Back then, years ago, Linux distributions were trying to provide you with a safe operating system. With signed packages, built from a web of trust. Some even work on reproducible builds. > But then, everything got Windows-ized. “Apps” were the rage, which you download and run, without being concerned about security, or the ability to upgrade the application to the next version. Because “you only live once”. These two a both very valid and great approaches for solving different problems. Sometimes you're just a regular user without any valuable data that just wants to do things in a quick and convenient way. And sometimes, you're a system administrator that needs to evaluate the whole build pipeline and plug all the holes for production deployment. Both alternatives should exist, and one doesn't cancel the other.
- pferde 9y agoYes, but when "system administrators" with valuable data just want to do things in a quick and convenient way in production, we've got problems.
- wwweston 9y agoPerhaps interestingly, I've had some similar complaints about package managers like homebrew. I've noticed over the last decade that a certain level of knowledge about building from source seems harder to find via search or engaging in a community forum/chat. The assumption is often that everyone will be using the package manager (I'd say that's especially true on macOS, but it might be an artifact of me spending more time there since it's been my most used dev machine), so if you have trouble building something, the answer will frequently be "just use homebrew -- whoever maintains the formula will already have solved your problems for you." There's two problems with this: first, that's presumably true for the source build, and if you can hit a case that the source build doesn't work for, chances are pretty good you can hit a case that brew doesn't work for (my experience is that more often than not they go together). Second... I'm happier to use package managers for applications meant to live on my machine and never go live elsewhere (heaven knows there's lots of stuff I just want to install and get on with my life rather than fiddle with), but for applications I'm deploying, it seems to me it's generally wise if someone involved in the project has a picture of the build details and dependency graphs in their head. Some of the automation we're throwing at operations is really convenient. It's not simple, though. And depending on how much forethought is put into it, I'm starting to think of it as ... maybe not technical debt, but a technical credit card which is super convenient at times, but also can easily become debt if you're not careful with it.
- gbrown 9y agoYeah, project disclaimers like "We do not recommend building from source" are big red flags for code quality or dependency hells.
- FraKtus 9y agoI never installer homebrew on any mac I develop. When you see all the possibilities to customize projects, it's just ridiculous to want to build something with a one-liner and expect it will take thousand of decisions for you.
- zuni 9y agoIt is a mess, but one part of the ever transitioning "sysadmin role" that is valuable is, being able to code and understand it. That being said, good sysadmins (rare), can and do. Isn't most of this being driven by society in general though? Everyone wants everything now... we're generally feeding this trend to do whatever it takes in the shortest amount of time.
- endymi0n 9y agoI think the only reasonable answer is "it depends". After the new updated base OS images for our systems after Meltdown and Spectre were out, it literally took us ten minutes of human and about an hour of machine time to recompile all of our containers, run the tests and deploy them to production on our Kubernetes cluster, replacing the old insecure ones. At the scale of our systems, any grumpy sysadmin would have spent at least several days untangling the dependencies and carefully restarting all servers in the correct order after some manually `sudo apt-get`ing (and probably forgetting a few of the lesser used systems). Sure, typing "FROM ubuntu" (which by the way is a trusted and cryptographically image, contrary to the OPs concerns) leaves me at the mercy of whoever I trusted compiled that image. Then again, what difference does it make to trust whoever compiled that ubuntu-17.04.iso image I put on my CD? Or, as Brian Tracy taught me to say in these situations: "You may be right."
- kazen44 9y ago> At the scale of our systems, any grumpy sysadmin would have spent at least several days untangling the dependencies and carefully restarting all servers in the correct order after some manually `sudo apt-get`ing (and probably forgetting a few of the lesser used systems). if you are operating at suchs a large scale, your sysadmin should be automating things (not necesserly with docker mind you). This automation has been possible on *nix for the better part of a decade by now.
- mmt 9y agoAgreed. As a (potentially) "grumpy" sysadmin, I wouldn't be manually apt-getting or forgetting anything. In fact, one of the lessons I've learned is that, at large enough scale, automated mass upgrades can do the forgetting for me, so verifying (also automated) is important. As for any "tangle" of dependencies that may exist, I've never seen that be caused by any choice made by "Ops" but, rather, solely by those writing the appllication code being deployed. As such, it would apply just as much to a Docker image as to (what I view as) a traditional deployment.
- nova22033 9y ago>And since nobody is still able to compile things from scratch Is is true that we had better security when we had to "compile everything from scratch" whatever that means? Did people compile the OS from scratch?
- acheron 9y agoDid people compile the OS from scratch? ...yes. https://www.freebsd.org/doc/en/books/handbook/makeworld.html https://www.freebsd.org/doc/en/books/handbook/makeworld.html P.S., make sure you have a spare couple days before running make buildworld on a 150 MHz Pentium.
- Animats 9y agoIf you thought having to deal with old COBOL programs was a problem, it gets much worse. When there are 10 year old containers in production, and the parts to rebuild them are long gone, then you have a real problem. There's a reason that Google has internal systems which can and do rebuild everything from source.
- kazen44 9y agoThis is actually a very important point that gets ignored a lot. What will happen if public container images are no longer available for building your application? Your in a for a world of hurt if a major dependency of your application doesn't work anymore because someone pulled the image.
- Max_aaa 9y agoThis is why we have a local registry and for any container that is goes to testing or production or is needed for builds should be build-able from source. We still need to figure out how to best tackle the issue of online repositories being taken down or vanishing. For now we run apt-mirror.
- Spivak 9y agoThe same thing you do with any 3rd party dependency -- save a copy of the source. Pull from DockerHub all you want but keep the images and their sources in a private registry and deploy your actual services from that.
- nunez 9y agoThe right answer is to use your own repository and pull those images in-house where they can be scanned and verified, I.e. , run a container from that image, scan it using security tools, then action from there. You can also kind-of recreate a Dockerfile from an image using docker history: https://stackoverflow.com/questions/19104847/how-to-generate-a-dockerfile-from-an-image https://stackoverflow.com/questions/19104847/how-to-generate... In the general case, it’s the same as a Ruby Gem or NodeJS module going away from rubygems or npmjs; once its gone, its gone
- JohnStudio 9y agoI think the entire statosphere of DevOps is just about dead on the whole in 2018 .. in retro, after working with things like Docker .. and more specific industry variations beyond the Amazon tech, it makes no sense to dwell on the security / control of a dedicated systems admin professional since the tools are all outside the local domain anyway. The rest from VoIP to IoT to container services are managed whole-sale .... SysAdmin is a dino in the age of distributed tech and outsourced IT resources. I'm a programmer, so I'll take heat for it .. but I don't see a need for them anymore.
- mmt 9y agoI'm not convinced there was ever a need for sysadmins, especially from the point of view of programmers. Programmers have always been able to do that work themselves. It's just that they don't want to. They think it's boring, or perhaps even beneath them. It's a bit like tax preparation. I don't think that's really changed all that much, even today, even if it's now programming against the AWS API, judging by the number of job postings for such a role. Also, like with tax preparation, using a good expert instead of doing it oneself could save on a lot of money and/or future headaches, but it can come with "grumpy" old-fashioned-seeming nagging about procedures akin to annotating and saving receipts. Whether it's worth it depends on situation and, far more importantly temperament.
- tonyarkles 9y ago> I'm not convinced there was ever a need for sysadmins, especially from the point of view of programmers. Programmers have always been able to do that work themselves. > It's just that they don't want to. They think it's boring, or perhaps even beneath them. It's a bit like tax preparation. I'll half agree here, and half not. I'm coming at this as a programmer who has historically done ops as well, currently doing more ops than dev. There's parts of it that programmers don't want to manage, like Apache configuration, provisioning and sizing VMs appropriately, etc. That's the boring stuff, and I totally get it. I don't like managing most of that stuff either, and have got everything set up with service discovery to handle a good chunk of the configuration, and Terraform for managing resources instead of having to click through the Azure Portal, and Nomad for scheduling tasks in the clusters. Groovy. There's also the parts that the developers seem to have no clue about. They're very happy with all of these abstractions, but there's a lot of layers underneath. They think in terms of making HTTP requests; when those don't work, it's up to me to figure out that their library is keeping a pool of HTTP connections open, and the Azure load balancer silently drops those connections out of the NAT table after 4 minutes of idle time (without sending FIN or RST packets to either side, natch). When the app gets terminated because it's exceeded its cgroup memory limit, I'm the one helping them adjust their JVM heap size. And when their requests take too long, I'm there helping them look through their query logs to figure out what's happening. All of these cases have happened in recent memory, and it's never been a matter of "this is boring and/or beneath me", it's been a matter of "HALP! I HAVE NO IDEA WTF IS HAPPENING" Edit: so yes, I agree with you in the sense that I'm a developer who generally speaking doesn't need a sysadmin. I installed Linux for the first time when I was 12 years old, 22 years ago :), and I didn't have a sysadmin then. In those 22 years, I've seen a lot of weird shit happen, and most of the people I've worked with don't seem to be well equipped to dive 3 abstraction layers deeper than they're used to to figure out what's wrong.
- lasermike026 9y agoI'm out! I'm done. I started as a developer. I then migrated to sysadmin, then systems engineer, then devops, and back to developer. I am done with playing the platform game. None of it matters. What matters is writing code that does work leading to profits. Always be coding. CAPEX over OPEX.
- Nano2rad 9y agoThis complexity could make companies depend on commercial version even for open source software.
- Floegipoky 9y agoI feel like this has a lot to do with the "cattle, not pets" mindset and disposability of modern infrastructure. For instance the author talks about patching a container. But that's not idiomatic- instead you would include the patch in your build pipeline and deploy the new image.
- voltagex_ 9y agoUh, the Bigtop project mentioned in the Debian wiki doesn't seem to be working either. https://ci.bigtop.apache.org/job/Docker-Sandbox/112/BUILD_ENVIRONMENTS=debian-8,STACK=hdfs,label=docker-slave-06/console https://ci.bigtop.apache.org/job/Docker-Sandbox/112/BUILD_EN... Does anyone care?
- kokey 9y agoI think part of the problem is that Docker is the now popular answer to the problem of 'it works on my machine'. Unfortunately 'works on my machine' also involved going to the web site of a particular tool or library and following the steps recommended for quickly trying the tool out, which gives things like curl http://somewebsite.com http://somewebsite.com | sudo bash situation or following the steps in a blog post where someone quickly compiled a bleeding edge version of it, with all the dependencies to build it, on their Ubuntu laptop.
- jcastro 9y ago> Essentially, the Docker approach boils down to downloading an unsigned binary, running it, and hoping it doesn’t contain any backdoor into your companies network. I don't get why people still claim that this is "the Docker approach". This is not the docker approach, everyone hopefully knows this is an anti-pattern by now.
- batoure 9y agoI don't disagree with the OP here per say. However I do think OP made a mistake by using Hadoop as the example. Hadoop has much more in common with an OS at this point than a single application. The ability to download a single component of the eco system is mainly to support small scale testing and development. You wouldn't build Debian by going and getting each piece of the linux kernel from its source building those building all the pieces of the middle ware up and so forth... that is the whole point of having Debian in the first place. Do certain complex software eco systems need better support for fingerprinting their builds, Yes. Does all this mean the sky is falling... probably not.
- disordinary 9y agoThere's always been propriety code installed on systems, who knows what that big Oracle database is actually doing? Are these sysadmins actually looking through the open source code that they're compiling to make sure that there isn't a security flaw in plain sight? With Docker you can get containers maintained by a trusted source, as a sysadmin you don't have to deal with all the hassle of upgrading things, you can just replace the container with the latest version. With Docker content trust the containers are signed. There have always been cowboys out there and that has nothing to do with the tech stack.
- nineteen999 9y agoThank heavens people are starting to point this out. The last devops team I worked on had an obsession with shiny. Never mind that they couldn't bootstrap a new base database for their application anymore or automate the entire application deployment (even with a phased approach) on either VMware or AWS. They wanted to keep piling new tools (often with sub 1.0 version numbers) of new tools on top of an unstable foundation, and would just shrug when it fell apart in production (which it commonly did). I tried pointing out to them that by giving commit access to their internal puppet git repository to every developer in the building, they had effectively given root access to them as well. All I received were shrugs and blank looks all around. One thing from the article that doesn't seem to be discussed enough here in the comments is trend of pulling random Docker images from the internet and deploying on your infrastructure simply because it's easier to integrate random versions with feature X than maintain your own builds, or work with the vendor's provided packages to achieve the same result. The security implications of this in particular has been bugging me for years.
- mmt 9y agoThat helps to explain why there are so many DevOps job postings with nearly identical lists of tools, especially for startups here in the SFBA.
- nunez 9y ago> None of these “fancy” tools still builds by a traditional make command. Every tool has to come up with their own, incomptaible, and non-portable “method of the day” of building. 100% this. Learning Make a few years ago was one of the best decisions I’ve ever made. It’s simple (until it’s not), straightforward and available on just about any Linux and UNIX installation (version compatibility aside). Trying to make the case for POSIX-compatible Bash scripts has been tough, though. I also agree re: Docker. While you can secure a container image with USER statements in the Dockerfile and knowing how to give containers just the capabilities that they need to do their job, it is WAY too easy to run them as root and give them privileged access to things. It should’ve been the other way around. Also, every container orchestration platform seems like a really elaborate hack.
- nunez 9y agoI think a lot of stuff that I used to need to know —- OS kernel internals, hardware specifics, relatively deep network knowledge —— is less useful in the age of public cloud, containers and immutable infrastructure. There isn’t really a need to tune a kernel for performance or do deep troubleshooting to root cause OS issues and ensure uptime; if something seems off, kill the machine or container and let auto scaling or the container orchestrator take care of it. If something wrong with cloud networking, call AWS, as the most we can do is prove that it’s on them. Etc. While I still know these things, I haven’t had to employ them in a while. I will probably use those even less as I start getting into management. That’s a bit saddening. But the new stuff I’ve learned over the years, namely, designing systems like I’d design software, is amazing. Applying TDD to infrastructure isn’t something I thought I’d be doing, but here we are, and we have “modern DevOps” tooling (and lots of other things) to thank for that.
- auslander 9y ago> There isn’t really a need to tune a kernel for performance Maybe, but knowing SElinux, kernel namespaces and syscalls for Docker seccomp profiles are crucial for security, and security is the Next Big Thing. Sad you are getting into management. Lets push to undo that notion that manager is worth more salary than lead engineer. I'm pushing :D
- nunez 9y agoWhy is that sad?
- auslander 9y agoWell, you said yourself: > I start getting into management. That’s a bit saddening Less one engineer in the world?
- nunez 9y agoOh! I’m not at all sad about getting into management; I’m saddened that I haven’t put the lower-level stuff I know (sort of) into good use in a while. I want to go into management! We need more engineers in management!
- _bxg1 9y agoSeems like the security issue is easy to fix? Only installing binaries you trust and comparing them against a hash is basic computer admin 101; it's not specific to Docker. Just make sure you trust whoever made the docker image, and that they give you a hash to compare it against.
- auslander 9y agoSecurity is what the article is about, and it is spot on. Cloud infra today requires coding skills. Thats why so many web developers trying to do infra too. Without systems or infrastructure knowledge. True cloud automation engineer, a.k.a DevOps, a.k.a SRE, is the sysadmin learned to code more than bash. Python, REST, classes, methods, objects. Or web developer, learned Linux and networking in deep, down to userland to kernel system calls and routing protocols. These people are rare, hence the ruckus and mayhem. You will be hacked.