40 ms·

The sad state of sysadmin in the age of containers (2015)

428 points by maple3142 7y ago

_def 7y ago
I don't know that much about the state of container systems these days, but to me it seems it's the "comfortable" way in the "security vs comfort" tradeoff. Use where applicable and hope for better days.
- ThreeFx 7y ago
  But there are some environments where security is (almost) the only relevant factor, e.g. banking IT-infrastructure. I don't want my bank to pull arbitrary images from Dockerhub because that increases their comfort. Conversely, if I were sysadmin at a bank I would most definitely be concerned what was running in my network.
  - buzzkillington 7y ago
    >Conversely, if I were sysadmin at a bank I would most definitely be concerned what was running in my network. I've worked at a bank. It's a docker file. Running Hadoop. On top of a Linux Container. Running in Rhel 5. On top of a vmware hypervisor.
  - ownagefool 7y ago
    In 2015 I was running docker containers for a Gov organisation, where: - docker containers were built from source - Dockerfile published with the code - Built in a new CI environment - Pushed, Pulled and deployed from the sha - Collecting network traffic, undertaking protective monitoring, that looks for those backdoors. Just because you can pull arbitrary bullshit doesn't mean you have to. Though for the record, the same sysadmins that whine about newer tools are generally the same ones that implicitly trust their older toolsets. Just because you can compile it, doesn't make it secure, so you need to be running monitoring solutions and hedging your bets no matter the tech.
- arberavdullahu 7y ago
  I don't think you should put security at any case in tradeoff, by your argument one can save passwords in plaintext because its easy and comfortable, but its just not acceptable and this would effect not only your app but every app that your users use.
  - ThreeFx 7y ago
    I see the value when running e.g. home instrastructure on some old laptops or other non-critical stuff. Most people have NAT at home and thus whatever services they pull and listen in their local network aren't exposed to the outside world, and they can enjoy mopidy/NextCloud/whatever without going through a lot of hassle setting it up. Of course once we come to enterprise environments the opaque build/deployment process is atrocious.
    - BlueTemplar 7y ago
      30% of the Internet already uses IPv6. NAT is obsolete and was never supposed to be used for security anyway...
  - contravariant 7y ago
    If you don't make a tradeoff you end up with passwords to long to remember and security to cumbersome to not circumvent. Security has a cost, as does the lack of it, hence the trade-off.
  - x3n0ph3n3 7y ago
    The trade-off is always between security and accessibility. If you want a completely secure computer, bury it at the bottom of a mineshift and pour a few yards of concrete over it. No one is breaking into that anytime soon!
  - geocar 7y ago
    This is not "the tradeoff", but too easily a straw man because bcrypt (et al) cost so very little as to effectively cost nothing at all. Think about your server consuming resources: It needs a [machine] password stored someplace in order to authenticate itself to those resources. If it stores this encrypted, then an operator must be present to decrypt it when (re)starting the server. If this is stored unencrypted, then it is not and the server can be started unattended (or autoscaled or whatever). That is a security tradeoff because there are both real benefits and risks gained and lost with each approach. If docker was ever a tradeoff, it was between taking the time to have a thoughtful architecture versus trying to get acquired, and I think we know which way they went.
majewsky 7y ago
Discussed at the time (443 comments): https://news.ycombinator.com/item?id=9419188 https://news.ycombinator.com/item?id=9419188 And again in 2018 (426 comments): https://news.ycombinator.com/item?id=17083436 https://news.ycombinator.com/item?id=17083436
- mike_hock 7y ago
  But unfortunately, it's relevant to this day.
  - majewsky 7y ago
    More than ever.
upstandingdude 7y ago
ha +1 for the build tool mess. its like everyone goes "duuude this stuff is so complex lets build something on top to abstract it all away" and you end up with another level of complex stuff that builds the stuff that builds the that wraps the stuff that you originally wanted to use...
pram 7y ago
Terrible article IMO. Hadoop is an awful mess, but it has nothing to do with Docker, which is simplistic in comparison. "Ever tried to security update a container?" Yes, I have! In fact you can maintain patch compliance in a container pretty much the same way you'd maintain a VM or bare metal Linux installation!
- black_puppydog 7y ago
  > Hadoop is an awful mess, but it has nothing to do with Docker, which is simplistic in comparison. The fact that they can get away with a build system like this is very much due to docker (and curl | sudo bash) allow people to not feel the pain this mess causes, at least not right away. > Yes, I have! In fact you can maintain patch compliance in a container pretty much the same way you'd maintain a VM or bare metal Linux installation! But then you lose the declarative/immutable nature of docker, no? Except if you mean every time there's a security update, you rebuild your containers?
  - onion2k 7y ago
    Except if you mean every time there's a security update, you rebuild your containers? That's exactly what you should be doing, at least if you're relatively small scale. Google probably takes a different approach but most of us aren't working at that scale.
  - pram 7y ago
    First: No one installs Hadoop from scratch, and Hadoop isn't built with Docker. Companies use Ambari or other distros like EMR. Second: Yes, generally you can use a build system like Jenkins, and a registry like Artifactory to automate the process. The docker image is updated, and then you can push it out with your orchestration in whatever method you choose. It's not an obscure or difficult thing to manage..
  - adrianN 7y ago
    It's pretty hard to do truly declarative/immutable things with docker because the typical dockerfile starts with "apt-get update && apt-get install ...". For that use case I think nix is much better.
Yuval_Halevi 7y ago
I saw this comment now from 4 years ago on reddit: Software packaging and building seems to be becoming more complicated and more disconnected-- particularly as more specific tools continue to be developed. It seems every little corner has their own dependency management and build management solution. The Docker comments seem a little backhanded. If one looks at a Dockerfile, there's not much to complain about. It downloads a key, it adds a repository, installs a package, and some very simple docker-specific tweaks. If you distrust Docker's signed image, building your own is as simple as doing a git clone ..; cd docker-nginx; docker build. Docker encourages disposable containers, separating data, and making your own images. There's still a lot of work for Docker to do regarding signed images, but I'd argue running isolated images with documented changesets and simple build files is far different from blindly running 'curl | sudo bash' https://old.reddit.com/r/programming/comments/33ktc9/the_sad_state_of_sysadmin_in_the_age_of_containers/cqlxne5/ https://old.reddit.com/r/programming/comments/33ktc9/the_sad...
jve 7y ago
This all boils down to trust. `curl | sudo bash` is no different than .\install.exe. The question is about trusting the SOURCE and trusting the DISTRIBUTION channel (that HTTP download from scala-lang.org violates this). Where did you get it? from https://microsoft.com/.. https://microsoft.com/... or from https://micro.soft.com/... https://micro.soft.com/...? Whom you trust more? The same with pre-built VM image or whatever... do you trust the party that made this image/container available?
- LockAndLol 7y ago
  `curl | sudo bash` should never have become a thing in the first place. To me, it's the most "WTF?" mode of installing anything. Especially when the source is HTTP. In what world is that secure? Pity Qubes is so heavy, otherwise I'd be using it
  - eejjjj82 7y ago
    would you feel better if your Makefile looks like this: install: [ $(id -u) -ne 0 ] && echo 'please run sudo make install' || curl https//my.thing | /bin/sh it goes back to what jve said - it's a matter of trust. how often do you blindly run 'sudo make install' w/o reading the entirety of the build script? all the time I bet - it's because you trust the source
    - LockAndLol 7y ago
      No. I think I have never run `sudo make install`. Things I install come from package managers. Docker or a VM is used to test other software. And of course trust is important, but if I encounter things like `curl | bash`, my trust is lost.
      - yori 7y ago
        What difference does it make if you run `make install` or `curl | bash`? In both cases, you are running code you have not audited yourself. Or are you the kind who inspects every Makefile and installer script before installing? If not, why is one better than the other?
onion2k 7y ago
Ever tried to security update a container? The whole point of using a container is that you can destroy it and build a new one easily. The new one should be built using up-to-date packages with security patches applied (and tested, obvs). Using the 'pets versus cattle'[1] analogy, patching a container feels like you're treating it like a pet. You should just kill it and get a new one instead. [1] https://thenewstack.io/how-to-treat-your-kubernetes-clusters-like-cattle-not-pets/ https://thenewstack.io/how-to-treat-your-kubernetes-clusters...
- dsr_ 7y ago
  The whole point of having a distro with a good reputation is that you can incrementally outsource your trust to them, one package at a time. OpenSSL needs a fix? One package update per machine, and if you have lots of machines, you update that. With VMs, you can use the same tools you use with any other machine. With containers, you are responsible for figuring out which ones need which packages and how to re-build them. Your method is likely to be idiosyncratic, so you not only need to be your own security team, you need to be your own packaging maintainer and infrastructure maintainer. The two extreme versions of this are the neglect model, in which you just don't care about anything, and the bleeding-edge model, where you automatically build with the latest versions of everything pulled from their origins. Both ways end up with unexpected security disasters.
  - regularfry 7y ago
    The "one package at a time" model assumes you care about per-instance uptime and bandwidth costs. The "reset the world" model assumes rebuilds and reboots are cheap enough not to have to. I don't think either is necessarily wrong; there's not a huge distance between `apt-get install unattended-upgrades` and a daily (weekly, whatever) container rebuild except that you need to cron the latter. Unless you're using a different OS between the two cases, they can use the same package database, the same tooling, and so on.
    - dsr_ 7y ago
      You need to make the decisions, keep track, and do the work. If you've got the tools made for you, excellent. Otherwise you also have to build and maintain the tools -- which is my point.
jwr 7y ago
I recently had a similar discussion with people using npm for building a CSS framework library. I tried to explain the concept of getting a pre-downloaded tarball and using "make" (or similar) to produce target artifacts from source files in a deterministic, repeatable and reliable manner, without relying on any third-party servers being available and without pulling in dependencies that might have changed. It seems the concept was entirely alien to programmers younger than me.
- mateuszf 7y ago
  It can be done using package managers too, for example using Nix. The problem here is not related to distribution method but rather dependency hell in javascriptland.
  - weberc2 7y ago
    I would say the main issue is that Nix is super difficult and requires you to intimately know your entire dependency tree down into Linux particularities. JavaScript makes matters worse, but even packaging a nontrivial Python package pulled directly from Pypi is often difficult. I want to like Nix, but it is far too pedantic to be practical. And this isn’t even mentioning the usability issues.
    - glglwty 7y ago
      I think you only need to specify your immediate dependencies in nix?
      - weberc2 7y ago
        Someone has to write the package definitions for those dependencies, and the public package repository doesn’t have broad coverage for many popular packages in many languages. This is understandable in that this is a massive effort, but that’s also the point—the effort to manage packages is huge and the extra effort compared to other tools is not a good bargain for many projects, nor is it much of consolation for project maintainers who would otherwise like to use it.
1337shadow 7y ago
The holy grail of sysadmin has always been deployment of immutable images, instead of host system pollution, source code mutation during deployment ... It was doable with things like Packer, KVM or OpenStack, now it's become easier with containers. You can still orchestrate your stuff with ansible or bash scripts. I most certainly remember how I was doing isolated deployments 10 years ago: https://github.com/jpic/bashworks/tree/master/vps https://github.com/jpic/bashworks/tree/master/vps HINT: it's much cleaner and much more automated with containers: with GitLab-CI it's trivial to host a private container bulding infrastructure with a registry and other automation ...
media-share-iq 7y ago
We providing A Feature-Rich Enterprise Video Platform That's Also Affordable and Easy-to-Use services like 1] Enterprise Video Platform 2] Video Management Platform, MEDIASHAREiQ E-mail : ems.marketing19@gmail.com Website : https://www.mediashareiq.com/ https://www.mediashareiq.com/ Mobile : +1-877-367-5050(USA)
buzzkillington 7y ago
Looked at the Debian wiki the OP linked to: >Debian currently does not include Hadoop packages. There are a number of reasons for this; in particular the Hadoop build process will load various dependencies via Maven instead of using distribution-supplied packages. Java projects like this are unfortunately not easy to package because of interdependencies; and unfortunately the Hadoop stack is full of odd dependencies (including Apache Forrest, which for a long time only worked with Java = 1.5). >If you want to build Debian packages, the most complete efforts can be found at the Apache Bigtop project http://bigtop.apache.org/ http://bigtop.apache.org/ . Unfortunately, the build process for these packages is currently of a disastrous quality, and should only be attempted within disposable virtual machines, as it requires root permissions and will install non-packaged software. >These will allow building Hadoop packages on a recent Debian system. However, the resulting packages do not live up to Debian quality standards. In particular, they include copies of .jar files from other packages, and will thus not benefit from security updates done to these packages. >If you are interested in getting Hadoop packages into Debian, Coordinate with the Java packaging team Coordinate with upstream Apache Bigtop >to avoid duplicate efforts. Thank you. https://wiki.debian.org/Hadoop https://wiki.debian.org/Hadoop Jesus Christ.
- alfozan 7y ago
  Who still uses Hadoop anyway? https://spark.apache.org/ https://spark.apache.org/ https://www.iguazio.com/data-science-post-hadoop/ https://www.iguazio.com/data-science-post-hadoop/
  - buzzkillington 7y ago
    Spark is worse because you need Scala as well as regular Java. I've tried building it for my day job, I would rather have a colonoscopy without sedation. It's more pleasant and dignified.
    - itg 7y ago
      And then add PySpark on top of that. Couldn't leave my last job fast enough when they decided to use Hadoop/PySpark when the largest incoming files we received were at most a few GBs.
_pmf_ 7y ago
Give me rsync and a sysadmin any day over a circle jerking dev ops team.
- Annatar 7y ago
  Replace rsync with OS packages and you got yourself a deal and a half.
- ex_amazon_sde 7y ago
  Amazon and some other FAANGs build and deploy OS packages, and by choice. Sad to see how anything that is not container gets downvoted here.
bureaucrat 7y ago
Ok boomer.
- Hnrobert42 7y ago
  Ad hominem attacks are unproductive and specifically disallowed on HN.
_wldu 7y ago
Standard Makefiles are really under appreciated today. They are simple and can be used with most any language, version controlled and have been around for decades.
- neuronic 7y ago
  But it's not a product you can sell or artificially inflate its monetary value so out the window it goes.
- adgasf 7y ago
  Makefiles are too low level to be convenient for anything other than simple projects on one platform.
  - gerbilly 7y ago
    If the alternative you are hinting at is CMake, I can't really agree with you. I tried it recently and was shocked at how awful the syntax and usability was. It's begging for a tool to auto-detect your dependencies from source. I was trying to port an existing project to it and I gave up when I realised that I'd have to manually create about 30 CMakeList.txt files or whatever they call them. Meanwhile during the same weekend, autoconf allowed me to build gcc from source with three commands. I know I know, I'm not supposed to like autoconf, and the orthodoxy is that it is garbage ... It seems every generation ignores the previous generation's tools and invents worse ones to replace them. If this continues we'll soon go back to banging rocks and sticks together.
    - adgasf 7y ago
      I am definitely not hinting at CMake! Better options are Bazel, Buck and Please. Please is the most light-weight option.
    - jeltz 7y ago
      Take a look at Ninja and Bazel. I ahree with you that CMake is worse than autoconf but I would argue that the tools which came after CMake are better than CMake and autoconf. So while there certainly are regressions not every new generation is worse than the previous.
t0astbread 7y ago
I'm looking at this from a relatively neutral newbie POV. I've tipped my toes in both administering machines the "classic" way and with tools like Docker and Docker Compose. To me both have their benefits and drawbacks but Docker looks a bit better overall. Here's why: 1) The idea of immutable systems rocks. It feels like the logical next step from containers. 2) This is maybe just because of me but I have the feeling services that can easily be (automatically) updated (like with Docker) are gonna be more up-to-date on average than when I have to build every new version myself. (Although that could be automated probably.)
dig1 7y ago
The same applies today for nodejs and go, but the same was happening with python, ruby and perl libraries in the past. However, this only shows that standard distro packaging approach is broken or was broken all the time. First, if you want to distribute you application (or library) via distro packages, you'd either have to build them and publish on your site or have to go through long mailing process to satisfy distro specific packaging guidelines and hope that your package will end up in standard (or extended) distribution. Debian is also notorious for "reviewing" and modifying source code, without contacting authors (remember Debian SSH fiasco). I had, not once, unpleasant experience with this process. Today, things are much easier - pack your whatever and put on dockerhub, maven or github. No middleman and no headaches just to reach your users. Sure, there are drawbacks (like mentioned in article), but we'd have to leverage things like GPG and proper package signing. Now, that is totally different story, because cryptography, even the basic one, is PITA for most developers and users. You can't have both :D
lmm 7y ago
You've got to look at this in a context where platform package managers like apt are simultaneously 1) platform-specific 2) jealous, insisting that every language has to conform to their way of doing things and 3) fundamentally not very good, having very limited ability to do things like install packages for a single user or install multiple versions of the same package. Platform package managers like maven have been far more successful because they provide a much better user experience, and frankly I suspect this is if anything because they're so thoroughly isolated from the platform package manager.
- marmaduke 7y ago
  No I don't. I use Debian derivatives because I can apt get all my stuff without thinking hard, because those package maintainers have done the hard work.
  - lmm 7y ago
    Well, clearly not, since you can't install Hadoop. Before you blame that on Hadoop, remember that it only requires some really quite basic things from its package manager, which apt is nevertheless completely unable to do: use libraries in the normal recommended way, play nice with Java, work cross-platform.
    - rumanator 7y ago
      > Well, clearly not, since you can't install Hadoop. You are complaining that you can't automatically install broken packages out of the box through the official repo. And packaging an application is the responsibility of the people working on that application, not the OS. What's your point?
      - int_19h 7y ago
        For Linux distros, the primary responsibility for packaging is actually with the distro maintainers, since packaging is distro-specific. That said, a well-written app should not present any difficulty for the package maintainer.
Havoc 7y ago
And it seems to be getting worse lately with everyone talking about their cloud-y build "pipeline". Which invariably seems to involve stringing together a bunch of different service offerings from startups that might not be here tomorrow. It's hard to imagine something more fragile
de_watcher 7y ago
http://kmkeen.com/maintainers-matter/ http://kmkeen.com/maintainers-matter/
AndyMcConachie 7y ago
Things will break and I will need to be able to debug them. New versions will be released and I will need to be able to upgrade to them. Bugs will be found and I may need to know if I or my users are experiencing them. Security holes will be found and I will need to either upgrade, or mitigate them in some other fashion. All of the above realities mean I want well defined and understandable software. The less well defined and understandable a deployment is, the less I am interested in deploying it. Complexity is a cost, and this cost cannot be waived by simply chucking everying into a container. Simplicity means cheap. It means time not wasted and it means fewer headaches.
rbc 7y ago
The only part of this article that I really disagree with is the title. The main issue here is release engineering, or rather, the lack of it. Software development and system administration have well developed traditions, but release engineering doesn't seem to get much attention in comparison. Given that supply chain attacks are now all the rage in the black hat community, maybe release engineering will start to get the attention that it deserves.
jillesvangurp 7y ago
The maven/gradle ecosystem actually does use signatures. That's one of the things they got right. Maven central has no unsigned artifacts. Also, the signatures are actually checked on download. Other maven servers may be a bit more sloppy of course. In any case, if you are running that on a production system, you are doing it wrong. You run that as part of CI. The output of CI is containers which you start on a production machine. The point about containers is that they should be immutable. Running chef/puppet in kubernetes is not a thing (I hope, probably somebody went there). Updating a container means replacing it with a new one, that hopefully you or somebody else built in a responsible way. That last part is indeed a problem that we have moved instead of solved. 20 year ago, people were just sticking whatever they downloaded on hardware they bought at the store and banged on it until it worked (very literally sometimes). So, I guess this is progress but indeed hardly ideal. I remember using puppet. Can't say that that is with any level of fondness. I vastly prefer Dockerfile and having CI systems produce containers from those. Installing things on a filesystem is no longer that common at deploy time unless that filesystem is that of the container you are building using a Dockerfile or you are self hosting kubernetes (aka. reinventing a lot of wheels at great cost). Puppet/chef etc. are still of use if you are doing that but otherwise it has a limited role in IAAS type architectures. The closest thing is perhaps pre-baking AMI images using packer and some tool like ansible, which is nice if you want to avoid having a lot of startup overhead. Hadoop is complicated, which is why companies exist that host that for you or will help you hosting it on premise in a responsible way. If you want to DYI, you indeed have to do a lot of things and do them properly. Kubernetes has moved that space forward in recent years; so it's easier but this is not for everyone. If on the other hand you are messing with puppet to get this stuff done, maybe reflect on the wisdom of not standing on the shoulders of giants rather than blaming the internet for your self inflicted pain.
axegon 7y ago
Talking from a developer's point of view, I've thought about this for a while and I think it has something to do with what I refer to as "anti-imposter syndrome" - the notion that anyone can make software. You see, over the past 10 years or so, self-proclaimed "educational" services/websites/institutions have been shoving down everyone's throat the idea that anyone will be able to create the next FAANG from scratch after just 3 months of training. Which is the same as claiming that I can become an F1 driver in 3 months. Where do I sign up?!?!?! The thing is, F1 results are a lot more visible since you have a point of reference - the top dogs. Chances of scoring a fastest lap or coming even close to them - little to none. Software - not so much - you have a shiny interface and what it does underneath is always a mystery to the end user. And people who have gone through those magical 3 months of training are often lead to believe that the way they are doing things is what everyone is doing and that is how it should be done. Often people who have signed up for those courses are people who have just a smudge above average technical knowledge - they have no idea how OS'es work, what should be considered safe or even why. Don't hate me for saying it, but essentially Windows users. And this is the software they end up building and distributing. 10 years down the line, thousands have picked up random scraps of knowledge from here and there and tried to mash something together. Don't get me wrong, I think technologies like Docker are astonishing and an incredible tools in the hands of people with knowledge and experience. However way too many people have picked up some scraps from there and created the unholy mess the author is talking about.
cuillevel3 7y ago
> I’m not complaining about old-school sysadmins. They know how to keep systems running, manage update and upgrade paths. Sadly these ways of olde didn't scale. They could only do it for a handful of servers, change request took months and the systems were not to be 'touched'. > And since nobody is still able to compile things from scratch, everybody just downloads precompiled binaries from random websites. Often without any authentication or signature. Right, that would have never happened back then, when packages were not signed and freshmeat.net was still a thing. For some reason compiling C code from some website (`wget;./configure;make;make install`) seems to be more secure than `go install`, maybe because nobody understands automake anymore.
- zeveb 7y ago
  > They could only do it for a handful of servers, change request took months and the systems were not to be 'touched'. Those old ways of doing things were indeed slower than they needed to be in many cases, and I'm glad that we sped them up. I don't believe that we needed to throw the baby away with the bathwater though. The problem now is that we build systems which are unmanageable and unmanaged. 'Throw it in a container and let it run' is not scalable either: it scales neither security not maintainability. Instead of scaling, it just punts. Enabling one to do something one shouldn't do (i.e., deploy insecure software into production) is not a virtue.
  - CathedralBorrow 7y ago
    > The problem now is that we build systems which are unmanageable and unmanaged Are all the systems out there in production actually unmanageable and unmanaged?
  - cuillevel3 7y ago
    After a few decades it's no longer a baby ;) Software is becoming more complex and moving faster than ever. 'Controlling your dependencies' was always an illusion and a trade-off at best. Often a trade-off against security, inheriting the folder of JAR files from your predecessor and such. I think of containers and all these new tools as designs which are supposed to help us manage existing complexity. They don't create it, it's already there in the requirements and real-world deployments. The friction we feel, is that some of those tools are not very good (yet), but maybe better than the previous generation (sometimes). On the negative side, there is a lot of nostalgia ("ah, do you remember bare metal...") and unwillingness to change and learn.
llcoolv 7y ago
OK Boomer :D
z3t4 7y ago
So you dont want to run (insert software here, like your IDE) in the cloud, and you dont want to run it yourself because you dont trust it (its open source). Then how do you want it delivered? Snaps? Docker? KVM image?
- disintegore 7y ago
  In an InstallShield package, with a 7-step license activation process. Let's face it, security is an asymptote. I just want someone I can call and scream at when we eventually get pwned and I'm ready to pay for that privilege.
bitwize 7y ago
<OK-Boomer>Those of us smart enough to set up ham radio network links were likely also smart enough to not use AOL.</OK-Boomer>
znpy 7y ago
https://xkcd.com/1988/ https://xkcd.com/1988/
turk73 7y ago
Before containers, I had sysadmins who ran our AIX cluster. Three nodes, not a single one configured like the other. There was zero consistency where it counted and these jokers would go on a terminal and manually configure Websphere and other components and screw it up constantly. We even tried to give them a terminal tool where you could type in one window and it would mirror on other boxes, but they refused to use it. Puppet, Chef, Ansible, all led the way to something better. Docker and Kubernetes feel so far advanced from what we had in 2012.
PaulHoule 7y ago
Uh... I've built Hadoop from source, it is not that bad.
- icedchai 7y ago
  I've also built it from source, admittedly back in 2012. It wasn't that bad, either! My concerns were building an AMI image, not "packaging" it in a more traditional sense.
  - PaulHoule 7y ago
    I think most of the conventional image building tools are stupid. (e.g. they take something that could be done in 10 minutes and make it take an hour, are prone to failure because of too many round trips, ...) I made rather complex images that included a large database (e.g. 30 GB) and Java apps, and found the easy way to do it was write a program that analyzes the problem, writes a shell script that builds a bastion and builds the software, then put that in the cloud-init... The machine "phones home" via SQS to tell the controlling server to shut it down and image it.
duelingjello 7y ago
Yes, it's worse than that on a couple of fronts. Cloudera, Amazon and others benefit financially from situations of over-complicated messiness, while average SWEs, SAs and others are left with wasted time, money and lack of trust. And worse, lax engineering and operations practices introduce attack surfaces to fraudsters, thieves, organized crime and state-level actors to backdoor, destroy, steal, extort and monetize infrastructure of organizations large and small.
kube-system 7y ago
Using stuff blindly from Docker Hub is a bit sketch, but it’s not a hard problem to fix. Read the dockerfile before using it, adjust if you don’t like it, and build and push to your own repo. Then run your vulnerability scans of choice on them nightly.
rb808 7y ago
I agree the last decade has been really tough for sysadmins, but the payoff is developers have had a great time and I dont have to rely on those guys any more.
- peterwwillis 7y ago
  This is a weird mentality that I think comes from being disconnected from ops. It's like you're a mechanical engineer building a truck, and now that you have a 3D Printer that can make any part, you think you don't have to rely on or communicate with the people driving the trucks on the road.
  - dividedbyzero 7y ago
    As long as what you're doing isn't too involved, I guess running without ops is completely realistic nowadays. If all you need is a database and a relatively simple monolithic app plus stuff like logging, metrics, and if a little downtime here and there isn't the end of the world, there's lots and lots of managed/cloud offerings that'll cut down ops to a point where it's very much manageable for capable devs. A lot of small shops run like this and do so successfully.
    - peterwwillis 7y ago
      If a little downtime here and there doesn't matter, then either customers don't care about the product, or the company doesn't care about the customer. Ops's job is to keep customers happy by keeping the applications from going down; it's not like they think a dev can't set up a service in AWS themselves. But you're right, you don't need a dedicated ops person if a dev has the time and knowledge to maintain a service reliably.
- opsthrowaway0 7y ago
  Flash bulletin from the engine room: you're still relying on "these guys" (and gals), you're just cost-sharing our salaries across whatever pile-o-garbage SaaS you convinced your architect to engage, including my own large cloud provider. Hint: it'd be more effective (and cheaper) to employ us directly, but you do you. That said, at the end of the day, I'm still sitting in my Aeron and your "great time" just means concentrating an entire discipline of professionals into cloud companies and ISPs. In this dream scenario (for who?), you get to sit around waiting for ticket updates while your revenue craters instead of picking up the phone or using that hole on your face to communicate with a peer. I get to focus on systems at scale and on monitoring and metrics instead of an annoying developer who escalates to their SVP and makes an inter-org ruckus when their TLS certificate expires and they can't find the error in their voluminous Javascript "logs". To be clear: I'm having a great time. There's a suspicious TAM in between me and you now, and you don't have my extension. I prayed for this day across many developer escalations and still can't believe it's here. Seriously, people who are likely replaceable with Microsoft Access and a few Thoughtworks clones shouldn't throw stones at the professions completely supporting their Blue Bottle habit. Oh yeah, the Internet keeps on quietly working, too. Tip jar is to your left.
ascotan 7y ago
This post reminded me of an old article: https://blog.codinghorror.com/vampires-programmers-versus-werewolves-sysadmins/ https://blog.codinghorror.com/vampires-programmers-versus-we... The big win for 'containers' is that they separate the concerns of sysadmins from developers. The days of the sysadmin administering the tomcat and apache installs are shortly coming to an end. For my part, i think this is a good thing because devs and sysadmins have different concerns that don't overlap all the time.
disordinary 7y ago
I don't agree. Systems are more complicated to build with a lot of dependencies that's true, but this is mitigated by shipping pre-built binaries and containers which have the dependencies bundled. Yes, those binaries can be from untrusted sources but if you use those you're taking a risk - the same as if you use code from an untrusted source (just because it's not compiled doesn't make it safe, when was the last time your organisation audited anything), ultimately a business should be using banaries from businesses they trust. Containers actually increase security in the following ways: 1. They run isolated from the host system and if you set things up properly they should be non root. 2. They're immutable and cannot be tampered with. 3. Because they work on layers you can change them without affecting the underlying container, to upgrade you can just replace the layer that you inherit from, simplifying the upgrade path (and being immutable, stateless, and versioned there's no risk to upgrading, you can very easily rollback). 4. In the example listed above like Haddoop you can install the outdated version of Java that it depends on without compromising the host or other workloads on that server. Of course, this all breaks down if you don't use containers from trusted sources and if you don't have processes in place. It works in a modern environment with pipelines and automation, it doesn't work as well in a more traditional environment with sysadmins hand wrangling deployments. Of course the most secure way to do things is to have deterministic builds of a codebase which creates a hash which you can compare with a canonical source, but this is tricky and not all build toolchains support this properly so you'll end up with mixed results. For a streamlined business the best compromise is to get applications from trusted vendors and rely on static analysis and the ability to quickly upgrade if a vulnerability is discovered. Ultimately, the one of the best ways to secure a system is to establish an unbroken, audited, and automated trust chain and remove the ability for human intervention.
Russtopia 7y ago
Amen. I avoid any container-based or blindly 'curl |sudo' install (the latter only after manual inspection of such a script). Apps which only offer containers are untrustworthy black boxes. And it indicates devs who are too ignorant and/or lazy to make even basic efforts at distribution-neutral or portable code. Gentoo and Funtoo keep devs honest to some extent, as packages must be built, not just slurped down in whatever alien form the project decided to use.
- james-mcelwain 7y ago
  I'm not sure about other container solutions, but Docker isn't a black box at all. It's fully introspectable -- I often dump a container's file system to see what's actually included.
IOT_Apprentice 7y ago
So nearly 5 years later, is any of this any better? Also, why haven't Linux vendors standardized on a single package manager that is the default for ALL Linux distros? This seems like the equivalent hell that npm gives to NodeJS.
- mappu 7y ago
  > why haven't Linux vendors standardized on a single package manager that is the default for ALL Linux distros? The two distro package managers that "matter" (apt and rpm) are similar enough that this isn't a problem in practice. You can have fpm spit out a deb or rpm or arch pkg with just one flag change, and host it on your own website. The real differences aren't just mechanical packaging differences, but more a question of philosophies, there are a lot of differences in upstream distro packaging policies themselves. Debian isn't going to let you upload it directly unless it is a good citizen on their distro, stays up-to-date with their versions of all dependencies (so that you only end up with one python, libjpeg etc on your system) and is certifiably free software down to the bones. Whereas Arch will take literally anything in the AUR, and the fast way into CentOS is to be employed by RH (paraphrasing). (I tried to package a certain app in RPM, but it turns out that OpenSuSE and CentOS have different names for the tzdata package, and until very recently rpm didn't even have a system for specifying alternate dependencies.)
cosmodisk 7y ago
"Consider for example Hadoop. Nobody seems to know how to build Hadoop from scratch. It’s an incredible mess of dependencies, version requirements and build tools." And that's exactly what's wrong with a half of the software industry. Some people even take pride in thinking that having dependency hell is a good thing. A lot of build tools look like they are created by aliens to aliens with little thought of how the end user would use it. And if someone dares to say how crappy the whole thing is, they get dismissed for not being smart enough or simply lack dedication to spend unlimited hours trying to build some 500 line code.

428 points by maple3142 7y ago

_def 7y ago

I don't know that much about the state of container systems these days, but to me it seems it's the "comfortable" way in the "security vs comfort" tradeoff. Use where applicable and hope for better days.

ThreeFx 7y ago
But there are some environments where security is (almost) the only relevant factor, e.g. banking IT-infrastructure. I don't want my bank to pull arbitrary images from Dockerhub because that increases their comfort. Conversely, if I were sysadmin at a bank I would most definitely be concerned what was running in my network.
- buzzkillington 7y ago
  >Conversely, if I were sysadmin at a bank I would most definitely be concerned what was running in my network. I've worked at a bank. It's a docker file. Running Hadoop. On top of a Linux Container. Running in Rhel 5. On top of a vmware hypervisor.
- ownagefool 7y ago
  In 2015 I was running docker containers for a Gov organisation, where: - docker containers were built from source - Dockerfile published with the code - Built in a new CI environment - Pushed, Pulled and deployed from the sha - Collecting network traffic, undertaking protective monitoring, that looks for those backdoors. Just because you can pull arbitrary bullshit doesn't mean you have to. Though for the record, the same sysadmins that whine about newer tools are generally the same ones that implicitly trust their older toolsets. Just because you can compile it, doesn't make it secure, so you need to be running monitoring solutions and hedging your bets no matter the tech.
arberavdullahu 7y ago
I don't think you should put security at any case in tradeoff, by your argument one can save passwords in plaintext because its easy and comfortable, but its just not acceptable and this would effect not only your app but every app that your users use.
- ThreeFx 7y ago
  I see the value when running e.g. home instrastructure on some old laptops or other non-critical stuff. Most people have NAT at home and thus whatever services they pull and listen in their local network aren't exposed to the outside world, and they can enjoy mopidy/NextCloud/whatever without going through a lot of hassle setting it up. Of course once we come to enterprise environments the opaque build/deployment process is atrocious.
  - BlueTemplar 7y ago
    30% of the Internet already uses IPv6. NAT is obsolete and was never supposed to be used for security anyway...
- contravariant 7y ago
  If you don't make a tradeoff you end up with passwords to long to remember and security to cumbersome to not circumvent. Security has a cost, as does the lack of it, hence the trade-off.
- x3n0ph3n3 7y ago
  The trade-off is always between security and accessibility. If you want a completely secure computer, bury it at the bottom of a mineshift and pour a few yards of concrete over it. No one is breaking into that anytime soon!
- geocar 7y ago
  This is not "the tradeoff", but too easily a straw man because bcrypt (et al) cost so very little as to effectively cost nothing at all. Think about your server consuming resources: It needs a [machine] password stored someplace in order to authenticate itself to those resources. If it stores this encrypted, then an operator must be present to decrypt it when (re)starting the server. If this is stored unencrypted, then it is not and the server can be started unattended (or autoscaled or whatever). That is a security tradeoff because there are both real benefits and risks gained and lost with each approach. If docker was ever a tradeoff, it was between taking the time to have a thoughtful architecture versus trying to get acquired, and I think we know which way they went.

majewsky 7y ago

Discussed at the time (443 comments): https://news.ycombinator.com/item?id=9419188 https://news.ycombinator.com/item?id=9419188 And again in 2018 (426 comments): https://news.ycombinator.com/item?id=17083436 https://news.ycombinator.com/item?id=17083436

mike_hock 7y ago
But unfortunately, it's relevant to this day.
- majewsky 7y ago
  More than ever.

upstandingdude 7y ago

ha +1 for the build tool mess. its like everyone goes "duuude this stuff is so complex lets build something on top to abstract it all away" and you end up with another level of complex stuff that builds the stuff that builds the that wraps the stuff that you originally wanted to use...

pram 7y ago

Terrible article IMO. Hadoop is an awful mess, but it has nothing to do with Docker, which is simplistic in comparison. "Ever tried to security update a container?" Yes, I have! In fact you can maintain patch compliance in a container pretty much the same way you'd maintain a VM or bare metal Linux installation!

black_puppydog 7y ago
> Hadoop is an awful mess, but it has nothing to do with Docker, which is simplistic in comparison. The fact that they can get away with a build system like this is very much due to docker (and curl | sudo bash) allow people to not feel the pain this mess causes, at least not right away. > Yes, I have! In fact you can maintain patch compliance in a container pretty much the same way you'd maintain a VM or bare metal Linux installation! But then you lose the declarative/immutable nature of docker, no? Except if you mean every time there's a security update, you rebuild your containers?
- onion2k 7y ago
  Except if you mean every time there's a security update, you rebuild your containers? That's exactly what you should be doing, at least if you're relatively small scale. Google probably takes a different approach but most of us aren't working at that scale.
- pram 7y ago
  First: No one installs Hadoop from scratch, and Hadoop isn't built with Docker. Companies use Ambari or other distros like EMR. Second: Yes, generally you can use a build system like Jenkins, and a registry like Artifactory to automate the process. The docker image is updated, and then you can push it out with your orchestration in whatever method you choose. It's not an obscure or difficult thing to manage..
- adrianN 7y ago
  It's pretty hard to do truly declarative/immutable things with docker because the typical dockerfile starts with "apt-get update && apt-get install ...". For that use case I think nix is much better.

Yuval_Halevi 7y ago

I saw this comment now from 4 years ago on reddit: Software packaging and building seems to be becoming more complicated and more disconnected-- particularly as more specific tools continue to be developed. It seems every little corner has their own dependency management and build management solution. The Docker comments seem a little backhanded. If one looks at a Dockerfile, there's not much to complain about. It downloads a key, it adds a repository, installs a package, and some very simple docker-specific tweaks. If you distrust Docker's signed image, building your own is as simple as doing a git clone ..; cd docker-nginx; docker build. Docker encourages disposable containers, separating data, and making your own images. There's still a lot of work for Docker to do regarding signed images, but I'd argue running isolated images with documented changesets and simple build files is far different from blindly running 'curl | sudo bash' https://old.reddit.com/r/programming/comments/33ktc9/the_sad_state_of_sysadmin_in_the_age_of_containers/cqlxne5/ https://old.reddit.com/r/programming/comments/33ktc9/the_sad...

jve 7y ago

This all boils down to trust. `curl | sudo bash` is no different than .\install.exe. The question is about trusting the SOURCE and trusting the DISTRIBUTION channel (that HTTP download from scala-lang.org violates this). Where did you get it? from https://microsoft.com/.. https://microsoft.com/... or from https://micro.soft.com/... https://micro.soft.com/...? Whom you trust more? The same with pre-built VM image or whatever... do you trust the party that made this image/container available?

LockAndLol 7y ago
`curl | sudo bash` should never have become a thing in the first place. To me, it's the most "WTF?" mode of installing anything. Especially when the source is HTTP. In what world is that secure? Pity Qubes is so heavy, otherwise I'd be using it
- eejjjj82 7y ago
  would you feel better if your Makefile looks like this: install: [ $(id -u) -ne 0 ] && echo 'please run sudo make install' || curl https//my.thing | /bin/sh it goes back to what jve said - it's a matter of trust. how often do you blindly run 'sudo make install' w/o reading the entirety of the build script? all the time I bet - it's because you trust the source
  - LockAndLol 7y ago
    No. I think I have never run `sudo make install`. Things I install come from package managers. Docker or a VM is used to test other software. And of course trust is important, but if I encounter things like `curl | bash`, my trust is lost.
    - yori 7y ago
      What difference does it make if you run `make install` or `curl | bash`? In both cases, you are running code you have not audited yourself. Or are you the kind who inspects every Makefile and installer script before installing? If not, why is one better than the other?

onion2k 7y ago

Ever tried to security update a container? The whole point of using a container is that you can destroy it and build a new one easily. The new one should be built using up-to-date packages with security patches applied (and tested, obvs). Using the 'pets versus cattle'[1] analogy, patching a container feels like you're treating it like a pet. You should just kill it and get a new one instead. [1] https://thenewstack.io/how-to-treat-your-kubernetes-clusters-like-cattle-not-pets/ https://thenewstack.io/how-to-treat-your-kubernetes-clusters...

dsr_ 7y ago
The whole point of having a distro with a good reputation is that you can incrementally outsource your trust to them, one package at a time. OpenSSL needs a fix? One package update per machine, and if you have lots of machines, you update that. With VMs, you can use the same tools you use with any other machine. With containers, you are responsible for figuring out which ones need which packages and how to re-build them. Your method is likely to be idiosyncratic, so you not only need to be your own security team, you need to be your own packaging maintainer and infrastructure maintainer. The two extreme versions of this are the neglect model, in which you just don't care about anything, and the bleeding-edge model, where you automatically build with the latest versions of everything pulled from their origins. Both ways end up with unexpected security disasters.
- regularfry 7y ago
  The "one package at a time" model assumes you care about per-instance uptime and bandwidth costs. The "reset the world" model assumes rebuilds and reboots are cheap enough not to have to. I don't think either is necessarily wrong; there's not a huge distance between `apt-get install unattended-upgrades` and a daily (weekly, whatever) container rebuild except that you need to cron the latter. Unless you're using a different OS between the two cases, they can use the same package database, the same tooling, and so on.
  - dsr_ 7y ago
    You need to make the decisions, keep track, and do the work. If you've got the tools made for you, excellent. Otherwise you also have to build and maintain the tools -- which is my point.

jwr 7y ago

I recently had a similar discussion with people using npm for building a CSS framework library. I tried to explain the concept of getting a pre-downloaded tarball and using "make" (or similar) to produce target artifacts from source files in a deterministic, repeatable and reliable manner, without relying on any third-party servers being available and without pulling in dependencies that might have changed. It seems the concept was entirely alien to programmers younger than me.

mateuszf 7y ago
It can be done using package managers too, for example using Nix. The problem here is not related to distribution method but rather dependency hell in javascriptland.
- weberc2 7y ago
  I would say the main issue is that Nix is super difficult and requires you to intimately know your entire dependency tree down into Linux particularities. JavaScript makes matters worse, but even packaging a nontrivial Python package pulled directly from Pypi is often difficult. I want to like Nix, but it is far too pedantic to be practical. And this isn’t even mentioning the usability issues.
  - glglwty 7y ago
    I think you only need to specify your immediate dependencies in nix?
    - weberc2 7y ago
      Someone has to write the package definitions for those dependencies, and the public package repository doesn’t have broad coverage for many popular packages in many languages. This is understandable in that this is a massive effort, but that’s also the point—the effort to manage packages is huge and the extra effort compared to other tools is not a good bargain for many projects, nor is it much of consolation for project maintainers who would otherwise like to use it.

1337shadow 7y ago

The holy grail of sysadmin has always been deployment of immutable images, instead of host system pollution, source code mutation during deployment ... It was doable with things like Packer, KVM or OpenStack, now it's become easier with containers. You can still orchestrate your stuff with ansible or bash scripts. I most certainly remember how I was doing isolated deployments 10 years ago: https://github.com/jpic/bashworks/tree/master/vps https://github.com/jpic/bashworks/tree/master/vps HINT: it's much cleaner and much more automated with containers: with GitLab-CI it's trivial to host a private container bulding infrastructure with a registry and other automation ...

media-share-iq 7y ago

We providing A Feature-Rich Enterprise Video Platform That's Also Affordable and Easy-to-Use services like 1] Enterprise Video Platform 2] Video Management Platform, MEDIASHAREiQ E-mail : ems.marketing19@gmail.com Website : https://www.mediashareiq.com/ https://www.mediashareiq.com/ Mobile : +1-877-367-5050(USA)

buzzkillington 7y ago

Looked at the Debian wiki the OP linked to: >Debian currently does not include Hadoop packages. There are a number of reasons for this; in particular the Hadoop build process will load various dependencies via Maven instead of using distribution-supplied packages. Java projects like this are unfortunately not easy to package because of interdependencies; and unfortunately the Hadoop stack is full of odd dependencies (including Apache Forrest, which for a long time only worked with Java = 1.5). >If you want to build Debian packages, the most complete efforts can be found at the Apache Bigtop project http://bigtop.apache.org/ http://bigtop.apache.org/ . Unfortunately, the build process for these packages is currently of a disastrous quality, and should only be attempted within disposable virtual machines, as it requires root permissions and will install non-packaged software. >These will allow building Hadoop packages on a recent Debian system. However, the resulting packages do not live up to Debian quality standards. In particular, they include copies of .jar files from other packages, and will thus not benefit from security updates done to these packages. >If you are interested in getting Hadoop packages into Debian, Coordinate with the Java packaging team Coordinate with upstream Apache Bigtop >to avoid duplicate efforts. Thank you. https://wiki.debian.org/Hadoop https://wiki.debian.org/Hadoop Jesus Christ.

alfozan 7y ago
Who still uses Hadoop anyway? https://spark.apache.org/ https://spark.apache.org/ https://www.iguazio.com/data-science-post-hadoop/ https://www.iguazio.com/data-science-post-hadoop/
- buzzkillington 7y ago
  Spark is worse because you need Scala as well as regular Java. I've tried building it for my day job, I would rather have a colonoscopy without sedation. It's more pleasant and dignified.
  - itg 7y ago
    And then add PySpark on top of that. Couldn't leave my last job fast enough when they decided to use Hadoop/PySpark when the largest incoming files we received were at most a few GBs.

_pmf_ 7y ago

Give me rsync and a sysadmin any day over a circle jerking dev ops team.

Annatar 7y ago
Replace rsync with OS packages and you got yourself a deal and a half.
ex_amazon_sde 7y ago
Amazon and some other FAANGs build and deploy OS packages, and by choice. Sad to see how anything that is not container gets downvoted here.

bureaucrat 7y ago

Ok boomer.

Hnrobert42 7y ago
Ad hominem attacks are unproductive and specifically disallowed on HN.

_wldu 7y ago

Standard Makefiles are really under appreciated today. They are simple and can be used with most any language, version controlled and have been around for decades.

neuronic 7y ago
But it's not a product you can sell or artificially inflate its monetary value so out the window it goes.
adgasf 7y ago
Makefiles are too low level to be convenient for anything other than simple projects on one platform.
- gerbilly 7y ago
  If the alternative you are hinting at is CMake, I can't really agree with you. I tried it recently and was shocked at how awful the syntax and usability was. It's begging for a tool to auto-detect your dependencies from source. I was trying to port an existing project to it and I gave up when I realised that I'd have to manually create about 30 CMakeList.txt files or whatever they call them. Meanwhile during the same weekend, autoconf allowed me to build gcc from source with three commands. I know I know, I'm not supposed to like autoconf, and the orthodoxy is that it is garbage ... It seems every generation ignores the previous generation's tools and invents worse ones to replace them. If this continues we'll soon go back to banging rocks and sticks together.
  - adgasf 7y ago
    I am definitely not hinting at CMake! Better options are Bazel, Buck and Please. Please is the most light-weight option.
  - jeltz 7y ago
    Take a look at Ninja and Bazel. I ahree with you that CMake is worse than autoconf but I would argue that the tools which came after CMake are better than CMake and autoconf. So while there certainly are regressions not every new generation is worse than the previous.

t0astbread 7y ago

I'm looking at this from a relatively neutral newbie POV. I've tipped my toes in both administering machines the "classic" way and with tools like Docker and Docker Compose. To me both have their benefits and drawbacks but Docker looks a bit better overall. Here's why: 1) The idea of immutable systems rocks. It feels like the logical next step from containers. 2) This is maybe just because of me but I have the feeling services that can easily be (automatically) updated (like with Docker) are gonna be more up-to-date on average than when I have to build every new version myself. (Although that could be automated probably.)

dig1 7y ago

The same applies today for nodejs and go, but the same was happening with python, ruby and perl libraries in the past. However, this only shows that standard distro packaging approach is broken or was broken all the time. First, if you want to distribute you application (or library) via distro packages, you'd either have to build them and publish on your site or have to go through long mailing process to satisfy distro specific packaging guidelines and hope that your package will end up in standard (or extended) distribution. Debian is also notorious for "reviewing" and modifying source code, without contacting authors (remember Debian SSH fiasco). I had, not once, unpleasant experience with this process. Today, things are much easier - pack your whatever and put on dockerhub, maven or github. No middleman and no headaches just to reach your users. Sure, there are drawbacks (like mentioned in article), but we'd have to leverage things like GPG and proper package signing. Now, that is totally different story, because cryptography, even the basic one, is PITA for most developers and users. You can't have both :D

lmm 7y ago

You've got to look at this in a context where platform package managers like apt are simultaneously 1) platform-specific 2) jealous, insisting that every language has to conform to their way of doing things and 3) fundamentally not very good, having very limited ability to do things like install packages for a single user or install multiple versions of the same package. Platform package managers like maven have been far more successful because they provide a much better user experience, and frankly I suspect this is if anything because they're so thoroughly isolated from the platform package manager.

marmaduke 7y ago
No I don't. I use Debian derivatives because I can apt get all my stuff without thinking hard, because those package maintainers have done the hard work.
- lmm 7y ago
  Well, clearly not, since you can't install Hadoop. Before you blame that on Hadoop, remember that it only requires some really quite basic things from its package manager, which apt is nevertheless completely unable to do: use libraries in the normal recommended way, play nice with Java, work cross-platform.
  - rumanator 7y ago
    > Well, clearly not, since you can't install Hadoop. You are complaining that you can't automatically install broken packages out of the box through the official repo. And packaging an application is the responsibility of the people working on that application, not the OS. What's your point?
    - int_19h 7y ago
      For Linux distros, the primary responsibility for packaging is actually with the distro maintainers, since packaging is distro-specific. That said, a well-written app should not present any difficulty for the package maintainer.

Havoc 7y ago

And it seems to be getting worse lately with everyone talking about their cloud-y build "pipeline". Which invariably seems to involve stringing together a bunch of different service offerings from startups that might not be here tomorrow. It's hard to imagine something more fragile

de_watcher 7y ago

http://kmkeen.com/maintainers-matter/ http://kmkeen.com/maintainers-matter/

AndyMcConachie 7y ago

Things will break and I will need to be able to debug them. New versions will be released and I will need to be able to upgrade to them. Bugs will be found and I may need to know if I or my users are experiencing them. Security holes will be found and I will need to either upgrade, or mitigate them in some other fashion. All of the above realities mean I want well defined and understandable software. The less well defined and understandable a deployment is, the less I am interested in deploying it. Complexity is a cost, and this cost cannot be waived by simply chucking everying into a container. Simplicity means cheap. It means time not wasted and it means fewer headaches.

rbc 7y ago

The only part of this article that I really disagree with is the title. The main issue here is release engineering, or rather, the lack of it. Software development and system administration have well developed traditions, but release engineering doesn't seem to get much attention in comparison. Given that supply chain attacks are now all the rage in the black hat community, maybe release engineering will start to get the attention that it deserves.

jillesvangurp 7y ago

The maven/gradle ecosystem actually does use signatures. That's one of the things they got right. Maven central has no unsigned artifacts. Also, the signatures are actually checked on download. Other maven servers may be a bit more sloppy of course. In any case, if you are running that on a production system, you are doing it wrong. You run that as part of CI. The output of CI is containers which you start on a production machine. The point about containers is that they should be immutable. Running chef/puppet in kubernetes is not a thing (I hope, probably somebody went there). Updating a container means replacing it with a new one, that hopefully you or somebody else built in a responsible way. That last part is indeed a problem that we have moved instead of solved. 20 year ago, people were just sticking whatever they downloaded on hardware they bought at the store and banged on it until it worked (very literally sometimes). So, I guess this is progress but indeed hardly ideal. I remember using puppet. Can't say that that is with any level of fondness. I vastly prefer Dockerfile and having CI systems produce containers from those. Installing things on a filesystem is no longer that common at deploy time unless that filesystem is that of the container you are building using a Dockerfile or you are self hosting kubernetes (aka. reinventing a lot of wheels at great cost). Puppet/chef etc. are still of use if you are doing that but otherwise it has a limited role in IAAS type architectures. The closest thing is perhaps pre-baking AMI images using packer and some tool like ansible, which is nice if you want to avoid having a lot of startup overhead. Hadoop is complicated, which is why companies exist that host that for you or will help you hosting it on premise in a responsible way. If you want to DYI, you indeed have to do a lot of things and do them properly. Kubernetes has moved that space forward in recent years; so it's easier but this is not for everyone. If on the other hand you are messing with puppet to get this stuff done, maybe reflect on the wisdom of not standing on the shoulders of giants rather than blaming the internet for your self inflicted pain.

axegon 7y ago

Talking from a developer's point of view, I've thought about this for a while and I think it has something to do with what I refer to as "anti-imposter syndrome" - the notion that anyone can make software. You see, over the past 10 years or so, self-proclaimed "educational" services/websites/institutions have been shoving down everyone's throat the idea that anyone will be able to create the next FAANG from scratch after just 3 months of training. Which is the same as claiming that I can become an F1 driver in 3 months. Where do I sign up?!?!?! The thing is, F1 results are a lot more visible since you have a point of reference - the top dogs. Chances of scoring a fastest lap or coming even close to them - little to none. Software - not so much - you have a shiny interface and what it does underneath is always a mystery to the end user. And people who have gone through those magical 3 months of training are often lead to believe that the way they are doing things is what everyone is doing and that is how it should be done. Often people who have signed up for those courses are people who have just a smudge above average technical knowledge - they have no idea how OS'es work, what should be considered safe or even why. Don't hate me for saying it, but essentially Windows users. And this is the software they end up building and distributing. 10 years down the line, thousands have picked up random scraps of knowledge from here and there and tried to mash something together. Don't get me wrong, I think technologies like Docker are astonishing and an incredible tools in the hands of people with knowledge and experience. However way too many people have picked up some scraps from there and created the unholy mess the author is talking about.

cuillevel3 7y ago

> I’m not complaining about old-school sysadmins. They know how to keep systems running, manage update and upgrade paths. Sadly these ways of olde didn't scale. They could only do it for a handful of servers, change request took months and the systems were not to be 'touched'. > And since nobody is still able to compile things from scratch, everybody just downloads precompiled binaries from random websites. Often without any authentication or signature. Right, that would have never happened back then, when packages were not signed and freshmeat.net was still a thing. For some reason compiling C code from some website (`wget;./configure;make;make install`) seems to be more secure than `go install`, maybe because nobody understands automake anymore.

zeveb 7y ago
> They could only do it for a handful of servers, change request took months and the systems were not to be 'touched'. Those old ways of doing things were indeed slower than they needed to be in many cases, and I'm glad that we sped them up. I don't believe that we needed to throw the baby away with the bathwater though. The problem now is that we build systems which are unmanageable and unmanaged. 'Throw it in a container and let it run' is not scalable either: it scales neither security not maintainability. Instead of scaling, it just punts. Enabling one to do something one shouldn't do (i.e., deploy insecure software into production) is not a virtue.
- CathedralBorrow 7y ago
  > The problem now is that we build systems which are unmanageable and unmanaged Are all the systems out there in production actually unmanageable and unmanaged?
- cuillevel3 7y ago
  After a few decades it's no longer a baby ;) Software is becoming more complex and moving faster than ever. 'Controlling your dependencies' was always an illusion and a trade-off at best. Often a trade-off against security, inheriting the folder of JAR files from your predecessor and such. I think of containers and all these new tools as designs which are supposed to help us manage existing complexity. They don't create it, it's already there in the requirements and real-world deployments. The friction we feel, is that some of those tools are not very good (yet), but maybe better than the previous generation (sometimes). On the negative side, there is a lot of nostalgia ("ah, do you remember bare metal...") and unwillingness to change and learn.

llcoolv 7y ago

OK Boomer :D

z3t4 7y ago

So you dont want to run (insert software here, like your IDE) in the cloud, and you dont want to run it yourself because you dont trust it (its open source). Then how do you want it delivered? Snaps? Docker? KVM image?

disintegore 7y ago
In an InstallShield package, with a 7-step license activation process. Let's face it, security is an asymptote. I just want someone I can call and scream at when we eventually get pwned and I'm ready to pay for that privilege.

bitwize 7y ago

<OK-Boomer>Those of us smart enough to set up ham radio network links were likely also smart enough to not use AOL.</OK-Boomer>

znpy 7y ago

https://xkcd.com/1988/ https://xkcd.com/1988/

turk73 7y ago

Before containers, I had sysadmins who ran our AIX cluster. Three nodes, not a single one configured like the other. There was zero consistency where it counted and these jokers would go on a terminal and manually configure Websphere and other components and screw it up constantly. We even tried to give them a terminal tool where you could type in one window and it would mirror on other boxes, but they refused to use it. Puppet, Chef, Ansible, all led the way to something better. Docker and Kubernetes feel so far advanced from what we had in 2012.

PaulHoule 7y ago

Uh... I've built Hadoop from source, it is not that bad.

icedchai 7y ago
I've also built it from source, admittedly back in 2012. It wasn't that bad, either! My concerns were building an AMI image, not "packaging" it in a more traditional sense.
- PaulHoule 7y ago
  I think most of the conventional image building tools are stupid. (e.g. they take something that could be done in 10 minutes and make it take an hour, are prone to failure because of too many round trips, ...) I made rather complex images that included a large database (e.g. 30 GB) and Java apps, and found the easy way to do it was write a program that analyzes the problem, writes a shell script that builds a bastion and builds the software, then put that in the cloud-init... The machine "phones home" via SQS to tell the controlling server to shut it down and image it.

duelingjello 7y ago

Yes, it's worse than that on a couple of fronts. Cloudera, Amazon and others benefit financially from situations of over-complicated messiness, while average SWEs, SAs and others are left with wasted time, money and lack of trust. And worse, lax engineering and operations practices introduce attack surfaces to fraudsters, thieves, organized crime and state-level actors to backdoor, destroy, steal, extort and monetize infrastructure of organizations large and small.

kube-system 7y ago

Using stuff blindly from Docker Hub is a bit sketch, but it’s not a hard problem to fix. Read the dockerfile before using it, adjust if you don’t like it, and build and push to your own repo. Then run your vulnerability scans of choice on them nightly.

rb808 7y ago

I agree the last decade has been really tough for sysadmins, but the payoff is developers have had a great time and I dont have to rely on those guys any more.

peterwwillis 7y ago
This is a weird mentality that I think comes from being disconnected from ops. It's like you're a mechanical engineer building a truck, and now that you have a 3D Printer that can make any part, you think you don't have to rely on or communicate with the people driving the trucks on the road.
- dividedbyzero 7y ago
  As long as what you're doing isn't too involved, I guess running without ops is completely realistic nowadays. If all you need is a database and a relatively simple monolithic app plus stuff like logging, metrics, and if a little downtime here and there isn't the end of the world, there's lots and lots of managed/cloud offerings that'll cut down ops to a point where it's very much manageable for capable devs. A lot of small shops run like this and do so successfully.
  - peterwwillis 7y ago
    If a little downtime here and there doesn't matter, then either customers don't care about the product, or the company doesn't care about the customer. Ops's job is to keep customers happy by keeping the applications from going down; it's not like they think a dev can't set up a service in AWS themselves. But you're right, you don't need a dedicated ops person if a dev has the time and knowledge to maintain a service reliably.
opsthrowaway0 7y ago
Flash bulletin from the engine room: you're still relying on "these guys" (and gals), you're just cost-sharing our salaries across whatever pile-o-garbage SaaS you convinced your architect to engage, including my own large cloud provider. Hint: it'd be more effective (and cheaper) to employ us directly, but you do you. That said, at the end of the day, I'm still sitting in my Aeron and your "great time" just means concentrating an entire discipline of professionals into cloud companies and ISPs. In this dream scenario (for who?), you get to sit around waiting for ticket updates while your revenue craters instead of picking up the phone or using that hole on your face to communicate with a peer. I get to focus on systems at scale and on monitoring and metrics instead of an annoying developer who escalates to their SVP and makes an inter-org ruckus when their TLS certificate expires and they can't find the error in their voluminous Javascript "logs". To be clear: I'm having a great time. There's a suspicious TAM in between me and you now, and you don't have my extension. I prayed for this day across many developer escalations and still can't believe it's here. Seriously, people who are likely replaceable with Microsoft Access and a few Thoughtworks clones shouldn't throw stones at the professions completely supporting their Blue Bottle habit. Oh yeah, the Internet keeps on quietly working, too. Tip jar is to your left.

ascotan 7y ago

This post reminded me of an old article: https://blog.codinghorror.com/vampires-programmers-versus-werewolves-sysadmins/ https://blog.codinghorror.com/vampires-programmers-versus-we... The big win for 'containers' is that they separate the concerns of sysadmins from developers. The days of the sysadmin administering the tomcat and apache installs are shortly coming to an end. For my part, i think this is a good thing because devs and sysadmins have different concerns that don't overlap all the time.

disordinary 7y ago

I don't agree. Systems are more complicated to build with a lot of dependencies that's true, but this is mitigated by shipping pre-built binaries and containers which have the dependencies bundled. Yes, those binaries can be from untrusted sources but if you use those you're taking a risk - the same as if you use code from an untrusted source (just because it's not compiled doesn't make it safe, when was the last time your organisation audited anything), ultimately a business should be using banaries from businesses they trust. Containers actually increase security in the following ways: 1. They run isolated from the host system and if you set things up properly they should be non root. 2. They're immutable and cannot be tampered with. 3. Because they work on layers you can change them without affecting the underlying container, to upgrade you can just replace the layer that you inherit from, simplifying the upgrade path (and being immutable, stateless, and versioned there's no risk to upgrading, you can very easily rollback). 4. In the example listed above like Haddoop you can install the outdated version of Java that it depends on without compromising the host or other workloads on that server. Of course, this all breaks down if you don't use containers from trusted sources and if you don't have processes in place. It works in a modern environment with pipelines and automation, it doesn't work as well in a more traditional environment with sysadmins hand wrangling deployments. Of course the most secure way to do things is to have deterministic builds of a codebase which creates a hash which you can compare with a canonical source, but this is tricky and not all build toolchains support this properly so you'll end up with mixed results. For a streamlined business the best compromise is to get applications from trusted vendors and rely on static analysis and the ability to quickly upgrade if a vulnerability is discovered. Ultimately, the one of the best ways to secure a system is to establish an unbroken, audited, and automated trust chain and remove the ability for human intervention.

Russtopia 7y ago

Amen. I avoid any container-based or blindly 'curl |sudo' install (the latter only after manual inspection of such a script). Apps which only offer containers are untrustworthy black boxes. And it indicates devs who are too ignorant and/or lazy to make even basic efforts at distribution-neutral or portable code. Gentoo and Funtoo keep devs honest to some extent, as packages must be built, not just slurped down in whatever alien form the project decided to use.

james-mcelwain 7y ago
I'm not sure about other container solutions, but Docker isn't a black box at all. It's fully introspectable -- I often dump a container's file system to see what's actually included.

IOT_Apprentice 7y ago

So nearly 5 years later, is any of this any better? Also, why haven't Linux vendors standardized on a single package manager that is the default for ALL Linux distros? This seems like the equivalent hell that npm gives to NodeJS.

mappu 7y ago
> why haven't Linux vendors standardized on a single package manager that is the default for ALL Linux distros? The two distro package managers that "matter" (apt and rpm) are similar enough that this isn't a problem in practice. You can have fpm spit out a deb or rpm or arch pkg with just one flag change, and host it on your own website. The real differences aren't just mechanical packaging differences, but more a question of philosophies, there are a lot of differences in upstream distro packaging policies themselves. Debian isn't going to let you upload it directly unless it is a good citizen on their distro, stays up-to-date with their versions of all dependencies (so that you only end up with one python, libjpeg etc on your system) and is certifiably free software down to the bones. Whereas Arch will take literally anything in the AUR, and the fast way into CentOS is to be employed by RH (paraphrasing). (I tried to package a certain app in RPM, but it turns out that OpenSuSE and CentOS have different names for the tzdata package, and until very recently rpm didn't even have a system for specifying alternate dependencies.)

cosmodisk 7y ago

"Consider for example Hadoop. Nobody seems to know how to build Hadoop from scratch. It’s an incredible mess of dependencies, version requirements and build tools." And that's exactly what's wrong with a half of the software industry. Some people even take pride in thinking that having dependency hell is a good thing. A lot of build tools look like they are created by aliens to aliens with little thought of how the end user would use it. And if someone dares to say how crappy the whole thing is, they get dismissed for not being smart enough or simply lack dedication to spend unlimited hours trying to build some 500 line code.