6 ms·

Terrible article IMO. Hadoop is an awful mess, but it has nothing to do with Docker, which is simplistic in comparison. "Ever tried to security update a contai

• black_puppydog 7y ago
  > Hadoop is an awful mess, but it has nothing to do with Docker, which is simplistic in comparison. The fact that they can get away with a build system like this is very much due to docker (and curl | sudo bash) allow people to not feel the pain this mess causes, at least not right away. > Yes, I have! In fact you can maintain patch compliance in a container pretty much the same way you'd maintain a VM or bare metal Linux installation! But then you lose the declarative/immutable nature of docker, no? Except if you mean every time there's a security update, you rebuild your containers?
  • onion2k 7y ago
    Except if you mean every time there's a security update, you rebuild your containers? That's exactly what you should be doing, at least if you're relatively small scale. Google probably takes a different approach but most of us aren't working at that scale.
  • pram 7y ago
    First: No one installs Hadoop from scratch, and Hadoop isn't built with Docker. Companies use Ambari or other distros like EMR. Second: Yes, generally you can use a build system like Jenkins, and a registry like Artifactory to automate the process. The docker image is updated, and then you can push it out with your orchestration in whatever method you choose. It's not an obscure or difficult thing to manage..
  • adrianN 7y ago
    It's pretty hard to do truly declarative/immutable things with docker because the typical dockerfile starts with "apt-get update && apt-get install ...". For that use case I think nix is much better.
    • doliveira 7y ago
      Linters complain when you don't pin the version, though
      • adrianN 7y ago
        That's what I tried doing. The result was that the docker container didn't build anymore because at least one of the packages we used had a a patch release every other day and apt only serves the most up to date version.
• buzzkillington 7y ago
  >Unfortunately, the build process for these packages is currently of a disastrous quality, and should only be attempted within disposable virtual machines, as it requires root permissions and will install non-packaged software. https://wiki.debian.org/Hadoop https://wiki.debian.org/Hadoop If Debian, a distro on the lower side of dramatic, calls it disastrous, it's pretty bad.
  • regularfry 7y ago
    Debian is (mostly) low on the drama, but high on the expected package quality. It's not surprising they'd have problems here.
    • quicksilver03 7y ago
      I couldn't disagree more, by looking at how badly Debian dares to diverge on basic packages such as Apache, nginx and exim it's clear to me that they don't care at all about quality.
      • regularfry 7y ago
        Divergence from the upstream isn't a marker for or against quality, as Debian chooses to define it.
• tomtomtom777 7y ago
  The article focuses on Hadoop, but the generalization to containers is spot on. Containers have become the first stop method to hide overly complicated build processes. When you've noticed your wiring has become a complete mess of knots, just put it in a box so no one will trip. It seems to me that containers are useful for deploying final setups, but should be a no go for packaging tools.
  • apexalpha 7y ago
    I would see it more as to dividing your wiring into segments with breakers in between so that one rogue device doesn't bring down the whole network. Seems a good thing, to me at least.
  • indigochill 7y ago
    >When you've noticed your wiring has become a complete mess of knots, just put it in a box so no one will trip. On the other side of the coin, putting stuff in boxes is the essence of architecture. CPU opcodes? Boxes. Compiling readable code into bytecode? Boxes. Objects? Boxes. Functions? Boxes. Sure, it can be abused. There might be spaghetti in those boxes. The spaghetti should be chopped into digestible chunks (smaller boxes!). But putting it in a box makes that an implementation detail. Putting stuff in boxes is still progress.
    • bluGill 7y ago
      All problems in programming can be solved by adding another layer of abstraction (ie a box) - except too many layers of abstraction. Good architecture is not about putting things into boxes, but about figuring out what boxes you need and which you do not. Sometimes a box is good, sometimes a box is bad. It isn't the fault of the box that it does't contain the right thing.
  • hvidgaard 7y ago
    I don't think it's fair to blame containers for that. Just because someone programs a shitty product in Visual Studio, does not mean it's a problem with Visual Studio. What containers does beautifully is dependency containment. I make something and put it inside a container, with the exact dependency it needs and it'll work alongside other containers with a program that might have a different incompatible dependency. What we're seeing is not so much that developers suddenly decides to throw best practice into the wind. They just do not have to put the same amount of effort into fixing the broken process exactly because we have dependency containment. I believe we would see the same mess without containers, and the process change that would fix one would fix the other.
• the_duke 7y ago
  You missed the bigger point. Just look at the popular images on Docker hub. A lot of them involve messy build steps, including downloading binaries or source tarballs without verification. It's often hard to know which dependencies a Docker image has, and therefore hard to track vulnerabilities and redeploy fixed images. A lot of docker containers end up either running for a long time, or get rebuilt and redeployed often, but have pinned versions of base images or dependencies which still leaves vulnerabilities in place. This is compensated a bit by the fact that Docker does provide somewhat decent isolation, and most containers are run in a cloud environment, behind load balancers and with decent security constraints. But there are many pitfalls in the whole process. Containers provide a lot of advantages, and are certainly where things will continue to go. But we need to figure out the tooling and ecosystem story to build, verify and update container deployments securely.
  • Annatar 7y ago
    "But we need to figure out the tooling and ecosystem story to build, verify and update container deployments securely." Already done: Solaris zones. Available in the SmartOS distribution near you. Combine with OS packaging, imgadm and vmadm commands for maximum impact.
    • regularfry 7y ago
      I suspect there's a similar story around LXC available.
      • deleted 7y ago
        [deleted]
    • ti_ranger 7y ago
      I've spend 1 minute skimming some of the docs, but it's unclear whether these tools can do a better job than the docker-like tooling. The vmadm "build an image from scratch with a json file" doesn't help in building Hadoop any more than the oldest versions of docker did using a minimal image and a Dockerfile. How does provide visibility into every dependency inside an image? Especially when using 3rd-party-maintained images that consist of significant hours of work per image to get the image to work. (BTW., I built "live CD images" for use in physical and virtual machines since ~2001, using tools such as mklivecd, livecd-tools etc.).
• gchamonlive 7y ago
  I was confused when I read the article. I kept thinking it was actually a problem with hadoop and not docker, since docker is nothing but an abstraction on top of the Linux kernel. I remember trying to produce a viable alpine based docker image for a flask backend service we had thar produced pdf using latex. Dependencies were really hard to pin down, had to resort to open issues on github, but utimately all packages were on the distribution, so it is not a problem with the tech itself, but the archaic way Java lets people to produce applications
• HorstG 7y ago
  Agreed, up to a point: Ever tried to security-update a vendor-provided container which you should never touch because of the vendor's support and warranty conditions? And even if allowed to touch it, the mess that it usually is: app running as root in the container, stuff chmodded a+rwx -R, weird base distro, build script is just a wget, cp -R or "this binary magically appears from somewhere". With a helping of " only works if the directory structure and environment look like the devs machine". Containers are the ultimate expression of "it compiles, let's ship it" laziness
  • tetha 7y ago
    But that's vendors, not containers. You can say exactly the same about vendors on a VM - applications running as root, bad passwords set, reboots impossible, patching impossible, backups... Or a virtual appliance. Or a third party application you installed and cannot upgrade due to a weird dependency on libc, or java, or a specific windows versions. Vendors like that are always a mess, no matter the technology. Containers can turn into an equal mess, yes, but that's up to the build chains and ecosystem built around the containers.
    • HorstG 7y ago
      Agreed. Most vendors supply an unholy mess of weird setup.exe, shellscript installers, omnibus packets or appliances as well, containers are not alone here. Edit: To clarify, containers and VMs are somewhat special in that it is very easy to do the wrong thing, and that you get a lot of rope to hang yourself with. The application developer by. default takes responsibility for the whole OS stack (maybe except the kernel with containers) but usually doesn't handle it very well. Whereas with packages or even setup.exe, the default case in most IDEs and tools is to package very little, and only upon request (using omnibus e.g.) to include the whole shebang edit2: the definition of 'vendor' of course includes a lot of OSS projects who are just as bad as most commercial vendors in this regard, as exemplified by all the aforementioned Apache Foundation Java Crapware
• fargle 7y ago
  I think the article is right on and makes a good point. The title does imply that containers are somehow bad for system administration. But the point is not that a container with a nicely installed, maintained, sensible image and applications pre-loaded on it would be a problem for system administration. Nor would it be anything but trivial to update patches in this situation. The real point the author is making is that many times containers are misused to paper over problems like lack of reproducibility, dependency hell, etc. They aren't reproducible, certainly not by you. If they were you'd not need the container in the first place, or you could build your own easily. So for this kind of misuse, the analogy to Tucows Windows 95 shareware is pretty close. Hadoop and Docker are just the examples to make the point. And they are good enough ones, because I could understand exactly what the author meant. There are certainly hundreds of other examples.
• Russtopia 7y ago
  I suspect Hadoop was just used as an example. For another, I would suggest Concourse (build/CI tool). When I tried it, it was docker images for the default install, vague instructions, and a dev forum which seemed rather hostile to anyone asking for support to build or configure outside of containers. Might have changed since then, but a big turn off.