7 ms·

As a "major theme", the author takes: > Consider for example Hadoop. Nobody seems to know how to build Hadoop from scratch. It’s an incredible mess of dependen

As a "major theme", the author takes: > Consider for example Hadoop. Nobody seems to know how to build Hadoop from scratch. It’s an incredible mess of dependencies, version requirements and build tools. And as the major introduction to the blog post: > I’m not complaining about old-school sysadmins. They know how to keep systems running, manage update and upgrade paths. Huh? Old-school sysadmins know how to keep systems running, manage updates and upgrades. At the same time nobody knows how to build Hadoop from scratch. At the same time, Hadoop build instructions themselves have curl|sh scripts or mirrors and the wiki page is outdated. And it uses Java (and thus maven/ivy). And that downloads the internet. According to the blog, Hadoop, maven/ivy/sbt/any dependency manager, package managers, and everything is broken. But the tagline is: > This rant is about containers, prebuilt VMs What does any of this have to do with the "Age of containers" and pre-built VMs? Is the author just talking about Gentoo/LFS-style "compile the whole system from scratch"? This feels like an incredibly rushed rant. I can only envision the author requiring to setup hadoop for the first time, breaking their head for a few days (it happens), and taking it out on everything.

carrja99 8y ago
It's a rant, my inclination is that it has to do with a very specific situation the author is facing at work.
- djsumdog 8y ago
  The author should have grabbed the .sdeb or the debian build scripts and tore them apart if they really wanted to make a point (if, upon examining the build, there was one to make). I mean there is a lot of cognitive load/disconnect we're talk about. As an ops guy, I can't look into every package. That's why I trust the package manager (apt-get, yum, whatever) and all the build maintainers who either volunteer or work on for Redhat/Canonical/SuSE/IBM/whoever. Things get through. That's why we have all those security people out there who are digging around for bug bounties and find crap like the recent Ubuntu Snap package craziness. Docker containers can be good. You can use an official Ubuntu or Alpine image, build your base, and create scripts to make sure your base containers don't go out of date. Most people don't do that. The official Docker containers are kinda a mess, but at least they're maintained. Grabbing some random container off Dockerhub? Yea that's not going to end well; unless you just use their source to build your own. Or if it's a container continually maintained but the person/company who wrote the service. Docker containers do need better security introspection and that's going to be a big deal going forward. But this article is all rant and some, but not enough, substance.
  - mcguire 8y ago
    "Docker containers do need better security introspection and that's going to be a big deal going forward." Exactly! And npm. And maven. And every damn package system for every damn programming language since package systems are now a requirement.
    - smaddox 8y ago
      Yes, but shouldn't you have separate "build" and "deploy" container images? You should "build" a particular version once, "deploy" the result into a test environment, test it thoroughly, and then "deploy" to production, right? This is not my job (yet). Please tell me if I'm wrong, because I'll need to do it in the next few months.
emilsedgh 8y ago
I think the logic is, if we didn't rely on Containers and prebuilt VM's, Hadoop had to be easier to build to be useful.
- icebraining 8y ago
  I think that is the author's logic. Except it's not very logic, since Hadoop (or Bigtop) doesn't use either.
  - mcguire 8y ago
    Picture this: you need to use Hadoop. Do you: A) work through building it yourself, or B) get a container that claims to have a running Hadoop and hope it works for you? If B wasn't on the table, what would happen?
    - icebraining 8y ago
      If I need to use Hadoop, I'll download one of the pre-built binaries that they offer on their site. You'll notice that the Debian Wiki users have given up on building it since 2010. That was three years before Docker even appeared. Almost nobody was using containers back then.
    - icedchai 8y ago
      Hadoop isn't even that difficult to set up. I've built it from source, and installed it from binaries. Containers are totally unnecessary here, just as they are for most java apps.
- badloginagain 8y ago
  The point everyone seems to be missing, and the one I think most important, is that we're no longer building from trusted sources. Build systems just download and run random code from the internet without verifying that its the correct code, from the correct source. Its a ticking time bomb.
  - 8y ago
mkirklions 8y ago
Can I complain about pre-built VMs? I swear by the time I figure out my path problems I have 4 folders named /code/ And my working file system is /code/code/index.php So when I run node.js (to compile my .scss) it messes up the folder pathing and ugh... why cant we do everything on production? /rant
AlexCoventry 8y ago
The author has added an update to the bottom of the post which I think makes his main intended message clearer: Update: it was pointed out that this started way before Docker: »Docker is the new ‘curl | sudo bash‘«. That’s right, but it’s now pretty much mainstream to download and run untrusted software in your “datacenter”. That is bad, really bad. Before, admins would try hard to prevent security holes, now they call themselves “devops” and happily introduce them to the network themselves!
commandlinefan 8y ago
I've seen this brewing for a while, and getting worse and worse. Back in the 80's and 90's, there were developers who would code their own sorting or hashing routines rather than linking in some external library to handle this "solved" problem. The perjorative term "Not Invented Here" (NIH) grew to describe those developers and they were shamed into reusing code whenever there was code to reuse. And in some cases (like sort routines), it makes perfect sense. However, NIH accusations have grown to "if there's something vaguely similar to what you're writing, you must use it, even if that involves more custom coding to artificially bend it to the case at hand than you would have developed in the first place", culminating in things like the completely empty, useless (but enormous) Spring "framework" or, to a lesser extent, things like Angular that sort of do some things, but create far more problems than they solve (and definitely add more development overhead than they remove).
- ironjunkie 8y ago
  Interesting take on the NIH term. I thought this was more an ego thing for the big tech companies. They love to reinvent existing things to look like geniuses
  - commandlinefan 8y ago
    I think you're correct on the origin of the term, and I should have mentioned that - but in the past few decades, I've been accused of "NIH"-ing whenever I've "rolled something" of my own from authentication to IoC. Just because there's a library that has a particular description attached to it doesn't mean that it should be used as often as it can be.

carrja99 8y ago

It's a rant, my inclination is that it has to do with a very specific situation the author is facing at work.

djsumdog 8y ago
The author should have grabbed the .sdeb or the debian build scripts and tore them apart if they really wanted to make a point (if, upon examining the build, there was one to make). I mean there is a lot of cognitive load/disconnect we're talk about. As an ops guy, I can't look into every package. That's why I trust the package manager (apt-get, yum, whatever) and all the build maintainers who either volunteer or work on for Redhat/Canonical/SuSE/IBM/whoever. Things get through. That's why we have all those security people out there who are digging around for bug bounties and find crap like the recent Ubuntu Snap package craziness. Docker containers can be good. You can use an official Ubuntu or Alpine image, build your base, and create scripts to make sure your base containers don't go out of date. Most people don't do that. The official Docker containers are kinda a mess, but at least they're maintained. Grabbing some random container off Dockerhub? Yea that's not going to end well; unless you just use their source to build your own. Or if it's a container continually maintained but the person/company who wrote the service. Docker containers do need better security introspection and that's going to be a big deal going forward. But this article is all rant and some, but not enough, substance.
- mcguire 8y ago
  "Docker containers do need better security introspection and that's going to be a big deal going forward." Exactly! And npm. And maven. And every damn package system for every damn programming language since package systems are now a requirement.
  - smaddox 8y ago
    Yes, but shouldn't you have separate "build" and "deploy" container images? You should "build" a particular version once, "deploy" the result into a test environment, test it thoroughly, and then "deploy" to production, right? This is not my job (yet). Please tell me if I'm wrong, because I'll need to do it in the next few months.

emilsedgh 8y ago

I think the logic is, if we didn't rely on Containers and prebuilt VM's, Hadoop had to be easier to build to be useful.

icebraining 8y ago
I think that is the author's logic. Except it's not very logic, since Hadoop (or Bigtop) doesn't use either.
- mcguire 8y ago
  Picture this: you need to use Hadoop. Do you: A) work through building it yourself, or B) get a container that claims to have a running Hadoop and hope it works for you? If B wasn't on the table, what would happen?
  - icebraining 8y ago
    If I need to use Hadoop, I'll download one of the pre-built binaries that they offer on their site. You'll notice that the Debian Wiki users have given up on building it since 2010. That was three years before Docker even appeared. Almost nobody was using containers back then.
  - icedchai 8y ago
    Hadoop isn't even that difficult to set up. I've built it from source, and installed it from binaries. Containers are totally unnecessary here, just as they are for most java apps.
badloginagain 8y ago
The point everyone seems to be missing, and the one I think most important, is that we're no longer building from trusted sources. Build systems just download and run random code from the internet without verifying that its the correct code, from the correct source. Its a ticking time bomb.
- 8y ago

mkirklions 8y ago

Can I complain about pre-built VMs? I swear by the time I figure out my path problems I have 4 folders named /code/ And my working file system is /code/code/index.php So when I run node.js (to compile my .scss) it messes up the folder pathing and ugh... why cant we do everything on production? /rant

AlexCoventry 8y ago

The author has added an update to the bottom of the post which I think makes his main intended message clearer: Update: it was pointed out that this started way before Docker: »Docker is the new ‘curl | sudo bash‘«. That’s right, but it’s now pretty much mainstream to download and run untrusted software in your “datacenter”. That is bad, really bad. Before, admins would try hard to prevent security holes, now they call themselves “devops” and happily introduce them to the network themselves!

commandlinefan 8y ago

I've seen this brewing for a while, and getting worse and worse. Back in the 80's and 90's, there were developers who would code their own sorting or hashing routines rather than linking in some external library to handle this "solved" problem. The perjorative term "Not Invented Here" (NIH) grew to describe those developers and they were shamed into reusing code whenever there was code to reuse. And in some cases (like sort routines), it makes perfect sense. However, NIH accusations have grown to "if there's something vaguely similar to what you're writing, you must use it, even if that involves more custom coding to artificially bend it to the case at hand than you would have developed in the first place", culminating in things like the completely empty, useless (but enormous) Spring "framework" or, to a lesser extent, things like Angular that sort of do some things, but create far more problems than they solve (and definitely add more development overhead than they remove).

ironjunkie 8y ago
Interesting take on the NIH term. I thought this was more an ego thing for the big tech companies. They love to reinvent existing things to look like geniuses
- commandlinefan 8y ago
  I think you're correct on the origin of the term, and I should have mentioned that - but in the past few decades, I've been accused of "NIH"-ing whenever I've "rolled something" of my own from authentication to IoC. Just because there's a library that has a particular description attached to it doesn't mean that it should be used as often as it can be.