4 ms·

I think the logic is, if we didn't rely on Containers and prebuilt VM's, Hadoop had to be easier to build to be useful.

by emilsedgh 8y ago

icebraining 8y ago
I think that is the author's logic. Except it's not very logic, since Hadoop (or Bigtop) doesn't use either.
- mcguire 8y ago
  Picture this: you need to use Hadoop. Do you: A) work through building it yourself, or B) get a container that claims to have a running Hadoop and hope it works for you? If B wasn't on the table, what would happen?
  - icebraining 8y ago
    If I need to use Hadoop, I'll download one of the pre-built binaries that they offer on their site. You'll notice that the Debian Wiki users have given up on building it since 2010. That was three years before Docker even appeared. Almost nobody was using containers back then.
  - icedchai 8y ago
    Hadoop isn't even that difficult to set up. I've built it from source, and installed it from binaries. Containers are totally unnecessary here, just as they are for most java apps.
badloginagain 8y ago
The point everyone seems to be missing, and the one I think most important, is that we're no longer building from trusted sources. Build systems just download and run random code from the internet without verifying that its the correct code, from the correct source. Its a ticking time bomb.
- dullgiulio 8y ago
  There is SSL/TLS, unless it's done wrong (invalid certificates get ignored by the dependencies manager), it's safer than the old "md5 of the file" systems. Now, some dependencies are fraudolent (especially true in the Javascript world because it eventually targets a lot of user browsers), but nobody ever checked the sources anyway...
  - cesarb 8y ago
    TLS only verifies that have connected to the correct server. It can't verify whether the package on the server has been replaced by a malicious one. For that, you need a "md5 of the file" (these days, a sha256, because md5 has long been broken).
    - dullgiulio 8y ago
      You need to make sure the hash is also not tampered, both on server and in flight to the user. How do you do that? If the answer is: use TLS, there is no point in having the file hash at all.
      - dijit 8y ago
        No, the answer is to use PGP and a manifest hash. This is how package managers work. TLS doesn't replace those.
mcguire 8y ago
Which isn't really true; as a sysadmin (I'd say "former", but once you're a sysadmin, you're always a sysadmin), I've seen lots of things with horrible build and dependency nightmares, and that was before package managers, containers, and virtual machine images became de rigeuer. Think of a self-hosting programming language: you can't build it without a running installation of a previous version. (Anyone remembering "On Trusting Trust" at this point?) Or any application in an image-based language like Smalltalk. Development becomes path-dependent. It's inevitable to get into a situation where A and B cannot be made to work together, except in a derivative of a version that someone, somewhere made while holding their mouth the right way. Pre-built containers and VMs are an admission that path-dependence is the way stuff is supposed to be.
rdsubhas 8y ago
That's like saying, "If we didn't invent the internet, we would have never had privacy issues". OK, so if we didn't rely on containers - would hadoop have had a perfect set of packages for every distribution? Let's say that the packages for Arch linux were broken. What next? That's the whole problem with the article. It takes a problem (building Hadoop was bad), correlates it to a completely different tool (because we have docker, hadoop build scripts are bad), and goes on to rant about everything else.

I think the logic is, if we didn't rely on Containers and prebuilt VM's, Hadoop had to be easier to build to be useful.

icebraining 8y ago

I think that is the author's logic. Except it's not very logic, since Hadoop (or Bigtop) doesn't use either.

mcguire 8y ago
Picture this: you need to use Hadoop. Do you: A) work through building it yourself, or B) get a container that claims to have a running Hadoop and hope it works for you? If B wasn't on the table, what would happen?
- icebraining 8y ago
  If I need to use Hadoop, I'll download one of the pre-built binaries that they offer on their site. You'll notice that the Debian Wiki users have given up on building it since 2010. That was three years before Docker even appeared. Almost nobody was using containers back then.
- icedchai 8y ago
  Hadoop isn't even that difficult to set up. I've built it from source, and installed it from binaries. Containers are totally unnecessary here, just as they are for most java apps.

badloginagain 8y ago

The point everyone seems to be missing, and the one I think most important, is that we're no longer building from trusted sources. Build systems just download and run random code from the internet without verifying that its the correct code, from the correct source. Its a ticking time bomb.

dullgiulio 8y ago
There is SSL/TLS, unless it's done wrong (invalid certificates get ignored by the dependencies manager), it's safer than the old "md5 of the file" systems. Now, some dependencies are fraudolent (especially true in the Javascript world because it eventually targets a lot of user browsers), but nobody ever checked the sources anyway...
- cesarb 8y ago
  TLS only verifies that have connected to the correct server. It can't verify whether the package on the server has been replaced by a malicious one. For that, you need a "md5 of the file" (these days, a sha256, because md5 has long been broken).
  - dullgiulio 8y ago
    You need to make sure the hash is also not tampered, both on server and in flight to the user. How do you do that? If the answer is: use TLS, there is no point in having the file hash at all.
    - dijit 8y ago
      No, the answer is to use PGP and a manifest hash. This is how package managers work. TLS doesn't replace those.

mcguire 8y ago

Which isn't really true; as a sysadmin (I'd say "former", but once you're a sysadmin, you're always a sysadmin), I've seen lots of things with horrible build and dependency nightmares, and that was before package managers, containers, and virtual machine images became de rigeuer. Think of a self-hosting programming language: you can't build it without a running installation of a previous version. (Anyone remembering "On Trusting Trust" at this point?) Or any application in an image-based language like Smalltalk. Development becomes path-dependent. It's inevitable to get into a situation where A and B cannot be made to work together, except in a derivative of a version that someone, somewhere made while holding their mouth the right way. Pre-built containers and VMs are an admission that path-dependence is the way stuff is supposed to be.

rdsubhas 8y ago

That's like saying, "If we didn't invent the internet, we would have never had privacy issues". OK, so if we didn't rely on containers - would hadoop have had a perfect set of packages for every distribution? Let's say that the packages for Arch linux were broken. What next? That's the whole problem with the article. It takes a problem (building Hadoop was bad), correlates it to a completely different tool (because we have docker, hadoop build scripts are bad), and goes on to rant about everything else.