6 ms·

Looked at the Debian wiki the OP linked to: >Debian currently does not include Hadoop packages. There are a number of reasons for this; in particular the Hadoo

Looked at the Debian wiki the OP linked to: >Debian currently does not include Hadoop packages. There are a number of reasons for this; in particular the Hadoop build process will load various dependencies via Maven instead of using distribution-supplied packages. Java projects like this are unfortunately not easy to package because of interdependencies; and unfortunately the Hadoop stack is full of odd dependencies (including Apache Forrest, which for a long time only worked with Java = 1.5). >If you want to build Debian packages, the most complete efforts can be found at the Apache Bigtop project http://bigtop.apache.org/ http://bigtop.apache.org/ . Unfortunately, the build process for these packages is currently of a disastrous quality, and should only be attempted within disposable virtual machines, as it requires root permissions and will install non-packaged software. >These will allow building Hadoop packages on a recent Debian system. However, the resulting packages do not live up to Debian quality standards. In particular, they include copies of .jar files from other packages, and will thus not benefit from security updates done to these packages. >If you are interested in getting Hadoop packages into Debian, Coordinate with the Java packaging team Coordinate with upstream Apache Bigtop >to avoid duplicate efforts. Thank you. https://wiki.debian.org/Hadoop https://wiki.debian.org/Hadoop Jesus Christ.

alfozan 7y ago
Who still uses Hadoop anyway? https://spark.apache.org/ https://spark.apache.org/ https://www.iguazio.com/data-science-post-hadoop/ https://www.iguazio.com/data-science-post-hadoop/
- buzzkillington 7y ago
  Spark is worse because you need Scala as well as regular Java. I've tried building it for my day job, I would rather have a colonoscopy without sedation. It's more pleasant and dignified.
  - itg 7y ago
    And then add PySpark on top of that. Couldn't leave my last job fast enough when they decided to use Hadoop/PySpark when the largest incoming files we received were at most a few GBs.
    - boxy310 7y ago
      I once had a consulting gig where the customer desperately wanted to build a Spark/Scala ML pipeline, for a dataset that was 10 MB. We spent 3 months hammering it together for a flat Python process that would've taken us 2 weeks.
      - snaky 7y ago
        > This find xargs mawk pipeline gets us down to a runtime of about 12 seconds, or about 270MB/sec, which is around 235 times faster than the Hadoop implementation. https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
      - buzzkillington 7y ago
        If you'd sent it off to mechanical Turk it would have been done in an afternoon.
  - marcinzm 7y ago
    I build it in a container for work and didn't find it that difficult to be honest. And Google has plenty of example Dockerfiles that show the steps needed. The only real system dependencies are Java8, maven and texlive (and Python/R if you build for that). Then it's `make-distribution.sh` with the appropriate flags. Scala and everything else that is needed is downloaded by maven. The resulting directory is self-contained assuming you have java8 runtime on your target machine.
  - dcolkitt 7y ago
    FWIW, the below linked Dockerfile will download, build and install Spark in a single step. https://gist.github.com/Mister-Meeseeks/1ebf875b6e1262449cbc45c5342f592a https://gist.github.com/Mister-Meeseeks/1ebf875b6e1262449cbc...
    - deleted 7y ago
      [deleted]
- dig1 7y ago
  Sell talk and buzzwords. Either author has no idea that Hadoop is ecosystem and Spark depends on it or deliberately mix Hadoop and Kubernetes, which aren't much related. And good luck running Spark without Hadoop ;)
  - smabie 7y ago
    Spark doesn’t have a hard dependency on Hadoop. Spark doesn’t have a storage engine, but you don’t necessarily need one.
    - dig1 7y ago
      Yes, but my comment was about serious, production grade setup.
    - atomicity 7y ago
      Spark still depends on Hadoop for a lot: - Using Parquet files = parquet-mr which is tied to Hadoop MR https://github.com/apache/spark/tree/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet https://github.com/apache/spark/tree/master/sql/core/src/mai... - Using S3 instead of HDFS = Hadoop S3a connector Even if you don't run HDFS and YARN, you aren't escaping Hadoop. And if some configuration goes wrong, and you'll probably need to look into the Hadoop conf files. The original comment was about the mass of libraries that Hadoop brings in. Spark isn't a solution that allows you to leave the mess. If you try to dockerize spark, you'll still see that you have 300 MB size images full of JARs that came from wherever.
- kylek 7y ago
  Spark and dependencies are just as much of a tire fire and you’ll often want Hadoop as well.
- javabean22 7y ago
  Author doesn't know what they are talking about. Amazon EMR is just Hadoop in the could.
- mixmastamyk 7y ago
  Sounds like this is the root issue, and not significantly about package managers. No one is willing to invest the time to untangle the build process and fix compat issues in the software. The project is slowly dying out.
0xFFFE 7y ago
I recently went as far as setting up Bigtop on a VM to build the Deb packages, I gave up after 2 days. I would rather suffer through 2 root canal procedures back to back without anesthesia.
- mitchell_h 7y ago
  On a bet a couple years ago someone gave me a week to compile and stand up most of the core hadoop services distributed by horton. 4 days in I lost my shit and desk flipped.
adamc 7y ago
That triggered memories... happier in Python.

alfozan 7y ago

Who still uses Hadoop anyway? https://spark.apache.org/ https://spark.apache.org/ https://www.iguazio.com/data-science-post-hadoop/ https://www.iguazio.com/data-science-post-hadoop/

buzzkillington 7y ago
Spark is worse because you need Scala as well as regular Java. I've tried building it for my day job, I would rather have a colonoscopy without sedation. It's more pleasant and dignified.
- itg 7y ago
  And then add PySpark on top of that. Couldn't leave my last job fast enough when they decided to use Hadoop/PySpark when the largest incoming files we received were at most a few GBs.
  - boxy310 7y ago
    I once had a consulting gig where the customer desperately wanted to build a Spark/Scala ML pipeline, for a dataset that was 10 MB. We spent 3 months hammering it together for a flat Python process that would've taken us 2 weeks.
    - snaky 7y ago
      > This find xargs mawk pipeline gets us down to a runtime of about 12 seconds, or about 270MB/sec, which is around 235 times faster than the Hadoop implementation. https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
    - buzzkillington 7y ago
      If you'd sent it off to mechanical Turk it would have been done in an afternoon.
- marcinzm 7y ago
  I build it in a container for work and didn't find it that difficult to be honest. And Google has plenty of example Dockerfiles that show the steps needed. The only real system dependencies are Java8, maven and texlive (and Python/R if you build for that). Then it's `make-distribution.sh` with the appropriate flags. Scala and everything else that is needed is downloaded by maven. The resulting directory is self-contained assuming you have java8 runtime on your target machine.
- dcolkitt 7y ago
  FWIW, the below linked Dockerfile will download, build and install Spark in a single step. https://gist.github.com/Mister-Meeseeks/1ebf875b6e1262449cbc45c5342f592a https://gist.github.com/Mister-Meeseeks/1ebf875b6e1262449cbc...
  - deleted 7y ago
    [deleted]
dig1 7y ago
Sell talk and buzzwords. Either author has no idea that Hadoop is ecosystem and Spark depends on it or deliberately mix Hadoop and Kubernetes, which aren't much related. And good luck running Spark without Hadoop ;)
- smabie 7y ago
  Spark doesn’t have a hard dependency on Hadoop. Spark doesn’t have a storage engine, but you don’t necessarily need one.
  - dig1 7y ago
    Yes, but my comment was about serious, production grade setup.
  - atomicity 7y ago
    Spark still depends on Hadoop for a lot: - Using Parquet files = parquet-mr which is tied to Hadoop MR https://github.com/apache/spark/tree/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet https://github.com/apache/spark/tree/master/sql/core/src/mai... - Using S3 instead of HDFS = Hadoop S3a connector Even if you don't run HDFS and YARN, you aren't escaping Hadoop. And if some configuration goes wrong, and you'll probably need to look into the Hadoop conf files. The original comment was about the mass of libraries that Hadoop brings in. Spark isn't a solution that allows you to leave the mess. If you try to dockerize spark, you'll still see that you have 300 MB size images full of JARs that came from wherever.
kylek 7y ago
Spark and dependencies are just as much of a tire fire and you’ll often want Hadoop as well.
javabean22 7y ago
Author doesn't know what they are talking about. Amazon EMR is just Hadoop in the could.
mixmastamyk 7y ago
Sounds like this is the root issue, and not significantly about package managers. No one is willing to invest the time to untangle the build process and fix compat issues in the software. The project is slowly dying out.

0xFFFE 7y ago

I recently went as far as setting up Bigtop on a VM to build the Deb packages, I gave up after 2 days. I would rather suffer through 2 root canal procedures back to back without anesthesia.

mitchell_h 7y ago
On a bet a couple years ago someone gave me a week to compile and stand up most of the core hadoop services distributed by horton. 4 days in I lost my shit and desk flipped.

adamc 7y ago

That triggered memories... happier in Python.