10 ms·

This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=command%20line%20too

by wenc 5y ago

This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=command%20line%20tools%20hadoop&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.

commandlinefan 5y ago
Well, a lot of AWS and GCP infrastructure use it, but they hide it from you.
- fulafel 5y ago
  Any specifics about which services?
  - ryapric 5y ago
    AWS: Glue, and by proxy Athena GCP: Dataproc Those are just the obvious ones though.
    - mr_toad 5y ago
      Glue often uses EMR under the hood, which is often Spark. And Athena is PrestoDB, as far as I know it has nothing to do with Hadoop other than you can use it to query Hadoop data stores.
      - trashtester 5y ago
        The way I see it, Hadoop is still in common use as the storage layer for Spark and related implementations, whether that is in the form of HDFS or something like EMRFS: Quote from AWS: "EMRFS is an implementation of the Hadoop file system ..." https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-file-systems.html https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-p...
        commandlinefan 5y ago
        Ah - that's a good point. Usually when people say "Hadoop", I assume they're referring to HDFS, but there is (was...) the Hadoop MR system that ran on top of it that has been almost entirely replaced by Spark.
jamal-kumar 5y ago
Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or whatever streaming APIs you write with your own preferred language?
- bpodgursky 5y ago
  I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern). I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.
  - mohanmcgeek 5y ago
    But I can write SQL in Spark just fine. Can't I?
    - bpodgursky 5y ago
      You can write some SQL in Spark, but 1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc. 2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is doing on the backend to support their SQL performance, but it's really freaking fast. There's just no other way to cut it. I've spent 5-10 years eking every ounce of performance I can get out of a Hadoop/Spark cluster. I'm not trying to be unreasonable about this. I would love for OSS to be competitive; it's great for the world, and it would be great for my skill set and earning potential. But it's not a contest, and if you think standalone Spark is going to be a viable competitor in a couple years, you are deluding yourself. Make informed choices about your career and investment.
      - rxin 5y ago
Puts 5y ago
Maybe we need a reminder every now and then that text-files are quite capable and except from being faster to parse in a lot of situation has the benefits of being "future safe" and easy to backup and compress as well.
- spullara 5y ago
  I mean, it isn't like Hadoop wasn't used to parse text files. Also, it is all fun and games until you need a join.
  - kragen 5y ago
    comm(1) and join(1) can do joins on text files. Make sure they're sorted in the same locale you're joining them in; LANG=C tends to be the fastest.
- IanCal 5y ago
  The common text file for lots of this, CSVs, are absolutely awful. They're fine until they totally aren't, they were just the best option for a lot of use cases. I'd argue that's now been entirely replaced with parquet, significantly faster and broad support, proper types and more.
  - stingraycharles 5y ago
    I’d argue that for as long as Excel doesn’t support Parquet files, we haven’t seen the last of CSVs for a long time. Parquet is great, but it’s simply nowhere near as ubiquitous as CSV.
    - gehen88 5y ago
      I hadn't even heard of Parquet until now, and I'm sure this goes for lots of developers who don't do much data engineering.
      - jsjohnst 5y ago
StreamBright 5y ago
> Outside of legacy systems, Hadoop isn't widely used anymore. This is not true at all. Almost all the cloud providers have their own Hadoop distributions that is used a lot in many companies.
- rizkeyz 5y ago
  Exactly. Basically most big data lives in hdfs and hdfs is part of hadoop. Even if you use Spark and Flink I would classify that as using Hadoop (under the hood). > However, a very common setup is to use Flink to analyze data stored in the Hadoop Distributed File System (HDFS). -- https://wints.github.io/flink-web//faq.html https://wints.github.io/flink-web//faq.html
tus666 5y ago
What about if you had 500 petabytes of data to analyze? Big data does exist in the wild.
- usrbinbash 5y ago
  > What about if you had 500 petabytes of data to analyze? Then we probably need the big tools. > Big data does exist in the wild. So does little data. The problem is that a "one-size-fits-all" approach has become common, not just in data analysis; think of all the low-medium traffic webpages that use giant, complex frameworks and huge distributed systems just to display essentially a small CRUD app that would have been ALOT easier to cobble together in plain JS on a simple LAMP server. What the article shows is the importance on deciding for the right tool for the job: When I want to plant a little tree in my backyard, bringing one of these https://upload.wikimedia.org/wikipedia/commons/0/01/Bucket_wheel_excavator_in_Ferropolis.jpg https://upload.wikimedia.org/wikipedia/commons/0/01/Bucket_w... to dig the hole is proooobably overengineering it a tiny little bit, and will likely take longer than getting a shovel.
- HelloNurse 5y ago
  Those 500 PB imply a competent staff, a world-class infrastructure, and worthwhile applications. These things don't materialize suddenly: they start maturing when the data set is 500 GB or even just planned and they usually mature enough to switch technology multiple times as the scale rises.
auggierose 5y ago
So what is used instead of Hadoop currently?
- serial_dev 5y ago
  Whether a technology can replace Hadoop in an organization depends on many factors, but some technologies that solve at least in part similar problem are Apache Storm, Spark, Flink, Kafka Streams, and maybe BigQuery? Or, as the original article says, some companies just use some command line tools, shell scripts. It's been a couple of years since I was interested in Data Engineering, so my knowledge on this topic is some years behind.
  - riknox 5y ago
    I've not seen Storm being used anywhere sane for a few years at least now, and from a glance at job postings it looks unlikely. Spark, Kafka Streams etc. are definitely used in a modern data platform from my experience. I think we're seeing a big shift with Hadoop-like workloads being moved onto cloud providers, so BigQuery, Amazon EMR etc.
    - bsenftner 5y ago
      I'm curious what constitutes "big data" anymore. In an intermediate machine learning course, we train on nearly a petabyte of data using Google Colab and Jupyter Notebooks. Nobody discusses the size of the data requiring any special treatment due to its size... would not 95% of a petabyte be "big data"?
      - happymellon 5y ago
        Big data is a shifting concept as computers gain more storage and faster commodity processors. My general rule of thumb is whether it is too big to put on my laptop. So greater than a couple of Tb's.
      - pilotneko 5y ago
        What course are you taking? Imagenet is only 150 GB, and Common Crawl is only 320 TB. Big data is a moving target, but I’m comfortable defining it as data too large to fit in memory. Obviously, you can always get a bigger node, my rule is thumb is that if you need generators, you are working with big data.
morelisp 5y ago
The article still applies to the things which have replaced Hadoop. The operational, development, and computational overhead of distributing/re-aggregating work remains huge.

by wenc 5y ago

commandlinefan 5y ago

Well, a lot of AWS and GCP infrastructure use it, but they hide it from you.

fulafel 5y ago
Any specifics about which services?
- ryapric 5y ago
  AWS: Glue, and by proxy Athena GCP: Dataproc Those are just the obvious ones though.
  - mr_toad 5y ago
    Glue often uses EMR under the hood, which is often Spark. And Athena is PrestoDB, as far as I know it has nothing to do with Hadoop other than you can use it to query Hadoop data stores.
    - trashtester 5y ago
      The way I see it, Hadoop is still in common use as the storage layer for Spark and related implementations, whether that is in the form of HDFS or something like EMRFS: Quote from AWS: "EMRFS is an implementation of the Hadoop file system ..." https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-file-systems.html https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-p...
      - commandlinefan 5y ago
        Ah - that's a good point. Usually when people say "Hadoop", I assume they're referring to HDFS, but there is (was...) the Hadoop MR system that ran on top of it that has been almost entirely replaced by Spark.

jamal-kumar 5y ago

Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or whatever streaming APIs you write with your own preferred language?

bpodgursky 5y ago
I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern). I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.
- mohanmcgeek 5y ago
  But I can write SQL in Spark just fine. Can't I?
  - bpodgursky 5y ago
    You can write some SQL in Spark, but 1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc. 2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is doing on the backend to support their SQL performance, but it's really freaking fast. There's just no other way to cut it. I've spent 5-10 years eking every ounce of performance I can get out of a Hadoop/Spark cluster. I'm not trying to be unreasonable about this. I would love for OSS to be competitive; it's great for the world, and it would be great for my skill set and earning potential. But it's not a contest, and if you think standalone Spark is going to be a viable competitor in a couple years, you are deluding yourself. Make informed choices about your career and investment.
    - rxin 5y ago

Puts 5y ago

Maybe we need a reminder every now and then that text-files are quite capable and except from being faster to parse in a lot of situation has the benefits of being "future safe" and easy to backup and compress as well.

spullara 5y ago
I mean, it isn't like Hadoop wasn't used to parse text files. Also, it is all fun and games until you need a join.
- kragen 5y ago
  comm(1) and join(1) can do joins on text files. Make sure they're sorted in the same locale you're joining them in; LANG=C tends to be the fastest.
IanCal 5y ago
The common text file for lots of this, CSVs, are absolutely awful. They're fine until they totally aren't, they were just the best option for a lot of use cases. I'd argue that's now been entirely replaced with parquet, significantly faster and broad support, proper types and more.
- stingraycharles 5y ago
  I’d argue that for as long as Excel doesn’t support Parquet files, we haven’t seen the last of CSVs for a long time. Parquet is great, but it’s simply nowhere near as ubiquitous as CSV.
  - gehen88 5y ago
    I hadn't even heard of Parquet until now, and I'm sure this goes for lots of developers who don't do much data engineering.
    - jsjohnst 5y ago

StreamBright 5y ago

> Outside of legacy systems, Hadoop isn't widely used anymore. This is not true at all. Almost all the cloud providers have their own Hadoop distributions that is used a lot in many companies.

rizkeyz 5y ago
Exactly. Basically most big data lives in hdfs and hdfs is part of hadoop. Even if you use Spark and Flink I would classify that as using Hadoop (under the hood). > However, a very common setup is to use Flink to analyze data stored in the Hadoop Distributed File System (HDFS). -- https://wints.github.io/flink-web//faq.html https://wints.github.io/flink-web//faq.html

tus666 5y ago

What about if you had 500 petabytes of data to analyze? Big data does exist in the wild.

usrbinbash 5y ago
> What about if you had 500 petabytes of data to analyze? Then we probably need the big tools. > Big data does exist in the wild. So does little data. The problem is that a "one-size-fits-all" approach has become common, not just in data analysis; think of all the low-medium traffic webpages that use giant, complex frameworks and huge distributed systems just to display essentially a small CRUD app that would have been ALOT easier to cobble together in plain JS on a simple LAMP server. What the article shows is the importance on deciding for the right tool for the job: When I want to plant a little tree in my backyard, bringing one of these https://upload.wikimedia.org/wikipedia/commons/0/01/Bucket_wheel_excavator_in_Ferropolis.jpg https://upload.wikimedia.org/wikipedia/commons/0/01/Bucket_w... to dig the hole is proooobably overengineering it a tiny little bit, and will likely take longer than getting a shovel.
HelloNurse 5y ago
Those 500 PB imply a competent staff, a world-class infrastructure, and worthwhile applications. These things don't materialize suddenly: they start maturing when the data set is 500 GB or even just planned and they usually mature enough to switch technology multiple times as the scale rises.

auggierose 5y ago

So what is used instead of Hadoop currently?

serial_dev 5y ago
Whether a technology can replace Hadoop in an organization depends on many factors, but some technologies that solve at least in part similar problem are Apache Storm, Spark, Flink, Kafka Streams, and maybe BigQuery? Or, as the original article says, some companies just use some command line tools, shell scripts. It's been a couple of years since I was interested in Data Engineering, so my knowledge on this topic is some years behind.
- riknox 5y ago
  I've not seen Storm being used anywhere sane for a few years at least now, and from a glance at job postings it looks unlikely. Spark, Kafka Streams etc. are definitely used in a modern data platform from my experience. I think we're seeing a big shift with Hadoop-like workloads being moved onto cloud providers, so BigQuery, Amazon EMR etc.
  - bsenftner 5y ago
    I'm curious what constitutes "big data" anymore. In an intermediate machine learning course, we train on nearly a petabyte of data using Google Colab and Jupyter Notebooks. Nobody discusses the size of the data requiring any special treatment due to its size... would not 95% of a petabyte be "big data"?
    - happymellon 5y ago
      Big data is a shifting concept as computers gain more storage and faster commodity processors. My general rule of thumb is whether it is too big to put on my laptop. So greater than a couple of Tb's.
    - pilotneko 5y ago
      What course are you taking? Imagenet is only 150 GB, and Common Crawl is only 320 TB. Big data is a moving target, but I’m comfortable defining it as data too large to fit in memory. Obviously, you can always get a bigger node, my rule is thumb is that if you need generators, you are working with big data.

morelisp 5y ago

The article still applies to the things which have replaced Hadoop. The operational, development, and computational overhead of distributing/re-aggregating work remains huge.