3 ms·

Reconsider whether you really need to: https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html (a

Reconsider whether you really need to: https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html (although it might look good on your CV)

heartsucker 10y ago
I would second this. I've used Hadoop at two jobs, and both times the added overhead in terms of operational and programming complexity wasn't worth it. Most everything could have been done on a beefy machine with the most basic knowledge of parallel programming.
vkjv 10y ago
I don't recall where I heard this analogy, but I repeat it frequently. "Not using a hadoop is like cutting down a forest with a single chainsaw. Using it is like cutting a forest down with unlimited hatchets. It will usually be faster and cheaper with the chainsaw."
SmellTheGlove 10y ago
It's definitely a resume builder. The "Teradata" on my resume has sort of gone out of style. Other than cost, though, I've really not run into a practical situation where Hadoop would help me with something that Teradata wouldn't do. I work for large companies that can generally afford Teradata and to scale it as needed (Teradata is awesome when you need to scale easily), so my professional Hadoop experience is near zero. I read the link and agree with most of it. All except for when he gets to the 5TB part, but I get that it was written in 2013 and probably not to a fortune 500 audience. If Teradata Cloud could get a reasonably priced entry level tier going, I think they'd be talked about a bit more.
hubatrix 10y ago
I am learning this as part of academics, but I belive Map reduce was introduced for handling google page rank and it seemed to handle it well, so SHOULD I SAY "USE HADOOP BUT IN A SENSIBLE WAY WHERE IT REALLY HELPS" or is it like never use it ?
- codepie 10y ago
  Since you mentioned page rank, you can also check out graph processing frameworks, eg Pregel/Giraph, which are designed specifically to run graph algorithms on big data.
  - heartsucker 10y ago
    Replying to this again. Using Scala and Scalding or Spark makes it super simple to do algorithmic stuff. I haven't used Pregel/Giraph ever, so I can't say how it stacks up. And OP, if you see this, using raw M/R to do things like graph processing is going to be a HUGE pain in comparison to all the libraries built around it.

Reconsider whether you really need to: https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html (although it might look good on your CV)

heartsucker 10y ago

I would second this. I've used Hadoop at two jobs, and both times the added overhead in terms of operational and programming complexity wasn't worth it. Most everything could have been done on a beefy machine with the most basic knowledge of parallel programming.

vkjv 10y ago

I don't recall where I heard this analogy, but I repeat it frequently. "Not using a hadoop is like cutting down a forest with a single chainsaw. Using it is like cutting a forest down with unlimited hatchets. It will usually be faster and cheaper with the chainsaw."

SmellTheGlove 10y ago

It's definitely a resume builder. The "Teradata" on my resume has sort of gone out of style. Other than cost, though, I've really not run into a practical situation where Hadoop would help me with something that Teradata wouldn't do. I work for large companies that can generally afford Teradata and to scale it as needed (Teradata is awesome when you need to scale easily), so my professional Hadoop experience is near zero. I read the link and agree with most of it. All except for when he gets to the 5TB part, but I get that it was written in 2013 and probably not to a fortune 500 audience. If Teradata Cloud could get a reasonably priced entry level tier going, I think they'd be talked about a bit more.

hubatrix 10y ago

I am learning this as part of academics, but I belive Map reduce was introduced for handling google page rank and it seemed to handle it well, so SHOULD I SAY "USE HADOOP BUT IN A SENSIBLE WAY WHERE IT REALLY HELPS" or is it like never use it ?

codepie 10y ago
Since you mentioned page rank, you can also check out graph processing frameworks, eg Pregel/Giraph, which are designed specifically to run graph algorithms on big data.
- heartsucker 10y ago
  Replying to this again. Using Scala and Scalding or Spark makes it super simple to do algorithmic stuff. I haven't used Pregel/Giraph ever, so I can't say how it stacks up. And OP, if you see this, using raw M/R to do things like graph processing is going to be a HUGE pain in comparison to all the libraries built around it.