5 ms·

Hadoop: When grownups do open source

61 points by sanj 18y ago

th0ma5 18y ago
heh... "Along with the data processing framework, Doug Cutting also included a fault tolerant, replicated, distributed file system with Hadoop just because fuck you"
danielrhodes 18y ago
Amazing article, only because it called out so many types of people simultaneously.
- william42 18y ago
  It's like Zed Shaw without the factual accuracy.
gcv 18y ago
And people complain about Zed Shaw's expletive-filled writing style? At least Zed has a sense of humor and blasts just about everything in sight. As for this article, let's see. Hero worship? Check. Ad hominem attacks? Check. Uninformed comments ("RubyForge? What... is that?")? Check.
- ajross 18y ago
  Yeah, this is disappointing. Mostly because Hadoop is, in fact, a pretty nifty piece of software. But it's a cluster thing, and individual hackers don't do clusters for some pretty fundamental economic reasons. So to Dziuba, that means that the Hadoop people at Yahoo and elsewhere are "grownups". Sigh.
  - scott_s 18y ago
    I thought that was an odd distinction. For some reason, he equates open-source with the Web 2.0, Ruby crowd. The first things that come to my mind are Linux and gcc, which are two of the most important software projects on the planet.
    - pojo 18y ago
      No, he doesn't. He's clearly drawn a distinction between Web 2.0 open source and the "rest" of open source. His entire point is to belittle and laugh at Web 2.0 open source attempts.
- william42 18y ago
  And it seems that unlike Zed Shaw who wrote the webserver behind almost every Ruby site out there these guys' technology contributions are...a digg-based search engine.
sanj 18y ago
The question I had is whether Dziuba built something interesting. Zed has.
- william42 18y ago
  He built a search engine called Pressflip which has a neat idea(say which searches are good or bad) but fails on the execution.
paul 18y ago
This guy is so annoying. You might think that the non-success of his own startup (http://www.techcrunch.com/2008/07/19/pressflip-is-a-belly-flop/ http://www.techcrunch.com/2008/07/19/pressflip-is-a-belly-fl...) would make him slightly more humble, but apparently not.
- brlewis 18y ago
  Maybe he decided to focus on his most sellable skill.
  - paul 18y ago
    Yes, I apparently those that can, do, and those that can not, talk trash.
geuis 18y ago
Parts of the article made me laugh, especially the Twitter as drum major quip (me being an old band/drumcorps member). Not knowing much about hadoop or map reduce, I gather he doesn't really have a clue based on his dismissal of Twitter as a failure. Last time I checked, they're going strong.
- sah 18y ago
  I hope someday I manage to build something that people laugh at as much as they're laughing at Twitter.
orib 18y ago
The Register: When children do journalism. He may have some points in there, but any real content is drowned by the noise of the baseless insults, ad-hominem attacks, and poor writing. The angry rant style can be made to work, but it needs to be backed up by actual skill at writing, and it definitely doesn't substitute for content. This article makes me sad.
- gizmo 18y ago
  And yet, we immediately recognize the style of the article is uncov. And we recognize the author's name. It may be bad writing, but that doesn't matter. It's good enough.
  - orib 18y ago
    No, it's not. It's sufficiently bad that I won't waste my time reading more articles by that author, and I will probably take a pass on articles by the Register, since their quality standards are low enough to let that drivel on. (Actually, I've been avoiding The Register for a few years now, but I gave them the benefit of the doubt when this appeared on hacker news. It was a mistake.)
    - jackchristopher 18y ago
      With writing (or anything), if you can't say something with a calm demeanor, it's probably not worth listening to. Rants, raves, or a negative, cynical tone suggests an ill-conceived idea; it's likely an emotional reaction.
      - sah 18y ago
        I think the idea was to be entertaining.
      - orib 18y ago
        If you read my first post, I'm not objecting to the tone so much as the way that it's used to cover the emptiness of the article. I can appreciate a good angry rant -- I've written some myself, and I tend to enjoy Zed Shaw's rants -- but an angry rant that slams things without justification is simply childishness and poor journalism. Sorry. Edit: Oops, I totally misread your post. Sorry about that. For some reason I thought you were justifying the tone of the article. I hear that learning to read can solve this sort of problem ;-) I agree that a good writer can discuss something without an angry rant, but I disagree that an angry/cynical tone implies an ill-conceived idea. I'd suggest that bad writers will try to use the angry rant format to hide the poverty of their ideas, but just as often a good writer will use it well as a rhetorical device, making an otherwise dull article more amusing, and driving home the important points. But either way, the format of the writing is (or rather, should be) secondary to the content. Edit: Clarified what I wrote a bit
gfunk911 18y ago
Wow. That might be the most unprofessional thing I've ever read.
- william42 18y ago
  I, on the other hand, have read other articles from The Register. This is par for the course.
thomasmallen 18y ago
I'm pretty much done with that e-rag.
icey 18y ago
This is a troll of epic proportions. Well, not really epic. Mini-series proportion is probably more apt.
- dfranke 18y ago
  I gathered that much just from the title. If you trained a Bayesian filter to recognize trolling, "grownups" would probably be one of the strongest troll tokens. adequacy.org even subtitled itself "news for grownups".
jon_dahl 18y ago
Heh - I'm working on a MapReduce blog post as we speak (see http://tinyurl.com/6oylnr http://tinyurl.com/6oylnr). So I suppose I'm prone to his attack. I actually think he makes one good point, that virtually no one actually needs to use MapReduce, let alone implement it, let alone write a blog tutorial on how to implement it. That said, virtually no one needs to write a compiler either, and yet understanding compilers (or even writing one) is not a waste. Also, his comparison of a message queue (Starling) to MapReduce almost makes you wonder if he really understands MapReduce. Starling has nothing to do with MapReduce, and MapReduce is completely unrelated to Twitter's scaling problems.
- nickb 18y ago
  And, Starling is not the reason for Twitter's scaling problems. It's actually a very very simple and very useful and it works as advertised.
bporterfield 18y ago
Does anyone on HN have any first-hand experience configuring and using Hadoop? If so, how was the process - was it straightfoward or complex, did it require lots of peeking into the source, any major pitfalls, etc - and, are you satisfied with how it's currently working and with your ability to add new boxes easily?
- strlen 18y ago
  Hi, I've setup a hadoop cluster for a start-up I am working in. Initial reason? The amount of incoming log data was getting too slow to process on a single machine and having had a single graduate course in distributed systems I knew a) what is required to go from a single script to a system distributed across nodes b) that I am not in the capacity (alone) to create this system in a time window that will meet business needs. As a result, I saw hadoop as offering a solution to: implement a distribute process where by records are processes in parallel by multiple machines without having to worry about implementing locking and message passing; distribute disk seek times across a cluster of nodes, pool together disk space with ability to add disk space by simply adding new commodity machines to a cluster. If you have these specific needs: a) need to rapidly and robustly process more data than a single machine can handle (think of it this way -- if you do data processing with a Perl script on a single machine, will the data be simply too old by the time the Perl script is done?) b) ability to add storage for new data without powering machines down c) the data is non-relational (i.e. you aren't exporting data from MySQL and then re-importing it again - if your data can be stored effectively inside a MySQL cluster and queried in a relational manner, you don't need to use hadoop). Now, if you still think you need hadoop (in most of the cases, answer is "you don't"): To get hadoop going minimally to be useful right away is actually almost trivial. However, as your dataset grows, the amount of jobs and the size of the cluster grow as well as the expectation of hadoop to also be a reliable data store grow, you will run into plateaus that you'll need to overcome. To answer your questions directly: - Getting it going is easy, trivial. Do it first on your desktop (if you run Linux directly, if not under VMWare). Process is straight forward. - Peeking into the source is not required to get it going - Pitfalls: * Use enough memory, JVM is a memory hog. Always make sure you have sufficient swap (4x your memory) * Find the optimal size of input data for your task (if you are using compressed files for input). (Yes, I will write these learnings up eventually).
auston 18y ago
"...but rather as an act of mercy for his keyboard" -Classic.
tlrobinson 18y ago
The only thing I got from this article is reaffirmation that Ted Dziuba is a huge Hadoop fanboy and still hates Twitter.

61 points by sanj 18y ago

heh... "Along with the data processing framework, Doug Cutting also included a fault tolerant, replicated, distributed file system with Hadoop just because fuck you"

danielrhodes 18y ago

Amazing article, only because it called out so many types of people simultaneously.

william42 18y ago
It's like Zed Shaw without the factual accuracy.

gcv 18y ago

And people complain about Zed Shaw's expletive-filled writing style? At least Zed has a sense of humor and blasts just about everything in sight. As for this article, let's see. Hero worship? Check. Ad hominem attacks? Check. Uninformed comments ("RubyForge? What... is that?")? Check.

ajross 18y ago
Yeah, this is disappointing. Mostly because Hadoop is, in fact, a pretty nifty piece of software. But it's a cluster thing, and individual hackers don't do clusters for some pretty fundamental economic reasons. So to Dziuba, that means that the Hadoop people at Yahoo and elsewhere are "grownups". Sigh.
- scott_s 18y ago
  I thought that was an odd distinction. For some reason, he equates open-source with the Web 2.0, Ruby crowd. The first things that come to my mind are Linux and gcc, which are two of the most important software projects on the planet.
  - pojo 18y ago
    No, he doesn't. He's clearly drawn a distinction between Web 2.0 open source and the "rest" of open source. His entire point is to belittle and laugh at Web 2.0 open source attempts.
william42 18y ago
And it seems that unlike Zed Shaw who wrote the webserver behind almost every Ruby site out there these guys' technology contributions are...a digg-based search engine.

sanj 18y ago

The question I had is whether Dziuba built something interesting. Zed has.

william42 18y ago
He built a search engine called Pressflip which has a neat idea(say which searches are good or bad) but fails on the execution.

paul 18y ago

This guy is so annoying. You might think that the non-success of his own startup (http://www.techcrunch.com/2008/07/19/pressflip-is-a-belly-flop/ http://www.techcrunch.com/2008/07/19/pressflip-is-a-belly-fl...) would make him slightly more humble, but apparently not.

brlewis 18y ago
Maybe he decided to focus on his most sellable skill.
- paul 18y ago
  Yes, I apparently those that can, do, and those that can not, talk trash.

geuis 18y ago

Parts of the article made me laugh, especially the Twitter as drum major quip (me being an old band/drumcorps member). Not knowing much about hadoop or map reduce, I gather he doesn't really have a clue based on his dismissal of Twitter as a failure. Last time I checked, they're going strong.

sah 18y ago
I hope someday I manage to build something that people laugh at as much as they're laughing at Twitter.

orib 18y ago

The Register: When children do journalism. He may have some points in there, but any real content is drowned by the noise of the baseless insults, ad-hominem attacks, and poor writing. The angry rant style can be made to work, but it needs to be backed up by actual skill at writing, and it definitely doesn't substitute for content. This article makes me sad.

gizmo 18y ago
And yet, we immediately recognize the style of the article is uncov. And we recognize the author's name. It may be bad writing, but that doesn't matter. It's good enough.
- orib 18y ago
  No, it's not. It's sufficiently bad that I won't waste my time reading more articles by that author, and I will probably take a pass on articles by the Register, since their quality standards are low enough to let that drivel on. (Actually, I've been avoiding The Register for a few years now, but I gave them the benefit of the doubt when this appeared on hacker news. It was a mistake.)
  - jackchristopher 18y ago
    With writing (or anything), if you can't say something with a calm demeanor, it's probably not worth listening to. Rants, raves, or a negative, cynical tone suggests an ill-conceived idea; it's likely an emotional reaction.
    - sah 18y ago
      I think the idea was to be entertaining.
    - orib 18y ago
      If you read my first post, I'm not objecting to the tone so much as the way that it's used to cover the emptiness of the article. I can appreciate a good angry rant -- I've written some myself, and I tend to enjoy Zed Shaw's rants -- but an angry rant that slams things without justification is simply childishness and poor journalism. Sorry. Edit: Oops, I totally misread your post. Sorry about that. For some reason I thought you were justifying the tone of the article. I hear that learning to read can solve this sort of problem ;-) I agree that a good writer can discuss something without an angry rant, but I disagree that an angry/cynical tone implies an ill-conceived idea. I'd suggest that bad writers will try to use the angry rant format to hide the poverty of their ideas, but just as often a good writer will use it well as a rhetorical device, making an otherwise dull article more amusing, and driving home the important points. But either way, the format of the writing is (or rather, should be) secondary to the content. Edit: Clarified what I wrote a bit

gfunk911 18y ago

Wow. That might be the most unprofessional thing I've ever read.

william42 18y ago
I, on the other hand, have read other articles from The Register. This is par for the course.

thomasmallen 18y ago

I'm pretty much done with that e-rag.

icey 18y ago

This is a troll of epic proportions. Well, not really epic. Mini-series proportion is probably more apt.

dfranke 18y ago
I gathered that much just from the title. If you trained a Bayesian filter to recognize trolling, "grownups" would probably be one of the strongest troll tokens. adequacy.org even subtitled itself "news for grownups".

jon_dahl 18y ago

Heh - I'm working on a MapReduce blog post as we speak (see http://tinyurl.com/6oylnr http://tinyurl.com/6oylnr). So I suppose I'm prone to his attack. I actually think he makes one good point, that virtually no one actually needs to use MapReduce, let alone implement it, let alone write a blog tutorial on how to implement it. That said, virtually no one needs to write a compiler either, and yet understanding compilers (or even writing one) is not a waste. Also, his comparison of a message queue (Starling) to MapReduce almost makes you wonder if he really understands MapReduce. Starling has nothing to do with MapReduce, and MapReduce is completely unrelated to Twitter's scaling problems.

nickb 18y ago
And, Starling is not the reason for Twitter's scaling problems. It's actually a very very simple and very useful and it works as advertised.

bporterfield 18y ago

Does anyone on HN have any first-hand experience configuring and using Hadoop? If so, how was the process - was it straightfoward or complex, did it require lots of peeking into the source, any major pitfalls, etc - and, are you satisfied with how it's currently working and with your ability to add new boxes easily?

strlen 18y ago
Hi, I've setup a hadoop cluster for a start-up I am working in. Initial reason? The amount of incoming log data was getting too slow to process on a single machine and having had a single graduate course in distributed systems I knew a) what is required to go from a single script to a system distributed across nodes b) that I am not in the capacity (alone) to create this system in a time window that will meet business needs. As a result, I saw hadoop as offering a solution to: implement a distribute process where by records are processes in parallel by multiple machines without having to worry about implementing locking and message passing; distribute disk seek times across a cluster of nodes, pool together disk space with ability to add disk space by simply adding new commodity machines to a cluster. If you have these specific needs: a) need to rapidly and robustly process more data than a single machine can handle (think of it this way -- if you do data processing with a Perl script on a single machine, will the data be simply too old by the time the Perl script is done?) b) ability to add storage for new data without powering machines down c) the data is non-relational (i.e. you aren't exporting data from MySQL and then re-importing it again - if your data can be stored effectively inside a MySQL cluster and queried in a relational manner, you don't need to use hadoop). Now, if you still think you need hadoop (in most of the cases, answer is "you don't"): To get hadoop going minimally to be useful right away is actually almost trivial. However, as your dataset grows, the amount of jobs and the size of the cluster grow as well as the expectation of hadoop to also be a reliable data store grow, you will run into plateaus that you'll need to overcome. To answer your questions directly: - Getting it going is easy, trivial. Do it first on your desktop (if you run Linux directly, if not under VMWare). Process is straight forward. - Peeking into the source is not required to get it going - Pitfalls: * Use enough memory, JVM is a memory hog. Always make sure you have sufficient swap (4x your memory) * Find the optimal size of input data for your task (if you are using compressed files for input). (Yes, I will write these learnings up eventually).

auston 18y ago

"...but rather as an act of mercy for his keyboard" -Classic.

tlrobinson 18y ago

The only thing I got from this article is reaffirmation that Ted Dziuba is a huge Hadoop fanboy and still hates Twitter.