3 ms·
Sites down; but, I wanted to add that at the peak of Hadoop hype, an unnamed company asked how I'd tally up occurrences of a value in a 1gb flat file, where the
by MoOmer 12y ago
Sites down; but, I wanted to add that at the peak of Hadoop hype, an unnamed company asked how I'd tally up occurrences of a value in a 1gb flat file, where the job would need to be done recurringly on multiple files, quickly. Without wanting to overthink it, I suggested a simple script iterating over the file, incrementing the key/values as they occur.
Apparently the correct answer was a Hadoop job.
Now, given that my experience with Hadoop was limited at the time, and even now only includes use of small clusters, that was surprising. It's sooooooo slow in comparison, and for single 1gb files, it still doesn't quite seem the best tool for the task to me.
Questions/Answers will sometimes be biased based on what the interviewer prefers.
- IndianAstronaut 12y agoI used to use Python/Pandas for up to 9gb files.
- kyllo 12y agoiterating over the file, incrementing the key/values as they occur Isn't that exactly what the Hadoop job would have to do anyway? The whole point of Hadoop is to process the data in a single pass, which is what your method does. As long as you iterate over the file without reading the whole thing into memory first.
- justizin 12y agoYeah, if you described an actual fucking program, and their answer was the name of a software package that you could learn in a day and they've been trying to hire an expert in for six months, and you would still just use that software package to run your program, be glad you didn't get the job. ;)