6 ms·
First, I don't believe that Map-Reduce is not used inside Google anymore. Second, the map-reduce pattern is actually very useful basic, common computation. It'
by random3 12y ago
First, I don't believe that Map-Reduce is not used inside Google anymore.
Second, the map-reduce pattern is actually very useful basic, common computation. It's equivalent to the SQL JOIN and it's not something that you can really do without.
Perhaps the large chunk (batch) approach is not ideal for many of the use-cases Map-Reduce is being tried with (e.g. like "interactive" querying with Pig or Hive). But that doesn't mean it's not useful. If you're optimizing for throughput you'll generally want to read/process in the batches optimized for some underlying sizes (it could be page size, it could be blocks, etc.).
Also a system is a lot more than just the compute framework. It needs to deal with various inputs/outputs, do scheduling, etc.
Between the distributed storage and the distributed processing, I'm not sure it's easy to decide which one could be more difficult, either.
Saying that Hadoop is a disaster, is not far from saying we live in an awful world. Working with them for years doesn't make me the most objective person, however, given the huge adoption, I'd say they may not be that bad. More, using something like Cloudera Manager makes it trivial (which sometimes makes me wonder why the vanilla version hasn't been improved...) (BTW there's QFS and other distributed file systems).
I wonder why is the Apache ecosystem horrible?
I get it that you don't like Java. Fair enough. What would be your language of choice for the next gen, stable distributed file system? Go, Rust, JavaScript?
- deleted 12y ago[deleted]
- wyager 12y ago>What would be your language of choice for the next gen, stable distributed file system? Here's my heavily biased subjective opinion on this entirely hypothetical software: I think we should do one or both of two things: A) Do it in very clean, fast, simple C. Put an emphasis on speed and simplicity. B) Do it in very reliable, secure, simple Haskell. Put an emphasis on correctness and simplicity. With some effort, the C one could be correct and the Haskell one could be fast. I mention these two languages because they compile to native code and have very good cross-platform support. You won't have any trouble running either of these on embedded devices (which I can't say for Java or Go. Go has some weird compiler bugs on ARM platforms, and the JVM is frequently too memory intensive for embedded). C has an advantage of allowing the absolute minimal implementation, and Haskell has an advantage of allowing a massively concurrent implementation. Yada yada yada Of course, it could be that the question is completely irrelevant. Just define a spec for a DFS, and then let different implementations pop up in whatever language is best suited to that implementation's specific details.
- nl 12y agoYou won't have any trouble running either of these on embedded devices (which I can't say for Java or Go. Go has some weird compiler bugs on ARM platforms, and the JVM is frequently too memory intensive for embedded). Why is this important in this use-case? If the DFS is being used for data processing then presumably the nodes are reasonably capable machines. There may well be a difference use-case for a DFS for embedded and resource-constrained devices. That's not what Google or Hadoop is doing though.
- delian66 12y agoCPU and memory resources spend on an inefficient filesystem implementation are just wasted resources, not available for your workload. Keep in mind that the inefficiencies are multiplied over all your cluster nodes.
- vidarh 12y agoThe biggest limiting factor even in our relatively low-density populated rack is heat and power. With off the shelf servers and relatively low density, I can trivially exceed the highest power allocations our colo provider will normally allow per rack. The more power you waste on inefficient CPU usage, the less you can devote to putting more drives in.
- nl 12y agoThe OP's claim is that memory is the limiting factor in the case of Java. I don't entirely agree, but even if I did it would almost certainly be a fixed overhead per machine, and unlikely to be a problem on server class machines. Also, the read/processing characteristics of compute nodes often means the CPU is underutilized while filesystem operations are ongoing.
- srean 12y agoI will leave with an elliptical meta-comment, for those whose competitive advantage lies in others not getting it right, have little interest in correcting misconceptions. You might have interest in this anecdote https://news.ycombinator.com/item?id=7948170 https://news.ycombinator.com/item?id=7948170
- deleted 12y ago[deleted]
- pedroo 12y ago> It's equivalent to the SQL JOIN... Don't you mean GROUP BY?
- e12e 12y agoI think parent meant equivalent in the sense of being "[a] very useful basic, common computation. (...) not something that you can really do without."
- walshemj 12y agoFortran or PL/1G gets my vote writing Map Reduce in java is painful compared to PL/1G mappers I was writing back in the 80's on PR1ME mini computers.
- shiven 12y agoWait, what? A distributed file system written in Fortran? Is it even possible in a language like that?
- gtirloni 12y agoPossibly yes, since it can interoperate with C. That's not what most people use it for (networking) so you would be swimming against the current. I don't think it would attract a following among developers anyway... perhaps if you emphasize it compiles to Javascript, who knows ;)
- walshemj 12y agono my bad thought that they where asking what would you use instead of java for he mapper/reducer code. having said that at BT we did have a distributed ISAM database where the core code was F77!