3 ms·

Even after choosing a technology (like Hadoop), there's a lot of difference between the best and the worst algorithm to solve a problem. For example, the reduc

by C4stor 8y ago

Even after choosing a technology (like Hadoop), there's a lot of difference between the best and the worst algorithm to solve a problem. For example, the reduce phase of MapReduce is pretty much "apply the algorithm you want on this List", and that's where knowing good list algorithms (for example) shine.

kamaal 8y ago
Sorry, in almost all cases its really choosing over several ways of Joining the data, not writing SELECT *, not writing cross joins and learning SQL well. Which why I ask again. Please list your problem which is so novel it requires you to invent a novel algorithm. Please note statistics is a science that has existed for centuries now. Unless you are in a university, its highly unlikely you have a problem that will need to you to work on something that novel.
- C4stor 8y ago
  I don't understand your point. It seems to me you mix statisticals approaches and algorithms, meaning you completely ignore the fact that code runs on computers, and that computers have mechanical characteristics, making two implementations of the same statistical approach wildly different in performance. I used to work at a company where some guys spent literally weeks inventing a fast way to do a dot product over huge vectors. Which doesn't mean they invented the concept of dot product.
  - kamaal 8y ago
    >>I used to work at a company where some guys spent literally weeks inventing a fast way to do a dot product over huge vectors. Very nice that you bought up this point. This is exactly what I was trying to point out. If a team of full time working people took weeks to arrive at a working solution, despite a massive body of open knowledge available as a reference to prior solutions, there is absolutely no way a candidate can give even arrive at a decent direction to the solution in under 45 minutes in a interview situation. No one is saying algorithms are not useful. But there is a very big difference between knowing an algorithm and inventing an algorithm. Testing the former is no indication of the latter, as illustrated by your own example.

by C4stor 8y ago

kamaal 8y ago

Sorry, in almost all cases its really choosing over several ways of Joining the data, not writing SELECT *, not writing cross joins and learning SQL well. Which why I ask again. Please list your problem which is so novel it requires you to invent a novel algorithm. Please note statistics is a science that has existed for centuries now. Unless you are in a university, its highly unlikely you have a problem that will need to you to work on something that novel.

C4stor 8y ago
I don't understand your point. It seems to me you mix statisticals approaches and algorithms, meaning you completely ignore the fact that code runs on computers, and that computers have mechanical characteristics, making two implementations of the same statistical approach wildly different in performance. I used to work at a company where some guys spent literally weeks inventing a fast way to do a dot product over huge vectors. Which doesn't mean they invented the concept of dot product.
- kamaal 8y ago
  >>I used to work at a company where some guys spent literally weeks inventing a fast way to do a dot product over huge vectors. Very nice that you bought up this point. This is exactly what I was trying to point out. If a team of full time working people took weeks to arrive at a working solution, despite a massive body of open knowledge available as a reference to prior solutions, there is absolutely no way a candidate can give even arrive at a decent direction to the solution in under 45 minutes in a interview situation. No one is saying algorithms are not useful. But there is a very big difference between knowing an algorithm and inventing an algorithm. Testing the former is no indication of the latter, as illustrated by your own example.