4 ms·
Doesn't need it. If you're computing a linear/logistic regression via gradient descent, performance-oriented C code on a single machine using local filesystem/
by jbooth 12y ago
Doesn't need it. If you're computing a linear/logistic regression via gradient descent, performance-oriented C code on a single machine using local filesystem/caches will beat a hadoop-based algorithm for just about any size dataset.
- riffraff 12y agoI meant that shops are likely to already have data sitting in hdfs/hive/whatever which they can trivially use with mahout, while they may not have it sitting on a single file system on a beefy machine.
- discardorama 12y agoVW works on Hadoop just fine, though it's not very "clean" and native. But it works.
- jbooth 12y agoThey can hadoop fs -get it or query it from hive. Because you want to coalesce all your model updates for every pass over the data, the long startup-time for hadoop jobs actually plays into this. You can have hundreds of millions of samples, in a sparse space of hundreds of millions of parameters, and do it faster on a single node using VW than in a hadoop cluster of equivalent nodes running Mahout.