3 ms·
Engineers are better equipped to collect and store massive amounts of data than to analyze it to draw meaningful conclusions from it or to drive decisions. It t
by karterk 13y ago
Engineers are better equipped to collect and store massive amounts of data than to analyze it to draw meaningful conclusions from it or to drive decisions. It takes business acumen to ask the right questions to the data.
The way "big data" is stored today also poses problems with respect to queryability. NoSQL systems are great at storing huge amounts of data efficiently but don't help us slice and dice the data easily. Having to write map reduce jobs for every "query" is painful and time consuming. Tools like Hive, Pig and Cascading help in writing MR jobs succinctly but are still very slow when someone wants to quickly filter and explore the data.
- izendejas 13y agoYou should look into Spark: http://spark-project.org/ http://spark-project.org/ and Shark (Spark + Hive). I personally find that using the Spark repl is both efficient and flexible; and being able to cache data in memory makes for very fast iterations, especially when doing something more sophisticated. There's also Cloudera Impala, but I've never used it, nor do I think I'll ever have to given what the AMPLab folks are doing.