No problem, glad I could help. Your use case sounds pretty interesting, HDFS should fit the bill for sure. You should take a look at Parquet (http://parquet.io

No problem, glad I could help. Your use case sounds pretty interesting, HDFS should fit the bill for sure. You should take a look at Parquet (http://parquet.io/ http://parquet.io/) for storing your data. It's an open source columnar format that was designed for Hadoop, it even supports nesting (https://github.com/Parquet/parquet-mr/wiki/The-striping-and-assembly-algorithms-from-the-Dremel-paper https://github.com/Parquet/parquet-mr/wiki/The-striping-and-... <-- really interesting). Also, it already works with a lot of the Hadoop ecosystem components (MR, Hive, Pig, Cascading, Impala, ..), so your data doesn't have to move once it's in HDFS. Good luck!