3 ms·

We have made a step towards unifying HDFS with a distributed SQL layer by doing something different- we put HDFS' metadata in MySQL Cluster (NDB) - an in-memory

We have made a step towards unifying HDFS with a distributed SQL layer by doing something different- we put HDFS' metadata in MySQL Cluster (NDB) - an in-memory distributed DB that scales to 10s of nodes. NDB is not your father's MySQL. The result is HopsFS that scales to at least 16X HDFS on throughput (Usenix FAST 2017). See here: www.hops.io We have since adapted Hive SQL to move its metadata to the same DB as HopsFS (MySQL Cluster). We added foreign keys from Hive tables to the backing directories in HDFS. The result is strongly consistent metadata for Hive. That is, you can remove a backing directorty for a DB or table in HopsFS, and Hive's DB or table is automatically deleted. If you have ideas for next steps, we're all ears. It's all open-source: https://github.com/hopshadoop https://github.com/hopshadoop See 22'40 in here: https://www.youtube.com/watch?v=mTRsrjH5WLI https://www.youtube.com/watch?v=mTRsrjH5WLI

elvinyung 9y ago
Very nice! I actually read the FAST '17 paper a while back. I'm curious if HopsFS has any support for partition-pruned secondary index scans? (If not, is it on the roadmap?) I feel like the lack of it in many current big-data systems is a major bottleneck to adoption for many kinds of workloads.
- jamesblonde 9y ago
  Do you mean Hive on Hops, I guess? HopsFS is a filesystem (drop-in replacement for HDFS with distributed metadata). Partition-pruned index scans are key for high performance in distributed databases. Index scans and full-table scans hit all shards and kill performance. Hive supports partition-pruned index scans and push-down filtering for projections.