3 ms·
That's a lot of technologies and issues that you touch on. Your points have validity (apart from heap sizes - several HDFS teams manage 300+GB heaps), but you a
by jamesblonde 7y ago
That's a lot of technologies and issues that you touch on. Your points have validity (apart from heap sizes - several HDFS teams manage 300+GB heaps), but you are missing the fundamental challenge - HDFS' semantics (POSIX-like) and its implications on the concurrency model. You can weaken the semantics of HDFS (remove atomic rename) and scale. If you don't want to do that you have to deal with the single-writer, multiple-reader concurrency semantics for the namespace. Speed up the RPC, separate namespace from blockspace (Ozone - btw, how do they handle global states like safe-mode without consensus between both parts?) - you will still bottleneck on mutations of the namespace.
We provided a multi-writer, multi-reader concurrency solution (see paper) based on cross-partition transactions, normalized schemas, row-level locking, and an application-level protocol for subtree ops. That is the fundamental (hard) problem for scaling hierarchical FS'.