5 ms·
Ask HN: Distributed Filesystems other than HDFS?
I'm interested in doing some analysis over a large set of data where the performance would benefit from distribution across multiple CPUs and machines, namely to use a large bucket of RAM.
We are a Python shop, and less interested in Python on top of layers like Spark - projects like Dask are more interesting. HDFS is of course an option, but as we don't need any other Hadoop components, there is overhead that takes up some of that needed RAM.
My first lab was built on MapR-FS, as it is NFS mountable, does not require namenodes, and looks like a regular filesystem to any non-Hadoop API. However, balancing available cache for the filesystem and for apps is a bit of a challenge - there is still a lot of JVM threads involved.
Are there any similar distributed filesystems that are "lightweight", transparent to apps, easy to manage, and POSIX? Or is this yet to be invented?