3 ms·
"But you don’t have to install Hadoop at all, unless you decide to use HDFS as Spark’s distributed file system. If you don’t, you can choose one of the other DF
by capkutay 12y ago
"But you don’t have to install Hadoop at all, unless you decide to use HDFS as Spark’s distributed file system. If you don’t, you can choose one of the other DFS’s that it supports"
What other file systems does spark support? And how seamless is that integration? I know with spark, I can write a Spark SQL query that will go directly against data persisted in HDFS. Should I expect the same behavior if I swap out HDFS with some other DFS?
- tupshin 12y agoSpark doesn't depend on much of anything from the Hadoop ecosystem, including HDFS. It supports S3, as well as NFS or any other locally mountable filesystem right out of the box. The FAQ talks a bunch about Spark's weak ties to Hadoop. http://spark.apache.org/faq.html http://spark.apache.org/faq.html