4 ms·
Anyone know if it has the same single point of failure issues as HDFS (NameNode)? Also, bummer that it's written in Java. I know that was for easy pluggability
by cgh 14y ago
Anyone know if it has the same single point of failure issues as HDFS (NameNode)?
Also, bummer that it's written in Java. I know that was for easy pluggability into Hadoop, but it makes it tough to write bindings for other languages.
- agibsonccc 14y agoYou may look in to: http://hadoop.apache.org/docs/r1.1.2/streaming.html http://hadoop.apache.org/docs/r1.1.2/streaming.html for that. Hadoop's HTTPFS (http://hadoop.apache.org/docs/r2.0.3-alpha/hadoop-hdfs-httpfs/index.html http://hadoop.apache.org/docs/r2.0.3-alpha/hadoop-hdfs-httpf...) maybe. If this is a dropin replacement, a REST wrapper around the API to just take advantage of it with client bindings in random libraries could work as well. Edit: https://github.com/amplab/tachyon/blob/master/src/main/java/tachyon/UnderFileSystemHdfs.java https://github.com/amplab/tachyon/blob/master/src/main/java/... It appears to just be a thin wrapper around HDFS, so the namenode and what not probably applies here.
- jey 14y agoActually, any language with a Thrift binding can talk to Tachyon: https://github.com/amplab/tachyon/blob/master/src/thrift/tachyon.thrift https://github.com/amplab/tachyon/blob/master/src/thrift/tac...
- agibsonccc 14y agoThere it is. I didn't dig much in to that.
- dchichkov 14y agoI guess using non-native API would introduce significant overhead and kill all the advantages of using this Tachyon filesystem. I take it, the resulting DFS can't be mounted, am I right? If yes, that's a pity. By the way, are there DFS with similar properties (very fast, aggressive memory use) that can be mounted?
- jey 14y agoWhy not? As far as I can tell, it'd be feasible to write a FUSE adapter that uses Thrift to talk to the Tachyon servers.
- codewright 14y agoFUSE can be really ugly sometimes though, kernel modules are better if you need any real reliability.
- rubinelli 14y agoHadoop 2.x lets you run a standby namenode with automatic failover.
- eclark 14y agoAdditionally the newish QJM will allow those namenodes to write to a quorum of servers.
- felixr 14y ago> Anyone know if it has the same single point of failure issues as HDFS (NameNode)? I think the NameNode SPOF issue in HDFS is also being dealt with: "Prior to Hadoop 2.0.0, the NameNode was a single point of failure (SPOF) in an HDFS cluster. [...] The HDFS High Availability feature addresses the above problems by providing the option of running two redundant NameNodes [...]" http://hadoop.apache.org/docs/r2.0.3-alpha/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html#Background http://hadoop.apache.org/docs/r2.0.3-alpha/hadoop-yarn/hadoo...