3 ms·
HDFS supports RS/XOR erasure coding which gives you same fault tolerance guarantees as 3x at much lower replication factor. This is essentially the same method
by dilyevsky 5y ago
HDFS supports RS/XOR erasure coding which gives you same fault tolerance guarantees as 3x at much lower replication factor. This is essentially the same method aws/gcp use under the hood - there’s no magic involved here
- genewitch 5y agoCan I use this over NFS or samba or is it just for Java and Hadoop?
- dilyevsky 5y agoYes you can* for new projects I would go with ceph+nfsv4 *) https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/ha...
- genewitch 5y agoceph+nfs was actually what i was going to do. Probably some learning pains but at least i don't have to "lose" half my storage because the sizes aren't paired with anything.