2 ms·
If you need to minimize the amount of copies then yes, you need to have some "risk management" software to estimate which machines are more reliable, and which
by EGreg 12y ago
If you need to minimize the amount of copies then yes, you need to have some "risk management" software to estimate which machines are more reliable, and which files are more important, and then assign those files to enough replicas to be able to statistically guarantee some SLA. Then you need failover where at least one if the replicas is always available.
The routing table should be small enough to fit in RAM on every machine, and consulted for request. It would be updated when failover occurs. The table would consist of general rules with temporary exceptions for specific partition ranges that are being failed over.
You can store indexes in files, in a similar way. Just avoid joins and make like a graph database: first load documents from the index and then do mapreduce to get the related documents.
But besides that, I can see how maybe multi user concurrent access might necessitate eventual consistency algorithms for each app, but that's it.