2 ms·
a GFS clone was the angle I was looking at. Cassandra looks pretty fun - are you suggesting that as the database right? Im thinking a quick python implementati
by ErrantX 17y ago
a GFS clone was the angle I was looking at.
Cassandra looks pretty fun - are you suggesting that as the database right? Im thinking a quick python implementation for PUT (maybe DELETE) and meta operations, using cassandra as a backend and Lighttpd for the GET (high performance) might work..... cheers.
- simonw 17y agoIf you want a GFS clone, you /really/ want HDFS from the Hadoop project - I'm reading the new O'Reilly book on Hadoop on the moment which is excellent.
- Tichy 17y agoThat is probably what I meant, too - I read that one of those OS projects had a clone of the GFS, only better (supposedly). Must have been Hadoop, as I haven't looked at many others.
- skorgu 17y agoNo first hand experience here but my understanding is that HDFS and Hadoop in general are geared towards high-throughput and not necessarily low latency. You'll likely have to put a caching layer in front of it if this is still [1] the case. There is already some [2] HTTP accessibility for HDFS, I'm again not 100% clear on its status but if it works well enough a nice caching proxy (varnish) with plenty of memory should help with latency on heavily-accessed files. [1] http://www.nabble.com/Using-HDFS-to-serve-www-requests-td22725659.html http://www.nabble.com/Using-HDFS-to-serve-www-requests-td227... [2] https://issues.apache.org/jira/browse/HADOOP-5010 https://issues.apache.org/jira/browse/HADOOP-5010