3 ms·
MemcacheDB: A distributed key-value storage system designed for persistence
- geuis 18y agoDoes this support a REST command interface?
- mdasen 18y agoNo. It supports the memcached interface. Most languages have libraries for this as memcached is decently popular.
- chops 18y agoThis seems very cool. I could see this going a step further to merge it with being an actual cache, keeping much of the frequently used data in memory, while still saving the data so that it's persistent. I'm for sure going to tinker around with this. MemcacheQ (message queuing system) linked on that page seems like it will solve a problem I had been thinking about recently for my system. On first glance, this looks like some pretty useful stuff in the overall.
- schtono 18y agoMemcacheQ seems especially interesting! Anyways, the limitation of 64k-1 of body length is a bit frustrating...
- mdasen 18y agoFor those who might not know how this speedup is possible: Traditional databases use data structures like b-tree indexes. These are awesome! They look up data in log(n) time (n being the size of your dataset). That means that as your data grows, the time to execute a single query doesn't grow so much. So, what's the problem with databases? Well, the problem is that as your data set grows, you're likely getting more queries as well. So, you end up with m * log(n) where m is the number of queries that users are making. Boo! What about throwing hardware at it? I hear that's all the rage! The problem with that is that it's really, really difficult to split data across more than a handful of machines. Replication is good for a few machines, but eventually you hit a wall with writes since writes must happen on every server in the cluster. Or do they? There's a technique called sharding that you may have heard of. Basically, it's putting some articles on server 1 and some articles on server 2. Then, when you need article 450, you need to figure out what server it's on and get it. This is what MemcacheDB (and memcached) do. They hash the key you give it and that tells them what server it will be on without having to actually ask each server if they have it. That's a constant time process so it scales infinitely (unlike those hosting companies that claim they scale infinitely). All you have to do is keep throwing boxes at it as you grow. Awesome! That's so much better than MySQL! Well, no. I mean, it is in some ways, but you can't say something like "get me the latest article that someone has commented on". That might be useful to your site, might not. All you can do is lookup by the key. It gets hard when you want to do something such as getting the comments for an article. You can't say, get me the comments for article 450. You have to actually store a list of the comment ids in the article, get the article, and then request the comments based on that list in the article object. Worse, lets say two people comment on an article at the same time. You'd better make sure you start a transaction before reading the object, modify the list, and save it. If you don't, you'll hit race conditions that can see a comment's association lost (something you don't have to worry about with foreign-key associations). Plus, beyond that trouble, you still can't say "get me the article with the most comments". To an extent, I'm wondering how this is much better than memcached. If you still need/want an RDBMS for query capabilities, do you really need another key-store that saves to disk? memcached can always be reconstituted from the RDBMS if a server fails, but I guess MemcacheDB forgoes that warm-up process (similar to the Shop.com Cache) and that can be a bonus. However, it's a lot slower than memcached due to the disk access. It looks really interesting, but I'm not sure how useful it would be (I should say, in my line of work). I won't give up queries and their site says emphatically that it's not a cache. If you're doing something that needs a simple key-value store, this is going to be an amazing product.
- bd 18y agoIt looks similar to Google App Engine Datastore [1] or Amazon SimpleDB [2], so I guess it could have similar use cases. If you already architect your application to use these cloud services, it should be easier to switch to/from MemcacheDB on your own servers. [1] http://code.google.com/appengine/docs/datastore http://code.google.com/appengine/docs/datastore [2] http://aws.amazon.com/simpledb http://aws.amazon.com/simpledb
- mdasen 18y agoIt isn't similar. With App Engine and SimpleDB, you can query on any field in the model. With MemcacheDB you can only query on the key. So, with App Engine and SimpleDB you can say "give me all the people records where their age is > 18". You can't do that with MemcacheDB. With MemcacheDB, you can only say, get me the records with the IDs x, y, and z. What App Engine and SimpleDB eliminate is joins. MemcacheDB would be more scalable than either App Engine or SimpleDB, but it doesn't allow you to do much. That's what makes memcached a good cache. You have a strict limitation in how you can get data, but it scales. SimpleDB and App Engine are better than most RDBMSs in terms of scalability, but they still use index lookups when you query them. To put it in SQL terms: App Engine/SimpleDB: "SELECT * FROM people where age > 18"; "SELECT * from articles where comment_count > 10"; "SELECT * FROM articles where author_id = 28" These are all valid. Most queries that don't involve joins are valid. MemcacheDB: "SELECT * FROM people where id = 8"; "SELECT * FROM people where id IN (8, 16, 27)" Those are the only types of queries that you can do with MemcacheDB. None of the queries above that work with App Engine and SimpleDB work with MemcacheDB.
- bd 18y agoThanks for the clarification. MemcacheDB does indeed look like App Engine Datastore/SimpleDB sans extra query machinery. Which is still interesting, as this is anyway their most efficient use pattern (in terms of resource consumption, not ease of development). Attribute-based queries on App Engine can hit restrictive CPU quotas quite easily, while key-based queries are very fast.
- strlen 18y agoI've used an earlier version of this - and unless things have changed since Spring 2008 - this isn't usable with a high volume of traffic: I benchmarked it by having multiple fork()'d processes load data from a "partitioned" MySQL database - at one point, memcachedb just gave up and quit reading from the UNIX domain socket. A possible alternative is Tokyo Cabinet: Tokyo Cabinet is a DBM implementation (BDB - on which memcachedb is based - is another) that memory maps the entire hash bucket. Tokyo Cabinet also has an RPC layer built on top of it which specifically has a memcached compatibility mode. I haven't yet tried, but I'd be interested to see what results people have with it: http://tokyocabinet.sourceforge.net/index.html http://tokyocabinet.sourceforge.net/index.html http://tokyocabinet.sourceforge.net/tyrantdoc/ http://tokyocabinet.sourceforge.net/tyrantdoc/
- ezmobius 18y agoI've played with this and I like the idea a lot. but I found that BDB sucks still and gets corrupted too easily. So Instead o've been using tokyo tyrant. It supports memcached protocol as well as HTTP and native client protocols. http://tokyocabinet.sourceforge.net/tyrantdoc/ http://tokyocabinet.sourceforge.net/tyrantdoc/ Much better then memcachedb and serves the same purpose.
- chadr 18y agoDo you have any more info on the corruption? I have found bdb to be very reliable. I used it as the backend storage engine for Sun's LDAP server product.
- moonpolysoft 18y agoI've been working a lot on the performance aspects of Dynomite and I think it is a better solution now than memcachedb. One big issue with memcachedb is that it leaves replication management completely to the user. So if you want to be able to seamlessly handle machine failure with a high traffic site, you need to do a lot of work to set up replicas, do clever things with DNS, and even then you won't get a guarantee of availability for that data. Dynomite handles all of that transparently, plus it isn't tied to a single storage engine. So that you can use a different storage engine based on the type of load you plan on generating. The code is here: http://github.com/cliffmoon/dynomite http://github.com/cliffmoon/dynomite