4 ms·
For those who might not know how this speedup is possible: Traditional databases use data structures like b-tree indexes. These are awesome! They look up dat
by mdasen 18y ago
For those who might not know how this speedup is possible:
Traditional databases use data structures like b-tree indexes. These are awesome! They look up data in log(n) time (n being the size of your dataset). That means that as your data grows, the time to execute a single query doesn't grow so much. So, what's the problem with databases? Well, the problem is that as your data set grows, you're likely getting more queries as well. So, you end up with m * log(n) where m is the number of queries that users are making. Boo!
What about throwing hardware at it? I hear that's all the rage! The problem with that is that it's really, really difficult to split data across more than a handful of machines. Replication is good for a few machines, but eventually you hit a wall with writes since writes must happen on every server in the cluster.
Or do they? There's a technique called sharding that you may have heard of. Basically, it's putting some articles on server 1 and some articles on server 2. Then, when you need article 450, you need to figure out what server it's on and get it. This is what MemcacheDB (and memcached) do. They hash the key you give it and that tells them what server it will be on without having to actually ask each server if they have it. That's a constant time process so it scales infinitely (unlike those hosting companies that claim they scale infinitely). All you have to do is keep throwing boxes at it as you grow.
Awesome! That's so much better than MySQL! Well, no. I mean, it is in some ways, but you can't say something like "get me the latest article that someone has commented on". That might be useful to your site, might not. All you can do is lookup by the key.
It gets hard when you want to do something such as getting the comments for an article. You can't say, get me the comments for article 450. You have to actually store a list of the comment ids in the article, get the article, and then request the comments based on that list in the article object. Worse, lets say two people comment on an article at the same time. You'd better make sure you start a transaction before reading the object, modify the list, and save it. If you don't, you'll hit race conditions that can see a comment's association lost (something you don't have to worry about with foreign-key associations).
Plus, beyond that trouble, you still can't say "get me the article with the most comments".
To an extent, I'm wondering how this is much better than memcached. If you still need/want an RDBMS for query capabilities, do you really need another key-store that saves to disk? memcached can always be reconstituted from the RDBMS if a server fails, but I guess MemcacheDB forgoes that warm-up process (similar to the Shop.com Cache) and that can be a bonus. However, it's a lot slower than memcached due to the disk access.
It looks really interesting, but I'm not sure how useful it would be (I should say, in my line of work). I won't give up queries and their site says emphatically that it's not a cache. If you're doing something that needs a simple key-value store, this is going to be an amazing product.
- bd 18y agoIt looks similar to Google App Engine Datastore [1] or Amazon SimpleDB [2], so I guess it could have similar use cases. If you already architect your application to use these cloud services, it should be easier to switch to/from MemcacheDB on your own servers. [1] http://code.google.com/appengine/docs/datastore http://code.google.com/appengine/docs/datastore [2] http://aws.amazon.com/simpledb http://aws.amazon.com/simpledb
- mdasen 18y agoIt isn't similar. With App Engine and SimpleDB, you can query on any field in the model. With MemcacheDB you can only query on the key. So, with App Engine and SimpleDB you can say "give me all the people records where their age is > 18". You can't do that with MemcacheDB. With MemcacheDB, you can only say, get me the records with the IDs x, y, and z. What App Engine and SimpleDB eliminate is joins. MemcacheDB would be more scalable than either App Engine or SimpleDB, but it doesn't allow you to do much. That's what makes memcached a good cache. You have a strict limitation in how you can get data, but it scales. SimpleDB and App Engine are better than most RDBMSs in terms of scalability, but they still use index lookups when you query them. To put it in SQL terms: App Engine/SimpleDB: "SELECT * FROM people where age > 18"; "SELECT * from articles where comment_count > 10"; "SELECT * FROM articles where author_id = 28" These are all valid. Most queries that don't involve joins are valid. MemcacheDB: "SELECT * FROM people where id = 8"; "SELECT * FROM people where id IN (8, 16, 27)" Those are the only types of queries that you can do with MemcacheDB. None of the queries above that work with App Engine and SimpleDB work with MemcacheDB.
- bd 18y agoThanks for the clarification. MemcacheDB does indeed look like App Engine Datastore/SimpleDB sans extra query machinery. Which is still interesting, as this is anyway their most efficient use pattern (in terms of resource consumption, not ease of development). Attribute-based queries on App Engine can hit restrictive CPU quotas quite easily, while key-based queries are very fast.
- 18y ago