7 ms·

This seems to be a pretty good implementation of FriendFeed's schema-less MySQL ( http://bret.appspot.com/entry/how-friendfeed-uses-mysql http://bret.appspot.co

by amix 17y ago

This seems to be a pretty good implementation of FriendFeed's schema-less MySQL ( http://bret.appspot.com/entry/how-friendfeed-uses-mysql http://bret.appspot.com/entry/how-friendfeed-uses-mysql ) and this option would probably beat most of the simple key-value stores in terms of perfomance, backup and replication. One should note thought that more modern NoSQL databases aren't only key-value stores. For example, Redis offers KEYS command which can scan the store by a pattern. Tokyo Tyrant (another "key-value" store) offers Lua scripting.

viraptor 17y ago
> this option would probably beat most of the simple key-value stores in terms of performance, backup and replication. I wonder why do you think so? Here's a sample test of plain and memcached mysql vs tokyo tyrant (http://www.mysqlperformanceblog.com/2009/10/19/mysql_memcached_tyrant_part3/ http://www.mysqlperformanceblog.com/2009/10/19/mysql_memcach...) It was done without the additional inserts for the index table... I don't see how it can be anywhere close to the TT's performance for example. While mysql's replication is very easy, TT's replication is trivial. Backup is probably comparable - just flush and snapshot... Unless you're comparing it to mysql+MC. Then yes - mysql's performance at 99.8% hits to MC can be ignored. We're left comparing MC with something else. Things might change dramatically if you have a write-heavy application. Or when you have more data than ram. Author's benchmarks will always show great results - let's wait for some real-world data.
- amix 17y ago
  Most != all :) Tokyo Tyrant and Redis are some of the fastest key-value stores and when I stated that it wasn't with them in mind. There are tons of other key-value stores that can't beat the perfomance of MySQL. I agree that replication in Tokyo Tyrant is generally good, but MySQL's is much more mature and one can get help more easily. It should be stated that I am the author of LightCloud (a distributed db built on top of TT) and I am running big installations of both TT and MySQL, so I speak from experience.
- jamesgolick 17y ago
  My data is from a real-world application. One that gets plenty of traffic. Friendly has served almost 50 million hits since we put it in to production two weeks ago. It actually happens to be quite a write-intensive workload. We have definitely seen a slow down on writes, but the speed up on reads has more than compensated for it. We do have more data than RAM. But, most of the data that actually gets read is fresh, so it's hot in cache. Of course Friendly isn't going to be right for everybody. But, I just want to make it clear that my data isn't from synthetic benchmarks. It's from production.
  - viraptor 17y ago
    I didn't mean to imply that those benchmarks are synthetic. What I had in mind is that when you write some software, you will improve it to solve your problem better than other software (why would you do it otherwise ;) ). I meant real-world as in -- from various sources and applications -- just so that we can see how useful it is and in what situations it's better / worse.
    - jamesgolick 17y ago
      Totally fair.
uggedal 17y ago
Tokyo Tyrant's Lua scripting of the db server has been great when developing http://wasitup.com http://wasitup.com - The following example would be hard to implement in an efficient manner with a traditional key/value store without scripting capabilities: -- Return keys with a given prefix for a particular node where there are a -- known total of nodes. Uses the reamainder of the keys' hash values to -- segment the keys. function slice(prefix, index_and_total) local args = _split(index_and_total, "\t") if #args ~= 2 then return nil end local index = tonumber(args[1]) local total = tonumber(args[2]) local res = {} local keys = _fwmkeys(prefix) for i = 1, #keys do local key = keys[i] local url = string.sub( key, string.find(key, ":", string.find(key, ":", 1, true) + 1, true) + 1 ) local modulo = tonumber(string.sub(_hash("md5", url), 1, 10), 16) % total if modulo == index then table.insert(res, key) end end if #res == 0 then return nil end return table.concat(res, "\t") end

by amix 17y ago

viraptor 17y ago

> this option would probably beat most of the simple key-value stores in terms of performance, backup and replication. I wonder why do you think so? Here's a sample test of plain and memcached mysql vs tokyo tyrant (http://www.mysqlperformanceblog.com/2009/10/19/mysql_memcached_tyrant_part3/ http://www.mysqlperformanceblog.com/2009/10/19/mysql_memcach...) It was done without the additional inserts for the index table... I don't see how it can be anywhere close to the TT's performance for example. While mysql's replication is very easy, TT's replication is trivial. Backup is probably comparable - just flush and snapshot... Unless you're comparing it to mysql+MC. Then yes - mysql's performance at 99.8% hits to MC can be ignored. We're left comparing MC with something else. Things might change dramatically if you have a write-heavy application. Or when you have more data than ram. Author's benchmarks will always show great results - let's wait for some real-world data.

amix 17y ago
Most != all :) Tokyo Tyrant and Redis are some of the fastest key-value stores and when I stated that it wasn't with them in mind. There are tons of other key-value stores that can't beat the perfomance of MySQL. I agree that replication in Tokyo Tyrant is generally good, but MySQL's is much more mature and one can get help more easily. It should be stated that I am the author of LightCloud (a distributed db built on top of TT) and I am running big installations of both TT and MySQL, so I speak from experience.
jamesgolick 17y ago
My data is from a real-world application. One that gets plenty of traffic. Friendly has served almost 50 million hits since we put it in to production two weeks ago. It actually happens to be quite a write-intensive workload. We have definitely seen a slow down on writes, but the speed up on reads has more than compensated for it. We do have more data than RAM. But, most of the data that actually gets read is fresh, so it's hot in cache. Of course Friendly isn't going to be right for everybody. But, I just want to make it clear that my data isn't from synthetic benchmarks. It's from production.
- viraptor 17y ago
  I didn't mean to imply that those benchmarks are synthetic. What I had in mind is that when you write some software, you will improve it to solve your problem better than other software (why would you do it otherwise ;) ). I meant real-world as in -- from various sources and applications -- just so that we can see how useful it is and in what situations it's better / worse.
  - jamesgolick 17y ago
    Totally fair.

uggedal 17y ago

Tokyo Tyrant's Lua scripting of the db server has been great when developing http://wasitup.com http://wasitup.com - The following example would be hard to implement in an efficient manner with a traditional key/value store without scripting capabilities: -- Return keys with a given prefix for a particular node where there are a -- known total of nodes. Uses the reamainder of the keys' hash values to -- segment the keys. function slice(prefix, index_and_total) local args = _split(index_and_total, "\t") if #args ~= 2 then return nil end local index = tonumber(args[1]) local total = tonumber(args[2]) local res = {} local keys = _fwmkeys(prefix) for i = 1, #keys do local key = keys[i] local url = string.sub( key, string.find(key, ":", string.find(key, ":", 1, true) + 1, true) + 1 ) local modulo = tonumber(string.sub(_hash("md5", url), 1, 10), 16) % total if modulo == index then table.insert(res, key) end end if #res == 0 then return nil end return table.concat(res, "\t") end