5 ms·

OpenKeyval.org, a free data store/social experiment

46 points by markchristian 16y ago

Howdy, HN'ers; A few weeks ago, we had an idea for a fun side project: a sorta-Memcache-like data storage system that could be trivially used for building simple web apps. After a few evenings of hacking, @dustball and I are pleased to present OpenKeyval. OpenKeyval is a completely open key-value data store, exposed as a drop-dead simple web service. The goal is to make this a very easy way to persist data in web applications. http://openkeyval.org/ As a data storage system, it's pretty basic: you can store any data you want under a particular key, and then fetch it later. You can also serve it back with a particular MIME type, so it can be used to serve small resources. The keyword small is operative here: the data is limited to 64 KiB, which is plenty of space for storing interesting data, but not really big enough for storing images or any other sort of media. As a social experiment, I think it's sort of fun: there's no authentication of any kind -- just come up with sufficiently unique keys to make it unlikely that anyone else will stumble upon them. Scouts honour that anything you put here is safe in our hands. I'm curious as to what people think this might be useful for, and if you have any feature requests. I'd love to see people try to build something with it -- but go easy on us, since it's just an experiment. :)

markchristian 16y ago
Oh, I forgot to mention -- it's open source, too. http://github.com/shinyplasticbag/openkeyval http://github.com/shinyplasticbag/openkeyval
- digispaghetti 16y ago
  I've already commented on the blog - but I'll say here as well - Looks like a great project, but seriously look at nodejs over PHP since JSON is native to nodejs, and you'll probably get more throughput on it. Also, check out nStore and node-supermarket for existing native key/val stores already done in nodejs
aaronblohowiak 16y ago
Clickable: http://openkeyval.org/ http://openkeyval.org/ If you want to help people make webapps, promote the jsonp store to the top of the documentation and explanation of how it is used (including the fact that you aren't going to be able to store anywhere near 64kib with it -- more like ~2kb). While it is less neat, it seems to be the only viable solution for todays non-IE limitations on cross-site form posting.
bhiggins 16y ago
hey sorry to burst your bubble but this is pointless.
- markchristian 16y ago
  Nothing is pointless! Everything is an adventure. :)
  - heretoo 16y ago
    Exactly. I was trying to think how to build a chat program on this, and establish keys between two chatees for communicating, but it seems impossible to make a public channel/key for registering interest in chatting in the first place, without using an untrusted readwrite channel/key. An append only store might solve it, with a key that is globally known. Some sort of broadcast channel. An append only, with limited size appends, and cleared at regular intervals.
    - markchristian 16y ago
      Good challenge! Here's a start: Say you and I want to chat with each other. We both pick a random key to store our messages in. Let's call those key[markchristian] and key[heretoo]. Each of those has a read-only version -- readonly[markchristian] and readonly[heretoo]. I'll post the key readonly[markchristian] to "markchristian-to-heretoo". You post the key readonly[heretoo] to "heretoo-to-markchristian". Our chat clients can just poll those keys, looking for new messages -- whatever hand-wavy format we want to have. Problems: 1. Requires us to agree to chat ahead of time and know each other's usernames. 2. Nothing to stop a malicious third-party from putting a different readonly key at the "heretoo-to-markchristian" pointer keys.
  - bhiggins 16y ago
    your enthusiasm is infectious in a bad way.
- aaronblohowiak 16y ago
  This post is a great example of being "a hater". You aren't disagreeing about the application of a tool to a problem, you are dismissing all opportunity for anyone to enjoy or get value out of the item in question. Since the creators put effort in to creating this, it at least has value to them. So do the people that have upvoted it. So you're dismissing their opinion without offering any information, which is not very helpful.
thepumpkin1979 16y ago
Interesting, but maybe you can create some sort of namespaces so we don't override each other's keys.
- markchristian 16y ago
  You can create your own namespaces by prefixing your keys. The whole idea of this is a flat, totally open namespace.
thepumpkin1979 16y ago
Share your keys on Twitter: #openkeyval
- markchristian 16y ago
  Nice. :)
naz 16y ago
Nice idea, super simple, and poses some interesting questions. There could be security issues with this given you can serve arbitrary MIME types. XSS won't be a problem because there are no cookies to steal. Maybe you could make a script that self-propagates across the key-space using AJAX.
kh411d 16y ago
hmm...
acme 16y ago
Reminds me of OpenDHT, "a publicly accessible distributed hash table (DHT) service", which has since been taken down: http://www.opendht.org/ http://www.opendht.org/ I do think being able to store information without too much hassle is pretty useful.
- pudquick 16y ago
  ... which in turn reminded me of the Vanish project http://vanish.cs.washington.edu/research.html http://vanish.cs.washington.edu/research.html which leveraged both OpenDHT and Vuze's (aka Azureus) DHT to store data (specifically: redundantly stored encryption keys, with the intent of letting them expire).
wazoox 16y ago
Instead of storing the data in files, this looks like a perfect use case for TokyoCabinet or similar storage engines.
rlpb 16y ago
Have you considered supporting the memcached protocol? It might help with takeup as client libraries already exist. http://github.com/memcached/memcached/blob/master/doc/protocol.txt http://github.com/memcached/memcached/blob/master/doc/protoc...
- markchristian 16y ago
  We have discussed it, and even considered just building an HTTP layer on top of it, but haven't done anything with it yet. As for supporting the memcached protocol, we considered that, too, and at this point, we're not targeting that use case. Personally, I've been planning on using this for entirely client-side JavaScript apps, where a memcache layer wouldn't be useful at all. (That being said, the code is up on github, and patches are welcome. :)
stagas 16y ago
I'm using a similar approach in a node.js module if you'd like to try it. http://github.com/stagas/chaos http://github.com/stagas/chaos stores the data in files using the sha1 hash key like you do, you could easily implement an http api layer on top of it. I get about 2k writes/s and 4k reads/s on a regular hd, and 40k-60k writes/reads per second on a ssd.

46 points by markchristian 16y ago

markchristian 16y ago

Oh, I forgot to mention -- it's open source, too. http://github.com/shinyplasticbag/openkeyval http://github.com/shinyplasticbag/openkeyval

digispaghetti 16y ago
I've already commented on the blog - but I'll say here as well - Looks like a great project, but seriously look at nodejs over PHP since JSON is native to nodejs, and you'll probably get more throughput on it. Also, check out nStore and node-supermarket for existing native key/val stores already done in nodejs

aaronblohowiak 16y ago

Clickable: http://openkeyval.org/ http://openkeyval.org/ If you want to help people make webapps, promote the jsonp store to the top of the documentation and explanation of how it is used (including the fact that you aren't going to be able to store anywhere near 64kib with it -- more like ~2kb). While it is less neat, it seems to be the only viable solution for todays non-IE limitations on cross-site form posting.

bhiggins 16y ago

hey sorry to burst your bubble but this is pointless.

markchristian 16y ago
Nothing is pointless! Everything is an adventure. :)
- heretoo 16y ago
  Exactly. I was trying to think how to build a chat program on this, and establish keys between two chatees for communicating, but it seems impossible to make a public channel/key for registering interest in chatting in the first place, without using an untrusted readwrite channel/key. An append only store might solve it, with a key that is globally known. Some sort of broadcast channel. An append only, with limited size appends, and cleared at regular intervals.
  - markchristian 16y ago
    Good challenge! Here's a start: Say you and I want to chat with each other. We both pick a random key to store our messages in. Let's call those key[markchristian] and key[heretoo]. Each of those has a read-only version -- readonly[markchristian] and readonly[heretoo]. I'll post the key readonly[markchristian] to "markchristian-to-heretoo". You post the key readonly[heretoo] to "heretoo-to-markchristian". Our chat clients can just poll those keys, looking for new messages -- whatever hand-wavy format we want to have. Problems: 1. Requires us to agree to chat ahead of time and know each other's usernames. 2. Nothing to stop a malicious third-party from putting a different readonly key at the "heretoo-to-markchristian" pointer keys.
- bhiggins 16y ago
  your enthusiasm is infectious in a bad way.
aaronblohowiak 16y ago
This post is a great example of being "a hater". You aren't disagreeing about the application of a tool to a problem, you are dismissing all opportunity for anyone to enjoy or get value out of the item in question. Since the creators put effort in to creating this, it at least has value to them. So do the people that have upvoted it. So you're dismissing their opinion without offering any information, which is not very helpful.

thepumpkin1979 16y ago

Interesting, but maybe you can create some sort of namespaces so we don't override each other's keys.

markchristian 16y ago
You can create your own namespaces by prefixing your keys. The whole idea of this is a flat, totally open namespace.

thepumpkin1979 16y ago

Share your keys on Twitter: #openkeyval

markchristian 16y ago
Nice. :)

naz 16y ago

Nice idea, super simple, and poses some interesting questions. There could be security issues with this given you can serve arbitrary MIME types. XSS won't be a problem because there are no cookies to steal. Maybe you could make a script that self-propagates across the key-space using AJAX.

kh411d 16y ago

hmm...

acme 16y ago

Reminds me of OpenDHT, "a publicly accessible distributed hash table (DHT) service", which has since been taken down: http://www.opendht.org/ http://www.opendht.org/ I do think being able to store information without too much hassle is pretty useful.

pudquick 16y ago
... which in turn reminded me of the Vanish project http://vanish.cs.washington.edu/research.html http://vanish.cs.washington.edu/research.html which leveraged both OpenDHT and Vuze's (aka Azureus) DHT to store data (specifically: redundantly stored encryption keys, with the intent of letting them expire).

wazoox 16y ago

Instead of storing the data in files, this looks like a perfect use case for TokyoCabinet or similar storage engines.

rlpb 16y ago

Have you considered supporting the memcached protocol? It might help with takeup as client libraries already exist. http://github.com/memcached/memcached/blob/master/doc/protocol.txt http://github.com/memcached/memcached/blob/master/doc/protoc...

markchristian 16y ago
We have discussed it, and even considered just building an HTTP layer on top of it, but haven't done anything with it yet. As for supporting the memcached protocol, we considered that, too, and at this point, we're not targeting that use case. Personally, I've been planning on using this for entirely client-side JavaScript apps, where a memcache layer wouldn't be useful at all. (That being said, the code is up on github, and patches are welcome. :)

stagas 16y ago

I'm using a similar approach in a node.js module if you'd like to try it. http://github.com/stagas/chaos http://github.com/stagas/chaos stores the data in files using the sha1 hash key like you do, you could easily implement an http api layer on top of it. I get about 2k writes/s and 4k reads/s on a regular hd, and 40k-60k writes/reads per second on a ssd.