7 ms·

Riak 1.0

166 points by franze 15y ago

geoffhill 15y ago
LevelDB support! Would love to see how it compares to Bitcask in terms of speed.
- Ixiaus 15y ago
  I imagine it won't be as fast? The cool thing about Bitcask is that all of the keys are in memory - I imagine that would also be beneficial with secondary indexes now supported... LevelDB seems mostly well suited for data that becomes (in terms of key size and number of keys) bigger than your RAM...
  - dsl 15y ago
    I think it will be a welcome change for anyone who runs a decent sized Riak deployment. We are currently adding machines simply to increase available RAM in the cluster.
    - nirvana 15y ago
      Why not bring a node down, and then replace it with a node that has more RAM? Are you exceeding the size of a node you can supply (in terms of RAM) for your cluster? I'd be very curious to know a bit about the character of your data, the size of your cluster, etc. (I've only run test clusters at this point, so hearing from someone doing production work would be informative.)
      - dsl 15y ago
        Replacement vs. addition is a situational trade-off, but ultimately the problem remains that you need to bring more RAM to the party. My biggest RAM consumer stores historical data for a goods trading platform. Each trade is a unique key, with all the trade data being the value. Access speed is important, but not as critical as the other goodies I get from Riak (replication and automated rebalancing). Metadata is stored separately, but I hope to change that with Riak 1.0 secondary indexes.
  - varworld 15y ago
    Secondary indexes are currently only supported on levelDB.
    - Ixiaus 15y ago
      Interesting - well, that would get me to choose leveldb then!
- willbmoss 15y ago
  Bitcask can guarantee one disk seek, whereas LevelDB will do one disk seek per level, so at least from that perspective, it can't be better. Level also has to look down the entire tree if a key is missing. This means inserts end up being more expensive than reads or updates (which are all just a hash lookup in Bitcask).
  - nosequel 15y ago
    LevelDB is there as the replacement for those who are currently using Innostore as their backend and not for those who have a dataset that fits bitcask.
  - fizx 15y ago
    "Bitcask can guarantee one disk seek, whereas LevelDB will do one disk seek per level, so at least from that perspective, it can't be better." Yep, this is a standard tradeoff. When you want your data to be iterable, you have to take the hit. In practice (I oversee a large cassandra cluster), this hit happens about ~1% of the time, which is either a lot, or a little, depending on your constraints. "Level also has to look down the entire tree if a key is missing." This is why Cassandra has a bloom filter on top of a very similar data store.
tsuraan 15y ago
Notice that this is a bit of a pre-announcement: Riak 1.0 will be available later this month. To preview some of the new features, download Riak, or to inquire about a commercial deployment, please visit http://www.basho.com. Their github page still just has 1.0-rc1 tagged. I'm excited though.
superjared 15y ago
"Riak 1.0 will be available later this month." I can't stand pre-announcements like this.
- timf 15y ago
  Further, there are only three days left.. The HN title should be changed away from "Riak 1.0".
- pjscott 15y ago
  This is actually an announcement of RC1 being released: http://downloads.basho.com/riak/riak-1.0.0rc1/ http://downloads.basho.com/riak/riak-1.0.0rc1/ I just did a rolling upgrade from 1.0b4, and it went smoothly. I love the LevelDB support, and it's holding up well under the considerable load that I'm throwing at it.
- nosequel 15y ago
  Riak 1.0.0 RC2 is out now, and "later this month" is only 3 more days.
- nirvana 15y ago
  I think when you've got your release candidate, you're basically done, barring any unexpected surprises. Given stable Riak has been during the beta period, I don't think they're really jumping the gun much.
jibs 15y ago
A bit tangential to this particular announcement - but i've been musing about using Riak, though so far put off by their (seemingly) open-core, rather than open-source implementation. Are the paid, enterprise functions stuff you eventually need in most use cases? the lack of multi-site replication in particular is curious; would this mean I can replicate between nodes on the cluster, as long as they are in the same datacenter, but not across the interwebs until i hand over some $$$?
- skeltoac 15y ago
  Enterprise includes a ring replication layer designed for higher-latency connections. There is nothing preventing you setting up a cluster that spans continents. What will deter you is the poor performance of the cluster due to the added latency between nodes.
  - jibs 15y ago
    Interesting - from what I remember of the the original Amazon Dynamo paper it seems the ring replication is pretty central to the thing (if we are both talking about the ring replication used for the distribution of keys across the nodes). This is sounding like crippleware :(
    - skeltoac 15y ago
      Replication of keys around the ring is free. What you pay for is their solution to the problem of significant latency between nodes: code that replicates the whole ring in several sites and coordinates the communication between the sites.
    - rzezeski 15y ago
      Replicas (or as you put it, ring replication) is critical, and Riak very much has replicas. What it doesn't have in it's open source version is multi-ring replication (cross DC) which is a separate concern. In the Dynamo paper the ring spans DCs but they also have a very different network than most that allows them to do that. In Riak it is recommended that each ring is contained in a single DC. If you want ring-to-ring replication from Basho then you can pay for Riak EDS. You could also build it yourself as others have mentioned (Kresten Krab Thorup has done something like this in Riak Mobile [1]). Nothing is stopping you from running a single ring (cluster) across DCs, and it might even be okay for certain apps, but it's not a choice that should be taken lightly. In general, if you don't understand the tradeoffs you're making in that regard then it's best to stick to one ring, one DC. [1]: http://www.erlang-factory.com/upload/presentations/413/ErlangFactory2011_KrestenKrabThorup_RiakSync.pdf http://www.erlang-factory.com/upload/presentations/413/Erlan...
vdm 15y ago
> Interested in Riak Enterprise for your company? Contact Us Now I don't have a company. Yet. What I am interested in is a go-away button on this obnoxious ad bar so I can read your webpages on my vertically-challenged 11" screen.
lwat 15y ago
Is anyone here paying for the Enterprise level Riak? I'd love to hear how they charge and whether you think it's worth it. Currently we're looking towards the Denali release later this year but Riak is looking more interesting by the day.
latch 15y ago
I haven't used Riak, but I did look into it for a project short while back. One problem I had was that the documentation on their website is heavily focused on what Riak is, vs how to use it. It's great that you can get such a fundamental understanding of Riak as-a-dynamo-implementation, and they do a great job writing that stuff, but its completely out of touch with what I expected/needed. Technically, what eventually put me off, is that I couldn't figure out how to maintain a clean secondary index. If you have a: SiteId, UserId, Data, and you want data to be accessible by SiteId or SiteId+UserId, I couldn't figure out a nice atomic way to maintain the secondary index. This is pretty basic stuff. I'm glad to see 1.0 will support native secondary indexes, but I think my inability to figure it out shows that their documentation is poor (or it could be that I suck).
- bgentry 15y ago
  The secondary index stuff is actually very new, so it's entirely possible that it didn't exist when you last looked at Riak. More info here http://blog.basho.com/2011/09/14/Secondary-Indexes-in-Riak/ http://blog.basho.com/2011/09/14/Secondary-Indexes-in-Riak/
socratic 15y ago
I am increasingly interested in Riak, in part because a very vocal minority on HN seems to think it is the One True NoSQL solution. However, I still don't quite get it. What is Riak? It seems to be some sort of Dynamo implementation (like the ironically ill fated Cassandra), but apparently it has a workflow engine? What do people use it for? What is it best at? Right now we're using PostgreSQL, Redis, and S3. PostgreSQL gives us ACID, Redis gives us fast in-memory access, and S3 gives us an infinite KV store. Is there some reason to use Riak? Would Riak just replace S3?
- amock 15y ago
  If your values are very small Riak is probably a good replacement for S3. If your values are large then S3 is probably better than Riak.
  - siculars 15y ago
    I'm not so certain. A value written to riak has something on the order of 450bytes of overhead (as of version 0.14, not entirely certain of the exact overhead in 1.0). Basically riak will write your value to disk with a bunch of other data that it uses internally to do its thing. Writing a stream of integers, one integer per riak key, would be a bad idea (tm), imho.
- kennystone 15y ago
  Ultimately it is a eventually consistent key-value store. They have map-reduce and a new meta-data thing that lets you query documents, but you really only want to use it if your data maps well to key/value (and lots of data does!). The really cool things are that it is truly horizontal scale - a new node brings read+write+map-reduce power, and that it will help you sleep better with its dynamo-ism.
- dschobel 15y ago
  Why do you say that Cassandra is ill-fated?
  - socratic 15y ago

Riak 1.0

166 points by franze 15y ago

geoffhill 15y ago

LevelDB support! Would love to see how it compares to Bitcask in terms of speed.

Ixiaus 15y ago
I imagine it won't be as fast? The cool thing about Bitcask is that all of the keys are in memory - I imagine that would also be beneficial with secondary indexes now supported... LevelDB seems mostly well suited for data that becomes (in terms of key size and number of keys) bigger than your RAM...
- dsl 15y ago
  I think it will be a welcome change for anyone who runs a decent sized Riak deployment. We are currently adding machines simply to increase available RAM in the cluster.
  - nirvana 15y ago
    Why not bring a node down, and then replace it with a node that has more RAM? Are you exceeding the size of a node you can supply (in terms of RAM) for your cluster? I'd be very curious to know a bit about the character of your data, the size of your cluster, etc. (I've only run test clusters at this point, so hearing from someone doing production work would be informative.)
    - dsl 15y ago
      Replacement vs. addition is a situational trade-off, but ultimately the problem remains that you need to bring more RAM to the party. My biggest RAM consumer stores historical data for a goods trading platform. Each trade is a unique key, with all the trade data being the value. Access speed is important, but not as critical as the other goodies I get from Riak (replication and automated rebalancing). Metadata is stored separately, but I hope to change that with Riak 1.0 secondary indexes.
- varworld 15y ago
  Secondary indexes are currently only supported on levelDB.
  - Ixiaus 15y ago
    Interesting - well, that would get me to choose leveldb then!
willbmoss 15y ago
Bitcask can guarantee one disk seek, whereas LevelDB will do one disk seek per level, so at least from that perspective, it can't be better. Level also has to look down the entire tree if a key is missing. This means inserts end up being more expensive than reads or updates (which are all just a hash lookup in Bitcask).
- nosequel 15y ago
  LevelDB is there as the replacement for those who are currently using Innostore as their backend and not for those who have a dataset that fits bitcask.
- fizx 15y ago
  "Bitcask can guarantee one disk seek, whereas LevelDB will do one disk seek per level, so at least from that perspective, it can't be better." Yep, this is a standard tradeoff. When you want your data to be iterable, you have to take the hit. In practice (I oversee a large cassandra cluster), this hit happens about ~1% of the time, which is either a lot, or a little, depending on your constraints. "Level also has to look down the entire tree if a key is missing." This is why Cassandra has a bloom filter on top of a very similar data store.

tsuraan 15y ago

Notice that this is a bit of a pre-announcement: Riak 1.0 will be available later this month. To preview some of the new features, download Riak, or to inquire about a commercial deployment, please visit http://www.basho.com. Their github page still just has 1.0-rc1 tagged. I'm excited though.

superjared 15y ago

"Riak 1.0 will be available later this month." I can't stand pre-announcements like this.

timf 15y ago
Further, there are only three days left.. The HN title should be changed away from "Riak 1.0".
pjscott 15y ago
This is actually an announcement of RC1 being released: http://downloads.basho.com/riak/riak-1.0.0rc1/ http://downloads.basho.com/riak/riak-1.0.0rc1/ I just did a rolling upgrade from 1.0b4, and it went smoothly. I love the LevelDB support, and it's holding up well under the considerable load that I'm throwing at it.
nosequel 15y ago
Riak 1.0.0 RC2 is out now, and "later this month" is only 3 more days.
nirvana 15y ago
I think when you've got your release candidate, you're basically done, barring any unexpected surprises. Given stable Riak has been during the beta period, I don't think they're really jumping the gun much.

jibs 15y ago

A bit tangential to this particular announcement - but i've been musing about using Riak, though so far put off by their (seemingly) open-core, rather than open-source implementation. Are the paid, enterprise functions stuff you eventually need in most use cases? the lack of multi-site replication in particular is curious; would this mean I can replicate between nodes on the cluster, as long as they are in the same datacenter, but not across the interwebs until i hand over some $$$?

skeltoac 15y ago
Enterprise includes a ring replication layer designed for higher-latency connections. There is nothing preventing you setting up a cluster that spans continents. What will deter you is the poor performance of the cluster due to the added latency between nodes.
- jibs 15y ago
  Interesting - from what I remember of the the original Amazon Dynamo paper it seems the ring replication is pretty central to the thing (if we are both talking about the ring replication used for the distribution of keys across the nodes). This is sounding like crippleware :(
  - skeltoac 15y ago
    Replication of keys around the ring is free. What you pay for is their solution to the problem of significant latency between nodes: code that replicates the whole ring in several sites and coordinates the communication between the sites.
  - rzezeski 15y ago
    Replicas (or as you put it, ring replication) is critical, and Riak very much has replicas. What it doesn't have in it's open source version is multi-ring replication (cross DC) which is a separate concern. In the Dynamo paper the ring spans DCs but they also have a very different network than most that allows them to do that. In Riak it is recommended that each ring is contained in a single DC. If you want ring-to-ring replication from Basho then you can pay for Riak EDS. You could also build it yourself as others have mentioned (Kresten Krab Thorup has done something like this in Riak Mobile [1]). Nothing is stopping you from running a single ring (cluster) across DCs, and it might even be okay for certain apps, but it's not a choice that should be taken lightly. In general, if you don't understand the tradeoffs you're making in that regard then it's best to stick to one ring, one DC. [1]: http://www.erlang-factory.com/upload/presentations/413/ErlangFactory2011_KrestenKrabThorup_RiakSync.pdf http://www.erlang-factory.com/upload/presentations/413/Erlan...

vdm 15y ago

> Interested in Riak Enterprise for your company? Contact Us Now I don't have a company. Yet. What I am interested in is a go-away button on this obnoxious ad bar so I can read your webpages on my vertically-challenged 11" screen.

lwat 15y ago

Is anyone here paying for the Enterprise level Riak? I'd love to hear how they charge and whether you think it's worth it. Currently we're looking towards the Denali release later this year but Riak is looking more interesting by the day.

latch 15y ago

I haven't used Riak, but I did look into it for a project short while back. One problem I had was that the documentation on their website is heavily focused on what Riak is, vs how to use it. It's great that you can get such a fundamental understanding of Riak as-a-dynamo-implementation, and they do a great job writing that stuff, but its completely out of touch with what I expected/needed. Technically, what eventually put me off, is that I couldn't figure out how to maintain a clean secondary index. If you have a: SiteId, UserId, Data, and you want data to be accessible by SiteId or SiteId+UserId, I couldn't figure out a nice atomic way to maintain the secondary index. This is pretty basic stuff. I'm glad to see 1.0 will support native secondary indexes, but I think my inability to figure it out shows that their documentation is poor (or it could be that I suck).

bgentry 15y ago
The secondary index stuff is actually very new, so it's entirely possible that it didn't exist when you last looked at Riak. More info here http://blog.basho.com/2011/09/14/Secondary-Indexes-in-Riak/ http://blog.basho.com/2011/09/14/Secondary-Indexes-in-Riak/

socratic 15y ago

I am increasingly interested in Riak, in part because a very vocal minority on HN seems to think it is the One True NoSQL solution. However, I still don't quite get it. What is Riak? It seems to be some sort of Dynamo implementation (like the ironically ill fated Cassandra), but apparently it has a workflow engine? What do people use it for? What is it best at? Right now we're using PostgreSQL, Redis, and S3. PostgreSQL gives us ACID, Redis gives us fast in-memory access, and S3 gives us an infinite KV store. Is there some reason to use Riak? Would Riak just replace S3?

amock 15y ago
If your values are very small Riak is probably a good replacement for S3. If your values are large then S3 is probably better than Riak.
- siculars 15y ago
  I'm not so certain. A value written to riak has something on the order of 450bytes of overhead (as of version 0.14, not entirely certain of the exact overhead in 1.0). Basically riak will write your value to disk with a bunch of other data that it uses internally to do its thing. Writing a stream of integers, one integer per riak key, would be a bad idea (tm), imho.
kennystone 15y ago
Ultimately it is a eventually consistent key-value store. They have map-reduce and a new meta-data thing that lets you query documents, but you really only want to use it if your data maps well to key/value (and lots of data does!). The really cool things are that it is truly horizontal scale - a new node brings read+write+map-reduce power, and that it will help you sleep better with its dynamo-ism.
dschobel 15y ago
Why do you say that Cassandra is ill-fated?
- socratic 15y ago