4 ms·

Ditto (http://dropc.am http://dropc.am). Very write heavy load for us, which Riak handles without blinking. Fault tolerant, robust, and easy to administer. E

by grourk 15y ago

Ditto (http://dropc.am http://dropc.am). Very write heavy load for us, which Riak handles without blinking. Fault tolerant, robust, and easy to administer. Every machine is identical, no special "master" nodes or anything like that.

grncdr 15y ago
I've heard this about Riak and I was quite excited to test it out for a new project, but in the limited testing I've done Cassandra and HBase both absolutely smoke Riak in terms of write performance. Not really apples to apples I suppose, but I was really surprised at how slow Riak was when handling many (millions) of small writes. We haven't finished our testing/profiling phase yet, so any hints on how to optimize a large number of small writes (on the order of ~dozen bytes each) would be appreciated.
- jamwt 15y ago
  Without going too much into specifics and picking on individual databases (which I could do, boy do I have the scars...) When you hit a certain traffic level, scalability, latency and robustness become far more important than single-node ops/s. I need to be able to add nodes and repair failed nodes while under load--I need the 99.9% latency mark to stay ~100ms while doing so. I don't really care how many bajillions of ops a second your database can do in some concocted scenario, b/c you're not going to do that many in the real world anyway (trust me, we tried). The disk subsystem is going to give you a few hundred, maybe a few thousand if you're lucky, IOPS, then your latency will spike to hell and your phone will wake you up at night. Maybe in the world where 99% of ops are reads, you will put up impressive numbers, but now you're just showing you are pretty good at using the disk cache. That's a relatively easy problem. The riak guys seem to get all this better than most: http://blog.basho.com/2011/05/11/Lies-Damn-Lies-And-NoSQL/ http://blog.basho.com/2011/05/11/Lies-Damn-Lies-And-NoSQL/ So, to give you a short answer to your direct question: Use SLC SSDs + md + RAID-0. Have at least 5 nodes. Use bitcask, but realize that your keys will need to fit in memory. Also, realize that really small values aren't a great fit for Riak in some ways b/c the overhead per value is at least a few hundred bytes. Also, it's important to note this is where I'm at right now, but maybe not where you (generally) are at. Riak may not make you happy at server #1, but it will make you pretty happy at server 10 and server 100. Riak's sweet spot is people with scaling pains. If you only need a server or two to try some stuff, and you don't have any users yet, you might cause yourself more headaches than you need. Sometimes you don't need a locomotive, you need a motorcycle. (These guys have a pretty great motorcycle: http://rethinkdb.com/ http://rethinkdb.com/ )
  - bretthoerner 15y ago
    Was your experience with Cassandra different? Happy at server 10 and 100, that is?
    - jamwt 15y ago
      TBH, we didn't seriously pursue Cassandra when we considered distributed database systems b/c the vast majority of "back-reference checks" we did on the YC network and other area startups was "stay away." We got some very frank advice from some people whose opinions on databases I take very seriously to stay away, including reports from within FB. Having said that, I cannot claim to have firsthand proven or disproven anything about Cassandra.
      - espeed 15y ago
        Quora's decisions to not use Cassandra and Adam's answer regarding it lead me to the same conclusion (http://www.quora.com/Quora-Infrastructure/Why-does-Quora-use-MySQL-as-the-data-store-instead-of-NoSQLs-such-as-Cassandra-MongoDB-CouchDB-etc http://www.quora.com/Quora-Infrastructure/Why-does-Quora-use...). Evidently few from Facebook are advocating it.
        dehora 15y ago
        Quora on MySQL failed outright when AWS EBS failed, companies on AWS using Cassandra like SimpleGeo and NetFlix did not. To their credit, Facebook were clear enough on their reasons for using HBase over MySQL and Cassandra, such as wanting to double down on their current Hadoop system/knowledge and having easily obtainable ordering guarantees on messages. It's also clear they've invested in making HBase good enough. At large loads and footprints, imvho, Riak, Cassandra and HBase present viable options. But there are some factors to consider that don't seem to get mentioned in the pop tech press - What are you able to operate in production? - What are you able/willing to debug and patch? - What hardware options do you have? - What are your workloads? - Which variable of C.A.P, when you lose it, most damages your business? - Will your company's choices be evaluated in the press? - Does your board/investors have capital tied up in business's that are using something else? - What architecture tradeoffs and styles sit well with you? - What kind of data access and consumption patterns make you money? - Can you pay for help? The right choice is context sensitive, and I'm fairly sure for this class of systems at this point in time, there's no free lunch. That means you have to do the legwork for yourself and make your own choices and commitments; doing what you heard worked for someone else is a cargo cult.

by grourk 15y ago

grncdr 15y ago

I've heard this about Riak and I was quite excited to test it out for a new project, but in the limited testing I've done Cassandra and HBase both absolutely smoke Riak in terms of write performance. Not really apples to apples I suppose, but I was really surprised at how slow Riak was when handling many (millions) of small writes. We haven't finished our testing/profiling phase yet, so any hints on how to optimize a large number of small writes (on the order of ~dozen bytes each) would be appreciated.

jamwt 15y ago
Without going too much into specifics and picking on individual databases (which I could do, boy do I have the scars...) When you hit a certain traffic level, scalability, latency and robustness become far more important than single-node ops/s. I need to be able to add nodes and repair failed nodes while under load--I need the 99.9% latency mark to stay ~100ms while doing so. I don't really care how many bajillions of ops a second your database can do in some concocted scenario, b/c you're not going to do that many in the real world anyway (trust me, we tried). The disk subsystem is going to give you a few hundred, maybe a few thousand if you're lucky, IOPS, then your latency will spike to hell and your phone will wake you up at night. Maybe in the world where 99% of ops are reads, you will put up impressive numbers, but now you're just showing you are pretty good at using the disk cache. That's a relatively easy problem. The riak guys seem to get all this better than most: http://blog.basho.com/2011/05/11/Lies-Damn-Lies-And-NoSQL/ http://blog.basho.com/2011/05/11/Lies-Damn-Lies-And-NoSQL/ So, to give you a short answer to your direct question: Use SLC SSDs + md + RAID-0. Have at least 5 nodes. Use bitcask, but realize that your keys will need to fit in memory. Also, realize that really small values aren't a great fit for Riak in some ways b/c the overhead per value is at least a few hundred bytes. Also, it's important to note this is where I'm at right now, but maybe not where you (generally) are at. Riak may not make you happy at server #1, but it will make you pretty happy at server 10 and server 100. Riak's sweet spot is people with scaling pains. If you only need a server or two to try some stuff, and you don't have any users yet, you might cause yourself more headaches than you need. Sometimes you don't need a locomotive, you need a motorcycle. (These guys have a pretty great motorcycle: http://rethinkdb.com/ http://rethinkdb.com/ )
- bretthoerner 15y ago
  Was your experience with Cassandra different? Happy at server 10 and 100, that is?
  - jamwt 15y ago
    TBH, we didn't seriously pursue Cassandra when we considered distributed database systems b/c the vast majority of "back-reference checks" we did on the YC network and other area startups was "stay away." We got some very frank advice from some people whose opinions on databases I take very seriously to stay away, including reports from within FB. Having said that, I cannot claim to have firsthand proven or disproven anything about Cassandra.
    - espeed 15y ago
      Quora's decisions to not use Cassandra and Adam's answer regarding it lead me to the same conclusion (http://www.quora.com/Quora-Infrastructure/Why-does-Quora-use-MySQL-as-the-data-store-instead-of-NoSQLs-such-as-Cassandra-MongoDB-CouchDB-etc http://www.quora.com/Quora-Infrastructure/Why-does-Quora-use...). Evidently few from Facebook are advocating it.
      - dehora 15y ago
        Quora on MySQL failed outright when AWS EBS failed, companies on AWS using Cassandra like SimpleGeo and NetFlix did not. To their credit, Facebook were clear enough on their reasons for using HBase over MySQL and Cassandra, such as wanting to double down on their current Hadoop system/knowledge and having easily obtainable ordering guarantees on messages. It's also clear they've invested in making HBase good enough. At large loads and footprints, imvho, Riak, Cassandra and HBase present viable options. But there are some factors to consider that don't seem to get mentioned in the pop tech press - What are you able to operate in production? - What are you able/willing to debug and patch? - What hardware options do you have? - What are your workloads? - Which variable of C.A.P, when you lose it, most damages your business? - Will your company's choices be evaluated in the press? - Does your board/investors have capital tied up in business's that are using something else? - What architecture tradeoffs and styles sit well with you? - What kind of data access and consumption patterns make you money? - Can you pay for help? The right choice is context sensitive, and I'm fairly sure for this class of systems at this point in time, there's no free lunch. That means you have to do the legwork for yourself and make your own choices and commitments; doing what you heard worked for someone else is a cargo cult.