4 ms·

I like Cassandra and I think it's something you should look at hard if what you want is an eventually consistent database because multi-DC and availability are

I like Cassandra and I think it's something you should look at hard if what you want is an eventually consistent database because multi-DC and availability are important to the problem you are solving. The blog post fails to make that case anywhere. I am guessing that is because it is an ad for GCE and not Cassandra. I know these kinds of benchmarks are necessary to get and keep your widget on the map, but they always annoy me because I know N other ways to arrive at the same or better numbers with a different set of tradeoffs. Demonstrating scale out is not great for differentiation because it says nothing about the total cost or complexity of use. Why is it that only GCE + Cassandra could have done a million of these particular kind of writes in a cost effective way? I also feel like they are fudging how elasticity works. It's not instant. Bringing up a cluster of a given size is not the same as adding node at time to a running cluster to arrive at that size and actually receiving full benefit from the additional capacity. From the wording it sounds like the entire cluster was brought up at once. Overall my beef is that the blog post fails to inform the reader and is mediocre as a benchmark of anything other than GCE. And let's not even start with the complete lack of random reads, does GCE offer SSD instances yet?

ivansmf 13y ago
I ran those tests. One of the challenges in benchmarking is to pick the workload, so I chose one our customers do more often. For IO testing I prefer to run FIO tests, and for networking iperf, but unless you do this for a living it is hard to relate to microbenchmarks. It would also leave out the Java settings. The reason I left the tests running for an hour is because at some point Cassandra gets into compactions, which are CPU and read intensive. The engine memory maps the files, so the IO subsystem backing the storage sees a lot of random IOs as page faults kick in. Streaming IOs get in during fsyncs. Again, easier to see using FIO. The cluster was brought up at once, and I added the ramp up time on the chart so folks could see it. I had three goals for this test: - Show low latency is possible with remote storage and proper capacity planning. This is why I used quorum commit, which forces the client to wait for at least two nodes to commit. - Share the settings I used. If you open the GIST and download the tarball you'll find all changes I did to the cassandra.yaml and cassandra-env.sh. This benefits our customers directly because it gives them a starting point. - Recommend that customers look at all samples, not only the middle 80% the stress tool reports. Otherwise the cluster looks much better than it is. Thank you for the comments! I can assure you I read them, and I'll incorporate suggestions as possible.
- ivansmf 13y ago
  Here is the GIST link: https://gist.github.com/ivansmf/6ec2197b69d1b7b26153 https://gist.github.com/ivansmf/6ec2197b69d1b7b26153
- arielweisberg 13y ago
  Thanks I missed the link to the tarball in the GIST. Now that I can see the workload I see that it is 100 million keys for the entire cluster of 330 nodes? That is 8 terabytes (n1-standard-8) of RAM for 18.6 * RF gigabytes of data or am I wrong? The entire thing should fit in the page cache assuming compaction keeps up. There are no deletes so no tombstones. On an overwrite workload I don't know what the space amplification will be and whether you will actually run out of RAM for caching. 90% of people who read a benchmark aren't going to look at it the way I do. They don't have a cost model that says how fast things should be and how much they should cost. I do and when things are fitting in memory I have a different a different set of expectations. If I am looking at the wrong instance type please let me know. For the workload you described Cassandra shouldn't be doing random IO. I would expect there to be three + N streams of sequential IO. The write ahead log, memtable flushing, compaction output, and N streams reading tables for compaction. All the write IO can be deferred and rescheduled heavily because fsyncs are infrequent. Reads for compaction are done by background tasks and shouldn't effect foreground latency. If read ahead is not working for compaction (and killing disk throughput) that may be something that needs to be addressed. Compaction should be requesting IOs large enough to amortize the cost of seeking. Page faults in memory mapped files don't stop read ahead and I think that the kernel will even detect sequential access and double read ahead. For a workload like this with no random reads you could configure the kernel to read ahead 2-4 megabytes and the page cache would probably absorb it fine. For the workload you described the only fsyncs would be the log (every 10 seconds) and memtable flushes/compaction finishing and that would normally be so infrequent as to not move the needle on overall IO capacity (although obviously it consumes sequential IO). You set trickle fsync so we are still talking about 10 megabyte writes. Granted there are many things I don't know about Cassandra. I've plumbed a lot of it, but it isn't my day job. Using a 24 gigabyte heap and a 600 megabyte young generation is questionable to me. I think Cassandra can do better, and I also think there are several tools that would do the same job with 10-20x less nodes by exploiting the fact that they never have to do random reads from disk.

ivansmf 13y ago

I ran those tests. One of the challenges in benchmarking is to pick the workload, so I chose one our customers do more often. For IO testing I prefer to run FIO tests, and for networking iperf, but unless you do this for a living it is hard to relate to microbenchmarks. It would also leave out the Java settings. The reason I left the tests running for an hour is because at some point Cassandra gets into compactions, which are CPU and read intensive. The engine memory maps the files, so the IO subsystem backing the storage sees a lot of random IOs as page faults kick in. Streaming IOs get in during fsyncs. Again, easier to see using FIO. The cluster was brought up at once, and I added the ramp up time on the chart so folks could see it. I had three goals for this test: - Show low latency is possible with remote storage and proper capacity planning. This is why I used quorum commit, which forces the client to wait for at least two nodes to commit. - Share the settings I used. If you open the GIST and download the tarball you'll find all changes I did to the cassandra.yaml and cassandra-env.sh. This benefits our customers directly because it gives them a starting point. - Recommend that customers look at all samples, not only the middle 80% the stress tool reports. Otherwise the cluster looks much better than it is. Thank you for the comments! I can assure you I read them, and I'll incorporate suggestions as possible.

ivansmf 13y ago
Here is the GIST link: https://gist.github.com/ivansmf/6ec2197b69d1b7b26153 https://gist.github.com/ivansmf/6ec2197b69d1b7b26153
arielweisberg 13y ago
Thanks I missed the link to the tarball in the GIST. Now that I can see the workload I see that it is 100 million keys for the entire cluster of 330 nodes? That is 8 terabytes (n1-standard-8) of RAM for 18.6 * RF gigabytes of data or am I wrong? The entire thing should fit in the page cache assuming compaction keeps up. There are no deletes so no tombstones. On an overwrite workload I don't know what the space amplification will be and whether you will actually run out of RAM for caching. 90% of people who read a benchmark aren't going to look at it the way I do. They don't have a cost model that says how fast things should be and how much they should cost. I do and when things are fitting in memory I have a different a different set of expectations. If I am looking at the wrong instance type please let me know. For the workload you described Cassandra shouldn't be doing random IO. I would expect there to be three + N streams of sequential IO. The write ahead log, memtable flushing, compaction output, and N streams reading tables for compaction. All the write IO can be deferred and rescheduled heavily because fsyncs are infrequent. Reads for compaction are done by background tasks and shouldn't effect foreground latency. If read ahead is not working for compaction (and killing disk throughput) that may be something that needs to be addressed. Compaction should be requesting IOs large enough to amortize the cost of seeking. Page faults in memory mapped files don't stop read ahead and I think that the kernel will even detect sequential access and double read ahead. For a workload like this with no random reads you could configure the kernel to read ahead 2-4 megabytes and the page cache would probably absorb it fine. For the workload you described the only fsyncs would be the log (every 10 seconds) and memtable flushes/compaction finishing and that would normally be so infrequent as to not move the needle on overall IO capacity (although obviously it consumes sequential IO). You set trickle fsync so we are still talking about 10 megabyte writes. Granted there are many things I don't know about Cassandra. I've plumbed a lot of it, but it isn't my day job. Using a 24 gigabyte heap and a 600 megabyte young generation is questionable to me. I think Cassandra can do better, and I also think there are several tools that would do the same job with 10-20x less nodes by exploiting the fact that they never have to do random reads from disk.