4 ms·

TITLE DISCLAIMER: 695k using 20 fairly large machines I was, at first glance, pretty impressed with the 695k TPS title claim. Upon reading the article, it beca

TITLE DISCLAIMER: 695k using 20 fairly large machines I was, at first glance, pretty impressed with the 695k TPS title claim. Upon reading the article, it became apparent that this was based on running 8 8-core EC2 m2.4xlarge instances for node.js in addition to 12 8-core EC2 m2.4xlarge instances for VoltDB. In terms of server architecture, this is quite a large setup. Just for an idea of how big of a machine we're talking for each of these instances: High-Memory Quadruple Extra Large Instance 68.4 GB of memory 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each) 1690 GB of instance storage 64-bit platform I/O Performance: High API name: m2.4xlarge $1.800 an hour running Linux on demand $864.00 a day baseline for 20 of these This isn't meant to discredit the author or work, just to make sure it's apparent that this solution is for serious businesses only with deep pockets and boatloads of traffic. You'd be spending $26,784 for the servers alone over a 31 day period if using them on demand. The results are definitely impressive for the right target audience.

stock_toaster 15y ago
Also near the bottom. > On the database side, I configured VoltDB to run at K=0, > which means the database was running with HA switched off. > Most applications would not run that way in a production > environment, but my goal was to test client-side throughput > and scaling; I wasn’t really focused the database side of > the tests. Testament to the author for mentioning it. It makes me interested in voltdb, whereas before it was unknown to me. But the numbers should indeed be taken with a grain of salt I think.
spullara 15y ago
I'm not sure if I am supposed to be impressed or not. On my macbook pro with 1 redis master and 1 redis slave I can do about 300k updates / second to the master replicated to the slave. Comparable? Who knows? These benchmarks don't mean much to me -- what matters is performance for your application.
- arielweisberg 15y ago
  I'll start by saying I dig Redis. I wish we had a storage engine that offered some of the kinds of flexibility that Redis does. If I were making Volt from scratch I would embed Redis or something with flexible schema and secondary indexes and spend less time working on SQL and more time working on distributed functionality. >These benchmarks don't mean much to me -- what matters is performance for your application. Very true. Think about how you would implement an application like Voter in Redis. Voter is an event counting app that applies business logic server side to do what is basically fraud detection. Let's take a look at what each of these transactions is. https://github.com/VoltDB/voltdb/blob/master/examples/voter/src/voter/procedures/Vote.java https://github.com/VoltDB/voltdb/blob/master/examples/voter/... 3 statements and an entire round trip to validate data is kosher. 1 insert/update statement that is actually updating two different summary views You would need to implement your own locking to get the isolation necessary to implement the business logic and there would be at least two round trips for each vote. You would also have to write your own code to maintain the materialized view used to print the summary. Sure you could implement the validation logic client side, but then you lose the ability to transactionally update said logic and the state it depends on. With the new scripting functionality in Redis you would be able to do it in one round trip but you would still end up writing your own code for the view and rollback. How you will attach a debugger to debug your server side logic is also worth considering. I mistakenly wrote that Redis doesn't cache Lua scripts, but it actually does. Now that you have server side logic you will also probably want to start maintaining replicated state to make some of your transactions single shard transactions. With Redis you are on your own for updating replicated state in a way that stays consistent across the cluster even when restoring different shards. Also, no matter how fast Redis is you may reach a point where you have run out of disk/network IO, or even CPU. Now you are looking at sharding Redis. You are probably going to want to maintain replicated state to make more of your transactions single shard transactions. With Redis you are on your own for updating replicated state in a way that stays consistent across the cluster even when restoring different shards. Don't forget re-sharding and backups which are now your problem. You definitely want to be able to roll back to before that last application update that corrupted your data. Don't forget that you will need to bring the replicas up in the correct state as well. Preferably in parallel with bringing up the master since you are down right now, and it would be nice if it resharded at the same time since you took this opportunity to add capacity. Redis cluster is coming, but it isn't here today and depending on the application you may not want to risk it not being ready when the time comes. You should at least double the number of ops (where an op is roundtrip to the DB) done by Volt if not more. That said, doing half as much work inside the procedure doesn't make it twice as fast and doing twice as much doesn't make it twice as slow. The dominant cost for such a trivial operation is not the operation itself. Redis is much better at this because it is single threaded and written in C. My bet is that the number of instructions and cache misses to accept and execute a command in Redis is much smaller because the execution engine isn't dealing with stored procedures, schema, undo logging, SQL, and code expecting to deal with distributed transactions even if none are in play. >I'm not sure if I am supposed to be impressed or not. If you haven't tl;dr-ed already I'd call it good.
brn5 15y ago
The benchmark was very much about whether you CAN scale if you need it. You'd probably not rent this size hardware at Amazon if you needed it on a daily basis. So $26,784 is probably not realistic. At any rate, as I mention in my longer report on the benchmark [1] the high total translates into very LOW hardware requirements for lesser needs. The number to go by may rather be "10k+ TPS per core for linear scaling ACID transactions." I think that's the number that matters. Linear 10k+/core ACID TPS. [1] http://community.voltdb.com/sites/default/files/NodejsBenchmarkReport_April_2012.pdf http://community.voltdb.com/sites/default/files/NodejsBenchm...

stock_toaster 15y ago

Also near the bottom. > On the database side, I configured VoltDB to run at K=0, > which means the database was running with HA switched off. > Most applications would not run that way in a production > environment, but my goal was to test client-side throughput > and scaling; I wasn’t really focused the database side of > the tests. Testament to the author for mentioning it. It makes me interested in voltdb, whereas before it was unknown to me. But the numbers should indeed be taken with a grain of salt I think.

spullara 15y ago

I'm not sure if I am supposed to be impressed or not. On my macbook pro with 1 redis master and 1 redis slave I can do about 300k updates / second to the master replicated to the slave. Comparable? Who knows? These benchmarks don't mean much to me -- what matters is performance for your application.

arielweisberg 15y ago
I'll start by saying I dig Redis. I wish we had a storage engine that offered some of the kinds of flexibility that Redis does. If I were making Volt from scratch I would embed Redis or something with flexible schema and secondary indexes and spend less time working on SQL and more time working on distributed functionality. >These benchmarks don't mean much to me -- what matters is performance for your application. Very true. Think about how you would implement an application like Voter in Redis. Voter is an event counting app that applies business logic server side to do what is basically fraud detection. Let's take a look at what each of these transactions is. https://github.com/VoltDB/voltdb/blob/master/examples/voter/src/voter/procedures/Vote.java https://github.com/VoltDB/voltdb/blob/master/examples/voter/... 3 statements and an entire round trip to validate data is kosher. 1 insert/update statement that is actually updating two different summary views You would need to implement your own locking to get the isolation necessary to implement the business logic and there would be at least two round trips for each vote. You would also have to write your own code to maintain the materialized view used to print the summary. Sure you could implement the validation logic client side, but then you lose the ability to transactionally update said logic and the state it depends on. With the new scripting functionality in Redis you would be able to do it in one round trip but you would still end up writing your own code for the view and rollback. How you will attach a debugger to debug your server side logic is also worth considering. I mistakenly wrote that Redis doesn't cache Lua scripts, but it actually does. Now that you have server side logic you will also probably want to start maintaining replicated state to make some of your transactions single shard transactions. With Redis you are on your own for updating replicated state in a way that stays consistent across the cluster even when restoring different shards. Also, no matter how fast Redis is you may reach a point where you have run out of disk/network IO, or even CPU. Now you are looking at sharding Redis. You are probably going to want to maintain replicated state to make more of your transactions single shard transactions. With Redis you are on your own for updating replicated state in a way that stays consistent across the cluster even when restoring different shards. Don't forget re-sharding and backups which are now your problem. You definitely want to be able to roll back to before that last application update that corrupted your data. Don't forget that you will need to bring the replicas up in the correct state as well. Preferably in parallel with bringing up the master since you are down right now, and it would be nice if it resharded at the same time since you took this opportunity to add capacity. Redis cluster is coming, but it isn't here today and depending on the application you may not want to risk it not being ready when the time comes. You should at least double the number of ops (where an op is roundtrip to the DB) done by Volt if not more. That said, doing half as much work inside the procedure doesn't make it twice as fast and doing twice as much doesn't make it twice as slow. The dominant cost for such a trivial operation is not the operation itself. Redis is much better at this because it is single threaded and written in C. My bet is that the number of instructions and cache misses to accept and execute a command in Redis is much smaller because the execution engine isn't dealing with stored procedures, schema, undo logging, SQL, and code expecting to deal with distributed transactions even if none are in play. >I'm not sure if I am supposed to be impressed or not. If you haven't tl;dr-ed already I'd call it good.

brn5 15y ago

The benchmark was very much about whether you CAN scale if you need it. You'd probably not rent this size hardware at Amazon if you needed it on a daily basis. So $26,784 is probably not realistic. At any rate, as I mention in my longer report on the benchmark [1] the high total translates into very LOW hardware requirements for lesser needs. The number to go by may rather be "10k+ TPS per core for linear scaling ACID transactions." I think that's the number that matters. Linear 10k+/core ACID TPS. [1] http://community.voltdb.com/sites/default/files/NodejsBenchmarkReport_April_2012.pdf http://community.voltdb.com/sites/default/files/NodejsBenchm...