12 ms·

you need to think in terms of reads/sec vs latency, not just latency in other words 1) what is the throughput (load) you expect your data store to sustain in r

by helwr 16y ago

you need to think in terms of reads/sec vs latency, not just latency in other words 1) what is the throughput (load) you expect your data store to sustain in reads/sec 2) what is the acceptable latency in msec for your application under expected load (see 1) and 3) how many simultaneous connections your db has to support to figure out which data store suits you best in terms of capacity/performance you have two choices: 1) run a benchmark on your hardware, ideally with your own client 2) run a benchmark on your hardware, ideally with your own client i'd recommend starting with this: http://wiki.github.com/brianfrankcooper/YCSB/ http://wiki.github.com/brianfrankcooper/YCSB/ dont forget that the tools you mentioned are intended for storing terabytes of data on hundreds/thousands of machines and their key benefit is scaling the bulk throughput rather than optimizing performance check out MongoDB, Redis, Tokyo Tyrant if you have up to 10-20GB, these are superfast tools but limited to storage/IO/memory capacity of a single box also look at some of these benchmarks (but running your own test is always better) http://research.yahoo.com/files/ycsb.pdf http://research.yahoo.com/files/ycsb.pdf http://www.slideshare.net/schubertzhang/hbase-0200-performance-evaluation http://www.slideshare.net/schubertzhang/hbase-0200-performan... http://code.google.com/p/redis/wiki/Benchmarks http://code.google.com/p/redis/wiki/Benchmarks http://capttofu.livejournal.com/23381.html http://capttofu.livejournal.com/23381.html http://colinhowe.wordpress.com/2009/04/27/redis-vs-mysql/ http://colinhowe.wordpress.com/2009/04/27/redis-vs-mysql/

ghotli 16y ago
I've benchmarked Tokyo Tyrant, MongoDB, and Redis for a smaller dataset (~20gb). We ended up going with Tyrant because it stored our serialized data more efficiently than the others. That index lives in memory on a big ec2 instance. The data I'm talking about will easily be around half a terabyte or more and I need efficient random access within it. At least initially. I assume that in the medium term this amount of data will grow quite a bit larger. I really do just want to use something like Tokyo Tyrant, but the data management overhead of sharding the datasets across enough high memory ec2 machines to get acceptable performance is prohibitive to going that route. I've kind of been driven to the dynamo / bigtable approach if only because it can store amounts of data in these sizes and can easily be ramped up by adding more servers. That said, I'm still in the analysis stage of where our architecture needs to go next. It very well may be cost prohibitive to go with one of the technologies I mentioned. Thank you for the benchmark tools and guidelines. They're immediately applicable to the analysis I'm performing this week.

by helwr 16y ago

ghotli 16y ago

I've benchmarked Tokyo Tyrant, MongoDB, and Redis for a smaller dataset (~20gb). We ended up going with Tyrant because it stored our serialized data more efficiently than the others. That index lives in memory on a big ec2 instance. The data I'm talking about will easily be around half a terabyte or more and I need efficient random access within it. At least initially. I assume that in the medium term this amount of data will grow quite a bit larger. I really do just want to use something like Tokyo Tyrant, but the data management overhead of sharding the datasets across enough high memory ec2 machines to get acceptable performance is prohibitive to going that route. I've kind of been driven to the dynamo / bigtable approach if only because it can store amounts of data in these sizes and can easily be ramped up by adding more servers. That said, I'm still in the analysis stage of where our architecture needs to go next. It very well may be cost prohibitive to go with one of the technologies I mentioned. Thank you for the benchmark tools and guidelines. They're immediately applicable to the analysis I'm performing this week.