3 ms·
I was running a website that was doing millions of writes a day to SDB for real time analytics. The biggest PITA feature of SDB is that it throttles writes in a
by pierrefar 17y ago
I was running a website that was doing millions of writes a day to SDB for real time analytics. The biggest PITA feature of SDB is that it throttles writes in a very horrible stringent way. You can barely tickle a domain and it will throttle you. I never had any consistently good batch puts - they all eventually fail.
After talking with SDB folks, they recommended that I shard my data because each domain maps to a different network computer cluster. I'm glad it's the first recommendation in the OP list because it seriously is the best thing you can do.
Another trick that I experimented with: use multiple EC2 instance to write to the same domain. I managed to convince myself that the throttling is per EC2 instance per domain, not a global per domain. However, cost ruled this solution out.
Reading was much more consistent but was also throttled, especially at high write-loads. The solution was two-fold:
1. Cache everything "indefinitely" and break the cache when you know its contents will change. For the real time stuff, you can't cache. I used memcached, and looked at other solutions like Tokyo Tyrant, memcachedb and redis. Use what you feel comfortable using really.
2. Read as little as possible. Doing a "select * from domain where..." is horrible compared to doing "select attribute1, attribute2 from domain where...". Once you read, cache.