3 ms·
I don't think it's a straight-forward decision and is influenced by a number of things; 1. desired time to market. 2. operating cost. 3. people cost. If you
by natbobc 9y ago
I don't think it's a straight-forward decision and is influenced by a number of things;
1. desired time to market.
2. operating cost.
3. people cost.
If you're dealing at that level of traffic you're not going to go broke unless you lack a decent business model. One of the companies I worked at was spending $100,000+ per month on AWS (more than they should've for the incoming traffic level but they survived for a number of years). A friend was working at another company that was spending $3,000,000+/month. The motivation for using DynamoDB/Datastore over say S3 is that you can do range queries, field sub-selection, offloading of systems administration, some elements of security, etc.
Most traffic patterns have windows and tend to be diurnal in nature. This depends on the number of timezones your service covers. Obvious exception being IoT type time-series. So you might experience peaks of 16k/s but that will be for perhaps a few windows in the day. I also wouldn't be treating DynamoDB as permanent "long-term" storage for granular data. I would be using a sliding window of whatever I consider to be a valuable retention period (e.g. sliding window of 30 days). If you don't you'd need to scale C* as well.
In order to run Cassandra and get streaming aggregation via Spark you need to run "2 regions". One as the primary online region for serving requests and a second region for stream processing using Spark. Minimum recommended memory requirement is 16GB for a C* node. For a C* + Spark node minimum is 32GB. The more the better for both. If you've selected your keys well C* will scale almost linearly with the addition of nodes. Using Netflix 1,000,000 writes/second as a model you could roughly state that dividing nodes (285) by 60 should yield an approximately similar throughput requirement to the articles. So about 5-6 t2.xlarge nodes could potentially work for the FE cluster, and whatever gives you adequate streaming performance for the streaming cluster, as a minimum 3-5 x t2.2xlarge. Let's say ~$2,000/month.
Another thing to consider is that the management and maintenance of a C* cluster doesn't come for free either. So figure you'll need a minimum of 2 sys-admins to administer the cluster. Depending on where you are that'll likely cost the company upwards of $400,000 per year or $33,000k/month once you combine base salary, pension contributions, taxes, etc. You could say 1 sysadmin but then you're going to burn out your sysadmins if 1 guy is always on-call.
So assuming a sliding window of 30 days that's $2,160/day for DynamoDB. With a C* cluster you're looking at a minimum of $1,100/day. More depending on how well it performs for what your system is doing. If you're not using a sliding window you'd also need to scale you're C* cluster as well. In order to configure the C* cluster in a way that provides back-ups, auto-recovery, etc you're probably looking at about 2w-2m depending on experience. Whereas DynamoDB time to market will be probably a 1-2w.