3 ms·
I did the math a while back, don't have the notes at the moment, but scaling an AWS system I built enough to collect 600m points of data each minute and compute
by harlanji 9y ago
I did the math a while back, don't have the notes at the moment, but scaling an AWS system I built enough to collect 600m points of data each minute and compute on data within 100ms and retain it for a few minutes would run a bit over $10k usd/mo to operate. I operated it at about 3m events/min with a good amount of compute per including ip to geo lookup... Zookeeper would be the only bottleneck in this case assuming good enough partitioning.
- tinix 9y agoUsing AWS is the problem here, and that's why it's so expensive. You could do this on bare metal WAY faster and more efficiently, and then you own the hardware forever, for the price you paid to do it for a month with a third party. AWS does not scale this way, you can't just throw more resources at a problem and expect to be profitable.
- user5994461 9y agoPut everything in Google BigQuery or Google DataProc. Cheaper than both and hardly any maintenance required.
- harlanji 9y agoAgreed it could be done cheaper over long term, just wanted to share about an actual prod system. This also had 3x replication via Kafka to avoid stampedes etc if anything failed and keep going with an at-least-once guarantee. In my opinion tho, even that price point is pretty accessible to keep tabs on all citizens with that resolution which was my hypothetical case.
- greenleafjacob 9y agoYou would own the hardware until it died which is not forever.
- tinix 9y agoJust because it quits working doesn't mean you don't still own it. Might wanna lay off the green leaf, bro. Hahaha