3 ms·
Hi! OP here! Just wanted to clear up on a few things! 1. Yes it's VPC we are talking about, and yes there is an added cost of $2 per region per day. Instances
by martincozzi 11y ago
Hi! OP here! Just wanted to clear up on a few things!
1. Yes it's VPC we are talking about, and yes there is an added cost of $2 per region per day. Instances are also slightly more expensive when running in a dedicated VPC. Overall this increased our bill by roughly %20-25. Having a support account also increases the overall cost greatly since it's percentage based of the total bill. However we've been very mindful of the money we spend on AWS and our bill as of Today is still roughly equivalent if not cheaper than the salary of a senior engineer.
2. Regarding efficiency of messages sent vs instances, you can't really divive num of messages processed by instances and get an accurate picture. Every service that we roll out is deployed across 3 AZs. So that means at a minimum 3 instances. Some services don't need the capacity of 3 servers, but redundancy is important to us, and we build everything with that in mind. We also invest a lot in monitoring, logging and metric tracking. Think Riemann, Sensu, Statsd, Logstash (Indexers + Elasticsearch Cluster). I'd say about 1/5 of the infrastructure is dedicated to monitoring/alerting/tracking.
Most days our API backend runs off or 3 instances, our workers out of 3 instances. So if you must calculate it that way, it's closer to 2M / 6. But even that math feels weird. Then we have various services like click tracking, photo resizing, video encoding etc. Also our frontends are decoupled from the backend so that's another 4 to 5 services on top of my mind ( so another 12-15 instances). And then there is our data pipeline, crons, our distribution service (apt/gems) and a few other things that you need to run in a VPC like bastion hosts, NAT instances etc. Those all add up and contribute to the service but don't process messages.
What's nice about running most of those behind AutoScalingGroups is that it's very easy to adjust and "scale down" on the hardware to keep the costs down when not necessary. We found that c3.larges work well for most Rails related stuff. Frontends can run just fine on m1.smalls and scale horizontally when needed etc.
Hope this was helpful! Let me know if you have more questions I'd be happy to answer them!
- rubiquity 11y agoNice work and thank you for sharing this. I'm currently working on a migration plan to AWS for an app that needs to be HIPAA compliant and stories like yours are useful.