7 ms·

How Cotap Is Building a HIPAA-Compliant Messaging Service on AWS

32 points by sergiotapia 11y ago

fasteo 11y ago
>>> process about 2 million messages each month... >>> ... We run about 60 instances in production Seems like a very low efficiency in either the hardware or the software. We run a comparable service with roughly 2 million messages per day in a 6 server dedicated cluster.
- Loic 11y ago
  60 servers for less than one message per second. I suppose they do something important and CPU intensive with each message.
- dantiberian 11y ago
  Agreed, that stuck out to me too 2,000,000 / 30 / 24 / 60 / 60 = 0.7 requests/second/cluster. However it's probably worth extending them the benefit of the doubt, there may be other factors that they didn't cover.
  - fasteo 11y ago
    Yes, that's also my feeling. They mention ElasticSearch; this alone can consume several servers but, again, storing 2 million/messages per month plus associated data in ElasticSearch does not explain that many production instances.
  - count 11y ago
    Is your message handling a linear deal over the month? They may process 1.5M of those in the last 8 hours of the day, which would be 3750 requests a second instead...
    - thwarted 11y ago
      I'm not sure how you arrived at this 3750 per second number. Processing 1.5 million messages for ⅓ of the time (1500000 messages / 30d / 8h / 60m / 60s) is an average of 1.7361 messages per second during those 8 hours per day. If the remaining 500k messages were spread evenly over the remaining 16 hours a day for a month, that's an average of 0.963 messages per second ((((500000 / 30 / 8 / 60 / 60 )×2)+1.7361)/3) over the entire month. Even if you were to process 1.5million messages during the last 8 hours of the month, that is only 52.08 ( 1500000 messages / 8h / 60m / 60s ) messages per second during that 8 hour period.
      - count 11y ago
        I, uh, don't know how I got that either (must have fat fingered in calc). I was talking about all in one 8 hour deal though. Thanks for the clarification.
- ianlevesque 11y ago
  From the article: > Our main API is backed by Rails (messages, authentication and authorization). There are a lot of reasons to run rails, but pure efficiency per request is not one of them. I'm sure they could replace their bottlenecks later as needed.
comrade1 11y ago
They have to use Amazon's VPC service - AWS has a 'secure' vpc service where you don't share the server with other customers which is a requirement for HIPAA compliance. That service gets really pricey. I run a similar service but for SMS messaging that is 21cfr11 compliant. We rent a 1/4 rack with our own door/lock and throw in pretty much disposable servers as needed. It's much cheaper than doing the equivalent on AWS, and it's not that difficult to set up your own secure servers once you have a few recipes to follow. We started it about 9 years ago before a secure service with amazon existed. Every year or two I price alternative solutions like AWS's secure service, VPCs, etc and it's always cheaper even taking account my labor to stick with our 1/4 closet. In fact, it seems they base their costs on the going rate for rack space, plus a premium for using the convenience of using their servers. Oh, and it's funny - we have almost the same number of users and we're processing more messages per month than them, all on a old-fashioned webserver-appserver-dataserver arrangement - three servers (with fallover redunandancy not counted) and it's barely breaking a sweat.
- count 11y ago
  You dramatically misunderstand VPC. Everybody on AWS today uses VPC, unless you happened to be grandfathered in to EC2 'classic'. There is NO COST to using VPC (aside from admin over head maybe?). It's a huge security upgrade, as it gives you the ability to isolate things much better. The specifics of HIPAA/HITECH compliance on AWS are available under their BAA (which is confidential), and do require some things that cost more money. VPC is not one of them. If you have only a quarter rack of servers and want the minimum s for physical security, yeah, you can probably make that work cheaper than AWS. When you have a whole datacenter worth of gear and now need a more sophisticated security / entry system, etc. the argument becomes much closer. With that said, you misunderstanding how VPC works and what it is makes me wonder if your pricing is coming out right: it's really complicated to price out a full AWS cost, just as much as to price out your own costs.
  - krallin 11y ago
    GP is most likely referring to VPCs with dedicated tenancy. Those are not free, and they're pricey [0]. [0]: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/dedicated-instance.html
martincozzi 11y ago
Hi! OP here! Just wanted to clear up on a few things! 1. Yes it's VPC we are talking about, and yes there is an added cost of $2 per region per day. Instances are also slightly more expensive when running in a dedicated VPC. Overall this increased our bill by roughly %20-25. Having a support account also increases the overall cost greatly since it's percentage based of the total bill. However we've been very mindful of the money we spend on AWS and our bill as of Today is still roughly equivalent if not cheaper than the salary of a senior engineer. 2. Regarding efficiency of messages sent vs instances, you can't really divive num of messages processed by instances and get an accurate picture. Every service that we roll out is deployed across 3 AZs. So that means at a minimum 3 instances. Some services don't need the capacity of 3 servers, but redundancy is important to us, and we build everything with that in mind. We also invest a lot in monitoring, logging and metric tracking. Think Riemann, Sensu, Statsd, Logstash (Indexers + Elasticsearch Cluster). I'd say about 1/5 of the infrastructure is dedicated to monitoring/alerting/tracking. Most days our API backend runs off or 3 instances, our workers out of 3 instances. So if you must calculate it that way, it's closer to 2M / 6. But even that math feels weird. Then we have various services like click tracking, photo resizing, video encoding etc. Also our frontends are decoupled from the backend so that's another 4 to 5 services on top of my mind ( so another 12-15 instances). And then there is our data pipeline, crons, our distribution service (apt/gems) and a few other things that you need to run in a VPC like bastion hosts, NAT instances etc. Those all add up and contribute to the service but don't process messages. What's nice about running most of those behind AutoScalingGroups is that it's very easy to adjust and "scale down" on the hardware to keep the costs down when not necessary. We found that c3.larges work well for most Rails related stuff. Frontends can run just fine on m1.smalls and scale horizontally when needed etc. Hope this was helpful! Let me know if you have more questions I'd be happy to answer them!
- rubiquity 11y ago
  Nice work and thank you for sharing this. I'm currently working on a migration plan to AWS for an app that needs to be HIPAA compliant and stories like yours are useful.

32 points by sergiotapia 11y ago

>>> process about 2 million messages each month... >>> ... We run about 60 instances in production Seems like a very low efficiency in either the hardware or the software. We run a comparable service with roughly 2 million messages per day in a 6 server dedicated cluster.

Loic 11y ago
60 servers for less than one message per second. I suppose they do something important and CPU intensive with each message.
dantiberian 11y ago
Agreed, that stuck out to me too 2,000,000 / 30 / 24 / 60 / 60 = 0.7 requests/second/cluster. However it's probably worth extending them the benefit of the doubt, there may be other factors that they didn't cover.
- fasteo 11y ago
  Yes, that's also my feeling. They mention ElasticSearch; this alone can consume several servers but, again, storing 2 million/messages per month plus associated data in ElasticSearch does not explain that many production instances.
- count 11y ago
  Is your message handling a linear deal over the month? They may process 1.5M of those in the last 8 hours of the day, which would be 3750 requests a second instead...
  - thwarted 11y ago
    I'm not sure how you arrived at this 3750 per second number. Processing 1.5 million messages for ⅓ of the time (1500000 messages / 30d / 8h / 60m / 60s) is an average of 1.7361 messages per second during those 8 hours per day. If the remaining 500k messages were spread evenly over the remaining 16 hours a day for a month, that's an average of 0.963 messages per second ((((500000 / 30 / 8 / 60 / 60 )×2)+1.7361)/3) over the entire month. Even if you were to process 1.5million messages during the last 8 hours of the month, that is only 52.08 ( 1500000 messages / 8h / 60m / 60s ) messages per second during that 8 hour period.
    - count 11y ago
      I, uh, don't know how I got that either (must have fat fingered in calc). I was talking about all in one 8 hour deal though. Thanks for the clarification.
ianlevesque 11y ago
From the article: > Our main API is backed by Rails (messages, authentication and authorization). There are a lot of reasons to run rails, but pure efficiency per request is not one of them. I'm sure they could replace their bottlenecks later as needed.

comrade1 11y ago

They have to use Amazon's VPC service - AWS has a 'secure' vpc service where you don't share the server with other customers which is a requirement for HIPAA compliance. That service gets really pricey. I run a similar service but for SMS messaging that is 21cfr11 compliant. We rent a 1/4 rack with our own door/lock and throw in pretty much disposable servers as needed. It's much cheaper than doing the equivalent on AWS, and it's not that difficult to set up your own secure servers once you have a few recipes to follow. We started it about 9 years ago before a secure service with amazon existed. Every year or two I price alternative solutions like AWS's secure service, VPCs, etc and it's always cheaper even taking account my labor to stick with our 1/4 closet. In fact, it seems they base their costs on the going rate for rack space, plus a premium for using the convenience of using their servers. Oh, and it's funny - we have almost the same number of users and we're processing more messages per month than them, all on a old-fashioned webserver-appserver-dataserver arrangement - three servers (with fallover redunandancy not counted) and it's barely breaking a sweat.

count 11y ago
You dramatically misunderstand VPC. Everybody on AWS today uses VPC, unless you happened to be grandfathered in to EC2 'classic'. There is NO COST to using VPC (aside from admin over head maybe?). It's a huge security upgrade, as it gives you the ability to isolate things much better. The specifics of HIPAA/HITECH compliance on AWS are available under their BAA (which is confidential), and do require some things that cost more money. VPC is not one of them. If you have only a quarter rack of servers and want the minimum s for physical security, yeah, you can probably make that work cheaper than AWS. When you have a whole datacenter worth of gear and now need a more sophisticated security / entry system, etc. the argument becomes much closer. With that said, you misunderstanding how VPC works and what it is makes me wonder if your pricing is coming out right: it's really complicated to price out a full AWS cost, just as much as to price out your own costs.
- krallin 11y ago
  GP is most likely referring to VPCs with dedicated tenancy. Those are not free, and they're pricey [0]. [0]: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/dedicated-instance.html

martincozzi 11y ago

Hi! OP here! Just wanted to clear up on a few things! 1. Yes it's VPC we are talking about, and yes there is an added cost of $2 per region per day. Instances are also slightly more expensive when running in a dedicated VPC. Overall this increased our bill by roughly %20-25. Having a support account also increases the overall cost greatly since it's percentage based of the total bill. However we've been very mindful of the money we spend on AWS and our bill as of Today is still roughly equivalent if not cheaper than the salary of a senior engineer. 2. Regarding efficiency of messages sent vs instances, you can't really divive num of messages processed by instances and get an accurate picture. Every service that we roll out is deployed across 3 AZs. So that means at a minimum 3 instances. Some services don't need the capacity of 3 servers, but redundancy is important to us, and we build everything with that in mind. We also invest a lot in monitoring, logging and metric tracking. Think Riemann, Sensu, Statsd, Logstash (Indexers + Elasticsearch Cluster). I'd say about 1/5 of the infrastructure is dedicated to monitoring/alerting/tracking. Most days our API backend runs off or 3 instances, our workers out of 3 instances. So if you must calculate it that way, it's closer to 2M / 6. But even that math feels weird. Then we have various services like click tracking, photo resizing, video encoding etc. Also our frontends are decoupled from the backend so that's another 4 to 5 services on top of my mind ( so another 12-15 instances). And then there is our data pipeline, crons, our distribution service (apt/gems) and a few other things that you need to run in a VPC like bastion hosts, NAT instances etc. Those all add up and contribute to the service but don't process messages. What's nice about running most of those behind AutoScalingGroups is that it's very easy to adjust and "scale down" on the hardware to keep the costs down when not necessary. We found that c3.larges work well for most Rails related stuff. Frontends can run just fine on m1.smalls and scale horizontally when needed etc. Hope this was helpful! Let me know if you have more questions I'd be happy to answer them!

rubiquity 11y ago
Nice work and thank you for sharing this. I'm currently working on a migration plan to AWS for an app that needs to be HIPAA compliant and stories like yours are useful.