9 ms·

A Billion User Load Test on Healthcare.gov

127 points by shashashasha 10y ago

kasparsklavins 10y ago
> The current observed peak load for HealthCare.gov’s Open Enrollment in 2016 was on Dec 14 2016 Dec 14? Did I miss something?
- ProAm 10y ago
  Probably Dec 14 2015 for the 2016 enrollment year
deleted 10y ago
[deleted]
jakupovic 10y ago
Next thing to look at is why only 2k/sec users can be added to the DB.
coldcode 10y ago
I wonder if this healthcare.gov will exist a year from now. It is nice to see how far it has come in 2 years.
- LargeCompanies 10y ago
  Doubtful Trump is getting rid of it. Personally I preferred the previous way of signing up for coverage ... call up a private insurer and sign up? The cost was less too as I wasn't paying for insurance for lazy sally or Joe Schmoe! So for me Obamacare/social medicine forced those who do and work to pay and cover for those who don't. See ya Obamacare(love obama though). Also the $300 plan I paid each month for sucked.. they fought me tooth and nail to cover things my doc prescribed.. I.e. Various scans(cat scan, etc).
  - delecti 10y ago
    Based on my understanding of the purpose of the Individual Mandate is that it's less that the well employed are subsidizing the under/unemployed, and more that the healthy are subsidizing the unhealthy. With everyone left to their own devices, the elderly/unhealthy are strongly incentivized to have insurance, and the young and healthy might reasonably conclude that their best bet is to not pay for insurance, that results in higher average premiums.
    - _delirium 10y ago
      People waiting until they get sick to sign up, essentially. The insurers' previous solution to this was no coverage of "pre-existing conditions". But people hated this generally, for various more or less good reasons. Banning exclusion of preexisting conditions is one of the handful of parts of Obamacare that's wildly popular across the political spectrum, at least among non-politicians (it polls something like 70-80% support). Obamacare's main replacement for that is the individual mandate requiring you to always have insurance, rather than to go without until you get seriously ill and only then purchase it. (There are also a few secondary things to try to discourage buying insurance only right before you're going to use it, like the open enrollment period being only once a year.)
      - delecti 10y ago
        Huh, I had never realized that purpose of the open enrollment period. It always seemed unnecessarily restrictive until now.
  - robbiemitchell 10y ago
    > call up a private insurer and sign up? For many (most?) cases you can still buy individual market directly with the insurer via phone/website. My understanding is you have to go through exchanges for certain subsidy situations. > The cost was less too > Also the $300 plan I paid each month for sucked First off, the old way was for an employer to handle it for you. If you had your own solo insurance before, I'm impressed. I had a hard time finding health coverage as a freelancer years ago. But more to your point, the cost was less, and sucked, because insurers could offer barebones, high-deductible plans that didn't really cover much except for catastrophic needs. Hence why you encountered high payments every time you actually needed healthcare. In addition, premiums cost more now because: - The highest risk and most expensive people were excluded from coverage entirely, either because they couldn't afford it or insurers refused to cover them, which they are no longer allowed to do. As a result, insurers are paying out more to cover medical treatment. - Providers and drug companies continue to raise prices, while insurance premiums are regulated and locked annually. - Government subsidies that were initially promised to insurers to help them cover gaps as they figured out the first few years of the ACA market were eliminated prematurely (cf. Marco Rubio). As a result, annual increases have jumped dramatically as they adjust to increased care, increasing costs, and drops in subsidies. So premiums are up because (a) you have better coverage (due to regulated minimum levels of coverage), (b) there are more sick people being covered (either because they couldn't afford coverage before or were excluded), and (c) net costs have risen. > forced those who do and work to pay and cover for those who don't I think that's an overly simplistic way of interpreting the situation.
sulam 10y ago
8K transactions / second is hardly extreme load. It's good they can support it, and I doubt they need more, but many, many sites handle more load than that.
- notyourwork 10y ago
  Agreed, it is good they can support it. However, is 8k TPS really a big deal? The team I work with handles 1M TPS.
  - ProAm 10y ago
    it's the government, we're thrilled they can do anything.
    - kartan 10y ago
      It all depends how much politics are involved, or how much public workers can freely do their jobs: https://www.nasa.gov/ https://www.nasa.gov/ http://www.darpa.mil/ http://www.darpa.mil/ And is Healthcare.gov developed by the government or just bough to an external company? It just an observation, I don't know if Healthcare.gov is a good or bad technology.
      - foota 10y ago
        I believe it was one of the first projects of the digital service. (https://www.usds.gov/ https://www.usds.gov/)
        vertex-four 10y ago
        Not quite. It was designed by a contractor, developed by several others, and once it was released to the public and shown to be dreadful, another contractor was put in charge of fixing it. It's still primarily developed by contract to this day. The first head of the USDS was involved in coordinating the fixing of Healthcare.gov, though.
0xdeadbeefbabe 10y ago
> (of only 2 machines!) with a total of 6000 worker threads. What about the bugs that arise from many connections from many machines? Those must exist too.
lcw 10y ago
I'm glad heathcare.gov is getting better. I would assume however the most important part of tests for open enrollment are the writes to user management where reads and writes simultaneously are exercised against datastores. Looks like users were manually created not sure if they included updates in the tests. I'm wondering if the tests were just auth based? That would make sense why they could easily scale horizontally since auth is a lot of times just limited to cpu capacity.
mooneater 10y ago
7754 tps over 1 hour is 28M. They may have 1B users in the db but they only tested activity on at most 28M.
- billsmithaustin 10y ago
  True, but running at 7754 tps over an hour with a database of 28M users is easier than with a database of 1B users.
  - mooneater 10y ago
    It is much easier to test with a small # of active users and a large (mostly unused) database, than with a large # of active users. To me the title implies 1B active users ("billion user load test").
    - user5994461 10y ago
      They are two different problems. If you have many active users, you're testing for the write load of user tokens and sessions. If you have many accounts, you're testing for the read load of the account database. Technically speaking, user id and sessions should be two different database systems, because they have different requirements and they're conflicting. For the first one, the problem is to have lots of write and enough performances. Plus having the storage if you keep tokens forever (typical for auditing). For the second one, the problem is to be able to store all the accounts, plus read performances (which can always be hacked by adding layers of caching). Funfact: if each account is only email + password (64 + 64 char), 1 billion accounts consume 128 GB, just for storing the ASCII characters. A lot more in practise with metadata, empty space and other fields. They say that they only had 1 maria DB on a single EBS P-IOPS volume. That's kinda weird that a 1 billion users database could fit in that -and zero errors, really?-. Maybe they just had a single integer field (1,2,3,4) to define the user and the password :D
rebootthesystem 10y ago
I am sure my wife will be very happy to learn about this test. It will definitely make up for our insurance going from about $12K per year to $26K per year. That, along with keeping our doctor and saving $2,500 per year have been fantastic. News Flash: Engineers think technology while, in the real world, the problem has nothing whatsoever to do with it. It's like tuning your Ferrarri for more power while your reality is you commute on the 405. In other words, irrelevant. But, hey, congrats, glad to see what I am paying for.
- foota 10y ago
  Fairly certain you're paying for the unemployed person down the street's insurance. (I say this not pejoratively toward's the unemployed)
  - rgbrenner 10y ago
    Insurance companies don't insure the unemployed for free. They're either paying their own premiums, or they're on medicaid/medicare. If they're on medicaid, it was paid for by the state/fed government from taxes... not from insurance premiums paid by someone else.
    - maxerickson 10y ago
      The cost of Medicaid and Medicare may show up in insurance premiums. http://kff.org/report-section/uncompensated-care-for-the-uninsured-in-2013-a-detailed-examination-cost-shifting-and-remaining-uncompensated-care-costs-8596/ http://kff.org/report-section/uncompensated-care-for-the-uni... There are other discussions of cost shifting that will make more aggressive claims; I've chosen to link Kaiser Family Foundation because I think they are pretty credible.
      - rgbrenner 10y ago
        the amount potentially associated with uncompensated care cost shifting is only 2.3 percent of private health insurance costs in 2013.
sixbit 10y ago
Healthcare.gov is the absolute worst. They had a button "End coverage for 2017" I clicked it around the end of November because I chose a non-marketplace plan for 2017, and the end result of that click was Cigna my current marketplace insurer for 2016 received a transmission from them, and interpreted it incorrectly, which made my 2016 plan inactive and retroactively terminated me back to 08/31 despite having paid all my premiums and processed claims up until end of November. Their escalation process is useless, 30 days to resolve (if I'm that lucky). Will try not to get injured in the meantime. Glitches and data sync issues are unreal, this is not an isolated incident. Every year it's been something. 2014 I was on Covered California and had to take them to administrative court to resolve data and tax form issues (canceled then due to a move and it completely wiped out my 2014 enrollment). Then in 2015 on HealthCare.gov they had me enrolled in the same plan twice and the insurer wanted double the premiums until they could resolve it.
- jlebar 10y ago
  Now we see the real test of whether healthcare.gov has reached the level of consumer tech: Does bug-report-via-top-ranked-HN-comment get anything done?
  - BinaryIdiot 10y ago
    After doing 6 years of government contracting and also interviewing with Nava I get the impression they're a higher grade of government contractor than what you typically work with. The initial failure of Healthcare.gov was no surprise to me (in fact I was surprised they even had a homepage working based on my government contracting experiences) and having Nava come in and fix things in what was likely a pretty horrific codebase is nothing short of impressive. Having said that there is still a ton of red tape and issues surrounding not just government contracting in general but also around the fact that different states and insurance providers have different systems that all work differently but somehow need to work together under this single application. So I wouldn't expect much even though the Nava folks, in my short time dealing with them and comparing them to other contractors, seem fairly solid.
  - llimllib 10y ago
SilverSurfer972 10y ago
A week to fill a DB and 70 m3x-large is way too expensive to find out how far you can scale. Me and my friend built stacktical (https://www.stacktical.com https://www.stacktical.com) so that with very minimal load testing it's possible to understand and share how many concurrent users your current and upcoming application infrastructure can handle. Testers are welcome
patja 10y ago
From the information provided I am doubtful that this is really a load test of healthcare.gov. It sounds like they are testing the login and account management system in isolation from the overall workflow of entering your financial data, dependents, etc. and shopping for plans, which is where the sausage gets made. It is an interesting post but it seems like this is about the most trivial component of the overall healthcare.gov solution, unless I am missing something.
pierotofy 10y ago
The website is terrible; they keep spamming my e-mail and calling my number with the same alerts over and over again.
user5994461 10y ago
I'd like to see this kind of authentication done over OpenAM and OpenDJ. That's the IAM systems done by Sun Microsystem (later Oracle, then Forgerock). That should be rock solid. IAM: Identity and Access Management Forgerock: https://www.forgerock.com/ https://www.forgerock.com/
rco8786 10y ago
Why load test for > 3x the number of potential users?
walter_bishop 10y ago
What's it cost to run Healthcare.gov on AWS, how many people-hours were spent in designing, configuring and deploying the website as compared to a non-cloud solution?

127 points by shashashasha 10y ago

> The current observed peak load for HealthCare.gov’s Open Enrollment in 2016 was on Dec 14 2016 Dec 14? Did I miss something?

ProAm 10y ago
Probably Dec 14 2015 for the 2016 enrollment year

deleted 10y ago

[deleted]

jakupovic 10y ago

Next thing to look at is why only 2k/sec users can be added to the DB.

coldcode 10y ago

I wonder if this healthcare.gov will exist a year from now. It is nice to see how far it has come in 2 years.

LargeCompanies 10y ago
Doubtful Trump is getting rid of it. Personally I preferred the previous way of signing up for coverage ... call up a private insurer and sign up? The cost was less too as I wasn't paying for insurance for lazy sally or Joe Schmoe! So for me Obamacare/social medicine forced those who do and work to pay and cover for those who don't. See ya Obamacare(love obama though). Also the $300 plan I paid each month for sucked.. they fought me tooth and nail to cover things my doc prescribed.. I.e. Various scans(cat scan, etc).
- delecti 10y ago
  Based on my understanding of the purpose of the Individual Mandate is that it's less that the well employed are subsidizing the under/unemployed, and more that the healthy are subsidizing the unhealthy. With everyone left to their own devices, the elderly/unhealthy are strongly incentivized to have insurance, and the young and healthy might reasonably conclude that their best bet is to not pay for insurance, that results in higher average premiums.
  - _delirium 10y ago
    People waiting until they get sick to sign up, essentially. The insurers' previous solution to this was no coverage of "pre-existing conditions". But people hated this generally, for various more or less good reasons. Banning exclusion of preexisting conditions is one of the handful of parts of Obamacare that's wildly popular across the political spectrum, at least among non-politicians (it polls something like 70-80% support). Obamacare's main replacement for that is the individual mandate requiring you to always have insurance, rather than to go without until you get seriously ill and only then purchase it. (There are also a few secondary things to try to discourage buying insurance only right before you're going to use it, like the open enrollment period being only once a year.)
    - delecti 10y ago
      Huh, I had never realized that purpose of the open enrollment period. It always seemed unnecessarily restrictive until now.
- robbiemitchell 10y ago
  > call up a private insurer and sign up? For many (most?) cases you can still buy individual market directly with the insurer via phone/website. My understanding is you have to go through exchanges for certain subsidy situations. > The cost was less too > Also the $300 plan I paid each month for sucked First off, the old way was for an employer to handle it for you. If you had your own solo insurance before, I'm impressed. I had a hard time finding health coverage as a freelancer years ago. But more to your point, the cost was less, and sucked, because insurers could offer barebones, high-deductible plans that didn't really cover much except for catastrophic needs. Hence why you encountered high payments every time you actually needed healthcare. In addition, premiums cost more now because: - The highest risk and most expensive people were excluded from coverage entirely, either because they couldn't afford it or insurers refused to cover them, which they are no longer allowed to do. As a result, insurers are paying out more to cover medical treatment. - Providers and drug companies continue to raise prices, while insurance premiums are regulated and locked annually. - Government subsidies that were initially promised to insurers to help them cover gaps as they figured out the first few years of the ACA market were eliminated prematurely (cf. Marco Rubio). As a result, annual increases have jumped dramatically as they adjust to increased care, increasing costs, and drops in subsidies. So premiums are up because (a) you have better coverage (due to regulated minimum levels of coverage), (b) there are more sick people being covered (either because they couldn't afford coverage before or were excluded), and (c) net costs have risen. > forced those who do and work to pay and cover for those who don't I think that's an overly simplistic way of interpreting the situation.

sulam 10y ago

8K transactions / second is hardly extreme load. It's good they can support it, and I doubt they need more, but many, many sites handle more load than that.

notyourwork 10y ago
Agreed, it is good they can support it. However, is 8k TPS really a big deal? The team I work with handles 1M TPS.
- ProAm 10y ago
  it's the government, we're thrilled they can do anything.
  - kartan 10y ago
    It all depends how much politics are involved, or how much public workers can freely do their jobs: https://www.nasa.gov/ https://www.nasa.gov/ http://www.darpa.mil/ http://www.darpa.mil/ And is Healthcare.gov developed by the government or just bough to an external company? It just an observation, I don't know if Healthcare.gov is a good or bad technology.
    - foota 10y ago
      I believe it was one of the first projects of the digital service. (https://www.usds.gov/ https://www.usds.gov/)
      - vertex-four 10y ago
        Not quite. It was designed by a contractor, developed by several others, and once it was released to the public and shown to be dreadful, another contractor was put in charge of fixing it. It's still primarily developed by contract to this day. The first head of the USDS was involved in coordinating the fixing of Healthcare.gov, though.

0xdeadbeefbabe 10y ago

> (of only 2 machines!) with a total of 6000 worker threads. What about the bugs that arise from many connections from many machines? Those must exist too.

lcw 10y ago

I'm glad heathcare.gov is getting better. I would assume however the most important part of tests for open enrollment are the writes to user management where reads and writes simultaneously are exercised against datastores. Looks like users were manually created not sure if they included updates in the tests. I'm wondering if the tests were just auth based? That would make sense why they could easily scale horizontally since auth is a lot of times just limited to cpu capacity.

mooneater 10y ago

7754 tps over 1 hour is 28M. They may have 1B users in the db but they only tested activity on at most 28M.

billsmithaustin 10y ago
True, but running at 7754 tps over an hour with a database of 28M users is easier than with a database of 1B users.
- mooneater 10y ago
  It is much easier to test with a small # of active users and a large (mostly unused) database, than with a large # of active users. To me the title implies 1B active users ("billion user load test").
  - user5994461 10y ago
    They are two different problems. If you have many active users, you're testing for the write load of user tokens and sessions. If you have many accounts, you're testing for the read load of the account database. Technically speaking, user id and sessions should be two different database systems, because they have different requirements and they're conflicting. For the first one, the problem is to have lots of write and enough performances. Plus having the storage if you keep tokens forever (typical for auditing). For the second one, the problem is to be able to store all the accounts, plus read performances (which can always be hacked by adding layers of caching). Funfact: if each account is only email + password (64 + 64 char), 1 billion accounts consume 128 GB, just for storing the ASCII characters. A lot more in practise with metadata, empty space and other fields. They say that they only had 1 maria DB on a single EBS P-IOPS volume. That's kinda weird that a 1 billion users database could fit in that -and zero errors, really?-. Maybe they just had a single integer field (1,2,3,4) to define the user and the password :D

rebootthesystem 10y ago

I am sure my wife will be very happy to learn about this test. It will definitely make up for our insurance going from about $12K per year to $26K per year. That, along with keeping our doctor and saving $2,500 per year have been fantastic. News Flash: Engineers think technology while, in the real world, the problem has nothing whatsoever to do with it. It's like tuning your Ferrarri for more power while your reality is you commute on the 405. In other words, irrelevant. But, hey, congrats, glad to see what I am paying for.

foota 10y ago
Fairly certain you're paying for the unemployed person down the street's insurance. (I say this not pejoratively toward's the unemployed)
- rgbrenner 10y ago
  Insurance companies don't insure the unemployed for free. They're either paying their own premiums, or they're on medicaid/medicare. If they're on medicaid, it was paid for by the state/fed government from taxes... not from insurance premiums paid by someone else.
  - maxerickson 10y ago
    The cost of Medicaid and Medicare may show up in insurance premiums. http://kff.org/report-section/uncompensated-care-for-the-uninsured-in-2013-a-detailed-examination-cost-shifting-and-remaining-uncompensated-care-costs-8596/ http://kff.org/report-section/uncompensated-care-for-the-uni... There are other discussions of cost shifting that will make more aggressive claims; I've chosen to link Kaiser Family Foundation because I think they are pretty credible.
    - rgbrenner 10y ago
      the amount potentially associated with uncompensated care cost shifting is only 2.3 percent of private health insurance costs in 2013.

sixbit 10y ago

Healthcare.gov is the absolute worst. They had a button "End coverage for 2017" I clicked it around the end of November because I chose a non-marketplace plan for 2017, and the end result of that click was Cigna my current marketplace insurer for 2016 received a transmission from them, and interpreted it incorrectly, which made my 2016 plan inactive and retroactively terminated me back to 08/31 despite having paid all my premiums and processed claims up until end of November. Their escalation process is useless, 30 days to resolve (if I'm that lucky). Will try not to get injured in the meantime. Glitches and data sync issues are unreal, this is not an isolated incident. Every year it's been something. 2014 I was on Covered California and had to take them to administrative court to resolve data and tax form issues (canceled then due to a move and it completely wiped out my 2014 enrollment). Then in 2015 on HealthCare.gov they had me enrolled in the same plan twice and the insurer wanted double the premiums until they could resolve it.

jlebar 10y ago
Now we see the real test of whether healthcare.gov has reached the level of consumer tech: Does bug-report-via-top-ranked-HN-comment get anything done?
- BinaryIdiot 10y ago
  After doing 6 years of government contracting and also interviewing with Nava I get the impression they're a higher grade of government contractor than what you typically work with. The initial failure of Healthcare.gov was no surprise to me (in fact I was surprised they even had a homepage working based on my government contracting experiences) and having Nava come in and fix things in what was likely a pretty horrific codebase is nothing short of impressive. Having said that there is still a ton of red tape and issues surrounding not just government contracting in general but also around the fact that different states and insurance providers have different systems that all work differently but somehow need to work together under this single application. So I wouldn't expect much even though the Nava folks, in my short time dealing with them and comparing them to other contractors, seem fairly solid.
- llimllib 10y ago

SilverSurfer972 10y ago

A week to fill a DB and 70 m3x-large is way too expensive to find out how far you can scale. Me and my friend built stacktical (https://www.stacktical.com https://www.stacktical.com) so that with very minimal load testing it's possible to understand and share how many concurrent users your current and upcoming application infrastructure can handle. Testers are welcome

patja 10y ago

From the information provided I am doubtful that this is really a load test of healthcare.gov. It sounds like they are testing the login and account management system in isolation from the overall workflow of entering your financial data, dependents, etc. and shopping for plans, which is where the sausage gets made. It is an interesting post but it seems like this is about the most trivial component of the overall healthcare.gov solution, unless I am missing something.

pierotofy 10y ago

The website is terrible; they keep spamming my e-mail and calling my number with the same alerts over and over again.

user5994461 10y ago

I'd like to see this kind of authentication done over OpenAM and OpenDJ. That's the IAM systems done by Sun Microsystem (later Oracle, then Forgerock). That should be rock solid. IAM: Identity and Access Management Forgerock: https://www.forgerock.com/ https://www.forgerock.com/

rco8786 10y ago

Why load test for > 3x the number of potential users?

walter_bishop 10y ago

What's it cost to run Healthcare.gov on AWS, how many people-hours were spent in designing, configuring and deploying the website as compared to a non-cloud solution?