4 ms·

> Officials say that too many people tried to log on at once; within three days, 8.6m had visited healthcare.gov. That is an order of magnitude less traffic th

by Encosia 13y ago

> Officials say that too many people tried to log on at once; within three days, 8.6m had visited healthcare.gov. That is an order of magnitude less traffic than I expected. 33 visits per second isn't child's play, but my WordPress blog hosted on a 1GB Linode VPS has easily handled bursts of that kind of traffic in the past. That's totally an apples and oranges comparison, of course, but it's more and more difficult to understand how this project has failed so miserably. I half bought the excuse that it was one of the highest traffic site launches in history, but apparently that was more speculation than truth.

jasonlotito 13y ago
Except your making the classic mistake of assuming that it's 33 visits per second because you are looking at the 8.6m visitors happening evenly over the course of 3 days. You're also making the mistake of confusing the number of visitors with the number of hits on the site. 20 requests to the server for information is still one visit. And yes, 33 hits per second on a static site, cached at various points is far, far easier to develop. > but it's more and more difficult to understand how this project has failed so miserably. So, imagine someone estimated 8.6m visitors over three days, and decided to load test the site for 33 visits per second... =) Then it becomes really easy to see why it could fall over. Disclaimer: I've been doing load testing the last few weeks, and have been beating up on other peoples server side applications. It's a lot of fun when it's not your code getting hammered. =)
- jacques_chester 13y ago
  Agreed. Capacity planning is not about the average case; it's about the 99th percentile case and whether you can afford it.
- deleted 13y ago
  [deleted]
  - jasonlotito 13y ago
    That's incorrect. The entire system, far beyond just the website included those costs. The company that actually built the site got it for a $93.7 million contract. The HALF A BILLION DOLLARS you are remarking also include exchanges facilitated by the government for individual states. Finally, that $93.7 million was over 3 years, 2011, 2012, and 2013. Now, you can armchair things all you want, but the technical elements of building the healthcare.gov website did not cost HALF A BILLION DOLLARS. CAPS LOCK IS CRUISE CONTROL FOR COOL!!1!
    - deleted 13y ago
      [deleted]
    - hga 13y ago
      You make some good points, so how about the point that's really getting to me? THREE AND A HALF YEARS That is, March 23, 2010 to October 1, 2013, 2 days after the Congress was finished with it. The Administration's CTO should have been bird dogging this from about then, along with the HHS specialists who we know have a clue (CMS.gov and Medicare.gov). They've had plenty of time to get the basic signup system working ... that should have been undergoing stress testing a long time ago, not failing on people as attested to in this discussion. We often talk about the triad of time, money and quality. They've had plenty of time, and obviously quality was punted. Is anyone claiming they didn't have enough money to the point it could have caused this level of failure?
      - jasonlotito 13y ago
        Okay, so whoever I replied to deleted their message. They are the ones who wrote "HALF A BILLION DOLLARS." Anyways... > THREE AND A HALF YEARS Seriously... I just did it to highlight the fact that's it stupid. Please tell me you're only doing it for the same reason? =) But anyways, no, the site didn't take three and a half years to make, as is clear by anyone taking a few minutes to do a bit of research. Anyways, getting to the heart of the matter: > Is anyone claiming they didn't have enough money to the point it could have caused this level of failure? I don't know and I don't care. I honestly think that question is irrelevant. Who cares if anyone is asking that?
jonknee 13y ago
Your WordPress blog doesn't interact with hundreds of APIs with varying quality and reliability. Healthcare.gov does and while many departments it interacts with are all furlough.
jpollock 13y ago
In phone networks, the difference between peak and average load is typically at least one order of magnitude higher. Every day there is a difference of 4x between min and max, with the peak being sustained for multiple hours. That's why telco systems are sized as "BHCA" (busy hour call attempts) and "CAPS" (call attempts per second), which are peak rates, not average rates. Well, they are both averages, but not over the whole day or year! Then, as others have pointed out, there's a difference between a static site and a non-static one. This is mostly determined by I/O. A single disk can do between 100-150 IO operations per second (IOPS). Reading a row from a database will take several IOPS. In the systems I work with, we have to keep the entire database in memory to meet our performance numbers, and even then we are still O(database reads). Doing a SOAP request to a legacy platform that was designed to only support batch requests? Lots of opportunity for slowness. Particularly when you start to run into TCP timers controlling port reuse. Doesn't excuse the 404's though. That stuff is easy to test automatically.

by Encosia 13y ago

jasonlotito 13y ago

Except your making the classic mistake of assuming that it's 33 visits per second because you are looking at the 8.6m visitors happening evenly over the course of 3 days. You're also making the mistake of confusing the number of visitors with the number of hits on the site. 20 requests to the server for information is still one visit. And yes, 33 hits per second on a static site, cached at various points is far, far easier to develop. > but it's more and more difficult to understand how this project has failed so miserably. So, imagine someone estimated 8.6m visitors over three days, and decided to load test the site for 33 visits per second... =) Then it becomes really easy to see why it could fall over. Disclaimer: I've been doing load testing the last few weeks, and have been beating up on other peoples server side applications. It's a lot of fun when it's not your code getting hammered. =)

jacques_chester 13y ago
Agreed. Capacity planning is not about the average case; it's about the 99th percentile case and whether you can afford it.
deleted 13y ago
[deleted]
- jasonlotito 13y ago
  That's incorrect. The entire system, far beyond just the website included those costs. The company that actually built the site got it for a $93.7 million contract. The HALF A BILLION DOLLARS you are remarking also include exchanges facilitated by the government for individual states. Finally, that $93.7 million was over 3 years, 2011, 2012, and 2013. Now, you can armchair things all you want, but the technical elements of building the healthcare.gov website did not cost HALF A BILLION DOLLARS. CAPS LOCK IS CRUISE CONTROL FOR COOL!!1!
  - deleted 13y ago
    [deleted]
  - hga 13y ago
    You make some good points, so how about the point that's really getting to me? THREE AND A HALF YEARS That is, March 23, 2010 to October 1, 2013, 2 days after the Congress was finished with it. The Administration's CTO should have been bird dogging this from about then, along with the HHS specialists who we know have a clue (CMS.gov and Medicare.gov). They've had plenty of time to get the basic signup system working ... that should have been undergoing stress testing a long time ago, not failing on people as attested to in this discussion. We often talk about the triad of time, money and quality. They've had plenty of time, and obviously quality was punted. Is anyone claiming they didn't have enough money to the point it could have caused this level of failure?
    - jasonlotito 13y ago
      Okay, so whoever I replied to deleted their message. They are the ones who wrote "HALF A BILLION DOLLARS." Anyways... > THREE AND A HALF YEARS Seriously... I just did it to highlight the fact that's it stupid. Please tell me you're only doing it for the same reason? =) But anyways, no, the site didn't take three and a half years to make, as is clear by anyone taking a few minutes to do a bit of research. Anyways, getting to the heart of the matter: > Is anyone claiming they didn't have enough money to the point it could have caused this level of failure? I don't know and I don't care. I honestly think that question is irrelevant. Who cares if anyone is asking that?

jonknee 13y ago

Your WordPress blog doesn't interact with hundreds of APIs with varying quality and reliability. Healthcare.gov does and while many departments it interacts with are all furlough.

jpollock 13y ago

In phone networks, the difference between peak and average load is typically at least one order of magnitude higher. Every day there is a difference of 4x between min and max, with the peak being sustained for multiple hours. That's why telco systems are sized as "BHCA" (busy hour call attempts) and "CAPS" (call attempts per second), which are peak rates, not average rates. Well, they are both averages, but not over the whole day or year! Then, as others have pointed out, there's a difference between a static site and a non-static one. This is mostly determined by I/O. A single disk can do between 100-150 IO operations per second (IOPS). Reading a row from a database will take several IOPS. In the systems I work with, we have to keep the entire database in memory to meet our performance numbers, and even then we are still O(database reads). Doing a SOAP request to a legacy platform that was designed to only support batch requests? Lots of opportunity for slowness. Particularly when you start to run into TCP timers controlling port reuse. Doesn't excuse the 404's though. That stuff is easy to test automatically.