8 ms·

Why the New Obamacare Website Is Going to Work This Time

76 points by brandonb 12y ago

akoumjian 12y ago
The question I have is, how do we get government to hire teams like this in the first place?
- bicknergseng 12y ago
  Build a federal gov run IT branch that isn't the NSA instead of contracting things like this out?
  - brandonb 12y ago
    Good idea. If you're interested in this type of thing, check out 18F, which is a brand-new engineering office run from within the government in San Francisco and DC: https://18f.gsa.gov/ https://18f.gsa.gov/
    - bicknergseng 12y ago
      Very cool. Taking a look and signed up for the Eventbrite above.
  - mseebach 12y ago
    And what are the dynamics that will make this work better, especially recalling the iron law of bureaucracy? http://www.jerrypournelle.com/reports/jerryp/iron.html http://www.jerrypournelle.com/reports/jerryp/iron.html
    - nhaehnle 12y ago
      One important dynamic is that contracting, due to its slippery nature, attracts the kind of people whose skill is siphoning money away from government while minimizing what they deliver. Think about it: Even contracted physical construction gets screwed up in large scale government projects, and building good contracts about software development is much harder that contracts about physical construction. In-house development simply doesn't have the kind of indirection where the bloodsuckers can get in.
      - mseebach 12y ago
        You'll cut one kind of bloodsucker out of the loop and let another in: the kind that thrives building fiefdoms in bureaucracies. And you're not addressing the main issue: projects being way too big and unwieldy (if now somewhat less expensive) even before they go over budget, over time or flatly fails altogether. If the CGIs of the world actually delivered a functional product on time and on schedule, the fact that they earn handsomely in the process is at best a secondary concern. Next to the cost of the staggering embarrassment that was the failure of the first Obamacare website, whatever CGI quoted is a rounding error.
- enjo 12y ago
  It's incredibly difficult. When government money is involved the good old boy network swings into action. Contractors are picked based on political connections first and actual capabilities second.
  - nickff 12y ago
    The connections are more often bureaucratic than political.
    - stcredzero 12y ago
      And such behavior is also typical in large enterprises.
  - jinushaun 12y ago
    I wouldn't be so cynical. Contracts are really awarded to companies that can jump through the bureaucratic hoops--and there's an entire industry dedicated to doing that while over-charging and under-delivering.
brandonb 12y ago
A bunch of the folks in this article are Y Combinator or Google alumni, and we're giving a talk on Saving HealthCare.gov next Thursday in SF for anyone who's interested: http://www.eventbrite.com/e/saving-healthcaregov-tickets-11850173187?aff=hn http://www.eventbrite.com/e/saving-healthcaregov-tickets-118... Let us know if you have questions! Or if you want to help. UPDATE: we seem to be sold out. Trying to find a larger venue. In the meantime, feel free to sign up on the waiting list and you'll get an email as soon as we find a room with more capacity.
- rch 12y ago
  Any chance we could live stream this in Boulder?
  - brandonb 12y ago
    Unfortunately we're not doing any recordings -- it's hard to be candid and recorded for all history at the same time. :(
    - protomyth 12y ago
      You're talking about a government website and not recording? I am a little worried that you don't think you can be candid about what is being done with our tax money.
      - srj 12y ago
        I imagine recording a discussion is particularly stifling when it pertains to government contracted work. In government it's all about how blame should be assigned and not about moving forward productively.
    - yeukhon 12y ago
      If you can't get professional to record the event, just launch a Google Hangout broadcast on the speaker's computer so the sldie is recorded, add a microphone to him/her under $20 and you good to go. If the speaker is close to the computer just use the built-in microphone.
bkurtz13 12y ago
> David Chang, a 2102 MIT graduate So that's how they're doing it, they got a programmer from the future to help!
- jusben1369 12y ago
  Well the US federal government has more resources than most so it strikes me as a smart move to do this. I suspect they were holding off announcing that they've solved time travel closer to the congressional elections to help out the Democrats but the cats out of the bag now.
  - deleted 12y ago
    [deleted]
dgroves 12y ago
It is so odd to me that the second highest budget outlay is for a system "known as healthcare" for which, I receive none, unless I pay for insurance, which also doesn't pay for anything - rather they require I pay. This same system lost 272 billion to fraud last year. Rather than clean up the fraud and corruption - we are getting a website?!?, and fines if we don't pay an insurance company to take our money and provide no benefits. America doesn't have a health care system it has an illness system whence you become ill they are allowed to relieve you of all of your assets and leave you to die.
sp332 12y ago
Why is this using Amazon servers? Wouldn't it make more sense to use purpose-built machines for this?
- toomuchtodo 12y ago
  No. You serve the static assets from a CDN, but the authentication and dynamic content generation need to scale very quickly when you have an in rush of traffic. This is the perfect use case for Amazon AWS (GovCloud, specifically). EDIT: Disclaimer: I did not work on Healthcare.gov, but did study up on the entire architecture while trying for a gig on the recovery team.
  - sp332 12y ago
    How many servers are we talking here, anyway? Like 1 rack's worth or what?
    - ceejayoz 12y ago
      It's highly variable because of the way health insurance in the US works. There's an "enrollment" period where you can switch/reenroll in your plan. It only lasts a few weeks to a few months in the end of the year. During that period, load is immense. Outside that period, there's extremely low traffic, as only people who have "qualifying events" are eligible to shop for insurance - if you get married, divorced, lose a job, etc. Letting someone else figure out what to do with 95% of your server capacity for 10ish months out of the year is a pretty decent cost savings for the government, I'd imagine.
- ianstallings 12y ago
  The government has their own amazon cloud, but to the point I think it actually would be better to use a virtualized setup because then you can scale power as needed and not have to worry too much about bringing servers online. Then when the inevitable slow period comes you can scale down significantly. This is my experience at least.
vinhboy 12y ago
So... are we going to get any of our taxpayer money back from the people who fucked up the first time? On another note I am really delighted to see that the government was forced to step out of their bubble to make this happen.
- sp332 12y ago
  Nope, in fact we're still giving them money for new projects. http://www.cgi.com/en/media-announcement http://www.cgi.com/en/media-announcement
- Sambdala 12y ago
  This is a very important issue that failed in our last attempt. As we put so many resources behind that attempt, this proves it's a difficult problem to solve that requires more resources. It logically follows that more taxpayer money is required for the next iteration. How else could we solve such an important problem?
- jayd16 12y ago
  Could you imagine if every shitty software project ended in a refund?
spacemanmatt 12y ago
We built a NEW website, and it's going to WORK this time. Not like that last time, that only took us up 30 feet into the air then dropped us. And when it DOES work, you're gonna turn to me and say, "Dude, this rocks." And I'll be like, "Dude, I TOLD you it was gonna be sweet."
Subetei36 12y ago
How do they measure success? If something costs 1000x what it would have in the market and still strives for adequacy it's already proof of failure. If any of us failed so spectacularly we'd go bankrupt and another service would be used (and there would be plenty of choices if regulations/poor law services didn't dissuade people from creating this kind of service). Now we have people that were productive at companies making services we pay for of our own volition spending time on this thing that doesn't work as a symptom, not a cause. So confused why this is a thing.
Agathos 12y ago
There are plenty more government IT projects for them to save next. The scale of these failures is mind-boggling, and Healthcare.gov wasn't nearly the worst. Here's one I just read about yesterday, might be good for their next project: http://www.thefiscaltimes.com/Articles/2013/11/26/Another-Failed-Gov-t-Tech-Project-Cost-11-Billion http://www.thefiscaltimes.com/Articles/2013/11/26/Another-Fa... That's $1.1 billion lost, another $4 billion planned. I can't even imagine what I'd demand of a $4 billion piece of software.
- evv 12y ago
  I can't find a reference for this, but I've heard that the IRS has attempted several multi-billion-dollar system overhauls, with no success. It's fascinating how much of the world is run by decades-old software, with no serious hope of an upgrade.
- brandonb 12y ago
  If you're interested in helping with this or any other government project, feel free to email jobs@hcgov.us or attend the talk mentioned elsewhere in the thread! There are lots of ways to get involved.
dreamdu5t 12y ago
Talk is cheap.
jonatanheyman 12y ago
I hope they're performing load tests this time, which apparently was not done properly for the first version.
deleted 12y ago
[deleted]

76 points by brandonb 12y ago

akoumjian 12y ago

The question I have is, how do we get government to hire teams like this in the first place?

bicknergseng 12y ago
Build a federal gov run IT branch that isn't the NSA instead of contracting things like this out?
- brandonb 12y ago
  Good idea. If you're interested in this type of thing, check out 18F, which is a brand-new engineering office run from within the government in San Francisco and DC: https://18f.gsa.gov/ https://18f.gsa.gov/
  - bicknergseng 12y ago
    Very cool. Taking a look and signed up for the Eventbrite above.
- mseebach 12y ago
  And what are the dynamics that will make this work better, especially recalling the iron law of bureaucracy? http://www.jerrypournelle.com/reports/jerryp/iron.html http://www.jerrypournelle.com/reports/jerryp/iron.html
  - nhaehnle 12y ago
    One important dynamic is that contracting, due to its slippery nature, attracts the kind of people whose skill is siphoning money away from government while minimizing what they deliver. Think about it: Even contracted physical construction gets screwed up in large scale government projects, and building good contracts about software development is much harder that contracts about physical construction. In-house development simply doesn't have the kind of indirection where the bloodsuckers can get in.
    - mseebach 12y ago
      You'll cut one kind of bloodsucker out of the loop and let another in: the kind that thrives building fiefdoms in bureaucracies. And you're not addressing the main issue: projects being way too big and unwieldy (if now somewhat less expensive) even before they go over budget, over time or flatly fails altogether. If the CGIs of the world actually delivered a functional product on time and on schedule, the fact that they earn handsomely in the process is at best a secondary concern. Next to the cost of the staggering embarrassment that was the failure of the first Obamacare website, whatever CGI quoted is a rounding error.
enjo 12y ago
It's incredibly difficult. When government money is involved the good old boy network swings into action. Contractors are picked based on political connections first and actual capabilities second.
- nickff 12y ago
  The connections are more often bureaucratic than political.
  - stcredzero 12y ago
    And such behavior is also typical in large enterprises.
- jinushaun 12y ago
  I wouldn't be so cynical. Contracts are really awarded to companies that can jump through the bureaucratic hoops--and there's an entire industry dedicated to doing that while over-charging and under-delivering.

brandonb 12y ago

A bunch of the folks in this article are Y Combinator or Google alumni, and we're giving a talk on Saving HealthCare.gov next Thursday in SF for anyone who's interested: http://www.eventbrite.com/e/saving-healthcaregov-tickets-11850173187?aff=hn http://www.eventbrite.com/e/saving-healthcaregov-tickets-118... Let us know if you have questions! Or if you want to help. UPDATE: we seem to be sold out. Trying to find a larger venue. In the meantime, feel free to sign up on the waiting list and you'll get an email as soon as we find a room with more capacity.

rch 12y ago
Any chance we could live stream this in Boulder?
- brandonb 12y ago
  Unfortunately we're not doing any recordings -- it's hard to be candid and recorded for all history at the same time. :(
  - protomyth 12y ago
    You're talking about a government website and not recording? I am a little worried that you don't think you can be candid about what is being done with our tax money.
    - srj 12y ago
      I imagine recording a discussion is particularly stifling when it pertains to government contracted work. In government it's all about how blame should be assigned and not about moving forward productively.
  - yeukhon 12y ago
    If you can't get professional to record the event, just launch a Google Hangout broadcast on the speaker's computer so the sldie is recorded, add a microphone to him/her under $20 and you good to go. If the speaker is close to the computer just use the built-in microphone.

bkurtz13 12y ago

> David Chang, a 2102 MIT graduate So that's how they're doing it, they got a programmer from the future to help!

jusben1369 12y ago
Well the US federal government has more resources than most so it strikes me as a smart move to do this. I suspect they were holding off announcing that they've solved time travel closer to the congressional elections to help out the Democrats but the cats out of the bag now.
- deleted 12y ago
  [deleted]

dgroves 12y ago

It is so odd to me that the second highest budget outlay is for a system "known as healthcare" for which, I receive none, unless I pay for insurance, which also doesn't pay for anything - rather they require I pay. This same system lost 272 billion to fraud last year. Rather than clean up the fraud and corruption - we are getting a website?!?, and fines if we don't pay an insurance company to take our money and provide no benefits. America doesn't have a health care system it has an illness system whence you become ill they are allowed to relieve you of all of your assets and leave you to die.

sp332 12y ago

Why is this using Amazon servers? Wouldn't it make more sense to use purpose-built machines for this?

toomuchtodo 12y ago
No. You serve the static assets from a CDN, but the authentication and dynamic content generation need to scale very quickly when you have an in rush of traffic. This is the perfect use case for Amazon AWS (GovCloud, specifically). EDIT: Disclaimer: I did not work on Healthcare.gov, but did study up on the entire architecture while trying for a gig on the recovery team.
- sp332 12y ago
  How many servers are we talking here, anyway? Like 1 rack's worth or what?
  - ceejayoz 12y ago
    It's highly variable because of the way health insurance in the US works. There's an "enrollment" period where you can switch/reenroll in your plan. It only lasts a few weeks to a few months in the end of the year. During that period, load is immense. Outside that period, there's extremely low traffic, as only people who have "qualifying events" are eligible to shop for insurance - if you get married, divorced, lose a job, etc. Letting someone else figure out what to do with 95% of your server capacity for 10ish months out of the year is a pretty decent cost savings for the government, I'd imagine.
ianstallings 12y ago
The government has their own amazon cloud, but to the point I think it actually would be better to use a virtualized setup because then you can scale power as needed and not have to worry too much about bringing servers online. Then when the inevitable slow period comes you can scale down significantly. This is my experience at least.

vinhboy 12y ago

So... are we going to get any of our taxpayer money back from the people who fucked up the first time? On another note I am really delighted to see that the government was forced to step out of their bubble to make this happen.

sp332 12y ago
Nope, in fact we're still giving them money for new projects. http://www.cgi.com/en/media-announcement http://www.cgi.com/en/media-announcement
Sambdala 12y ago
This is a very important issue that failed in our last attempt. As we put so many resources behind that attempt, this proves it's a difficult problem to solve that requires more resources. It logically follows that more taxpayer money is required for the next iteration. How else could we solve such an important problem?
jayd16 12y ago
Could you imagine if every shitty software project ended in a refund?

spacemanmatt 12y ago

We built a NEW website, and it's going to WORK this time. Not like that last time, that only took us up 30 feet into the air then dropped us. And when it DOES work, you're gonna turn to me and say, "Dude, this rocks." And I'll be like, "Dude, I TOLD you it was gonna be sweet."

Subetei36 12y ago

How do they measure success? If something costs 1000x what it would have in the market and still strives for adequacy it's already proof of failure. If any of us failed so spectacularly we'd go bankrupt and another service would be used (and there would be plenty of choices if regulations/poor law services didn't dissuade people from creating this kind of service). Now we have people that were productive at companies making services we pay for of our own volition spending time on this thing that doesn't work as a symptom, not a cause. So confused why this is a thing.

Agathos 12y ago

There are plenty more government IT projects for them to save next. The scale of these failures is mind-boggling, and Healthcare.gov wasn't nearly the worst. Here's one I just read about yesterday, might be good for their next project: http://www.thefiscaltimes.com/Articles/2013/11/26/Another-Failed-Gov-t-Tech-Project-Cost-11-Billion http://www.thefiscaltimes.com/Articles/2013/11/26/Another-Fa... That's $1.1 billion lost, another $4 billion planned. I can't even imagine what I'd demand of a $4 billion piece of software.

evv 12y ago
I can't find a reference for this, but I've heard that the IRS has attempted several multi-billion-dollar system overhauls, with no success. It's fascinating how much of the world is run by decades-old software, with no serious hope of an upgrade.
brandonb 12y ago
If you're interested in helping with this or any other government project, feel free to email jobs@hcgov.us or attend the talk mentioned elsewhere in the thread! There are lots of ways to get involved.

dreamdu5t 12y ago

Talk is cheap.

jonatanheyman 12y ago

I hope they're performing load tests this time, which apparently was not done properly for the first version.

deleted 12y ago

[deleted]