5 ms·

Important but not mentioned in the article is the amount that the government paid for that incompetence driven disaster. $840 million!

unethical_ban 3y ago
Right? Don't they know software is free and insurance is a simple industry?
- ceejayoz 3y ago
  It's not free, but the initial contractor CGI definitely hosed taxpayers.
- renewiltord 3y ago
  The rescue proved something everyone knows: some software engineers suck; others rule.
- fallingknife 3y ago
  And that is an excuse for spending damn near a billion for this? > Testing was largely a manual process. Releases and deployment required lengthy overnight downtimes and often failed, required equally-lengthy rollbacks. The core data layer lacked a schema. Provisioning hosting resources was slow and not automated. Critically, there was a pervasive lack of monitoring of the site. No APM, no bog-standard metrics about CPU, memory, disk, network, etc., that could be viewed by anyone on the team.
- lazide 3y ago
  The “beauty” of contracting for the gov’t - if you know how to play the game - is that their internal management and incentive structure is usually so screwed up (and will never get better) that manipulating the customer is easy and lucrative. If you know how to do it without being indicted anyway. Constant back and forth change orders, pointless delays to serve constantly changing stakeholders, constant demos with no real product, no one in the customer with any power who has experience to call BS on timelines or delivery, etc. And if done correctly, everyone in the ‘customer’ is simultaneously happy and angry and so it’s easy to deflect blame and point fingers back into their own org. Or so I’ve heard from folks who are involved. I try to avoid it like a plague, the BS I deal with hurts my soul enough already.
  - baggachipz 3y ago
    Having worked for a Beltway Bandit "consulting" company in the past, you are exactly right. It's not about getting work done, it's about spending hours so you can receive more.
  - hondo77 3y ago
    That was one of the takeaways from this whole episode. In its wake, the Technology Transformation Services and 18F resulted. Part of what they do is show government agencies a better way of procurement (a "modular" or "agile" approach). Performance-based contracting, and where the source code is, say, in a GitHub repo rather than on the contractor's servers. Contractor not performing? Get rid of them and get someone who will. Need to change the deliverables? No problem because you're not contracting for a deliverable anymore. The federal government is a big beast, though, so it takes a while for the idea to spread.
    - fallingknife 3y ago
      What good does it do to show them a better way? They are a monopoly that takes their revenue by force. What is their incentive to improve efficiency?
      - lazide 3y ago
        That depends on how obvious it is. They can’t be clearly doing it on purpose, or they could be indicted too. If they look like they’re doing their best and following the law, but it doesn’t work? That’s perfectly fine.
- deleted 3y ago
  [deleted]
- _ea1k 3y ago
  Of course not, but the people getting paid the bills also understand how to say "it is a complicated and difficult regulatory environment" frequently. If leadership isn't careful (and it seldom pays to be careful), it becomes an excuse for every real wrong.
jnovek 3y ago
Unfortunately the cost of healthcare.gov has become politicized, but it was extremely, absurdly expensive. Wikipedia says the total cost actually landed somewhere between 1.7b and 2.1b. https://en.wikipedia.org/wiki/HealthCare.gov https://en.wikipedia.org/wiki/HealthCare.gov
- screwturner68 3y ago
  I worked in the "rescue team" for a couple of years and the politics was one of the reasons I left, you never knew when the supreme court would stick their nose in and you'd be out of a job. I used to describe that job as having 100 bosses, 55 of those bosses wanted you to do your job for the least amount of money as possible and the other 45 wanted you to fail.
  - llimllib 3y ago
    and also the terror that you might get hauled into congress to testify
    - screwturner68 3y ago
      I actually used to use that line with I'd talk to some of our vendors support people. They'd start dragging that feet, wanting to get rid of me and I'd say, "you do know what project this is right...." and then I'd say if we can't fix this really quickly it will be on the news and your company will get dragged into it and nobody wants to testify to congress that the market place site was down because of a bug in your equipment. It was pretty amazing how people went above and beyond when you put things in the right perspective.
- winter_blue 3y ago
  The crazy thing is that a small team of around 10 really good engineers could probably have put together a robust reliable working Healthcare.gov-like system that could handle the potential 300 million users in 1-2 years. If you paid all 10 of them them $300k each a year, for 2 years, that's 10 * 300 * 2 = $6 million. Worst case, Healthcare.gov should have cost 5 times that, $30 million. I feel like the sort of egregious and copious waste in government should borderline be illegal.
pstuart 3y ago
My limited understanding of the system is that it is a feature, not a bug. Both sides of the aisle have patrons that feed at this trough and I doubt we'll ever get consensus better operational insights and less waste and fraud. Like so many other issues, the only thing holding us back from achieving this is will.
- tacocataco 3y ago
  Introduce competition into the electoral process by doing away with First Past The Post voting in favor of Ranked Choice voting/Single Transferable vote/STAR voting/Alternative vote. People should be free to vote for someone that best represents them, without worrying about a spoiler effect or strategic voting practices.
  - pstuart 3y ago
    That would be a great start for democracy. Real change requires Congress to do its job, and it's pretty much paralyzed these days. There's plenty of input to consider on how to improve things, we just need to find the collective will to act on it. https://www.brookings.edu/articles/reforming-federal-procurement-and-acquisitions-policies/ https://www.brookings.edu/articles/reforming-federal-procure...
hooverd 3y ago
Unfortunately the federal government is allergic to doing anything in-house.
- fullstop 3y ago
  The US Digital Service does fantastic work. The first head of the project was the person involved in the rescue of Healthcare.gov.
  - hooverd 3y ago
    18F too. Although their name is unfortunate. Maybe they'll open a Cali office someday.
thomastjeffery 3y ago
That's nothing. It's estimated that US healthcare per-capita spending is 20% more than countries who use a single-payer program (that's nearly every other developed country in the world).
- spiralx 3y ago
  Only the UK and Taiwan have a nation-wide single-payer healthcare system, although Canada comes sort of close with each state having its own separate single-payer system. Most countries use some sort of insurance model with varying levels of public/private provision and government and employer contribution. Switzerland is the only country with 100% private healthcare provision, but it offsets that with generous subsidies and having insurers offer a mandatory not-for-profit plan meeting legislated minimums. https://www.vox.com/2020/1/13/21055327/everybody-covered https://www.vox.com/2020/1/13/21055327/everybody-covered

unethical_ban 3y ago

Right? Don't they know software is free and insurance is a simple industry?

ceejayoz 3y ago
It's not free, but the initial contractor CGI definitely hosed taxpayers.
renewiltord 3y ago
The rescue proved something everyone knows: some software engineers suck; others rule.
fallingknife 3y ago
And that is an excuse for spending damn near a billion for this? > Testing was largely a manual process. Releases and deployment required lengthy overnight downtimes and often failed, required equally-lengthy rollbacks. The core data layer lacked a schema. Provisioning hosting resources was slow and not automated. Critically, there was a pervasive lack of monitoring of the site. No APM, no bog-standard metrics about CPU, memory, disk, network, etc., that could be viewed by anyone on the team.
lazide 3y ago
The “beauty” of contracting for the gov’t - if you know how to play the game - is that their internal management and incentive structure is usually so screwed up (and will never get better) that manipulating the customer is easy and lucrative. If you know how to do it without being indicted anyway. Constant back and forth change orders, pointless delays to serve constantly changing stakeholders, constant demos with no real product, no one in the customer with any power who has experience to call BS on timelines or delivery, etc. And if done correctly, everyone in the ‘customer’ is simultaneously happy and angry and so it’s easy to deflect blame and point fingers back into their own org. Or so I’ve heard from folks who are involved. I try to avoid it like a plague, the BS I deal with hurts my soul enough already.
- baggachipz 3y ago
  Having worked for a Beltway Bandit "consulting" company in the past, you are exactly right. It's not about getting work done, it's about spending hours so you can receive more.
- hondo77 3y ago
  That was one of the takeaways from this whole episode. In its wake, the Technology Transformation Services and 18F resulted. Part of what they do is show government agencies a better way of procurement (a "modular" or "agile" approach). Performance-based contracting, and where the source code is, say, in a GitHub repo rather than on the contractor's servers. Contractor not performing? Get rid of them and get someone who will. Need to change the deliverables? No problem because you're not contracting for a deliverable anymore. The federal government is a big beast, though, so it takes a while for the idea to spread.
  - fallingknife 3y ago
    What good does it do to show them a better way? They are a monopoly that takes their revenue by force. What is their incentive to improve efficiency?
    - lazide 3y ago
      That depends on how obvious it is. They can’t be clearly doing it on purpose, or they could be indicted too. If they look like they’re doing their best and following the law, but it doesn’t work? That’s perfectly fine.
deleted 3y ago
[deleted]
_ea1k 3y ago
Of course not, but the people getting paid the bills also understand how to say "it is a complicated and difficult regulatory environment" frequently. If leadership isn't careful (and it seldom pays to be careful), it becomes an excuse for every real wrong.

jnovek 3y ago

Unfortunately the cost of healthcare.gov has become politicized, but it was extremely, absurdly expensive. Wikipedia says the total cost actually landed somewhere between 1.7b and 2.1b. https://en.wikipedia.org/wiki/HealthCare.gov https://en.wikipedia.org/wiki/HealthCare.gov

screwturner68 3y ago
I worked in the "rescue team" for a couple of years and the politics was one of the reasons I left, you never knew when the supreme court would stick their nose in and you'd be out of a job. I used to describe that job as having 100 bosses, 55 of those bosses wanted you to do your job for the least amount of money as possible and the other 45 wanted you to fail.
- llimllib 3y ago
  and also the terror that you might get hauled into congress to testify
  - screwturner68 3y ago
    I actually used to use that line with I'd talk to some of our vendors support people. They'd start dragging that feet, wanting to get rid of me and I'd say, "you do know what project this is right...." and then I'd say if we can't fix this really quickly it will be on the news and your company will get dragged into it and nobody wants to testify to congress that the market place site was down because of a bug in your equipment. It was pretty amazing how people went above and beyond when you put things in the right perspective.
winter_blue 3y ago
The crazy thing is that a small team of around 10 really good engineers could probably have put together a robust reliable working Healthcare.gov-like system that could handle the potential 300 million users in 1-2 years. If you paid all 10 of them them $300k each a year, for 2 years, that's 10 * 300 * 2 = $6 million. Worst case, Healthcare.gov should have cost 5 times that, $30 million. I feel like the sort of egregious and copious waste in government should borderline be illegal.

pstuart 3y ago

My limited understanding of the system is that it is a feature, not a bug. Both sides of the aisle have patrons that feed at this trough and I doubt we'll ever get consensus better operational insights and less waste and fraud. Like so many other issues, the only thing holding us back from achieving this is will.

tacocataco 3y ago
Introduce competition into the electoral process by doing away with First Past The Post voting in favor of Ranked Choice voting/Single Transferable vote/STAR voting/Alternative vote. People should be free to vote for someone that best represents them, without worrying about a spoiler effect or strategic voting practices.
- pstuart 3y ago
  That would be a great start for democracy. Real change requires Congress to do its job, and it's pretty much paralyzed these days. There's plenty of input to consider on how to improve things, we just need to find the collective will to act on it. https://www.brookings.edu/articles/reforming-federal-procurement-and-acquisitions-policies/ https://www.brookings.edu/articles/reforming-federal-procure...

hooverd 3y ago

Unfortunately the federal government is allergic to doing anything in-house.

fullstop 3y ago
The US Digital Service does fantastic work. The first head of the project was the person involved in the rescue of Healthcare.gov.
- hooverd 3y ago
  18F too. Although their name is unfortunate. Maybe they'll open a Cali office someday.

thomastjeffery 3y ago

That's nothing. It's estimated that US healthcare per-capita spending is 20% more than countries who use a single-payer program (that's nearly every other developed country in the world).

spiralx 3y ago
Only the UK and Taiwan have a nation-wide single-payer healthcare system, although Canada comes sort of close with each state having its own separate single-payer system. Most countries use some sort of insurance model with varying levels of public/private provision and government and employer contribution. Switzerland is the only country with 100% private healthcare provision, but it offsets that with generous subsidies and having insurers offer a mandatory not-for-profit plan meeting legislated minimums. https://www.vox.com/2020/1/13/21055327/everybody-covered https://www.vox.com/2020/1/13/21055327/everybody-covered