7 ms·

California’s $327 million web site in operation

94 points by tjr 13y ago

vehementi 13y ago
Is there some balance sheet that explains how it cost $327M? Is this some massive contract awarded to some goverment guy's buddy?
- jbooth 13y ago
  It's actually the opposite. All of the regulations preventing it from going to some guy's buddy mean that only bloated, f'd up bureaucracies can clear the hurdles to land the gig. And in the end, it still helps to have some buddies, of course.
  - wpietri 13y ago
    Yes. The risks of doing a government contract are enormous. Once an old family friend with a lot of state government contracting experience brought me in to help out with some technical stuff. One of the clauses in the contract basically said that they could just decide not to pay me. After I got done spluttering, it was explained to me that they needed the clause in case of certain weird emergencies that never happened. (I can't remember now, but I imagine government shutdown is one of those cases.) So I said, "Whatever," and signed, because I could afford to take the risk. But there is no way I'd do that for a substantial project. The bigger the client, the bigger the crazy. To be able to take $5m gambles, I'd want to be doing $50-100m/year. Which means I'm running a bloated, f'd-up bureaucracy for sure. No thanks; if I need more crazy in my life, I'll just go read some comments on YouTube or newspaper sites.
- r0h1n 13y ago
  Accenture seems to be the biggest contractor - http://blogs.law.harvard.edu/philg/2012/09/15/californias-state-government-builds-itself-a-327-million-web-site/ http://blogs.law.harvard.edu/philg/2012/09/15/californias-st...
  - tootie 13y ago
    Having worked with Accenture on a government project in the past, they will milk CA dry and staff up with an army of worthless college grads at inflated bill rates and send their profits to the Cayman Islands to avoid taxes.
ck2 13y ago
It's amazing how even technical minded folks don't forgive launch day bugs and initial traffic overload. That said, some of these bugs definitely should have been easily spotted during beta testing. But I too would like to see an itemization of $327 million.
- timr 13y ago
  Initial reports were that insurance.gov got 1M uniques before 8AM on launch day. One can imagine that California's exchange site was also slammed. I'm not going to argue that this project's budget wasn't bloated, but it's pretty naive to think that your average low-rent startup team could handle it any better. Doing tens of millions of uniques on launch day is a challenge for any team. Most commercial websites with that much traffic have technology budgets in the tens of millions of dollars, if not more.
  - ck2 13y ago
    Plus given the politics around it, we don't know what devious behavior might have been going on against it either. Hopefully none, but you never know. Their stats may have been javascript based and not physical server hits.
bluecalm 13y ago
Sounds unreal, anybody have any details justifying it ?
- VMG 13y ago
  They don't need to. Their customers are unable to switch.
timje1 13y ago
Is there anything being done here that looks like it should cost more than 500k ? ... Look over here, I've got a bridge to sell that I think you're going to love...
jameswyse 13y ago
Is this the correct site? https://v.calheers.ca.gov/ https://v.calheers.ca.gov/ Come Back Soon Due to the success of the Covered California Marketplace we are experiencing higher than expected users. To make sure everyone has a great experience while on the site, we need you to come back in a little bit We're sorry for the inconvenience and we look forward to your return in a few hours.
- Domenic_S 13y ago
  > we are experiencing higher than expected users. Dammit, Californians, stop smoking all that pot so we can get to the obamacare website!
- deleted 13y ago
  [deleted]
steveplace 13y ago
I think I found their budget plan. Very, very big numbers. http://www.healthexchange.ca.gov/BoardMeetings/Documents/CHBE_Level2Grant_BudgetNarrative.pdf http://www.healthexchange.ca.gov/BoardMeetings/Documents/CHB...
- conover 13y ago
  688 Program Technician III... 88 Associate Gov. Program Analyst... 73 Office Assistant... Numbers are FTE
- wpietri 13y ago
  To be fair to them, California's population is 38 million. Divide the budgets by that number to get a per-capita cost and it doesn't seem crazy to me. As an aside the way we citizens run our government, budget bloat is guaranteed. We complain a little about bloat, but we complain a lot when things don't go perfectly, and we also go looking for individuals to blame. The strong incentive is to be very risk-averse. That's especially true in organizations, like government, where stability is prized and careers are long. I really appreciate a lot of the work that government people do, but I would never put up with half the bullshit they do.
wehadfun 13y ago
How much health insurance could have simply been bought with $327 million?
- Timothee 13y ago
  Probably not much actually. Google tells me that California's population is about 38 million people. So, on average, you get less than $10 per person as a one-time "credit" (or however you'd call this). Even if only 1% of people needed coverage, they could get $1,000 worth of coverage which doesn't take you very far. I'm not saying that that budget is reasonable or unreasonable, but that puts the amount in perspective a little bit.
- deleted 13y ago
  [deleted]
smalldaddy 13y ago
Government business is great when you can get it. The problem is navigating the purchasing process. In this case I like to think of it as stimulus for the economy.
- justinsteele 13y ago
  Everyone's tax money goes to a select few who screwed the state over (or.. went through their loopholes successfully, however you want to look at it). None of that sounds like a good stimulus to me.
kylelibra 13y ago
Apparently I need to find some government institutions for clients.
- sybhn 13y ago
  That was my conclusion too ;-) Why do we try so hard right?
  - 31reasons 13y ago
    Mine as well. Seems pretty lucrative.
- dc_ploy 13y ago
  Join the rat race. Just make sure to either of the following: black, women, veteran, disabled if you want to go out on your own. If not, just sub contract from a minority owned business.
Yhippa 13y ago
Storytime! Who wants to hear another story of going all-in on poor State IT management? http://www.inthepublicinterest.org/case/virginia-statewide-information-technology-system http://www.inthepublicinterest.org/case/virginia-statewide-i...
robertwalsh0 13y ago
I'd love to see a collection of how much these sites cost to build per state.
ogig 13y ago
My small province wasted 1 million Euros in a bad looking, useless, magento site that will be close after next elections. This is how I imagine it works here in my local administration: -EU just sent 1 million euros for helping us out! -What's destined for? -Technological development -Ok, i'll call (my cousin) Jon, and ask him to make a website for 750.000, you go and call (your brother) Mike and task him to make an iPhone app for 250.000. -Good, what will be the site for? -Mmmm, whatever Jon wants, does it matter? -Guess not... Lets go have a drink a celebrate our good job? -Deal.
ams6110 13y ago
State IT project. Unbelievably overstaffed, massive budget. Mediocre results, if not outright failure. Par for the course. Nobody should be surprised.
timr 13y ago
Lots of snark here; very little thoughtful commentary. If the document that steveplace posted [1] is accurate, the budget for this program includes things like the 800+ person call center to handle consumer requests, and the staff to figure out what kinds of health plans to offer in the first place. The line item for "technical" is $161M, which is only 23% of the budget. I'm not going to suggest that's a low number for this kind of project, but then again, it's kind of hard to make it all about the horribleness of public projects when 94% of the technical budget is going to "contractual services". It looks like the software was mostly done by private contractors. In other words: let's not allow facts and thoughtful discourse to get in the way of a good government bashing! [1] http://www.healthexchange.ca.gov/BoardMeetings/Documents/CHBE_Level2Grant_BudgetNarrative.pdf http://www.healthexchange.ca.gov/BoardMeetings/Documents/CHB...
- arbuge 13y ago
  Well, $161m is more than 23% of $327m to start with. The other thing is that although the government is using private contractors, that does not mean the latter are solely to blame. If the government ends up paying millions of dollars for a wordpress site sporting a $40 theme, the contractors charging for it may be immoral, but the government who awarded them the job is at best massively incompetent.
  - timr 13y ago
    "Well, $161m is more than 23% of $327m to start with." Read the linked document instead of just relying on Greenspun's numbers. The $327m number he cites appears to be incorrect.
- tootie 13y ago
  That's all true and the $327M price tag is misleading, but they still spent on the order of several million on the actual buildout of the website and fucked it up. It does not inspire confidence that the rest of the money was well spent.
  - timr 13y ago
mikestew 13y ago
At least CA's site is up and running. WA's site (https://www.wahealthplanfinder.org https://www.wahealthplanfinder.org) has been down since the first. My project for down moments today is to try and determine how much it cost to develop. (Oops, I guess it's up now, just slow, and has an invalid cert.)
deleted 13y ago
[deleted]

94 points by tjr 13y ago

Is there some balance sheet that explains how it cost $327M? Is this some massive contract awarded to some goverment guy's buddy?

jbooth 13y ago
It's actually the opposite. All of the regulations preventing it from going to some guy's buddy mean that only bloated, f'd up bureaucracies can clear the hurdles to land the gig. And in the end, it still helps to have some buddies, of course.
- wpietri 13y ago
  Yes. The risks of doing a government contract are enormous. Once an old family friend with a lot of state government contracting experience brought me in to help out with some technical stuff. One of the clauses in the contract basically said that they could just decide not to pay me. After I got done spluttering, it was explained to me that they needed the clause in case of certain weird emergencies that never happened. (I can't remember now, but I imagine government shutdown is one of those cases.) So I said, "Whatever," and signed, because I could afford to take the risk. But there is no way I'd do that for a substantial project. The bigger the client, the bigger the crazy. To be able to take $5m gambles, I'd want to be doing $50-100m/year. Which means I'm running a bloated, f'd-up bureaucracy for sure. No thanks; if I need more crazy in my life, I'll just go read some comments on YouTube or newspaper sites.
r0h1n 13y ago
Accenture seems to be the biggest contractor - http://blogs.law.harvard.edu/philg/2012/09/15/californias-state-government-builds-itself-a-327-million-web-site/ http://blogs.law.harvard.edu/philg/2012/09/15/californias-st...
- tootie 13y ago
  Having worked with Accenture on a government project in the past, they will milk CA dry and staff up with an army of worthless college grads at inflated bill rates and send their profits to the Cayman Islands to avoid taxes.

ck2 13y ago

It's amazing how even technical minded folks don't forgive launch day bugs and initial traffic overload. That said, some of these bugs definitely should have been easily spotted during beta testing. But I too would like to see an itemization of $327 million.

timr 13y ago
Initial reports were that insurance.gov got 1M uniques before 8AM on launch day. One can imagine that California's exchange site was also slammed. I'm not going to argue that this project's budget wasn't bloated, but it's pretty naive to think that your average low-rent startup team could handle it any better. Doing tens of millions of uniques on launch day is a challenge for any team. Most commercial websites with that much traffic have technology budgets in the tens of millions of dollars, if not more.
- ck2 13y ago
  Plus given the politics around it, we don't know what devious behavior might have been going on against it either. Hopefully none, but you never know. Their stats may have been javascript based and not physical server hits.

bluecalm 13y ago

Sounds unreal, anybody have any details justifying it ?

VMG 13y ago
They don't need to. Their customers are unable to switch.

timje1 13y ago

Is there anything being done here that looks like it should cost more than 500k ? ... Look over here, I've got a bridge to sell that I think you're going to love...

jameswyse 13y ago

Is this the correct site? https://v.calheers.ca.gov/ https://v.calheers.ca.gov/ Come Back Soon Due to the success of the Covered California Marketplace we are experiencing higher than expected users. To make sure everyone has a great experience while on the site, we need you to come back in a little bit We're sorry for the inconvenience and we look forward to your return in a few hours.

Domenic_S 13y ago
> we are experiencing higher than expected users. Dammit, Californians, stop smoking all that pot so we can get to the obamacare website!
deleted 13y ago
[deleted]

steveplace 13y ago

I think I found their budget plan. Very, very big numbers. http://www.healthexchange.ca.gov/BoardMeetings/Documents/CHBE_Level2Grant_BudgetNarrative.pdf http://www.healthexchange.ca.gov/BoardMeetings/Documents/CHB...

conover 13y ago
688 Program Technician III... 88 Associate Gov. Program Analyst... 73 Office Assistant... Numbers are FTE
wpietri 13y ago
To be fair to them, California's population is 38 million. Divide the budgets by that number to get a per-capita cost and it doesn't seem crazy to me. As an aside the way we citizens run our government, budget bloat is guaranteed. We complain a little about bloat, but we complain a lot when things don't go perfectly, and we also go looking for individuals to blame. The strong incentive is to be very risk-averse. That's especially true in organizations, like government, where stability is prized and careers are long. I really appreciate a lot of the work that government people do, but I would never put up with half the bullshit they do.

wehadfun 13y ago

How much health insurance could have simply been bought with $327 million?

Timothee 13y ago
Probably not much actually. Google tells me that California's population is about 38 million people. So, on average, you get less than $10 per person as a one-time "credit" (or however you'd call this). Even if only 1% of people needed coverage, they could get $1,000 worth of coverage which doesn't take you very far. I'm not saying that that budget is reasonable or unreasonable, but that puts the amount in perspective a little bit.
deleted 13y ago
[deleted]

smalldaddy 13y ago

Government business is great when you can get it. The problem is navigating the purchasing process. In this case I like to think of it as stimulus for the economy.

justinsteele 13y ago
Everyone's tax money goes to a select few who screwed the state over (or.. went through their loopholes successfully, however you want to look at it). None of that sounds like a good stimulus to me.

kylelibra 13y ago

Apparently I need to find some government institutions for clients.

sybhn 13y ago
That was my conclusion too ;-) Why do we try so hard right?
- 31reasons 13y ago
  Mine as well. Seems pretty lucrative.
dc_ploy 13y ago
Join the rat race. Just make sure to either of the following: black, women, veteran, disabled if you want to go out on your own. If not, just sub contract from a minority owned business.

Yhippa 13y ago

Storytime! Who wants to hear another story of going all-in on poor State IT management? http://www.inthepublicinterest.org/case/virginia-statewide-information-technology-system http://www.inthepublicinterest.org/case/virginia-statewide-i...

robertwalsh0 13y ago

I'd love to see a collection of how much these sites cost to build per state.

ogig 13y ago

My small province wasted 1 million Euros in a bad looking, useless, magento site that will be close after next elections. This is how I imagine it works here in my local administration: -EU just sent 1 million euros for helping us out! -What's destined for? -Technological development -Ok, i'll call (my cousin) Jon, and ask him to make a website for 750.000, you go and call (your brother) Mike and task him to make an iPhone app for 250.000. -Good, what will be the site for? -Mmmm, whatever Jon wants, does it matter? -Guess not... Lets go have a drink a celebrate our good job? -Deal.

ams6110 13y ago

State IT project. Unbelievably overstaffed, massive budget. Mediocre results, if not outright failure. Par for the course. Nobody should be surprised.

timr 13y ago

Lots of snark here; very little thoughtful commentary. If the document that steveplace posted [1] is accurate, the budget for this program includes things like the 800+ person call center to handle consumer requests, and the staff to figure out what kinds of health plans to offer in the first place. The line item for "technical" is $161M, which is only 23% of the budget. I'm not going to suggest that's a low number for this kind of project, but then again, it's kind of hard to make it all about the horribleness of public projects when 94% of the technical budget is going to "contractual services". It looks like the software was mostly done by private contractors. In other words: let's not allow facts and thoughtful discourse to get in the way of a good government bashing! [1] http://www.healthexchange.ca.gov/BoardMeetings/Documents/CHBE_Level2Grant_BudgetNarrative.pdf http://www.healthexchange.ca.gov/BoardMeetings/Documents/CHB...

arbuge 13y ago
Well, $161m is more than 23% of $327m to start with. The other thing is that although the government is using private contractors, that does not mean the latter are solely to blame. If the government ends up paying millions of dollars for a wordpress site sporting a $40 theme, the contractors charging for it may be immoral, but the government who awarded them the job is at best massively incompetent.
- timr 13y ago
  "Well, $161m is more than 23% of $327m to start with." Read the linked document instead of just relying on Greenspun's numbers. The $327m number he cites appears to be incorrect.
tootie 13y ago
That's all true and the $327M price tag is misleading, but they still spent on the order of several million on the actual buildout of the website and fucked it up. It does not inspire confidence that the rest of the money was well spent.
- timr 13y ago

mikestew 13y ago

At least CA's site is up and running. WA's site (https://www.wahealthplanfinder.org https://www.wahealthplanfinder.org) has been down since the first. My project for down moments today is to try and determine how much it cost to develop. (Oops, I guess it's up now, just slow, and has an invalid cert.)

deleted 13y ago

[deleted]