6 ms·
Healthcare.gov technology stack (infographic)
- nathas 13y ago$293M.... I'm just consistently angry over how much this cost. There are so many moving pieces, and while I understand that, $293M for one piece. Half a billion for a website. Christ.
- segmondy 13y agoSorry, but you obviously have no idea what you are talking about when you call it, "a website" Or is google just a website? Is yahoo.com just a website? Did you look at the infograph? How much do you think it will take to talk to 5 agencies and 300 insurance providers that have 4500+ health plans, and get them to agree/adhere, understand their EXISTING system and build a site? I'm not justifying the expense, but let's not trivialize the task at hand and call it just a website.
- ddoolin 13y agoPresumably the interfacing with insurance providers is done on the backend (not included in the $295M). It's possible that QSSI took all the content from the providers and standardized it for display in the front, near trivializing what you mention. Just a hunch, and not saying it isn't complex, just not $295M complex.
- dm2 13y agoSeriously, that number is staggering. CGI Federal literally ripped off the government, screwed the American people badly, and "stole" hundreds of millions. They should never be allowed to work on another project again and should be required to refund the government. The project managers should be fired and publicly humiliated. I STILL cannot even login to the website. The site has said "The system is down at the moment." for the past two weeks. The only reasonable explanation that I can imagine is sabotage by someone who didn't want Obamacare to succeed, the same thing that happened in Congress recently.
- joering2 13y ago> The only reasonable explanation I can imagine is sabotage by someone who didn't want Obamacare to succeed It depends how you define "success". While most people assume success as a well-oiled healthcare machine that is affordable and cares about you then yes, they have failed. However, if you have done your homework on Obama and Democratic party (in biggest part), you will understand that they already greatly succeed - that is they shoved a huge tax (that over 10 years grow to 10% of your income, starting with 1% next year) in american's throat. While congress pass the law and Supreme Court upheld it, rest is just a statistic noise. Who care congress has 11% approval? Who care Obama has vanishing approval rate and his administration cannot catch up with any "phoney" scandals they getting into every month it seems? Its all irrelevant. After all, they did not hire 16,000 new physicians; they hired 16,000 new IRS workers that will be responsible for collecting what you lawfully owe. That's all! Its all about the taxes; its never been about you or your health. The only way Obama ACA would be more successful is that every single american throw a towel and say "screw it, I will pay the fine, this is too complicated!". That would be the best scenario for the Obama and the gov. > the same thing that happened in Congress recently. I think you mean the shutdown. It was so pathethic that it is funny. Imagine situation when you drive to work every day. And then every weekend you go out and drink and drive. The cop pulled you over twice but he let you go with a warning. You keep drinking and driving. Then one day he pulled you over, boom! DUI, license revoked. Then you go to your friends and family and you complain that because of this cop now you cannot do your work, because you drove to the office every day. If you agree with this reasoning then yes Reps were responsible for shutting down the Gov and not letting Congress do their work.
- Iftheshoefits 13y agoI think you misunderstand what goal government contractors, and the government officials that manage their contracts, are working toward. It's not a full-featured, working product. It's money. The contractors want more contract money. The government workers want visibility and job security. These things "work" in the contracting system by having contractors that play the same game the government workers play: which is needlessly and pointlessly extending contracts, mainly by fucking things up while fulfilling the technical letter of the contract.
- 31reasons 13y agoIf you hire 10 web developers fulltime for 2 years, it would come down to ~$7000 hourly rate per developer.
- pan69 13y ago10 web developers couldn't even remotely come close to building/implementing a system like this. This is not a web development project but an enterprise integration project. What you see as "the website" is just the tip of the ice berg. It's like saying that a car is just a steering wheel and four wheels because that's all you "see" but you don't know what's going on "under the hood". However, I do agree that the costs for this are extravagant and there are certainly signs of "government milking" visible here.
- 31reasons 13y agoSeriously ? Give me specifics where exactly do you think $300 million go? Look at the slides its not for the complete system but the "front end" of it. Most of the software they use is Open Source, Cloud services/hardware is dirt cheap and Software Developer salaries are not that high.
- showerst 13y agoNot that it becomes a reasonable number, but according to the Post[1], it's more like 175 million. So 20% of a billion dollars... http://www.washingtonpost.com/blogs/fact-checker/wp/2013/10/24/how-much-did-healthcare-gov-cost/ http://www.washingtonpost.com/blogs/fact-checker/wp/2013/10/...
- spoiledtechie 13y agoThats just the contract cost.... Numbers are estimated near $340M and the GAO just came out with current costs sitting at $320M but harder liners are coming out with numbers near $620M. By the time its done and fixed, its estimated at $1B. Put that in your head...
- us0r 13y agothats just the federal part. CA, HI, WA, KY, and many other states are hundreds of millions (billions) more.
- hpritch5 13y agoToo bad I didn't get this contract...
- talmand 13y agoI know what you mean, I could've totally made a failing website for $100 million max. I would have saved them money.
- icn2 13y agoFor government projects, sometimes they don't want to save money at all. They are more interested in how money can get in someone's pocket.
- talmand 13y agoFine then, my budget for the failing website is now $500 million. Savings be damned! Oh crap, sorry, inflation. Price tag's now $750 million.
- us0r 13y agoToo bad it wasnt even bid out.
- guiomie 13y ago"I can tell that this system consists of a JBoss Application Server with data access components and RESTful web services developed using Java. Given my prior experience with JBoss and Java, although they are great for middleware development, they’re known to be a bit slow." I always had this non founded impression that something running on the JVM would be slow ... this author also. Anyone with data to back this impression ?
- redthrowaway 13y agoIt has nothing to do with "running on the JVM". The JVM is about as fast as you'll get any VM offering GC and running an IR like Java bytecode. It's only slow compared to compiled, native code (and even then nowhere near as relatively slow as it used to be), which no competing tech stack would be doing. It's much faster than Ruby, Python, etc. That said, my experiences with JBoss haven't left me enamoured of it, and it certainly wouldn't be my first choice.
- curveship 13y agoIn techempower's web framework benchmarks[1], Java-base frameworks top most categories. One C++-based framework, CPoll, gets a few percentage points more in a few tests, but Java servlets are the highest performing across all tests. (As always with micro-benchmarks, take all this with a grain of salt, but at least it's some data.) 1 - http://www.techempower.com/benchmarks/#section=data-r6 http://www.techempower.com/benchmarks/#section=data-r6
- edwinnathaniel 13y ago"Anyone with data to back this impression ?" No. Because it's not the AS that is slow. Someone else's code that is not efficient enough or doing too much DB calls.
- tmarthal 13y agohttp://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/ There are a ton of Java frameworks in the top performing benchmarks (See Lng:Jav). Whatever conclusion you've made from Java Swing applications running on top of WindowsXP machines in the early 00's are not applicable to modern web server performance.
- patrickg_zill 13y agoA friend analyzed the page and noted that they didn't minify or combine anything. "One obvious reason is that they're serving unminified and unconcatenated Javascript and CSS files. A single page load causes 92 requests to the server, and they're not using a CDN for any of that. They're also not setting the Cache-Control and Expires headers, so the browser isn't caching many of those files, including on 80k file full of response strings. For the non-technical, this means the site is essentially killing itself. An optimized and properly tuned production website would load one (1) HTML document, one (1) CSS document, and one (1!) JS document, they're loading 92. "
- deleted 13y ago[deleted]
- evan_ 13y agoTrue, but it's not like minifying and serving from a CDN will make anything worse, and it could improve things quite a bit.
- WalterSear 13y agoIt is, however more than just a smell: a front end of such an egregious turditude doesn't just uninspire confidence in any of the system - these poor practices demonstrate that whoever is in charge of the checks and balances hasn't done their job.
- thejerz 13y agoWhile that is an egregious amount of requests, it's not the reason the site is broken. Based on my experience as a user and poking around quickly in Firebug, the issues are largely server-side.
- patrickg_zill 13y agoTrue, it is just an additional data point. You have to wonder, if they can't minify their JS or even run the pages through the Yahoo YSlow tool, what is the quality of the backend?
- pgrote 13y agoWhere are the links to IRS and social security admin? During the sign up process they verify tax return status and ssns of dependents including children? Is that what CMS does?
- skyraider 13y agoI think the interesting lesson here is that even marketplaces should be competitively constructed - i.e., there should be significant price-competitive pressure to deliver goods and services; just letting the government to facilitate marketplaces removes price competition from the marketplace itself. I'm not suggesting that the government create a meta-marketplace for health insurance marketplaces. Just that it isn't as efficient as some people think to introduce a powerful federal competitor with 'unfair advantages' like public funding into the health insurance selection market.
- ngcazz 13y agoThe government should have had specified the APIs and see what vendors created the best implementations.
- icn2 13y agoAt least they didn't use windows/.net stack this time.
- sourc3 13y agoWhat makes you say that? Having built systems in this space with very large number of users using the .net platform, I wonder why you stated this. I am a supporter of open and non-MS software but I don't think we can blame it on the technology this time. In my experience so far a well engineered system, will perform regardless of the technology stack picked. Picking technology X or Y will not compensate for bad architecture and engineering practices.
- newsreader 13y agoCouldn’t agree with you more.
- newsreader 13y agoAmen to that. This thread and all others that mention the healthcare.gov debacle would go on forever talking about MS and ignoring the fundamental issues of bad architecture and government waste.
- Iftheshoefits 13y agoIt's not just government waste. The private, for-profit companies that are awarded these contracts are part of the problem. It's not like "government waste" is even the biggest problem here. The biggest problem is the private companies involved in this. Not the government. Although all parties deserve some blame, for sure, because they have a "scratch my back, I'll scratch yours" approach.
- newsreader 13y agoI agree that all parties deserve some blame, but spreading the blame around does not excuse the amount of money the government spent on this and other projects such as recovery.gov. For profit private companies are in it to make money so I don’t blame them for taking the money; I blame them for bad design and architecture.
- usaphp 13y ago$293Mln...Ok so it's like hiring a team of 100 web developers, pay each of them $100K a year, and they will work non-stop for 30 years... Edit: Working non stop for 30 years.
- patmcc 13y agoYou're off by a factor of 10, but yes, it's terrible.
- edwinnathaniel 13y agoDon't forget: - Software licenses - Software support/contracts (Redhat -> JBoss AS) - Hardware - Office space - Benefits - Infrastructure contracts (for 10 years... or 100 years)
- dm2 13y agoRealistically you would need to double that 100k to pay for other expenses (hardware, benefits, office, management). Even then, CGI Federal screwed the American people and should be fined for what they've done.
- snowwrestler 13y agoThe project almost certainly had thousands (multiple) working on it.
- ryanobjc 13y agoPeople appear to be missing the point that _the website_ is literally the smallest part of the project. While it's easy for people to crap on things like js-minifying, cache control, etc - the hard reality is the stuff that is the hardest is so hard that most people have never done anything even remotely close. Having worked on 10m+ loc codebases, life gets very hard. Dynamic languages tend to be unworkable. Module control, strong interfaces, and a lot of division of responsibility becomes the order of the day. And just figuring out how to build the thing becomes it's own team. For those who are just thinking "so, just slap together some JSON/web services and call it a day" - I invite you to research EDI and then report back here. Assuming you're still alive.
- splawn 13y agoIf only there was a way to remove the hundreds of private insurance companies they have to interface with. I assume they each have their own flavor of EDI, greatly adding to the complexity of this thing. (I have had to write interop EDI stuff with some insurance companies a while back and iirc they were snowflakes)
- daigoba66 13y agoGiven the complexity of the system, the most shocking thing to me is that how little time they had to actually build it. And from what's been made public there was virtually no time for testing (load testing was certainly absent). It should been "turned on" months before the public go-live to iron out all of these issues.
- spikels 13y agoThat's one thing I never understood. The law was passed 3.5 years before they had to go live. Seems like they should have had plenty of time but it seems like almost all of the coding was done in the past year with little testing along the way.
- mhurron 13y ago> I invite you to research EDI and then report back here. Assuming you're still alive. Which EDI standar ... ohhh I see what you're getting at. When you look at it like a single point aggregating all this information from all these separate insurance companies, it's amazing it works at all. When you look at the website failing the way it is though, you can't help but ask 'did they even test this thing' even when you know better. When you look at it from a user point of view, it's too bad it appears to not work at all.
- wil421 13y agoThe problem isnt the part that you can see, the Front-End. The problem is the legacy systems on the Back-End that need to talk to each other. And if one of these requests time out, the whole request fails, doesnt matter if the 5 systems before responded it has to go through the process again.
- neurotixz 13y agoThe web part is just a tiy part of a major project. Extracting informations from 55 databases/unrelated systems in a coherent manner is a nightmare by definition. The number of different 3rd party libraries that are required, the type incompatibilities, matching fields, accounting for changes in the schema, and talking to legacy systems (I am sure there are many of them in there...), all in 5 months? Good luck. Add to this the overhead of talking to IT and management of insurance companies, the IRS and others, and just the fact that it does answer some queries is fairly impressive in my book. Also I am sure that most of those systems are failry deep inside the corporate networks of these insurers, just getting access to them probably was not an easy task. That said the authentication layers failures and lack of testing are far from impressive.