6 ms·
Health Exchange Tech Problems Point To A Thornier Issue
- niels_olson 13y agoI work at a DoD hospital. Trying to do some research. I have been told 1) Python is not allowed 2) Open source software is insecure 3 Perl is allowed 4) there is a list of approved software. 5) I can't see it 6) Enthought is on at least one version of the list 7) I can verify there are several python projects on forge.mil
- jessaustin 13y agoAre these rules in force on any host connected to the network? Do they have different networks with differing rules? How are the rules enforced? (That is, I can imagine switches and firewalls enforcing network rules, but "don't install python"?)
- kreitje 13y agoThey do have different networks with different rules. When I was on the Michigan DOD network, we had the Microsoft Document Image Viewer. When I was deployed, the DOD network there did not have the Microsoft Document Image Viewer on the "approved" list and therefore could not be installed. It was such a pain working with TIFFs without it.
- staunch 13y ago#3 solves all your problems then :-)
- deleted 13y ago[deleted]
- niels_olson 13y agoAre you serious? Is there a perl implementation of python? Did I mention I am having to formally present to the executive steering committee to get permission to run perl?
- staunch 13y agoHah. Sounds terribly bureaucratic. But yes, I'm serious. It may not be trendy, but there's nothing Python can do that Perl can not. The differences are fairly cosmetic. The key is to write good quality Perl, so it's fun: • http://modernperlbooks.com/books/modern_perl/chapter_00.html http://modernperlbooks.com/books/modern_perl/chapter_00.html • http://my.safaribooksonline.com/book/programming/perl/0596001738 http://my.safaribooksonline.com/book/programming/perl/059600...
- frozenport 13y ago@2 1. Maybe they know of vulnerabilities that you don't. Remember the bug in Debian where cryptography was being seeded by the process id? 2. They might have put a lot of time into ensuring that one particular product was 100% secure and trust only that one. This is often the case for hardware, where vetting for security compliance can be difficult.
- jaggederest 13y agoHa, sounds a lot like most big companies. Why get and train competent staff to build a software development process when you can pay 10x as much to a vendor for software that doesn't really work.
- maaku 13y agoBecause in many cases hiring competent staff (aka civil servants) requires an act of congress.
- vinhboy 13y agoI have never worked on a high traffic site before, so I am just mostly talking out of my ass. But when I looked at the html source for the coveredca website, it had a lot of of basic problems: there were 3 DOCTYPES on one page, developer comments were still visible (you can actually see the names of all the developers who worked on the project), Java error pages were visible, they disabled the back button completely by enforcing javascript refreshing, they did not minify any of their assets (which would also remove the developer comments) AND they were actually loading many copies of the same assets multiple times because, like I said earlier, their site had 3-DOCTYPES in one html page. As someone who has never worked on a huge site, I don't want to naively say that it could have been done better. But I feel like those are basic issues that could have easily saved them a lot of bandwidth. Also, it seems to me that if they had so many issues on the front-end, it's a hint that maybe they also had many similar issues in the backend as well.
- ericd 13y agoYeah, the CA version (https://www.coveredca.com/ https://www.coveredca.com/) had so many visual glitches and various problems (404s and 500s), I threw up in my mouth a little when I found out that Accenture charged the state over 180M to build it and another 174M to operate it for four years. I don't know how that's possible.
- scarmig 13y agoAre you serious? Christ.
- akjetma 13y agoDo you have a source for those numbers? I need to vomit, but I can't without a source. edit: Nevermind: http://newsroom.accenture.com/news/accenture-chosen-to-implement-california-health-insurance-exchange.htm http://newsroom.accenture.com/news/accenture-chosen-to-imple... Vomit everywhere.
- frozenport 13y agoI had a job interview with Accenture that had 2 parts a dinner and a technical chat. 1. At the dinner their representative spoke about how life at Accenture was a breeze and how as a undergraduate he spent most of this time drinking. He drank on the weekdays but his favorite time to drink was on Sundays. He said he used to do some SQL but now he had moved onto the business side of things. One of the candidates told us how his hobby of skydiving let him see the big picture and another told us that his desi background would enabled to come up culture specific features for software services. I probed this question further, and he said that in India showing your muscles was considered hubris, while in America it made you look strong. On his facebook page he wouldn't show his bare chest... Nobody at the table `looked` like they could do any programming work. 2. During the technical interview I was asked what `clubs` I was in. I told them I wasn't in many clubs and instead spent my time doing research work (which by that time had yield publications) and administrating websites. I told them my hobby was programming. I wanted to tell them how I dealt with EC2 sight outages, worked with clients and their clients to write WordPress plugins. I wanted to talk about my experience engineering our production and outsourcing workflow. I wanted to summarize the techniques I learned overseeing our production staff. They didn't comprehend and asked he what `clubs` I was in. I was scheduled to have 3 interviews, I walked out after the first. Don't work for Accenture, they are a joke because their employees are a joke, which is much in agreement with this article. They also have 275,000 employees.
- ars 13y ago"that contractors built a system that could handle 50,000 people using the site at once" There are over 100 million families in the US and they build for 50,000??? Is that number at least 50,000 pages / second? (Which would be reasonable.)
- smartician 13y agoI think 50,000 concurrent users is actually a fairly reasonable number, given that not all US households are in the market for health insurance on the exchange, and that not everyone will be using the site at the same time.
- ars 13y agoYes, 50,000 pages per second is quite reasonable. But I suspect that's not what the number means here.
- deleted 13y ago[deleted]
- warfangle 13y agoConcurrent, perhaps. But a thing like this? You're going to see spikes of many, many times more than that opening [insert time period]. See also: the Slashdot Effect, the Digg Effect, et cetera.
- dangrossman 13y agoEven if they could theoretically make a site that can handle every person in the country using it at exactly the same time, it'd still be bottlenecked by the much older systems it relies on at the IRS, SS, VA, DOD, DHS, etc. As the article alluded to, the Healthcare.gov site checks your family size, AGI, social security benefit status, citizenship status, incarceration status, veteran status and enrollment in other government health programs during the signup process to determine what you're eligible for.
- 13y ago
- PaulHoule 13y agoMore generally the problem is a lack of standards in technology management. Malpractice is endemic in the IT space and nobody, either technologists or managers, are held accountable. For instance, the issues @vinhboy talks about are "bad smells" that reek of unprofessionalism -- like the primary keys that aren't unique, the predictable race conditions, and all of the problems that recur again and again.
- drewpc 13y agoThe article mentioned only one side of the problem: the acquisition process. The government must follow the Federal Acquisition Regulations (FAR - http://www.acquisition.gov/far/ http://www.acquisition.gov/far/) when contracting purchases of equipment and/or services. The FAR is not agile and is so strict that you CANNOT already know what you want, you must make the contract as generic as possible. Especially on large contracts (over $5M). For example, you can't say that you need laptops with Intel Core i5 processors because that would unfairly exclude other processor vendors from competing. The other side of the problem, arguably the bigger side, is that most of the people in the acquisition process do not know how to write a contract that is flexible and really gets the government what it needs. For such a large project as the health care website, the people at Health and Human Services most certainly did not know how to articulate what they wanted. They could've hired a company to help them determine their requirements and write a proposal, but that's a significant (and long) undertaking itself. In short, government systems will never be user friendly, scalable, or secure because we don't know how to acquire such a system.
- MichaelGG 13y agoI thought you can get around excluding vendors just by specifying generic requirements that only one vendor happens to fulfill. Like "must have inter-processor bandwidth of at least Xgbps" or whatever generic-yet-specific thing you can find.
- interstitial 13y agoI actually know someone who has worked for the VA in medical coding contracts. He still has Cobol programmers on his speed dial because some of the live code is still written in it.
- trimbo 13y agoThey might have COBOL around, lots of banks and government agencies do. But the VA's critical system, VistA, is written in MUMPS[1]. It's NoSQL! [1] - http://en.wikipedia.org/wiki/MUMPS http://en.wikipedia.org/wiki/MUMPS
- justinsb 13y agoApparently integrating with "legacy" systems was a big bottleneck (e.g. to verify your SSN). I don't understand why this verification had to be done "live" though; the obvious alternative would be to collect all the data, verify it asynchronously/overnight, and then flag up those people that had issues. You enter your birthday, SSN, etc and then the 1% of people that got their details wrong get an email the next day saying "we weren't able to verify your details, please log in and fix it".
- jes 13y agoTo what degree is the sheer complexity of the underlying government legislation responsible for cost, delays and complexity in the IT systems government pays to have built? No wonder we are going broke. It's nothing but special cases, all the way down.
- siliconc0w 13y agoJust saying there were like 20 articles about the all star obama campaign tech team that built a super scalable cloud infrastructure and ab-tested, email spammed, and narwhal'ed their way to victory. Couldn't get any of them for the actual governing?
- integraton 13y agoFTA: "One might look at this and go: Why can't we get the smartest people from Facebook and Google and from Twitter to come and work out these problems?" Johnson says. "The problem is that the way that federal contracting works is so burdensome that the only people who get contracts like this are experts at lobbying and experts at regulations that require you to get these sorts of contracts. And they're not experts at doing the job of building these websites."
- TWAndrews 13y agoNo, quite literally not. The campaign team is a private organization, not subject to rules for hiring federal employees or spending tax money. This is precisely the sort of shit that make people believe that the government is incompetent. It really is for certain undertakings that require some degree of operational flexibility.
- smsm42 13y agoWait, isn't the government the one who makes the rules? So if they made the rules that makes them incapable of doing things that they are supposed to do, while the people, technologies and ability to do the same thing is readily available and the same thing can be and is done by others - isn't that a very definition of incompetence? Not doing what is perfectly doable because of self-inflicted inhibitions? It doesn't "make people believe", it is incompetent. And, unfortunately, the very capable and competent people are working hard at making it worse by electing exactly the people who would perpetuate and expand the problem.
- scarmig 13y agoTwo points: 1) Our government has a huge set of rules about procurement. Who's allowed to get their bid approved, the right kinds of projects, who's allowed to manage them, who's allowed to set project requirements (different than the former), who's allowed to pay them, etc. In theory this is all really smart and awesome: you don't get someone giving their brother-in-law a swell multi-million dollar contract, physically impaired people can access your site, etc.* 2) Governments and campaigns face different incentives. Campaigns don't have many cross cutting concerns: the goal is to get a candidate elected. Sure, some high powered consultants are simultaneously angling for a new job after, but IT and most staff are relatively immune from that. Your job is to get something done, and you want to get somethng done. Government, though, has no such incentives: you can do shitty work and not face consequences, and the mission is inherently less aspirational and more shoveling shit. Much of the time government bureaucracies also double as a social welfare program, or a private welfare program if a well-connected corporation has a well-placed surrogate legislator or bureaucrat. *Reality, needless to say, can turn out a bit different.
- Spooky23 13y agoBlaming IT is an easy scapegoat. The notion that the Feds and various states should be operating marketplaces in the first place is absurd on its face. California and New York aren't subject to federal procurement guidelines, and their systems are basically non functional as well. Government is successful with complex public facing IT in areas where the processes are well understood. The DMV, tax collection, and unemployment are good examples in most states. Medicare billing is a success story in the federal space. Most other things are not very successful unless they are limited in scope. The whole system is a screwed up farce. "Obamacare" is a flawed system to begin with, to the point that nobody really understood what was needed for months. So the Feds and states were forced to roll with some highly generic requirements and specifics were made up as they went along. That scenario is a dream for Accenture (Cali) or CSC (ny) to churn hundreds of millions of dollars.
- ajross 13y agoCan you elaborate on how selling an insurance policy is not "well understood" or how it's inherently more complicated than administering the tax code (!) or running medicare? I don't follow that at all. Insurance has been sold successfully for well over a century. Hell, governments have been running and regulating markets for all sorts of things (many of them much more complicated than a handful of insurance plan choices!) for the whole of modern history. That doesn't mean that this site isn't a mess, but I don't see where the "screwed up farce" bit comes from. I rather suspect it's driven more by your choice of political tribe than any real intuition about what the government is capable of.
- threeseed 13y agoHilarious considering that Medicare had far more serious problems throughout its inception and you consider the billing component to be successful. Like all website launch problems in the months and years from now people are going to forget they ever happened. And instead will find having an independent marketplace to be incredibly useful. The states that have exchanges have already seen insurance premiums drop significantly.
- itbeho 13y ago
- downandout 13y agoThe government IT procurement process puts people that don't know what they are doing in charge of multi-million dollar budgets. The primary determinant of whether or not a company gets hired is its ability to navigate the procurement process - not their ability to actually do the job. In fact, many companies specialize in winning government contracts and then subcontract the work out to others. It's a sad, broken system that will almost certainly never change - in part because we don't vote for the people in charge of these things. It really doesn't matter who is in the White House or in control of Congress. The fiefdoms within the government that control these issues have maintained a culture of mediocrity for decades, and it will likely remain that way.
- dnautics 13y agohow do you propose reconfiguring the procurement process? Remember, it's got to be fair. You have to figure out how to charge a fair price. Everything has to be documented, so that it can be audited by the bureaucrat in a few months and given a second rubber stamp. Also we have to score the service providers, don't forget to ask them if they are minority-, veteran-, or disabled- owned, because each of those gives them +20 points.
- lisper 13y ago> it's got to be fair Why? It's not fair now.
- dragonwriter 13y ago> how do you propose reconfiguring the procurement process? For most of the problem areas, the answer isn't to mend it, but to end it. Stop contracting out core functions.
- nhebb 13y agoI believe the exchanges are run by the states. Since there is overlap - a federal law being implemented at the state level - the procurement process may have had to follow federal guidelines, but I wouldn't be surprised if this were a case of 50 states with 50 different procurement processes.
- balloot 13y agoIt should be noted that healthcare.gov might be the highest traffic website launch in the history of the internet. With millions of visitors on day one, I certainly can't think of one that would even compare. They are dealing with scaling issues that are almost completely unprecedented. Every other e-commerce site that deals with many millions of visitors a day has had months, if not years, to figure out the touchy parts of their infrastructure and make adjustments while the traffic ramps up. Given this, I can't help but think that 90% of the scorn heaped upon healthcare.gov is politically motivated. When Twitter spent YEARS going down on a regular basis, or Simcity just completely shut down on launch day (with a fraction of the users of Obamacare), nobody demanded the companies that running them be shut down. Shit happens when you build a website that gets incredible amounts of traffic. It will work out just fine.
- deleted 13y ago[deleted]
- thwarted 13y agoEvery other e-commerce site that deals with many millions of visitors a day has had months, if not years, to figure out the touchy parts of their infrastructure and make adjustments while the traffic ramps up. And how much those months or years of experience were leveraged in figuring out the touchy parts of the infrastructure? This shouldn't have been the first rodeo for anyone involved. There are numerous blog posts, books, videos, etc with ideas, what works, and what doesn't, for running websites of all sizes. Many of them are posted and debated on this very site.
- akjetma 13y agoI don't think what happened with SimCity supports your argument. People were outraged about it for weeks.
- pandaman 13y agoSimcity is exchanging orders of magnitude more data between each client and the server with much stricter latency requirements. Same applies to Twitter (though the data density is less than in a game there are more connections and the volume of data is higher). I don't see how a website that serves static (in the sense they do not change over time, like twitter feeds or game sessions) pages is comparable to either.
- joshuaheard 13y agoStart at 1:00 http://www.nbc.com/saturday-night-live/video/weekend-update-winnerslosers/n41636 http://www.nbc.com/saturday-night-live/video/weekend-update-...
- robomartin 13y agoThe real scary part is that we entrust these people with our most private information as well as decision making matrices that might very well determine what, when and if you receive the appropriate care. To me these incidents, the broken procurement systems, the mediocrity (or worst) that permeates government and many other factors only serve to reinforce the idea that we need massively less government reach into our lives, not more. Frankly, the combination of healthcare and the IRS scares the living daylights out of me.
- rurounijones 13y agoGood luck changing the procurement process. The UK Gov has been trying to get more SMBs involved in UK Gov procurement[1] in hardware and software is failing pretty miserably at the moment. [1] http://www.informationweek.com/smb/services/uk-pushes-govt-it-to-use-smb-suppliers/240149701 http://www.informationweek.com/smb/services/uk-pushes-govt-i... [2] Nuts, cannot find the link. Was an article on theregister.co.uk for what it is worth
- nostromo 13y agoThe jaw-dropper here is how much this website cost: $634,320,919. http://www.digitaltrends.com/opinion/obamacare-healthcare-gov-website-cost/ http://www.digitaltrends.com/opinion/obamacare-healthcare-go... With all the state exchanges included, the websites for this program must be over a billion dollars spent. Amazing.
- ericd 13y agoYeah, CA alone brings it almost to $1B (another $330M). It's pretty unbelievable how wasteful the procurement process apparently is.
- perfunctory 13y ago> In order for the exchange marketplace software to check on which coverage and subsidies you're eligible for, it has to ping requests to the ... Homeland Security data Really?
- tantalor 13y agoThis is the same company that build MapBox?
- quannah 13y agoNo. But the company that built the working sections of healthcare.gov - that is, everything except /marketplace* - and open-sourced it (https://github.com/CMSgov/healthcare.gov https://github.com/CMSgov/healthcare.gov) is Development Seed. They are indeed the makers of MapBox.
- philipashlock 13y agoFor those wanting deeper technical analysis around the problems and building off of some of the arguments Clay made about procurement, check out https://news.ycombinator.com/item?id=6518739 https://news.ycombinator.com/item?id=6518739
- pasbesoin 13y agoMake it work. Then make it pretty. You missed a step.