12 ms·
How a team of young people helped rebuild healthcare.gov (2015)
- whamlastxmas 10y agoSince I find non-descriptive headlines like this annoying (edit: looks like it's been changed on HN to be much better), here is a quote from the article that surprisingly provides its own TL;DR: >Here is the tl;dr version of their story: Marketplace Lite, or “MPL” as they came to be known, devoted months to rewriting Healthcare.gov functions in full, working as a startup within the government and replacing contractor-made apps with ones costing one-fiftieth of the price. And when, nearly a year after the initial launch of Healthcare.gov, the website’s second open-enrollment proved much healthier than its first, it was the MPL team who celebrated.
- dak1 10y agoAnd perhaps the biggest takeaway of the whole article, buried in the middle of it: > Instead, if the successes of the MPL team confirm any guiding principle for the future, it is this: Technical workers—not only engineers but designers—have to be involved with a process from the beginning. They will know that features must be described separately from needs, and that, when building software, smaller teams often perform better than larger teams.
- caminante 10y agomore tl;dr, here's the prior HN discussion: https://news.ycombinator.com/item?id=9857662 https://news.ycombinator.com/item?id=9857662 Many HN'ers suspected this was a "submarine" piece.
- donpdonp 10y agoUpvoted for not using the term whiz-kids.
- joshkpeterson 10y agoThis team became Nava: http://navahq.com/ http://navahq.com/
- Tiquor 10y agoI'm curious how this team got the call for the project.
- sumobob 10y agoExactly, some person somewhere who was involved in the traditional process of government software contracting, decided to go rogue.
- gabesmed 10y agoTodd Park, the ex-CTO of the United States, started to process of pulling in Silicon Valley talent. We were called "Todd's kids" for the longest time :) Todd reached out to the CEO of Civis Analytis (Dan Wagner I believe?), who had pitched in with the Obama campaign. Through him Todd reached Mikey Dickerson, who was at Google at the time (Mikey is now the head of USDS). Mikey looped in a few other googlers...and so it spread. I found the MPL team through friends in the startup world, many of whom were ex YC founders
- dpritchett 10y agoIt's pretty wild to think they had to go to the opposite coast to source a node/mysql/nginx solution. I know people who were doing such things much closer to DC. Granted, we are lower profile and maybe less pre-certified as "sure things". I wonder if borrowing Googlers is the government tech fire fighting equivalent of "nobody ever got fired for choosing IBM"?
- serge2k 10y ago> I wonder if borrowing Googlers is the government tech fire fighting equivalent of "nobody ever got fired for choosing IBM"? probably. Reasonably high chance of getting the smarts you need, plus you borrow a name.
- deleted 10y ago[deleted]
- rmason 10y agoThe entire culture has to change. Had a female programmer who works in Washington D.C. tell me a few years back the unwritten rule is that unless you're at one of the top rungs of the GS federal pay scale you aren't supposed to talk at all. She had a boss who met with developers and told them nobody cares about your opinions. You see million dollar mistakes being made and you either accept it or leave government which is what she did.
- dforrestwilson1 10y agoI too experienced several million dollar efforts that went nowhere. Governments in general are not geared to think about returns on investment, but our current one is quite terrible.
- wslack 10y agoI think government, in general, is built to be slow and deliberate, but as the pace of tech change has accelerated, the responsiveness of government hasn't kept up. Git was invested in 2005, for example. ref: https://bhc3.files.wordpress.com/2008/05/consumption-spreads-faster-today.png https://bhc3.files.wordpress.com/2008/05/consumption-spreads...
- vaadu 10y agoThe government is geared towards risk aversion and ensuring that the friends of the people at the top are taken care of. Results are secondary. This is possible because there's no accountability for not getting results.
- ams6110 10y agoAnd I'll say, outside of the military, the public sector does not attract competence. There are exceptions, certainly. But the pay and the level of bullshit tends to attract those who either want to play politics, or who want a secure paycheck and benefits with little accountability, or just can't really perform at the level most private sector work demands.
- 10y ago
- gabesmed 10y agoHey all, i'm one of the team members of Nava, the public benefit corporation that emerged from the MPL team to continue to improve how the government serves its people...happy to answer any questions!
- dforrestwilson1 10y agoHow much pushback have you seen from contractors and others who benefit from the current system?
- gabesmed 10y agoA lot of the most effective pushback we end up never seeing at all -- it happens in decisions and conversations we aren't party to. We do think in the long run delivering a radically better product will win out... But the presence of pushback does really demonstrate the need for tech-savvy partners inside the government, like those we have been lucky to work with at the Center for Medicare and Medicaid Services, and a reformed contracting process. The only way there will be better technology in government long term is if the people signing the checks are able to discern between the modern way and the old way, and insist on a better way of doing things. (Apologies for brief responses - on mobile)
- bogomipz 10y agoI asked this separately but since you are offering. What is the relationship between Nava and 18F?
- gabesmed 10y agoWe love 18f!!! We work a little more closely with USDS as our relationship with them is more complementary - they find opportunities in government and will often consult with us and other modern firms on how to approach big problems like identity management, identity verification, etc. we don't work directly with 18F but feel very much like we are part of the same movement towards reforming the ecosystem
- annoncoward 10y agoHow credible is this story I wonder? Feels a bit like a plug. 3 noobs brought in to replace a broken system built by CGI run on a single VM in the Verizon Terremark cloud? All as Google, Red Hat, and Oracle were called in? They couldn't get permission to use AWS yet it was already running on a similar cloud provider? Something doesn't smell right.
- bogomipz 10y agoInteresting, does anybody know how or if this relates to 18F? Is there a correlation? I ask because there is currently another post on HN today about 18F.
- imh 10y ago>AWS is a data center of supercomputers How the hell was this published?
- ryanmarsh 10y agoHow would you explain AWS to the layperson in a phrase?
- slapresta 10y agoWith words that have meanings? To the layperson, "data center" and "supercomputers" translates to "tech stuff". "AWS is a service by Amazon that runs software for you in computers connected to the internet" is still inaccurate, but has a meaning for the layperson other than "tech buzzwords with superlatives".
- ryanmarsh 10y agoIt's reasonable to expect the typical Atlantic reader to know what a supercomputer is and to ascertain what a data center is in this context. Painting with a broad brush that's what AWS is.
- jaibot 10y agoEh, it's not that inaccurate. AWS is a bunch of data centers full of big computers that simulate a bunch of smaller computers.
- killbrad 10y agoCan you say that in a generally digestible way?
- boyaka 10y agoAWS is a computing service built on "datacenters" located all around the world, which are large warehouses packed tightly with racks of computers that are networked together and efficiently powered and cooled. I'm missing a description of virtual machines / containers but that might be going into too much detail.
- mcorrand 10y agoThis is a hard problem to solve perfectly. I'm still stuck at the ID verification step, after sending in my documents. Never heard back. I'm the perfect corner case though, having gotten a green card recently, left a job, moved to another state, so on so forth...
- waterphone 10y agoSame, I was never able to get past this point, apparently because I don't exist in Experian's system due to having no credit history. Attempts to proceed beyond have never been successful the few times I've tried since, so I've given up.
- superswordfish 10y agoThis article was written by Robinson Meyer, for anybody else confused by the lack of a byline.
- lllorddino 10y agoHalf way into the article and still no mention of what the back end was like. Sigh. Also I wonder why they weren't allowed to use AWS.
- deleted 10y ago[deleted]
- gabesmed 10y agonode/MySQL/nginx for both app2 and SLS. both are on AWS -- the first federal govt systems to use them. Even with the urgency, needed to wait almost a year for them to be approved, mostly around security concerns.
- deleted 10y ago[deleted]
- wslack 10y ago"cloud" can be a scary topic, because it implies a loss of control and accountability.
- fucking_tragedy 10y agoI'd have serious problems if my healthcare information was processed and stored on servers controlled by a company like Amazon.
- dastbe 10y agoGiven that AWS is HIPAA compliant and you can use most database products for PHI (https://aws.amazon.com/compliance/hipaa-compliance/ https://aws.amazon.com/compliance/hipaa-compliance/), the likelihood is that you're going to see both public and private healthcare information moving into AWS (or Azure or GCE), if its not already there. But what about Amazon makes you not want healthcare organizations to use it?
- killbrad 10y agoSo, is it that you don't get how encryption works, or you think Amazon can access that data somehow?
- JustSomeNobody 10y agoAs a developer in my 40's, I see things like this and worry that the software industry will come to rely only on the young because they don't ... I don't want to say care ... but put as much emphasis maybe? ... on work-life balance. Developement-wise, I can hang with the 20 something crowd, no problems. I just can't compete with the single/no kids thing.
- wslack 10y agoI think that's the case right now with many jobs in mgmt consulting / investment banking. You can end up with a negative situation where people devote every waking hour then feel all subsequent success is owed to them - not great.
- dleslie 10y agoMy experience is similar; it's sad, really, because though I enjoy what I do and am often among the most productive, my unwillingness to forgo time with my family in order to work unreasonable hours, often a result of poor leadership decisions, has become somewhat of a liability. After a career of small companies and the occasional startup, I am eyeing the cubicle-and-collars world of B2B software development with more interest.
- gabesmed 10y agoThe youth helped us mostly because we were able to hop up and live in that house for months in order to be close to CMS - know our customers! But even for us those days are over - we have offices in SF and DC and have multiple more established folks on our team. Just like major tech companies, we want to provide a comfortable environment for skilled engineers of any age and family situation.
- ryanSrich 10y agoAs a married person with no kids I feel the same way. I'm not 40, or even 30 yet, but I look at my peers and wonder if they'll grow to regret putting in 60 hour work weeks just to make someone else rich. I'm guilty of doing it too, for sure, but it wears you down quick.
- 10y ago
- rev_null 10y agoThe new site may be easier to use, but is also fraught with privacy violations. https://www.eff.org/deeplinks/2015/01/healthcare.gov-sends-personal-data https://www.eff.org/deeplinks/2015/01/healthcare.gov-sends-p...
- trhway 10y agothe original healthcare.gov was made using SCRUM. SCRUM fails by design on projects involving more than one team with dependencies between them, and the result in such cases is usually complete devastation like the original healthcare.gov. There is no surprise that the [small] "team of young people" were able to incrementally build a [simple] webapp and an app using SCRUM - that is the sweetspot of SCRUM, pretty much the only situation where it doesn't screws things.
- cmonaghan 10y agoI'm a member of the Marketplace Lite team, and of the resulting public benefit corporation, Nava. I'm happy to answer any questions!
- rvanlaar 10y agoDo you think the success was caused by the team being young? Or just building what the customer, i.e. the people really needed in an agile way?
- cmonaghan 10y agoI think a key factor for our success was in being a small team, using a modern technology stack, and building in an agile way. I think the article overemphasizes the youth of the team. While many of us were quite young, our team included individuals in their 40s and 50s. I think the skew towards youth is more a byproduct of the risk profile of the project, and the need for us to spend significant time in Baltimore/DC.
- lostcolony 10y agoSpeaking as someone who worked in the past for a government contractor (DoD, so these issues may not be entirely the same with the feds, but I bet they're very similar), the issue isn't solely that bureaucrats don't get tech, it's also that the bureaucratic red tape that evolved to try and keep costs down has instead led to them rising, -especially- with software. From what I saw, government contracting requires all requirements up front. It then floats a bid, and the cheapest bidder wins. Whether or not the contract has been met is determined by that list of requirements; even if you fail to deliver on a non-functional requirement that is obvious ("the system should be able to scale out and handle X simultaneous users"), you aren't penalized if it wasn't one of the requirements. In fact, delivering on obvious things that aren't part of the contract -hurts the contractor-; it requires development time and effort (and thus eats into profits), and the government is legally prevented from taking such things into account on future contracts (that is, they can't say "Well, company A is the cheapest, but the last contract they took was a broken piece of shit; company B is a little pricier, but they delivered quality, we'll go with company B". No, you go with the cheapest, period, because if they fail to deliver what you specify you can sue, and if they fail to deliver what you -wanted-, but failed to specify, too bad). It's possible to renegotiate the contract mid-stream, but it's expensive and time consuming. So the contractor has every incentive to make sure that there is no time spent on quality, security, stability, scalability, etc, unless that's -explicitly- what was asked for. This article points out that having developers involved from the get go on the Healthare.gov rework led to it being successful, and that's very much true; you need technical people who care about more than just making bank to point out non-functional requirements that are missing. And, in general, trying to list all requirements up front is a fool's game, but such is the government's bidding process.
- vonmoltke 10y agoI was a little too quick on the upvote for your comment. Your first paragraph is spot-on, but your second is not. > the government is legally prevented from taking such things into account on future contracts (that is, they can't say "Well, company A is the cheapest, but the last contract they took was a broken piece of shit; company B is a little pricier, but they delivered quality, we'll go with company B". No, you go with the cheapest, period This is absolutely wrong. There is no requirement that the federal government must go with the cheapest bidder or that they cannot take performance into account. The government can and does regularly reject bids because they are unrealistically low; in fact, I was on one DOD program where both bidders were told to go home and come back with something realistic. Past performance definitely factors in to what the government considers reasonable. It's one of the reasons the government requires labor tracking even in situations where the labor expended doesn't affect the final cost, such as fixed-price contracts and uncompensated overtime.
- exabrial 10y agoAfter how many millions was given away though?
- tssva 10y agoThe site can only be considered a success compared to what it replaced. It is still slow, the workflow is confusing and this past enrollment period actually being able to login was still an issue.
- tuna-piano 10y agoI was struck by this part "The MPL team was not the ideal workforce: They were (and remain) contractors to contractors. They were not protected by a union, nor did they enjoy the many benefits of working as public employees. They are coders-for-hire who could relocate across the country quickly, and they reflect the larger industry they work in: Mostly young, mostly male, and highly educated. It does not have to be this way. This kind of precarious employment—lucrative, quasi-nomadic—is as much the result of poor planning as a natural consequence of writing code for a living." Why are any of those things not ideal? What's wrong with being contractors to contractors,non-unionized, young, male or educated? Or am I misunderstanding the author's point?
- dastbe 10y agoI think the focus was on the temporary employment part. The government does not need fly-by-night developers who come in when things are completely broken. they need a strong technical core that builds things well and with low costs the first time.
- serge2k 10y ago> non-unionized How much would they have made if they were unionized workers who got OT for the countless hours? Would they even have been asked to? Could they have gotten a proper office if they weren't non-union contractor to contractor workers? What sort of benefits do they get? You're working yourself at an insane pace, do you get health benefits and some guarantee of time off at some point? > young, male or educated At the exclusion of older, female, and minority workers.
- bmmayer1 10y agoThe sad thing about this is that said young people didn't get paid anywhere near the $250-500 million paid to the contractors and cronies[1] who built the broken healthcare.gov in the first place. [1]http://www.bloomberg.com/news/articles/2014-08-28/all-the-companies-making-money-from-healthcare-dot-gov-in-one-chart http://www.bloomberg.com/news/articles/2014-08-28/all-the-co...
- shitgoose 10y ago101 comments into discussion and no one mentioned CGI? the actual morons that fucked up the project and wasted 100's of millions of taxpayers' money? how many times should they piss in our face till we actually start noticing things?
- annoncoward 10y agoI mentioned CGI..
- alexmingoia 10y agoIt's a disgrace that no one is held responsible for healthcare.gov. We get article after article about "Silicon Valley" and their noble role in this shit show, yet not a single article about the criminally idiotic and corrupt politicians and their contractors.
- mmaunder 10y ago'Young people' are so cute when they're productive. Who knew?
- serge2k 10y agoBut... entitled millenials...
- kuharich 10y agoPrevious discussion: https://news.ycombinator.com/item?id=9857662 https://news.ycombinator.com/item?id=9857662
- beckler 10y agoI saw Mikey Dickerson talk about his experience in the initial recovery phase of healthcare.gov at Velocity in 2014. I think it shows just how insane the situation was. https://www.youtube.com/watch?v=7Vc8sxhy2I4 https://www.youtube.com/watch?v=7Vc8sxhy2I4