10 ms·
Greetings, folks: as we know, headlines don't tell the full story, and no, spoiler alert, Go alone did not save HealthCare.gov. A lot of people pulling together
by paulsmith 6y ago
Greetings, folks: as we know, headlines don't tell the full story, and no, spoiler alert, Go alone did not save HealthCare.gov. A lot of people pulling together and working very hard over many months did. There are very many stories about the rescue. This is but one part about how indeed, a modest but important service written in Go, yes, did help, a lot.
It is worth noting that, 7 years later, almost all of HealthCare.gov has been rebuilt, and substantial parts of it are in Go. This is in part what the company I co-founded after the rescue has been up to. (If this kind of thing -- building modern, reliable digital infrastructure for government -- seems appealing to you, [we are hiring][1].) This is a pretty interesting story in its own right.
[1]: https://adhoc.team/join/ https://adhoc.team/join/
- RangerScience 6y agoNeat. Curious: What's your stance on open-sourcing all civic infrastructure, like Healthcare.gov?
- paulsmith 6y agoI think it's a good idea (not a magic silver bullet, but considering tradeoffs, on balance better than proprietary platforms), but it's entirely up to the government folks in charge of particular projects.
- gpm 6y agoI'm curious what you think the greatest downsides would be? It seems the typical ones (you can't sell open source stuff) don't really apply.
- gunsch 6y agoVA has attempted a lot of open-sourcing as well, including most of VA.gov [1], including the project management of it, mostly as an ideological stance from the VA product owners we work with. Open-sourcing hasn't cultivated much in the way of public engagement with the projects, but it's done a lot in terms of making development easier for the range of contractors + VA employees we have, and (IMO) nudged toward better decision-making with the underlying knowledge that everything we do is publicly viewable. Other federal agencies routinely come to VA asking to learn from their digital modernization efforts, and I suspect the open-source stance has been a big part of that. (I also work at Ad Hoc. It's great!) [1] https://department-of-veterans-affairs.github.io/va.gov-team/ https://department-of-veterans-affairs.github.io/va.gov-team..., repository links at the bottom
- ElijahLynn 6y agoNicely stated gunsch: > Open-sourcing hasn't cultivated much in the way of public engagement with the projects, but it's done a lot in terms of making development easier for the range of contractors + VA employees we have, and (IMO) nudged toward better decision-making with the underlying knowledge that everything we do is publicly viewable.
- TOGoS 6y agoCare to define "gunsch"? Other than the expected gross definitions from Urban Dictionary I'm coming up empty-handed.
- jonchang 6y agoThat is the handle of the person who wrote the comment.
- dboreham 6y agoBut now it's a word, meaning: to mistake someone's username for an unfamiliar natural language word. To gunsch v.
- lazyasciiart 6y agoThe name of the commenter they replied to.
- trimbo 6y agoIt's worth noting that the VA's most critical software -- its EHR system, VistA -- is public domain with the source code available. However, despite being a critical, successful piece of open source software that had massive investment over decades, it's being abandoned for a commercial system, Cerner, in a $16bn project. https://ehrintelligence.com/news/va-cerner-implementation-contract-balloons-to-16-billion https://ehrintelligence.com/news/va-cerner-implementation-co...
- 6y ago
- ElijahLynn 6y agoThere is a start on VA.gov already happening in this direction. The front-end codebase and backend (Drupal 8) CMS are both fully opensource for VA.gov. * Frontend - https://github.com/department-of-veterans-affairs/vets-website/ https://github.com/department-of-veterans-affairs/vets-websi... * Backend - https://github.com/department-of-veterans-affairs/va.gov-cms/ https://github.com/department-of-veterans-affairs/va.gov-cms... Paul's company is working on this project too (whom I work alongside with but with a different company)! The U.S. Digital Service is really a big factor in this, see the playbook, specifically play 13: > If the codebase has not been released under an open source license, explain why. source: https://playbook.cio.gov/#play13 https://playbook.cio.gov/#play13. So, see how this is now flipped. It is now "hey, you need to open source this by default, and if not, you need to explain why you didn't." and not the other way around! Update: See Andrew Gunsch's comment in this thread too!
- dheera 6y agoIt would only make sense to open source something like that if people would actually go work on it. There may be lots of security flaws in the system but it's not guaranteed that open sourcing it will result in a fast fix from the community. This isn't OpenSSL. A better approach might be to just budget for and pay a team of good developers a market wage to make it better.
- randmeerkat 6y agoTo be fair OpenSSL was mostly ignored by industry and the community until heartbleed. “At the time of the Heartbleed attack, the OpenSSL website listed just 15 active developers, most of whom contributed to the project on a volunteer basis. But not all changes to the OpenSSL software are written by these 15 people. Rather, these developers help to filter and organize suggested changes from a larger community of people who make occasional contributions.” https://www.vox.com/2014/6/19/18076318/heartbleed https://www.vox.com/2014/6/19/18076318/heartbleed If you want the government to have more competitive Federal salaries for developers then write your Senators. As a note, I’ve seen some truly awful software written by highly paid devs, so, just paying more won’t necessarily fix the problem.
- WesolyKubeczek 6y ago> As a note, I’ve seen some truly awful software written by highly paid devs, so, just paying more won’t necessarily fix the problem. High pay is necessary, but not sufficient. With high pay, you MAY get a good product — if you attract competent people on all levels. With low pay, the mind-numbing godawfulness of the end result, if it's ever delivered, is practically guaranteed.
- Chris2048 6y agoSometimes, the high pay is to offset the terrible nature of the work, not to attract better developers, e.g. having to work on the terrible ad-hoc, micro-managed code of many previous devs.
- mrmonkeyman 6y agoWhy not work out in the open? Nothing innovative or secretive going on, right? Bridges being build are not walled off with armed military around them right?
- mbesto 6y ago> building modern, reliable digital infrastructure for government And how about well supported? How resilient is the technology going to be in 15 years and how easy will it be to find developers? How much are those resources going to cost versus say .NET, PHP, Java, etc.?
- casept 6y agoGo is sufficiently trivial that anyone who dares call themselves a programmer can learn it in two weeks if they already have experience in some even remotely-related language.
- quotemstr 6y agoYou could have said the same about pre-generics Java. The problem is that when the language is simple and minimalist, people do things with the language that are byzantine and arcane, and you can't learn those in two weeks. Example: classic Spring's XML configuration inner platform.
- socialdemocrat 6y agoNo you couldn’t. I watched both Java and Go since they got released and Java really quickly went into the over-engineering and complexity camp long before generics. I worked as a consultant doing Java long before generics and the culture of over-engineering was already there. The language also really encourage it. Go is an entirely different beast. The language, libraries and whole community is really worshipping simplicity in a way Java never did. Just look around a bit at the standard library. There are pretty much no setters and getters. No inheritance hierarchy to speak of. Not even constructors most of the time. It is not without reason that Go angers a lot of people that think they know how to do software. It breaks all the rules they have kept sacred, that they think all “real” programmers should follow. Think about it this way. If Java is SLS assembled in a clean room never getting done, then Go is more like SpaceX assembling a rocket in open space with guy who normally weld water towers. It isn’t how it is supposed to be done but it works and they get shit done 20x faster.
- 7thaccount 6y agoMy understanding was healthcare.gov was a failure until a small team took over and redid the full thing in ruby on rails just to get it all working, and then later it was ported to something more performant. Is that true?
- zacharycohn 6y agoThere are many gross oversimplifications of the healthcare.gov rescue experience.
- MengerSponge 6y agoIt was a long ago, but here's hoping someone from 18F can chime in: https://18f.gsa.gov/ https://18f.gsa.gov/
- paulsmith 6y agoNo. The original site was a Java/J2EE application, and that's what the small team you refer to helped to stabilize on the rescue. We got it working well enough to make it through that first enrollment year. There was no rewrite into Ruby on Rails. Later, and over time, key components of HealthCare.gov were redesigned and rebuilt. Rails was used for some, Go for others, and there is still some Java from the original design.
- leafmeal 6y agoI was checking out the application for one of your positions and found this required question: > Please provide your salary requirements or range. I feel frustrated when I see that you haven't listed a salary range for your open positions, yet expect applicants to provide this information. I think it puts prospective employees at a disadvantage. Also it can waste applicants' time if you'll never pay as much as they need or expect. I think your mission is valuable. I want public services to serve the public well. I hope you consider being more open about potential salary ranges because people like me would feel more comfortable applying.
- paulryanrogers 6y agoThis may be illegal in California where employers must share their range and cannot ask candidates their salary history.
- mmcclure 6y agoNo, I don't see anything here that would be illegal in California. There's no requirement to include the salary range in a job posting or anything upfront, the obligation is to disclose the pay scale for a position "upon reasonable request." In fact, this is specifically called out as being after an initial interview.[1] Employers can't ask candidates their salary history, but they can definitely ask the candidate for salary expectations. [1] CA Labor Code Section 432.3 (c) An employer, upon reasonable request, shall provide the pay scale for a position to an applicant applying for employment. For purposes of this section, “pay scale” means a salary or hourly wage range. For purposes of this section “reasonable request” means a request made after an applicant has completed an initial interview with the employer.
- dialamac 6y agoLike many CA laws I don’t see how this is practical. Unless there is some reference to market ranges or something might as well say range starts at minimum and up to whatever made up budgeted amount, depending on experience.
- lazyasciiart 6y agoFYI, looking at your job descriptions as a non-citizen, this line is pretty cryptic: "Some federal contracts require U.S. citizenship to be eligible for employment." Is this a really vague way of saying that only US citizens should apply?
- kfrzcode 6y agoNot likely. It's actually quite literal: Some federal contracts do require U.S. citizenship to be eligible for employment. Government contractors often have postings for multiple contracts or positions and not all may have this requirement.
- sethetter 6y agoAd Hoc employee here -- this is spot on.
- lazyasciiart 6y agoThis abysmal response suggests that there are no non-US citizens currently employed there and nobody knows how it would work if one was hired. Good to know! I’ll be interested to hear when someone finally guinea pigs it.
- lazyasciiart 6y agoYea, I do know that. What I don't know is whether that makes up 95% of their work and the other 5% is maintaining a COBOL app? Yea no thanks. edit: "quite literal" is absolutely worthless. It would be "quite literal" to post a job saying "You will definitely be paid", and it would convey no information at all.
- unishark 6y agoI think the point of the question is not what are the literal facts, but rather what do they mean in practical terms for applicants.
- worldsayshi 6y agoI think it would be very important to spread the word about how healthcare.gov was saved. Are there any good sources on this? It's not just US that fails with these kind of projects. In Sweden we have seen many expensive government IT infrastructure projects that effectively flush tax money down the drain. The users find them horrible to work with and they are riddled with security issues. As a developer in Sweden I feel very frustrated about this. If I knew of a similar company in Sweden I would seriously consider joining it. It could be done so much better. But how did you do it? Edit: And I'm thinking not so much about the technology part of the problem but everything else, since finding competent developers that can solve the problem once you have a clear direction and decent financing seems like the easier part? Edit 2: It feels like one of those things that could make a really eye opening documentary given the right direction.
- tgsovlerkhgsel 6y agoHaving seen such failures in Germany, I think the key reason is that government IT contracting requires two very different skills: Government contracting, and IT. And "contracting" skills are much more important than the latter: It doesn't matter how good your IT skills are, if you aren't good at contracting, you aren't getting the job, but if you're good at contracting, you can get the job, deliver a shitty product, and still make money. Good IT firms are busy working on free-market projects where they have to deal with 10% of the bureaucracy. It's not worth competing against established players who know how to play the government contractor game much better than you, especially if bidding rules may force the government to pick a contractor that they know will deliver a subpar product because it's cheaper/because their bid is written by someone who knows exactly how to write them. I suspect that if you want to compete in the market, you need to have a "bureaucracy" team that's at least as big, if not significantly bigger than, your actual software development team. Even in an ideal market that's going to drive prices up. Edit: It seems like in this case, this quagmire was avoided by having the government build a skilled team in-house, avoiding all the overhead of an army of government bureaucrats interfacing with an army of contractor bureaucrats (which is necessary due to the processes that ensure fair contract awards).
- 6y ago
- deepakhj 6y agoPaul, I love the Ad Hoc Homework resource. Thanks to your team for sharing it!
- peterwwillis 6y agoHoly crap dude. It looks like a sea of white faces. (it's partly the glare, but still) Are you facing any challenges in finding more diverse candidates?