17 ms·
How Go helped save HealthCare.gov
- gravypod 6y agoIs go what saved the project or good technical leadership and cultural changes in the thinking of design and architecture of software?
- taspeotis 6y agoThe title is “Go helped save” not “Go saved.”
- acdha 6y agoYes, but what little detail there is sounds like something which almost any language could have done acceptably. I'd believe that Go would require a few less servers than, say, async Python but the title really should have been something like “A little architecture goes a long way”. The approach they described is no doubt considerably more polished but people built websites using the same techniques back in the 90s to avoid SPOFs and then XHR made that even easier to do robustly. To be clear: this isn't to say that their work wasn't good but really seconding the point they're making about how the original design was the wrong architecture for a public website and there's no easy way to band-aid around that.
- dekhn 6y agoThis was my complaint about a go port at my employer. There was an old C++ service that was really slow, so the Go advocate made a bug-for-bug compatible version of it in Go, and they migrated over to that system. A lot of work went into touting how awesome it was, until I asked "so, you could have done this in C++, or really any other compiled language?" and they said "yeah, I just wanted to help build enthusiam for go. there's nothing specific about the language that made the resulting system any better"
- acdha 6y agoThere's always a tension between maintaining an existing codebase and trying something new but in that case I'd personally jump on a Go codebase instead of C/C++ in a heartbeat just for the maintainability improvements. There is a meaningful difference between “can a turing-complete language do this?” and “how much time will you spend dealing with avoidable problems or overhead?” and in the case of unsafe languages I find the arguments relatively compelling for picking anything safer to use in practice.
- vmception 6y agoand obviously they can now get hired for Go and you can't. what's the complaint, stop falling for these pedantic loops and win the game
- ravi-delia 6y agoThe complaint is that as long as you're transparently wasting employer resources on a resume gambit you may as well just embezzle anyway and avoid inconveniencing your coworkers.
- dekhn 6y agoNeither that code author nor I has any trouble being employed. I learned go and then stopped using it because (in my field) C++ and Python are what is used most commonly.
- benlivengood 6y agohttps://medium.com/the-u-s-digital-service/mikey-dickerson-to-sxsw-why-we-need-you-in-government-f31dab3263a0 https://medium.com/the-u-s-digital-service/mikey-dickerson-t... from the guy on the front lines. He has fascinating stories.
- oh_sigh 6y agoOhh, can he tell us about "Project Birmingham"? That would be a fun story. > A secret effort to influence the 2017 Senate election in Alabama used tactics inspired by Russian disinformation teams, including the creation of fake accounts to deliver misleading messages on Facebook to hundreds of thousands of voters to help elect Democrat Doug Jones in the deeply red state, according to a document obtained by The Washington Post. https://www.seattletimes.com/nation-world/nation-politics/secret-campaign-to-use-russian-inspired-tactics-in-2017-ala-election-stirs-anxiety-for-democrats/ https://www.seattletimes.com/nation-world/nation-politics/se... Of course, Mikey is an extremely hands on guy, and loves to dig into problems he's tasked with. But he threw all those personality traits all out the window when Reid Hoffman gave him almost a million dollars, and he (supposedly) had no idea what the money was being spent on.
- ineedasername 6y agoProject Birmingham is legitimately a concerning incident in dirty politics, but the inflammatory tone of your post makes it very easy to downvote you based on the appearance of flame bait for anyone who isn't already familiar with Project Birmingham.
- Ericson2314 6y agoHow fitting: American healthcare pre ACA : ACA : Public heathcare :: Untyped scripting languages : Go : Decent programming languages Or maybe I am being too charitable to Go here.
- javert 6y agoWhat do you dislike about Go?
- kupopuffs 6y agosomething something generics something something too simple
- RhodesianHunter 6y agoNot the person you're replying to, but the lack of generics is the big killer for me. I'm used to being able to manipulate data efficiently via streams/sequences and a rock-solid stdlib which just isn't possible in Go due to a lack of generics. A close second would be the lack of good performant libraries, which is somewhat due to number 1 and somewhat due to it being a relatively newer language. Try finding a good drop-in loading cache akin to Guava caching or Caffeine in Golang, it doesn't exist.
- BoorishBears 6y agoMy take on Golang: Use it for a project, get into the Go mindset of doing more with less. Then back switch to a language that's actually suited for useful work with getting mired in dogmatism and bring the mindset with you. Kotlin is a good one if you like Goroutines, Flow/Channels map pretty cleanly to them. Proponents of Go will say the dogmatism is a strength... it is until it isn't
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- sailfast 6y agoSpecifically: the author used Go to create a routine that would collect user data from submitted JSON requests to the site and then email them at a point later in the day when server load was low so they could sign up for healthcare. (Pre overall operational fixes / improvements) Transcript Anchor: https://changelog.com/gotime/154#transcript-83 https://changelog.com/gotime/154#transcript-83
- deleted 6y ago[deleted]
- _qulr 6y agoI just want to point out that I was unable to fill out my 2021 marketplace application on healthcare.gov until I stopped blocking optimizely.com in Little Snitch.
- sneak 6y agoThat’s terrible. Nobody should be compelled to provide their private health information to a single, unelected private company to obtain government benefits or comply with the law.
- jcims 6y agoIt's been like that since the initial release unfortunately. Realistically private companies support all sorts of government operations and have access to all sorts of extremely sensitive information. There's just something extra creepy about seeing it happen in your browser. (It's not immediately obvious what can be gleaned but most likely more than what you would gather from a cursory glance.) Here's a few of the domains from just the front page: adsrvr.org advertising.com chartbeat.com chartbeat.net digitalgov.gov doubleclick.net google-analytics.com google.com googleapis.com googletagmanager.com govdelivery.com gstatic.com healthcare.gov newrelic.com nr-data.net optimizely.com qualtrics.com quantummetric.com tealiumiq.com tiqcdn.com Edit: After a bit more looking I really think they need a bug bounty or something. This site has been dinged numerous times for infosec practices and it doesn't seem like the message is getting through.
- _qulr 6y agoI have quite a few of those domains blocked, but other than optimizely, blocking seems to have no effect on functionality.
- jcims 6y agoThis is what you want to see in your Optimizely code from Healthcare.gov: <!-- Facebook Pixel Code --> <script> !function(f,b,e,v,n,t,s) {if(f.fbq)return;n=f.fbq=function(){n.callMethod? n.callMethod.apply(n,arguments):n.queue.push(arguments)}; if(!f._fbq)f._fbq=n;n.push=n;n.loaded=!0;n.version='2.0'; n.queue=[];t=b.createElement(e);t.async=!0; t.src=v;s=b.getElementsByTagName(e)[0]; s.parentNode.insertBefore(t,s)}(window, document,'script', 'https://connect.facebook.net/en_US/fbevents.js'); fbq('init', '1706186366368131'); fbq('track', 'PageView'); fbq('track', 'Lead'); </script> lol
- paulsmith 6y agoGreetings, folks: as we know, headlines don't tell the full story, and no, spoiler alert, Go alone did not save HealthCare.gov. A lot of people pulling together and working very hard over many months did. There are very many stories about the rescue. This is but one part about how indeed, a modest but important service written in Go, yes, did help, a lot. It is worth noting that, 7 years later, almost all of HealthCare.gov has been rebuilt, and substantial parts of it are in Go. This is in part what the company I co-founded after the rescue has been up to. (If this kind of thing -- building modern, reliable digital infrastructure for government -- seems appealing to you, [we are hiring][1].) This is a pretty interesting story in its own right. [1]: https://adhoc.team/join/ https://adhoc.team/join/
- RangerScience 6y agoNeat. Curious: What's your stance on open-sourcing all civic infrastructure, like Healthcare.gov?
- paulsmith 6y agoI think it's a good idea (not a magic silver bullet, but considering tradeoffs, on balance better than proprietary platforms), but it's entirely up to the government folks in charge of particular projects.
- gpm 6y agoI'm curious what you think the greatest downsides would be? It seems the typical ones (you can't sell open source stuff) don't really apply.
- gunsch 6y agoVA has attempted a lot of open-sourcing as well, including most of VA.gov [1], including the project management of it, mostly as an ideological stance from the VA product owners we work with. Open-sourcing hasn't cultivated much in the way of public engagement with the projects, but it's done a lot in terms of making development easier for the range of contractors + VA employees we have, and (IMO) nudged toward better decision-making with the underlying knowledge that everything we do is publicly viewable. Other federal agencies routinely come to VA asking to learn from their digital modernization efforts, and I suspect the open-source stance has been a big part of that. (I also work at Ad Hoc. It's great!) [1] https://department-of-veterans-affairs.github.io/va.gov-team/ https://department-of-veterans-affairs.github.io/va.gov-team..., repository links at the bottom
- bsaul 6y agoI think people on this thread saying "any language would have worked" miss the point. The solution was to find a simple, obvious to deploy and maintain and monitor piece of code, doing precisely the thing needed to solve the very specific problem they had, and nothing more. And this is precisely the kind of culture that has been built around the go language from day one. So, in a way, it's absolutely not surprising that go was chosen.
- socialdemocrat 6y agoFinally somebody said it! You cannot divorce language from its engineering culture. Just compare how Go libraries and tools are made compared to Java. In the Java world everything is over-engineered to an extreme and if you as a new developer try to keep thing simple you will be quickly put in your place and told “that is not how proper Java programs are written.” And Java is not alone in having that kind of engineering culture. You find it a lot of places but it is the community where I find it most noticeable.
- cutler 6y agoMaybe Kotlin will change this, given time.
- aenario 6y agoUnfortunately Java seems to bleed its culture everywhere: Modern PHP development seems to be about how much of a pyramid of class you can build to serve a single request. Then you have all the new typescript frameworks, proudly inspired by Spring.
- tha0xf 6y agoWe specifically ban Go here at a large start-up that everyone's heard of so that Go developers don't come and spread their cancerous SJW delusions. Just like Linus banning C++ from the kernel to keep C++ developers away. Results: huge improvement in moral and worker efficiency.
- gpm 6y agoI know I shouldn't be responding to the troll... but is there any reason to believe that go developers are at all more sjw/liberal/political/anything else than the average other developer?
- tha0xf 6y agohttps://go-review.googlesource.com/c/go/+/236857/ https://go-review.googlesource.com/c/go/+/236857/ https://golang.org/ https://golang.org/ (still has the BLM banner)
- philosopher1234 6y agoWhat startup is it? Want to know so I can avoid it.
- tha0xf 6y agoDon't worry about it, you wouldn't make it past the first interview. We only accept the top of the top of the talent pool.
- philosopher1234 6y agoOh, good, you're screening out talented devs. Explains you working there.
- tha0xf 6y ago>Oh, good, you're screening out talented devs. Indeed, that's the nature of the beast when you only hire the top of the top. Lots of talented devs can't cut it. Not that you'd fall into the "talented" group anyway, so, again, no reason for you to worry. >Explains you working there. "We only accept the top of the top of the talent pool." Indeed, that does explain me working there.
- oh_sigh 6y agoI think the biggest thing that helped save HealthCare.gov is having a blank check from the government. As of 2014, the site cost $2.1B to create and operate: https://www.bloomberg.com/news/articles/2014-09-24/obamacare-website-costs-exceed-2-billion-study-finds https://www.bloomberg.com/news/articles/2014-09-24/obamacare...
- moocowtruck 6y agothe saddest part about this type of stuff is the gov has software devs on staff not from consultants who are capable of "saving" gov sites. It is often a political reason why things are they way they are(making everything 10x more convoluted for on staff devs)...and people profit more from having external contractors do the work. Parts of govt I'm in..contractors can use cloud services but our teams cannot, and we have to rely on internal systems and bureaucracy.. it's honestly all bullshit.
- vippy 6y agoThe real challenge is getting the clearance and funding to work on things, and having enough staff developers within an agency, and technology leaders embedded in the business organization to help guide decision makers, etc. It's painful watching the government spend billions on commercial "off-the-shelf" software that requires years of "customization" (read: development) and, at the end of the day, the code isn't owned by the public, the platform company can put the screws to the government and charge more, prevent the government from accessing its own data without charging literally tens of millions of dollars, etc.
- rablackburn 6y agoYep, one of the core value propositions for bringing in consultants is “they tell the exec what your staff already know. But because it cost a lot of money to have them say it people are more likely to listen” It’s a real service that provides real value. But companies can avoid the cost if they have a strong culture of trust internally (which is no simple thing to foster)
- moocowtruck 6y agoyep trust! exactly so little of it for whatever reason.. but somehow the contractor company we have to hound for the features they originally promised for that money get away with it, at least for enough years to get millions upon millions
- tehjoker 6y agoI deeply respect the efforts that went into saving HC.gov though the initial rollout was a stunning blunder by the Obama admin for not keeping careful track of progress. I further would note that such a citizen facing service would not need to have been stood up if Medicare for All was passed. It could all be handled on the backend and people would just walk into the doctor's office or hospital.
- nwsm 6y agoI interviewed with Ad Hoc a few years ago. They told the healthcare.gov story (and I googled it as well) but never mentioned Go. Seemed like a good group though
- RandyRanderson 6y agoHmmm.... label0: Pick any project x, and a technology y. If x is a success, publish content on how y saved x. else goto label0. This also works for compound x: label1: Pick any ailment x, and a compound y. If x improves with y, publish content on how y cures x. else goto label1.
- ineedasername 6y agoYes, it's the word "save" that's the problem. A title like "Go's role in improving Healthcare.gov" would be much better, less click-baity. Because I can say SQL saved my data analysis project. MS Word saved my writeup of the project. Email saved my distribution of my final report on the project. The phone saved my ability to have meaningful follow-up conversations with stakeholders on the project.
- FpUser 6y agoI did not listen to podcast as I prefer reading but looking at the title I'd say that if one is in dire straits one needs smart programmers. The language is irrelevant and unless some very specific requirements are to be met good software can be written in any language.
- doctor_eval 6y agoFYI the transcript at the link is really good
- FpUser 6y agoOops, sorry did not notice it. Thank you.
- exabrial 6y agoI don't know if I'd be boasting about wasting $2.1 billion dollars :/
- nojito 6y agoHow is that even a lot of money? Do people not understand how many failed projects there are in the US budget?
- DudeInBasement 6y agoI guess wasting other peoples money is ok as long as we waste less than the other failed projects?
- morsma 6y agoWere you trying to put forth an argument? Because I don't think you did.
- dsabanin 6y agoFunny title. Meanwhile, title in an alternative universe, where the project didn't go as planned: "How Go has caused Obamacare to fail".
- zacharycohn 6y ago(Former 18Fer here) If you're interested in getting involved in government-focused work, there are a ton of fantastic organizations hiring right now. One warning, before you consider applying to any of these orgs: please do not see yourself as a savior. There are a ton of smart people already in government who have tried everything before. More smart, talented people are great, but appreciate and respect those who have come before, who have been doing the Hard Work for a long, long time. Be humble, be willing to learn, and be ready to listen. Gov agencies to start: 18f.gsa.gov usds.gov Gov contractors to start: http://adhocteam.us/ http://adhocteam.us/ https://www.navapbc.com/ https://www.navapbc.com/ https://truss.works/ https://truss.works/ And selfishly, I'm Head of Product at http://www.demandstar.com http://www.demandstar.com where we're building software to help city and local governments bring the procurement process out of excel files and outlook inboxes and onto the web. Hiring Lead and Senior frontend engineers. Email me zcohn@demandstar.com for more info.
- lattalayta 6y agoAre there any open-source projects you'd recommend helping out with?
- zacharycohn 6y ago[Disclaimer: the government is a Big Place and there is lots of variety.] With the exception of open source infrastructural tools that are used by the whole world, there probably are not many open source government-run projects that allow contributions from non-governmental employees. There are all sorts of reasons for this: - Some are legal. Lawyers get anxious about the soliciting or accepting free work. There are ethical issues around contributions - how do you know it isn't a bribe? - Many are technical. Someone has to review that code before it gets merged in. Who does that? Probably needs to be a government employee, but many government agencies don't have much, if any, in-house engineering resources - and what few resources they do have are already quite busy maintaining the current crop of software projects. How is this maintained once the open source contributors go on to other projects? - Many are a mix of both. How do you ensure the code meets proper security and compliance standards? Accessibility requirements? Records retention requirements? Who is responsible for fixing one of these issues when an issue is discovered? What happens if there is a support issue? Who can the case management worker who is out in the field contact to get help? This isn't to say these issues can't be solved. But it would take a lot of work to solve them all, and it would be difficult to solve them all in a way that would be cheaper than just hiring internal resources, buying a COTS solution, or hiring a vendor to build what you need. That being said - if you aren't ready to switch jobs, there are lots of government-adjacent open source projects - especially in the Open Data world. I'd recommend checking out your local Code for America brigade as a good starting point: https://brigade.codeforamerica.org/?_ga=2.169184225.1948638540.1604991891-909336803.1604991891 https://brigade.codeforamerica.org/?_ga=2.169184225.19486385...
- batt4good 6y agoMaybe if the team who built healthcare.gov wasn't shit all of this could have been avoided? Those who cannot work for real tech companies... build poorly performing webpages for Uncle Sam?
- forest_dweller 6y agoI've worked in gov in the UK and it isn't always to do with the quality of contractors or full time employees. A lot of the time projects are mismanaged or BI haven't done their job properly and the requirements don't actually make any sense e.g. instead of validating a phone number by sending a text message and sending them a code to fill in on the site, they instead want you to validate via regex. Or you are having to work with some archaic piece of software that is provided by a third party and has it own domain specific language that doesn't really provide any benefit and is impossible to debug when it doesn't work. This creates an attitude where everyone is kinda resigned to failure or you just can't actually get things done.
- xvilka 6y agoDid they rewrote COBOL parts in Go too?
- mikeruhl 6y agoI liked this, Paul Smith had some good stories. This episode and the following one about Github CLI got me thinking a lot how we choose architecture and languages and how it’s an iterative process. Rarely do companies/orgs get it right on the first try. Needs change, estimates are way off, and budgets run out.
- vips7L 6y agoOh God this has nothing to do with Go.
- followben 6y agoGreat writeup. AdHoc sounds like a really interesting company to work for. Does anyone know if there are similar firms working to improve online govt services in Australia?
- justinclift 6y ago> And like a SAND attached to it for network storage SAND -> SAN Hopefully that's a typo by the transcriber. ;)