3 ms·
Silicon valley did ride to the rescue. Obama brought in Mikey Dickerson, head of google SRE, as well as several top people in Facebook (at least those were the
by hayst4ck 3y ago
Silicon valley did ride to the rescue.
Obama brought in Mikey Dickerson, head of google SRE, as well as several top people in Facebook (at least those were the people I had directly heard about) and humbly said "we failed, we need your help." From what I remember hearing he literally brought top valley people into the oval office and gave a "your country needs you" speech.
"Install new relic" really minimizes what was actually happening. "Can we install this tool that helps us understand how the system works" sounds a lot more enlightened doesn't it? Understand what's broken is the first step to fixing something and that's exactly what "installing new relic" does. If you don't have a monitoring system, that shows an extremely critical failure of technical leadership. It means no one knows what's happening. No one is measuring how things work. The google SRE book devotes chapters to monitoring, probably 25%+ of the book is explicitly about monitoring.
If you listen to Mikey present his work on healtchare.gov, he does not emphasize the solution, he emphasizes the problem: No one knew how things worked, no one knew how the pieces fit together, no one could measure why the site was performing so poorly.
The step after "installing new relic" was getting 50+ contractors in the same room and making them justify themselves. The contractors sold themselves to the government, but there wasn't a person on the government side with enough understanding to know you don't need 3 different load balancers at the same level for the same traffic and every system does need monitoring. How many different data stores do you really need?
If you listen to Mikey Dickerson give his presentation live, to the people he hopes to recruit, then it was very poorly managed at the technical level. Were those people stupid? probably not, but they were giving out lucrative contracts to any company that said they could help. So the makers of the original healtchare.gov (and by proxy the American public and Obama administration) were being taken advantage of for their ignorance.
> It's that they were operating under constraints and incentives that emphasize rules and compliance over making the whole system work well.
It's that trust was given to American companies who directly benefited from taking advantage of the government, and not to individuals who work on behalf of the public with specific expertise.
Mikey ends his presentation with a powerful call to action. If you aren't going to help the government with your own expertise, and make that sacrifice on behalf of the public, then some contractor will happily fill that role.
- wolverine876 3y ago> Were those people stupid? probably not, but they were giving out lucrative contracts to any company that said they could help. So the makers of the original healtchare.gov (and by proxy the American public and Obama administration) were being taken advantage of for their ignorance. Good points but that sounds to me like too simple a story. The people managing healthcare.gov weren't working their first day or managing their first project; you don't get that job without some experience. And if you have some experience, you know what you don't know, you know how to deal with those situations. (I'm not implying some other specific story - I don't know what happened. I'm just wary of such simple answers.)
- adonovan 3y agoI saw Mikey Dickerson talk about this at the time, and he was very explicit that the engineers on the project were all perfectly capable, but they were hobbled by an IT services culture that made them all terrified to be the person blamed for any mistake. He also said there was no culture of real-time monitoring. They literally had a TV running CNN, and whenever the chyron said "healthcare.gov is down again" they would spring into action. Amazing.
- enra 3y agoI can't really phantom that. When I was six grade or something, I had old PC with Debian in my room as "server" to play around and serve some random stuff from. And I had real time monitoring on that.
- wolverine876 3y agoThe cultural explanation makes more intuitive sense to me. And the lack of monitoring follows from the culture - they don't want to know.