5 ms·

A bunch of smart Silicon Valley hackers cannot fix what is wrong with government IT. Government IT problems stem from the fact that government IT contracts are

A bunch of smart Silicon Valley hackers cannot fix what is wrong with government IT. Government IT problems stem from the fact that government IT contracts are worth Billions, and they are doled out through privileged networks of insiders. There are plenty of smart companies in America that can fix America's IT problems if they are paid to do so. There is no political will to do so. That is why companies you never heard of come up again and again as subcontractors for things like Healthcare.gov Believe it or not, there are some very smart people working in government. They actually don't need us in Silicon Valley to tell them how to do stuff. They know what they could do with the latest technology. They are not allowed to do so because of bureaucracy.

Bahamut 12y ago
Sadly, part of the cause of this is due to the small government mantra preached by "conservatives". This results in many tasks being offloaded onto contractors who end up bloating the cost and using various other backchannels. If a lot of these things were done in-house in the government itself, things would be a lot more efficient.
- refurb 12y ago
  Why would the lack of efficiency suddenly disappear if these tasks were brought in house? Jesus, the VA system in in house and look at that mess.
  - aceperry 12y ago
    Part of the VA's problems stem from the fact that it was overwhelmed by returning vets from our wars in the middle east, and funding cuts to the VA.
    - danuker 12y ago
      The passion it must have gone through.
  - Lawtonfogle 12y ago
    There is a greater possibility to reduce the cost and increase the efficiency. At the same time, I think chances are those savings will not be realized due to all the buracracy that can happen as people flex their power muscles. For example, needing JQuery to be reviewed in its entirety before being allowed due to security concerns and an official rule that all code used must either be developed in house, purchased (aka, having someone else to blame if something happens), or fully reviewed before use.
ohashi 12y ago
To be fair, Andrew lives in DC and has for a while afaik. Not sure why this is being posed as Silicon Valley fixing government IT. It seems like a few smart people trying to make government better. Nothing is going to be perfect, but it's at least a step in the right direction. I respect Andrew taking this step, there's a lot of civic hacking in DC, this is just a step further and making it a full time gig from the look of it. At least he's doing something instead of simply whining.
jedberg 12y ago
Here's the difference. I've talked to some folks in the digital service. They aren't career government. They have no risk in committing "political suicide" by suggesting something that everyone knows is the right thing to do. Basically, if someone doesn't like what they said, they can respond with, "well I can always got back to my old job". The government workers don't have that freedom.
TazeTSchnitzel 12y ago
Ehh. The UK's Government Digital Service successfully made http://gov.uk/ http://gov.uk/ which was a massive improvement over what it replaced (DirectGov), making the UK Government's services easy to find and easy to use. Why can't the US Digital Service? It seems to be trying to do the same sort of thing.
brandonb 12y ago
"There is no political will to do so." That was true a couple of years ago, but healthcare.gov was a wakeup call, and everybody from the president on down takes reforming government IT seriously. I was part of the healthcare.gov "tech surge" and I can tell you firsthand that the government is willing to change. In the last year, the healthcare.gov UI was streamlined from 76 to 16 screens for the average user. Large parts of the site now run on AWS and more is moving. The most problematic individual component, the login system, was rewritten from scratch with an API-compatible, drop-in replacement which now serves 100% of traffic. And this has been done within the same agency responsible for the original site. It is not a story of "smart hackers" parachuting in and wresting control away from government bureaucrats, but rather partnering with the agencies and creating reform from within. USDS itself, although started by the White House, now has bipartisan support and a real budget to grow. Although you are right that there are powerful entrenched interests in the beltway, the political will to change is there, and now the challenge is to recruit people like Andrew to seize the opportunity.
- _asummers 12y ago
  > Large parts of the site now run on AWS and more is moving I'm curious what, if any, security obstacles you faced with this, especially considering HIPAA and all that.
  - ckozlowski 12y ago
    I'm guessing this is on GovCloud. That environment has its own security accreditation http://aws.amazon.com/govcloud-us/ http://aws.amazon.com/govcloud-us/.
  - brandonb 12y ago
    Short answer: no real security issues. AWS itself runs everything from credit card processors to hospital software, they have great documentation on how to design your system so that you comply with the major certifications like HIPAA, PCI-DSS, and FedRAMP: http://aws.amazon.com/compliance/ http://aws.amazon.com/compliance/ http://aws.amazon.com/compliance/fedramp-faqs/ http://aws.amazon.com/compliance/fedramp-faqs/ Our security audit was apparently the first one in years to pass with no "high findings" (high-priority security issues that must be fixed before the system is approved to go live). The main barrier we ran into was that although AWS was already FedRAMP-certified, it had not gone through CMS's own internal security review process, which requires hundreds of pages of additional documentation and an audit of the code, team, and penetration testing of the live system. At the end of that process, you get an Authority to Operate, or ATO, and that's what every site in the government legally must have in order to launch. You can get a sense of how much paperwork is involved here: http://www.cms.gov/Research-Statistics-Data-and-Systems/CMS-Information-Technology/InformationSecurity/downloads/ATO_Package_Guide.PDF http://www.cms.gov/Research-Statistics-Data-and-Systems/CMS-... Navigating this ATO process was the single largest roadblock in developing the various parts of healthcare.gov 2.0. It meant that what we expected to take two months ended up taking more like eight. The good news is that once we had gotten an ATO for AWS [1], any project within CMS--not just healthcare.gov--could start using AWS. [2] And within weeks of the ATO, we started hearing about other groups building their new projects on AWS, including Accenture, who's now the primary healthcare.gov contractor. And that also means that if a startup wants to work with CMS, rather than the typical set of DC contractors, now they can use AWS too and have one fewer barrier keeping them out of the system. [1] as a small technical point, we split the ATO into "infrastructure" and "application" portions so that the infrastructure (AWS) portion could be reused. [2] If you're wondering why it matters that the government can use AWS, it's because most data centers that cater to the government are really terrible. In the initial data center used for healthcare.gov, the way you provisioned a new server was to send a word document to a sales representative, listing the unix packages you want installed, and then a few weeks later they would give you a virtual machine which may or may not have what you asked for. You can't be agile at all in that type of environment, and you certainly can't do DevOps well because you can't script anything.
ckozlowski 12y ago
Government IT contracts are much less insidious than it seems. The reason you have the same companies has very little to do with "insider" status and much more to do with the fact that these companies have made it their business to be able to operate in the federal environment. Bureaucracy, far from being a conscious, malevolent tool, is simply the accumulated barriers and obstacles erected over time for every instance there was an anomaly or problem. Accountability was demanded, and new guidelines put in place. These stack up over time, long after the original incidence has ceased to be a major issue. Unlike the private sector where companies and organizations can simply be wiped out and start with a clean slate, government doesn't (and for various reasons, can't) do that. There's still money to be made however, for those who are familar with the process. Simply being eligable to bid for government work means getting approved on a contract vehicle, complying with various laws on accountability and record keeping, and ticking off various checkboxes for equal representation of genders and minority groups. These regulations are removed and unfeeling of special circumstances or facts on the ground; they are the accumulated "good intentions" of laws passed that now stack up like strata. People who know this system well are intimidated by all of this, which is why you see many government and defense retirees move on to the private sector to steer these small companies into this kind of work. They know how it works, and just knowing that is a skillset in and of itself. It's not evil. And there's far less cronyism than one might suspect. (I'm not claiming for a moment that it doesn't happen, but somewhat surprisingly, it's minimal.) Improvements are not consciously forbidden, and making things better is not discouraged. But like an enormous traffic jam, no single person has room to do much of anything. Coordinating a number of people to try and move in a better direction can improve things a little, until they are stopped by another group who has no room to move. It's a mistake to think of government IT as insidious. It's a problem we have built over time that we have created ourselves. To fix it will require enormous bravery to do away with volumes of regulations that, by themselves, probably did some good. And it will require tolerance of practices done in private that would never have seen the light of day, but in government, has the spotlight of oversight shining on. It will be a challenge to solve.
- jrochkind1 12y ago
  You did not succeed in convincing me that the system you describe is not insidious. I think you probably describe it accurately, and it sounds quite insidious to me. Yes, of course the companies that get government contracts need to be experts at getting government contracts -- the problem is that they do not need to be experts at what they actually _do_, what they're getting paid for. There's in fact very little room left to be experts at what they do, after all the expertise they need to develop (and resources spent on same) in getting and keeping contracts. If you're a company and your motive is making a profit, then of course there's plenty of money to be made, exactly. The system you describe sounds pretty insidious to me. "Insidious. adj. proceeding in a gradual, subtle way, but with harmful effects." The harmful effects are exactly the privileging of expertise at government contracting over expertise at doing a good job. That's harmful only if you prioritize doing a good job, if you prioritize making a profit I suppose it's just fine. > But like an enormous traffic jam, no single person has room to do much of anything. Coordinating a number of people to try and move in a better direction can improve things a little, until they are stopped by another group who has no room to move. Yes, quite, that's quite insidious, exactly. It has some unique features to government, but also some commonalities with any large bureaucracy.
DanielBMarkham 12y ago
"Believe it or not, there are some very smart people working in government. They actually don't need us in Silicon Valley to tell them how to do stuff." I know -- very condescending, isn't it? Gee, it's a good thing these really smart folks are showing up. What would we have done without them? A broken system must obviously be full of incompetent people. I wish these guys the best, but this is 90% bullshit and 10% reality. Just to cite one, small (?) problem: in commercial IT, if you don't like the work somebody is producing? You fire them. In government IT, no such thing exists. Managers spend years shuffling around poor-performing employees in a desperate attempt just to keep the lights on. I could go on, but it would all just wash over folks. This is a movement, dammit! We all feel so very excited! This time is different! So I will help do whatever I can -- blog, comment, debug, whatever. But I will not become suddenly blind to the realities of the system that we are attempting to change. That doesn't do anybody any good. That ain't cynicism, that's reality. Hope is not a strategy, and you can't fix a system you can't honestly describe. (or understand how it got that way)
- dnesting 12y ago
  Hi Daniel, thanks for your perspective. My own perspective: There are amazingly smart people working in government, and even working for many of the contractors commonly cited as examples of The Problem. Many of these individuals are not effective not because they're incompetent (and therefore need to be told what to do), but because they are not empowered, or they lack confidence or a team of people they can use to propel good solutions forward. Bureaucracy and internal politics are rampant. One way to look at what we're trying to do here is to act as a nucleus of tech expertise inside these agencies, to attract and empower those individuals to do the right thing while providing air cover from the White House when they're prevented from doing so. That being said, there is a vanishingly small number of actual engineers working for the government. Normally this role belongs to the contracting companies. The incentives here frequently encourage overly large, excessively complex designs, and implementations that maximize the number of humans rather than good engineering practices. By improving technical literacy within government we improve the government's ability to identify bad contracts and methods to prevent these situations from happening in the future. We are not blind to the fact that there have been other attempts before us. The key differences I see are (a) support from the top and (b) funding.

Bahamut 12y ago

Sadly, part of the cause of this is due to the small government mantra preached by "conservatives". This results in many tasks being offloaded onto contractors who end up bloating the cost and using various other backchannels. If a lot of these things were done in-house in the government itself, things would be a lot more efficient.

refurb 12y ago
Why would the lack of efficiency suddenly disappear if these tasks were brought in house? Jesus, the VA system in in house and look at that mess.
- aceperry 12y ago
  Part of the VA's problems stem from the fact that it was overwhelmed by returning vets from our wars in the middle east, and funding cuts to the VA.
  - danuker 12y ago
    The passion it must have gone through.
- Lawtonfogle 12y ago
  There is a greater possibility to reduce the cost and increase the efficiency. At the same time, I think chances are those savings will not be realized due to all the buracracy that can happen as people flex their power muscles. For example, needing JQuery to be reviewed in its entirety before being allowed due to security concerns and an official rule that all code used must either be developed in house, purchased (aka, having someone else to blame if something happens), or fully reviewed before use.

ohashi 12y ago

To be fair, Andrew lives in DC and has for a while afaik. Not sure why this is being posed as Silicon Valley fixing government IT. It seems like a few smart people trying to make government better. Nothing is going to be perfect, but it's at least a step in the right direction. I respect Andrew taking this step, there's a lot of civic hacking in DC, this is just a step further and making it a full time gig from the look of it. At least he's doing something instead of simply whining.

jedberg 12y ago

Here's the difference. I've talked to some folks in the digital service. They aren't career government. They have no risk in committing "political suicide" by suggesting something that everyone knows is the right thing to do. Basically, if someone doesn't like what they said, they can respond with, "well I can always got back to my old job". The government workers don't have that freedom.

TazeTSchnitzel 12y ago

Ehh. The UK's Government Digital Service successfully made http://gov.uk/ http://gov.uk/ which was a massive improvement over what it replaced (DirectGov), making the UK Government's services easy to find and easy to use. Why can't the US Digital Service? It seems to be trying to do the same sort of thing.

brandonb 12y ago

"There is no political will to do so." That was true a couple of years ago, but healthcare.gov was a wakeup call, and everybody from the president on down takes reforming government IT seriously. I was part of the healthcare.gov "tech surge" and I can tell you firsthand that the government is willing to change. In the last year, the healthcare.gov UI was streamlined from 76 to 16 screens for the average user. Large parts of the site now run on AWS and more is moving. The most problematic individual component, the login system, was rewritten from scratch with an API-compatible, drop-in replacement which now serves 100% of traffic. And this has been done within the same agency responsible for the original site. It is not a story of "smart hackers" parachuting in and wresting control away from government bureaucrats, but rather partnering with the agencies and creating reform from within. USDS itself, although started by the White House, now has bipartisan support and a real budget to grow. Although you are right that there are powerful entrenched interests in the beltway, the political will to change is there, and now the challenge is to recruit people like Andrew to seize the opportunity.

_asummers 12y ago
> Large parts of the site now run on AWS and more is moving I'm curious what, if any, security obstacles you faced with this, especially considering HIPAA and all that.
- ckozlowski 12y ago
  I'm guessing this is on GovCloud. That environment has its own security accreditation http://aws.amazon.com/govcloud-us/ http://aws.amazon.com/govcloud-us/.
- brandonb 12y ago
  Short answer: no real security issues. AWS itself runs everything from credit card processors to hospital software, they have great documentation on how to design your system so that you comply with the major certifications like HIPAA, PCI-DSS, and FedRAMP: http://aws.amazon.com/compliance/ http://aws.amazon.com/compliance/ http://aws.amazon.com/compliance/fedramp-faqs/ http://aws.amazon.com/compliance/fedramp-faqs/ Our security audit was apparently the first one in years to pass with no "high findings" (high-priority security issues that must be fixed before the system is approved to go live). The main barrier we ran into was that although AWS was already FedRAMP-certified, it had not gone through CMS's own internal security review process, which requires hundreds of pages of additional documentation and an audit of the code, team, and penetration testing of the live system. At the end of that process, you get an Authority to Operate, or ATO, and that's what every site in the government legally must have in order to launch. You can get a sense of how much paperwork is involved here: http://www.cms.gov/Research-Statistics-Data-and-Systems/CMS-Information-Technology/InformationSecurity/downloads/ATO_Package_Guide.PDF http://www.cms.gov/Research-Statistics-Data-and-Systems/CMS-... Navigating this ATO process was the single largest roadblock in developing the various parts of healthcare.gov 2.0. It meant that what we expected to take two months ended up taking more like eight. The good news is that once we had gotten an ATO for AWS [1], any project within CMS--not just healthcare.gov--could start using AWS. [2] And within weeks of the ATO, we started hearing about other groups building their new projects on AWS, including Accenture, who's now the primary healthcare.gov contractor. And that also means that if a startup wants to work with CMS, rather than the typical set of DC contractors, now they can use AWS too and have one fewer barrier keeping them out of the system. [1] as a small technical point, we split the ATO into "infrastructure" and "application" portions so that the infrastructure (AWS) portion could be reused. [2] If you're wondering why it matters that the government can use AWS, it's because most data centers that cater to the government are really terrible. In the initial data center used for healthcare.gov, the way you provisioned a new server was to send a word document to a sales representative, listing the unix packages you want installed, and then a few weeks later they would give you a virtual machine which may or may not have what you asked for. You can't be agile at all in that type of environment, and you certainly can't do DevOps well because you can't script anything.

ckozlowski 12y ago

Government IT contracts are much less insidious than it seems. The reason you have the same companies has very little to do with "insider" status and much more to do with the fact that these companies have made it their business to be able to operate in the federal environment. Bureaucracy, far from being a conscious, malevolent tool, is simply the accumulated barriers and obstacles erected over time for every instance there was an anomaly or problem. Accountability was demanded, and new guidelines put in place. These stack up over time, long after the original incidence has ceased to be a major issue. Unlike the private sector where companies and organizations can simply be wiped out and start with a clean slate, government doesn't (and for various reasons, can't) do that. There's still money to be made however, for those who are familar with the process. Simply being eligable to bid for government work means getting approved on a contract vehicle, complying with various laws on accountability and record keeping, and ticking off various checkboxes for equal representation of genders and minority groups. These regulations are removed and unfeeling of special circumstances or facts on the ground; they are the accumulated "good intentions" of laws passed that now stack up like strata. People who know this system well are intimidated by all of this, which is why you see many government and defense retirees move on to the private sector to steer these small companies into this kind of work. They know how it works, and just knowing that is a skillset in and of itself. It's not evil. And there's far less cronyism than one might suspect. (I'm not claiming for a moment that it doesn't happen, but somewhat surprisingly, it's minimal.) Improvements are not consciously forbidden, and making things better is not discouraged. But like an enormous traffic jam, no single person has room to do much of anything. Coordinating a number of people to try and move in a better direction can improve things a little, until they are stopped by another group who has no room to move. It's a mistake to think of government IT as insidious. It's a problem we have built over time that we have created ourselves. To fix it will require enormous bravery to do away with volumes of regulations that, by themselves, probably did some good. And it will require tolerance of practices done in private that would never have seen the light of day, but in government, has the spotlight of oversight shining on. It will be a challenge to solve.

jrochkind1 12y ago
You did not succeed in convincing me that the system you describe is not insidious. I think you probably describe it accurately, and it sounds quite insidious to me. Yes, of course the companies that get government contracts need to be experts at getting government contracts -- the problem is that they do not need to be experts at what they actually _do_, what they're getting paid for. There's in fact very little room left to be experts at what they do, after all the expertise they need to develop (and resources spent on same) in getting and keeping contracts. If you're a company and your motive is making a profit, then of course there's plenty of money to be made, exactly. The system you describe sounds pretty insidious to me. "Insidious. adj. proceeding in a gradual, subtle way, but with harmful effects." The harmful effects are exactly the privileging of expertise at government contracting over expertise at doing a good job. That's harmful only if you prioritize doing a good job, if you prioritize making a profit I suppose it's just fine. > But like an enormous traffic jam, no single person has room to do much of anything. Coordinating a number of people to try and move in a better direction can improve things a little, until they are stopped by another group who has no room to move. Yes, quite, that's quite insidious, exactly. It has some unique features to government, but also some commonalities with any large bureaucracy.

DanielBMarkham 12y ago

"Believe it or not, there are some very smart people working in government. They actually don't need us in Silicon Valley to tell them how to do stuff." I know -- very condescending, isn't it? Gee, it's a good thing these really smart folks are showing up. What would we have done without them? A broken system must obviously be full of incompetent people. I wish these guys the best, but this is 90% bullshit and 10% reality. Just to cite one, small (?) problem: in commercial IT, if you don't like the work somebody is producing? You fire them. In government IT, no such thing exists. Managers spend years shuffling around poor-performing employees in a desperate attempt just to keep the lights on. I could go on, but it would all just wash over folks. This is a movement, dammit! We all feel so very excited! This time is different! So I will help do whatever I can -- blog, comment, debug, whatever. But I will not become suddenly blind to the realities of the system that we are attempting to change. That doesn't do anybody any good. That ain't cynicism, that's reality. Hope is not a strategy, and you can't fix a system you can't honestly describe. (or understand how it got that way)

dnesting 12y ago
Hi Daniel, thanks for your perspective. My own perspective: There are amazingly smart people working in government, and even working for many of the contractors commonly cited as examples of The Problem. Many of these individuals are not effective not because they're incompetent (and therefore need to be told what to do), but because they are not empowered, or they lack confidence or a team of people they can use to propel good solutions forward. Bureaucracy and internal politics are rampant. One way to look at what we're trying to do here is to act as a nucleus of tech expertise inside these agencies, to attract and empower those individuals to do the right thing while providing air cover from the White House when they're prevented from doing so. That being said, there is a vanishingly small number of actual engineers working for the government. Normally this role belongs to the contracting companies. The incentives here frequently encourage overly large, excessively complex designs, and implementations that maximize the number of humans rather than good engineering practices. By improving technical literacy within government we improve the government's ability to identify bad contracts and methods to prevent these situations from happening in the future. We are not blind to the fact that there have been other attempts before us. The key differences I see are (a) support from the top and (b) funding.