15 ms·

Inside the Obama Tech Surge as It Hacks the Pentagon and VA

162 points by brensudol 10y ago

impostervt 10y ago
"U.S. Digital Service salaries are set in accordance with the General Schedule Pay Scale, which is capped at $160,300." That's not bad, but not great for really high talent employees. I wonder how many people get that rate though. https://www.usds.gov/join#compensation https://www.usds.gov/join#compensation
- vonmoltke 10y ago
  I'm not sure that's much of an issue, since it is pretty good money (even in DC) for the 99% of us who aren't "really high talent employees".
  - impostervt 10y ago
    True, but I wonder what the "normal" rate is. As in, do they really get that much, or by stating that rate on their website are they really just trying to say "if you make a lot more than this and care a lot about money, don't bother".
    - ryanmarsh 10y ago
      Which is basically why I'm saving to go work there.
      - zacharycohn 10y ago
        Apply soon! We're hungry for help. [Disclaimer: I work at 18F.]
- sbov 10y ago
  Even if I made more than that in the private sector, I would be willing to take a minor pay cut for this sort of position. If you think about it, you're helping put taxpayer dollars to more efficient use, which includes your own.
  - macintux 10y ago
    Not sure how many people are going to take a 10% pay cut in hopes that 5 years from now their taxes might be a few bucks cheaper (which of course they won't be because the money would just be spent elsewhere). Admirable intent, I'm suspicious of its prevalence.
    - ryanmarsh 10y ago
      They call it a tour of duty for a reason. $163k would be a massive paycut for me but I'd do it and happily burn savings to make a difference for the lives of my fellow veterans.
      - sailfast 10y ago
        Love your second sentence, but really wish they would stop calling it a "tour of duty". It's still a pretty solid amount of money (nevermind Federal benefits) but to really transform we need people willing to work over the long haul not just parachute in for short terms. Also (slightly biased of course) I bet you get a bit addicted to the mission and that provides a non-monetary incentive to stick around a bit longer :)
        zacharycohn 10y ago
        [Disclaimer: I work at 18F.] It's referred to as a tour of duty because most of us are on 2 year appointments. At the end of that 2 years, there's an optional 2 year extension. These orgs haven't been around long enough for anyone to run out their second two years yet. Personally, I love this format. It never would have even dawned on me to consider applying for a career position in the federal government, but when the job is time-limited, there's some decision making bug in the brain (even though I could have applied for a normal career position and then just left after 4 years!). It's a great way to entice people to apply who never would have before. We also need people in for the long haul. But lets solve one problem at a time!
        10y ago
- spacecowboy_lon 10y ago
  Presumably you get the pension that goes with that
  - vonmoltke 10y ago
    Depends. My wife only has the $9000 in her Thrift Savings Plan after four years at the White House. She was a "political appointee", though, so she didn't get normal GS benefits.
- deleted 10y ago
  [deleted]
- liyanchang 10y ago
  Disclosure: I'm an engineer at USDS and these are my own opinions. That's right. The US Government is probably not going to win on compensation - salary limits, no stock options, no lunch. However, while it's not a money making enterprise, it's enough to do just fine. I recognize the sacrifices that many of my colleagues have made to be here, not to mention the sacrifices that federal employees and contractors have made (some who are very good and could be making more in the private sector) and that makes this even more worthwhile. The thing that government can offer is impact. I've always known that government has a big impact on people's lives but not sure if I could personally make an impact. That's what USDS, 18F, a number of other opportunities are offering. This is not a job for everyone. But if you're a certain type, there's nothing quite like it.
  - mattmcknight 10y ago
    The government could do a lot more by adjusting the differentials for technical competence on the GS scale to be somewhere near market (up and down). However, assessing technical competence isn't exactly straightforward.
- steven777400 10y ago
  As I mentioned in another thread, I work for state government and our IT salaries are capped at a max of around $87k/year (most positions are capped lower). As a result, it's extremely difficult to hire someone who isn't a net negative to the team. Since we have no power to adjust salaries, we're looking at other avenues, such as doing more "direct from community college" in-training hires as well as non-traditional employment paths. Unlike USDS and 18F, we don't even have exciting projects, a shortcut around bureaucracy, or cutting edge technologies to sell with. The only sales tool we have for candidates is "if you complete your 6 months probationary, you can essentially never be fired or laid off." Sometimes, that attracts the wrong sort of candidate, as you can probably imagine. :)
mtgx 10y ago
I wonder how many of these people will end up being recruited by the NSA. Sounds like the "NSA boycott" is about to be over soon. http://www.npr.org/2015/03/31/395829446/after-snowden-the-nsa-faces-recruitment-challenge http://www.npr.org/2015/03/31/395829446/after-snowden-the-ns...
vonmoltke 10y ago
Great to hear Ash Carter is so behind this. That makes me want to sign up for the Defense Digital Service, assuming they have work that isn't frontend-heavy.
elicash 10y ago
I love this part: Another project involves working on the Global Positioning System. Lynch still can’t believe that he can work on something that cool while serving his country. “I didn’t even know that we ran GPS, right?” he says. “That’s crazy shit.”
- MrMullen 10y ago
  It is funny to explain to people that don't know anything about GPS that the government not only runs it, the Department of Defense is the steward of it. I guess people assume that something as "cool" as GPS system can't be owned and operated by the US government.
pp19dd 10y ago
> These entities cannot continue to operate as both buyer and seller. There is an inherent conflict of interest when the entity that consults on how to solve a problem then also offers to sell you the solution. The government outlawed such arrangements for contractors, so they should not be permitted to create such a dynamic internally. That's a very strong accusation from Hodgkins (linked blog post in the article).
Klonoar 10y ago
This article does a good job of showing it, but everyone talked about is seriously amazing. I made it through all the interview pieces and ran into some other life complications so wound up not getting to work with them, but having been by their space in the Pentagon they're doing some of the coolest stuff imaginable right now. If you're looking for a new job, I'd highly recommend giving it a shot. Definitely felt like an opportunity unmatched elsewhere right now.
cs702 10y ago
The federal government hires a team of young, talented, motivated engineers and managers, puts them in charge of failed software projects, gives them the resources and authority they require to turns things around, and -- surprise! -- it turns out they do a GREAT job. In hindsight, this shouldn't be too surprising. What might be surprising is that the same logic should apply to ALL government functions, not just software development. Who says government projects and agencies have to be poorly run? Who says we can't devise better systems and/or mechanisms (e.g., market-based mechanisms) for attracting and retaining great people in government? Throughout history, there are examples of government bureaucracies that have been reasonably well run, in some cases for centuries. The Habsburg empire, in particular, comes to mind: http://www.independent.co.uk/news/world/europe/habsburg-empire-territories-europe-have-higher-trust-in-justice-systems-a6893946.html http://www.independent.co.uk/news/world/europe/habsburg-empi...
- vonmoltke 10y ago
  > Why do government projects and agencies have to be poorly run? Because lots of people, inside and outside of government, derive significant money, power, or both from government dysfunction. A classic case of goal misalignment.
  - mtbcoder 10y ago
    To provide a quote from the article: ..two lobbyists who managed to find their way onto the five-person panel testifying. They represented the interests of traditional IT contractors, who seem to believe it is their right to overcharge taxpayers for complex computer systems that don’t work. Even though, as one congressperson at the hearing put it, those special interests are likely to view the USDS and 18F “with a jaundiced eye,” they were given the opportunity to seize on the authority issue as a way to cast doubts on the Obama tech surge.
  - GVIrish 10y ago
    That's part of it. Back in the mid-nineties the idea that the private sector could do government functions cheaper/better really took hold and the result is that whole core sets of responsibilities are almost completed outsourced to contractors. That in turn has made lobbyists and recently retired senior government and military employees wealthy when they join those same contractors. However, a problem that has been around a long time is that it is difficult to fire government people for under performance so you have some people in positions of power crippling a whole host of IT efforts through their incompetence. That's not so much about money, it's just that the dead wood is hard to get rid of.
kkhire 10y ago
how cool would it be if this obama tech team really bolstered recruitment of more software engineers into gov't? I'm talking federal, state, and local level. Just like each state has senators and reps; 1 engineer for every 1M residents that live in the state. each state's teams would tackle problems with software, share solutions to widespread, formulaic issues with one another. it'd be a iconic and fresh-breath-of-air kind of way to speed up efficiency and effectiveness (government and big corps are SO SLOW)
GVIrish 10y ago
It's heartening to see some new approaches government IT actually getting high level support to make a positive impact. Funnily enough I see someone I used to work with in one of the pictures. Hopefully this will lead to more substantive change to how government IT is done. It's great to have teams that can drop into problem areas with a clear mandate to get things moving, but the whole system needs some serious reforms so that these IT quagmires don't get created in the first place. This article heaps a lot of blame on contractors but fact of the matter is, contractors are often hamstrung by what the customer wants them to do and what the customer allows them to do. And some of that comes back to what was specified in the original contract. Ultimately there are multiple components to the abysmal state of government IT projects: * The government has moved to a model where very few full time employees are technical people. The idea is that almost all IT will be outsourced to the private sector * Because fewer and fewer FTE's are technical, procurement and project management decision-making is compromised. * There is little accountability for government employees, and by extension, contractors. Botched projects where millions of dollars are waste rarely have major career consequences for FTE's, and the bigger contracting companies don't have their ability to get new contracts affected at all. * Building software for government often involves arduous interpretation of regulations, trying to build to undocumented and byzantine business processes, or waiting an inordinate amount of time for the bureaucracy to make important decisions (then change their mind!). I feel like empowering the technical people is a very important first step, and getting them executive backing may create the momentum needed to change some of the common pitfalls. But I think there are some big changes that have to happen to the whole ecosystem to really get things to where they need to be. It's all well and good to give small teams exceptional power to get around the bureaucracy but that approach doesn't fix the underlying problems.
- pjc50 10y ago
  The contractors are hamstrung by being contractors. Because the contracting process is adversarial, and geared around a fixed objective against which fixed lowest bids can be made, the kind of iterative development which produces great results is ruled out.
- jbooth 10y ago
  The contractors are hamstrung by their own business model. I'm sure a lot of the people on the ground are trying to do a good job, but when you're got a set-in-stone spec negotiated by people 4 levels of reporting away from the actual work in the trenches.. yeah that's probably not going to be a spec to do the right thing. Whether they're contracted or government employees, we need to devolve authority from big top-down specs and towards programmers working alongside the actual users.
gumby 10y ago
Great article. For me this is the money quote: “The culture is very persecutory. If something goes wrong it’s usually followed by a witch-hunt, and the result is very chilling. People are afraid to take risks.” This applies top to bottom (and has nothing to do with the current rage between republicans and democrats). The 1400 pages of travel regulations are a classic symptom.
pjungwir 10y ago
I'm curious if anyone here from USDS or 18F has deployed Postgres in a FIPS 140-2 environment, and if so what challenges they had. The VA seems to be saying that Postgres should not be used: http://www.va.gov/TRM/ToolPage.asp?tid=5692&tab=2 http://www.va.gov/TRM/ToolPage.asp?tid=5692&tab=2 contrast that with Oracle: http://www.va.gov/TRM/ToolPage.asp?tid=9&tab=2 http://www.va.gov/TRM/ToolPage.asp?tid=9&tab=2 (Don't miss the difference in tone on the "Analysis" tabs.) I'm sure some of that is due to lobbyists, but nonetheless it seems to me that there are legitimate challenges in making Postgres meet FIPS 140-2 requirements. I've been able to recompile Postgres to use the FIPS OpenSSL wrapper, but storing passwords with MD5 is a harder issue to fix, and of course there is no definitive list. Does anyone have some experience with this?
- Poiesis 10y ago
  Contractor here, not USDS or 18F. I usually see Postgres configured to use PAM auth, often against LDAP. This can it make it easier to meet password complexity requirements (don't get me started) and lockouts after incorrect attempts (same).
- saosebastiao 10y ago
  I'm pretty sure the only OS that is approved for electronic voting systems is Windows XP, which has obviously become a problem, but a new OS would likely take two or more years to go through the approval process. Ubuntu changes so much in two years that by the time it is approved, it would have lost 2/3rds of its life even as an LTR. I would assume that the main reason that Oracle is preferred is due to lobbying, but also the fact that they don't really innovate anymore so it is non-trivial for them to provide support and bug fixes for a decade or two.
  - nickparker 10y ago
    Does anyone else find it extremely discomforting that the voting machines even have a full fledged OS? It's a throwback to this xkcd https://xkcd.com/463/ https://xkcd.com/463/ In my mind, the ideal electronic voting machine: - Has mechanical buttons which simultaneously punch a paper ballot in a manner observable to the voter - Runs on a small (open source) microcontroller that's been audited for backdoors. - Runs (open source) crypto directly on the metal - Publishes the results in a cryptographically verifiable way. IMO, anything else is evidence of at least government/contractor ineptitude, and potentially malfeasance.
ryanmarsh 10y ago
It's my dream to work at USDS. First I need to save enough money to cover the difference in salary for 2-3 years.
- gmarx 10y ago
  I worked on a government contract years ago. The government employees always told me that their pay was better than it seemed because of the amazing benefits including a great vacation policy. I never made a spreadsheet to test this assertion
- Klonoar 10y ago
  Mmmm, I've been through the process and gotten an offer, and it wasn't that far off what the industry standard is (and honestly beats regular DC rates). They're really solid overall. Granted everyone's got their own experiences, so YMMV I guess. Just curious where your data comes from?
  - ryanmarsh 10y ago
    I read (sorry I don't recall where) a comment on GS levels for USDS. It was like half my current income. Perhaps I should just apply and find out for sure though.
    - knowaveragejoe 10y ago
      https://www.usds.gov/join#compensation https://www.usds.gov/join#compensation
saosebastiao 10y ago
What I would love to see is an 18F-like organization that creates open source software that helps user-facing government organizations at the state and local level. All of these things should be simple but (typically) are not: * Paying a utility bill online * Registering to vote * Finding out how to get from place to place using public transportation * Paying for public transportation passes * Signing up for unemployment benefits * Proving residency * Getting an ID card * Registering a vehicle * Paying taxes * Entering pleas to local courts for minor offenses and infractions without taking a day off work * Reporting non-emergency code violations (like construction noise after hours) * Knowing who your representatives are * Communicating with your representatives when you do know who they are We actually use and interact with local and state governments far more than we do with the federal government. And yet when it comes down to it, they are the most difficult to work with because their resources are so constrained. I honestly believe that local governments could benefit from some resource allocation on their behalf at the federal level for the creation of highly scalable technological public goods that will only ever be used at the local level.
- vonmoltke 10y ago
  Granted this is one anecdote, but in my experience state and local governments in North Texas are generally much easier to deal with than the federal government. Some are better than others, but I haven't run into any yet that are really bad. Some, like the Dallas Police Department and Dallas Area Rapid Transit, even have mobile apps to facilitate activities that are often done from a phone but awkward to do via the web.
- liyanchang 10y ago
  Disclosure: I'm an engineer at USDS and these are my own opinions. I absolutely agree that the interface that most citizens use to government can be better. President Obama agrees with you (from his sxsw remarks): "I could change the politics of America faster than just about anything if I could just take control of all the DMVs in the country." "If their primary interaction with government is the IRS, you just don’t have a good association with government when you’re writing that check." USDS has been working with the IRS to help them get their Transcript service back online with better identity proofing though admittedly just a small step in a better online experience. (irs.gov/transcript) A number of state and local governments have been making improvements as well (much harder to see if your state or your local government has not been). I'd encourage you to search them out or even start it.
glup 10y ago
A note on personal experience as a tech contractor at a federal agency: I worked as a contractor at the US Geological Survey (USGS), in the Department of the Interior each summer between 2006 and 2010. The most useful thing I worked on was a program for calculating air-sea carbon flux: I did the OS X GUI and helped with the core logic (computing air-sea carbon flux is nontrivial). At the time I was appalled that we spent $10k on what could have just been an R package around a wrapper for the C++ core. But since then the program has become a very common tool for ocean acidification research, with ~175 citations, many of these have since been cited in all sorts of climate change policy documents. It's admittedly hard to assess value, but it's definitely more than 10k. And definitely a public good that the market cannot be counted on to deliver. I also found out at some point that the contractor I was working for (a Fortune 500 engineering and defense company) billed the USGS more than twice what I received (I got no real benefits). The federal employees I worked with detested the contracting company because they knew how much it was skimming from the contract employees, many of whom were full-time and had been there for years. When I figured this all out, I quit, started my own company, and was hired directly as an independent contractor. Unfortunately, this was not an option for most people, as this wouldn't have worked for a full-time, year-round position.
- liyanchang 10y ago
  Disclosure: I'm an engineer at USDS and these are my own opinions. I'm personally appreciative of the work that you've done and your continued work, in spite of the many challenges and frustrations. There are many ways to serve. It's always incredibly rewarding to be working along side dedicated and talented public servants and contractors. Thank you.
knowaveragejoe 10y ago
Some really heartening news here. I hope it continues under the next president and doesn't succumb to the lobbying efforts by the larger contractors.
deleted 10y ago
[deleted]
dkrich 10y ago
Unfortunately, there's quite a bit of misinformation in this article. The author makes it sound as simple as hiring fresh hackers from Silicon Valley to take over failed projects from stodgy software developers from the large D.C. contracting firms. Having worked on government projects like those cited in this article for several years, I can tell you that it is the government agencies themselves, and not so much the developers who are to blame. To build a cool iPhone app for a private service, you can pretty much use whatever tools and timetables you want. Not so with government tech. With even small projects, you are beholden to policies that were written several levels up. Things like no cloud-based hosting, no sites that don't use government-written style guidelines, and in many cases, no development that occurs outside the government facility. If you hamstrung pretty much any private tech company with the same restrictions government contractors have to deal with, I can guarantee you they would look a lot different. That's not to say that change can't occur, but to attack the contractors is to misunderstand the root cause of why government software is historically so bad. There are actually many, many very talented developers and software engineers steeped in the latest lean startup theories inside the beltway, but they can seldom use any of that because government policies stand in their way. For most projects, you are dealing with literally thousands of regulations throughout the development process, and most software apps have to account for ever-changing laws and policies that are specific to each application. However I have already seen a great deal of positive change, with many agencies opening their minds to newer, faster methods of development and allowing better tools to be used.
- jessriedel 10y ago
  So it's an accident that this involved fresh hackers from Silicon Valley?
  - dkrich 10y ago
    Not an accident- selectivity bias. Ask those hackers to build an app for the USPS, FDA, the DOE or any other agency that imposes strict regulations and has to implement business logic to account for tens of thousands of rules (not an exaggeration) and see how agile and innovative they are. There's a reason that most of these contracts are worth between 7 and 8 figures. That's the kind of manpower required to engineer them.
mbesto 10y ago
My favorite part of this whole situation: "Start Up Life Meets the Compliance Riddled Federal IT Market for 18F and USDS" by A.R. "Trey" Hodgkins (ITI) https://www.itic.org/news-events/techwonk-blog/start-up-life-meets-the-compliance-riddled-federal-it-market-for-18f-and-usds https://www.itic.org/news-events/techwonk-blog/start-up-life... 18F’s and USDS’s heavy reliance on open source and custom coding raises serious concerns. While open source is an entirely appropriate option for agencies to consider when looking to acquire software solutions for mission needs, we need to ensure that these are not the only options available as different missions have different demands. Making open source software the only preferred solution doesn’t adhere to the tenets of tech neutrality. It also limits options and competition, which means we may not be achieving best value for the taxpayer.
arh68 10y ago
I'm excited, skeptical, and interested all at once. When the hackers leave, who's going to end up doing maintenance on these projects? It's not unusual for a gov't developer to work on a project just long enough to be productive, only to have them leave for greener pastures (often a promotion or relocation). In this case, most will want to stop suffering at lower salary and go back to SV. So it seems like it's going to be USDS holding the bag; unless there's a solid plan to transition the workload back to the VA (either organic or contractor), then USDS will have to maintain every project they start (forever). On the other hand, I bet there are more than a couple good developers within the VA that would love to take over the projects, if only there were a permanent position they could do it from. Why can't Ash Carter set up more permanent positions? Worst case scenario, a new President comes in and sunsets USDS, requiring VA to maintain the workload. VA balks, they hand it to Deloitte/BAH/etc, and we're back to square one. Best case scenario, USDS build quality solutions that last decades and can become the next legacy system that VA builds on. And maybe USDS (if it's still around) can re-revamp the system, etc etc. Also, I really don't understand the lobbyist's angle on open source: isn't all code written by a federal employee already in the public domain? (with obvious exceptions re: security etc)
infinite8s 10y ago
There's something about the second photo showing their 'funky' office. That looks like the team member was stuck in an unfinished basement (evoking memories of Office Space). https://cdn-images-1.medium.com/max/600/1*Og10aHFHpbFXrwD7cGYzkw.jpeg https://cdn-images-1.medium.com/max/600/1*Og10aHFHpbFXrwD7cG...

162 points by brensudol 10y ago

"U.S. Digital Service salaries are set in accordance with the General Schedule Pay Scale, which is capped at $160,300." That's not bad, but not great for really high talent employees. I wonder how many people get that rate though. https://www.usds.gov/join#compensation https://www.usds.gov/join#compensation

vonmoltke 10y ago
I'm not sure that's much of an issue, since it is pretty good money (even in DC) for the 99% of us who aren't "really high talent employees".
- impostervt 10y ago
  True, but I wonder what the "normal" rate is. As in, do they really get that much, or by stating that rate on their website are they really just trying to say "if you make a lot more than this and care a lot about money, don't bother".
  - ryanmarsh 10y ago
    Which is basically why I'm saving to go work there.
    - zacharycohn 10y ago
      Apply soon! We're hungry for help. [Disclaimer: I work at 18F.]
sbov 10y ago
Even if I made more than that in the private sector, I would be willing to take a minor pay cut for this sort of position. If you think about it, you're helping put taxpayer dollars to more efficient use, which includes your own.
- macintux 10y ago
  Not sure how many people are going to take a 10% pay cut in hopes that 5 years from now their taxes might be a few bucks cheaper (which of course they won't be because the money would just be spent elsewhere). Admirable intent, I'm suspicious of its prevalence.
  - ryanmarsh 10y ago
    They call it a tour of duty for a reason. $163k would be a massive paycut for me but I'd do it and happily burn savings to make a difference for the lives of my fellow veterans.
    - sailfast 10y ago
      Love your second sentence, but really wish they would stop calling it a "tour of duty". It's still a pretty solid amount of money (nevermind Federal benefits) but to really transform we need people willing to work over the long haul not just parachute in for short terms. Also (slightly biased of course) I bet you get a bit addicted to the mission and that provides a non-monetary incentive to stick around a bit longer :)
      - zacharycohn 10y ago
        [Disclaimer: I work at 18F.] It's referred to as a tour of duty because most of us are on 2 year appointments. At the end of that 2 years, there's an optional 2 year extension. These orgs haven't been around long enough for anyone to run out their second two years yet. Personally, I love this format. It never would have even dawned on me to consider applying for a career position in the federal government, but when the job is time-limited, there's some decision making bug in the brain (even though I could have applied for a normal career position and then just left after 4 years!). It's a great way to entice people to apply who never would have before. We also need people in for the long haul. But lets solve one problem at a time!
        10y ago
spacecowboy_lon 10y ago
Presumably you get the pension that goes with that
- vonmoltke 10y ago
  Depends. My wife only has the $9000 in her Thrift Savings Plan after four years at the White House. She was a "political appointee", though, so she didn't get normal GS benefits.
deleted 10y ago
[deleted]
liyanchang 10y ago
Disclosure: I'm an engineer at USDS and these are my own opinions. That's right. The US Government is probably not going to win on compensation - salary limits, no stock options, no lunch. However, while it's not a money making enterprise, it's enough to do just fine. I recognize the sacrifices that many of my colleagues have made to be here, not to mention the sacrifices that federal employees and contractors have made (some who are very good and could be making more in the private sector) and that makes this even more worthwhile. The thing that government can offer is impact. I've always known that government has a big impact on people's lives but not sure if I could personally make an impact. That's what USDS, 18F, a number of other opportunities are offering. This is not a job for everyone. But if you're a certain type, there's nothing quite like it.
- mattmcknight 10y ago
  The government could do a lot more by adjusting the differentials for technical competence on the GS scale to be somewhere near market (up and down). However, assessing technical competence isn't exactly straightforward.
steven777400 10y ago
As I mentioned in another thread, I work for state government and our IT salaries are capped at a max of around $87k/year (most positions are capped lower). As a result, it's extremely difficult to hire someone who isn't a net negative to the team. Since we have no power to adjust salaries, we're looking at other avenues, such as doing more "direct from community college" in-training hires as well as non-traditional employment paths. Unlike USDS and 18F, we don't even have exciting projects, a shortcut around bureaucracy, or cutting edge technologies to sell with. The only sales tool we have for candidates is "if you complete your 6 months probationary, you can essentially never be fired or laid off." Sometimes, that attracts the wrong sort of candidate, as you can probably imagine. :)

mtgx 10y ago

I wonder how many of these people will end up being recruited by the NSA. Sounds like the "NSA boycott" is about to be over soon. http://www.npr.org/2015/03/31/395829446/after-snowden-the-nsa-faces-recruitment-challenge http://www.npr.org/2015/03/31/395829446/after-snowden-the-ns...

vonmoltke 10y ago

Great to hear Ash Carter is so behind this. That makes me want to sign up for the Defense Digital Service, assuming they have work that isn't frontend-heavy.

elicash 10y ago

I love this part: Another project involves working on the Global Positioning System. Lynch still can’t believe that he can work on something that cool while serving his country. “I didn’t even know that we ran GPS, right?” he says. “That’s crazy shit.”

MrMullen 10y ago
It is funny to explain to people that don't know anything about GPS that the government not only runs it, the Department of Defense is the steward of it. I guess people assume that something as "cool" as GPS system can't be owned and operated by the US government.

pp19dd 10y ago

> These entities cannot continue to operate as both buyer and seller. There is an inherent conflict of interest when the entity that consults on how to solve a problem then also offers to sell you the solution. The government outlawed such arrangements for contractors, so they should not be permitted to create such a dynamic internally. That's a very strong accusation from Hodgkins (linked blog post in the article).

Klonoar 10y ago

This article does a good job of showing it, but everyone talked about is seriously amazing. I made it through all the interview pieces and ran into some other life complications so wound up not getting to work with them, but having been by their space in the Pentagon they're doing some of the coolest stuff imaginable right now. If you're looking for a new job, I'd highly recommend giving it a shot. Definitely felt like an opportunity unmatched elsewhere right now.

cs702 10y ago

The federal government hires a team of young, talented, motivated engineers and managers, puts them in charge of failed software projects, gives them the resources and authority they require to turns things around, and -- surprise! -- it turns out they do a GREAT job. In hindsight, this shouldn't be too surprising. What might be surprising is that the same logic should apply to ALL government functions, not just software development. Who says government projects and agencies have to be poorly run? Who says we can't devise better systems and/or mechanisms (e.g., market-based mechanisms) for attracting and retaining great people in government? Throughout history, there are examples of government bureaucracies that have been reasonably well run, in some cases for centuries. The Habsburg empire, in particular, comes to mind: http://www.independent.co.uk/news/world/europe/habsburg-empire-territories-europe-have-higher-trust-in-justice-systems-a6893946.html http://www.independent.co.uk/news/world/europe/habsburg-empi...

vonmoltke 10y ago
> Why do government projects and agencies have to be poorly run? Because lots of people, inside and outside of government, derive significant money, power, or both from government dysfunction. A classic case of goal misalignment.
- mtbcoder 10y ago
  To provide a quote from the article: ..two lobbyists who managed to find their way onto the five-person panel testifying. They represented the interests of traditional IT contractors, who seem to believe it is their right to overcharge taxpayers for complex computer systems that don’t work. Even though, as one congressperson at the hearing put it, those special interests are likely to view the USDS and 18F “with a jaundiced eye,” they were given the opportunity to seize on the authority issue as a way to cast doubts on the Obama tech surge.
- GVIrish 10y ago
  That's part of it. Back in the mid-nineties the idea that the private sector could do government functions cheaper/better really took hold and the result is that whole core sets of responsibilities are almost completed outsourced to contractors. That in turn has made lobbyists and recently retired senior government and military employees wealthy when they join those same contractors. However, a problem that has been around a long time is that it is difficult to fire government people for under performance so you have some people in positions of power crippling a whole host of IT efforts through their incompetence. That's not so much about money, it's just that the dead wood is hard to get rid of.

kkhire 10y ago

how cool would it be if this obama tech team really bolstered recruitment of more software engineers into gov't? I'm talking federal, state, and local level. Just like each state has senators and reps; 1 engineer for every 1M residents that live in the state. each state's teams would tackle problems with software, share solutions to widespread, formulaic issues with one another. it'd be a iconic and fresh-breath-of-air kind of way to speed up efficiency and effectiveness (government and big corps are SO SLOW)

GVIrish 10y ago

It's heartening to see some new approaches government IT actually getting high level support to make a positive impact. Funnily enough I see someone I used to work with in one of the pictures. Hopefully this will lead to more substantive change to how government IT is done. It's great to have teams that can drop into problem areas with a clear mandate to get things moving, but the whole system needs some serious reforms so that these IT quagmires don't get created in the first place. This article heaps a lot of blame on contractors but fact of the matter is, contractors are often hamstrung by what the customer wants them to do and what the customer allows them to do. And some of that comes back to what was specified in the original contract. Ultimately there are multiple components to the abysmal state of government IT projects: * The government has moved to a model where very few full time employees are technical people. The idea is that almost all IT will be outsourced to the private sector * Because fewer and fewer FTE's are technical, procurement and project management decision-making is compromised. * There is little accountability for government employees, and by extension, contractors. Botched projects where millions of dollars are waste rarely have major career consequences for FTE's, and the bigger contracting companies don't have their ability to get new contracts affected at all. * Building software for government often involves arduous interpretation of regulations, trying to build to undocumented and byzantine business processes, or waiting an inordinate amount of time for the bureaucracy to make important decisions (then change their mind!). I feel like empowering the technical people is a very important first step, and getting them executive backing may create the momentum needed to change some of the common pitfalls. But I think there are some big changes that have to happen to the whole ecosystem to really get things to where they need to be. It's all well and good to give small teams exceptional power to get around the bureaucracy but that approach doesn't fix the underlying problems.

pjc50 10y ago
The contractors are hamstrung by being contractors. Because the contracting process is adversarial, and geared around a fixed objective against which fixed lowest bids can be made, the kind of iterative development which produces great results is ruled out.
jbooth 10y ago
The contractors are hamstrung by their own business model. I'm sure a lot of the people on the ground are trying to do a good job, but when you're got a set-in-stone spec negotiated by people 4 levels of reporting away from the actual work in the trenches.. yeah that's probably not going to be a spec to do the right thing. Whether they're contracted or government employees, we need to devolve authority from big top-down specs and towards programmers working alongside the actual users.

gumby 10y ago

Great article. For me this is the money quote: “The culture is very persecutory. If something goes wrong it’s usually followed by a witch-hunt, and the result is very chilling. People are afraid to take risks.” This applies top to bottom (and has nothing to do with the current rage between republicans and democrats). The 1400 pages of travel regulations are a classic symptom.

pjungwir 10y ago

I'm curious if anyone here from USDS or 18F has deployed Postgres in a FIPS 140-2 environment, and if so what challenges they had. The VA seems to be saying that Postgres should not be used: http://www.va.gov/TRM/ToolPage.asp?tid=5692&tab=2 http://www.va.gov/TRM/ToolPage.asp?tid=5692&tab=2 contrast that with Oracle: http://www.va.gov/TRM/ToolPage.asp?tid=9&tab=2 http://www.va.gov/TRM/ToolPage.asp?tid=9&tab=2 (Don't miss the difference in tone on the "Analysis" tabs.) I'm sure some of that is due to lobbyists, but nonetheless it seems to me that there are legitimate challenges in making Postgres meet FIPS 140-2 requirements. I've been able to recompile Postgres to use the FIPS OpenSSL wrapper, but storing passwords with MD5 is a harder issue to fix, and of course there is no definitive list. Does anyone have some experience with this?

Poiesis 10y ago
Contractor here, not USDS or 18F. I usually see Postgres configured to use PAM auth, often against LDAP. This can it make it easier to meet password complexity requirements (don't get me started) and lockouts after incorrect attempts (same).
saosebastiao 10y ago
I'm pretty sure the only OS that is approved for electronic voting systems is Windows XP, which has obviously become a problem, but a new OS would likely take two or more years to go through the approval process. Ubuntu changes so much in two years that by the time it is approved, it would have lost 2/3rds of its life even as an LTR. I would assume that the main reason that Oracle is preferred is due to lobbying, but also the fact that they don't really innovate anymore so it is non-trivial for them to provide support and bug fixes for a decade or two.
- nickparker 10y ago
  Does anyone else find it extremely discomforting that the voting machines even have a full fledged OS? It's a throwback to this xkcd https://xkcd.com/463/ https://xkcd.com/463/ In my mind, the ideal electronic voting machine: - Has mechanical buttons which simultaneously punch a paper ballot in a manner observable to the voter - Runs on a small (open source) microcontroller that's been audited for backdoors. - Runs (open source) crypto directly on the metal - Publishes the results in a cryptographically verifiable way. IMO, anything else is evidence of at least government/contractor ineptitude, and potentially malfeasance.

ryanmarsh 10y ago

It's my dream to work at USDS. First I need to save enough money to cover the difference in salary for 2-3 years.

gmarx 10y ago
I worked on a government contract years ago. The government employees always told me that their pay was better than it seemed because of the amazing benefits including a great vacation policy. I never made a spreadsheet to test this assertion
Klonoar 10y ago
Mmmm, I've been through the process and gotten an offer, and it wasn't that far off what the industry standard is (and honestly beats regular DC rates). They're really solid overall. Granted everyone's got their own experiences, so YMMV I guess. Just curious where your data comes from?
- ryanmarsh 10y ago
  I read (sorry I don't recall where) a comment on GS levels for USDS. It was like half my current income. Perhaps I should just apply and find out for sure though.
  - knowaveragejoe 10y ago
    https://www.usds.gov/join#compensation https://www.usds.gov/join#compensation

saosebastiao 10y ago

What I would love to see is an 18F-like organization that creates open source software that helps user-facing government organizations at the state and local level. All of these things should be simple but (typically) are not: * Paying a utility bill online * Registering to vote * Finding out how to get from place to place using public transportation * Paying for public transportation passes * Signing up for unemployment benefits * Proving residency * Getting an ID card * Registering a vehicle * Paying taxes * Entering pleas to local courts for minor offenses and infractions without taking a day off work * Reporting non-emergency code violations (like construction noise after hours) * Knowing who your representatives are * Communicating with your representatives when you do know who they are We actually use and interact with local and state governments far more than we do with the federal government. And yet when it comes down to it, they are the most difficult to work with because their resources are so constrained. I honestly believe that local governments could benefit from some resource allocation on their behalf at the federal level for the creation of highly scalable technological public goods that will only ever be used at the local level.

vonmoltke 10y ago
Granted this is one anecdote, but in my experience state and local governments in North Texas are generally much easier to deal with than the federal government. Some are better than others, but I haven't run into any yet that are really bad. Some, like the Dallas Police Department and Dallas Area Rapid Transit, even have mobile apps to facilitate activities that are often done from a phone but awkward to do via the web.
liyanchang 10y ago
Disclosure: I'm an engineer at USDS and these are my own opinions. I absolutely agree that the interface that most citizens use to government can be better. President Obama agrees with you (from his sxsw remarks): "I could change the politics of America faster than just about anything if I could just take control of all the DMVs in the country." "If their primary interaction with government is the IRS, you just don’t have a good association with government when you’re writing that check." USDS has been working with the IRS to help them get their Transcript service back online with better identity proofing though admittedly just a small step in a better online experience. (irs.gov/transcript) A number of state and local governments have been making improvements as well (much harder to see if your state or your local government has not been). I'd encourage you to search them out or even start it.

glup 10y ago

A note on personal experience as a tech contractor at a federal agency: I worked as a contractor at the US Geological Survey (USGS), in the Department of the Interior each summer between 2006 and 2010. The most useful thing I worked on was a program for calculating air-sea carbon flux: I did the OS X GUI and helped with the core logic (computing air-sea carbon flux is nontrivial). At the time I was appalled that we spent $10k on what could have just been an R package around a wrapper for the C++ core. But since then the program has become a very common tool for ocean acidification research, with ~175 citations, many of these have since been cited in all sorts of climate change policy documents. It's admittedly hard to assess value, but it's definitely more than 10k. And definitely a public good that the market cannot be counted on to deliver. I also found out at some point that the contractor I was working for (a Fortune 500 engineering and defense company) billed the USGS more than twice what I received (I got no real benefits). The federal employees I worked with detested the contracting company because they knew how much it was skimming from the contract employees, many of whom were full-time and had been there for years. When I figured this all out, I quit, started my own company, and was hired directly as an independent contractor. Unfortunately, this was not an option for most people, as this wouldn't have worked for a full-time, year-round position.

liyanchang 10y ago
Disclosure: I'm an engineer at USDS and these are my own opinions. I'm personally appreciative of the work that you've done and your continued work, in spite of the many challenges and frustrations. There are many ways to serve. It's always incredibly rewarding to be working along side dedicated and talented public servants and contractors. Thank you.

knowaveragejoe 10y ago

Some really heartening news here. I hope it continues under the next president and doesn't succumb to the lobbying efforts by the larger contractors.

deleted 10y ago

[deleted]

dkrich 10y ago

Unfortunately, there's quite a bit of misinformation in this article. The author makes it sound as simple as hiring fresh hackers from Silicon Valley to take over failed projects from stodgy software developers from the large D.C. contracting firms. Having worked on government projects like those cited in this article for several years, I can tell you that it is the government agencies themselves, and not so much the developers who are to blame. To build a cool iPhone app for a private service, you can pretty much use whatever tools and timetables you want. Not so with government tech. With even small projects, you are beholden to policies that were written several levels up. Things like no cloud-based hosting, no sites that don't use government-written style guidelines, and in many cases, no development that occurs outside the government facility. If you hamstrung pretty much any private tech company with the same restrictions government contractors have to deal with, I can guarantee you they would look a lot different. That's not to say that change can't occur, but to attack the contractors is to misunderstand the root cause of why government software is historically so bad. There are actually many, many very talented developers and software engineers steeped in the latest lean startup theories inside the beltway, but they can seldom use any of that because government policies stand in their way. For most projects, you are dealing with literally thousands of regulations throughout the development process, and most software apps have to account for ever-changing laws and policies that are specific to each application. However I have already seen a great deal of positive change, with many agencies opening their minds to newer, faster methods of development and allowing better tools to be used.

jessriedel 10y ago
So it's an accident that this involved fresh hackers from Silicon Valley?
- dkrich 10y ago
  Not an accident- selectivity bias. Ask those hackers to build an app for the USPS, FDA, the DOE or any other agency that imposes strict regulations and has to implement business logic to account for tens of thousands of rules (not an exaggeration) and see how agile and innovative they are. There's a reason that most of these contracts are worth between 7 and 8 figures. That's the kind of manpower required to engineer them.

mbesto 10y ago

My favorite part of this whole situation: "Start Up Life Meets the Compliance Riddled Federal IT Market for 18F and USDS" by A.R. "Trey" Hodgkins (ITI) https://www.itic.org/news-events/techwonk-blog/start-up-life-meets-the-compliance-riddled-federal-it-market-for-18f-and-usds https://www.itic.org/news-events/techwonk-blog/start-up-life... 18F’s and USDS’s heavy reliance on open source and custom coding raises serious concerns. While open source is an entirely appropriate option for agencies to consider when looking to acquire software solutions for mission needs, we need to ensure that these are not the only options available as different missions have different demands. Making open source software the only preferred solution doesn’t adhere to the tenets of tech neutrality. It also limits options and competition, which means we may not be achieving best value for the taxpayer.

arh68 10y ago

I'm excited, skeptical, and interested all at once. When the hackers leave, who's going to end up doing maintenance on these projects? It's not unusual for a gov't developer to work on a project just long enough to be productive, only to have them leave for greener pastures (often a promotion or relocation). In this case, most will want to stop suffering at lower salary and go back to SV. So it seems like it's going to be USDS holding the bag; unless there's a solid plan to transition the workload back to the VA (either organic or contractor), then USDS will have to maintain every project they start (forever). On the other hand, I bet there are more than a couple good developers within the VA that would love to take over the projects, if only there were a permanent position they could do it from. Why can't Ash Carter set up more permanent positions? Worst case scenario, a new President comes in and sunsets USDS, requiring VA to maintain the workload. VA balks, they hand it to Deloitte/BAH/etc, and we're back to square one. Best case scenario, USDS build quality solutions that last decades and can become the next legacy system that VA builds on. And maybe USDS (if it's still around) can re-revamp the system, etc etc. Also, I really don't understand the lobbyist's angle on open source: isn't all code written by a federal employee already in the public domain? (with obvious exceptions re: security etc)

infinite8s 10y ago

There's something about the second photo showing their 'funky' office. That looks like the team member was stuck in an unfinished basement (evoking memories of Office Space). https://cdn-images-1.medium.com/max/600/1*Og10aHFHpbFXrwD7cGYzkw.jpeg https://cdn-images-1.medium.com/max/600/1*Og10aHFHpbFXrwD7cG...