6 ms·

I'm honestly surprised this is on HN, but it's good that it is. I worked with Nic on and off for almost his entire tenure while I was CTO for Kessel Run and I

I'm honestly surprised this is on HN, but it's good that it is. I worked with Nic on and off for almost his entire tenure while I was CTO for Kessel Run and I can state with full confidence that this is at best him mis-representing his importance and the problems with the DoD IT; and at worst this is his attempt to spin his being fired (or being asked to resign ala Nixon) by the incoming Secretary (timing here is not just a coincidence). A couple of core points, that are important to keep in mind that have nothing to do with Nic's character, integrity, communication style or technical capabilities (which is a separate and important topic but not suitable for this public forum IMO): - The CSO position was made up by him, it's not related to any GSA Schedule and it had about the kind of charter you would expect for the position: Namely ill-defined and loosely empowered. - There was no office of the CSO in the sense that it was not congressionally funded, had no budget, no personnel and no real authority for writing, implementing policy or actually doing engineering. - Nic never held a clearance, and as a result was never actually involved or aware of most of the programs that he intended to impact - His primary mission seemed to be to push any organization that was developing software for the USAF to immediately adopt microservices architectures, containers/kubernetes and a couple of very specific "DevSecOps" practices - and specifically to the specifications that he approved/suggested. Make of that what you will. That said, a lot of what he says is true and IT/network infrastructure, development and test etc... in the DoD is far from modern and in some places completely broken. Other places, where it matters a lot it's like nothing you've ever seen or will likely see in the commercial sector for decades. Bottom line, I suggest taking this tirade with an EXTREME amount of salt.

ElijahLynn 5y ago
I encourage you to leave this comment on the article itself on The Register since you already made it public here.
phkahler 5y ago
>> Other places, where it matters a lot it's like nothing you've ever seen or will likely see in the commercial sector for decades. That's something I'd really like to see. How does that kind of difference come about? My guess is that it requires a certain degree of funding and commitment that may be impossible in wallstreet companies. But what else does it take for an organization to get there?
Ziggy_Zaggy 5y ago
This is a very insightful and contrasting response. Do you have any other articles/materials that we can reference for additional information related to this topic?
ryanmarsh 5y ago
Other places, where it matters a lot it's like nothing you've ever seen or will likely see in the commercial sector for decades. It’s weird how the federal govt is like this across the board. Most things are “fine” being held together with bubblegum and duct tape. Some things matter a lot though, and when they do you get to see really smart people apply themselves in ways that are cooler than the movies.
enkid 5y ago
The idea that he could fix Air Force IT in 6 months if empowered seems absolutely ridiculous given the size of the organization. What do you think the US gov needs to change to get better at it?
- tyingq 5y ago
  He did mention that's it's frustrating that officers put in charge of IT work have no IT experience. That's not true for many other specialties in the military, where the officers will have extensive experience in the area they are leading. Doesn't solve anything by itself, but it would help.
  - Pokepokalypse 5y ago
    The fact is: you can't wave a magic wand, and suddenly crap out a person with high-level experience in these very young tools, who also happens to have officer-level rank in the USAF (or at the very least: experience with satisfying the operational and compliance requirements in this industry). I have no doubt such people exist. But not in the numbers that would be needed. It will take years or decades of focused effort to build that workforce.
    - mpyne 5y ago
      Exactly right, but the military services haven't even started to implement the changes that would be needed to grow and maintain a cadre of digital talent. Fred Brooks (a name some of you might recognize) recommended in a 1987 report for Defense Science Board on problems with DoD software that DoD needed to improve its uniformed and government software expertise. Even there he said such personnel would likely have to be heavily supplemented by contractor help -- but at least they'd know how to do the technical management. Fast forward 30+ years and there are pockets of talent but if anything the problems are larger as digital products have become more important to business operations around the world. Recent example from Reddit on DoD IT: https://i.redd.it/dyp2nyaca5l71.jpg https://i.redd.it/dyp2nyaca5l71.jpg
    - tyingq 5y ago
gfosco 5y ago
Likewise I will take this comment with an equal amount of salt.
deleted 5y ago
[deleted]
_bnmd 5y ago
Yeah. I aint saying who I am, but I'll anonymously corroborate this. Nic had a lot of good ideas, but he micromanaged the hell out of low-level product decisions, forced us to use specific broken products that made me swear the vendors must have naked photos of him somewhere. He threw new requirements out of left field in the middle of demos that were not real requirements written down anywhere, but for whatever reason, we had to change course in the middle of development and do it anyway. I'm sitting here right now on a Friday afternoon while the Air Force is on a four-day weekend ahead of labor day, trying to deploy to a broken ass application his DevSecOps reference architecture forces everyone to use, but it doesn't work because it uses a checksum algorithm disabled by FIPS-compliance hardening in this environment, which we have absolutely no control over. The biggest impediment to even getting this far was another vendor enterprise service he forced us to use that was broken until July. We were stuck just waiting on a bug fix to be provided. And, of course, we have to use Iron Bank container images for everything, but Iron Bank container images are themselves perpetually broken. They do security hardening, but no functionality testing, and their process of pushing breaking changes to the same tags can break you in production unexpectedly. And you can't pin to the actual sha because Harbor only holds onto five orphaned shas at a time that don't correspond to a tag. He's touting a lot of accomplishments here that are accomplishments because pushing broken functionality and calling it done is a very easy way to say you delivered faster than a normal DoD program that has to actually prove it does what it says it does.
- nchaillan 5y ago
  This is Nicolas. I'd welcome to get your proposed fixed so we can fix both Iron Bank and the other issues you mentioned here. Do remember that I'm not the one mandating FIPS validation. You know, there is a concept called cybersecurity.
  - _bnmd 5y ago
    The issue with FIPS compliance is with Anchore. It's actually fixed in 10.2, but that version is not yet in Iron Bank, so delaying deployment to IL5. The issue with Iron Bank itself is a lot more structural. We report it every time a specific container breaks, and it eventually gets fixed on a report-by-report basis, but the base issue is they're doing two things completely wrong: 1) The process requiring disconnected builds in the container build stage to avoid pulling in dependencies from the Internet, combined with lack of expertise on the part of people assigned to the container hardening teams in terms of how to actually build software, results in them usually copying down the upstream official container, and making a naive copy of the desired executable from that container into a UBI base image. That happens to work for dynamically linked executables when the upstream and UBI have the same glibc version, usually right after a UBI release. Then it later breaks spectacularly, and when the issue is explained to Iron Bank engineers, they don't understand and simply call it fixed when UBI releases again and glibc is aligned by chance with upstream. 2) They push functional changes to tags. You can pull some <image>:8.4 one day, and the next day the exactly same tag will have different environment variables set, different paths, will have executables removed (notably, the jq image used to include aws-cli for some reason, which it shouldn't have, but once it did, you can't just remove it). Normally, the fix for this is pin to a sha instead of the tag, but Harbor doesn't hold onto the shas once you have republished to the same tag five times, and Iron Bank is continuously rebuilding those tags. We've reported it and Iron Bank technical leadership is at least aware this is a problem, but figuring out how to fix it has never been a priority for them.
nchaillan 5y ago
To be clear, I am not gone nor was I asked to leave. In fact, I was asked to stay. I do have a clearance as well and yes sorry I didn't let us get locked into Pivotal like you did and cost taxpayer millions.
- Ziggy_Zaggy 5y ago
  Thanks for commenting on the thread. Can you refer us to your opinions on the matters expressed in the article in a public friendly fashion?
nchaillan 5y ago
Of course Andrew did his best to get KR locked in to PCF which is now taking us 2 years to walk away from. Well done friend.
poulsbohemian 5y ago
> Other places, where it matters a lot it's like nothing you've ever seen or will likely see in the commercial sector for decades. Would love to more about this as it sounds like you are implying that their practices and technology are significantly ahead of the commercial sector.
spfzero 5y ago
Thanks for posting this, the lack of a security clearance seems telling. Even contractors often have them. Also, a person in a real position of responsibility would never behave this way, and to be fair to him, they would never be treated this way.
hamburgerwah 5y ago
^^ lol, political hit jobs representing in these comments
hiram112 5y ago
> - Nic never held a clearance, and as a result was never actually involved or aware of most of the programs that he intended to impact I honestly don't know how he'd ever expect to succeed in a job like this without a clearance. Without it, he'd have little practical knowledge of how the actual air gapped networks (SIPR, JWICS, etc.) are designed and implemented. Even if he didn't know the classified technical details, there's a lot of differences between deploying software in these environments vs the wide open public internet. From experience, the type of exploits that would be devastating to a big company just aren't a big risk for most apps on these networks. OTOH, classified apps have all sorts of bureaucratic rules that the unclassified world doesn't have to consider. For example, if this guy is coming from Silicon Valley, he might spend 80% of his time worrying about exploits that cost corporate American billions / year and which are due to unpatched systems which become targets for script-kiddies pounding every IP on the internet again and again, looking to take advantage of dozens of known vulnerabilities. On the classified networks, these types of exploits just aren't as likely. There aren't fleets of bots in China and Russia trying known root passwords on every machine with port 22 open or trying to access open Mongo or Elastic Search systems on whole IP ranges. On the other hand, a private sector SecOps guy isn't even considering that they absolutely cannot store multiple big program's databases in the same data center, staffed by the same contracting company. Because there's a huge risk that some rogue IT contractor with general root access in some small signals intelligence "datacenter" in Hawaii or Germany will end up having access to way too much sensitive data that is shared by multiple intel agencies and military branches, and it'd be disastrous if he decided to start taking raw disks home with him each night a la Snowden. The Air Force and other DOD agencies certainly must ensure that a single contractor doesn't have access to too many data sources. And with everyone moving as fast as they can to AWS, Azure, etc, I'm guessing these type of procedure are far more important to the Air Force than worrying about some development team deploying an app on an older docker image based on some old version of Java or Python with a few known exploits, which could cost a private sector company a lot of pain and money the minute it was deployed on the open internet.
- Aeolun 5y ago
  I don’t think you want your airgap to be the only thing standing between your IT infra and total disaster. Especially in the military it seems entirely plausible that bad people gain access to your equipment.

ElijahLynn 5y ago

I encourage you to leave this comment on the article itself on The Register since you already made it public here.

phkahler 5y ago

>> Other places, where it matters a lot it's like nothing you've ever seen or will likely see in the commercial sector for decades. That's something I'd really like to see. How does that kind of difference come about? My guess is that it requires a certain degree of funding and commitment that may be impossible in wallstreet companies. But what else does it take for an organization to get there?

Ziggy_Zaggy 5y ago

This is a very insightful and contrasting response. Do you have any other articles/materials that we can reference for additional information related to this topic?

ryanmarsh 5y ago

Other places, where it matters a lot it's like nothing you've ever seen or will likely see in the commercial sector for decades. It’s weird how the federal govt is like this across the board. Most things are “fine” being held together with bubblegum and duct tape. Some things matter a lot though, and when they do you get to see really smart people apply themselves in ways that are cooler than the movies.

enkid 5y ago

The idea that he could fix Air Force IT in 6 months if empowered seems absolutely ridiculous given the size of the organization. What do you think the US gov needs to change to get better at it?

tyingq 5y ago
He did mention that's it's frustrating that officers put in charge of IT work have no IT experience. That's not true for many other specialties in the military, where the officers will have extensive experience in the area they are leading. Doesn't solve anything by itself, but it would help.
- Pokepokalypse 5y ago
  The fact is: you can't wave a magic wand, and suddenly crap out a person with high-level experience in these very young tools, who also happens to have officer-level rank in the USAF (or at the very least: experience with satisfying the operational and compliance requirements in this industry). I have no doubt such people exist. But not in the numbers that would be needed. It will take years or decades of focused effort to build that workforce.
  - mpyne 5y ago
    Exactly right, but the military services haven't even started to implement the changes that would be needed to grow and maintain a cadre of digital talent. Fred Brooks (a name some of you might recognize) recommended in a 1987 report for Defense Science Board on problems with DoD software that DoD needed to improve its uniformed and government software expertise. Even there he said such personnel would likely have to be heavily supplemented by contractor help -- but at least they'd know how to do the technical management. Fast forward 30+ years and there are pockets of talent but if anything the problems are larger as digital products have become more important to business operations around the world. Recent example from Reddit on DoD IT: https://i.redd.it/dyp2nyaca5l71.jpg https://i.redd.it/dyp2nyaca5l71.jpg
  - tyingq 5y ago

gfosco 5y ago

Likewise I will take this comment with an equal amount of salt.

deleted 5y ago

[deleted]

_bnmd 5y ago

Yeah. I aint saying who I am, but I'll anonymously corroborate this. Nic had a lot of good ideas, but he micromanaged the hell out of low-level product decisions, forced us to use specific broken products that made me swear the vendors must have naked photos of him somewhere. He threw new requirements out of left field in the middle of demos that were not real requirements written down anywhere, but for whatever reason, we had to change course in the middle of development and do it anyway. I'm sitting here right now on a Friday afternoon while the Air Force is on a four-day weekend ahead of labor day, trying to deploy to a broken ass application his DevSecOps reference architecture forces everyone to use, but it doesn't work because it uses a checksum algorithm disabled by FIPS-compliance hardening in this environment, which we have absolutely no control over. The biggest impediment to even getting this far was another vendor enterprise service he forced us to use that was broken until July. We were stuck just waiting on a bug fix to be provided. And, of course, we have to use Iron Bank container images for everything, but Iron Bank container images are themselves perpetually broken. They do security hardening, but no functionality testing, and their process of pushing breaking changes to the same tags can break you in production unexpectedly. And you can't pin to the actual sha because Harbor only holds onto five orphaned shas at a time that don't correspond to a tag. He's touting a lot of accomplishments here that are accomplishments because pushing broken functionality and calling it done is a very easy way to say you delivered faster than a normal DoD program that has to actually prove it does what it says it does.

nchaillan 5y ago
This is Nicolas. I'd welcome to get your proposed fixed so we can fix both Iron Bank and the other issues you mentioned here. Do remember that I'm not the one mandating FIPS validation. You know, there is a concept called cybersecurity.
- _bnmd 5y ago
  The issue with FIPS compliance is with Anchore. It's actually fixed in 10.2, but that version is not yet in Iron Bank, so delaying deployment to IL5. The issue with Iron Bank itself is a lot more structural. We report it every time a specific container breaks, and it eventually gets fixed on a report-by-report basis, but the base issue is they're doing two things completely wrong: 1) The process requiring disconnected builds in the container build stage to avoid pulling in dependencies from the Internet, combined with lack of expertise on the part of people assigned to the container hardening teams in terms of how to actually build software, results in them usually copying down the upstream official container, and making a naive copy of the desired executable from that container into a UBI base image. That happens to work for dynamically linked executables when the upstream and UBI have the same glibc version, usually right after a UBI release. Then it later breaks spectacularly, and when the issue is explained to Iron Bank engineers, they don't understand and simply call it fixed when UBI releases again and glibc is aligned by chance with upstream. 2) They push functional changes to tags. You can pull some <image>:8.4 one day, and the next day the exactly same tag will have different environment variables set, different paths, will have executables removed (notably, the jq image used to include aws-cli for some reason, which it shouldn't have, but once it did, you can't just remove it). Normally, the fix for this is pin to a sha instead of the tag, but Harbor doesn't hold onto the shas once you have republished to the same tag five times, and Iron Bank is continuously rebuilding those tags. We've reported it and Iron Bank technical leadership is at least aware this is a problem, but figuring out how to fix it has never been a priority for them.

nchaillan 5y ago

To be clear, I am not gone nor was I asked to leave. In fact, I was asked to stay. I do have a clearance as well and yes sorry I didn't let us get locked into Pivotal like you did and cost taxpayer millions.

Ziggy_Zaggy 5y ago
Thanks for commenting on the thread. Can you refer us to your opinions on the matters expressed in the article in a public friendly fashion?

nchaillan 5y ago

Of course Andrew did his best to get KR locked in to PCF which is now taking us 2 years to walk away from. Well done friend.

poulsbohemian 5y ago

> Other places, where it matters a lot it's like nothing you've ever seen or will likely see in the commercial sector for decades. Would love to more about this as it sounds like you are implying that their practices and technology are significantly ahead of the commercial sector.

spfzero 5y ago

Thanks for posting this, the lack of a security clearance seems telling. Even contractors often have them. Also, a person in a real position of responsibility would never behave this way, and to be fair to him, they would never be treated this way.

hamburgerwah 5y ago

^^ lol, political hit jobs representing in these comments

hiram112 5y ago

> - Nic never held a clearance, and as a result was never actually involved or aware of most of the programs that he intended to impact I honestly don't know how he'd ever expect to succeed in a job like this without a clearance. Without it, he'd have little practical knowledge of how the actual air gapped networks (SIPR, JWICS, etc.) are designed and implemented. Even if he didn't know the classified technical details, there's a lot of differences between deploying software in these environments vs the wide open public internet. From experience, the type of exploits that would be devastating to a big company just aren't a big risk for most apps on these networks. OTOH, classified apps have all sorts of bureaucratic rules that the unclassified world doesn't have to consider. For example, if this guy is coming from Silicon Valley, he might spend 80% of his time worrying about exploits that cost corporate American billions / year and which are due to unpatched systems which become targets for script-kiddies pounding every IP on the internet again and again, looking to take advantage of dozens of known vulnerabilities. On the classified networks, these types of exploits just aren't as likely. There aren't fleets of bots in China and Russia trying known root passwords on every machine with port 22 open or trying to access open Mongo or Elastic Search systems on whole IP ranges. On the other hand, a private sector SecOps guy isn't even considering that they absolutely cannot store multiple big program's databases in the same data center, staffed by the same contracting company. Because there's a huge risk that some rogue IT contractor with general root access in some small signals intelligence "datacenter" in Hawaii or Germany will end up having access to way too much sensitive data that is shared by multiple intel agencies and military branches, and it'd be disastrous if he decided to start taking raw disks home with him each night a la Snowden. The Air Force and other DOD agencies certainly must ensure that a single contractor doesn't have access to too many data sources. And with everyone moving as fast as they can to AWS, Azure, etc, I'm guessing these type of procedure are far more important to the Air Force than worrying about some development team deploying an app on an older docker image based on some old version of Java or Python with a few known exploits, which could cost a private sector company a lot of pain and money the minute it was deployed on the open internet.

Aeolun 5y ago
I don’t think you want your airgap to be the only thing standing between your IT infra and total disaster. Especially in the military it seems entirely plausible that bad people gain access to your equipment.