5 ms·

Yeah. I aint saying who I am, but I'll anonymously corroborate this. Nic had a lot of good ideas, but he micromanaged the hell out of low-level product decision

by _bnmd 5y ago

Yeah. I aint saying who I am, but I'll anonymously corroborate this. Nic had a lot of good ideas, but he micromanaged the hell out of low-level product decisions, forced us to use specific broken products that made me swear the vendors must have naked photos of him somewhere. He threw new requirements out of left field in the middle of demos that were not real requirements written down anywhere, but for whatever reason, we had to change course in the middle of development and do it anyway. I'm sitting here right now on a Friday afternoon while the Air Force is on a four-day weekend ahead of labor day, trying to deploy to a broken ass application his DevSecOps reference architecture forces everyone to use, but it doesn't work because it uses a checksum algorithm disabled by FIPS-compliance hardening in this environment, which we have absolutely no control over. The biggest impediment to even getting this far was another vendor enterprise service he forced us to use that was broken until July. We were stuck just waiting on a bug fix to be provided. And, of course, we have to use Iron Bank container images for everything, but Iron Bank container images are themselves perpetually broken. They do security hardening, but no functionality testing, and their process of pushing breaking changes to the same tags can break you in production unexpectedly. And you can't pin to the actual sha because Harbor only holds onto five orphaned shas at a time that don't correspond to a tag. He's touting a lot of accomplishments here that are accomplishments because pushing broken functionality and calling it done is a very easy way to say you delivered faster than a normal DoD program that has to actually prove it does what it says it does.

nchaillan 5y ago
This is Nicolas. I'd welcome to get your proposed fixed so we can fix both Iron Bank and the other issues you mentioned here. Do remember that I'm not the one mandating FIPS validation. You know, there is a concept called cybersecurity.
- _bnmd 5y ago
  The issue with FIPS compliance is with Anchore. It's actually fixed in 10.2, but that version is not yet in Iron Bank, so delaying deployment to IL5. The issue with Iron Bank itself is a lot more structural. We report it every time a specific container breaks, and it eventually gets fixed on a report-by-report basis, but the base issue is they're doing two things completely wrong: 1) The process requiring disconnected builds in the container build stage to avoid pulling in dependencies from the Internet, combined with lack of expertise on the part of people assigned to the container hardening teams in terms of how to actually build software, results in them usually copying down the upstream official container, and making a naive copy of the desired executable from that container into a UBI base image. That happens to work for dynamically linked executables when the upstream and UBI have the same glibc version, usually right after a UBI release. Then it later breaks spectacularly, and when the issue is explained to Iron Bank engineers, they don't understand and simply call it fixed when UBI releases again and glibc is aligned by chance with upstream. 2) They push functional changes to tags. You can pull some <image>:8.4 one day, and the next day the exactly same tag will have different environment variables set, different paths, will have executables removed (notably, the jq image used to include aws-cli for some reason, which it shouldn't have, but once it did, you can't just remove it). Normally, the fix for this is pin to a sha instead of the tag, but Harbor doesn't hold onto the shas once you have republished to the same tag five times, and Iron Bank is continuously rebuilding those tags. We've reported it and Iron Bank technical leadership is at least aware this is a problem, but figuring out how to fix it has never been a priority for them.
spfzero 5y ago
It's interesting how often someone high up pushes a vendor solution that won't work, and when you explain that it is not workable, and why, (and not just you but basically everyone knows) you would expect them to say "thanks for discovering this critical flaw, we'll can them immediately and select a better solution" or, "we'll have a couple of architects look into that and report back" at least. But instead, they look at you like you just created an embarrassing problem for them and now you're the enemy. One reason might be that they are receiving some kind of compensation from that vendor. A more charitable explanation would be that they want to cover up a big mistake by loudly affirming that it was not a mistake, and you are in the way of that. Either way, the fact that they don't really want to know whether it was a mistake, and if they know, don't want to fix it, tells you all you need to know about them.
Pokepokalypse 5y ago
>He threw new requirements out of left field in the middle of demos that were not real requirements written down anywhere, but for whatever reason, we had to change course in the middle of development and do it anyway. To me this sounds like the fundamental difference in approach between "agile" and "waterfall" (and underscores his accusation of the DoD using broken "water-agile-fall"; which is more commonly called "water-scrum-fall", but I digress). You can't use an agile process for a project whose contract demands requirements (and all that implies) up front. It would require a complete revamp of the entire DoD contracting process. Which may or may not be something that can or should be done, but with the understanding that it IS the way it is, for a very good reason: to prevent fraud. Which was a painful learning experience going back to the very beginning of the Federal practice of providing national defense. The current system is NOT perfect, but you don't throw the whole system away in pursuit of "cost savings" or you'll attract every grifter and con artist out there into this business looking to scam every penny they can from taxpayers. >He's touting a lot of accomplishments here that are accomplishments because pushing broken functionality and calling it done is a very easy way to say you delivered faster than a normal DoD program that has to actually prove it does what it says it does. 100%. But dang, wouldn't it be nice if we really COULD do all this? And given my limited exposure to P1, and the traditional delivery systems, it seems to me there's a very worthwhile middle-ground to pursue. It didn't seem he was interested in pursuing an incremental approach. It seemed like very much an all-or-nothing mission to him.
- mpyne 5y ago
  > It would require a complete revamp of the entire DoD contracting process. Which may or may not be something that can or should be done, but with the understanding that it IS the way it is, for a very good reason: to prevent fraud. Which was a painful learning experience going back to the very beginning of the Federal practice of providing national defense. The current system is NOT perfect, but you don't throw the whole system away in pursuit of "cost savings" or you'll attract every grifter and con artist out there into this business looking to scam every penny they can from taxpayers. The thing is, at least for software dev, DoD projects don't know their software requirements any more than private sector ones do. Plus, the current acquisition system is by no means more fair or less riddled with fraud. The WWII model of much easier acquisition regulations run by bureaus or offices with deep technical expertise and independent auditing did quite well. Just because we intend to reduce fraud with a given set of policies doesn't mean we should assume that the policy actually works, or that the other problems such policies introduce are to be ignored as a result.
mpyne 5y ago
> He's touting a lot of accomplishments here that are accomplishments because pushing broken functionality and calling it done is a very easy way to say you delivered faster than a normal DoD program that has to actually prove it does what it says it does. Is your experience that normal DoD software programs have to actually prove they do what they say they do? Because that has not been my experience at all in the Navy. We work on something for years until political and leadership support for working without concrete progress can finally no longer be maintained, throw whatever we have at that point at the users, and then force people to use it because "we've already invested $XXM into it and worked on it for years and all the rest of our plans assume that this system is there."
- _bnmd 5y ago
  My experience is admittedly entirely from the IC side before I came to Platform One last year. And yes, when it came to ground processing and dissemination of geointelligence products, we absolutely did prove we were doing what we were supposed to be doing, to an extremely exacting standard. I'm talking exact specs on false positive rates of cloud detections. Written explanations of pixel by pixel differences from the last release's test keys traced to an RFC. Verification that something worked equally well over cities, jungles, deserts, ocean, mountains. Sometimes getting dinged for ridiculous attention to detail stuff like accurate multi-tile pixel registration of a mosaic collection over Mount Everest, just to prove it still worked in the worst possible spot on the planet for our topographic earth models. When you see people above asking "what are the examples where we're a decade or more ahead of the commercial world," satellite imagery is one of the places where we're a decade or more ahead of the commercial world. I don't know if you recall Trump accidentally declassifying a collection on Twitter a few years back and imagery experts skeptical that it actually could have been taken from space given the resolution. That is not even remotely the tip of the iceberg of what we can do. I would not have even believed what we can do until I saw it. So I actually am used to being in an environment run by technical experts in physics and image science who know exactly what they want, write down clear, specific, and exacting requirements, and actually meeting those requirements. But you know what? We had the infrastructure to do it. A complete clone of the operational system with a cloned production data flow, putting the next release through exactly the same load, so we could detect the differences and any bugs immediately. Actual continuous integration because there was somewhere we could integrate to. Then I come to Platform One and it's here's an AWS account and some buzzword tech products that say they enable GitOps and CI. Figure out how to use them and stand up your own servers from scratch. And you know what? I'm a confident person. I perfectly believe I can do that. But not very quickly. And if my five person team is supposed to do design, development, test, operations, and maintenance all by ourselves, it is never going to happen quickly. Process can't save you from underprovisioning of resources. But apparently the government is just not willing to spend money any more. It's hard to see how. It's not like the federal budget is getting any smaller. But I have no idea where that money is going. At least some of that used to be going toward extremely superior radical technology that the public is skeptical of because it's all classified and they don't hear about it, but trust me, it's there.

by _bnmd 5y ago

nchaillan 5y ago

This is Nicolas. I'd welcome to get your proposed fixed so we can fix both Iron Bank and the other issues you mentioned here. Do remember that I'm not the one mandating FIPS validation. You know, there is a concept called cybersecurity.

_bnmd 5y ago
The issue with FIPS compliance is with Anchore. It's actually fixed in 10.2, but that version is not yet in Iron Bank, so delaying deployment to IL5. The issue with Iron Bank itself is a lot more structural. We report it every time a specific container breaks, and it eventually gets fixed on a report-by-report basis, but the base issue is they're doing two things completely wrong: 1) The process requiring disconnected builds in the container build stage to avoid pulling in dependencies from the Internet, combined with lack of expertise on the part of people assigned to the container hardening teams in terms of how to actually build software, results in them usually copying down the upstream official container, and making a naive copy of the desired executable from that container into a UBI base image. That happens to work for dynamically linked executables when the upstream and UBI have the same glibc version, usually right after a UBI release. Then it later breaks spectacularly, and when the issue is explained to Iron Bank engineers, they don't understand and simply call it fixed when UBI releases again and glibc is aligned by chance with upstream. 2) They push functional changes to tags. You can pull some <image>:8.4 one day, and the next day the exactly same tag will have different environment variables set, different paths, will have executables removed (notably, the jq image used to include aws-cli for some reason, which it shouldn't have, but once it did, you can't just remove it). Normally, the fix for this is pin to a sha instead of the tag, but Harbor doesn't hold onto the shas once you have republished to the same tag five times, and Iron Bank is continuously rebuilding those tags. We've reported it and Iron Bank technical leadership is at least aware this is a problem, but figuring out how to fix it has never been a priority for them.

spfzero 5y ago

It's interesting how often someone high up pushes a vendor solution that won't work, and when you explain that it is not workable, and why, (and not just you but basically everyone knows) you would expect them to say "thanks for discovering this critical flaw, we'll can them immediately and select a better solution" or, "we'll have a couple of architects look into that and report back" at least. But instead, they look at you like you just created an embarrassing problem for them and now you're the enemy. One reason might be that they are receiving some kind of compensation from that vendor. A more charitable explanation would be that they want to cover up a big mistake by loudly affirming that it was not a mistake, and you are in the way of that. Either way, the fact that they don't really want to know whether it was a mistake, and if they know, don't want to fix it, tells you all you need to know about them.

Pokepokalypse 5y ago

>He threw new requirements out of left field in the middle of demos that were not real requirements written down anywhere, but for whatever reason, we had to change course in the middle of development and do it anyway. To me this sounds like the fundamental difference in approach between "agile" and "waterfall" (and underscores his accusation of the DoD using broken "water-agile-fall"; which is more commonly called "water-scrum-fall", but I digress). You can't use an agile process for a project whose contract demands requirements (and all that implies) up front. It would require a complete revamp of the entire DoD contracting process. Which may or may not be something that can or should be done, but with the understanding that it IS the way it is, for a very good reason: to prevent fraud. Which was a painful learning experience going back to the very beginning of the Federal practice of providing national defense. The current system is NOT perfect, but you don't throw the whole system away in pursuit of "cost savings" or you'll attract every grifter and con artist out there into this business looking to scam every penny they can from taxpayers. >He's touting a lot of accomplishments here that are accomplishments because pushing broken functionality and calling it done is a very easy way to say you delivered faster than a normal DoD program that has to actually prove it does what it says it does. 100%. But dang, wouldn't it be nice if we really COULD do all this? And given my limited exposure to P1, and the traditional delivery systems, it seems to me there's a very worthwhile middle-ground to pursue. It didn't seem he was interested in pursuing an incremental approach. It seemed like very much an all-or-nothing mission to him.

mpyne 5y ago
> It would require a complete revamp of the entire DoD contracting process. Which may or may not be something that can or should be done, but with the understanding that it IS the way it is, for a very good reason: to prevent fraud. Which was a painful learning experience going back to the very beginning of the Federal practice of providing national defense. The current system is NOT perfect, but you don't throw the whole system away in pursuit of "cost savings" or you'll attract every grifter and con artist out there into this business looking to scam every penny they can from taxpayers. The thing is, at least for software dev, DoD projects don't know their software requirements any more than private sector ones do. Plus, the current acquisition system is by no means more fair or less riddled with fraud. The WWII model of much easier acquisition regulations run by bureaus or offices with deep technical expertise and independent auditing did quite well. Just because we intend to reduce fraud with a given set of policies doesn't mean we should assume that the policy actually works, or that the other problems such policies introduce are to be ignored as a result.

mpyne 5y ago

> He's touting a lot of accomplishments here that are accomplishments because pushing broken functionality and calling it done is a very easy way to say you delivered faster than a normal DoD program that has to actually prove it does what it says it does. Is your experience that normal DoD software programs have to actually prove they do what they say they do? Because that has not been my experience at all in the Navy. We work on something for years until political and leadership support for working without concrete progress can finally no longer be maintained, throw whatever we have at that point at the users, and then force people to use it because "we've already invested $XXM into it and worked on it for years and all the rest of our plans assume that this system is there."

_bnmd 5y ago
My experience is admittedly entirely from the IC side before I came to Platform One last year. And yes, when it came to ground processing and dissemination of geointelligence products, we absolutely did prove we were doing what we were supposed to be doing, to an extremely exacting standard. I'm talking exact specs on false positive rates of cloud detections. Written explanations of pixel by pixel differences from the last release's test keys traced to an RFC. Verification that something worked equally well over cities, jungles, deserts, ocean, mountains. Sometimes getting dinged for ridiculous attention to detail stuff like accurate multi-tile pixel registration of a mosaic collection over Mount Everest, just to prove it still worked in the worst possible spot on the planet for our topographic earth models. When you see people above asking "what are the examples where we're a decade or more ahead of the commercial world," satellite imagery is one of the places where we're a decade or more ahead of the commercial world. I don't know if you recall Trump accidentally declassifying a collection on Twitter a few years back and imagery experts skeptical that it actually could have been taken from space given the resolution. That is not even remotely the tip of the iceberg of what we can do. I would not have even believed what we can do until I saw it. So I actually am used to being in an environment run by technical experts in physics and image science who know exactly what they want, write down clear, specific, and exacting requirements, and actually meeting those requirements. But you know what? We had the infrastructure to do it. A complete clone of the operational system with a cloned production data flow, putting the next release through exactly the same load, so we could detect the differences and any bugs immediately. Actual continuous integration because there was somewhere we could integrate to. Then I come to Platform One and it's here's an AWS account and some buzzword tech products that say they enable GitOps and CI. Figure out how to use them and stand up your own servers from scratch. And you know what? I'm a confident person. I perfectly believe I can do that. But not very quickly. And if my five person team is supposed to do design, development, test, operations, and maintenance all by ourselves, it is never going to happen quickly. Process can't save you from underprovisioning of resources. But apparently the government is just not willing to spend money any more. It's hard to see how. It's not like the federal budget is getting any smaller. But I have no idea where that money is going. At least some of that used to be going toward extremely superior radical technology that the public is skeptical of because it's all classified and they don't hear about it, but trust me, it's there.