8 ms·

After 15 years of being a DoD contractor, it is frustrating to see yet another sole source entity getting the contract. Prices will inflate and there is no com

After 15 years of being a DoD contractor, it is frustrating to see yet another sole source entity getting the contract. Prices will inflate and there is no competition. HP and Oracle have been reaping the benefits of this for at least two decades now. We put databases on Oracle that should be handled by Postgres or MariaDB since DoD prefers Oracle. We would buy useless HP software because we were a HP shop. I fought to get a non HP solid state array for our data, it was an epic battle (in the end I won, on the extreme we had 6 to 7 hour processes cut down to under an hour, the HP equivalent could not replicate that at the time). So i can see DoD moving to Azure and then get the vendor lock in and in 10 years if they want to move the cost will be so extreme it will either cause taxpayers a ton of money or not be realistic. As impossible as it sounds, and somewhat impractical, i would rather see a vendor agnostic approach and DoD spread across multiple gov clouds. i guess years has gotten me jaded with government spending (wait, what, how did we buy 2 extra $50k Cisco chassis and then keep them in storage for 3 years....).

jszymborski 7y ago
I understand that the existing approach results in a worse and more expensive product, but doesn't this approach also allow agencies to focus their efforts on assuring the vendors of their critical infrastructure aren't comprimised by bad actors etc...? (This isn't my area expertise, so I'm open to the idea of being super wrong)
- siffland 7y ago
  You can lock down YOUR infrastructure, but then are 100% dependent on the cloud environment to maintain and patch theirs. Since you have no insight to what the underlying infrastructure consists of you really have no way of knowing if they are secure. Do their storage arrays have open CVE's? are they employing people who are mentally sane? You just need to trust them. So in the cloud just migrates the a lot of the security to another team. I do not know for a fact, but I am pretty sure the DoD cannot just show up at the AWS or Azure facilities and start auditing them (maybe they can and it is in a contract, someone else might know).
  - mrep 7y ago
    Considering AWS already has 2 entire airgapped dataceters for the US government, I'm pretty sure this contract will entail microsoft building entirely separate airgapped datacenters for the DOD and thus they probably will be able to just show up to their datacenter because nobody else is going to be running anything in those datacenters.
    - topkai22 7y ago
      Correct, but those air gapped data centers already exist- there are separate Azure regions for certain restricted civilian government workloads, DoD unclassified work, and cleared work.
  - thefreeman 7y ago
    They are almost certainly getting their own privately built cloud which they can 100% show up and audit whenever they please.
akiselev 7y ago
Is it really unusual or inefficient for the Department of Defense logistics chain to have lots of spare capacity all over the place w.r.t the Cisco chassis? Especially with global force projection, I'd expect there would be massive, unavoidable waste just to maintain operational capabilities, stretching into every corner of the industry.
Iwillgetby 7y ago
My cubicle at my last job was surrounded by a mountain of expensive printers because they would loose some funding if they didn't spend all of their budget each year. If we ordered a server it would show up in 6 to 12 months and sometimes would be sent to the wrong place. The people in charge of ordering would replace SSDs with spinning disk and we wouldn't know about the change until the wrong parts arrive months later. Fun times.
sysadmindotfail 7y ago
As a former longtime contractor I totally agree. One thing often overlooked is this 'Silver Tsunami' that will wreak havoc on the old guard (existing giant contracting companies). The existing workforce is aging, the new blood is uninterested in the old ways (cruft, old tech) and frankly there's tremendous opportunity for a 'Space-X' style small contractor to get a serious foothold. Some of these companies need to replace 40% of their workforce in the next decade. Who would a new grad choose?
- mtberatwork 7y ago
  > there's tremendous opportunity for a 'Space-X' style small contractor to get a serious foothold. Sure, but the procurement process is incredibly difficult to crack into for small companies. There is tremendous amounts of red tape designed to keep other players out. > Some of these companies need to replace 40% of their workforce in the next decade. Who would a new grad choose? Most will go to where the money is...unless you go FAANG, it's harder to find the kind of high salaries that defense contractors can throw around in the private sector.
  - blaser-waffle 7y ago
    > Most will go to where the money is...unless you go FAANG, it's harder to find the kind of high salaries that defense contractors can throw around in the private sector. There is a reason the DC suburbs of Northern VA and Southern MD have some of the richest counties in the US...
  - ta999999171 7y ago
    > There is tremendous amounts of red tape designed to keep other players out. Call it what it is - bureaucracy, and corruption.
derefr 7y ago
> We put databases on Oracle that should be handled by Postgres or MariaDB since DoD prefers Oracle. I mean, if you’ve already budgeted the CapEx for some additional Oracle licenses, the OpEx efficiencies of having unified tooling and a unified ops doctrine are no joke. I haven’t worked with an Oracle DBMS, but I think this is analogous: I’d sure hate to have to manage a cloud infrastructure where parts were on AWS, parts on GCP, and parts on Azure. Sure, there are generic tools that treat all three the same, or over-layers like K8s that don’t care about substrate—but what if the projects on each platform were taking advantage of that platform’s specialties? What if I was using SNS on AWS, or BigQuery on GCP? To bring that back through the analogy, what if our Oracle projects were tuned using Oracle-specific query-planner hints, while our Postgres projects did their ETL using PG-specific Foreign Data Wrapper connectors? In both cases, the only real solution is hiring and retaining O(N) specialized ops headcount, one team for each stack. And that cost gets a lot higher than just paying for another darn Oracle license.
- awinder 7y ago
  I'm sure that these deals are complicated at the size they're going at and maybe laymens pricing models just get tossed aside, but one of the biggest things I spend time on in cloud architecting is all around data ingress/egress in order to control costs. I simply don't understand how it's possible to go multi-cloud and control those costs, I feel like you'd either blow costs out not caring, or blow costs out throwing engineers at very complex solutions.
- vasco 7y ago
  Even with an abstraction layer like kubernetes you still have a lot of duplicate work if you're multi-cloud. Services need to be exposed with load balancers, and those will have different configurations to be setup. Same with any Volumes. And then you have platform updates on both sides, bugs, quirks. plus the maintenance and upkeep of two clouds - two bills to inspect, two account managers to deal with, two sets of permissions and overall account configuration to setup, maintain and audit. Multi-cloud is really a set of requirements that changes the whole game in terms of operational overhead. And we limited our example to kubernetes. I imagine any non-fictional-for-the-sake-of-example company would want to make use of other platform specific tools as you mention.
awinder 7y ago
What's hysterical to me is that Microsoft was actually making this point in their proposal because they very much expected for AWS to solely receive go-ahead with this contract.
arwineap 7y ago
> how did we buy 2 extra $50k Cisco chassis and then keep them in storage for 3 years Military budgets... Spend it or lose it. It's better you find spare cisco chassis, than hundreds of zip disks or obsolete laptops >.>
- siffland 7y ago
  This was an accident, they meant to buy 2 blades for more SAN ports and they ordered the wrong parts. They did this with a few database servers as well, they were supposed to order 8 core processors, but they ordered 10 core instead (it was a typo), they had 4 processors per machine it was a $40k extra mess not including the extra Oracle licenses needed to buy since Oracle licenses by cores (8 extra cores per machine and 3 machines). But i do understand the spend it or lose it, which is another reason no one tries to be efficient.
BareNakedCoder 7y ago
Not just governments. I worked at a F100 company that had a special type of internal funding called SQP for spending on (mostly) IT people-hours on projects. Year after year, the biz would penny-pinch on their SQP to the point that we struggled to keep core contractors on staff. Then around Oct/Nov, biz would come to us begging to spend their SQP on anything (else they could lose it next year) as long as that project stop billing at fiscal year end (not a day afterwards; as that would be charged to next year's SQP). That meant we now struggled because we didn't have enough people on staff to burn up all the SQP. Famine then feast! I would suggest (half jokingly) that we should head over to the nearest retirement home and get a bus load of old folks to join us at $150/hr for two months to just sit there doing nothing but burn SQP.
- Answerawake 7y ago
  We are always seeing threads with junior people/new grads talking about how they are having so much trouble breaking into the industry. Just hire them and give them interesting tiny projects. Would be super helpful to so many people to get going and still allow you to spend all your money.
  - joejerryronnie 7y ago
    You can't ramp up hiring that quickly. By the time you've onboarded several new devs, the year (and budget period) will be over.

jszymborski 7y ago

I understand that the existing approach results in a worse and more expensive product, but doesn't this approach also allow agencies to focus their efforts on assuring the vendors of their critical infrastructure aren't comprimised by bad actors etc...? (This isn't my area expertise, so I'm open to the idea of being super wrong)

siffland 7y ago
You can lock down YOUR infrastructure, but then are 100% dependent on the cloud environment to maintain and patch theirs. Since you have no insight to what the underlying infrastructure consists of you really have no way of knowing if they are secure. Do their storage arrays have open CVE's? are they employing people who are mentally sane? You just need to trust them. So in the cloud just migrates the a lot of the security to another team. I do not know for a fact, but I am pretty sure the DoD cannot just show up at the AWS or Azure facilities and start auditing them (maybe they can and it is in a contract, someone else might know).
- mrep 7y ago
  Considering AWS already has 2 entire airgapped dataceters for the US government, I'm pretty sure this contract will entail microsoft building entirely separate airgapped datacenters for the DOD and thus they probably will be able to just show up to their datacenter because nobody else is going to be running anything in those datacenters.
  - topkai22 7y ago
    Correct, but those air gapped data centers already exist- there are separate Azure regions for certain restricted civilian government workloads, DoD unclassified work, and cleared work.
- thefreeman 7y ago
  They are almost certainly getting their own privately built cloud which they can 100% show up and audit whenever they please.

akiselev 7y ago

Is it really unusual or inefficient for the Department of Defense logistics chain to have lots of spare capacity all over the place w.r.t the Cisco chassis? Especially with global force projection, I'd expect there would be massive, unavoidable waste just to maintain operational capabilities, stretching into every corner of the industry.

Iwillgetby 7y ago

My cubicle at my last job was surrounded by a mountain of expensive printers because they would loose some funding if they didn't spend all of their budget each year. If we ordered a server it would show up in 6 to 12 months and sometimes would be sent to the wrong place. The people in charge of ordering would replace SSDs with spinning disk and we wouldn't know about the change until the wrong parts arrive months later. Fun times.

sysadmindotfail 7y ago

As a former longtime contractor I totally agree. One thing often overlooked is this 'Silver Tsunami' that will wreak havoc on the old guard (existing giant contracting companies). The existing workforce is aging, the new blood is uninterested in the old ways (cruft, old tech) and frankly there's tremendous opportunity for a 'Space-X' style small contractor to get a serious foothold. Some of these companies need to replace 40% of their workforce in the next decade. Who would a new grad choose?

mtberatwork 7y ago
> there's tremendous opportunity for a 'Space-X' style small contractor to get a serious foothold. Sure, but the procurement process is incredibly difficult to crack into for small companies. There is tremendous amounts of red tape designed to keep other players out. > Some of these companies need to replace 40% of their workforce in the next decade. Who would a new grad choose? Most will go to where the money is...unless you go FAANG, it's harder to find the kind of high salaries that defense contractors can throw around in the private sector.
- blaser-waffle 7y ago
  > Most will go to where the money is...unless you go FAANG, it's harder to find the kind of high salaries that defense contractors can throw around in the private sector. There is a reason the DC suburbs of Northern VA and Southern MD have some of the richest counties in the US...
- ta999999171 7y ago
  > There is tremendous amounts of red tape designed to keep other players out. Call it what it is - bureaucracy, and corruption.

derefr 7y ago

> We put databases on Oracle that should be handled by Postgres or MariaDB since DoD prefers Oracle. I mean, if you’ve already budgeted the CapEx for some additional Oracle licenses, the OpEx efficiencies of having unified tooling and a unified ops doctrine are no joke. I haven’t worked with an Oracle DBMS, but I think this is analogous: I’d sure hate to have to manage a cloud infrastructure where parts were on AWS, parts on GCP, and parts on Azure. Sure, there are generic tools that treat all three the same, or over-layers like K8s that don’t care about substrate—but what if the projects on each platform were taking advantage of that platform’s specialties? What if I was using SNS on AWS, or BigQuery on GCP? To bring that back through the analogy, what if our Oracle projects were tuned using Oracle-specific query-planner hints, while our Postgres projects did their ETL using PG-specific Foreign Data Wrapper connectors? In both cases, the only real solution is hiring and retaining O(N) specialized ops headcount, one team for each stack. And that cost gets a lot higher than just paying for another darn Oracle license.

awinder 7y ago
I'm sure that these deals are complicated at the size they're going at and maybe laymens pricing models just get tossed aside, but one of the biggest things I spend time on in cloud architecting is all around data ingress/egress in order to control costs. I simply don't understand how it's possible to go multi-cloud and control those costs, I feel like you'd either blow costs out not caring, or blow costs out throwing engineers at very complex solutions.
vasco 7y ago
Even with an abstraction layer like kubernetes you still have a lot of duplicate work if you're multi-cloud. Services need to be exposed with load balancers, and those will have different configurations to be setup. Same with any Volumes. And then you have platform updates on both sides, bugs, quirks. plus the maintenance and upkeep of two clouds - two bills to inspect, two account managers to deal with, two sets of permissions and overall account configuration to setup, maintain and audit. Multi-cloud is really a set of requirements that changes the whole game in terms of operational overhead. And we limited our example to kubernetes. I imagine any non-fictional-for-the-sake-of-example company would want to make use of other platform specific tools as you mention.

awinder 7y ago

What's hysterical to me is that Microsoft was actually making this point in their proposal because they very much expected for AWS to solely receive go-ahead with this contract.

arwineap 7y ago

> how did we buy 2 extra $50k Cisco chassis and then keep them in storage for 3 years Military budgets... Spend it or lose it. It's better you find spare cisco chassis, than hundreds of zip disks or obsolete laptops >.>

siffland 7y ago
This was an accident, they meant to buy 2 blades for more SAN ports and they ordered the wrong parts. They did this with a few database servers as well, they were supposed to order 8 core processors, but they ordered 10 core instead (it was a typo), they had 4 processors per machine it was a $40k extra mess not including the extra Oracle licenses needed to buy since Oracle licenses by cores (8 extra cores per machine and 3 machines). But i do understand the spend it or lose it, which is another reason no one tries to be efficient.

BareNakedCoder 7y ago

Not just governments. I worked at a F100 company that had a special type of internal funding called SQP for spending on (mostly) IT people-hours on projects. Year after year, the biz would penny-pinch on their SQP to the point that we struggled to keep core contractors on staff. Then around Oct/Nov, biz would come to us begging to spend their SQP on anything (else they could lose it next year) as long as that project stop billing at fiscal year end (not a day afterwards; as that would be charged to next year's SQP). That meant we now struggled because we didn't have enough people on staff to burn up all the SQP. Famine then feast! I would suggest (half jokingly) that we should head over to the nearest retirement home and get a bus load of old folks to join us at $150/hr for two months to just sit there doing nothing but burn SQP.

Answerawake 7y ago
We are always seeing threads with junior people/new grads talking about how they are having so much trouble breaking into the industry. Just hire them and give them interesting tiny projects. Would be super helpful to so many people to get going and still allow you to spend all your money.
- joejerryronnie 7y ago
  You can't ramp up hiring that quickly. By the time you've onboarded several new devs, the year (and budget period) will be over.