6 ms·

Federal Gov employee here. Can't speak for a project with this scope, but the procurement middlemen get into everything, far for the worse. Two years ago our

Federal Gov employee here. Can't speak for a project with this scope, but the procurement middlemen get into everything, far for the worse. Two years ago our team wanted to buy a small cluster (~300 cores, ~$50K). We talked directly to two good vendors (good recommendations from university partners) and came up with a fine machine and 2 bids for it. Sent recommendations to procurement. Procurement put it out for bid, and a fly-by-night company undercut the bid by $10K... by noticing that procurement had not specified details of service level (that were in the bids we'd gotten and forwarded). Procurement, once it goes there, is a true black box. No communication, no understanding. Five months later, we were basically delivered 2 pallets of unassembled parts and no instructions. Believe me, we spent 3-4x as much in labor as the $10K savings to get it working, and it's been plagued with issues that would have been under the onsite service warranties for the better companies. The biggest irony is: I firmly believe that procurement acts this way not because the government is fundamentally incompetent, but because the Public, and thus Congress, BELIEVES we are incompetent, so puts so many levels of "check" bureaucracy in place that the people who know what they want can't participate directly in getting it.

joshdick 13y ago
Procurement rules are more to guard against corruption and cronyism than incompetence. Without them, you could reward your friends with fat government contracts, regardless of what's in the public's interest.
- squidfood 13y ago
  Yes. I should have added that I understand the need for basic checks and auditing (and that the public expects us to be uniformly fraudulent as well as incompetent), but that the level of checks and removal from the process of experts who know requirements goes far beyond this need. It's really all about optics rather than reasonable checks - no department wants to be the one that Congress targets, so especially in poisonous political environments the "checks" are significantly more expensive IMO than the actual "waste, fraud, and abuse" they guard against.
- deleted 13y ago
  [deleted]
- phy6 13y ago
  And like most things with government, you end up getting both: the bureaucracy to prevent cronyism as well as the cronyism. Speaking from experience. Sure it might be more obvious if the bureaucracy wasn't in place, but it's still there.
- apendleton 13y ago
  The irony is that they don't really do that. It's common in both procurement and hiring in the federal government to see bizarrely-specific requirements whose fairly transparent intention is to limit the potential bidder or applicant pool to a single company or individual.
  - brianpgordon 13y ago
    * Must have a company name which SHA-1 hashes to e8b06511aa36381fc2306eb6f8181204585c5453. Hey, there's a theoretically infinite number of company names which meet this requirement...
- DannyBee 13y ago
  "Without them, you could reward your friends with fat government contracts, regardless of what's in the public's interest. " Except this happens anyway. Much like patents, people just become better at drafting. What is currently done is not an effective mechanism for stopping cronyism and corruption at all. In fact makes it easier in a lot of cases, because it provides plausible deniability (It's not that we gave it to our friend, it's that you didn't meet the requirements!)
- mcone 13y ago
  > Procurement rules are more to guard against corruption and cronyism than incompetence. Former federal IT contractor here. The procurement rules were designed for that, yes. But in real world scenarios, those rules effectively do exactly the opposite. Since there are so many hoops for potential vendors to jump through, only the most established players get to bid on most contracts. And in my experience corruption and cronyism is still alive and well in federal IT contracting.
marcosdumay 13y ago
Well, federal government worker here too, but not on the US government. Those rules are there because a malicious worker can cause a huge amount of damage. They are a pain (one entity I worked once spent about $15k (in people time) contracting $100 worth of ssl certificates, in a process that took more than a year (so, no certificates for the site during a period), we were forbiden from contracting the service for more than a year... and the contractor was another governmental entity. The rules are maddening, but they are necessary for a democracy. The problem is that governmental IT is out of place. The government will never be competent in contracting software development - the only known tool that works in keeping government contracts honest is auctioning, and agile is simply not compatible with auctioning. The only possible way out is by doing IT in-house.
- narag 13y ago
  An in-house team serving several departments can work well: providing infrastructure, defining communication protocols, quality and security standars, helping subcontracting.
apendleton 13y ago
I just had a conversation with a federal employee last week talking about the issue of procurement offices being a blackbox and procurement officers not understanding the technical specifics of what they're bidding out. His solution was just to become certified as a procurement officer (apparently it's 72 hours of coursework) and deal with the bids himself. He said these kinds of problems have pretty much disappeared for him since then.
- VladRussian2 13y ago
  > His solution was just to become certified as a procurement officer (apparently it's 72 hours of coursework) and deal with the bids himself. He said these kinds of problems have pretty much disappeared for him since then. sounds like a loophole one can procure a truck through :)
- allochthon 13y ago
  According to US federal procurementology, would it be a conflict of interest to be both procurer and recipient of what is purchased?
  - apendleton 13y ago
    He's overseeing the bidding process for contractors for projects that he will supervise; he doesn't personally benefit from the outcome of the selection (i.e., he doesn't own stock in companies that are bidding or anything like that), so I'm not sure how it would be a conflict. So no, I don't think so.
batgaijin 13y ago
maybe watson should handle procurment
emiliobumachar 13y ago
Out of curiosity, why aren't you naming the no-support supplier? Are there any rules forbidding you to do so?
- squidfood 13y ago
  Honestly, tiny company, couple years ago, don't remember now, I wouldn't be surprised if it was a couple dudes who look at Fed announcements and say "hey, I can buy that off the shelf and ship it to the feds for a profit".
nostromo 13y ago
> I firmly believe that procurement acts this way not because the government is fundamentally incompetent, but because the Public, and thus Congress, BELIEVES we are incompetent There's almost 3 million federal workers. Many more if you include people who work on government contracts. The Federal Government is by far the biggest enterprise in the US by both employees and revenue. With such a large organization, there are undoubtedly large swaths of both incompetence and competence. The challenge with any large organization is that the rules are there to reign in the bad people, but are equally enforced on the good. (For example, most people won't abuse their company's T&E policy, but some will, so everyone has to be treated as suspect.) Honestly I'm not sure what a good solution would look like, but I don't think it's as simple as "trust us."
- deleted 13y ago
  [deleted]
- squidfood 13y ago
  > Honestly I'm not sure what a good solution would look like, but I don't think it's as simple as "trust us." Didn't mean to imply that any sufficiently large organization shouldn't have an audit trail and reasonable accountability!
  - mwsherman 13y ago
    I would offer that those two concepts are optimized for ex-post-facto blame. It’s not simply gov’t, as you point out, it’s large organizations. To contrast this with more modern techniques, ‘audit trail’ is simply source control. And most of us don’t go into source control looking for a smoking gun. It’s rarely necessary if a process is agile/iterative. Bugs will be (relatively) small and recent in time. So the notion of going back six months and figuring out ‘what went wrong’ is just not a thing. Wrong happens every day, in small amounts, transparently. Conversely, a bug if truly large and undetected, and explodes a year from its creation, then the whole team is to blame. We’ve all looked at the code hundreds of times in that period.
- dllthomas 13y ago
  'Honestly I'm not sure what a good solution would look like, but I don't think it's as simple as "trust us."' This might be a good starting point: http://www.amazon.com/Liars-Outliers-Enabling-Society-Thrive/dp/1118143302 http://www.amazon.com/Liars-Outliers-Enabling-Society-Thrive...
mbesto 13y ago
Gov't procurement is actually no different from any other bureaucratic procurement organization in Big Co. They are always seen as a blocker (rarely an enabler) and the people who sit in their roles usually hate their responsibilities. (I'm generalizing of course, but you get the point) The consulting company I used to work implemented techniques for procurement orgs based mainly on behaviors.[1] It's the first time I've seen such a thing and believe it is something very necessary to scaling procurement organizations. [1]-http://www.youtube.com/watch?v=7Xgh-A1ZCfg http://www.youtube.com/watch?v=7Xgh-A1ZCfg
bane 13y ago
The federal procurement process is one of those cases where separation of concerns hurts things. I know that things are this way to try and prevent bribery and cronyism etc. But it's clear that the pendulum has swung too far. I've been part of the procurement process a few times from the vendor side and the layers of nested black boxes makes solving procurement issues virtually impossible and once the procurement is made massive overruns are almost inevitable.
- dllthomas 13y ago
  "But it's clear that the pendulum has swung too far." While I think that's likely, I'm a little bit hesitant to claim things are clear when we can't observe the alternative.
allochthon 13y ago
This is distressing.
jasiek 13y ago
Yup. The heavy equipment process create an additional barrier to entry. If you want to bid for a govt contract, you need to hire someone who has done it before, otherwise you'll drown in paperwork. What's really sad about this situation is that it is stable - eg. existing stakeholders (contractors & govt procurement personnel) will work hand in hand to ensure that the process stays in place - the former to make more money and the latter to retain their jobs.

joshdick 13y ago

Procurement rules are more to guard against corruption and cronyism than incompetence. Without them, you could reward your friends with fat government contracts, regardless of what's in the public's interest.

squidfood 13y ago
Yes. I should have added that I understand the need for basic checks and auditing (and that the public expects us to be uniformly fraudulent as well as incompetent), but that the level of checks and removal from the process of experts who know requirements goes far beyond this need. It's really all about optics rather than reasonable checks - no department wants to be the one that Congress targets, so especially in poisonous political environments the "checks" are significantly more expensive IMO than the actual "waste, fraud, and abuse" they guard against.
deleted 13y ago
[deleted]
phy6 13y ago
And like most things with government, you end up getting both: the bureaucracy to prevent cronyism as well as the cronyism. Speaking from experience. Sure it might be more obvious if the bureaucracy wasn't in place, but it's still there.
apendleton 13y ago
The irony is that they don't really do that. It's common in both procurement and hiring in the federal government to see bizarrely-specific requirements whose fairly transparent intention is to limit the potential bidder or applicant pool to a single company or individual.
- brianpgordon 13y ago
  * Must have a company name which SHA-1 hashes to e8b06511aa36381fc2306eb6f8181204585c5453. Hey, there's a theoretically infinite number of company names which meet this requirement...
DannyBee 13y ago
"Without them, you could reward your friends with fat government contracts, regardless of what's in the public's interest. " Except this happens anyway. Much like patents, people just become better at drafting. What is currently done is not an effective mechanism for stopping cronyism and corruption at all. In fact makes it easier in a lot of cases, because it provides plausible deniability (It's not that we gave it to our friend, it's that you didn't meet the requirements!)
mcone 13y ago
> Procurement rules are more to guard against corruption and cronyism than incompetence. Former federal IT contractor here. The procurement rules were designed for that, yes. But in real world scenarios, those rules effectively do exactly the opposite. Since there are so many hoops for potential vendors to jump through, only the most established players get to bid on most contracts. And in my experience corruption and cronyism is still alive and well in federal IT contracting.

marcosdumay 13y ago

Well, federal government worker here too, but not on the US government. Those rules are there because a malicious worker can cause a huge amount of damage. They are a pain (one entity I worked once spent about $15k (in people time) contracting $100 worth of ssl certificates, in a process that took more than a year (so, no certificates for the site during a period), we were forbiden from contracting the service for more than a year... and the contractor was another governmental entity. The rules are maddening, but they are necessary for a democracy. The problem is that governmental IT is out of place. The government will never be competent in contracting software development - the only known tool that works in keeping government contracts honest is auctioning, and agile is simply not compatible with auctioning. The only possible way out is by doing IT in-house.

narag 13y ago
An in-house team serving several departments can work well: providing infrastructure, defining communication protocols, quality and security standars, helping subcontracting.

apendleton 13y ago

I just had a conversation with a federal employee last week talking about the issue of procurement offices being a blackbox and procurement officers not understanding the technical specifics of what they're bidding out. His solution was just to become certified as a procurement officer (apparently it's 72 hours of coursework) and deal with the bids himself. He said these kinds of problems have pretty much disappeared for him since then.

VladRussian2 13y ago
> His solution was just to become certified as a procurement officer (apparently it's 72 hours of coursework) and deal with the bids himself. He said these kinds of problems have pretty much disappeared for him since then. sounds like a loophole one can procure a truck through :)
allochthon 13y ago
According to US federal procurementology, would it be a conflict of interest to be both procurer and recipient of what is purchased?
- apendleton 13y ago
  He's overseeing the bidding process for contractors for projects that he will supervise; he doesn't personally benefit from the outcome of the selection (i.e., he doesn't own stock in companies that are bidding or anything like that), so I'm not sure how it would be a conflict. So no, I don't think so.

batgaijin 13y ago

maybe watson should handle procurment

emiliobumachar 13y ago

Out of curiosity, why aren't you naming the no-support supplier? Are there any rules forbidding you to do so?

squidfood 13y ago
Honestly, tiny company, couple years ago, don't remember now, I wouldn't be surprised if it was a couple dudes who look at Fed announcements and say "hey, I can buy that off the shelf and ship it to the feds for a profit".

nostromo 13y ago

> I firmly believe that procurement acts this way not because the government is fundamentally incompetent, but because the Public, and thus Congress, BELIEVES we are incompetent There's almost 3 million federal workers. Many more if you include people who work on government contracts. The Federal Government is by far the biggest enterprise in the US by both employees and revenue. With such a large organization, there are undoubtedly large swaths of both incompetence and competence. The challenge with any large organization is that the rules are there to reign in the bad people, but are equally enforced on the good. (For example, most people won't abuse their company's T&E policy, but some will, so everyone has to be treated as suspect.) Honestly I'm not sure what a good solution would look like, but I don't think it's as simple as "trust us."

deleted 13y ago
[deleted]
squidfood 13y ago
> Honestly I'm not sure what a good solution would look like, but I don't think it's as simple as "trust us." Didn't mean to imply that any sufficiently large organization shouldn't have an audit trail and reasonable accountability!
- mwsherman 13y ago
  I would offer that those two concepts are optimized for ex-post-facto blame. It’s not simply gov’t, as you point out, it’s large organizations. To contrast this with more modern techniques, ‘audit trail’ is simply source control. And most of us don’t go into source control looking for a smoking gun. It’s rarely necessary if a process is agile/iterative. Bugs will be (relatively) small and recent in time. So the notion of going back six months and figuring out ‘what went wrong’ is just not a thing. Wrong happens every day, in small amounts, transparently. Conversely, a bug if truly large and undetected, and explodes a year from its creation, then the whole team is to blame. We’ve all looked at the code hundreds of times in that period.
dllthomas 13y ago
'Honestly I'm not sure what a good solution would look like, but I don't think it's as simple as "trust us."' This might be a good starting point: http://www.amazon.com/Liars-Outliers-Enabling-Society-Thrive/dp/1118143302 http://www.amazon.com/Liars-Outliers-Enabling-Society-Thrive...

mbesto 13y ago

Gov't procurement is actually no different from any other bureaucratic procurement organization in Big Co. They are always seen as a blocker (rarely an enabler) and the people who sit in their roles usually hate their responsibilities. (I'm generalizing of course, but you get the point) The consulting company I used to work implemented techniques for procurement orgs based mainly on behaviors.[1] It's the first time I've seen such a thing and believe it is something very necessary to scaling procurement organizations. [1]-http://www.youtube.com/watch?v=7Xgh-A1ZCfg http://www.youtube.com/watch?v=7Xgh-A1ZCfg

bane 13y ago

The federal procurement process is one of those cases where separation of concerns hurts things. I know that things are this way to try and prevent bribery and cronyism etc. But it's clear that the pendulum has swung too far. I've been part of the procurement process a few times from the vendor side and the layers of nested black boxes makes solving procurement issues virtually impossible and once the procurement is made massive overruns are almost inevitable.

dllthomas 13y ago
"But it's clear that the pendulum has swung too far." While I think that's likely, I'm a little bit hesitant to claim things are clear when we can't observe the alternative.

allochthon 13y ago

This is distressing.

jasiek 13y ago

Yup. The heavy equipment process create an additional barrier to entry. If you want to bid for a govt contract, you need to hire someone who has done it before, otherwise you'll drown in paperwork. What's really sad about this situation is that it is stable - eg. existing stakeholders (contractors & govt procurement personnel) will work hand in hand to ensure that the process stays in place - the former to make more money and the latter to retain their jobs.