6 ms·

Isn't it interesting that governments can get their act together and come up with big distributed systems crunching tens of Gbps in real time but still continue

Isn't it interesting that governments can get their act together and come up with big distributed systems crunching tens of Gbps in real time but still continue to implode spectacularly when it comes to large-scale webapps (e.g. Obamacare, UK NHS systems, UK taxation systems.)

ShinyCyril 12y ago
I'd say UCAS should be on that list too (always buckled on results day), however I'm not sure that it's government affiliated.
objclxt 12y ago
You'd assume that if a GCHQ/NSA IT project went spectacularly off the rails (and I can virtually guarantee it has at some point or the other: you'd have to assume so given how often it happens with defense projects we know about) we wouldn't hear about it anyway.
secfirstmd 12y ago
Ha, yeh right. They have proven themselves just as bad as mangaging those projects. In fact it is one of the only areas where the UK Parliament Intelligence and Security Committee has really given them a bit of a kicking. "MI5 abandons multi-million pound IT project" http://www.telegraph.co.uk/technology/10053240/MI5-abandons-multi-million-pound-IT-project.html http://www.telegraph.co.uk/technology/10053240/MI5-abandons-...
cryoshon 12y ago
Different command structures could be part of the cause. Obamacare was a non-classified civilian project, under the leadership of civilians. The primary fumble was (due to corruption) using a federal-friendly (corrupt and ineffective but known to be loyal) contractor rather than an actual development firm. NSA etc are military operations with honeycombed classification schemes and cellularized access to information. Even if there were failures and waste, we'd never hear of it. I assume that corruption plays less of a role since the concern is on effectiveness, but I have nothing to base that on.
- lostlogin 12y ago
  The military isn't that well organized. Anyone here can name at least a dozen snafu from the last decade. A purchase, an invasion, a process a project. You name it.
  - rosser 12y ago
    My inner cynic suggests it might actually be more difficult to find examples of non-fuckups.
  - logfromblammo 12y ago
    At the heart of the military organization, there is an essential core of operations that ensures that anything that absolutely, positively must get done can be done. The remainder is simply there to exercise and maintain the massive required capital investment while it is functionally idle. The gigantic failures exist so that when there is an actual need for aerospace engineers to respond to an actual threat, rather than the theoretical projections based on the data from the last war, it won't take 25 years to train them up from scratch. The failures are not quite as horrible when you realize that it could be more expensive in the long run to do nothing at all. But then the questions turn toward the amount of pre-emptable core capabilities we need to maintain. For a long time, the doctrine in the U.S. has been the ability to sustain simultaneous nonnuclear war in two separate theaters plus nuclear second-strike capability sufficient to create a MAD deterrent. That is expensive. And since we haven't actually needed that much war materiel since 1945, that leaves a lot of room for waste. It could be argued that the limited regional wars of choice that the U.S. has pursued since then have simply been capital maintenance, to prevent the military from decaying into expensive uselessness. I think in its deepest backrooms, the generals still gauge their effectiveness by whether they could fight both Russia and China at the same time.
    - avn2109 12y ago
      This is really interesting. Can you suggest more reading on the two-simultaneous-front goals?
Nanzikambe 12y ago
> when it comes to large-scale webapps (e.g. Obamacare, UK NHS systems, UK taxation systems.) There's absolutely no reason to believe the spy tech isn't as poorly implemented and terrible value for money as the examples you mention. Given their secretive nature it's even more likely to be the case I'd suspect.
- drzaiusapelord 12y ago
  This is my thinking as well. A lot of what spy tech is, is the ability to throw a warrant or NS letter into someone's face and force them to open the gate. The mythical uber-competent security services seem to be a pretty questionable premise. The few criminals caught via these tools are almost always the low hanging fruit of unconnected crazies, who seem to be harmless cranks and not real terrorists. Five eyes didn't stop or predict Snowden, Assange/Manning, Boston bombers, the rise of ISIS, Putin's advance on Crimea and Eastern Ukraine, the massive cyberwar Russia and China have declared on us, etc. In fact, all these things were major surprises to these security services, at least from what we can tell. There's a lot of federal tax money spent on what is cargo cult science and cargo cult intelligence. The NSA has the ability to buy a lot of PhD's but they are stuck with the same batch of incompetent and profiteer federal contractors the rest of the government is stuck with.
  - secfirstmd 12y ago
    I think it's also related to policy-makers... These days too many of them are a bit intimidated by the mere presence and aura of people in uniform, intel people, cops etc...Years ago, many of them had served in the military or a had a very good understanding of what they can really do (not what they claim they can do) how they worked/the types of people behind them(JFK, Churchill etc) so were rightly sceptical of their claims. I mean, has anyone ever met a cop/spook/politician who didn't want more power to do X, more money to do Y, more people, more secrecy etc? One of the main problems for intel agencies is that they are merely one conduit to policy-makers, amongst many these days. Ultimately it is those policy-makers who need to (and increasingly are incapable of) using this information for strategic decision-making...Tactically it is easy to say "arrest/shoot that guy," because they get kudos and less blame if some thing goes wrong. Strategically it is harder to say "don't shoot that guy, keep him alive and in 5 years we can negotiate with him." It took the British a long time to learn how to do this in Northern Ireland, and it was only when they did that they managed to start to move towards peace with the IRA. I'm not sure would that be possibly in the bloggy, politically polarised, short-term, tabloid, Sky/Fox News type of world we live in these days. After all, even if they did predict the rise of ISIS, Boston, it is the policy-maker who must decide what action to take. Difficult when: a) There are sooo many warnings coming across their desk b) Limited resources C) Not many real policy instruments available. For example, the US public would not have been up for attacking for acting on intel available about ISIS 18 months ago, thus that policy option was thus not available.
rayiner 12y ago
It's much easier to write software that does something, than it is to write software that does the right thing every time. If an NSA system fails to see 10% of traffic, that's acceptable. If an NHS system incorrectly rejects 10% of claims, that's a disaster.
- godgod 12y ago
  > "If an NSA system fails to see 10% of traffic, that's acceptable." If you believe the NSA lies and assume their sole purpose for collecting global communications is to stop terrorism...missing 10%...that too is a disaster. If we missed 10% of the terrorist plots, that'd be a complete failure. On the other hand we know the NSA doesn't just collect this data to fight terrorism. The data is collected for economic advantage as much if not more than any other reason. Terrorism to the NSA data collection is just a red herring.
EthanHeilman 12y ago
Actually the TRAILBLAZER project was the NSAs attempt and it was a total failure, costing around 3 billion dollars. They abandoned it in 2006. SAIC the contractor that "built" it for the NSA got pretty rich though. "Trailblazer was a United States National Security Agency (NSA) program intended to develop a capability to analyze data carried on communications networks like the Internet. In 2005, NSA director Michael Hayden told a Senate hearing that the Trailblazer program was several hundred million dollars over budget and years behind schedule. In 2006 the program was shut down, after having cost billions of US Dollars. Several anonymous NSA sources told Hosenball of Newsweek later on that the project was a "wasteful failure". The new project replacing Trailblazer is called Turbulence." - http://en.wikipedia.org/wiki/Trailblazer_Project http://en.wikipedia.org/wiki/Trailblazer_Project
xorcist 12y ago
That's a very interesting question. My pet theory is that it has to do with outsourcing and the widespread belief in the private market that software development is a product just like any other that you buy on the market. When your average intelligence agency has a million dollars to spend on a project, it will keep a house full of smart people and tell to forth and create, paying their bills and keeping their backs. We might call this the academic model, if it wasn't so far from how academia works today. When a civilian government agency (or any large private entity for that matter as they are completely similar in this regard) have a million dollars to spend on a project they immediately start phoning the big boys: We've got a million to spend on this project, can do do it? It is all a bit exaggerated of course, but in this made up example one will get a working, if perhaps shoddy product, and one will get a server filled with very important powerpoints while the product itself tends to be late and slow. Or to put things in a simpler way and less of a rant: I suspect it is a lot easier to spend a million dollars on a highly specialized system, than to do the same on a very generic system anyone could build. The corollary is that the same intelligence agency would fail in the same way a private or civilian organization would when implementing a new payroll system.
- secfirstmd 12y ago
  Let me help ya there "No one ever got fired for hiring IBM" :)
runjake 12y ago
It's not that the Intelligence Community has their act together any better, it's that they have more money to throw at the problems. Less regulation and oversight probably "help" things, too.
Istof 12y ago
They fixed Obamacare's website and they probably have even more incentives to fix the spying applications (if they are broken)... catching "terrorists" might be a low priority goal on their list which would explain why the reporters say that the NSA surveillance played little role in foiling terror plots.
chicofarm 12y ago
Government department may be incompetent, private defense contractors are some of most capable organizations in the world. Different US government operations are being increasingly privatized. One difference between private contractors and government organizations, is contractors are not held accountable to oversight (to congress for example) as the government entities are.
pdkl95 12y ago
While the other replies here have some very good ideas, there might be another reason for this (perceived? real?) difference that is far simpler: running a webapp on the scale of Obamacare/NHS is probably a much harder job than tapping the internet backbone. For the NSA/GCHQ/etc, the actual engineering task of setting up some DPI boxes, beam-splitters, and other wiretap equipment is not particularly complicated on the engineering side[1]. The difficulty with that kind of project is access and legality. Even their massive DB in Utah, while (very) large, is probably a fairly simple schema compared to a many business DBs. Compare that to a healthcare website, that has to implement many random features mandated by law in a way that is compatible with a giant list of "standards", medical jargons, insurance requirements of all sorts, financial regulations, to name just a few of the details. And it has to work, because lives are going to rely on it. While this kind of website might be simple in concept, the random, contradictory, poorly-specified, "design by committee" project requirements are going to be nasty. [1] crypto may be different, but very little of what the NSA is currently doing has anything to do with crypto. They don't even need side-channel attacks when it's all in plaintext... sigh

ShinyCyril 12y ago

I'd say UCAS should be on that list too (always buckled on results day), however I'm not sure that it's government affiliated.

objclxt 12y ago

You'd assume that if a GCHQ/NSA IT project went spectacularly off the rails (and I can virtually guarantee it has at some point or the other: you'd have to assume so given how often it happens with defense projects we know about) we wouldn't hear about it anyway.

secfirstmd 12y ago

Ha, yeh right. They have proven themselves just as bad as mangaging those projects. In fact it is one of the only areas where the UK Parliament Intelligence and Security Committee has really given them a bit of a kicking. "MI5 abandons multi-million pound IT project" http://www.telegraph.co.uk/technology/10053240/MI5-abandons-multi-million-pound-IT-project.html http://www.telegraph.co.uk/technology/10053240/MI5-abandons-...

cryoshon 12y ago

Different command structures could be part of the cause. Obamacare was a non-classified civilian project, under the leadership of civilians. The primary fumble was (due to corruption) using a federal-friendly (corrupt and ineffective but known to be loyal) contractor rather than an actual development firm. NSA etc are military operations with honeycombed classification schemes and cellularized access to information. Even if there were failures and waste, we'd never hear of it. I assume that corruption plays less of a role since the concern is on effectiveness, but I have nothing to base that on.

lostlogin 12y ago
The military isn't that well organized. Anyone here can name at least a dozen snafu from the last decade. A purchase, an invasion, a process a project. You name it.
- rosser 12y ago
  My inner cynic suggests it might actually be more difficult to find examples of non-fuckups.
- logfromblammo 12y ago
  At the heart of the military organization, there is an essential core of operations that ensures that anything that absolutely, positively must get done can be done. The remainder is simply there to exercise and maintain the massive required capital investment while it is functionally idle. The gigantic failures exist so that when there is an actual need for aerospace engineers to respond to an actual threat, rather than the theoretical projections based on the data from the last war, it won't take 25 years to train them up from scratch. The failures are not quite as horrible when you realize that it could be more expensive in the long run to do nothing at all. But then the questions turn toward the amount of pre-emptable core capabilities we need to maintain. For a long time, the doctrine in the U.S. has been the ability to sustain simultaneous nonnuclear war in two separate theaters plus nuclear second-strike capability sufficient to create a MAD deterrent. That is expensive. And since we haven't actually needed that much war materiel since 1945, that leaves a lot of room for waste. It could be argued that the limited regional wars of choice that the U.S. has pursued since then have simply been capital maintenance, to prevent the military from decaying into expensive uselessness. I think in its deepest backrooms, the generals still gauge their effectiveness by whether they could fight both Russia and China at the same time.
  - avn2109 12y ago
    This is really interesting. Can you suggest more reading on the two-simultaneous-front goals?

Nanzikambe 12y ago

> when it comes to large-scale webapps (e.g. Obamacare, UK NHS systems, UK taxation systems.) There's absolutely no reason to believe the spy tech isn't as poorly implemented and terrible value for money as the examples you mention. Given their secretive nature it's even more likely to be the case I'd suspect.

drzaiusapelord 12y ago
This is my thinking as well. A lot of what spy tech is, is the ability to throw a warrant or NS letter into someone's face and force them to open the gate. The mythical uber-competent security services seem to be a pretty questionable premise. The few criminals caught via these tools are almost always the low hanging fruit of unconnected crazies, who seem to be harmless cranks and not real terrorists. Five eyes didn't stop or predict Snowden, Assange/Manning, Boston bombers, the rise of ISIS, Putin's advance on Crimea and Eastern Ukraine, the massive cyberwar Russia and China have declared on us, etc. In fact, all these things were major surprises to these security services, at least from what we can tell. There's a lot of federal tax money spent on what is cargo cult science and cargo cult intelligence. The NSA has the ability to buy a lot of PhD's but they are stuck with the same batch of incompetent and profiteer federal contractors the rest of the government is stuck with.
- secfirstmd 12y ago
  I think it's also related to policy-makers... These days too many of them are a bit intimidated by the mere presence and aura of people in uniform, intel people, cops etc...Years ago, many of them had served in the military or a had a very good understanding of what they can really do (not what they claim they can do) how they worked/the types of people behind them(JFK, Churchill etc) so were rightly sceptical of their claims. I mean, has anyone ever met a cop/spook/politician who didn't want more power to do X, more money to do Y, more people, more secrecy etc? One of the main problems for intel agencies is that they are merely one conduit to policy-makers, amongst many these days. Ultimately it is those policy-makers who need to (and increasingly are incapable of) using this information for strategic decision-making...Tactically it is easy to say "arrest/shoot that guy," because they get kudos and less blame if some thing goes wrong. Strategically it is harder to say "don't shoot that guy, keep him alive and in 5 years we can negotiate with him." It took the British a long time to learn how to do this in Northern Ireland, and it was only when they did that they managed to start to move towards peace with the IRA. I'm not sure would that be possibly in the bloggy, politically polarised, short-term, tabloid, Sky/Fox News type of world we live in these days. After all, even if they did predict the rise of ISIS, Boston, it is the policy-maker who must decide what action to take. Difficult when: a) There are sooo many warnings coming across their desk b) Limited resources C) Not many real policy instruments available. For example, the US public would not have been up for attacking for acting on intel available about ISIS 18 months ago, thus that policy option was thus not available.

rayiner 12y ago

It's much easier to write software that does something, than it is to write software that does the right thing every time. If an NSA system fails to see 10% of traffic, that's acceptable. If an NHS system incorrectly rejects 10% of claims, that's a disaster.

godgod 12y ago
> "If an NSA system fails to see 10% of traffic, that's acceptable." If you believe the NSA lies and assume their sole purpose for collecting global communications is to stop terrorism...missing 10%...that too is a disaster. If we missed 10% of the terrorist plots, that'd be a complete failure. On the other hand we know the NSA doesn't just collect this data to fight terrorism. The data is collected for economic advantage as much if not more than any other reason. Terrorism to the NSA data collection is just a red herring.

EthanHeilman 12y ago

Actually the TRAILBLAZER project was the NSAs attempt and it was a total failure, costing around 3 billion dollars. They abandoned it in 2006. SAIC the contractor that "built" it for the NSA got pretty rich though. "Trailblazer was a United States National Security Agency (NSA) program intended to develop a capability to analyze data carried on communications networks like the Internet. In 2005, NSA director Michael Hayden told a Senate hearing that the Trailblazer program was several hundred million dollars over budget and years behind schedule. In 2006 the program was shut down, after having cost billions of US Dollars. Several anonymous NSA sources told Hosenball of Newsweek later on that the project was a "wasteful failure". The new project replacing Trailblazer is called Turbulence." - http://en.wikipedia.org/wiki/Trailblazer_Project http://en.wikipedia.org/wiki/Trailblazer_Project

xorcist 12y ago

That's a very interesting question. My pet theory is that it has to do with outsourcing and the widespread belief in the private market that software development is a product just like any other that you buy on the market. When your average intelligence agency has a million dollars to spend on a project, it will keep a house full of smart people and tell to forth and create, paying their bills and keeping their backs. We might call this the academic model, if it wasn't so far from how academia works today. When a civilian government agency (or any large private entity for that matter as they are completely similar in this regard) have a million dollars to spend on a project they immediately start phoning the big boys: We've got a million to spend on this project, can do do it? It is all a bit exaggerated of course, but in this made up example one will get a working, if perhaps shoddy product, and one will get a server filled with very important powerpoints while the product itself tends to be late and slow. Or to put things in a simpler way and less of a rant: I suspect it is a lot easier to spend a million dollars on a highly specialized system, than to do the same on a very generic system anyone could build. The corollary is that the same intelligence agency would fail in the same way a private or civilian organization would when implementing a new payroll system.

secfirstmd 12y ago
Let me help ya there "No one ever got fired for hiring IBM" :)

runjake 12y ago

It's not that the Intelligence Community has their act together any better, it's that they have more money to throw at the problems. Less regulation and oversight probably "help" things, too.

Istof 12y ago

They fixed Obamacare's website and they probably have even more incentives to fix the spying applications (if they are broken)... catching "terrorists" might be a low priority goal on their list which would explain why the reporters say that the NSA surveillance played little role in foiling terror plots.

chicofarm 12y ago

Government department may be incompetent, private defense contractors are some of most capable organizations in the world. Different US government operations are being increasingly privatized. One difference between private contractors and government organizations, is contractors are not held accountable to oversight (to congress for example) as the government entities are.

pdkl95 12y ago

While the other replies here have some very good ideas, there might be another reason for this (perceived? real?) difference that is far simpler: running a webapp on the scale of Obamacare/NHS is probably a much harder job than tapping the internet backbone. For the NSA/GCHQ/etc, the actual engineering task of setting up some DPI boxes, beam-splitters, and other wiretap equipment is not particularly complicated on the engineering side[1]. The difficulty with that kind of project is access and legality. Even their massive DB in Utah, while (very) large, is probably a fairly simple schema compared to a many business DBs. Compare that to a healthcare website, that has to implement many random features mandated by law in a way that is compatible with a giant list of "standards", medical jargons, insurance requirements of all sorts, financial regulations, to name just a few of the details. And it has to work, because lives are going to rely on it. While this kind of website might be simple in concept, the random, contradictory, poorly-specified, "design by committee" project requirements are going to be nasty. [1] crypto may be different, but very little of what the NSA is currently doing has anything to do with crypto. They don't even need side-channel attacks when it's all in plaintext... sigh