7 ms·

Automation Myths: The NSA Can't Replace 90% of Its System Administrators

56 points by adfm 13y ago

deleted 13y ago
[deleted]
diminoten 13y ago
This reads like the rest of the industry's pitch for automation against the cries of, "but you're automating me out of a job!" Automation, done right, does replace low-skill labor. In QA (my perspective), automation replaces the need to run regression tests for each build. That means if a person's job is primarily comprised of executing regression tests, they're not likely going to have much to do once regression tests get automated. Similarly, if the NSA hired systems administrators to primarily do things like deploy servers etc., then those people can be replaced by automation. However, your "average" systems administrator job is as much about firefighting, troubleshooting, and customer service (internal customers) as it is deploying servers and managing other (automatable) technical problems. Some of NSA systems administrators are possibly working on optimizing and scaling, or implementing what the NSA's developers and architects have designed and written - stuff that can't realistically be automated. That said, they probably could play games with titles if they really wanted to get this done without losing their ability to function. Rename a bunch of sysadmin positions as "automation engineers", or reclassify their current sysadmins as "developers" or "systems engineers", and continue on their merry way, happy to have satisfied some congressional pressure without having slowed operations.
- rst 13y ago
  Automation, done right, does replace low-skilled labor --- but doing it right involves completely re-engineering the tasks to support it. In system administration, for instance, the scripts to configure (and monitor, and alert about malfunctions on) new servers don't write themselves, and they can be really tricky. So, achieving those benefits in a re-engineered version of an existing system can be a long, hard slog. That said, if NSA's really doing this to improve security (if only by reducing exposure to malcontents), then doing it right is the only option. So long as there are the same number of hands on the same number of keyboards, doing the same sort of thing, relabeling them does nothing to reduce the risk.
- quanticle 13y ago
  That reminds of a conversation I had with a sysadmin colleague about this very thing. He said, "Yeah, knowing the government, what they're going to do is rename a bunch of sysadmins as 'devops' or something and just carry on as if nothing happened."
- walshemj 13y ago
  But the NSA CIA etc is not low skill labor and has specialized needs I could see replacing 90% of the contractor Sysadmins bringing the job in house would mean better control. And in high security environments you want to strictly limit what one person can do this is security 101. The NSA has talked about implementing the two-man rule and presumably making much more use of discretionary access control which increases staff requirements.
  - diminoten 13y ago
    Why wouldn't there be some low-skill labor involved? Someone has to physically place the server blades into the racks, install operating systems and necessary libraries, manage raid setups, configure the network, etc. Sure these government orgs have special needs as well, but they don't magically do away with the mundane needs. They are, after all, building a huge data center - someone (or something) is going to have to manage the space. Sure, Ed Snowden wasn't a low-skill labor IT worker, but I bet when they did an internal audit, they discovered a bunch of low-skill IT workers with more access than they needed, and it was at that point they realized not everyone is doing a job a human has to. But again, I bet this is going to be more reclassification than it is going to be actually having less warm bodies on the project.
    - walshemj 13y ago
      Well building a secure system for a TLA isn't low skill labor It needs to be built to very high standards googles data centers have poor physical security for example. For this sort of use case you need to trust the people who are building your DC's just as much as your sysadmin with root access. Why do you think the FO in the UK has people whose job is to design and build secure sites and to oversee contractors - to avoid the unfortunate experience of finding bugs in your embassy as the USA has done in the past.
      - diminoten 13y ago
        Designing? Not low skill. Actually building? Possibly less skilled. Frank Gary doesn't need master carpenters to get his dream created.
dsr_ 13y ago
Hi, I'm a sysadmin. 90% was always a bogus value, but that's not the main problem that I see. Intelligence agencies are in the business of gathering, recording, analyzing and cross referencing information, and then producing reports -- all while maintaining compartmentalization to a degree that would be insane in most other contexts. You should never outsource your value proposition. What's the key differentiator between the NSA and, say Arbor or Stratfor? Ignore the lack of need to be profitable. The main difference is that the NSA needs to make security its religion. To do that, the principle of least privilege applies. The concept of Need To Know applies. And whenever possible, decisions need to be implemented by two or more people who can't/won't conspire with each other. Sysadmins set those systems up and run them. Sysadmins understand security as it actually is -- bits without Colour, c.f. yesterday's discussion -- and can turn the Colourful requirements documents into reasonable facsimiles of working systems. The NSA should never outsource computing expertise, be it development or systems administration. If they were to eliminate 90% of their contract sysadmins - by bringing them in-house - the problem would be with the remaining 10%. But reduce their operational workforce like that and claim automation can do it? That's so much bull.
- fixxer 13y ago
  > You should never outsource your value proposition. I really liked that. Well said. As for the NSA issuing statements of any kind, I just ignore them these days. I can't speak as to the requirements in terms of managing large systems of compute clusters, but I will speculate that no matter how "smart" their algos are, at some point, they've got to have a few hundred devoted souls doing QA/QC... probably young Air Force recruits. I simply do not believe they're getting value out of the data without substantial QA/QC. Alternatively, perhaps they're simply not getting any real value out of the system because it doesn't really work as a sieve. If you already have a target, then parallel construction is probably greatly improved.
- joe_the_user 13y ago
  You should never outsource your value proposition. Probably that literally is true. But the phrase itself generally goes with the unsaid assumption you should outsource something. It's been said that security is different from everything else in that it can't be bolted on later and it can't be the job of some of the organization. That's probably true too. And if the NSA's "value proposition" is security, then one assumes they shouldn't outsource anything. But there's this small problem that the NSA metastasizing has gone hand-in-hand with the whole government contracting revolution. Especially, the various private entities sucking the teet of the state are the ones best placed to lobby for more and more funds for the various agencies, the "revolving door" is now well established as the means by which agency officials are rewarded and so-forth. So you can't stop subcontracting yet you can't do it securely. Well, bummer for them. Hopefully there will be more, not fewer, breaches in the future. They seem more in our advantage than otherwise.
RougeFemme 13y ago
If the sysadmin staffing was at the level truly needed, I don't see how they could reduce it without playing the games described her in other posts - bringing work in house, reclassifying positions, etc. But it could be that the staffing level was bloated to begin with. . .not uncommon with government contracts. And - being slightly facetious here - they might just choose to reduce the multiple of human redundancy that the staffing level represents. . .from say quintuple to triple redundancy. Comb through the remaining system admins and remove those viewed as most likely to "leak" information. . .although they are obviously not that great at it.
deleted 13y ago
[deleted]
karlkatzke 13y ago
HN, and many of the technical writers covering this story, are missing out on a major subtext of this story. I'm on my phone, so I can't cite my source, but look at the actual quotes from NSA staff. They're not laying off 90% of the people that performed system administration tasks, they're revoking the administration rights of 90% of the people that have them. That means that their Windows domain was so badly administered that only 10% of the people who have Administrator roles were performing administration tasks. I doubt anyone is actually losing their job nor do I think they're adding significant automation.
- courtneywylie 13y ago
  Here's the Reuters quote from NSA Director Keith Alexander that most people (myself included) are using for this: "What we’re in the process of doing – not fast enough – is reducing our system administrators by about 90 percent."
  - karlkatzke 13y ago
    The piece that's missing that I can't find right now, but was fully quoted by the BBC, is the adjacent sentence, which was about how they had been too permissive with user permissions.
coldcode 13y ago
We can hope they make it impossible to get any work done. I say eliminate 100%.
mathattack 13y ago
I was thinking the same thing when I saw that. If you have too few people doing the job, the sh*t will really hit the fan. If everyone's too busy to do their own job, nobody will watch the watchers.
batgaijin 13y ago
Well it's possible if they are switching from a leviathan sharepoint setup.

56 points by adfm 13y ago

deleted 13y ago

[deleted]

This reads like the rest of the industry's pitch for automation against the cries of, "but you're automating me out of a job!" Automation, done right, does replace low-skill labor. In QA (my perspective), automation replaces the need to run regression tests for each build. That means if a person's job is primarily comprised of executing regression tests, they're not likely going to have much to do once regression tests get automated. Similarly, if the NSA hired systems administrators to primarily do things like deploy servers etc., then those people can be replaced by automation. However, your "average" systems administrator job is as much about firefighting, troubleshooting, and customer service (internal customers) as it is deploying servers and managing other (automatable) technical problems. Some of NSA systems administrators are possibly working on optimizing and scaling, or implementing what the NSA's developers and architects have designed and written - stuff that can't realistically be automated. That said, they probably could play games with titles if they really wanted to get this done without losing their ability to function. Rename a bunch of sysadmin positions as "automation engineers", or reclassify their current sysadmins as "developers" or "systems engineers", and continue on their merry way, happy to have satisfied some congressional pressure without having slowed operations.

rst 13y ago
Automation, done right, does replace low-skilled labor --- but doing it right involves completely re-engineering the tasks to support it. In system administration, for instance, the scripts to configure (and monitor, and alert about malfunctions on) new servers don't write themselves, and they can be really tricky. So, achieving those benefits in a re-engineered version of an existing system can be a long, hard slog. That said, if NSA's really doing this to improve security (if only by reducing exposure to malcontents), then doing it right is the only option. So long as there are the same number of hands on the same number of keyboards, doing the same sort of thing, relabeling them does nothing to reduce the risk.
quanticle 13y ago
That reminds of a conversation I had with a sysadmin colleague about this very thing. He said, "Yeah, knowing the government, what they're going to do is rename a bunch of sysadmins as 'devops' or something and just carry on as if nothing happened."
walshemj 13y ago
But the NSA CIA etc is not low skill labor and has specialized needs I could see replacing 90% of the contractor Sysadmins bringing the job in house would mean better control. And in high security environments you want to strictly limit what one person can do this is security 101. The NSA has talked about implementing the two-man rule and presumably making much more use of discretionary access control which increases staff requirements.
- diminoten 13y ago
  Why wouldn't there be some low-skill labor involved? Someone has to physically place the server blades into the racks, install operating systems and necessary libraries, manage raid setups, configure the network, etc. Sure these government orgs have special needs as well, but they don't magically do away with the mundane needs. They are, after all, building a huge data center - someone (or something) is going to have to manage the space. Sure, Ed Snowden wasn't a low-skill labor IT worker, but I bet when they did an internal audit, they discovered a bunch of low-skill IT workers with more access than they needed, and it was at that point they realized not everyone is doing a job a human has to. But again, I bet this is going to be more reclassification than it is going to be actually having less warm bodies on the project.
  - walshemj 13y ago
    Well building a secure system for a TLA isn't low skill labor It needs to be built to very high standards googles data centers have poor physical security for example. For this sort of use case you need to trust the people who are building your DC's just as much as your sysadmin with root access. Why do you think the FO in the UK has people whose job is to design and build secure sites and to oversee contractors - to avoid the unfortunate experience of finding bugs in your embassy as the USA has done in the past.
    - diminoten 13y ago
      Designing? Not low skill. Actually building? Possibly less skilled. Frank Gary doesn't need master carpenters to get his dream created.

dsr_ 13y ago

Hi, I'm a sysadmin. 90% was always a bogus value, but that's not the main problem that I see. Intelligence agencies are in the business of gathering, recording, analyzing and cross referencing information, and then producing reports -- all while maintaining compartmentalization to a degree that would be insane in most other contexts. You should never outsource your value proposition. What's the key differentiator between the NSA and, say Arbor or Stratfor? Ignore the lack of need to be profitable. The main difference is that the NSA needs to make security its religion. To do that, the principle of least privilege applies. The concept of Need To Know applies. And whenever possible, decisions need to be implemented by two or more people who can't/won't conspire with each other. Sysadmins set those systems up and run them. Sysadmins understand security as it actually is -- bits without Colour, c.f. yesterday's discussion -- and can turn the Colourful requirements documents into reasonable facsimiles of working systems. The NSA should never outsource computing expertise, be it development or systems administration. If they were to eliminate 90% of their contract sysadmins - by bringing them in-house - the problem would be with the remaining 10%. But reduce their operational workforce like that and claim automation can do it? That's so much bull.

fixxer 13y ago
> You should never outsource your value proposition. I really liked that. Well said. As for the NSA issuing statements of any kind, I just ignore them these days. I can't speak as to the requirements in terms of managing large systems of compute clusters, but I will speculate that no matter how "smart" their algos are, at some point, they've got to have a few hundred devoted souls doing QA/QC... probably young Air Force recruits. I simply do not believe they're getting value out of the data without substantial QA/QC. Alternatively, perhaps they're simply not getting any real value out of the system because it doesn't really work as a sieve. If you already have a target, then parallel construction is probably greatly improved.
joe_the_user 13y ago
You should never outsource your value proposition. Probably that literally is true. But the phrase itself generally goes with the unsaid assumption you should outsource something. It's been said that security is different from everything else in that it can't be bolted on later and it can't be the job of some of the organization. That's probably true too. And if the NSA's "value proposition" is security, then one assumes they shouldn't outsource anything. But there's this small problem that the NSA metastasizing has gone hand-in-hand with the whole government contracting revolution. Especially, the various private entities sucking the teet of the state are the ones best placed to lobby for more and more funds for the various agencies, the "revolving door" is now well established as the means by which agency officials are rewarded and so-forth. So you can't stop subcontracting yet you can't do it securely. Well, bummer for them. Hopefully there will be more, not fewer, breaches in the future. They seem more in our advantage than otherwise.

RougeFemme 13y ago

If the sysadmin staffing was at the level truly needed, I don't see how they could reduce it without playing the games described her in other posts - bringing work in house, reclassifying positions, etc. But it could be that the staffing level was bloated to begin with. . .not uncommon with government contracts. And - being slightly facetious here - they might just choose to reduce the multiple of human redundancy that the staffing level represents. . .from say quintuple to triple redundancy. Comb through the remaining system admins and remove those viewed as most likely to "leak" information. . .although they are obviously not that great at it.

deleted 13y ago

[deleted]

karlkatzke 13y ago

HN, and many of the technical writers covering this story, are missing out on a major subtext of this story. I'm on my phone, so I can't cite my source, but look at the actual quotes from NSA staff. They're not laying off 90% of the people that performed system administration tasks, they're revoking the administration rights of 90% of the people that have them. That means that their Windows domain was so badly administered that only 10% of the people who have Administrator roles were performing administration tasks. I doubt anyone is actually losing their job nor do I think they're adding significant automation.

courtneywylie 13y ago
Here's the Reuters quote from NSA Director Keith Alexander that most people (myself included) are using for this: "What we’re in the process of doing – not fast enough – is reducing our system administrators by about 90 percent."
- karlkatzke 13y ago
  The piece that's missing that I can't find right now, but was fully quoted by the BBC, is the adjacent sentence, which was about how they had been too permissive with user permissions.

coldcode 13y ago

We can hope they make it impossible to get any work done. I say eliminate 100%.

mathattack 13y ago

I was thinking the same thing when I saw that. If you have too few people doing the job, the sh*t will really hit the fan. If everyone's too busy to do their own job, nobody will watch the watchers.

batgaijin 13y ago

Well it's possible if they are switching from a leviathan sharepoint setup.