6 ms·

What saddens me is that, while red team pen testing is a very "hot" (high employer demand, high salaries) job market, people don't generally care about the blue

by sebcat 9y ago

What saddens me is that, while red team pen testing is a very "hot" (high employer demand, high salaries) job market, people don't generally care about the blue team. It's easy to get a pentesting gig that pays well, but employers don't ask for/value people with the competence to build/maintain secure applications/networks/solutions in my experience. Instead they pay for recurring pen tests which results in internal tickets/issues/BUGs, while the development/operation practices are kept the same.

solatic 9y ago
It's because internal econometrics are resulting in perverse incentives. Defensive security is a cost center without clear, deterministic metrics for success. Let's say you pay X on defensive security (which is an oversimplification when you're talking about a cultural change, but that cultural change involves people learning how to pay attention to security, and paying attention is a form of man-hours, for which a cost can be calculated). If you don't get attacked, is it because the X you paid is high enough to deter/foil attackers, or could you have paid less and achieved the same result? If you are attacked and the attackers get past your defenses, is it because the X you paid wasn't enough, or if you had spent more, would the attackers have succeeded anyway, because of their relative power and motivation? For defensive security, it's very, very hard to justify to bean counters that X was the correct amount of money spend, no matter what the real outcome is, because it's hard to understand X's affect on that outcome. Pentests which result in tickets/issues/etc. are much easier to justify. The company spent X on the pentest, and it got Y feedback in return. Simple, and effective, at least in the short-term. It's part of the overall challenge that organizations face when they become metrics-driven. People choose the path of least resistance, so if you ask people to measure data, they'll measure the data that's easiest to measure. Data that's harder to measure - culture and social attitudes - becomes "not a priority" to measure.
- dmitri1981 9y ago
  You probably meant Economics rather than Econometrics. Economics concerns itself with incentives and payoffs, whereas Econometrics is the statistical study of economic systems. Apologies for the pedantry
- logir 9y ago
  It's not only "internal". There isn't really a functioning market for "middleware" in much of software, creative areas excluded. I don't really believe that managers, users or anyone else is primarily responsible. At the end of the day most developers aren't very good at security and they aren't necessarily willing to pay for it either.
- golergka 9y ago
  Isn't a result of pentest a direct measure of defensive security as well?
j7ake 9y ago
In theory you could take advantage of this by starting a company that hires people from the blue side at a premium and create products with top notch security. Unfortunately I am not sure if currently consumers care that much about security of their products relative to convenience, price, and eye candy. But in theory those who spend lots on red side will end up having a more expensive product and a bad reputation, so perhaps investing more on blue will win in the long run.
- eterm 9y ago
  The trouble with that is that b2b contracts will specify pen tests so you'll incur that cost anyway to demonstrate you don't have security holes. "Regular pen test" is seen as demonstrating security, which is as little perverse because the results don't typically get published so you could be having the same issues year after year and look just as good as someone who gets a clean bill each time.
gcb0 9y ago
it's like everything in a big company. every round of red team on my team's applications we just sit and laugh as they find nothing, yet we serve data to billions of customers every hour, from a myriad of complex entry points. nobody cares, and when I mention that on my self reviews it looks like I am padding it. then the other teams only handle requests from the ios app they own, and red team finds tons of amateur attacks that work. they spend a quarter fixing it, and boast that they worked with the red team to patch hundreds of vulnerabilities. and everyone is promoted. but that is not new. it always happened with teams that causes outages, or teams that miss out obvious revenues stream for years. remedial action for some reason is always rewarded in troubled big corps.
- omginternets 9y ago
  >remedial action for some reason is always rewarded in troubled big corps. reply >for some reason I would go out on a limb to say it's definitional. A troubled Big Corp is troubled precisely because it focuses on the wrong thing.
- hellbanner 9y ago
  That's sad to hear. Possibly you could demonstrate "We defend against XXX attacks, compare that to the iOS app which defends against XX"
hexc0de 9y ago
I used to work for a pentest company and everytime we engage with a customer, we discover a lot of problems and security issues which we document and submit to the customer. We discover after a while that another company gets a contract 10x our price fixing the issues we discovered
- vinceguidry 9y ago
  Sounds like a missed opportunity to at least get a referral fee for resolution.
  - jacquesm 9y ago
    Conflict of interest.
    - MichaelBurge 9y ago
      Assuming the company is voluntarily hiring them, and not being required by a contract or law that needs an independent 3rd party for auditing purposes, it seems like the pentesting company would do an even better job on the inspection if they had a good chance of getting repair work down the line for every issue they found. If they make up a bunch of minor things that don't matter, you can ignore those and focus on the important ones. I suppose if you don't have any in-house expertise at all to evaluate what they say, the conflict would be more important?
      - jacquesm 9y ago
        It's simply bad practice to have the people that are involved in the 'checking' making money or being involved in the 'fixing' in any way shape or form. You'll see this in almost every situation that is somehow related to auditing.
        mhb 9y ago
        Like colleges.
deleted 9y ago
[deleted]
susam 9y ago
My experience is inconsistent with yours. In my 12 years of experience in the security industry, I have found that the blue teams (engineers who develop, maintain and secure network, data and applications) have higher demand and higher salaries than the red teams or pen-testing teams. My experience is limited to security software development in e-banking, e-commerce, network security and data security domains in technology areas like cryptography, PKI, deep packet inspection, and network protocols. I know my experience may not be representative of the entire security industry and there is a possible selection bias too (i.e. I may have seen more demand for blue team engineers because I have belonged to blue teams myself), but I thought I should share my experience here to present the other side of the story.
- hellbanner 9y ago
  Thanks for sharing this. You mention banking - protecting money seems like a much higher incentive for blue team than say, the security of a forum or online game server.
  - biggerbistro 9y ago
    Unfortunately not at the FIs I have worked for. Their approach seems to be "give the illusion of security while covering any actual losses with insurance". I think there needs to be greater punishment for companies that lose customer information. Only then will the incentives be large enough for something to be done.
    - geiseric 9y ago
      It's hard to put a price on good will and customer trust. Every C level should realize that their company is an information company and take appropriate action.
- geiseric 9y ago
  As a Blue team member who works in the medical field, I must agree, that demand is high. We hire a firm to do Red team, but we do have our internal folks too, along with us Blue team folks who learn Red team stuff. We worked with a major security firm and together we accidentally created Purple team. We had the Red team come in and while pentesting share his screen with us all. Another Red team member explained what he was doing and after an attack was launched and we would see if our tools detected the activity. If they didn't, we went out to find out why. This was huge. It showed us where we needed to tune some things and where we needed newer and/or different tools. This isn't the only way we get pen tested. They do their annual "regular" pentest. The Purple team thing was awesome though. We learned a ton. Since I happen to own most of our tools and am secondary on the ones I don't own, I have learned a tremendous amount and I've been in IT for 20 years.
jdc0589 9y ago
I'm lucky enough to be with a company with a couple people on both sides. Originally we were all blue team (that was VERY poorly integrated with the engineering group), but ive started shifting one guy over to full time pentesting and working on integration with engineering. > results in internal tickets/issues/BUGs, while the development/operation practices are kept the same. You could not be more accurate; this also applies to groups that maybe started out as corporate infosec (virus protection, simple application scanning, etc...) and were never really tightly coupled with engineering. We have identified essentially identical authorization issues in a pre-release version of one of our products two or three times this year, which was also present in the last 3rd party pentest of the same product before my time (which was pretty scathing). Its incredible.
jjguy 9y ago
This will change with time. I have thought, studied, talked, and written extensively on this general theme. What comes to mind to frame it for you is my "four principles:" [a] 1. Compromise is inevitable 2. Default-allow products always fail 3. 1 and 2 are not opinion or marketing spin, just simple truths 4. As an industry, we are still learning 1 and 2 Security is slowly shifting from an administrative IT function to an operational function. In IT, the business value comes from the products and people are a tax required to administer the products. In security operations, the business value comes from the people, products are just tools in their toolbag. [c] Keep walking this dog and you realize basic IT activities for core infrastructure are critical for security, to the point the CIO will report to the CISO -- unless the CIO steps up. [b] So - in short - your frustrations are accurate, but the winds are shifting. Companies will incresingly value top people for their internal staff/blue teams. It's going to take a few more years, but I believe it is inevitable. [a] - https://www.linkedin.com/pulse/my-four-cybersecurity-principles-j-j-guy https://www.linkedin.com/pulse/my-four-cybersecurity-princip... [b] - https://www.linkedin.com/pulse/cio-report-ciso-j-j-guy https://www.linkedin.com/pulse/cio-report-ciso-j-j-guy [c] - https://www.linkedin.com/pulse/cio-report-ciso-why-j-j-guy https://www.linkedin.com/pulse/cio-report-ciso-why-j-j-guy
brians 9y ago
I'm hiring for blue team at Akamai. We've picked up some fantastic early-career people into our training program; also looking for mid-career architects, senior architects. https://akamaijobs.referrals.selectminds.com/jobs/senior-lead-architect-information-security-4748 https://akamaijobs.referrals.selectminds.com/jobs/senior-lea... https://akamaijobs.referrals.selectminds.com/jobs/security-architect-4900 https://akamaijobs.referrals.selectminds.com/jobs/security-a... https://akamaijobs.referrals.selectminds.com/jobs/manager-information-security-5057 https://akamaijobs.referrals.selectminds.com/jobs/manager-in...
sporkenfang 9y ago
This. Not interested in pen testing at all, just want to build things that don't suck from a security perspective. Happy to break things along the way to gain perspective, and I like to know how things work, but I'm a software engineer who likes security. A builder, not a breaker, by nature.
SandwichTeeth 9y ago
Honestly, as someone who is leaving "blue-team" network security work for a "DevOps" production team (I know, I know), it's really a mixed bag. I've done blue-team for 2 companies now and honestly the job is more project management than anything else. I found that I was very rarely actually getting hands-on with technology. When implementing a new piece of security tech, we were simply directing other teams to perform most of the the actual technical work (this was the case at both of my "security engineering" jobs). I didn't get any of the satisfaction of building anything, solving problems etc. The other big thing to note is that a lot of companies have security teams solely to meet audit requirements. If you find yourself on a team like that, you'll be spending a lot of time just gathering evidence for audits, remediating findings and writing policy. I really loved security intellectually, but in practice, the blue-team side of things wasn't my cup of tea.
pdeuchler 9y ago
The company I work for is aggressively hiring blue teamers. The core of our product is essentially a security product, so we need someone who has at least some professional engineering experience as you would be responsible for helping to lead teams building certain parts of our infrastructure and product. Someone who can hop onto code reviews that deal with sensitive areas would be fantastic. We're located in Boulder but for the right candidate we'd consider remote, although that might involve relatively frequent travel. cGhpbGlwLmRldWNobGVyQGp1bXBjbG91ZC5jb20= for contact

by sebcat 9y ago

solatic 9y ago

It's because internal econometrics are resulting in perverse incentives. Defensive security is a cost center without clear, deterministic metrics for success. Let's say you pay X on defensive security (which is an oversimplification when you're talking about a cultural change, but that cultural change involves people learning how to pay attention to security, and paying attention is a form of man-hours, for which a cost can be calculated). If you don't get attacked, is it because the X you paid is high enough to deter/foil attackers, or could you have paid less and achieved the same result? If you are attacked and the attackers get past your defenses, is it because the X you paid wasn't enough, or if you had spent more, would the attackers have succeeded anyway, because of their relative power and motivation? For defensive security, it's very, very hard to justify to bean counters that X was the correct amount of money spend, no matter what the real outcome is, because it's hard to understand X's affect on that outcome. Pentests which result in tickets/issues/etc. are much easier to justify. The company spent X on the pentest, and it got Y feedback in return. Simple, and effective, at least in the short-term. It's part of the overall challenge that organizations face when they become metrics-driven. People choose the path of least resistance, so if you ask people to measure data, they'll measure the data that's easiest to measure. Data that's harder to measure - culture and social attitudes - becomes "not a priority" to measure.

dmitri1981 9y ago
You probably meant Economics rather than Econometrics. Economics concerns itself with incentives and payoffs, whereas Econometrics is the statistical study of economic systems. Apologies for the pedantry
logir 9y ago
It's not only "internal". There isn't really a functioning market for "middleware" in much of software, creative areas excluded. I don't really believe that managers, users or anyone else is primarily responsible. At the end of the day most developers aren't very good at security and they aren't necessarily willing to pay for it either.
golergka 9y ago
Isn't a result of pentest a direct measure of defensive security as well?

j7ake 9y ago

In theory you could take advantage of this by starting a company that hires people from the blue side at a premium and create products with top notch security. Unfortunately I am not sure if currently consumers care that much about security of their products relative to convenience, price, and eye candy. But in theory those who spend lots on red side will end up having a more expensive product and a bad reputation, so perhaps investing more on blue will win in the long run.

eterm 9y ago
The trouble with that is that b2b contracts will specify pen tests so you'll incur that cost anyway to demonstrate you don't have security holes. "Regular pen test" is seen as demonstrating security, which is as little perverse because the results don't typically get published so you could be having the same issues year after year and look just as good as someone who gets a clean bill each time.

gcb0 9y ago

it's like everything in a big company. every round of red team on my team's applications we just sit and laugh as they find nothing, yet we serve data to billions of customers every hour, from a myriad of complex entry points. nobody cares, and when I mention that on my self reviews it looks like I am padding it. then the other teams only handle requests from the ios app they own, and red team finds tons of amateur attacks that work. they spend a quarter fixing it, and boast that they worked with the red team to patch hundreds of vulnerabilities. and everyone is promoted. but that is not new. it always happened with teams that causes outages, or teams that miss out obvious revenues stream for years. remedial action for some reason is always rewarded in troubled big corps.

omginternets 9y ago
>remedial action for some reason is always rewarded in troubled big corps. reply >for some reason I would go out on a limb to say it's definitional. A troubled Big Corp is troubled precisely because it focuses on the wrong thing.
hellbanner 9y ago
That's sad to hear. Possibly you could demonstrate "We defend against XXX attacks, compare that to the iOS app which defends against XX"

hexc0de 9y ago

I used to work for a pentest company and everytime we engage with a customer, we discover a lot of problems and security issues which we document and submit to the customer. We discover after a while that another company gets a contract 10x our price fixing the issues we discovered

vinceguidry 9y ago
Sounds like a missed opportunity to at least get a referral fee for resolution.
- jacquesm 9y ago
  Conflict of interest.
  - MichaelBurge 9y ago
    Assuming the company is voluntarily hiring them, and not being required by a contract or law that needs an independent 3rd party for auditing purposes, it seems like the pentesting company would do an even better job on the inspection if they had a good chance of getting repair work down the line for every issue they found. If they make up a bunch of minor things that don't matter, you can ignore those and focus on the important ones. I suppose if you don't have any in-house expertise at all to evaluate what they say, the conflict would be more important?
    - jacquesm 9y ago
      It's simply bad practice to have the people that are involved in the 'checking' making money or being involved in the 'fixing' in any way shape or form. You'll see this in almost every situation that is somehow related to auditing.
      - mhb 9y ago
        Like colleges.

deleted 9y ago

[deleted]

susam 9y ago

My experience is inconsistent with yours. In my 12 years of experience in the security industry, I have found that the blue teams (engineers who develop, maintain and secure network, data and applications) have higher demand and higher salaries than the red teams or pen-testing teams. My experience is limited to security software development in e-banking, e-commerce, network security and data security domains in technology areas like cryptography, PKI, deep packet inspection, and network protocols. I know my experience may not be representative of the entire security industry and there is a possible selection bias too (i.e. I may have seen more demand for blue team engineers because I have belonged to blue teams myself), but I thought I should share my experience here to present the other side of the story.

hellbanner 9y ago
Thanks for sharing this. You mention banking - protecting money seems like a much higher incentive for blue team than say, the security of a forum or online game server.
- biggerbistro 9y ago
  Unfortunately not at the FIs I have worked for. Their approach seems to be "give the illusion of security while covering any actual losses with insurance". I think there needs to be greater punishment for companies that lose customer information. Only then will the incentives be large enough for something to be done.
  - geiseric 9y ago
    It's hard to put a price on good will and customer trust. Every C level should realize that their company is an information company and take appropriate action.
geiseric 9y ago
As a Blue team member who works in the medical field, I must agree, that demand is high. We hire a firm to do Red team, but we do have our internal folks too, along with us Blue team folks who learn Red team stuff. We worked with a major security firm and together we accidentally created Purple team. We had the Red team come in and while pentesting share his screen with us all. Another Red team member explained what he was doing and after an attack was launched and we would see if our tools detected the activity. If they didn't, we went out to find out why. This was huge. It showed us where we needed to tune some things and where we needed newer and/or different tools. This isn't the only way we get pen tested. They do their annual "regular" pentest. The Purple team thing was awesome though. We learned a ton. Since I happen to own most of our tools and am secondary on the ones I don't own, I have learned a tremendous amount and I've been in IT for 20 years.

jdc0589 9y ago

I'm lucky enough to be with a company with a couple people on both sides. Originally we were all blue team (that was VERY poorly integrated with the engineering group), but ive started shifting one guy over to full time pentesting and working on integration with engineering. > results in internal tickets/issues/BUGs, while the development/operation practices are kept the same. You could not be more accurate; this also applies to groups that maybe started out as corporate infosec (virus protection, simple application scanning, etc...) and were never really tightly coupled with engineering. We have identified essentially identical authorization issues in a pre-release version of one of our products two or three times this year, which was also present in the last 3rd party pentest of the same product before my time (which was pretty scathing). Its incredible.

jjguy 9y ago

This will change with time. I have thought, studied, talked, and written extensively on this general theme. What comes to mind to frame it for you is my "four principles:" [a] 1. Compromise is inevitable 2. Default-allow products always fail 3. 1 and 2 are not opinion or marketing spin, just simple truths 4. As an industry, we are still learning 1 and 2 Security is slowly shifting from an administrative IT function to an operational function. In IT, the business value comes from the products and people are a tax required to administer the products. In security operations, the business value comes from the people, products are just tools in their toolbag. [c] Keep walking this dog and you realize basic IT activities for core infrastructure are critical for security, to the point the CIO will report to the CISO -- unless the CIO steps up. [b] So - in short - your frustrations are accurate, but the winds are shifting. Companies will incresingly value top people for their internal staff/blue teams. It's going to take a few more years, but I believe it is inevitable. [a] - https://www.linkedin.com/pulse/my-four-cybersecurity-principles-j-j-guy https://www.linkedin.com/pulse/my-four-cybersecurity-princip... [b] - https://www.linkedin.com/pulse/cio-report-ciso-j-j-guy https://www.linkedin.com/pulse/cio-report-ciso-j-j-guy [c] - https://www.linkedin.com/pulse/cio-report-ciso-why-j-j-guy https://www.linkedin.com/pulse/cio-report-ciso-why-j-j-guy

brians 9y ago

I'm hiring for blue team at Akamai. We've picked up some fantastic early-career people into our training program; also looking for mid-career architects, senior architects. https://akamaijobs.referrals.selectminds.com/jobs/senior-lead-architect-information-security-4748 https://akamaijobs.referrals.selectminds.com/jobs/senior-lea... https://akamaijobs.referrals.selectminds.com/jobs/security-architect-4900 https://akamaijobs.referrals.selectminds.com/jobs/security-a... https://akamaijobs.referrals.selectminds.com/jobs/manager-information-security-5057 https://akamaijobs.referrals.selectminds.com/jobs/manager-in...

sporkenfang 9y ago

This. Not interested in pen testing at all, just want to build things that don't suck from a security perspective. Happy to break things along the way to gain perspective, and I like to know how things work, but I'm a software engineer who likes security. A builder, not a breaker, by nature.

SandwichTeeth 9y ago

Honestly, as someone who is leaving "blue-team" network security work for a "DevOps" production team (I know, I know), it's really a mixed bag. I've done blue-team for 2 companies now and honestly the job is more project management than anything else. I found that I was very rarely actually getting hands-on with technology. When implementing a new piece of security tech, we were simply directing other teams to perform most of the the actual technical work (this was the case at both of my "security engineering" jobs). I didn't get any of the satisfaction of building anything, solving problems etc. The other big thing to note is that a lot of companies have security teams solely to meet audit requirements. If you find yourself on a team like that, you'll be spending a lot of time just gathering evidence for audits, remediating findings and writing policy. I really loved security intellectually, but in practice, the blue-team side of things wasn't my cup of tea.

pdeuchler 9y ago

The company I work for is aggressively hiring blue teamers. The core of our product is essentially a security product, so we need someone who has at least some professional engineering experience as you would be responsible for helping to lead teams building certain parts of our infrastructure and product. Someone who can hop onto code reviews that deal with sensitive areas would be fantastic. We're located in Boulder but for the right candidate we'd consider remote, although that might involve relatively frequent travel. cGhpbGlwLmRldWNobGVyQGp1bXBjbG91ZC5jb20= for contact