5 ms·

Ask HN: How do you integrate corp security/risk people into agile projects?

With large orgs coming around to agile dev, as developers or product owners, how are you handling the gatekeeper review/approve role of security analysts? Edit: e.g. How do you get threat intelligence, compliance assessment, and a shareable security architecture to assure customers/stakeholders into a backlog?

toomuchtodo 8y ago
EDIT: lvh/latacora's answers above in thread are most likely better suited to your use case Cloud security architect at a financial services firm here (supporting thousands of devs). We're using the usual pull request review process between dev/ops teams and our risk management team (which is composed of various subject matter experts) with an SLA. We're also hard gating ("you shall not pass") using Sentinel rules for Terraform Enterprise and service control policies in AWS accounts. I highly recommend you put a training program together that incorporates a feedback loop from the above process. Before devs and ops folks start, they should be gated until they go through your training process. That should minimize the exceptions you'll need to catch in the review process, but when those exceptions occur, that should be noted for training improvement.
- hluska 8y ago
  A training program is an excellent idea. Thank you very much - that helps me a lot.
  - toomuchtodo 8y ago
    Happy to help.
- motohagiography 8y ago
  Training developers in is super valuable. Sounds like that is the lynchpin for the rest of the process.
lvh 8y ago
I don’t know what you had in mind for “large”, but I can tell you how it’s working out for us. - Most customers write design docs/tickets. This helps because we can suggest alternative approaches early. It’s rare that we tell people that an idea is fundamentally flawed, but it’s common that we suggest a different implementation approach or additional safeguards. - We review individual PRs. Here, we're looking for concrete individual appsec concerns (or infrasec if it’s a Terraform PR for example). Not the right place to have overarching product design conversations. One concern is that there are just too many PRs for a fledgling security team to review every single one. For this we rely on 2 mechanisms: 1) IC opt-in (they tag us on GitHub via @CLIENT/security when they would like a set of eyes on something or if a PR is part of a product change we’ve been involved in) 2) Automated heuristics. These have a hair trigger: lots of false positives. It’s just stuff like “are you adding middleware?”, “are you adding an endpoint?”, et cetera. That’s OK, because we don’t make it a gating part of CI: its job is to turn 100 PRs for us to look at into 10 PRs for us to skim and make a decision if it’s the 1 PR that deserves extensive scrutiny. So, overall: stop thinking of yourself as a gatekeeper and start thinking of yourself as an advisor. Our job is to make sure you understand what you’re doing and what your alternatives are. Security is never the only requirement: you have a business to run.
franzwong 8y ago
We use AppScan from IBM. https://www.ibm.com/security/application-security/appscan https://www.ibm.com/security/application-security/appscan
meowface 8y ago
Code/PR reviews (by other devs and by people who specialize in application security), static analysis, secure coding practices training, constant vulnerability assessments and penetration testing, and a strong security team who understands the need to balance productivity and security. Really, above all else it's just hiring or contracting (ideally hiring) good security people. Good appsec people to review code and look for vulns, people who just do education and training, people who just do penetration testing, people who just do security architecture and tooling. And of course a good security operations and incident response team to deal with the day-to-day "frontline" defense, investigation, and response. If they are full-time employees who are integrated with the company culture and are experienced enough with agile development, then odds are they're not going to set up more red tape than is necessary. If security is mostly offloaded to devs or operations and no independent security org exists at your company, you're in trouble.
- motohagiography 8y ago
  Thanks for this. Do product owners and developers get a sense of the product level threat model, e.g. "competitors reversing our code," "spammers will harvest our user identities," "a grad student is going to break this as their thesis project," or "a state level actor will try to establish persistence for Intel collection" ?
  - meowface 8y ago
    From what I've personally seen, it's typically the security organization's responsibility to construct threat models and risk profiles, which basically turns into more general downstream security guidance for product owners and devs. The people working on the products don't necessarily need to know or care about particular adversaries or scenarios or threats; they just need to understand application/network/data security concepts, be constantly vigilant, and not get too annoyed if a security team tells them they need to switch to a new way of doing something. A web developer should know what XSS and CSRF are and should know a S3 bucket with customer data and no ACL is bad, but they don't need to know what an APT or cyber kill chain is. Any vulnerability, misconfiguration, or exposure could be exploited by many different kinds of entities for many reasons. A company with a good security department will have a dedicated threat intelligence and research team to consider specific risks and adversaries, which they will share with adjacent security teams; that information probably isn't relevant to most people just working on a product. But sure, some of it may reach a product owner if there's a particular kind of risk inherent to it (like if the product is offering secure end-to-end encrypted communications as a selling point or if they're in a sector that's been heavily targeted by state actors). Every organization and product will differ a lot, but for any company with a not-insignificant presence in an industry, state actors (APTs) are pretty much always going to be a risk. The odds of being hit by an APT are pretty low for most small companies, but the risk is always there. In terms of what's most likely to regularly affect the average company, financially motivated cybercrime has pretty much always been and remains to be the biggest issue. Much like regular daily crime, I suppose. It's people just looking to make a buck. Commodity malware like ransomware and banking info stealers, vuln scanning bots that pop a shell on their web servers and sell access, phishing emails spreading malware or impersonating executives and requesting wire transfers are probably still the main risk for most companies out there, of all sizes. This is probably the biggest problem for most companies whether they're entirely tech/software-focused or just a retail outlet or restaurant. Sometimes you'll get more organized and sophisticated cybercrime actors targeting specific organizations in an APT-style manner, but it's still almost always using those tried-and-true basic tactics. And believe it or not, many state-sponsored APTs themselves often go with generic tactics, too. Most influential countries' intelligence services do seem to have at least one "A Team" conducting cyberwarfare/espionage, but some also seem to have a lot of grunts just looking for low-hanging fruit, and they're usually not much harder to detect or stop than more typical script kiddies you might otherwise encounter. Often the "A Teams" will intentionally stick to the most generic stuff they can find, too, to avoid tipping their hand that they're smarter than they seem and to hide the fact that they have access to non-public tools and exploits (both to prevent their "weapons" from being burned and to not seem like they're an APT). The most sophisticated groups will only pull out the big guns if all of the generic techniques fail. I'm not really well-versed in industrial espionage and competitors reverse engineering software, but I think it's not really something most companies need to care about. If your competitive advantage is actually at risk just because someone reverse engineered one of your binaries, there's probably something very wrong with your business model. The companies who need to care about that kind of stuff are usually already well-aware.
lvh 8y ago
'tptacek and I were just discussing this the other day for dependencies (both libraries and vendors). ICs in technical and non-technical roles ask us if a particular dependency is OK, and we evaluate it. Examples include the privileges a browser extension wants, the OAuth scopes a vendor wants, and the general security posture of a new library. Individual evaluations don't capture risk you get from just having a lot of dependencies. "What happens if this gets compromised" is something we consider individually, but it becomes more actionable when you look at your dependencies globally. Examples: someone loses their Chrome Store credentials, or someone accidentally pushes AWS keys to GitHub, or someone loses their npm creds in an eslint-scope style problem. Now the dependency itself is malicious. If it's a library, that means code execution on your developer laptops, CI systems, and possibly customer devices. Individual dependency requests are not the right place to push back on introducing a dependency. You need a second speed to address those by teaming up with e.g. the VPEng and usually someone in PeopleOps/HR/Sales/Marketing (for things like calendar mgmt, sales tools, ad-hoc CRMs) to periodically pare that list down.
lvh 8y ago
Train your ICs to be better at spotting risky behavior. They don't have to be able to exploit and fix the bugs: they just have to know when to ask for someone else to look at it. One thing we've found to be effective here: we start with a comprehensive audit (appsec, infrasec, corpsec, deployment security, external network footprint) that tells us about systemic problems you face. Then we can provide targeted training where you contextualize the problem for ICs. I'm sure Ryan McGee (magoo) has ways to tie that in to quantitatively making people better at risk estimates, but we're not trying to get ICs at clients to do that (yet).
jbob2000 8y ago
At my company (80,000+ people, bank), we have an app security department that handles everything. We give them a URL to our test environment for a dynamic scan and we give them the contents of our release branch for a static scan. We are not allowed to release until the security scans are done and reviewed with the app security team. We release about once a quarter because of this. We do not practice continuous delivery and doing agile is difficult. There's no gatekeeper, story grooming, product owners, nada. You do what app security says when they say to do it, we work on their schedule. As a tangent, working for a large company has shown me that agile doesn't really work at this scale. There's too many third parties that you have no control over. And the business is always trying to respond to the market, so projects are changing all the time and people are always shifting roles. I just come in, put my commits on the branch, and pray that the machine has my back.
- motohagiography 8y ago
  Wrt agile not scaling because of 3rd parties you don't control, would you attribute that more to unwillingness or inability of third parties like security to collaborate?
  - jbob2000 8y ago
    I don't want to call it an unwillingness or inability, because they are good people who do good work. I really just think that at our scale, we move with the world, and the world isn't agile nor does it care about agile, it just is. So really, we aren't agile because the world isn't agile. For example, we were preparing to run some world cup soccer promotions based on the teams that won. We literally had to wait until a soccer match finished in order to deliver our changes. How do you practice agile with that kind of restriction?

toomuchtodo 8y ago

EDIT: lvh/latacora's answers above in thread are most likely better suited to your use case Cloud security architect at a financial services firm here (supporting thousands of devs). We're using the usual pull request review process between dev/ops teams and our risk management team (which is composed of various subject matter experts) with an SLA. We're also hard gating ("you shall not pass") using Sentinel rules for Terraform Enterprise and service control policies in AWS accounts. I highly recommend you put a training program together that incorporates a feedback loop from the above process. Before devs and ops folks start, they should be gated until they go through your training process. That should minimize the exceptions you'll need to catch in the review process, but when those exceptions occur, that should be noted for training improvement.

hluska 8y ago
A training program is an excellent idea. Thank you very much - that helps me a lot.
- toomuchtodo 8y ago
  Happy to help.
motohagiography 8y ago
Training developers in is super valuable. Sounds like that is the lynchpin for the rest of the process.

lvh 8y ago

I don’t know what you had in mind for “large”, but I can tell you how it’s working out for us. - Most customers write design docs/tickets. This helps because we can suggest alternative approaches early. It’s rare that we tell people that an idea is fundamentally flawed, but it’s common that we suggest a different implementation approach or additional safeguards. - We review individual PRs. Here, we're looking for concrete individual appsec concerns (or infrasec if it’s a Terraform PR for example). Not the right place to have overarching product design conversations. One concern is that there are just too many PRs for a fledgling security team to review every single one. For this we rely on 2 mechanisms: 1) IC opt-in (they tag us on GitHub via @CLIENT/security when they would like a set of eyes on something or if a PR is part of a product change we’ve been involved in) 2) Automated heuristics. These have a hair trigger: lots of false positives. It’s just stuff like “are you adding middleware?”, “are you adding an endpoint?”, et cetera. That’s OK, because we don’t make it a gating part of CI: its job is to turn 100 PRs for us to look at into 10 PRs for us to skim and make a decision if it’s the 1 PR that deserves extensive scrutiny. So, overall: stop thinking of yourself as a gatekeeper and start thinking of yourself as an advisor. Our job is to make sure you understand what you’re doing and what your alternatives are. Security is never the only requirement: you have a business to run.

franzwong 8y ago

We use AppScan from IBM. https://www.ibm.com/security/application-security/appscan https://www.ibm.com/security/application-security/appscan

meowface 8y ago

Code/PR reviews (by other devs and by people who specialize in application security), static analysis, secure coding practices training, constant vulnerability assessments and penetration testing, and a strong security team who understands the need to balance productivity and security. Really, above all else it's just hiring or contracting (ideally hiring) good security people. Good appsec people to review code and look for vulns, people who just do education and training, people who just do penetration testing, people who just do security architecture and tooling. And of course a good security operations and incident response team to deal with the day-to-day "frontline" defense, investigation, and response. If they are full-time employees who are integrated with the company culture and are experienced enough with agile development, then odds are they're not going to set up more red tape than is necessary. If security is mostly offloaded to devs or operations and no independent security org exists at your company, you're in trouble.

motohagiography 8y ago
Thanks for this. Do product owners and developers get a sense of the product level threat model, e.g. "competitors reversing our code," "spammers will harvest our user identities," "a grad student is going to break this as their thesis project," or "a state level actor will try to establish persistence for Intel collection" ?
- meowface 8y ago
  From what I've personally seen, it's typically the security organization's responsibility to construct threat models and risk profiles, which basically turns into more general downstream security guidance for product owners and devs. The people working on the products don't necessarily need to know or care about particular adversaries or scenarios or threats; they just need to understand application/network/data security concepts, be constantly vigilant, and not get too annoyed if a security team tells them they need to switch to a new way of doing something. A web developer should know what XSS and CSRF are and should know a S3 bucket with customer data and no ACL is bad, but they don't need to know what an APT or cyber kill chain is. Any vulnerability, misconfiguration, or exposure could be exploited by many different kinds of entities for many reasons. A company with a good security department will have a dedicated threat intelligence and research team to consider specific risks and adversaries, which they will share with adjacent security teams; that information probably isn't relevant to most people just working on a product. But sure, some of it may reach a product owner if there's a particular kind of risk inherent to it (like if the product is offering secure end-to-end encrypted communications as a selling point or if they're in a sector that's been heavily targeted by state actors). Every organization and product will differ a lot, but for any company with a not-insignificant presence in an industry, state actors (APTs) are pretty much always going to be a risk. The odds of being hit by an APT are pretty low for most small companies, but the risk is always there. In terms of what's most likely to regularly affect the average company, financially motivated cybercrime has pretty much always been and remains to be the biggest issue. Much like regular daily crime, I suppose. It's people just looking to make a buck. Commodity malware like ransomware and banking info stealers, vuln scanning bots that pop a shell on their web servers and sell access, phishing emails spreading malware or impersonating executives and requesting wire transfers are probably still the main risk for most companies out there, of all sizes. This is probably the biggest problem for most companies whether they're entirely tech/software-focused or just a retail outlet or restaurant. Sometimes you'll get more organized and sophisticated cybercrime actors targeting specific organizations in an APT-style manner, but it's still almost always using those tried-and-true basic tactics. And believe it or not, many state-sponsored APTs themselves often go with generic tactics, too. Most influential countries' intelligence services do seem to have at least one "A Team" conducting cyberwarfare/espionage, but some also seem to have a lot of grunts just looking for low-hanging fruit, and they're usually not much harder to detect or stop than more typical script kiddies you might otherwise encounter. Often the "A Teams" will intentionally stick to the most generic stuff they can find, too, to avoid tipping their hand that they're smarter than they seem and to hide the fact that they have access to non-public tools and exploits (both to prevent their "weapons" from being burned and to not seem like they're an APT). The most sophisticated groups will only pull out the big guns if all of the generic techniques fail. I'm not really well-versed in industrial espionage and competitors reverse engineering software, but I think it's not really something most companies need to care about. If your competitive advantage is actually at risk just because someone reverse engineered one of your binaries, there's probably something very wrong with your business model. The companies who need to care about that kind of stuff are usually already well-aware.

lvh 8y ago

'tptacek and I were just discussing this the other day for dependencies (both libraries and vendors). ICs in technical and non-technical roles ask us if a particular dependency is OK, and we evaluate it. Examples include the privileges a browser extension wants, the OAuth scopes a vendor wants, and the general security posture of a new library. Individual evaluations don't capture risk you get from just having a lot of dependencies. "What happens if this gets compromised" is something we consider individually, but it becomes more actionable when you look at your dependencies globally. Examples: someone loses their Chrome Store credentials, or someone accidentally pushes AWS keys to GitHub, or someone loses their npm creds in an eslint-scope style problem. Now the dependency itself is malicious. If it's a library, that means code execution on your developer laptops, CI systems, and possibly customer devices. Individual dependency requests are not the right place to push back on introducing a dependency. You need a second speed to address those by teaming up with e.g. the VPEng and usually someone in PeopleOps/HR/Sales/Marketing (for things like calendar mgmt, sales tools, ad-hoc CRMs) to periodically pare that list down.

lvh 8y ago

Train your ICs to be better at spotting risky behavior. They don't have to be able to exploit and fix the bugs: they just have to know when to ask for someone else to look at it. One thing we've found to be effective here: we start with a comprehensive audit (appsec, infrasec, corpsec, deployment security, external network footprint) that tells us about systemic problems you face. Then we can provide targeted training where you contextualize the problem for ICs. I'm sure Ryan McGee (magoo) has ways to tie that in to quantitatively making people better at risk estimates, but we're not trying to get ICs at clients to do that (yet).

jbob2000 8y ago

At my company (80,000+ people, bank), we have an app security department that handles everything. We give them a URL to our test environment for a dynamic scan and we give them the contents of our release branch for a static scan. We are not allowed to release until the security scans are done and reviewed with the app security team. We release about once a quarter because of this. We do not practice continuous delivery and doing agile is difficult. There's no gatekeeper, story grooming, product owners, nada. You do what app security says when they say to do it, we work on their schedule. As a tangent, working for a large company has shown me that agile doesn't really work at this scale. There's too many third parties that you have no control over. And the business is always trying to respond to the market, so projects are changing all the time and people are always shifting roles. I just come in, put my commits on the branch, and pray that the machine has my back.

motohagiography 8y ago
Wrt agile not scaling because of 3rd parties you don't control, would you attribute that more to unwillingness or inability of third parties like security to collaborate?
- jbob2000 8y ago
  I don't want to call it an unwillingness or inability, because they are good people who do good work. I really just think that at our scale, we move with the world, and the world isn't agile nor does it care about agile, it just is. So really, we aren't agile because the world isn't agile. For example, we were preparing to run some world cup soccer promotions based on the teams that won. We literally had to wait until a soccer match finished in order to deliver our changes. How do you practice agile with that kind of restriction?