6 ms·

Stupid question, how does a security audit work? Do the consultants just read through the code? Do they try to find security bug like they do on bug bounty prog

by arrmn 10y ago

Stupid question, how does a security audit work? Do the consultants just read through the code? Do they try to find security bug like they do on bug bounty programs?

sb8244 10y ago
I'm not an expert in this field, but we recently did a security audit. The auditors get access to the code in order to evaluate it for vulnerabilities. In our ruby application, they also check gems that we are using (through open source tools albeit). They also did an in-app audit where they tried to break the application however they might see that. Having access to the code helps with this. When you get audited by potential customer, it usually involves not having code access and trying to penetrate the app without that access.
- BinaryIdiot 10y ago
  > When you get audited by potential customer, it usually involves not having code access and trying to penetrate the app without that access. Is this in reference to on-prem / enterprise software and is this typical? I haven't heard of customers doing this but it certainly makes sense (might as well invest thousands to test before spending magnitudes more on the product itself only to find it having a huge security hole). Then again I'm not sure I've worked with potential customers who have access to do something like that.
  - softawre 10y ago
    We just signed a big deal with a Google subsidiary, and part of that deal required us to go through a third party penetration test (no code access).
    - BinaryIdiot 10y ago
      Well today I learned. Thanks!
dsacco 10y ago
Good question. It can be all or none of the above. Here's what happens at a high level: Once a company decides it needs a security assessment performed on an application, it engages with a consulting firm. Consulting firms generally offer a variety of services, from web and mobile application penetration tests, to cryptanalysis (implementation and design), to reverse engineering and binary penetration testing, with source code audits sprinkled throughout (or as standalone assessments). Let's assume they move forward with a web application assessment. The company decides if it wants a source code audit, a penetration test or both. The most comprehensive assessments will include source code and unmitigated access to a staging environment that the consultants do not have to worry about destroying. However, they could also decide they don't want to hand over the code (common in things like sensitive financial applications or in applications with protective developers). I've worked on many assessments where I had no source code - this is called a "black-box" assessment. Conversely, an assessment might consist of a source code audit with no penetration test! This is less common, but it's particularly suited for engagements where the developers are fairly sure they've eliminated the most common issues and they are really focused on obscure errors, logic flaws and race conditions. It really depends on the type of security audit. You can have more exotic ones, like black-box cryptanalysis where a company hands Riscure a proprietary payment mechanism and there is heavy reverse engineering and side channel analysis. It can also be very vanilla, like the web application penetration tests that bug bounty programs attempt to simulate. Companies decide what they are going to do based on their application's profile and their goals. Putting this all together, these are the stages of a traditional security audit from a high-quality firm: Step 1: A company receives several proposals and decides which company to move forward with based on which statement of work most closely matches their security goals, timing, budget and desired expertise. Then they decide on a start date. Step 2: Representatives from the company (generally a technical manager, a security engineer or manager if the company has one and at least one developer) have a conference call with representatives from the security firm (generally, the security consultants performing the assessment, an account executive and a technical manager) to "kick off" the assessment with technical and logistical engagement planning. Things like "How will we access the staging environment?" and "Is there anything off-limits" are fleshed out here as well as reminders about scope and scheduling. Step 3: Things like source code, infrastructure/application/API documentation, PGP keys, etc. are securely exchanged and verified. This comes out of a list of mutual action items from the kick-off call. Step 4: The actual assessment happens, generally in a period of one to three weeks. I've never been involved in an assessment less than one week long, and assessments longer than four weeks usually need to re-scope or they become monolithic and difficult to coordinate. Progress reports with findings and testing data are securely sent to the company from the security firm. Step 5: The assessment is finished and a final deliverable is securely sent to the company from the security firm. An optional re-test assessment might happen a few weeks or months later to confirm if the findings have been satisfactorily resolved. This is based on my knowledge of having worked in security consultancies, engaging with them as an in-house security engineer and running my own consulting firm.
opinion1 10y ago
Yes. The way it works is that two smart hackers go into the office each day and spend eight hours trying to think of as many creative ways of tapping on the target application as possible. Nothing is off-limits except what is agreed up front, but you're obviously expected not to interfere with production operations. The client is generally expected to set up a testing environment substantially similar to production, but usually consultants just have to muddle through with whatever the clients give, which may not (and usually isn't) populated with production data. As long as consultants are given the ability to enter in data themselves, i.e. admin accounts, this is fine, because data entry is their job. Hacking is basically large-scale data entry, and it's as boring as it sounds: very tedious, interspersed with excitement when you see an XSS popup window or figure out a clever way to get a reverse shell. If after two weeks of this you found no medium-or-higher security vulnerabilities, you were generally considered to not be doing a very good job. The secret of the industry is that at the end of this process, you are deemed secure. That's the point of the security audit. But if it's not a repeating process, it doesn't work. It may work for that particular version of the application, and it may substantially improve the security in that old vulnerabilities are found and fixed. Let me abandon this train of thought and put it another way: This post is a press release saying that phpMyAdmin is secure. But that's not how this works. High-severity vulnerabilities are often found near the end of an audit. This is because the consultants have had time to become intimately familiar with the application. But the late stages of an audit are exactly when the consultant's time is mostly spent writing reports for the existing findings, and not doing pentesting. This means that two weeks is often just long enough to start finding serious vulns, since week one can be devoted to pentesting and week two is mostly reporting from Tuesday onward. But that "mostly reporting" process gets the consultant thinking about the application as they're doing the writeups, which -- you guessed it -- leads to realizing that there's something clever they could try. And when they try that clever thing, sometimes it yields a high-severity vuln. It's the opposite of a mechanical, thoughtless process. That means your results will vary depending on who, specifically, is doing the auditing. If you run your application through the consulting process twice -- same version, same staging data, same everything -- it's likely that you'll get wildly different results, because the pentesters are different people. It has to be an on-going process in order to be effective. And it can be highly effective. It just costs so much that only the most massive companies can afford this. That's not to say this audit wasn't effective. It's possible that whoever did the audit found substantially everything. But it was interesting to discover how often this was not the case, in a "How'd they miss this last time?" sort of way.
pg_is_a_butt 10y ago
yes. i had to do a security audit with NCC Group before... they need all your source code, all your build processes, all your environment setup scripts... basically everything. imagine training a whole new team now to build and maintain your company from scratch. then they tell you everything you already knew but management told you not to bother fixing. then management yells at you for not fixing it earlier. then everyone is fired and NCC Group takes over.
jquast 10y ago
The first few chapters of the book, "The Art of Software Security Assessment: Identifying and Preventing Software Vulnerabilities" outlines a very meticulous process of reviewing source code for vulnerabilities in a professional manner.

by arrmn 10y ago

Stupid question, how does a security audit work? Do the consultants just read through the code? Do they try to find security bug like they do on bug bounty programs?

sb8244 10y ago

I'm not an expert in this field, but we recently did a security audit. The auditors get access to the code in order to evaluate it for vulnerabilities. In our ruby application, they also check gems that we are using (through open source tools albeit). They also did an in-app audit where they tried to break the application however they might see that. Having access to the code helps with this. When you get audited by potential customer, it usually involves not having code access and trying to penetrate the app without that access.

BinaryIdiot 10y ago
> When you get audited by potential customer, it usually involves not having code access and trying to penetrate the app without that access. Is this in reference to on-prem / enterprise software and is this typical? I haven't heard of customers doing this but it certainly makes sense (might as well invest thousands to test before spending magnitudes more on the product itself only to find it having a huge security hole). Then again I'm not sure I've worked with potential customers who have access to do something like that.
- softawre 10y ago
  We just signed a big deal with a Google subsidiary, and part of that deal required us to go through a third party penetration test (no code access).
  - BinaryIdiot 10y ago
    Well today I learned. Thanks!

dsacco 10y ago

Good question. It can be all or none of the above. Here's what happens at a high level: Once a company decides it needs a security assessment performed on an application, it engages with a consulting firm. Consulting firms generally offer a variety of services, from web and mobile application penetration tests, to cryptanalysis (implementation and design), to reverse engineering and binary penetration testing, with source code audits sprinkled throughout (or as standalone assessments). Let's assume they move forward with a web application assessment. The company decides if it wants a source code audit, a penetration test or both. The most comprehensive assessments will include source code and unmitigated access to a staging environment that the consultants do not have to worry about destroying. However, they could also decide they don't want to hand over the code (common in things like sensitive financial applications or in applications with protective developers). I've worked on many assessments where I had no source code - this is called a "black-box" assessment. Conversely, an assessment might consist of a source code audit with no penetration test! This is less common, but it's particularly suited for engagements where the developers are fairly sure they've eliminated the most common issues and they are really focused on obscure errors, logic flaws and race conditions. It really depends on the type of security audit. You can have more exotic ones, like black-box cryptanalysis where a company hands Riscure a proprietary payment mechanism and there is heavy reverse engineering and side channel analysis. It can also be very vanilla, like the web application penetration tests that bug bounty programs attempt to simulate. Companies decide what they are going to do based on their application's profile and their goals. Putting this all together, these are the stages of a traditional security audit from a high-quality firm: Step 1: A company receives several proposals and decides which company to move forward with based on which statement of work most closely matches their security goals, timing, budget and desired expertise. Then they decide on a start date. Step 2: Representatives from the company (generally a technical manager, a security engineer or manager if the company has one and at least one developer) have a conference call with representatives from the security firm (generally, the security consultants performing the assessment, an account executive and a technical manager) to "kick off" the assessment with technical and logistical engagement planning. Things like "How will we access the staging environment?" and "Is there anything off-limits" are fleshed out here as well as reminders about scope and scheduling. Step 3: Things like source code, infrastructure/application/API documentation, PGP keys, etc. are securely exchanged and verified. This comes out of a list of mutual action items from the kick-off call. Step 4: The actual assessment happens, generally in a period of one to three weeks. I've never been involved in an assessment less than one week long, and assessments longer than four weeks usually need to re-scope or they become monolithic and difficult to coordinate. Progress reports with findings and testing data are securely sent to the company from the security firm. Step 5: The assessment is finished and a final deliverable is securely sent to the company from the security firm. An optional re-test assessment might happen a few weeks or months later to confirm if the findings have been satisfactorily resolved. This is based on my knowledge of having worked in security consultancies, engaging with them as an in-house security engineer and running my own consulting firm.

opinion1 10y ago

Yes. The way it works is that two smart hackers go into the office each day and spend eight hours trying to think of as many creative ways of tapping on the target application as possible. Nothing is off-limits except what is agreed up front, but you're obviously expected not to interfere with production operations. The client is generally expected to set up a testing environment substantially similar to production, but usually consultants just have to muddle through with whatever the clients give, which may not (and usually isn't) populated with production data. As long as consultants are given the ability to enter in data themselves, i.e. admin accounts, this is fine, because data entry is their job. Hacking is basically large-scale data entry, and it's as boring as it sounds: very tedious, interspersed with excitement when you see an XSS popup window or figure out a clever way to get a reverse shell. If after two weeks of this you found no medium-or-higher security vulnerabilities, you were generally considered to not be doing a very good job. The secret of the industry is that at the end of this process, you are deemed secure. That's the point of the security audit. But if it's not a repeating process, it doesn't work. It may work for that particular version of the application, and it may substantially improve the security in that old vulnerabilities are found and fixed. Let me abandon this train of thought and put it another way: This post is a press release saying that phpMyAdmin is secure. But that's not how this works. High-severity vulnerabilities are often found near the end of an audit. This is because the consultants have had time to become intimately familiar with the application. But the late stages of an audit are exactly when the consultant's time is mostly spent writing reports for the existing findings, and not doing pentesting. This means that two weeks is often just long enough to start finding serious vulns, since week one can be devoted to pentesting and week two is mostly reporting from Tuesday onward. But that "mostly reporting" process gets the consultant thinking about the application as they're doing the writeups, which -- you guessed it -- leads to realizing that there's something clever they could try. And when they try that clever thing, sometimes it yields a high-severity vuln. It's the opposite of a mechanical, thoughtless process. That means your results will vary depending on who, specifically, is doing the auditing. If you run your application through the consulting process twice -- same version, same staging data, same everything -- it's likely that you'll get wildly different results, because the pentesters are different people. It has to be an on-going process in order to be effective. And it can be highly effective. It just costs so much that only the most massive companies can afford this. That's not to say this audit wasn't effective. It's possible that whoever did the audit found substantially everything. But it was interesting to discover how often this was not the case, in a "How'd they miss this last time?" sort of way.

pg_is_a_butt 10y ago

yes. i had to do a security audit with NCC Group before... they need all your source code, all your build processes, all your environment setup scripts... basically everything. imagine training a whole new team now to build and maintain your company from scratch. then they tell you everything you already knew but management told you not to bother fixing. then management yells at you for not fixing it earlier. then everyone is fired and NCC Group takes over.

jquast 10y ago

The first few chapters of the book, "The Art of Software Security Assessment: Identifying and Preventing Software Vulnerabilities" outlines a very meticulous process of reviewing source code for vulnerabilities in a professional manner.