4 ms·

Could you automate this? Could you have some scripts running that automatically try various attacks on sites, present the results, or does each attack have to

Could you automate this? Could you have some scripts running that automatically try various attacks on sites, present the results, or does each attack have to be hand crafted?

arcwhite 10y ago
In my experience automated tools are generally fairly rubbish for this kind of work... which is not to say it's impossible, just that you'd want some real smarts in said tool
dsacco 10y ago
You could certainly try, but no off the shelf software, commercial or open source, could fully replicate what the author did. It's sort of the golden dream of the security world to fully automate this work to intelligent scanners like Acunetix or Fortify. When doing this work I personally find it helpful to frame vulnerabilities as areas in development where an engineer's mental model and design expectations were inadequate (or simply flawed). This is in contrast to the way most automated security scanning software implicitly understands vulnerabilities as lists of things that are known to go wrong. Both are helpful perspectives, but the reason why humans are much better at the actual work is because the former encompasses the latter without missing the more abstract issues. This is all a long-winded way of saying that in order to do this, you'd probably need to design an excellent machine learning system, preferably with supervised learning, and train it to classify vulnerabilities in source code and applications by running through the same things successful security engineers do. Depending on your definition of automated this would probably start to border on AGI in order to be really effective at fully removing all the schlep work. If I had to guess, I'd say the first company that will develop anything realistically close to this will be Hackerone or Bugcrowd. They are the companies with the most data, and they have already started working on machine learning systems that can classify bug bounty reports as likely signal or likely noise in production. I could even see a very long term Uber-esque plan where they attempt to replace their bug bounty participants with systems developed on their training data.
yeukhon 10y ago
Unfortunately, I will say no. Most blackbox and greybox testing are very manual. The reason is because we know what we are testing. Context matter. Computers don't understand /signup and /register are common page for user account registration. Automation tools thus far are very primitive. Let's imagine a user registration process. It seems simple, but really not scalable. What can you write? Some examples: * check HTTP security headers * check page and requests are made over HTTPS * try to verify CSRF token is working * try to fuzz input fields (for all kinds of injection attacks) First two automation are pretty simple to implement. CSRF validation automation is quite common and also not terribly hard to implement (if you use commonly used web framework the implementation should be fairly standard). But testing injection? There's reflected and there's stored. Stored can be quite tough. Supposed you finish your registration successfully, you either see stored XSS immediately from within your profile page, or you will see the stored XSS somewhere on the web site. In the latter case, you are waiting. Where do you go? You can pick a few hot spots (forum page, profile page, email, etc), but website A is different from website B, how can you write a tool that's scalable? Effectively, you are now writing a tool specifically for one website. Now move on something more interesting. A lot of the vulnerabilities depend on state. By state I mean not only session, but the path of how the user gets to that page. Going back to stored XSS, yes, the input is in the database, but the user may only see this XSS if and only if he was on this page performing this action. IMO, recording and learning how a real human would use the website is one of the ways to automate and to attack a website because your goal is to generate large set of test cases. If you are lucky you can hit a jackpot and reproduce a real vulnerability that someone is actually using against you. Change the input, replay the whole scenario, and repeat the cycle. Want to be smarter, not dumb fuzzer? Add some rules. Another way to replay is by leveraging your tests. For example, let say my API can create objects, well, first I need to register myself, then I need to authenticate myself, and I need to fill in a form with my authorization token or whatever. My tests would be making multiple API calls. What if now I do a permutation? Add/remove/modify one step at a time? Swap the steps around? Privilege escalation, maybe? Anything? Using your functional tests you can now generate new test cases and it is very possible you will find real vulnerability that way. There are other complications. Take gmail.com. The source is obfuscated and minified. Try to make sense of the source is a challenge. DOM can change, so now you have to update your script. The incredible Gmail.js made this easier, but still require author to maintain the ever-changing DOM. Really, the many little steps you take to perform penetration test can be automated. But there are too many steps and the number of states could overwhelm. Writing generic tool is very much a hard problem. In a way, I guess you can, creating an effective, kind of smart penetration automation software is like creating a model for determining which Spotify's Discovery recommendation algorithm is better. Very data driven and evolutionary-based.

Could you automate this? Could you have some scripts running that automatically try various attacks on sites, present the results, or does each attack have to be hand crafted?

arcwhite 10y ago

In my experience automated tools are generally fairly rubbish for this kind of work... which is not to say it's impossible, just that you'd want some real smarts in said tool

dsacco 10y ago

You could certainly try, but no off the shelf software, commercial or open source, could fully replicate what the author did. It's sort of the golden dream of the security world to fully automate this work to intelligent scanners like Acunetix or Fortify. When doing this work I personally find it helpful to frame vulnerabilities as areas in development where an engineer's mental model and design expectations were inadequate (or simply flawed). This is in contrast to the way most automated security scanning software implicitly understands vulnerabilities as lists of things that are known to go wrong. Both are helpful perspectives, but the reason why humans are much better at the actual work is because the former encompasses the latter without missing the more abstract issues. This is all a long-winded way of saying that in order to do this, you'd probably need to design an excellent machine learning system, preferably with supervised learning, and train it to classify vulnerabilities in source code and applications by running through the same things successful security engineers do. Depending on your definition of automated this would probably start to border on AGI in order to be really effective at fully removing all the schlep work. If I had to guess, I'd say the first company that will develop anything realistically close to this will be Hackerone or Bugcrowd. They are the companies with the most data, and they have already started working on machine learning systems that can classify bug bounty reports as likely signal or likely noise in production. I could even see a very long term Uber-esque plan where they attempt to replace their bug bounty participants with systems developed on their training data.

yeukhon 10y ago

Unfortunately, I will say no. Most blackbox and greybox testing are very manual. The reason is because we know what we are testing. Context matter. Computers don't understand /signup and /register are common page for user account registration. Automation tools thus far are very primitive. Let's imagine a user registration process. It seems simple, but really not scalable. What can you write? Some examples: * check HTTP security headers * check page and requests are made over HTTPS * try to verify CSRF token is working * try to fuzz input fields (for all kinds of injection attacks) First two automation are pretty simple to implement. CSRF validation automation is quite common and also not terribly hard to implement (if you use commonly used web framework the implementation should be fairly standard). But testing injection? There's reflected and there's stored. Stored can be quite tough. Supposed you finish your registration successfully, you either see stored XSS immediately from within your profile page, or you will see the stored XSS somewhere on the web site. In the latter case, you are waiting. Where do you go? You can pick a few hot spots (forum page, profile page, email, etc), but website A is different from website B, how can you write a tool that's scalable? Effectively, you are now writing a tool specifically for one website. Now move on something more interesting. A lot of the vulnerabilities depend on state. By state I mean not only session, but the path of how the user gets to that page. Going back to stored XSS, yes, the input is in the database, but the user may only see this XSS if and only if he was on this page performing this action. IMO, recording and learning how a real human would use the website is one of the ways to automate and to attack a website because your goal is to generate large set of test cases. If you are lucky you can hit a jackpot and reproduce a real vulnerability that someone is actually using against you. Change the input, replay the whole scenario, and repeat the cycle. Want to be smarter, not dumb fuzzer? Add some rules. Another way to replay is by leveraging your tests. For example, let say my API can create objects, well, first I need to register myself, then I need to authenticate myself, and I need to fill in a form with my authorization token or whatever. My tests would be making multiple API calls. What if now I do a permutation? Add/remove/modify one step at a time? Swap the steps around? Privilege escalation, maybe? Anything? Using your functional tests you can now generate new test cases and it is very possible you will find real vulnerability that way. There are other complications. Take gmail.com. The source is obfuscated and minified. Try to make sense of the source is a challenge. DOM can change, so now you have to update your script. The incredible Gmail.js made this easier, but still require author to maintain the ever-changing DOM. Really, the many little steps you take to perform penetration test can be automated. But there are too many steps and the number of states could overwhelm. Writing generic tool is very much a hard problem. In a way, I guess you can, creating an effective, kind of smart penetration automation software is like creating a model for determining which Spotify's Discovery recommendation algorithm is better. Very data driven and evolutionary-based.