6 ms·

CloudFlare vs Incapsula vs ModSecurity - Comparative penetration testing [pdf]

64 points by dsl 14y ago

antr 14y ago
Page 18: "Though CloudFlare is presented as, besides other things, a very proficient web application firewall, we concluded that that’s just a marketing sales point and nothing more. ... CloudFlare does NOT protect from web attacks!" I'd really like to hear from Cloudfare on this statement.
- dsl 14y ago
  Me too. They even upgraded to the Pro account which is supposed to include advanced protection.
- druiid 14y ago
  As a Cloudflare user whom has been protected by these kinds of attacks in the past... I too would like to know what the eff is going on here. They supposedly (according to tech support in the past) USE mod_security... so what did they break?
  - driverdan 14y ago
    You were protected from DDoS attacks or security attacks?
    - druiid 14y ago
      Both. We were victims of up to something like 21GBps DDoS attacks (sorry to any other clients on that segment of the Cloudflare network... whistles and walks away :P). We've also seen directed attempts from toolkits and the like for SQL injection attempts, etc.
- bluetidepro 14y ago
  See my comment above, it has a few notes from their co-founder about this.
corry 14y ago
Anyone else tried testing the security claims of CF? This particular test looks pretty bad but it's just 1 case...
- JimWestergren 14y ago
  Here is another test (October 2012) with similar findings: http://tonyonsecurity.com/2012/11/13/protecting-your-website-cloudflare-or-incapsula/ http://tonyonsecurity.com/2012/11/13/protecting-your-website... SQL Injection: Incapsula blocked all 30 attacks and CloudFlare blocked 1 XSS: Incapsula blocked all 12 attacks and CloudFlare blocked 0 RFI: Incapsula blocked 0 and CloudFlare blocked 0 I really like CloudFlare and hope they can improve this.
JimWestergren 14y ago
I am getting a "Invalid Http Request Header" when I try to reach incapsula.com with Firefox 19. Perhaps because I am currently located in Bolivia. A bad false positive and they lost a potential client.
bluetidepro 14y ago
I tweeted CloudFlare asking for a statement about this, as a few of us commenters have asked for (and wanted), and this was the response from their co-founder... > " @bluetidepro @CloudFlare flawed testing strategy that assumes a last-generation, rules-based WAF. We've always worked differently. " https://twitter.com/eastdakota/status/307604650436210688 https://twitter.com/eastdakota/status/307604650436210688 > " @bluetidepro @CloudFlare that said, we'll have a significant release soon that allows the people who want a rules based WAF to have it. " https://twitter.com/eastdakota/status/307604791054434304 https://twitter.com/eastdakota/status/307604791054434304 > " @bluetidepro @CloudFlare but it's kinda silly because it's certainly not the future. Feels like taking a step backward. " https://twitter.com/eastdakota/status/307604919605661698 https://twitter.com/eastdakota/status/307604919605661698 (My tweet: Hope @CloudFlare can make a statement about this: http://zeroscience.mk/files/wafreport2013.pdf http://zeroscience.mk/files/wafreport2013.pdf - I'm pretty annoyed as a paying customer.) https://twitter.com/bluetidepro/status/307603272645111808 https://twitter.com/bluetidepro/status/307603272645111808
- druiid 14y ago
  Wait, what? That is, I'm sorry... but a stupid response on their part. A SQL injection attempt (successful or not) is an SQL injection attempt. If it makes it to the server how is anything the 'future' or 'working differently', it still made it to the server and the service isn't doing the job one is paying for. This response is so bad that I just did a literal facepalm. Needless to say we will be getting in touch with them as well...
  - deleted 14y ago
    [deleted]
david_shaw 14y ago
I think it's worth mentioning that CloudFlare, while advertising its WAF as a "pro" feature[1], gives it a one-paragraph blurb halfway down its feature page. Don't get me wrong--I'm not saying that it's okay to fudge features--but CloudFlare is not primarily marketed as a WAF. Among other security features, CloudFlare is designed to load balance and protect sites from DDoS attacks, which it does well (though, like any service, not perfectly). Additionally, it looks like they assign visitors "reputation scores" that can have different effects based on the status of the application. They also seem to use heuristics to "learn" about new attacks, which while not the fastest way to block attacks can produce some alright results.[2] I'd be interested to see whether the Zero Science Lab study used CloudFlare's feature to allow them to manage mod_security rules or not, and whether that would have made a difference. The PDF does state: > Remember, we are using their Business Plan which should be an enterprise WAF solution for your company. ... but it's unclear which features they enabled. Last point: I've done a lot of web application security assessments, and while, yes, WAFs can help an organization defend against XSS, SQLi, LFI/RFI, CSRF, etc., they are really just bandages over the much more serious issue of insecure code. Developers have a tough job, sure, but there is no WAF that will protect you against all attacks. Maybe ' OR 1=1;-- won't work, but there are a lot of ways--even emerging attack vectors--that security software won't protect against. That said, though, if you're going to rely heavily on a WAF, going with a "sub-feature" like those presented in CloudFlare is definitely not the way to do it. -- As I finished up this comment, I realized that bluetidepro posted some comments from CloudFlare about the report: > @CloudFlare flawed testing strategy that assumes a last-generation, rules-based WAF. We've always worked differently. [3] This statement in particular furthers their own marketing; they're trying to create security via heuristic rules, rather than a static ruleset. Really, a combination would probably be best. -- Lastly: Although this comment might read as a defense of CloudFlare, I'd like to note that I have no affiliation with the organization whatsoever; it just seemed like a very one-sided argument. [1] https://www.cloudflare.com/features-security https://www.cloudflare.com/features-security [2] http://blog.cloudflare.com/thats-freaking-awesome-cloudflare-automatical http://blog.cloudflare.com/thats-freaking-awesome-cloudflare... [3] https://twitter.com/eastdakota/status/307604650436210688 https://twitter.com/eastdakota/status/307604650436210688
- dsl 14y ago
newman314 14y ago
Waiting to see if eastdakota or jgrahamc will chime in...
drakaal 14y ago
CloudFlare does about as much for protecting you from web attacks as a condom does in protecting you from a rabid pitt bull. In fact it might even make things worse, since CF can be tricked in to refreshing its cache of pages pretty easily, making it easier to create a DoS. Also with very little work you can trick CF in to thinking a site is under attack from Google's IP range which will trigger it to block Google, seriously hurting your search rankins.

64 points by dsl 14y ago

antr 14y ago

Page 18: "Though CloudFlare is presented as, besides other things, a very proficient web application firewall, we concluded that that’s just a marketing sales point and nothing more. ... CloudFlare does NOT protect from web attacks!" I'd really like to hear from Cloudfare on this statement.

dsl 14y ago
Me too. They even upgraded to the Pro account which is supposed to include advanced protection.
druiid 14y ago
As a Cloudflare user whom has been protected by these kinds of attacks in the past... I too would like to know what the eff is going on here. They supposedly (according to tech support in the past) USE mod_security... so what did they break?
- driverdan 14y ago
  You were protected from DDoS attacks or security attacks?
  - druiid 14y ago
    Both. We were victims of up to something like 21GBps DDoS attacks (sorry to any other clients on that segment of the Cloudflare network... whistles and walks away :P). We've also seen directed attempts from toolkits and the like for SQL injection attempts, etc.
bluetidepro 14y ago
See my comment above, it has a few notes from their co-founder about this.

corry 14y ago

Anyone else tried testing the security claims of CF? This particular test looks pretty bad but it's just 1 case...

JimWestergren 14y ago
Here is another test (October 2012) with similar findings: http://tonyonsecurity.com/2012/11/13/protecting-your-website-cloudflare-or-incapsula/ http://tonyonsecurity.com/2012/11/13/protecting-your-website... SQL Injection: Incapsula blocked all 30 attacks and CloudFlare blocked 1 XSS: Incapsula blocked all 12 attacks and CloudFlare blocked 0 RFI: Incapsula blocked 0 and CloudFlare blocked 0 I really like CloudFlare and hope they can improve this.

JimWestergren 14y ago

I am getting a "Invalid Http Request Header" when I try to reach incapsula.com with Firefox 19. Perhaps because I am currently located in Bolivia. A bad false positive and they lost a potential client.

bluetidepro 14y ago

I tweeted CloudFlare asking for a statement about this, as a few of us commenters have asked for (and wanted), and this was the response from their co-founder... > " @bluetidepro @CloudFlare flawed testing strategy that assumes a last-generation, rules-based WAF. We've always worked differently. " https://twitter.com/eastdakota/status/307604650436210688 https://twitter.com/eastdakota/status/307604650436210688 > " @bluetidepro @CloudFlare that said, we'll have a significant release soon that allows the people who want a rules based WAF to have it. " https://twitter.com/eastdakota/status/307604791054434304 https://twitter.com/eastdakota/status/307604791054434304 > " @bluetidepro @CloudFlare but it's kinda silly because it's certainly not the future. Feels like taking a step backward. " https://twitter.com/eastdakota/status/307604919605661698 https://twitter.com/eastdakota/status/307604919605661698 (My tweet: Hope @CloudFlare can make a statement about this: http://zeroscience.mk/files/wafreport2013.pdf http://zeroscience.mk/files/wafreport2013.pdf - I'm pretty annoyed as a paying customer.) https://twitter.com/bluetidepro/status/307603272645111808 https://twitter.com/bluetidepro/status/307603272645111808

druiid 14y ago
Wait, what? That is, I'm sorry... but a stupid response on their part. A SQL injection attempt (successful or not) is an SQL injection attempt. If it makes it to the server how is anything the 'future' or 'working differently', it still made it to the server and the service isn't doing the job one is paying for. This response is so bad that I just did a literal facepalm. Needless to say we will be getting in touch with them as well...
- deleted 14y ago
  [deleted]

david_shaw 14y ago

I think it's worth mentioning that CloudFlare, while advertising its WAF as a "pro" feature[1], gives it a one-paragraph blurb halfway down its feature page. Don't get me wrong--I'm not saying that it's okay to fudge features--but CloudFlare is not primarily marketed as a WAF. Among other security features, CloudFlare is designed to load balance and protect sites from DDoS attacks, which it does well (though, like any service, not perfectly). Additionally, it looks like they assign visitors "reputation scores" that can have different effects based on the status of the application. They also seem to use heuristics to "learn" about new attacks, which while not the fastest way to block attacks can produce some alright results.[2] I'd be interested to see whether the Zero Science Lab study used CloudFlare's feature to allow them to manage mod_security rules or not, and whether that would have made a difference. The PDF does state: > Remember, we are using their Business Plan which should be an enterprise WAF solution for your company. ... but it's unclear which features they enabled. Last point: I've done a lot of web application security assessments, and while, yes, WAFs can help an organization defend against XSS, SQLi, LFI/RFI, CSRF, etc., they are really just bandages over the much more serious issue of insecure code. Developers have a tough job, sure, but there is no WAF that will protect you against all attacks. Maybe ' OR 1=1;-- won't work, but there are a lot of ways--even emerging attack vectors--that security software won't protect against. That said, though, if you're going to rely heavily on a WAF, going with a "sub-feature" like those presented in CloudFlare is definitely not the way to do it. -- As I finished up this comment, I realized that bluetidepro posted some comments from CloudFlare about the report: > @CloudFlare flawed testing strategy that assumes a last-generation, rules-based WAF. We've always worked differently. [3] This statement in particular furthers their own marketing; they're trying to create security via heuristic rules, rather than a static ruleset. Really, a combination would probably be best. -- Lastly: Although this comment might read as a defense of CloudFlare, I'd like to note that I have no affiliation with the organization whatsoever; it just seemed like a very one-sided argument. [1] https://www.cloudflare.com/features-security https://www.cloudflare.com/features-security [2] http://blog.cloudflare.com/thats-freaking-awesome-cloudflare-automatical http://blog.cloudflare.com/thats-freaking-awesome-cloudflare... [3] https://twitter.com/eastdakota/status/307604650436210688 https://twitter.com/eastdakota/status/307604650436210688

dsl 14y ago

newman314 14y ago

Waiting to see if eastdakota or jgrahamc will chime in...

drakaal 14y ago

CloudFlare does about as much for protecting you from web attacks as a condom does in protecting you from a rabid pitt bull. In fact it might even make things worse, since CF can be tricked in to refreshing its cache of pages pretty easily, making it easier to create a DoS. Also with very little work you can trick CF in to thinking a site is under attack from Google's IP range which will trigger it to block Google, seriously hurting your search rankins.