4 ms·

If anyone is receiving these types of logs on AWS, please do yourself a favor and place AWS WAF in front of your VPC. It's not expensive and can significantly

If anyone is receiving these types of logs on AWS, please do yourself a favor and place AWS WAF in front of your VPC. It's not expensive and can significantly help you, saving you from many headaches in situations like this. While it might not block everything that arrives at your service, it can be a great help!

anamexis 3y ago
This is a good suggestion, but careful with the default rulesets. We turned on AWS WAF (in our case, the motivation was just SOC 2 compliance). There were a few overzealous rules that subtly broke parts of our app. There were request body rules that did things like block requests that contained "localhost" in the request body. There was also a rule that blocked requests without a User-Agent header, which we were not previously requiring on API requests, so we broke our entire API for a few users until we figured that out.
- everfrustrated 3y ago
  In my experience WAFs are not something that one should ever "just turn on". Complete due diligence is required to fully understand and realise the impact of the rules and should be tested like any software change by going through a testing phase. Ideally software teams should be fully trained and be responsible for their lifecycle.
  - macNchz 3y ago
    Yes you need to be familiar with the rulesets being applied, and prepared to closely monitor what is being blocked. Ideally I think I’d roll it out one ruleset at a time to limit the number of potential issues being introduced at once. Had a fun one after turning on the AWS WAF with some default rules–a small number of users reported they couldn’t upload new logo images anymore. Turned out some Adobe product was adding XML metadata to the image files, which the WAF picked up and blocked.
  - marcosdumay 3y ago
    Just to add, but only testing will never work well enough for something like this. This is one of the cases where you must understand what you are doing. There's no technique for doing it mindlessnessly.
  - arter4 3y ago
    Agree. There are so many ways WAF rules can unintentionally block legitimate traffic. From very long URLs (is that a DoS attempt?), to special characters in a POST with a file upload (is that = part of a SQL injection attempt or is that just part of a base64 encoded file?) and so on.
- nullindividual 3y ago
  I use the Azure equivalent of the AWS WAF but I have no direct experience with AWS WAF. Azure WAF leverages the OWASP ruleset[0] and many of those rules throw false-positives, SQL-related rules being one of the top offenders. As you note, it requires adjustment due to overzealous rules. OWASP has Paranoia Levels[1] which allow you to be more targeted. [0] https://github.com/coreruleset/coreruleset https://github.com/coreruleset/coreruleset [1] https://coreruleset.org/20211028/working-with-paranoia-levels/ https://coreruleset.org/20211028/working-with-paranoia-level...
bogota 3y ago
It is not that easy as using the AWS WAF with default rules for our application led to many valid requests and IPs being blocked. You need to know what is being blocked and verify at first or you will in some cases be losing customers.
- CubsFan1060 3y ago
  Your best plan is to start with all the rules in count mode. Let that sit for awhile and analyze anything that was counted. As you feel good about it, slowly start to move things into block.
  - mango7283 3y ago
    Problem is when you get owned during that window and get slammed for not blocking sooner. Then you block sooner and get slammed for blocking too soon. Repeat for 1000 web services.
    - CubsFan1060 3y ago
      Yeah. Sadly the WAF can't solve organizational problems.
fabian2k 3y ago
None of the attacks listed in the post would be an issue for any kind of modern web application. Why should I add a WAF for this?
remram 3y ago
WAF tends to ban widely, sometimes for dubious reasons. For example, researchers at my university study Twitter data, and the mere fact of following links from a small random sample of tweets means that our university's IPs are blocked by most WAF.
mac-chaffee 3y ago
In actuality, WAFs hurt more than help. They give a false sense of security since they are so easily bypassable, plus they have a significant performance cost and a significant chance of blocking legitimate traffic: https://www.macchaffee.com/blog/2023/wafs/ https://www.macchaffee.com/blog/2023/wafs/

anamexis 3y ago

This is a good suggestion, but careful with the default rulesets. We turned on AWS WAF (in our case, the motivation was just SOC 2 compliance). There were a few overzealous rules that subtly broke parts of our app. There were request body rules that did things like block requests that contained "localhost" in the request body. There was also a rule that blocked requests without a User-Agent header, which we were not previously requiring on API requests, so we broke our entire API for a few users until we figured that out.

everfrustrated 3y ago
In my experience WAFs are not something that one should ever "just turn on". Complete due diligence is required to fully understand and realise the impact of the rules and should be tested like any software change by going through a testing phase. Ideally software teams should be fully trained and be responsible for their lifecycle.
- macNchz 3y ago
  Yes you need to be familiar with the rulesets being applied, and prepared to closely monitor what is being blocked. Ideally I think I’d roll it out one ruleset at a time to limit the number of potential issues being introduced at once. Had a fun one after turning on the AWS WAF with some default rules–a small number of users reported they couldn’t upload new logo images anymore. Turned out some Adobe product was adding XML metadata to the image files, which the WAF picked up and blocked.
- marcosdumay 3y ago
  Just to add, but only testing will never work well enough for something like this. This is one of the cases where you must understand what you are doing. There's no technique for doing it mindlessnessly.
- arter4 3y ago
  Agree. There are so many ways WAF rules can unintentionally block legitimate traffic. From very long URLs (is that a DoS attempt?), to special characters in a POST with a file upload (is that = part of a SQL injection attempt or is that just part of a base64 encoded file?) and so on.
nullindividual 3y ago
I use the Azure equivalent of the AWS WAF but I have no direct experience with AWS WAF. Azure WAF leverages the OWASP ruleset[0] and many of those rules throw false-positives, SQL-related rules being one of the top offenders. As you note, it requires adjustment due to overzealous rules. OWASP has Paranoia Levels[1] which allow you to be more targeted. [0] https://github.com/coreruleset/coreruleset https://github.com/coreruleset/coreruleset [1] https://coreruleset.org/20211028/working-with-paranoia-levels/ https://coreruleset.org/20211028/working-with-paranoia-level...

bogota 3y ago

It is not that easy as using the AWS WAF with default rules for our application led to many valid requests and IPs being blocked. You need to know what is being blocked and verify at first or you will in some cases be losing customers.

CubsFan1060 3y ago
Your best plan is to start with all the rules in count mode. Let that sit for awhile and analyze anything that was counted. As you feel good about it, slowly start to move things into block.
- mango7283 3y ago
  Problem is when you get owned during that window and get slammed for not blocking sooner. Then you block sooner and get slammed for blocking too soon. Repeat for 1000 web services.
  - CubsFan1060 3y ago
    Yeah. Sadly the WAF can't solve organizational problems.

fabian2k 3y ago

None of the attacks listed in the post would be an issue for any kind of modern web application. Why should I add a WAF for this?

remram 3y ago

WAF tends to ban widely, sometimes for dubious reasons. For example, researchers at my university study Twitter data, and the mere fact of following links from a small random sample of tweets means that our university's IPs are blocked by most WAF.

mac-chaffee 3y ago

In actuality, WAFs hurt more than help. They give a false sense of security since they are so easily bypassable, plus they have a significant performance cost and a significant chance of blocking legitimate traffic: https://www.macchaffee.com/blog/2023/wafs/ https://www.macchaffee.com/blog/2023/wafs/