5 ms·

Not all attacks are equal: understanding and preventing DoS in web applications

46 points by ievans 6y ago

js4ever 6y ago
Summary: add a rate limiter to your api endpoints to block DoS, and a cdn like cloudflare to block DDOS
daghan 6y ago
You can use flask_limiter library for flask
lwansbrough 6y ago
Our business operates in a market that has a lot of wannabe hacker types: bored teenagers in the gaming community with the minimum knowledge required to order a DDOS. As such, we're very very careful about never leaking our IPs anywhere. This has been tricky at times but it's well worth it. Cloudflare has made our lives a lot easier as it handles the bulk of the requests. Still, a lot of what it takes to be DDOS resilient is just being able to scale to high concurrency very quickly: ability to cache everything at every level, read-only mode, CDN for everything including HTML when possible, API gateways and self imposed rate limiting to third party services. With that said, there's not much you can do if your IP leaks. Some often overlooked areas for IP leaks: user generated content (your server makes a request to a URL specified by the user), or even sending an email (even many services provide the origin sender IP - such as SendGrid - as a matter of policy, so we don't use those services.) Still to do: developing our own online ML based monitor which can detect abnormal user behaviour before it becomes a problem (ie. aggressive HTML scrapers, burp suite scans, etc.)
- necovek 6y ago
  That is indeed an interesting observation. I am not a network engineer so take all of the below with a handful of salt :) Can't you drop all incoming traffic on "protected" IPs other than that coming through known-trusted networks like CDN's one? Your networking equipment still needs to be able to handle that load as well, which is not a given. I guess DDoS attacker could fake the IP packet headers to appear as if they are coming from Cloudflare (in your case), though even then there'd be some obscurity in figuring what subnet you are allowing through. I guess CDN providers could provide a secret-key-seeded time-based (like TOTP) source network, and your firewall would let packets in by that. The benefit of this is that it is cheaper than doing full packet encryption, and probably much easier to implement in hardware/software.
- rovr138 6y ago
  >Still to do: developing our own online ML based monitor which can detect abnormal user behaviour before it becomes a problem (ie. aggressive HTML scrapers, burp suite scans, etc.) Search first. I would try something like, detect irregular OR abnormal behavior internet OR requests OR databases There are models for this and papers around it.
  - lwansbrough 6y ago
    The plan I have is to use an isolation forest with dimensions like requests per minute, unique pages per session, etc. and just present a recaptcha or inaccurate data when they're outside some threshold.
    - rovr138 6y ago
      That reminded me of this, https://blog.cloudflare.com/cleaning-up-bad-bots/ https://blog.cloudflare.com/cleaning-up-bad-bots/ Not sure if you had seen it. I like the approach. It'll still return a 200, but just slow them down. In a botnet, that's a pretty interesting approach instead of returning a captcha so they'll keep using the IP. Something to consider...
blackflame7000 6y ago
Some of the best DDOS attacks work by sending data as character by character as slow as possible before the connection times out so as to mimic extremely slow connections and consume handler threads. Also HTTP1.1 downgrade attacks can force the usage of more connections especially if keep-alive is false.
netsectoday 6y ago
This is very well written and practical. It's exactly how you think about security when you own a company. I'm guessing the author is/was holding a large amount of stock in Heroku/Salesforce. Summary: 1. Don't publicly expose endpoints that are either slow or require a ton of resources. 2. If you can't get rid of a slow endpoint; put authentication in front of it so you have a lever to pull in the event of an attack. 3. Throttle / rate limit everything with high barriers so not to impede normal traffic patterns. 4. Don't make it easy for someone to DoS you; reduce or eliminate well-know attack vectors and vulnerabilities. 5. Scan your app for regex and zip bombs. 6. The bad guys will sniff-out your N+1 queries, so fix them. 7. If necessary: pay for DDoS mitigation from a cloud provider.
- blackflame7000 6y ago
  8. Set timeouts for the maximum time a client can take to send a request. For Example, the attacker will send a GET request using many small packets with a large time gap between them. They do this by first sending the “G” in its own packet, after a while they will send the “E” and after some more time the “T” and so on. While this is a legitimate behavior according to the HTTP and TCP protocols, this behavior is consuming a lot of resources from the server – it has to keep the connection open, waiting for the full request to arrive. Since the connection pool is limited, it is very easy to reach pool saturation, with very little traffic.
  - netsectoday 6y ago
    https://en.wikipedia.org/wiki/Slowloris_(computer_security) https://en.wikipedia.org/wiki/Slowloris_(computer_security) "There are ways to mitigate or reduce the impact of such an attack. In general, these involve increasing the maximum number of clients the server will allow, limiting the number of connections a single IP address is allowed to make, imposing restrictions on the minimum transfer speed a connection is allowed to have, and restricting the length of time a client is allowed to stay connected."
    - blackflame7000 6y ago

46 points by ievans 6y ago

js4ever 6y ago

Summary: add a rate limiter to your api endpoints to block DoS, and a cdn like cloudflare to block DDOS

daghan 6y ago

You can use flask_limiter library for flask

lwansbrough 6y ago

Our business operates in a market that has a lot of wannabe hacker types: bored teenagers in the gaming community with the minimum knowledge required to order a DDOS. As such, we're very very careful about never leaking our IPs anywhere. This has been tricky at times but it's well worth it. Cloudflare has made our lives a lot easier as it handles the bulk of the requests. Still, a lot of what it takes to be DDOS resilient is just being able to scale to high concurrency very quickly: ability to cache everything at every level, read-only mode, CDN for everything including HTML when possible, API gateways and self imposed rate limiting to third party services. With that said, there's not much you can do if your IP leaks. Some often overlooked areas for IP leaks: user generated content (your server makes a request to a URL specified by the user), or even sending an email (even many services provide the origin sender IP - such as SendGrid - as a matter of policy, so we don't use those services.) Still to do: developing our own online ML based monitor which can detect abnormal user behaviour before it becomes a problem (ie. aggressive HTML scrapers, burp suite scans, etc.)

necovek 6y ago
That is indeed an interesting observation. I am not a network engineer so take all of the below with a handful of salt :) Can't you drop all incoming traffic on "protected" IPs other than that coming through known-trusted networks like CDN's one? Your networking equipment still needs to be able to handle that load as well, which is not a given. I guess DDoS attacker could fake the IP packet headers to appear as if they are coming from Cloudflare (in your case), though even then there'd be some obscurity in figuring what subnet you are allowing through. I guess CDN providers could provide a secret-key-seeded time-based (like TOTP) source network, and your firewall would let packets in by that. The benefit of this is that it is cheaper than doing full packet encryption, and probably much easier to implement in hardware/software.
rovr138 6y ago
>Still to do: developing our own online ML based monitor which can detect abnormal user behaviour before it becomes a problem (ie. aggressive HTML scrapers, burp suite scans, etc.) Search first. I would try something like, detect irregular OR abnormal behavior internet OR requests OR databases There are models for this and papers around it.
- lwansbrough 6y ago
  The plan I have is to use an isolation forest with dimensions like requests per minute, unique pages per session, etc. and just present a recaptcha or inaccurate data when they're outside some threshold.
  - rovr138 6y ago
    That reminded me of this, https://blog.cloudflare.com/cleaning-up-bad-bots/ https://blog.cloudflare.com/cleaning-up-bad-bots/ Not sure if you had seen it. I like the approach. It'll still return a 200, but just slow them down. In a botnet, that's a pretty interesting approach instead of returning a captcha so they'll keep using the IP. Something to consider...

blackflame7000 6y ago

Some of the best DDOS attacks work by sending data as character by character as slow as possible before the connection times out so as to mimic extremely slow connections and consume handler threads. Also HTTP1.1 downgrade attacks can force the usage of more connections especially if keep-alive is false.

netsectoday 6y ago

This is very well written and practical. It's exactly how you think about security when you own a company. I'm guessing the author is/was holding a large amount of stock in Heroku/Salesforce. Summary: 1. Don't publicly expose endpoints that are either slow or require a ton of resources. 2. If you can't get rid of a slow endpoint; put authentication in front of it so you have a lever to pull in the event of an attack. 3. Throttle / rate limit everything with high barriers so not to impede normal traffic patterns. 4. Don't make it easy for someone to DoS you; reduce or eliminate well-know attack vectors and vulnerabilities. 5. Scan your app for regex and zip bombs. 6. The bad guys will sniff-out your N+1 queries, so fix them. 7. If necessary: pay for DDoS mitigation from a cloud provider.

blackflame7000 6y ago
8. Set timeouts for the maximum time a client can take to send a request. For Example, the attacker will send a GET request using many small packets with a large time gap between them. They do this by first sending the “G” in its own packet, after a while they will send the “E” and after some more time the “T” and so on. While this is a legitimate behavior according to the HTTP and TCP protocols, this behavior is consuming a lot of resources from the server – it has to keep the connection open, waiting for the full request to arrive. Since the connection pool is limited, it is very easy to reach pool saturation, with very little traffic.
- netsectoday 6y ago
  https://en.wikipedia.org/wiki/Slowloris_(computer_security) https://en.wikipedia.org/wiki/Slowloris_(computer_security) "There are ways to mitigate or reduce the impact of such an attack. In general, these involve increasing the maximum number of clients the server will allow, limiting the number of connections a single IP address is allowed to make, imposing restrictions on the minimum transfer speed a connection is allowed to have, and restricting the length of time a client is allowed to stay connected."
  - blackflame7000 6y ago