4 ms·

Our business operates in a market that has a lot of wannabe hacker types: bored teenagers in the gaming community with the minimum knowledge required to order a

Our business operates in a market that has a lot of wannabe hacker types: bored teenagers in the gaming community with the minimum knowledge required to order a DDOS. As such, we're very very careful about never leaking our IPs anywhere. This has been tricky at times but it's well worth it. Cloudflare has made our lives a lot easier as it handles the bulk of the requests. Still, a lot of what it takes to be DDOS resilient is just being able to scale to high concurrency very quickly: ability to cache everything at every level, read-only mode, CDN for everything including HTML when possible, API gateways and self imposed rate limiting to third party services. With that said, there's not much you can do if your IP leaks. Some often overlooked areas for IP leaks: user generated content (your server makes a request to a URL specified by the user), or even sending an email (even many services provide the origin sender IP - such as SendGrid - as a matter of policy, so we don't use those services.) Still to do: developing our own online ML based monitor which can detect abnormal user behaviour before it becomes a problem (ie. aggressive HTML scrapers, burp suite scans, etc.)

necovek 6y ago
That is indeed an interesting observation. I am not a network engineer so take all of the below with a handful of salt :) Can't you drop all incoming traffic on "protected" IPs other than that coming through known-trusted networks like CDN's one? Your networking equipment still needs to be able to handle that load as well, which is not a given. I guess DDoS attacker could fake the IP packet headers to appear as if they are coming from Cloudflare (in your case), though even then there'd be some obscurity in figuring what subnet you are allowing through. I guess CDN providers could provide a secret-key-seeded time-based (like TOTP) source network, and your firewall would let packets in by that. The benefit of this is that it is cheaper than doing full packet encryption, and probably much easier to implement in hardware/software.
rovr138 6y ago
>Still to do: developing our own online ML based monitor which can detect abnormal user behaviour before it becomes a problem (ie. aggressive HTML scrapers, burp suite scans, etc.) Search first. I would try something like, detect irregular OR abnormal behavior internet OR requests OR databases There are models for this and papers around it.
- lwansbrough 6y ago
  The plan I have is to use an isolation forest with dimensions like requests per minute, unique pages per session, etc. and just present a recaptcha or inaccurate data when they're outside some threshold.
  - rovr138 6y ago
    That reminded me of this, https://blog.cloudflare.com/cleaning-up-bad-bots/ https://blog.cloudflare.com/cleaning-up-bad-bots/ Not sure if you had seen it. I like the approach. It'll still return a 200, but just slow them down. In a botnet, that's a pretty interesting approach instead of returning a captcha so they'll keep using the IP. Something to consider...

necovek 6y ago

That is indeed an interesting observation. I am not a network engineer so take all of the below with a handful of salt :) Can't you drop all incoming traffic on "protected" IPs other than that coming through known-trusted networks like CDN's one? Your networking equipment still needs to be able to handle that load as well, which is not a given. I guess DDoS attacker could fake the IP packet headers to appear as if they are coming from Cloudflare (in your case), though even then there'd be some obscurity in figuring what subnet you are allowing through. I guess CDN providers could provide a secret-key-seeded time-based (like TOTP) source network, and your firewall would let packets in by that. The benefit of this is that it is cheaper than doing full packet encryption, and probably much easier to implement in hardware/software.

rovr138 6y ago

>Still to do: developing our own online ML based monitor which can detect abnormal user behaviour before it becomes a problem (ie. aggressive HTML scrapers, burp suite scans, etc.) Search first. I would try something like, detect irregular OR abnormal behavior internet OR requests OR databases There are models for this and papers around it.

lwansbrough 6y ago
The plan I have is to use an isolation forest with dimensions like requests per minute, unique pages per session, etc. and just present a recaptcha or inaccurate data when they're outside some threshold.
- rovr138 6y ago
  That reminded me of this, https://blog.cloudflare.com/cleaning-up-bad-bots/ https://blog.cloudflare.com/cleaning-up-bad-bots/ Not sure if you had seen it. I like the approach. It'll still return a 200, but just slow them down. In a botnet, that's a pretty interesting approach instead of returning a captcha so they'll keep using the IP. Something to consider...