3 ms·

> * We're getting hit with a huge DDoS attack, repeatedly over 3 weeks, with no sign of stopping I've read the full blog post, I am not convinced it's a DDoS a

> * We're getting hit with a huge DDoS attack, repeatedly over 3 weeks, with no sign of stopping I've read the full blog post, I am not convinced it's a DDoS attack. Traffic patterns for web analytics will come from over the place and will look like a DDoS when it's not. For example, a customer misplacing their analytics in a JS loop and having a moderate traffic blog will generate billions of requests from all over the globe. Event tracking can be billions of requests as well by themselves. Never attribute to malice that which is adequately explained by simpler means. > * With zero access logs, there was no way to find patterns in the attack, and we had no way to block it It's a loss of time and energy. Your system should be able to handle these billions of requests. > * We were then able to identify a pattern and block the attack on Saturday Was it specific accounts? > * Without access logs (even redacted ones), this wasn't possible You can one-way hash the IP. So you can still look for pattern but you've lost the actual IP. And same one-way hash IP can block whichever IP seems devious in your firewall. (like md5 can be enough.) > But if you know a more privacy-focused way to block these attacks, I'm sure I'll buy you a few beers when we hang out. Haha no need to. But come say hi if you ever in Austin. julien _at_ serpapi.com.

JackWritesCode 6y ago
It was 100% a layer 7 DDoS, the attack was targeted and malicious. I can't say too much but the AWS team confirmed it. But I appreciate how something like you describe could happen. I like the idea of an MD5 hash. Although I'm not too certain why an IP would be a bad thing to log for 24 hours. From a privacy law perspective, the MD5 hash is considered PII. And if we see an IP address in an access log, we know that an IP visited one of the tens of thousands of websites Fathom runs on, but we don't know which one. Edit: Something else, with MD5 there's no way of finding patterns with the IPs, so you'd have to play whack a mole. Whereas raw IPs allow no real privacy invasion whilst allowing pattern detection
- hartator 6y ago
  > It was 100% a layer 7 DDoS, the attack was targeted and malicious. I can't say too much but the AWS team confirmed it. But I appreciate how something like you describe could happen. Layer 7 just means they are making ton of HTTP requests from lot of IPS however this is the nature of a web analytics. > we know that an IP visited one of the tens of thousands of websites Fathom runs on, but we don't know which one. How can the attackers know which websites have your analytics on? Crawling the web is super hard. You should be able to get back which accounts are affected from your analytics call: https://starman.fathomdns.com/?p=%2F&h=https%3A%2F%2Fusefathom.com&r=&sid=BIABKBRK&res=1440x900 Isn't sid=BIABKBRK the account? The access log should store this `sid` allowing you to just block the account. (or ask them for ton of money if it's a legitimate use :) )
  - JackWritesCode 6y ago
    > Layer 7 just means they are making ton of HTTP requests from lot of IPS however this is the nature of a web analytics. Exactly! Which is why it was so hard. There's no path pattern. Everything hits "/". So the only way to fight back is to match IP / header patterns (but even then, we have to redact sensitive headers). > How can the attackers know which websites have your analytics on? Crawling the web is super hard. The attacker went after some of our more high profile customers. They're known via testimonials or from Twitter. > The access log should store this `sid` allowing you to just block the account. (or ask them for ton of money if it's a legitimate use :) ) We could certainly temporarily block traffic to a site. The problem is, without some kind of firewall (e.g. WAF), our application has to absorb so much traffic, and that's the issue. We need to block it at the edge.
    - throwaway189262 6y ago
      > The problem is, without some kind of firewall (e.g. WAF), our application has to absorb so much traffic, and that's the issue. We need to block it at the edge. I think you need some high performance ingestion code. One day somebody is going to load test or misconfigure their site. Or you'll get a really big client. Or somebody will get Reddit frontpaged or slashdotted. Blocking huge volumes of traffic isn't always going to be an option. You should be able to handle it. I would get off all this "sexy" stuff like lamdba and SQS and build some old school battle boxes at a co-lo. Run some extreme performance framework like Actix or Vert.X built to handle giant piles of traffic. A single box with those frameworks and a 40 gig line can handle millions of requests a second. Use counting bloom filters synced between boxes for IP blocking. This avoids saving IP's and is needed to prevent ram exhaustion attacks anyways. Use two layers, one for individual IP's and a second for blocking subnets with lots of bad actors in them. Block for ~24h by using dual sets of bloom filters in a sliding window with 12h overlap where IP's get added to both. When a filter is 24h old, discard it. This needs to be done because you can't remove items from Bloom filters, they saturate over time. On these super boxes you can do filter, blocking, batching, and dedup. Then send the data off to wherever for further processing. You mentioned not wanting to use Cloudflare because they're a competitor. That's fine but I would take a look at their blog. They go over the tech they use for mass data ingestion and filtering, and it's basically what I just described.
jusssi 6y ago
> You can one-way hash the IP. So you can still look for pattern but you've lost the actual IP. And same one-way hash IP can block whichever IP seems devious in your firewall. The plaintext space (amount of possible IPs, even more so for IP blocks) is so small that that you can try all the possible plaintexts within seconds, essentially reversing the hash.
- corobo 6y ago
  Just off the top of my head, haven't given this much thought. Could you salt the hash with something way less predictable that gets discarded when the next one is generated? Assuming you only kept logs for a day for example you could regenerate it daily and store $salt somewhere only the most senior of senior techs can access it (if anyone at all)
- hartator 6y ago
  Then a longer to run hash function, like 256 times sha256 with a secret salt. It's reversible in theory but it will take a lot of resources.
  - tatersolid 6y ago
    You can’t afford to do that to every 1500-byte packet at 10-100 Gbps.
- throwaway189262 6y ago
  You can store "possible attacking" IP's in a counting Bloom filter and start blocking when it saturates. Random false positives are probably not an issue for an analytics service that's inherently lossy anyways. By using Bloom filter you avoid storing IP's, and save tons of space. They're basically required for rate limiting in IPV6 where the address space is so large an attacker can just run you out of ram by using so many different IP's. It's also pretty common to have several thresholds of blocking based on individual IP vs subnets. So if more than 2 ips get blocked in a small subnet, you block the whole subnet.
Aeolun 6y ago
> For example, a customer misplacing their analytics in a JS loop and having a moderate traffic blog will generate billions of requests from all over the globe. Potentially, but then customers don’t see a ton of spam on their dashboard. Just a lot of repeated requests. They don’t come in random waves either. Valid traffic is pretty well spaced out. > Your system should be sble to handle these billions of requests Potentially, but you really don’t want to pay for them. The only real way to go about stopping it is blocking offending ips.

JackWritesCode 6y ago

It was 100% a layer 7 DDoS, the attack was targeted and malicious. I can't say too much but the AWS team confirmed it. But I appreciate how something like you describe could happen. I like the idea of an MD5 hash. Although I'm not too certain why an IP would be a bad thing to log for 24 hours. From a privacy law perspective, the MD5 hash is considered PII. And if we see an IP address in an access log, we know that an IP visited one of the tens of thousands of websites Fathom runs on, but we don't know which one. Edit: Something else, with MD5 there's no way of finding patterns with the IPs, so you'd have to play whack a mole. Whereas raw IPs allow no real privacy invasion whilst allowing pattern detection

hartator 6y ago
> It was 100% a layer 7 DDoS, the attack was targeted and malicious. I can't say too much but the AWS team confirmed it. But I appreciate how something like you describe could happen. Layer 7 just means they are making ton of HTTP requests from lot of IPS however this is the nature of a web analytics. > we know that an IP visited one of the tens of thousands of websites Fathom runs on, but we don't know which one. How can the attackers know which websites have your analytics on? Crawling the web is super hard. You should be able to get back which accounts are affected from your analytics call: https://starman.fathomdns.com/?p=%2F&h=https%3A%2F%2Fusefathom.com&r=&sid=BIABKBRK&res=1440x900 Isn't sid=BIABKBRK the account? The access log should store this `sid` allowing you to just block the account. (or ask them for ton of money if it's a legitimate use :) )
- JackWritesCode 6y ago
  > Layer 7 just means they are making ton of HTTP requests from lot of IPS however this is the nature of a web analytics. Exactly! Which is why it was so hard. There's no path pattern. Everything hits "/". So the only way to fight back is to match IP / header patterns (but even then, we have to redact sensitive headers). > How can the attackers know which websites have your analytics on? Crawling the web is super hard. The attacker went after some of our more high profile customers. They're known via testimonials or from Twitter. > The access log should store this `sid` allowing you to just block the account. (or ask them for ton of money if it's a legitimate use :) ) We could certainly temporarily block traffic to a site. The problem is, without some kind of firewall (e.g. WAF), our application has to absorb so much traffic, and that's the issue. We need to block it at the edge.
  - throwaway189262 6y ago
    > The problem is, without some kind of firewall (e.g. WAF), our application has to absorb so much traffic, and that's the issue. We need to block it at the edge. I think you need some high performance ingestion code. One day somebody is going to load test or misconfigure their site. Or you'll get a really big client. Or somebody will get Reddit frontpaged or slashdotted. Blocking huge volumes of traffic isn't always going to be an option. You should be able to handle it. I would get off all this "sexy" stuff like lamdba and SQS and build some old school battle boxes at a co-lo. Run some extreme performance framework like Actix or Vert.X built to handle giant piles of traffic. A single box with those frameworks and a 40 gig line can handle millions of requests a second. Use counting bloom filters synced between boxes for IP blocking. This avoids saving IP's and is needed to prevent ram exhaustion attacks anyways. Use two layers, one for individual IP's and a second for blocking subnets with lots of bad actors in them. Block for ~24h by using dual sets of bloom filters in a sliding window with 12h overlap where IP's get added to both. When a filter is 24h old, discard it. This needs to be done because you can't remove items from Bloom filters, they saturate over time. On these super boxes you can do filter, blocking, batching, and dedup. Then send the data off to wherever for further processing. You mentioned not wanting to use Cloudflare because they're a competitor. That's fine but I would take a look at their blog. They go over the tech they use for mass data ingestion and filtering, and it's basically what I just described.

jusssi 6y ago

> You can one-way hash the IP. So you can still look for pattern but you've lost the actual IP. And same one-way hash IP can block whichever IP seems devious in your firewall. The plaintext space (amount of possible IPs, even more so for IP blocks) is so small that that you can try all the possible plaintexts within seconds, essentially reversing the hash.

corobo 6y ago
Just off the top of my head, haven't given this much thought. Could you salt the hash with something way less predictable that gets discarded when the next one is generated? Assuming you only kept logs for a day for example you could regenerate it daily and store $salt somewhere only the most senior of senior techs can access it (if anyone at all)
hartator 6y ago
Then a longer to run hash function, like 256 times sha256 with a secret salt. It's reversible in theory but it will take a lot of resources.
- tatersolid 6y ago
  You can’t afford to do that to every 1500-byte packet at 10-100 Gbps.
throwaway189262 6y ago
You can store "possible attacking" IP's in a counting Bloom filter and start blocking when it saturates. Random false positives are probably not an issue for an analytics service that's inherently lossy anyways. By using Bloom filter you avoid storing IP's, and save tons of space. They're basically required for rate limiting in IPV6 where the address space is so large an attacker can just run you out of ram by using so many different IP's. It's also pretty common to have several thresholds of blocking based on individual IP vs subnets. So if more than 2 ips get blocked in a small subnet, you block the whole subnet.

Aeolun 6y ago

> For example, a customer misplacing their analytics in a JS loop and having a moderate traffic blog will generate billions of requests from all over the globe. Potentially, but then customers don’t see a ton of spam on their dashboard. Just a lot of repeated requests. They don’t come in random waves either. Valid traffic is pretty well spaced out. > Your system should be sble to handle these billions of requests Potentially, but you really don’t want to pay for them. The only real way to go about stopping it is blocking offending ips.