4 ms·

Amazon also limits DNS queries - probably in a well meaning attempt to prevent DNS amplification attacks from originating within AWS. And I mean DNS queries acr

by ttul 8y ago

Amazon also limits DNS queries - probably in a well meaning attempt to prevent DNS amplification attacks from originating within AWS. And I mean DNS queries across their network whether or not they hit Amazon's DNS servers. This is _any_ port 53 UDP traffic. https://www.sparkpost.com/blog/undocumented-limit-dns-aws/ https://www.sparkpost.com/blog/undocumented-limit-dns-aws/

greglindahl 8y ago
I wonder if this is related to connection tracking?
- john37386 8y ago
  By default, Amazon uses stateless firewall. It means that by default it's not tracking connections.
  - spydum 8y ago
    I think you may be mistaken. Security Groups are stateful: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Securit... As suggested, it's very likely they hit the connection tracking limitation: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#security-group-connection-tracking https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-ne... I've personally witnessed teams hit this specifically for DNS (usually for internal, where you have explicitly permitted src/dst).
    - john37386 8y ago
      Yep I mistaken Security groups with Network Acl. Thanks. It's in the best practices to not track dns for big systems. From powerdns https://doc.powerdns.com/recursor/performance.html https://doc.powerdns.com/recursor/performance.html
    - gstaro 8y ago
      Thx for the clarification. Had the same misconception
kureikain 8y ago
I have got hit by this several time :(. AWS actually have well document about this DNS issue: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-limits https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.htm... This issue can easily get amplifier if you're using Kubernetes on AWS and some library that didn't cache on DNS on its own. Imagine you have a healthcheck every 3 seconds, do a bunch of DNS to its dependencies services, and a single server may have 10 pods.

by ttul 8y ago

greglindahl 8y ago

I wonder if this is related to connection tracking?

john37386 8y ago
By default, Amazon uses stateless firewall. It means that by default it's not tracking connections.
- spydum 8y ago
  I think you may be mistaken. Security Groups are stateful: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Securit... As suggested, it's very likely they hit the connection tracking limitation: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#security-group-connection-tracking https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-ne... I've personally witnessed teams hit this specifically for DNS (usually for internal, where you have explicitly permitted src/dst).
  - john37386 8y ago
    Yep I mistaken Security groups with Network Acl. Thanks. It's in the best practices to not track dns for big systems. From powerdns https://doc.powerdns.com/recursor/performance.html https://doc.powerdns.com/recursor/performance.html
  - gstaro 8y ago
    Thx for the clarification. Had the same misconception

kureikain 8y ago

I have got hit by this several time :(. AWS actually have well document about this DNS issue: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-limits https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.htm... This issue can easily get amplifier if you're using Kubernetes on AWS and some library that didn't cache on DNS on its own. Imagine you have a healthcheck every 3 seconds, do a bunch of DNS to its dependencies services, and a single server may have 10 pods.