5 ms·

Packets per Second Limitations in EC2

178 points by mbarlocker 8y ago

rbranson 8y ago
The (undocumented) PPS limitation on EC2 instance types before they added SR-IOV NICs is around 150K PPS. If you had your own full machine — usually the top size of a given instance class, but no guarantee — this would be pretty consistent. But it was a shared resource. This made running memcache clusters really painful on EC2, given that they’d easily get limited by packet throughput before CPU or bandwidth. With modern instance types it’s much better!
- ra1n85 8y ago
  ???? Where are you getting these numbers? For what instance types? For what protocols? This is just wrong.
  - edoceo 8y ago
    GP said it was undocumented (I presume empirical, but would like more details) Do you have a more accurate dataset? What are your observations? I'm assuming this affects loads of HN readers and I too am interested in what the facts are. Side note: you may be getting downvoted because a source, or other details are lacking. I too get downvoted for posts that lack these details.
    - samstave 8y ago
      I took him saying "just wrong" as being "im not OK with this", but i could be incorrect. However, with that said, i have always found that calling your rep and asking about specific un documented limits is the fastest way to get to the bottom of per-instance/account/vpc/whatever limits. Just as there are limits that can be changed if you agree, in writing, that you will be financially responsible for whatever the impact is (e.g. when you could tell them that you wanted spot price limits adjusted for you to be able to better bid above the scaling factors that were in place(not sure if this is still the case)) Some limits are global and cant be changed/negotiated, but other undocumented limits....
ttul 8y ago
Amazon also limits DNS queries - probably in a well meaning attempt to prevent DNS amplification attacks from originating within AWS. And I mean DNS queries across their network whether or not they hit Amazon's DNS servers. This is _any_ port 53 UDP traffic. https://www.sparkpost.com/blog/undocumented-limit-dns-aws/ https://www.sparkpost.com/blog/undocumented-limit-dns-aws/
- greglindahl 8y ago
  I wonder if this is related to connection tracking?
  - john37386 8y ago
    By default, Amazon uses stateless firewall. It means that by default it's not tracking connections.
    - spydum 8y ago
      I think you may be mistaken. Security Groups are stateful: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Securit... As suggested, it's very likely they hit the connection tracking limitation: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#security-group-connection-tracking https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-ne... I've personally witnessed teams hit this specifically for DNS (usually for internal, where you have explicitly permitted src/dst).
      - john37386 8y ago
        Yep I mistaken Security groups with Network Acl. Thanks. It's in the best practices to not track dns for big systems. From powerdns https://doc.powerdns.com/recursor/performance.html https://doc.powerdns.com/recursor/performance.html
      - gstaro 8y ago
        Thx for the clarification. Had the same misconception
ramshanker 8y ago
One thing I was looking for the entire article weather these limitations are imposed by the Instance or the TOR networking gear. Maybe random reallocation to a rack with newer switches will guarantee better baseline PPS.
- ra1n85 8y ago
  Limits are done on the physical host or the host’s NIC. The TORs are not involved.
- rbranson 8y ago
  It has to do with hardware support for virtualized networking. Instances capable of SR-IOV or with the ENA can sustain millions of PPS. It makes a huge difference.
  - ra1n85 8y ago
    Not correct. Instances are rate limited based on their type and protocol. Enhanced networking support increases total potential performance, but there are artificial limitations put in place.
  - dkhenry 8y ago
    They can sustain millions of PPS, but they still get artificially limited, no matter how many ENI's you seem to attach a given instance will top out at around the same amount of PPS
shaklee3 8y ago
Does anyone know if this applies to the dpdk ENA driver?
- dgemm 8y ago
  What difference would the driver make?
  - shaklee3 8y ago
    I'm assuming the limitation is in the smartnic, and perhaps a different driver is tuned differently.
- Gcplp 8y ago
  My guess is that there's a bridge between the NIC and the VM though which they're imposing the limits. Could be OVS with DPDK support, for example.
toast0 8y ago
Note that if your traffic hits the ec2 connection tracking security groups, you will also hit per instance limits on the number of tracked connections [1]. As far as I know, they don't come out and say they have a limit on the number of tracked connections, but they do, and it scales by instance type -- better to adjust your rules so the traffic is allowed in a stateless manner. I don't know, but wouldn't be surprised if connection tracked packets are more limited than packets that aren't tracked. [1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#security-group-connection-tracking https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-ne...
- greglindahl 8y ago
  That sure sounds like it's being processed by the standard Linux firewall. In which case, yeah, if you have (my favorite example) a web crawler operating on the general web, you'll hit serious limits.
  - toredash 8y ago
    There is a limit of you have a Security Group attached with a rule that is -not- 0.0.0.0/0. So for anything that is public / heavy utilized, the recommendation is to open the service up to 0.0.0.0/0.
Rapzid 8y ago
I guess technically it is, but I hesitate to call the need for PPS limitations in the DC as "over subscribing". Connect a single server to a network and it's oversubscribed. That's a bit hyperbolic, but even some beefy networks can be seriously burdened by just a single server spamming UDP packets without some sort of QoS.. Especially if they are bypassing user space and using the kernel to just replicate a bunch of packets onto the wire :) I'm probably a bit biased from having spent time setting up linux TC on xen hypervisors for this very reason; and I think we even settled at 50k pps for the per vm limit too..
- blazespin 8y ago
  They should still publish the limits. If they are reasonable, customers won't mind.
RA_Fisher 8y ago
The distributions are mixtures. This is my favorite package for modeling those: https://cran.r-project.org/web/packages/gamlss.mx/index.html https://cran.r-project.org/web/packages/gamlss.mx/index.html
kev009 8y ago
A more plausible explanation is that the xen networking path is simply expensive, the intel VFs are limited by queue count and silicon (i40e isn't a great ASIC), and the Annapurna part is really an ARM64 NPU. NPUs have been abandoned by most silicon vendors and have a tragic history. It's simply hard to make NPUs work right at attractive price/power/performance and at high speed versus fixed function scatter/gather I/O units coupled with general purpose CPUs running software network stacks. The only benefit Annapurna gives EC2 over a software device model is a hard security boundary of effectively another computer inside the computer for Nitro metal as a service. I think this is one reason why EC2 is limited to 25G while 100G has been commodity for a long time. Here is a demonstration of a software stack that can scale toward hardware limits without relying on a particular vendor https://www.slideshare.net/SeanChittenden/freebsd-vpc-introduction https://www.slideshare.net/SeanChittenden/freebsd-vpc-introd.... This approaches 100G line rate for large packets which is what it was optimized for. I don't know PPS at low packet size but do know what would be required to optimize that use case and it could be done pretty quickly.
- rbranson 8y ago
  James Hamilton talked about their commitment to 25GbE hardware at Reinvent in 2016. Fast forward to ~23m. https://youtu.be/AyOAjFNPAbA https://youtu.be/AyOAjFNPAbA
- discodave 8y ago
  Interesting theories on the EC2/Annapurna situation. Do GCP, Azure, or any other cloud providers offer 100G networking?
  - gstaro 8y ago
    Sounds like a marketing blurb but from just a few days ago: "Azure is breaking the speed barrier in cloud connectivity. ExpressRoute Direct provides 100G connectivity for customers with extreme bandwidth needs. This is 10x faster than other clouds." https://azure.microsoft.com/en-us/blog/azure-networking-fall-2018-update/ https://azure.microsoft.com/en-us/blog/azure-networking-fall... Can sb confirm that? Have a useful case in mind.
john37386 8y ago
EC2 throttles everything by default and PPS is no exception. What can you do when your system needs more bandwidth, cpu, ram or any other kind of resources? You can either scale vertically... which is not bad at beginning of a project, but sooner or later you will hit the ultimate limit. Or You can scale horizontally. Which means that you have enough nodes or instances to bypass those limits and make sure your projects grow well over time. Netflix runs on EC2 and they probably generate billions of PPS from Amazon. For sure it's several millions PPS a d they seem to not hit the limits mentionned in the article.
- spydum 8y ago
  What is it you think netflix runs on AWS? Content distribution is served from their Open Connect CDN, not AWS.. last I understood, most of Netflix cloud workloads were analytical/DWH, and services.. Not generally billions of PPS, and certainly not to single instances.
patrickg_zill 8y ago
Eventually people will realize how AWS overcharges for what they deliver. But of course there's nothing wrong with pricing yourself higher than the lowest cost option... This is perhaps already being seen in a piecemeal fashion as people compare eg S3 storage prices with other companies' prices.

178 points by mbarlocker 8y ago

The (undocumented) PPS limitation on EC2 instance types before they added SR-IOV NICs is around 150K PPS. If you had your own full machine — usually the top size of a given instance class, but no guarantee — this would be pretty consistent. But it was a shared resource. This made running memcache clusters really painful on EC2, given that they’d easily get limited by packet throughput before CPU or bandwidth. With modern instance types it’s much better!

ra1n85 8y ago
???? Where are you getting these numbers? For what instance types? For what protocols? This is just wrong.
- edoceo 8y ago
  GP said it was undocumented (I presume empirical, but would like more details) Do you have a more accurate dataset? What are your observations? I'm assuming this affects loads of HN readers and I too am interested in what the facts are. Side note: you may be getting downvoted because a source, or other details are lacking. I too get downvoted for posts that lack these details.
  - samstave 8y ago
    I took him saying "just wrong" as being "im not OK with this", but i could be incorrect. However, with that said, i have always found that calling your rep and asking about specific un documented limits is the fastest way to get to the bottom of per-instance/account/vpc/whatever limits. Just as there are limits that can be changed if you agree, in writing, that you will be financially responsible for whatever the impact is (e.g. when you could tell them that you wanted spot price limits adjusted for you to be able to better bid above the scaling factors that were in place(not sure if this is still the case)) Some limits are global and cant be changed/negotiated, but other undocumented limits....

ttul 8y ago

Amazon also limits DNS queries - probably in a well meaning attempt to prevent DNS amplification attacks from originating within AWS. And I mean DNS queries across their network whether or not they hit Amazon's DNS servers. This is _any_ port 53 UDP traffic. https://www.sparkpost.com/blog/undocumented-limit-dns-aws/ https://www.sparkpost.com/blog/undocumented-limit-dns-aws/

greglindahl 8y ago
I wonder if this is related to connection tracking?
- john37386 8y ago
  By default, Amazon uses stateless firewall. It means that by default it's not tracking connections.
  - spydum 8y ago
    I think you may be mistaken. Security Groups are stateful: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Securit... As suggested, it's very likely they hit the connection tracking limitation: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#security-group-connection-tracking https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-ne... I've personally witnessed teams hit this specifically for DNS (usually for internal, where you have explicitly permitted src/dst).
    - john37386 8y ago
      Yep I mistaken Security groups with Network Acl. Thanks. It's in the best practices to not track dns for big systems. From powerdns https://doc.powerdns.com/recursor/performance.html https://doc.powerdns.com/recursor/performance.html
    - gstaro 8y ago
      Thx for the clarification. Had the same misconception

ramshanker 8y ago

One thing I was looking for the entire article weather these limitations are imposed by the Instance or the TOR networking gear. Maybe random reallocation to a rack with newer switches will guarantee better baseline PPS.

ra1n85 8y ago
Limits are done on the physical host or the host’s NIC. The TORs are not involved.
rbranson 8y ago
It has to do with hardware support for virtualized networking. Instances capable of SR-IOV or with the ENA can sustain millions of PPS. It makes a huge difference.
- ra1n85 8y ago
  Not correct. Instances are rate limited based on their type and protocol. Enhanced networking support increases total potential performance, but there are artificial limitations put in place.
- dkhenry 8y ago
  They can sustain millions of PPS, but they still get artificially limited, no matter how many ENI's you seem to attach a given instance will top out at around the same amount of PPS

shaklee3 8y ago

Does anyone know if this applies to the dpdk ENA driver?

dgemm 8y ago
What difference would the driver make?
- shaklee3 8y ago
  I'm assuming the limitation is in the smartnic, and perhaps a different driver is tuned differently.
Gcplp 8y ago
My guess is that there's a bridge between the NIC and the VM though which they're imposing the limits. Could be OVS with DPDK support, for example.

toast0 8y ago

Note that if your traffic hits the ec2 connection tracking security groups, you will also hit per instance limits on the number of tracked connections [1]. As far as I know, they don't come out and say they have a limit on the number of tracked connections, but they do, and it scales by instance type -- better to adjust your rules so the traffic is allowed in a stateless manner. I don't know, but wouldn't be surprised if connection tracked packets are more limited than packets that aren't tracked. [1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#security-group-connection-tracking https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-ne...

greglindahl 8y ago
That sure sounds like it's being processed by the standard Linux firewall. In which case, yeah, if you have (my favorite example) a web crawler operating on the general web, you'll hit serious limits.
- toredash 8y ago
  There is a limit of you have a Security Group attached with a rule that is -not- 0.0.0.0/0. So for anything that is public / heavy utilized, the recommendation is to open the service up to 0.0.0.0/0.

Rapzid 8y ago

I guess technically it is, but I hesitate to call the need for PPS limitations in the DC as "over subscribing". Connect a single server to a network and it's oversubscribed. That's a bit hyperbolic, but even some beefy networks can be seriously burdened by just a single server spamming UDP packets without some sort of QoS.. Especially if they are bypassing user space and using the kernel to just replicate a bunch of packets onto the wire :) I'm probably a bit biased from having spent time setting up linux TC on xen hypervisors for this very reason; and I think we even settled at 50k pps for the per vm limit too..

blazespin 8y ago
They should still publish the limits. If they are reasonable, customers won't mind.

RA_Fisher 8y ago

The distributions are mixtures. This is my favorite package for modeling those: https://cran.r-project.org/web/packages/gamlss.mx/index.html https://cran.r-project.org/web/packages/gamlss.mx/index.html

kev009 8y ago

A more plausible explanation is that the xen networking path is simply expensive, the intel VFs are limited by queue count and silicon (i40e isn't a great ASIC), and the Annapurna part is really an ARM64 NPU. NPUs have been abandoned by most silicon vendors and have a tragic history. It's simply hard to make NPUs work right at attractive price/power/performance and at high speed versus fixed function scatter/gather I/O units coupled with general purpose CPUs running software network stacks. The only benefit Annapurna gives EC2 over a software device model is a hard security boundary of effectively another computer inside the computer for Nitro metal as a service. I think this is one reason why EC2 is limited to 25G while 100G has been commodity for a long time. Here is a demonstration of a software stack that can scale toward hardware limits without relying on a particular vendor https://www.slideshare.net/SeanChittenden/freebsd-vpc-introduction https://www.slideshare.net/SeanChittenden/freebsd-vpc-introd.... This approaches 100G line rate for large packets which is what it was optimized for. I don't know PPS at low packet size but do know what would be required to optimize that use case and it could be done pretty quickly.

rbranson 8y ago
James Hamilton talked about their commitment to 25GbE hardware at Reinvent in 2016. Fast forward to ~23m. https://youtu.be/AyOAjFNPAbA https://youtu.be/AyOAjFNPAbA
discodave 8y ago
Interesting theories on the EC2/Annapurna situation. Do GCP, Azure, or any other cloud providers offer 100G networking?
- gstaro 8y ago
  Sounds like a marketing blurb but from just a few days ago: "Azure is breaking the speed barrier in cloud connectivity. ExpressRoute Direct provides 100G connectivity for customers with extreme bandwidth needs. This is 10x faster than other clouds." https://azure.microsoft.com/en-us/blog/azure-networking-fall-2018-update/ https://azure.microsoft.com/en-us/blog/azure-networking-fall... Can sb confirm that? Have a useful case in mind.

john37386 8y ago

EC2 throttles everything by default and PPS is no exception. What can you do when your system needs more bandwidth, cpu, ram or any other kind of resources? You can either scale vertically... which is not bad at beginning of a project, but sooner or later you will hit the ultimate limit. Or You can scale horizontally. Which means that you have enough nodes or instances to bypass those limits and make sure your projects grow well over time. Netflix runs on EC2 and they probably generate billions of PPS from Amazon. For sure it's several millions PPS a d they seem to not hit the limits mentionned in the article.

spydum 8y ago
What is it you think netflix runs on AWS? Content distribution is served from their Open Connect CDN, not AWS.. last I understood, most of Netflix cloud workloads were analytical/DWH, and services.. Not generally billions of PPS, and certainly not to single instances.

patrickg_zill 8y ago

Eventually people will realize how AWS overcharges for what they deliver. But of course there's nothing wrong with pricing yourself higher than the lowest cost option... This is perhaps already being seen in a piecemeal fashion as people compare eg S3 storage prices with other companies' prices.