4 ms·

> but at scale Do you perhaps know what the actual scale is? Is it a 100 proxies around the world or is 1k, 10k, 100k ...? I imagine that number is not very hi

> but at scale Do you perhaps know what the actual scale is? Is it a 100 proxies around the world or is 1k, 10k, 100k ...? I imagine that number is not very high since the blog post says that pingora-origin can serve 35M+ requests per second. My wild guess would be not more than 100 deployments. If so, then the "at scale" argument loosens up quite a bit IMO. > it frees up the cycles for other software I see this more like a theoretical stance since with such argument we could micro-optimize the code all day every day. It's a reverse HTTP proxy intended to be high-performance and low-latency so I don't imagine that the server that is running it has more important software to run.

vlovich123 2y ago
> Do you perhaps know what the actual scale is? Literally every single Cloudflare edge server that processes requests runs this proxy. Based on other things you said, it seems like your mental model of how Cloudflare's network runs is inaccurate. Mapping pingora origin requests/s to number of servers is difficult because of how tiered caching works & not all traffic is tiered cached and from what I remember pingora origin refers to all origin requests and a good amount of them aren't tiered (e.g. Workers outbound requests to the origin go through pingora origin & don't typically tier) which means those requests are being made directly from the edge (also tiering isn't like there's 100 servers - it spreads the load out evenly to avoid overwhelming any single server). [1] suggests ballpark of 164 data centers and each data center is multiple machines each with multiple cores; [2] says the next gen servers have 128 cores while [3] suggests that the current gen has 96 and the previous has 48. We can assume that on average each server in a data center has 64 cores. Let's assume there are 20 machines in a data center on average (I don't remember the exact number so it's a wild guess). That comes out to an estimate of ~30k machines with ~2M total cores and this reverse proxy being responsible for ~20% of CPU time on average for every single machine (40k CPU cores from the article divided by 2M total cores across the network). > It's a reverse HTTP proxy intended to be high-performance and low-latency so I don't imagine that the server that is running it has more important software to run. In fact, the server that's running the proxy is also the server that's handling the inbound request as well as the Workers runtime, Cloudflare tunnels, firewall product, image optimizations, DDOS protections, etc etc etc & any other internal software they have to run to maintain their network (observability, keeping track of routing graphs, etc). So yes, CPU cycles are largely fungible and will be freed up for use on other tasks / to absorb growth as they acquire more traffic (& even if your server is underutilized this can help because it means the CPUs are switching contexts less often given how many threads are on these machines resulting in potentially better latencies at the extremes for neighboring services). There's more nuance here but this is a good first order approximation. And yes, it's a micro optimization but clearly helpful at scale when you add up many teams doing micro optimizations. For example, here's a sqlite blog post [4] explaining how micro optimizations that don't even show up in real world systems (unlike this one which does) over an extended period of time yielded an overall 50% improvement. Now how much is it worth to optimize global CPU usage by 0.34%? I don't know but Cloudflare does have a good way of converting CPU cycles saved to how much it saved in terms of number of servers they had to buy. Back of the envelope this added ~1k machines worth of compute time capacity to the network (0.34% * 2 million cores / 64 cores per machine). I think it's interesting the 0.34% is the actual CPU usage from their blog post and also comes up if you multiply the 20% of CPU time I estimated for the server by the 1.7% they said that hot function takes up but I think that's mostly an accident - they predicted 0.43% which suggests that the network is larger than 2M (~31k servers) and maybe they have 2.5M cores (~39k servers) and between offline servers and workloads varying due to time of day in practice they're probably closer to using ~2M cores total of their overall capacity. That optimization could still be worth doing since SRE's are responsible for maintaining fleet health while the Pingora team is responsible for their corner. [1] https://community.cloudflare.com/t/how-many-datacenters-does-cloudflare-have-exactly/56642 https://community.cloudflare.com/t/how-many-datacenters-does... [2] https://blog.cloudflare.com/cloudflare-gen-12-server-bigger-better-cooler-in-a-2u1n-form-factor/ https://blog.cloudflare.com/cloudflare-gen-12-server-bigger-... [3] https://blog.cloudflare.com/the-epyc-journey-continues-to-milan-in-cloudflares-11th-generation-edge-server/ https://blog.cloudflare.com/the-epyc-journey-continues-to-mi... [4] https://sqlite-users.sqlite.narkive.com/CVRvSKBs/50-faster-than-3-7-17 https://sqlite-users.sqlite.narkive.com/CVRvSKBs/50-faster-t...
- menaerus 2y ago
  > That comes out to an estimate of ~30k machines with ~2M total cores and this reverse proxy being responsible for ~20% of CPU time on average for every single machine (40k CPU cores from the article divided by 2M total cores across the network). I think you wanted to say 2% of CPU time on average and not 20%, no? > In fact, the server that's running the proxy is also the server that's handling the inbound request as well as the Workers runtime, Cloudflare tunnels, firewall product, image optimizations, DDOS protections, etc etc etc & any other internal software they have to run to maintain their network (observability, keeping track of routing graphs, etc). Sure, they have plethora of software that they need to run but that wasn't my point. My point was rather that I cannot imagine that a single VM that runs the low-latency service such as the proxy is also running any type of heavier weight services. Thus an argument of freeing up the CPU cycles for some other software when in reality there probably isn't any didn't make much sense to me. This would be a legitimate argument if there was a challenge to begin with. Such as "we noticed that our 64-core 128G VM that runs the proxy is becoming saturated and thus the tail-latency of our XYZ service running on the same VM is starting to degrade". Without the problem statement "freeing up cycles for other software" is otherwise purely theoretical goal that is difficult if not impossible to quantify. > And yes, it's a micro optimization but clearly helpful at scale when you add up many teams doing micro optimizations. Perhaps in companies such as Google where they have a dedicated team developing fundamental low-level data-structures and/or runtime which is then used by 1000's of their other internal services. I believe that's actually one of the reasons why GoogleBenchmark was designed. In majority of other companies micro-optimizations are only a very good brain gymnastics exercise, I think. > For example, here's a sqlite blog post [4] explaining how micro optimizations that don't even show up in real world systems (unlike this one which does) over an extended period of time yielded an overall 50% improvement. The hypothesis of "100s of micro-optimizations add up to an overall 50% improvement" is equally, or mathematically speaking less likely (due to the Amdahl's law), than the hypothesis of "there were a few changes but we don't know what are the exact changes that contributed to the majority of 50% improvement" I intentionally say hypothesis because that's what it is. There's no evidence to support either of the claims. If I wanted to support my time spent working on the micro-optimizations at the XYZ company, this is also how I would frame it. This is not to say that I think that the author of sqlite was trying to do exactly that but only that the reasoning outlined in the post is not convincing to me. That said, reducing the CPU cycles for certain workload by ~30% is a noble and quantifiable artifact especially if you're a database company hosting thousands of your database instances since this obviously directly translates to more $$$ due to the better hardware utilization. However, trie-hard micro-optimization is done in the software that constitutes 2% of their total CPU time (if your ballpark estimates are somewhat close). Since the micro-optimization cut down the CPU utilization from 1.71% to 0.34% of 2% of 100% of CPU time this, in other words, means that total fleet-wide net of CPU utilization went from 0.000342% (before the change) to 0.000068% (after the change).

vlovich123 2y ago

> Do you perhaps know what the actual scale is? Literally every single Cloudflare edge server that processes requests runs this proxy. Based on other things you said, it seems like your mental model of how Cloudflare's network runs is inaccurate. Mapping pingora origin requests/s to number of servers is difficult because of how tiered caching works & not all traffic is tiered cached and from what I remember pingora origin refers to all origin requests and a good amount of them aren't tiered (e.g. Workers outbound requests to the origin go through pingora origin & don't typically tier) which means those requests are being made directly from the edge (also tiering isn't like there's 100 servers - it spreads the load out evenly to avoid overwhelming any single server). [1] suggests ballpark of 164 data centers and each data center is multiple machines each with multiple cores; [2] says the next gen servers have 128 cores while [3] suggests that the current gen has 96 and the previous has 48. We can assume that on average each server in a data center has 64 cores. Let's assume there are 20 machines in a data center on average (I don't remember the exact number so it's a wild guess). That comes out to an estimate of ~30k machines with ~2M total cores and this reverse proxy being responsible for ~20% of CPU time on average for every single machine (40k CPU cores from the article divided by 2M total cores across the network). > It's a reverse HTTP proxy intended to be high-performance and low-latency so I don't imagine that the server that is running it has more important software to run. In fact, the server that's running the proxy is also the server that's handling the inbound request as well as the Workers runtime, Cloudflare tunnels, firewall product, image optimizations, DDOS protections, etc etc etc & any other internal software they have to run to maintain their network (observability, keeping track of routing graphs, etc). So yes, CPU cycles are largely fungible and will be freed up for use on other tasks / to absorb growth as they acquire more traffic (& even if your server is underutilized this can help because it means the CPUs are switching contexts less often given how many threads are on these machines resulting in potentially better latencies at the extremes for neighboring services). There's more nuance here but this is a good first order approximation. And yes, it's a micro optimization but clearly helpful at scale when you add up many teams doing micro optimizations. For example, here's a sqlite blog post [4] explaining how micro optimizations that don't even show up in real world systems (unlike this one which does) over an extended period of time yielded an overall 50% improvement. Now how much is it worth to optimize global CPU usage by 0.34%? I don't know but Cloudflare does have a good way of converting CPU cycles saved to how much it saved in terms of number of servers they had to buy. Back of the envelope this added ~1k machines worth of compute time capacity to the network (0.34% * 2 million cores / 64 cores per machine). I think it's interesting the 0.34% is the actual CPU usage from their blog post and also comes up if you multiply the 20% of CPU time I estimated for the server by the 1.7% they said that hot function takes up but I think that's mostly an accident - they predicted 0.43% which suggests that the network is larger than 2M (~31k servers) and maybe they have 2.5M cores (~39k servers) and between offline servers and workloads varying due to time of day in practice they're probably closer to using ~2M cores total of their overall capacity. That optimization could still be worth doing since SRE's are responsible for maintaining fleet health while the Pingora team is responsible for their corner. [1] https://community.cloudflare.com/t/how-many-datacenters-does-cloudflare-have-exactly/56642 https://community.cloudflare.com/t/how-many-datacenters-does... [2] https://blog.cloudflare.com/cloudflare-gen-12-server-bigger-better-cooler-in-a-2u1n-form-factor/ https://blog.cloudflare.com/cloudflare-gen-12-server-bigger-... [3] https://blog.cloudflare.com/the-epyc-journey-continues-to-milan-in-cloudflares-11th-generation-edge-server/ https://blog.cloudflare.com/the-epyc-journey-continues-to-mi... [4] https://sqlite-users.sqlite.narkive.com/CVRvSKBs/50-faster-than-3-7-17 https://sqlite-users.sqlite.narkive.com/CVRvSKBs/50-faster-t...

menaerus 2y ago
> That comes out to an estimate of ~30k machines with ~2M total cores and this reverse proxy being responsible for ~20% of CPU time on average for every single machine (40k CPU cores from the article divided by 2M total cores across the network). I think you wanted to say 2% of CPU time on average and not 20%, no? > In fact, the server that's running the proxy is also the server that's handling the inbound request as well as the Workers runtime, Cloudflare tunnels, firewall product, image optimizations, DDOS protections, etc etc etc & any other internal software they have to run to maintain their network (observability, keeping track of routing graphs, etc). Sure, they have plethora of software that they need to run but that wasn't my point. My point was rather that I cannot imagine that a single VM that runs the low-latency service such as the proxy is also running any type of heavier weight services. Thus an argument of freeing up the CPU cycles for some other software when in reality there probably isn't any didn't make much sense to me. This would be a legitimate argument if there was a challenge to begin with. Such as "we noticed that our 64-core 128G VM that runs the proxy is becoming saturated and thus the tail-latency of our XYZ service running on the same VM is starting to degrade". Without the problem statement "freeing up cycles for other software" is otherwise purely theoretical goal that is difficult if not impossible to quantify. > And yes, it's a micro optimization but clearly helpful at scale when you add up many teams doing micro optimizations. Perhaps in companies such as Google where they have a dedicated team developing fundamental low-level data-structures and/or runtime which is then used by 1000's of their other internal services. I believe that's actually one of the reasons why GoogleBenchmark was designed. In majority of other companies micro-optimizations are only a very good brain gymnastics exercise, I think. > For example, here's a sqlite blog post [4] explaining how micro optimizations that don't even show up in real world systems (unlike this one which does) over an extended period of time yielded an overall 50% improvement. The hypothesis of "100s of micro-optimizations add up to an overall 50% improvement" is equally, or mathematically speaking less likely (due to the Amdahl's law), than the hypothesis of "there were a few changes but we don't know what are the exact changes that contributed to the majority of 50% improvement" I intentionally say hypothesis because that's what it is. There's no evidence to support either of the claims. If I wanted to support my time spent working on the micro-optimizations at the XYZ company, this is also how I would frame it. This is not to say that I think that the author of sqlite was trying to do exactly that but only that the reasoning outlined in the post is not convincing to me. That said, reducing the CPU cycles for certain workload by ~30% is a noble and quantifiable artifact especially if you're a database company hosting thousands of your database instances since this obviously directly translates to more $$$ due to the better hardware utilization. However, trie-hard micro-optimization is done in the software that constitutes 2% of their total CPU time (if your ballpark estimates are somewhat close). Since the micro-optimization cut down the CPU utilization from 1.71% to 0.34% of 2% of 100% of CPU time this, in other words, means that total fleet-wide net of CPU utilization went from 0.000342% (before the change) to 0.000068% (after the change).