4 ms·

They don't explicitly say that in the blog post but I assume that the engineering team behind this code does not run any workload performance experiments themse

They don't explicitly say that in the blog post but I assume that the engineering team behind this code does not run any workload performance experiments themselves but they only run the local (micro)benchmarks. Because of this I think there's no way for them to answer your question or to say "hey, this could very well be a vector since we could not find any statistically significant impact between a vector, hash-map and trie in this code path for given workload". This is a common fallacy I've seen in too many places. In other words, investing non-trivial engineering effort into optimizing a code-path that resembles .000001% of your workload is a waste of resources. And due to the Amdahl's law > Optimizing operations that are already measured in microseconds may seem a little silly, but these small improvements add up. this is not how things usually work out in the end. Making small improvements in insignificant code-paths will still remain insignificant in the overall runtime performance.

vlovich123 2y ago
> If we compare the sampled performance of the different versions of clear_internal_headers, we can see that the results from the performance sampling closely match what our benchmarks predicted Look at the end where they validate that the performance wins show up in production in a similar way that the microbenchmarks predict.
- menaerus 2y ago
  They only measure the CPU utilization of clear_internal_headers and they don't show what was the impact of the change to the overall runtime performance. The latter is what I was referring to.
  - vlovich123 2y ago
    The overall performance change for that specific software is minimal but at scale it frees up the cycles for other software since it’s a multi tenant system which is why they’re showing how much CPU time they shaved across the fleet. Or am I misunderstanding what point you’re making?
    - menaerus 2y ago
      > but at scale Do you perhaps know what the actual scale is? Is it a 100 proxies around the world or is 1k, 10k, 100k ...? I imagine that number is not very high since the blog post says that pingora-origin can serve 35M+ requests per second. My wild guess would be not more than 100 deployments. If so, then the "at scale" argument loosens up quite a bit IMO. > it frees up the cycles for other software I see this more like a theoretical stance since with such argument we could micro-optimize the code all day every day. It's a reverse HTTP proxy intended to be high-performance and low-latency so I don't imagine that the server that is running it has more important software to run.
      - vlovich123 2y ago
        > Do you perhaps know what the actual scale is? Literally every single Cloudflare edge server that processes requests runs this proxy. Based on other things you said, it seems like your mental model of how Cloudflare's network runs is inaccurate. Mapping pingora origin requests/s to number of servers is difficult because of how tiered caching works & not all traffic is tiered cached and from what I remember pingora origin refers to all origin requests and a good amount of them aren't tiered (e.g. Workers outbound requests to the origin go through pingora origin & don't typically tier) which means those requests are being made directly from the edge (also tiering isn't like there's 100 servers - it spreads the load out evenly to avoid overwhelming any single server). [1] suggests ballpark of 164 data centers and each data center is multiple machines each with multiple cores; [2] says the next gen servers have 128 cores while [3] suggests that the current gen has 96 and the previous has 48. We can assume that on average each server in a data center has 64 cores. Let's assume there are 20 machines in a data center on average (I don't remember the exact number so it's a wild guess). That comes out to an estimate of ~30k machines with ~2M total cores and this reverse proxy being responsible for ~20% of CPU time on average for every single machine (40k CPU cores from the article divided by 2M total cores across the network). > It's a reverse HTTP proxy intended to be high-performance and low-latency so I don't imagine that the server that is running it has more important software to run. In fact, the server that's running the proxy is also the server that's handling the inbound request as well as the Workers runtime, Cloudflare tunnels, firewall product, image optimizations, DDOS protections, etc etc etc & any other internal software they have to run to maintain their network (observability, keeping track of routing graphs, etc). So yes, CPU cycles are largely fungible and will be freed up for use on other tasks / to absorb growth as they acquire more traffic (& even if your server is underutilized this can help because it means the CPUs are switching contexts less often given how many threads are on these machines resulting in potentially better latencies at the extremes for neighboring services). There's more nuance here but this is a good first order approximation. And yes, it's a micro optimization but clearly helpful at scale when you add up many teams doing micro optimizations. For example, here's a sqlite blog post [4] explaining how micro optimizations that don't even show up in real world systems (unlike this one which does) over an extended period of time yielded an overall 50% improvement. Now how much is it worth to optimize global CPU usage by 0.34%? I don't know but Cloudflare does have a good way of converting CPU cycles saved to how much it saved in terms of number of servers they had to buy. Back of the envelope this added ~1k machines worth of compute time capacity to the network (0.34% * 2 million cores / 64 cores per machine). I think it's interesting the 0.34% is the actual CPU usage from their blog post and also comes up if you multiply the 20% of CPU time I estimated for the server by the 1.7% they said that hot function takes up but I think that's mostly an accident - they predicted 0.43% which suggests that the network is larger than 2M (~31k servers) and maybe they have 2.5M cores (~39k servers) and between offline servers and workloads varying due to time of day in practice they're probably closer to using ~2M cores total of their overall capacity. That optimization could still be worth doing since SRE's are responsible for maintaining fleet health while the Pingora team is responsible for their corner. [1] https://community.cloudflare.com/t/how-many-datacenters-does-cloudflare-have-exactly/56642 https://community.cloudflare.com/t/how-many-datacenters-does... [2] https://blog.cloudflare.com/cloudflare-gen-12-server-bigger-better-cooler-in-a-2u1n-form-factor/ https://blog.cloudflare.com/cloudflare-gen-12-server-bigger-... [3] https://blog.cloudflare.com/the-epyc-journey-continues-to-milan-in-cloudflares-11th-generation-edge-server/ https://blog.cloudflare.com/the-epyc-journey-continues-to-mi... [4] https://sqlite-users.sqlite.narkive.com/CVRvSKBs/50-faster-than-3-7-17 https://sqlite-users.sqlite.narkive.com/CVRvSKBs/50-faster-t...

vlovich123 2y ago

> If we compare the sampled performance of the different versions of clear_internal_headers, we can see that the results from the performance sampling closely match what our benchmarks predicted Look at the end where they validate that the performance wins show up in production in a similar way that the microbenchmarks predict.

menaerus 2y ago
They only measure the CPU utilization of clear_internal_headers and they don't show what was the impact of the change to the overall runtime performance. The latter is what I was referring to.
- vlovich123 2y ago
  The overall performance change for that specific software is minimal but at scale it frees up the cycles for other software since it’s a multi tenant system which is why they’re showing how much CPU time they shaved across the fleet. Or am I misunderstanding what point you’re making?
  - menaerus 2y ago
    > but at scale Do you perhaps know what the actual scale is? Is it a 100 proxies around the world or is 1k, 10k, 100k ...? I imagine that number is not very high since the blog post says that pingora-origin can serve 35M+ requests per second. My wild guess would be not more than 100 deployments. If so, then the "at scale" argument loosens up quite a bit IMO. > it frees up the cycles for other software I see this more like a theoretical stance since with such argument we could micro-optimize the code all day every day. It's a reverse HTTP proxy intended to be high-performance and low-latency so I don't imagine that the server that is running it has more important software to run.
    - vlovich123 2y ago
      > Do you perhaps know what the actual scale is? Literally every single Cloudflare edge server that processes requests runs this proxy. Based on other things you said, it seems like your mental model of how Cloudflare's network runs is inaccurate. Mapping pingora origin requests/s to number of servers is difficult because of how tiered caching works & not all traffic is tiered cached and from what I remember pingora origin refers to all origin requests and a good amount of them aren't tiered (e.g. Workers outbound requests to the origin go through pingora origin & don't typically tier) which means those requests are being made directly from the edge (also tiering isn't like there's 100 servers - it spreads the load out evenly to avoid overwhelming any single server). [1] suggests ballpark of 164 data centers and each data center is multiple machines each with multiple cores; [2] says the next gen servers have 128 cores while [3] suggests that the current gen has 96 and the previous has 48. We can assume that on average each server in a data center has 64 cores. Let's assume there are 20 machines in a data center on average (I don't remember the exact number so it's a wild guess). That comes out to an estimate of ~30k machines with ~2M total cores and this reverse proxy being responsible for ~20% of CPU time on average for every single machine (40k CPU cores from the article divided by 2M total cores across the network). > It's a reverse HTTP proxy intended to be high-performance and low-latency so I don't imagine that the server that is running it has more important software to run. In fact, the server that's running the proxy is also the server that's handling the inbound request as well as the Workers runtime, Cloudflare tunnels, firewall product, image optimizations, DDOS protections, etc etc etc & any other internal software they have to run to maintain their network (observability, keeping track of routing graphs, etc). So yes, CPU cycles are largely fungible and will be freed up for use on other tasks / to absorb growth as they acquire more traffic (& even if your server is underutilized this can help because it means the CPUs are switching contexts less often given how many threads are on these machines resulting in potentially better latencies at the extremes for neighboring services). There's more nuance here but this is a good first order approximation. And yes, it's a micro optimization but clearly helpful at scale when you add up many teams doing micro optimizations. For example, here's a sqlite blog post [4] explaining how micro optimizations that don't even show up in real world systems (unlike this one which does) over an extended period of time yielded an overall 50% improvement. Now how much is it worth to optimize global CPU usage by 0.34%? I don't know but Cloudflare does have a good way of converting CPU cycles saved to how much it saved in terms of number of servers they had to buy. Back of the envelope this added ~1k machines worth of compute time capacity to the network (0.34% * 2 million cores / 64 cores per machine). I think it's interesting the 0.34% is the actual CPU usage from their blog post and also comes up if you multiply the 20% of CPU time I estimated for the server by the 1.7% they said that hot function takes up but I think that's mostly an accident - they predicted 0.43% which suggests that the network is larger than 2M (~31k servers) and maybe they have 2.5M cores (~39k servers) and between offline servers and workloads varying due to time of day in practice they're probably closer to using ~2M cores total of their overall capacity. That optimization could still be worth doing since SRE's are responsible for maintaining fleet health while the Pingora team is responsible for their corner. [1] https://community.cloudflare.com/t/how-many-datacenters-does-cloudflare-have-exactly/56642 https://community.cloudflare.com/t/how-many-datacenters-does... [2] https://blog.cloudflare.com/cloudflare-gen-12-server-bigger-better-cooler-in-a-2u1n-form-factor/ https://blog.cloudflare.com/cloudflare-gen-12-server-bigger-... [3] https://blog.cloudflare.com/the-epyc-journey-continues-to-milan-in-cloudflares-11th-generation-edge-server/ https://blog.cloudflare.com/the-epyc-journey-continues-to-mi... [4] https://sqlite-users.sqlite.narkive.com/CVRvSKBs/50-faster-than-3-7-17 https://sqlite-users.sqlite.narkive.com/CVRvSKBs/50-faster-t...