7 ms·

Debunking Cloudflare’s recent performance tests (2021)

185 points by promiseofbeans 3y ago

burglins 3y ago
It's kinda weird seeing a blunt response (criticism of their competitor) from a company, but I like it. Feels more genuine, I guess.
- zdragnar 3y ago
  Given that they compared rust-wasm on a selection of a rather odd number of nodes (673- what went into that particular selection?) and the title of "lies", it just reads as someone at fastly is bitter. All the article does (which they even admit, blaming it on "beta" status) is that, if you want to run javascript on an edge server, cloudflare is faster. On top of that, you can do it in a free tier, whereas fastly says their free offering is only meant for a short trial.
no_wizard 3y ago
First, I'd be shocked if Fastly didn't try to produce some counter arguments to the Cloudflare blogpost. I'm not saying Fastly nor Cloudflare are objectively right, however, there are a couple things that stood out to me here. In the immediate finds I'm seeing, these two matter most to me: >Cloudflare used a free Fastly trial account to conduct their tests. Free trial accounts are designed for limited use compared to paid accounts, and performance under load is not comparable between the two. >Cloudflare conducted their tests in a single hour, on a single day. This fails to normalize for daily traffic patterns or abnormal events and is susceptible to random distortion effects. If you ran several sets of tests at different times of day, it’s likely that at some point, you’d achieve your desired outcome. Neither of these are good. For one, paid uninhibited account would cost little to do this test properly. You don't know if you hit their rate limits, are de-prioritized over paid users etc. There are lots of reasons why this is could cause the metrics to be skewed. The second one is an even bigger deal. Sample size matters, as does variance of inputs. This is just sloppy all around. 1 hour on 1 day will tell little of performance. Ideally, this should be done quite randomly, with a bunch of different random workload variances, and sampled over weeks, and from different locations (east coast, west coast, midwest, Canada, Europe etc). This would give a real sense of the overall network, and leave a lot less room for Fastly to hit back if the benchmarks are accurate. Overall I think Fastly may have caught Cloudflare red handed in sloppiness of comparision. For Cloudflare, this is pretty poor execution. Unfortunately, if or when Cloudflare does a response piece and still backs their claim, they're going to look more suspect if they aren't more thorough with their methodology and reproducibility
- fiddlerwoaroof 3y ago
  Those might be bad, but comparing rust compiled to web assembly vs. JavaScript is much worse here
  - deleted 3y ago
    [deleted]
  - dmix 3y ago
    I'll assume you're right but can someone with more knowledge of why explain this? I get the whole compiled vs JS's event loop stuff but I've never worked with edge computing so I don't really understand the details of what matters here in the full stack of a request. From my limited research in the past for web servers JS is not that far behind Rust. In fact they often rank better. For this "single query" JS (just-js) ranks 1st with a rust one at 3rd: https://www.techempower.com/benchmarks/#section=data-r21&test=query&hw=cl https://www.techempower.com/benchmarks/#section=data-r21&tes... for "fortunes" requests JS is 5th place with Rust at 2nd and 4th: https://www.techempower.com/benchmarks/#section=data-r21 https://www.techempower.com/benchmarks/#section=data-r21 etc, but again not sure how relevant this is
    - quickthrower2 3y ago
      I am guessing warming up, i.e. JIT?
    - rizky05 3y ago
      FYI, just-js is a js runtime specifically built for beating techempower benchmark. It is not for general use case.
    - fiddlerwoaroof 3y ago
      The issue I have is that, if you’re claiming to test the performance of Cloudflare workers vs. Fastly’s similar service and, especially, if you’re going to write a blogpost about the statistical issues with Cloudflare’s benchmarks, you really should do your best to make the only difference the platform. So either JavaScript on both or rust on both so it’s obvious that the language and runtime aren’t the source of performance differences.
KyleSanderson 3y ago
2021.
Signez 3y ago
See the previous discussion in 2021: https://news.ycombinator.com/item?id=29465729 https://news.ycombinator.com/item?id=29465729 …in which Cloudflare's CEO directly address the first point (having a legacy clause in their Terms that used to prohibit benchmarking, clause they removed quickly after), and in which one of the main engineer behind Cloudflare Workers underline that there is "apple vs oranges" comparisons for a while from Fastly before that blogpost. I really recommand reading those previous discussions, you'll learn a lot about the context of this blogpost and what Cloudflare think of it!
ProAm 3y ago
Cloudflare is now too big to fail. They are in the category of being able to get away doing horrible, biased, unethical shit and no one can do a thing because its now a battle of lawyers vs lawyers. They are not all bad, but its like saying Google does shady shit? Of course they do, but what are you going to do about it? Money is power, and power is everything.
- _8j50 3y ago
  Far from it. They are popular among the tech crowd for sure bur akamai and others are very popular too. Most sites don't use them either. Many if their products overlap with cloud providers' offerings too.
  - ProAm 3y ago
    Im just saying they are big enough you arent going to fight them.. Look how loose their T/C are. They serve a purpose, but any liberty they decide to follow you are at their whim unless you have legitimate money to fight. And for 99% its not worth a 1 year fight and the amount of money. Just like all big tech, its largely un-realistically-regulated so they can do what they want.
    - djbusby 3y ago
      Yea, like other providers (eg Twilio, Stripe) they can pick/choose who to allow as clients. Kick ClientA out because they did some vague, hand-wavy violation but allow ClientB (a direct competitor to A) continued use. Its not just these mentioned, many corporate ToS give them too much power and it's insurmountable for individuals or small business to fight back.
    - catlifeonmars 3y ago
      Are there any competitive counter examples, or is this just par for the course? Not saying that makes it ok, but now I am curious.
    - tbarbugli 3y ago
      dont use them if you dont like them, it is a free world
Aeolun 3y ago
Wow, someone is salty. I note that Fastly doesn’t specify how they selected their catchpoint nodes either. And they decided to compare Rust against JS because that somehow makes more sense? That’s bizarre.
- youngtaff 3y ago
  Rust was the first language Fastly supported so their support is more mature, AssemblyScript was a later development Not sure how many nodes Catchpoint have so 675 might be the natural limit
deleted 3y ago
[deleted]
tomalaci 3y ago
This should really get "(2021)" added to the title, especially since the title implies it being recent (when it is not).
- saagarjha 3y ago
  Added
  - promiseofbeans 3y ago
    Thanks - I should have looked more carefully before posting
asdfasgasdgasdg 3y ago
> Lies, damned lies: debunking Cloudflare’s recent performance tests It's not recent any more. This blog post is a year and a half old.
- promiseofbeans 3y ago
  Sorry, I should have looked more carefully at the article (the date is at the bottom for some reason?). It looks like an admin has updated the title to be more accurate now though, which is helpful.
dpifke 3y ago
Fastly is the only official PyPI mirror, and I was once asked by a client to figure out why PyPI downloads weren't working, on a host that otherwise seemed to have good connectivity (downloads of similarly-sized files from non-Fastly sites worked just fine). It turned out that Fastly ignores/blocks ICMP unreachable messages, which breaks Path MTU discovery[0]. The host was behind a VPN and I verified using Wireshark that it was asking Fastly to lower its MTU to accomodate the VPN packet overhead, but Fastly was ignoring the messages. Setting MSS on the host's default route (`ip route change default advmss 1380`) fixed it, but it seems basic networking tips like "don't block all ICMP" should be table stakes for a CDN. I nicely emailed Fastly about this, figuring they might appreciate knowing they had something misconfigured, but never heard back. Ironically, Cloudflare has a really good blog post about this issue[1], which I used when explaining the problem and solution to my client. [0]: https://en.wikipedia.org/wiki/Path_MTU_Discovery https://en.wikipedia.org/wiki/Path_MTU_Discovery [1]: https://blog.cloudflare.com/path-mtu-discovery-in-practice/ https://blog.cloudflare.com/path-mtu-discovery-in-practice/
- r1ch 3y ago
  Unfortunately you can't really rely on PMTU discovery these days. Not only is it regularly filtered, it slows every connection down since you add a RTT for the ICMP packets for every TCP host you talk to. In this case I would say the VPN client would be at fault for not adjusting the TCP MSS.
  - devman0 3y ago
    Relying on MSS to not break layer 3 is going to be interesting when QUIC is more prevalent. PMTUD should be better supported.
    - r1ch 3y ago
      QUIC unfortunately defaults to 1370 bytes to avoid having to deal with PMTUD :(. Endpoints are free to negotiate higher and the protocol does provide a way to generate arbitrary sized ping frames, but it's likely most implementations will just stick with the default.
tgtweak 3y ago
IDK - anecdotally I've used fastly side by side with cloudflare enterprise and I don't think they're stretching the truth too far saying that they generally perform better.
tc313 3y ago
(2021)
deleted 3y ago
[deleted]
berkle4455 3y ago
In which year do we think we’ll get leaks confirming Cloudflare was an NSA operation?
deleted 3y ago
[deleted]
civilized 3y ago
If it weren't for Hacker News I never would have learned about this genre of blog post, where two for-profit companies talk shit about each other's products until they both look bad.
- Havoc 3y ago
  Seems particularly common for edge related companies for some reason
- nickstinemates 3y ago
  It's really entertaining.
sgammon 3y ago
Recent Cloudflare updates on performance (2023): https://blog.cloudflare.com/speed-week-2023-network-performance-update/ https://blog.cloudflare.com/speed-week-2023-network-performa...

It's kinda weird seeing a blunt response (criticism of their competitor) from a company, but I like it. Feels more genuine, I guess.

zdragnar 3y ago
Given that they compared rust-wasm on a selection of a rather odd number of nodes (673- what went into that particular selection?) and the title of "lies", it just reads as someone at fastly is bitter. All the article does (which they even admit, blaming it on "beta" status) is that, if you want to run javascript on an edge server, cloudflare is faster. On top of that, you can do it in a free tier, whereas fastly says their free offering is only meant for a short trial.

no_wizard 3y ago

First, I'd be shocked if Fastly didn't try to produce some counter arguments to the Cloudflare blogpost. I'm not saying Fastly nor Cloudflare are objectively right, however, there are a couple things that stood out to me here. In the immediate finds I'm seeing, these two matter most to me: >Cloudflare used a free Fastly trial account to conduct their tests. Free trial accounts are designed for limited use compared to paid accounts, and performance under load is not comparable between the two. >Cloudflare conducted their tests in a single hour, on a single day. This fails to normalize for daily traffic patterns or abnormal events and is susceptible to random distortion effects. If you ran several sets of tests at different times of day, it’s likely that at some point, you’d achieve your desired outcome. Neither of these are good. For one, paid uninhibited account would cost little to do this test properly. You don't know if you hit their rate limits, are de-prioritized over paid users etc. There are lots of reasons why this is could cause the metrics to be skewed. The second one is an even bigger deal. Sample size matters, as does variance of inputs. This is just sloppy all around. 1 hour on 1 day will tell little of performance. Ideally, this should be done quite randomly, with a bunch of different random workload variances, and sampled over weeks, and from different locations (east coast, west coast, midwest, Canada, Europe etc). This would give a real sense of the overall network, and leave a lot less room for Fastly to hit back if the benchmarks are accurate. Overall I think Fastly may have caught Cloudflare red handed in sloppiness of comparision. For Cloudflare, this is pretty poor execution. Unfortunately, if or when Cloudflare does a response piece and still backs their claim, they're going to look more suspect if they aren't more thorough with their methodology and reproducibility

fiddlerwoaroof 3y ago
Those might be bad, but comparing rust compiled to web assembly vs. JavaScript is much worse here
- deleted 3y ago
  [deleted]
- dmix 3y ago
  I'll assume you're right but can someone with more knowledge of why explain this? I get the whole compiled vs JS's event loop stuff but I've never worked with edge computing so I don't really understand the details of what matters here in the full stack of a request. From my limited research in the past for web servers JS is not that far behind Rust. In fact they often rank better. For this "single query" JS (just-js) ranks 1st with a rust one at 3rd: https://www.techempower.com/benchmarks/#section=data-r21&test=query&hw=cl https://www.techempower.com/benchmarks/#section=data-r21&tes... for "fortunes" requests JS is 5th place with Rust at 2nd and 4th: https://www.techempower.com/benchmarks/#section=data-r21 https://www.techempower.com/benchmarks/#section=data-r21 etc, but again not sure how relevant this is
  - quickthrower2 3y ago
    I am guessing warming up, i.e. JIT?
  - rizky05 3y ago
    FYI, just-js is a js runtime specifically built for beating techempower benchmark. It is not for general use case.
  - fiddlerwoaroof 3y ago
    The issue I have is that, if you’re claiming to test the performance of Cloudflare workers vs. Fastly’s similar service and, especially, if you’re going to write a blogpost about the statistical issues with Cloudflare’s benchmarks, you really should do your best to make the only difference the platform. So either JavaScript on both or rust on both so it’s obvious that the language and runtime aren’t the source of performance differences.

KyleSanderson 3y ago

2021.

Signez 3y ago

See the previous discussion in 2021: https://news.ycombinator.com/item?id=29465729 https://news.ycombinator.com/item?id=29465729 …in which Cloudflare's CEO directly address the first point (having a legacy clause in their Terms that used to prohibit benchmarking, clause they removed quickly after), and in which one of the main engineer behind Cloudflare Workers underline that there is "apple vs oranges" comparisons for a while from Fastly before that blogpost. I really recommand reading those previous discussions, you'll learn a lot about the context of this blogpost and what Cloudflare think of it!

ProAm 3y ago

Cloudflare is now too big to fail. They are in the category of being able to get away doing horrible, biased, unethical shit and no one can do a thing because its now a battle of lawyers vs lawyers. They are not all bad, but its like saying Google does shady shit? Of course they do, but what are you going to do about it? Money is power, and power is everything.

_8j50 3y ago
Far from it. They are popular among the tech crowd for sure bur akamai and others are very popular too. Most sites don't use them either. Many if their products overlap with cloud providers' offerings too.
- ProAm 3y ago
  Im just saying they are big enough you arent going to fight them.. Look how loose their T/C are. They serve a purpose, but any liberty they decide to follow you are at their whim unless you have legitimate money to fight. And for 99% its not worth a 1 year fight and the amount of money. Just like all big tech, its largely un-realistically-regulated so they can do what they want.
  - djbusby 3y ago
    Yea, like other providers (eg Twilio, Stripe) they can pick/choose who to allow as clients. Kick ClientA out because they did some vague, hand-wavy violation but allow ClientB (a direct competitor to A) continued use. Its not just these mentioned, many corporate ToS give them too much power and it's insurmountable for individuals or small business to fight back.
  - catlifeonmars 3y ago
    Are there any competitive counter examples, or is this just par for the course? Not saying that makes it ok, but now I am curious.
  - tbarbugli 3y ago
    dont use them if you dont like them, it is a free world

Aeolun 3y ago

Wow, someone is salty. I note that Fastly doesn’t specify how they selected their catchpoint nodes either. And they decided to compare Rust against JS because that somehow makes more sense? That’s bizarre.

youngtaff 3y ago
Rust was the first language Fastly supported so their support is more mature, AssemblyScript was a later development Not sure how many nodes Catchpoint have so 675 might be the natural limit

deleted 3y ago

[deleted]

tomalaci 3y ago

This should really get "(2021)" added to the title, especially since the title implies it being recent (when it is not).

saagarjha 3y ago
Added
- promiseofbeans 3y ago
  Thanks - I should have looked more carefully before posting

asdfasgasdgasdg 3y ago

> Lies, damned lies: debunking Cloudflare’s recent performance tests It's not recent any more. This blog post is a year and a half old.

promiseofbeans 3y ago
Sorry, I should have looked more carefully at the article (the date is at the bottom for some reason?). It looks like an admin has updated the title to be more accurate now though, which is helpful.

dpifke 3y ago

Fastly is the only official PyPI mirror, and I was once asked by a client to figure out why PyPI downloads weren't working, on a host that otherwise seemed to have good connectivity (downloads of similarly-sized files from non-Fastly sites worked just fine). It turned out that Fastly ignores/blocks ICMP unreachable messages, which breaks Path MTU discovery[0]. The host was behind a VPN and I verified using Wireshark that it was asking Fastly to lower its MTU to accomodate the VPN packet overhead, but Fastly was ignoring the messages. Setting MSS on the host's default route (`ip route change default advmss 1380`) fixed it, but it seems basic networking tips like "don't block all ICMP" should be table stakes for a CDN. I nicely emailed Fastly about this, figuring they might appreciate knowing they had something misconfigured, but never heard back. Ironically, Cloudflare has a really good blog post about this issue[1], which I used when explaining the problem and solution to my client. [0]: https://en.wikipedia.org/wiki/Path_MTU_Discovery https://en.wikipedia.org/wiki/Path_MTU_Discovery [1]: https://blog.cloudflare.com/path-mtu-discovery-in-practice/ https://blog.cloudflare.com/path-mtu-discovery-in-practice/

r1ch 3y ago
Unfortunately you can't really rely on PMTU discovery these days. Not only is it regularly filtered, it slows every connection down since you add a RTT for the ICMP packets for every TCP host you talk to. In this case I would say the VPN client would be at fault for not adjusting the TCP MSS.
- devman0 3y ago
  Relying on MSS to not break layer 3 is going to be interesting when QUIC is more prevalent. PMTUD should be better supported.
  - r1ch 3y ago
    QUIC unfortunately defaults to 1370 bytes to avoid having to deal with PMTUD :(. Endpoints are free to negotiate higher and the protocol does provide a way to generate arbitrary sized ping frames, but it's likely most implementations will just stick with the default.

tgtweak 3y ago

IDK - anecdotally I've used fastly side by side with cloudflare enterprise and I don't think they're stretching the truth too far saying that they generally perform better.

tc313 3y ago

(2021)

deleted 3y ago

[deleted]

berkle4455 3y ago

In which year do we think we’ll get leaks confirming Cloudflare was an NSA operation?

deleted 3y ago

[deleted]

civilized 3y ago

If it weren't for Hacker News I never would have learned about this genre of blog post, where two for-profit companies talk shit about each other's products until they both look bad.

Havoc 3y ago
Seems particularly common for edge related companies for some reason
nickstinemates 3y ago
It's really entertaining.

sgammon 3y ago

Recent Cloudflare updates on performance (2023): https://blog.cloudflare.com/speed-week-2023-network-performance-update/ https://blog.cloudflare.com/speed-week-2023-network-performa...