42 ms·

First, I'd be shocked if Fastly didn't try to produce some counter arguments to the Cloudflare blogpost. I'm not saying Fastly nor Cloudflare are objectively ri

First, I'd be shocked if Fastly didn't try to produce some counter arguments to the Cloudflare blogpost. I'm not saying Fastly nor Cloudflare are objectively right, however, there are a couple things that stood out to me here. In the immediate finds I'm seeing, these two matter most to me: >Cloudflare used a free Fastly trial account to conduct their tests. Free trial accounts are designed for limited use compared to paid accounts, and performance under load is not comparable between the two. >Cloudflare conducted their tests in a single hour, on a single day. This fails to normalize for daily traffic patterns or abnormal events and is susceptible to random distortion effects. If you ran several sets of tests at different times of day, it’s likely that at some point, you’d achieve your desired outcome. Neither of these are good. For one, paid uninhibited account would cost little to do this test properly. You don't know if you hit their rate limits, are de-prioritized over paid users etc. There are lots of reasons why this is could cause the metrics to be skewed. The second one is an even bigger deal. Sample size matters, as does variance of inputs. This is just sloppy all around. 1 hour on 1 day will tell little of performance. Ideally, this should be done quite randomly, with a bunch of different random workload variances, and sampled over weeks, and from different locations (east coast, west coast, midwest, Canada, Europe etc). This would give a real sense of the overall network, and leave a lot less room for Fastly to hit back if the benchmarks are accurate. Overall I think Fastly may have caught Cloudflare red handed in sloppiness of comparision. For Cloudflare, this is pretty poor execution. Unfortunately, if or when Cloudflare does a response piece and still backs their claim, they're going to look more suspect if they aren't more thorough with their methodology and reproducibility

fiddlerwoaroof 3y ago
Those might be bad, but comparing rust compiled to web assembly vs. JavaScript is much worse here
- deleted 3y ago
  [deleted]
- dmix 3y ago
  I'll assume you're right but can someone with more knowledge of why explain this? I get the whole compiled vs JS's event loop stuff but I've never worked with edge computing so I don't really understand the details of what matters here in the full stack of a request. From my limited research in the past for web servers JS is not that far behind Rust. In fact they often rank better. For this "single query" JS (just-js) ranks 1st with a rust one at 3rd: https://www.techempower.com/benchmarks/#section=data-r21&test=query&hw=cl https://www.techempower.com/benchmarks/#section=data-r21&tes... for "fortunes" requests JS is 5th place with Rust at 2nd and 4th: https://www.techempower.com/benchmarks/#section=data-r21 https://www.techempower.com/benchmarks/#section=data-r21 etc, but again not sure how relevant this is
  - quickthrower2 3y ago
    I am guessing warming up, i.e. JIT?
  - rizky05 3y ago
    FYI, just-js is a js runtime specifically built for beating techempower benchmark. It is not for general use case.
  - fiddlerwoaroof 3y ago
    The issue I have is that, if you’re claiming to test the performance of Cloudflare workers vs. Fastly’s similar service and, especially, if you’re going to write a blogpost about the statistical issues with Cloudflare’s benchmarks, you really should do your best to make the only difference the platform. So either JavaScript on both or rust on both so it’s obvious that the language and runtime aren’t the source of performance differences.
    - dmix 3y ago
      That sounds like the most reasonable take
    - sophacles 3y ago
      At the time this was going down, cloudflare had web assembly and fastly didn't. Fastly's complaint is "we don't offer the fast runtime so its unfair that you point out that you have a faster option while claiming you have a faster option."
zdragnar 3y ago
I'll agree that a one hour, one day test isn't the greatest, but cloudflare workers have a free tier- comparing to fastly's free performance isn't entirely dishonest. What is dishonest, as a sibling pointed out, is compiling rust to WASM and pretending it is a more fair comparison. Reading through both posts, I'm not exactly finding fastly's defense to be meaningful.
kkielhofner 3y ago
Title needs to be updated - this is from 2021! In the edge compute space that makes it irrelevant.
- imperfectcats 3y ago
  I wish articles like this had prominent dates!
- HWR_14 3y ago
  It makes the results of the comparison irrelevant. It does not make the accuracy (or honesty) of the testing methodologies employed and reported irrelevant.
- tayo42 3y ago
  what happened next then? were there more follow ups?
  - NicoJuicy 3y ago
    Cloudflare said game on and is now faster than fastly everywhere Well, except Liberia I added more details here, with what I remember from those days: https://news.ycombinator.com/item?id=36733580 https://news.ycombinator.com/item?id=36733580 ( Then noticed your post)
lopkeny12ko 3y ago
Uh, what? Why should I, as a customer of a hosted, managed, SaaS product, need to ever care or think about the time of day as a factor for the performance of my product? If Fastly, as a hosted SaaS, isn't managing its own capacity well enough to deliver consistent performance at all times, that's on them. Additionally, provisioning hidden limits between different types of accounts, especially limits that affect performance for a CDN whose objective should be delivering consistent performance, is deceptive at best, and malicious and false advertising at worst.
- nostrebored 3y ago
  If you think that any CDN doesn’t have performance variations, you’re going to be sorely mistaken. There’s a reason that many sites that require efficient content delivery have many different CDNs and optimize between them. Well, there are many reasons. But one of them is QoS.
  - lostmsu 3y ago
    Fastly could have picked a bad Cloudflare hour and do their own countertest.
    - deleted 3y ago
      [deleted]
    - lnxg33k1 3y ago
      For a product named Compute@Kintergarten?
    - operator-name 3y ago
      The article outlines - Cloudflare doesn't strictly allow benchmarking - Thus fastly compared against Cloudflare's own metric and numbers
  - SteveNuts 3y ago
    My biggest issue with this is it comes across as desperate, defensive, and whiny. Fastly would do better to go on the offensive and prove their worth rather than complain about everything a competitor says.
judge2020 3y ago
For point 1, Is/Has Fastly ever offered a self-service paid plan? Last I tried, you had to talk to sales if you wanted to be able to pay them any amount of money for a regular amount of traffic or to access products like Compute@Edge. So for Cloudflare's tests, I can see why they wouldn't want to directly say "we're benchmarking you, can we have an account", since that would require an entire contract to be set up and approved by the purchasing department, and it could give Fastly the time to flag their account to "over" prioritize it, potentially affecting benchmark results.
- bastawhiz 3y ago
  That doesn't change the fact that it's an apples to oranges comparison.
NicoJuicy 3y ago
This is bullshit. Eastdakota refuted fastly's blame game then and this year blew fastly out of the water. https://twitter.com/eastdakota/status/1671896844513734656?t=tEoTvI53zcwMr5qBWu6lZw&s=19 https://twitter.com/eastdakota/status/1671896844513734656?t=... > Love that the only country where Fastly is fastest is now Liberia.
- youngtaff 3y ago
  Of course this graph is probably bullshit too as we simply don’t know enough about how the data was generated… > We use Real User Measurements (RUM) and fetch a small file from Cloudflare, Akamai, CloudFront, Fastly and Google Cloud CDN. Browsers around the world report the performance of those providers from the perspective of the end-user network they are on Even the blog post they reference doesn’t cover much more detail Key thing is to be measuring via RUM they must be injecting the script into sites they serve (and presumably the free ones) so there may already be bias introduced here, also why choose a 100kb file, how does the data differ if say a 10kb or 200kb is used etc. Lots of questions about the methodology remain
  - NicoJuicy 3y ago
    There's always going to be something to complain about, I guess... To counterpoint, I always found it interesting that cloudflare talks about population density ( big cities) on where to perform faster. Since that's where most of the population lives. I haven't seen any other companies talk about that outside of cloudflare. And it seems a more important real-life benchmark than deciding the difference between 10 kb, 100 kb or 200 kb... Tbh, I think 100 kb. Is just the smallest relevant size which they thought, looking at their speedtest file sizes of 100 kb, 1 mb and 10 mb - https://speed.cloudflare.com/ https://speed.cloudflare.com/
    - youngtaff 3y ago
      > There's always going to be something to complain about, I guess... Cloudflare are making the claim that they’re faster than others, and they collected the data they’re basing their claim on They’re always going to try to present themselves favourably so that means we should be skeptical and ask questions about the gaps in the data and the methodology

fiddlerwoaroof 3y ago

Those might be bad, but comparing rust compiled to web assembly vs. JavaScript is much worse here

deleted 3y ago
[deleted]
dmix 3y ago
I'll assume you're right but can someone with more knowledge of why explain this? I get the whole compiled vs JS's event loop stuff but I've never worked with edge computing so I don't really understand the details of what matters here in the full stack of a request. From my limited research in the past for web servers JS is not that far behind Rust. In fact they often rank better. For this "single query" JS (just-js) ranks 1st with a rust one at 3rd: https://www.techempower.com/benchmarks/#section=data-r21&test=query&hw=cl https://www.techempower.com/benchmarks/#section=data-r21&tes... for "fortunes" requests JS is 5th place with Rust at 2nd and 4th: https://www.techempower.com/benchmarks/#section=data-r21 https://www.techempower.com/benchmarks/#section=data-r21 etc, but again not sure how relevant this is
- quickthrower2 3y ago
  I am guessing warming up, i.e. JIT?
- rizky05 3y ago
  FYI, just-js is a js runtime specifically built for beating techempower benchmark. It is not for general use case.
- fiddlerwoaroof 3y ago
  The issue I have is that, if you’re claiming to test the performance of Cloudflare workers vs. Fastly’s similar service and, especially, if you’re going to write a blogpost about the statistical issues with Cloudflare’s benchmarks, you really should do your best to make the only difference the platform. So either JavaScript on both or rust on both so it’s obvious that the language and runtime aren’t the source of performance differences.
  - dmix 3y ago
    That sounds like the most reasonable take
  - sophacles 3y ago
    At the time this was going down, cloudflare had web assembly and fastly didn't. Fastly's complaint is "we don't offer the fast runtime so its unfair that you point out that you have a faster option while claiming you have a faster option."

zdragnar 3y ago

I'll agree that a one hour, one day test isn't the greatest, but cloudflare workers have a free tier- comparing to fastly's free performance isn't entirely dishonest. What is dishonest, as a sibling pointed out, is compiling rust to WASM and pretending it is a more fair comparison. Reading through both posts, I'm not exactly finding fastly's defense to be meaningful.

kkielhofner 3y ago

Title needs to be updated - this is from 2021! In the edge compute space that makes it irrelevant.

imperfectcats 3y ago
I wish articles like this had prominent dates!
HWR_14 3y ago
It makes the results of the comparison irrelevant. It does not make the accuracy (or honesty) of the testing methodologies employed and reported irrelevant.
tayo42 3y ago
what happened next then? were there more follow ups?
- NicoJuicy 3y ago
  Cloudflare said game on and is now faster than fastly everywhere Well, except Liberia I added more details here, with what I remember from those days: https://news.ycombinator.com/item?id=36733580 https://news.ycombinator.com/item?id=36733580 ( Then noticed your post)

lopkeny12ko 3y ago

Uh, what? Why should I, as a customer of a hosted, managed, SaaS product, need to ever care or think about the time of day as a factor for the performance of my product? If Fastly, as a hosted SaaS, isn't managing its own capacity well enough to deliver consistent performance at all times, that's on them. Additionally, provisioning hidden limits between different types of accounts, especially limits that affect performance for a CDN whose objective should be delivering consistent performance, is deceptive at best, and malicious and false advertising at worst.

nostrebored 3y ago
If you think that any CDN doesn’t have performance variations, you’re going to be sorely mistaken. There’s a reason that many sites that require efficient content delivery have many different CDNs and optimize between them. Well, there are many reasons. But one of them is QoS.
- lostmsu 3y ago
  Fastly could have picked a bad Cloudflare hour and do their own countertest.
  - deleted 3y ago
    [deleted]
  - lnxg33k1 3y ago
    For a product named Compute@Kintergarten?
  - operator-name 3y ago
    The article outlines - Cloudflare doesn't strictly allow benchmarking - Thus fastly compared against Cloudflare's own metric and numbers
- SteveNuts 3y ago
  My biggest issue with this is it comes across as desperate, defensive, and whiny. Fastly would do better to go on the offensive and prove their worth rather than complain about everything a competitor says.

judge2020 3y ago

For point 1, Is/Has Fastly ever offered a self-service paid plan? Last I tried, you had to talk to sales if you wanted to be able to pay them any amount of money for a regular amount of traffic or to access products like Compute@Edge. So for Cloudflare's tests, I can see why they wouldn't want to directly say "we're benchmarking you, can we have an account", since that would require an entire contract to be set up and approved by the purchasing department, and it could give Fastly the time to flag their account to "over" prioritize it, potentially affecting benchmark results.

bastawhiz 3y ago
That doesn't change the fact that it's an apples to oranges comparison.

NicoJuicy 3y ago

This is bullshit. Eastdakota refuted fastly's blame game then and this year blew fastly out of the water. https://twitter.com/eastdakota/status/1671896844513734656?t=tEoTvI53zcwMr5qBWu6lZw&s=19 https://twitter.com/eastdakota/status/1671896844513734656?t=... > Love that the only country where Fastly is fastest is now Liberia.

youngtaff 3y ago
Of course this graph is probably bullshit too as we simply don’t know enough about how the data was generated… > We use Real User Measurements (RUM) and fetch a small file from Cloudflare, Akamai, CloudFront, Fastly and Google Cloud CDN. Browsers around the world report the performance of those providers from the perspective of the end-user network they are on Even the blog post they reference doesn’t cover much more detail Key thing is to be measuring via RUM they must be injecting the script into sites they serve (and presumably the free ones) so there may already be bias introduced here, also why choose a 100kb file, how does the data differ if say a 10kb or 200kb is used etc. Lots of questions about the methodology remain
- NicoJuicy 3y ago
  There's always going to be something to complain about, I guess... To counterpoint, I always found it interesting that cloudflare talks about population density ( big cities) on where to perform faster. Since that's where most of the population lives. I haven't seen any other companies talk about that outside of cloudflare. And it seems a more important real-life benchmark than deciding the difference between 10 kb, 100 kb or 200 kb... Tbh, I think 100 kb. Is just the smallest relevant size which they thought, looking at their speedtest file sizes of 100 kb, 1 mb and 10 mb - https://speed.cloudflare.com/ https://speed.cloudflare.com/
  - youngtaff 3y ago
    > There's always going to be something to complain about, I guess... Cloudflare are making the claim that they’re faster than others, and they collected the data they’re basing their claim on They’re always going to try to present themselves favourably so that means we should be skeptical and ask questions about the gaps in the data and the methodology