21 ms·

Debunking Cloudflare’s recent performance tests

556 points by mcone 5y ago

gopalv 5y ago
> _We used a Wasm binary compiled from Rust rather than JavaScript. > We know support for JavaScript is important to many customers, but we're not yet satisfied with the performance of Compute@Edge packages compiled from JavaScript. That's why it's in beta. When a product is ready for production, we remove the beta designation. I can't really square the idea that the 50-150ms time-delays in question comes down to the actual programming language, but it is absolutely believable that a longer test reduces the median latency rather than a high load test for a shorter duration. Having said that, I would notice anything over 150ms in my clicks, but wouldn't care whether something took under 100 or under 50 - except that the lower the latency the less the scale needed to serve the same number of active users (it becomes a question of cost rather than response time).
- kailanb 5y ago
  It definitely comes down to the language — JavaScript is an interpreted language, which requires an engine. Rust is statically compiled and runs by itself.
  - azakai 5y ago
    In general you'd be right, but this is not a benchmark of code running in those languages. There is no loop, no heavy computation - it's just returning the headers. That should take less than 1ms in either language. Startup time might be a factor. But the measurements of the same language running on the same provider's infrastructure go from around 10 to 50ms (on one provider) and 10 to 100ms (on the other). Those huge differences don't look like CPU latencies (the machines aren't 5x or 10x faster in different locations) - they are likely network latencies. So programming language is probably a very minor factor here (at least in the non-beta services, which is what was measured here - maybe it was an issue in the older measurements).
    - acdha 5y ago
      I'd also expect unrelated load to be a factor, especially if there's a mechanism which priorities paid customer traffic over the free tier. Given the complexities of loading and running JavaScript, I'd be especially interested in what memory contention looks like — it seems like it could be as simple as Rust needing far less memory than loading all of SpiderMonkey and thus being less likely to hit both low-level CPU cache and high-level runtime cache evictions.
      - azakai 5y ago
        Ah, good point, memory consumption might indeed be a factor here. If that's the case I'd expect higher variance in the measurements perhaps - looking for that in the raw data could be interesting.
    - technobabbler 5y ago
      > In general you'd be right, but this is not a benchmark of code running in those languages. There is no loop, no heavy computation - it's just returning the headers. That should take less than 1ms in either language. They can't have it both ways... Fastly was the one that complained that their JS engine still wasn't up to par (being in beta), so clearly the language of choice (or interpreter/compiler/whatever) DOES matter. Either the language doesn't matter (in which case, replicate the JS test) or it does, in which case this isn't even a useful comparison.
      - azakai 5y ago
        Sorry, I should have been clearer. The new benchmark data does not look like it is influenced by the language in a noticeable way (since we see 5x or 10x differences in the mean even when using the same language, and which provider is faster or slower depends on which location you measure at). But it's very possible the old data may have been influenced by the language. That's not very interesting, though: as Fastly said, it's just beta performance. And anyhow this is a benchmark with almost no CPU work anyhow.
  - vlovich123 5y ago
    On Cloudflare, JavaScript is a JITted language, not strictly interpreted (it uses V8). Code that's seeing any amount of traffic gets lowered into native code pretty darn quick & V8's profile-guided optimizer is pretty impressive at removing JS overheads. Rust is statically compiled but it runs under WASM. If I'm not mistaken, WASM is also typically interpreted & then JIT'ed to native code (at least it is on V8 IIRC). My guess would be that Fastly lowers it to native code AOT (or at least does a fair bit of optimization) but I'm not 100% sure. The question is whether Rust WASM will beat CF JS and that part is less clear because the workloads you'd choose for JS vs Rust are so different. Also, these platforms aren't static so it'll be interesting to watch the competition play out as we both push the boundaries of Edge compute. Disclaimer: I work at Cloudflare on R2. I think both teams do great technical work but I'm more excited about CF. I think our suite of Edge compute products built on top of Workers is stronger (cron triggers, unbound, durable objects & many other interesting things coming).
- acdha 5y ago
  > I can't really square the idea that the 50-150ms time-delays in question comes down to the actual programming language, but it is absolutely believable that a longer test reduces the median latency rather than a high load test for a shorter duration. It seems plausible to me: in Fastly's case, they're using WebAssembly via wasmtime[1] which does support AOT compilation but most JavaScript code is dynamic enough that they still need a runtime JIT engine. I believe the the current approach Fastly is using is to compile Mozilla's SpiderMonkey JIT engine itself into WebAssembly and they've done some really nice work making that load as quickly as possible: https://bytecodealliance.org/articles/making-javascript-run-fast-on-webassembly https://bytecodealliance.org/articles/making-javascript-run-... The catch, of course, is that this still leaves a fair amount work which a JavaScript program has to deal with at runtime compared to a Rust program which the compiler can spend minutes optimizing long before deployment. This is a classic tradeoff for dynamic languages and a lot of people are satisfied with the approach of defaulting to faster developer turnaround and later converting hot spots to something like C or Rust, but I think it's definitely dodgy to use a single example of something you know to be this complex and present the results as generally representative of the entire platform. I have no knowledge of how Cloudflare ran their tests or reason to suspect malice but not disclosing test methodology and a “no benchmarks” license clause is going to make accusations about the results inevitable. Veterans of the benchmarketing wars have stories about comparisons where someone used a configuration option which happened to disable key performance optimizations for their competitors or used examples which favored their design decisions. Since nobody can read someone's mind to tell their real intent, the best way to avoid acrimony is to have full public disclosure of the benchmarks and their methodology — and to allow other people to conduct benchmarks so they can reproduce results or fill gaps. 1. https://github.com/bytecodealliance/wasmtime https://github.com/bytecodealliance/wasmtime
xwdv 5y ago
This vicious attack by Fastly on Cloudflare will not go unnoticed. IMO Fastly should prepare for the retaliation.
- kailanb 5y ago
  Vicious attack? Or defending your place in the market?
  - floatingatoll 5y ago
    Based on their comment history, I interpreted this as unmarked-sarcasm slash pithy-witty driveby, that isn't meant to be taken seriously or contribute usefully to the discussion.
- Twirrim 5y ago
  Cloudflare started it by attacking Fastly in the first place.
  - NicoJuicy 5y ago
    Benchmarking isn't attacking...
redm 5y ago
I think statistics like these tests are easy to sway in your favor, hence why Fastly is responding. Its not clear that Fastly didn't do the same thing, but a good ole' CDN rivalry does make for a good read. Someone get more mud to sling.
deft 5y ago
point #2 is silly. So basically your product is slower but its ok cuz its a beta only... Fine. But then cf did not mislead anyone. Your beta is slow.
- invisible 5y ago
  Not really. Cloudflare could have used their rust implementation for doing a fair comparison, instead they compared a beta product to a stable product. It's not like CF didn't have a choice to make a fair comparison.
  - etchalon 5y ago
    Did CF pretend they were benchmarking something other than the product they said they were benchmarking?
    - invisible 5y ago
      > Today, we’re excited to report that Cloudflare Workers is 196% faster than Fastly’s Compute@Edge based on the time to first byte from the tests we ran on 50 nodes using Catchpoint’s data from across the world. Kind of? At minimum they made misleading representations. They used abbreviated explanations of how they're better and then expanded on what the test actually consisted of later on.
      - unityByFreedom 5y ago
        > At minimum they made misleading representations Fastly does not have a production-ready JavaScript product. From their blog post, > Their tests compare JavaScript running on Cloudflare Workers, a mature, generally available product, with JavaScript running on Compute@Edge. Although the Compute@Edge platform is now available for all in production, support for JavaScript on Compute@Edge is a beta product. We clearly identify in our documentation [2] that beta products are not ready for production use. A fairer test on this point would have compared Rust on Compute@Edge with JavaScript on Cloudflare Workers, which are at more comparable stages of the product lifecycle. Restricting any comparison isn't satisfactory since we care about both the supported languages and performance. Let there be writeups about both and allow users to make up our own minds. I think the Cloudflare article [2] could be amended to refer to the compared product as "Fastly's JavaScript on Compute@Edge" and all would be fine. Fastly will probably still say they advise against this comparison since it's "beta". Nonetheless, we should all feel free to compare publicly available products and write about them. [1] https://docs.fastly.com/products/fastly-product-lifecycle#:~:text=Fastly%20strongly%20advises,Terms%20of%20Service https://docs.fastly.com/products/fastly-product-lifecycle#:~.... [2] https://blog.cloudflare.com/network-performance-update-full-stack-week/ https://blog.cloudflare.com/network-performance-update-full-...
jjeaff 5y ago
As mentioned in the article, Cloudflare expressly prohibits benchmarking their services in the tos. I think it's rather disingenuous for Cloudflare to publish their own benchmarks calling out competitors when they won't allow anyone to run their own tests and comparisons.
- charcircuit 5y ago
  They don't prohibit it outright. They just require you to ask first. The article made no mention of if they asked or not.
  - deadmutex 5y ago
    Why tho?
    - hysan 5y ago
      Probably for the same reason that I've seen processes put in place for performance testing a service (internally or on production) - because it puts a significant load on the service and automatic rate limiting, flagging, etc can be triggered.
    - dekhn 5y ago
      if you benchmark your competitor, your competitor should have the ability to provide tuning so their numbers look as good as possible, before you run to the press.
      - deadmutex 5y ago
        Unless your competitor was offering a complimentary service to do that for all customers (tuning) then perhaps it shouldn't be done. I believe the idea of benchmarking is to get a sense of how the service would perform for a customer. Consider the following scenario: You calling your ISP that you're going to do a benchmark. Then the ISP gives you the best service, at the expense of other customers bandwidth, for the duration of the test. Then after the test, the ISP removes the QoS modification, and reverts to the old behavior. That could end up with a biased, unrealistic result.
jbergstroem 5y ago
In the context of "should I use Cloudflare or Fastly as my edge", I lean Cloudflare. That said, I enjoyed listening to the GraphCDN crew choosing Fastly[1] – one crucial feather in the hat being cache invalidation[2] (miss ya, Phil Karlton) – and it sounds like a solid choice. [1]: https://www.youtube.com/watch?v=lpmpTJc_SP0 https://www.youtube.com/watch?v=lpmpTJc_SP0 [2]: https://graphcdn.io/docs/how-to/purge-the-cache https://graphcdn.io/docs/how-to/purge-the-cache
- tylermenezes 5y ago
  I've used both Cloudflare and Fastly for many years, and the cache invalidation at Fastly is a way bigger deal than people think. They're not competing for a 5% improvement in performance, they're competing with a totally different set of features which mean you can cache 80% more things, as long as you design for it. (Which I'm sure is why they don't aggressively compete on free plans -- if you try to just throw it in front like Cloudflare you won't see much of a difference from Cloudflare.) With some careful design to take advantage of it, I've gotten 5-10x faster page loads using Fastly. (The last time I pointed this out, people accused me of being a sockpuppet account. Feel free to Google me!)
  - snowwrestler 5y ago
    Yes. Caching is a different animal when you can count on fast, reliable, targeted invalidation. You can cache basically everything with super long TTLs, and just sniper-shot individual cache objects as soon as the backend updates.
  - deleted 5y ago
    [deleted]
  - sadsoul21 5y ago
    I'm curious, what "totally different set of features" are you referring to?
    - tylermenezes 5y ago
breakingcups 5y ago
Cloudflare not allowing benchmarks in their TOS is very sketchy, that puts them in the same tier as Oracle. Cloudflare have pulled enough shady stuff now that they've fallen out of my favor. Their generous free product bought them a lot of community goodwill but their real face has been showing the past few years.
- pm90 5y ago
  I would give them a bit more wiggle room before giving up. We’re currently living under a monopoly of AWS (GCP is the most serious “competitor”, still far far behind in market share) and I would love to see more competition in this space. That said: this is definitely not the kind of behavior I expect from them and I’m hoping there’s a better reason than them not willing to submit their products to an impartial evaluation.
  - varsketiz 5y ago
    You are forgetting Azure, which is much bigger than GCP.
    - pm90 5y ago
      I’m not forgetting Azure, it’s a deliberate omission because it is not a Tier 1 cloud product. It might be better than OCI or IBM Cloud but it simply does not have the maturity of either AWS or GCP.
      - robertlagrant 5y ago
        Interesting. As someone who's only really used Azure (other than a little GCP) - what's missing?
  - latch 5y ago
    AWS is a monopoly if you're living in a HN bubble. The vast majority of "hosting" (internet or otherwise) doesn't use AWS. Large telco's (Telefonica, AT&T, China Unicom, ...) have numerous data centers. Conglomerates (Tata, ...) have numerous data centers. You have wholesalers (Equinix, Digital Realty, ...). You have dedicated providers (OVH, Hetzner, PhoenixNap, ...). VPS' (DO, Linode, Vultr,...) and shared hosting (GoDaddy, ...) Then you have a lot of private data centers, e.g. I worked at a major bank that had its own datacenters (multiple around the world). Admittedly, these are super small compared to an AWS DC, but there's literally thousands of these around the world (I know for banking and government, but I assume other industries do this too).
deleted 5y ago
[deleted]
merb 5y ago
btw. for what it's worth their javascript to wasm is opensource: - https://github.com/fastly/js-compute-runtime https://github.com/fastly/js-compute-runtime - https://github.com/tschneidereit/spidermonkey-wasi-embedding https://github.com/tschneidereit/spidermonkey-wasi-embedding and besides that it is slower than nodejs it is still plenty fast (no matter that it is not as fast as they want) btw. it's startup is faster than node. (maybe better pgo might help) its insane what they built, to do it and it will bring the whole wasm community forward if they succeed (and I hope they do)
- NicoJuicy 5y ago
  Gentle reminder: That's mostly because they acquired the team behind wasm. Which was already an opensource project at Mozilla. Post on HN: https://news.ycombinator.com/item?id=24897641 https://news.ycombinator.com/item?id=24897641
  - tschneidereit 5y ago
    It's really the other way around: we joined Fastly because we knew it's a place where we could do this kind of work in the open. None of the code involved here existed a year ago, and none of it was somehow forced to be open source. (Also, the code in these two repositories is in many ways the most boring part of this solution. See this blog post for an excellent overview of the more interesting parts: https://bytecodealliance.org/articles/making-javascript-run-fast-on-webassembly https://bytecodealliance.org/articles/making-javascript-run-...)
    - NicoJuicy 5y ago
      The article I linked too ( and the hn comments) in my original post seemed that fastly acquired the wasm team. Sorry if I was mistaking.
      - tschneidereit 5y ago
        Oh, no worries at all! While there was no acquisition involved, a whole group of folks working on WebAssembly at Mozilla (myself included) moved to Fastly last Fall. What I tried to emphasize is that instead of the projects at hand here being open source because they somehow had to be, we all joined Fastly because it's a place where this kind of project can be made open source (and be created in the first place!) :)
kentonv 5y ago
OK... So, I'm the tech lead for Cloudflare Workers. In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don't know anything about either beyond what's publicly visible. But as long as we're already mud slinging, I'd like to take the opportunity to get a little something off my chest. Fastly has been trumpeting for years that Compute@Edge has 35 microsecond cold starts, or whatever, and repeatedly posting blog posts comparing that against 5 milliseconds for Workers, and implying that they are 150x faster. If you look at the details, it turns out that 35 microsecond time is actually how long they take to start a new request, given that the application is already loaded in memory. A hot start, not a cold start. Whereas Workers' 5ms includes time to load the application from disk (which is the biggest contributor to total time). Our hot start time is also a few microseconds, but that doesn't seem like an interesting number? We never called this out, it didn't seem worth arguing over. But excuse me if I'm not impressed by claims of false comparisons... On a serious note, I've been saying for decades that benchmarks are almost always meaningless, because different technologies will have different strengths and weaknesses, so you usually can't tell anything about how your use case will perform unless you actually test that use case. So, I would encourage everyone to run your own test and don't just go on other people's numbers. It's great that Fastly has opened up C@E for self-service testing so that people can actually try it out.
- zimmerx 5y ago
  I think this is all super fair commentary and very balanced. As a Compute@Edge user though how do they test CloudFlare when the ToS says otherwise? Test and break the ToS and hope for the best? It's a bit of a shame that it exists.
- refulgentis 5y ago
  I suggest deleting this. I'm a completely neutral third-party with little interest in the subject at hand, and their post doesn't come off like mud-slinging, at all. This then makes your response look accusatory and helps further a broader narrative post about Cloudflare's behavior here, which it seems your goal is to dispel, given your statement on whether engineering was involved.* A suggestion for something actionable that'll make things better in this difficult moment: email product counsel about starting the process to override your EULA and enable Compute@Edge to do benchmarking. It'll take forever and it's the only way to have an objective, evolving conversation about benchmarks, which your leads will (at least, should) want after this happened. * intentionally vague here, in order to give you space to delete and leave no record
deleted 5y ago
[deleted]
rileymat2 5y ago
> Cloudflare used a free Fastly trial account to conduct their tests. Free trial accounts are designed for limited use compared to paid accounts, and performance under load is not comparable between the two. https://www.fastly.com/pricing/ https://www.fastly.com/pricing/ does not make a distinction it says that "Any size company looking to give Fastly a try." They make distinctions on paying for more bandwidth, not that they are giving you an inferior product performance wise.
- flumpcakes 5y ago
  > They make distinctions on paying for more bandwidth, not that they are giving you an inferior product performance wise. I think this is a non-sequitur. Bandwidth is a fundamental property of the performance of a system, if you can "pay for more bandwidth" then you are literally getting an inferior product when you do not pay. Consider the implementation for "paying for more bandwidth". I assume that it is the same copper cabling hitting the same switches for fee paying and non-fee paying customers so the only way to "offer more bandwidth" is to implement some kind of quality-of-service metric on the network. This is explicitly giving the non-fee paying customer an inferior product.
  - judge2020 5y ago
    > Test up to $50 of traffic for free. Pay as you go from there with bandwidth pricing. Looks like it's just post-use monitoring that'll cut off the account after it exceeds the trial limit or start billing per-GB; it doesn't say 'slower connection', so CF didn't realize that a trial account would affect their benchmarks regarding Compute@Edge
  - rileymat2 5y ago
    People have stopped using "bandwidth" as we used to. It now means a total sum of the amount transferred in a lot of cases, not the "width" of the pipe. It is more evident here: https://www.fastly.com/pricing/#bandwidthpricing https://www.fastly.com/pricing/#bandwidthpricing (To be clear, I agree with your definition, but it is not used)
alophawen 5y ago
<--- CRIPPLE FIGHT
deleted 5y ago
[deleted]
atom_arranger 5y ago
As a consumer it’s great that both companies are benchmarking. Even if Cloudflare’s was flawed. Maybe a case of, the best way to get the right answer is to post the wrong answer. We’re at least moving towards objective comparisons, talking about numbers. Cloudflare prohibiting benchmarking is not good, obviously.
rkwasny 5y ago
Popcorn time ;) Let's wait for @jgrahamc to wake up Simple solution is for Cloudflare to change TOS to allow benchmarking and for 3rd party to publish a reproducible benchmark suite. However TBH benchmarking a distributed system like Cloudflare/Fastly is going to be REALLY HARD. There are 1000s of servers involved and reproducing the results might be impossible.
- ksec 5y ago
  >Popcorn time ;) Indeed. We already have comments claiming Fastly are dismissive and evasive while others claiming the fault of Cloudflare. The community is divided, which is quite usual on recent HN. Let there be another holy war of CDN between Cloudflare and Fastly.
  - unityByFreedom 5y ago
    > The community is divided I don't see any division. It's clear to me who has experience to back up their words and who doesn't.
  - saagarjha 5y ago
    Ok, but let’s ideally not have the comments from that war wash up here.
- raggi 5y ago
  It's a decent use case for planetlab.
  - ethbr0 5y ago
    And reproducible benchmarks with cached, publicly accessible results. At some point, you either have to make free tiers worse, or deal with thousands of people using them to run the same benchmark over and over. Cloudflare loves alliances: start a distributed benchmarking alliance! :)
- unityByFreedom 5y ago
  > Simple solution is for Cloudflare to change TOS to allow benchmarking and for 3rd party to publish a reproducible benchmark suite. Why doesn't Fastly just gasp ask Cloudflare about performing a benchmark? If they say no, publish that. Fastly does not publish their pricing [1]. And, you need to contact a rep to sign up for commercial use [2], which implies everyone gets a different deal. It's absurd to say that contacting Cloudflare to do benchmarking means it's impossible for them to do a comparison, > [FASTLY BLOG] it's impossible for us to define our own terms for a comparison because Cloudflare’s terms of use prohibit benchmarking of their services >> [CLOUDFLARE TERMS] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent; That is not so different from Fastly's own sign-up process. [1] https://www.fastly.com/pricing/ https://www.fastly.com/pricing/ [2] https://news.ycombinator.com/item?id=29468455 https://news.ycombinator.com/item?id=29468455
technobabbler 5y ago
I was expecting a point-by-point comparison from Fastly. Instead, this comes off incredibly defensive and evasive, basically boiling down to two irrelevant points: * Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN) * Fastly performance throttles their free accounts (ok, but not Cloudflare's fault, and I definitely don't want to get into your pricing game if you're going to start offering different tiers of serverless workers... part of Cloudflare's allure is its simplicity in pricing). They do have two good points: * Benchmarks between CDNs should be done from the same client locations (fine, but why did you then use your own set) * Cloudflare should not prohibit benchmarking of their products (fair) But this whole post read like a feeble attempt at misdirection, and made me distrust Fastly as a company. The complaints about their methodology aren't solved by using your own, even more flawed, methodology that doesn't even use the same language. You could've just emailed them and asked to work together on an apple-to-apples test instead of creating an even more flawed benchmark.
- tylerhou 5y ago
  > * Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN) First, the JavaScript is compiled to hot bytecode (WASM-equivalent) by the VM, so the performance difference is smaller than you would expect than the normal performance difference between Rust and JavaScript. Second, when you’re measuring milliseconds for response latencies, the difference between JS taking 10us CPU and Rust-wasm taking 1us is negligible. And system overheads far outweigh the actual computation time.
  - sroussey 5y ago
    > First, the JavaScript is compiled to hot bytecode (WASM-equivalent) by the VM, so the performance difference is smaller than you would expect than the normal performance difference between Rust and JavaScript. Can you explain this? JS intermediate is wildly different from WASM, not to mention WASM requires a max memory album and a lot of other things resulting from the different VMs
    - 5y ago
imilk 5y ago
This comes across as incredibly defensive in tone from Fastly. They could have just reported the results of their test. But calling their competitors outright liars reflects worse on them than anyone else.
- bloodyplonker22 5y ago
  They do have to punch up because they are the little guy in marketing, but they did it in a very sloppy manner with shaky points.
- shah0048 5y ago
  When you're sucker punched with misleading stats, ya kind have to hit back. And didn't sound defensive at all, btw.
thecompilr 5y ago
Awesome, now where do I sign for the free fastly account? Oh right, they only do enterprise, so who the hell cares? Cloudflare OTOH caters for the whole spectrum.
yeskia 5y ago
Only semi-related, but why does Cloudflare perform so poorly in Oceania? I've noticed less than stellar performance in Oceania with Cloudflare's CDN, so to see similar stats with workers makes it seem it's not a product-specific issue.
- plasma 5y ago
  I'm based Australia and have used Cloudflare for personal/business; if you aren't on the $200/month plan for your website then you don't get (consistent) access to the Australian-based servers for your visitors (they may be routed to eg Singapore instead during some times of the day) which actually hurts performance if your service is located in (say) AWS/Azure Australia region, at least for the CDN aspect, I assume the same for Workers too.
  - kiwijamo 5y ago
    Same here in New Zealand too. Cloudflare peers only with certain ISPs. The ones they don't peer with sees some traffic (e.g. sites on free plans) routed to offshore Cloudflare POPs around the world instead of local Cloudflare POPs. I understand if the website is on a paid plan they will be more likely to route to a local POP (incurring more costs in doing so).
    - celsoazevedo 5y ago
      According to this 2016 post, the problem is Optus and Telstra, and the costs of peering with them (scroll down to "Oceania"): https://blog.cloudflare.com/bandwidth-costs-around-the-world/ https://blog.cloudflare.com/bandwidth-costs-around-the-world...
- LordAtlas 5y ago
  I'm in India and if you're on the free CF plan, you often seem to get served from the Singapore POP instead of the Indian ones. I've seen other people on various groups complain about this too.
foton1981 5y ago
Seriously, who cares about 5ms difference? Or even 25ms. I don't. And Asia is just used to longer latencies. Edge compute became a commodity even before it was born.
- jlund-molfese 5y ago
  I’d love to be able to (affordably) spin up a low-latency Windows desktop for a few hours at a time, that I could use for applications which don’t run well on my bottom-tier M1 Mac
- cortesoft 5y ago
  The whole point of edge workers is to be able to provide very low latency all around the world... once you have that, it opens up whole new classes of workloads. Yes, 25ms doesn't matter for most current use cases.... but that is partly because the use cases that required lower latency weren't possible before.
ddtaylor 5y ago
That is pretty scummy that Cloudflare won't allow benchmarks.
civilized 5y ago
It's been hilarious watching competing companies publish warring blog posts declaring each other full of shit. I mean, I knew, but thanks for airing each others' dirty underpants.
- gary_0 5y ago
  It reminds me of being on Slashdot 15 years ago. We've even got one of the engineers posting in the comment thread, starting their own little flamewar. The names of the characters have changed, but there's still a bit of the same old Internet here.
  - unityByFreedom 5y ago
    The fact that a lead engineer and CEO took the time to respond here is clarity not a flamewar. edit maybe I'm wrong, https://news.ycombinator.com/item?id=29469174 https://news.ycombinator.com/item?id=29469174
    - gary_0 5y ago
      I did say little flamewar. :)
      - kentonv 5y ago
        I definitely started some sort of a flamewar but somehow not the one I was trying to start.
- dmw_ng 5y ago
  Most insightful comment here IMHO. Vaguely reminiscent of the Glasgow Ice Cream Wars, I'd rather not buy from anyone involved
  - deleted 5y ago
    [deleted]
- unityByFreedom 5y ago
boynamedsue 5y ago
Cloudflare seems to encourage benchmarks of Workers Sites: https://blog.cloudflare.com/workers-sites/ https://blog.cloudflare.com/workers-sites/
JediPig 5y ago
I am glad I said no to working for both. When I saw both are politically active and showed child like behavior during the interviews, I said no, and decided to dedicate some of my time to dog and cat rescue.
- deleted 5y ago
  [deleted]
soheil 5y ago
I'm regularly surprised why Cloudflare gets so much love specially here on HN. Cloudflare is slow, it man-in-the-middles websites, it shows a redirect page randomly for no apparent reason on websites that it powers, it does questionable things at the DNS level, it randomly shows captchas for no apparent reason, the list goes on. Almost every website nowadays uses this monstrosity. We're handing over the internet on a silver platter to them. I think there are better alternatives and I'm so glad at least Fastly is debunking their bs experiment.
- celsoazevedo 5y ago
  > it man-in-the-middles websites That's just how reverse proxies work. I'm not saying that this is good, but anyone offering the same service is doing the same thing.
- sgtfrankieboy 5y ago
  >no apparent reason A big reason is making sure the website behind it stays online. Having to deal with weekly/daily DDOS attacks yourself takes a lot of time.
speedgoose 5y ago
I also read that Fastly is slower at specific point of times during a week, slower for free test accounts, slower at executing javascript (hence the switch to rust), and slower in locations carefully selected by CloudFlare. I'm not saying that the benchmark from CloudFlare is fair, but it's a bit funny that their response contain a "let's switch to rust" or "let's benchmark next to our datacenters".
- kamray23 5y ago
  I mean they switched to Rust because a comparison of two fully developed systems is fairer than a production system and one in testing. Besides, I believe the blog post said that they selected nodes at actual random?
tomasreimers 5y ago
"A fairer test on this point would have compared Rust on Compute@Edge with JavaScript on Cloudflare Workers, which are at more comparable stages of the product lifecycle." Are there any widely agreed upon benchmarks on how emscriptened rust compares to untyped js? It feels like a large portion of the reported Delta boils down to that.
chinathrow 5y ago
This reads as someone who his losing lots of customers to CF and whines about it.
deleted 5y ago
[deleted]
qwertyuiop_ 5y ago
Another appendage measuring contest.
seando-vajina 5y ago
wgaf
nickdothutton 5y ago
The “DeWitt Clause” or sometimes also known as the “Oracle Clause” has been a fixture of ToS for years. The fact that it features in Cloudflare’s docs only serves to remind us that entrenched practices die hard. https://www.brentozar.com/archive/2018/05/the-dewitt-clause-why-you-rarely-see-database-benchmarks/ https://www.brentozar.com/archive/2018/05/the-dewitt-clause-...

556 points by mcone 5y ago

> _We used a Wasm binary compiled from Rust rather than JavaScript. > We know support for JavaScript is important to many customers, but we're not yet satisfied with the performance of Compute@Edge packages compiled from JavaScript. That's why it's in beta. When a product is ready for production, we remove the beta designation. I can't really square the idea that the 50-150ms time-delays in question comes down to the actual programming language, but it is absolutely believable that a longer test reduces the median latency rather than a high load test for a shorter duration. Having said that, I would notice anything over 150ms in my clicks, but wouldn't care whether something took under 100 or under 50 - except that the lower the latency the less the scale needed to serve the same number of active users (it becomes a question of cost rather than response time).

kailanb 5y ago
It definitely comes down to the language — JavaScript is an interpreted language, which requires an engine. Rust is statically compiled and runs by itself.
- azakai 5y ago
  In general you'd be right, but this is not a benchmark of code running in those languages. There is no loop, no heavy computation - it's just returning the headers. That should take less than 1ms in either language. Startup time might be a factor. But the measurements of the same language running on the same provider's infrastructure go from around 10 to 50ms (on one provider) and 10 to 100ms (on the other). Those huge differences don't look like CPU latencies (the machines aren't 5x or 10x faster in different locations) - they are likely network latencies. So programming language is probably a very minor factor here (at least in the non-beta services, which is what was measured here - maybe it was an issue in the older measurements).
  - acdha 5y ago
    I'd also expect unrelated load to be a factor, especially if there's a mechanism which priorities paid customer traffic over the free tier. Given the complexities of loading and running JavaScript, I'd be especially interested in what memory contention looks like — it seems like it could be as simple as Rust needing far less memory than loading all of SpiderMonkey and thus being less likely to hit both low-level CPU cache and high-level runtime cache evictions.
    - azakai 5y ago
      Ah, good point, memory consumption might indeed be a factor here. If that's the case I'd expect higher variance in the measurements perhaps - looking for that in the raw data could be interesting.
  - technobabbler 5y ago
    > In general you'd be right, but this is not a benchmark of code running in those languages. There is no loop, no heavy computation - it's just returning the headers. That should take less than 1ms in either language. They can't have it both ways... Fastly was the one that complained that their JS engine still wasn't up to par (being in beta), so clearly the language of choice (or interpreter/compiler/whatever) DOES matter. Either the language doesn't matter (in which case, replicate the JS test) or it does, in which case this isn't even a useful comparison.
    - azakai 5y ago
      Sorry, I should have been clearer. The new benchmark data does not look like it is influenced by the language in a noticeable way (since we see 5x or 10x differences in the mean even when using the same language, and which provider is faster or slower depends on which location you measure at). But it's very possible the old data may have been influenced by the language. That's not very interesting, though: as Fastly said, it's just beta performance. And anyhow this is a benchmark with almost no CPU work anyhow.
- vlovich123 5y ago
  On Cloudflare, JavaScript is a JITted language, not strictly interpreted (it uses V8). Code that's seeing any amount of traffic gets lowered into native code pretty darn quick & V8's profile-guided optimizer is pretty impressive at removing JS overheads. Rust is statically compiled but it runs under WASM. If I'm not mistaken, WASM is also typically interpreted & then JIT'ed to native code (at least it is on V8 IIRC). My guess would be that Fastly lowers it to native code AOT (or at least does a fair bit of optimization) but I'm not 100% sure. The question is whether Rust WASM will beat CF JS and that part is less clear because the workloads you'd choose for JS vs Rust are so different. Also, these platforms aren't static so it'll be interesting to watch the competition play out as we both push the boundaries of Edge compute. Disclaimer: I work at Cloudflare on R2. I think both teams do great technical work but I'm more excited about CF. I think our suite of Edge compute products built on top of Workers is stronger (cron triggers, unbound, durable objects & many other interesting things coming).
acdha 5y ago
> I can't really square the idea that the 50-150ms time-delays in question comes down to the actual programming language, but it is absolutely believable that a longer test reduces the median latency rather than a high load test for a shorter duration. It seems plausible to me: in Fastly's case, they're using WebAssembly via wasmtime[1] which does support AOT compilation but most JavaScript code is dynamic enough that they still need a runtime JIT engine. I believe the the current approach Fastly is using is to compile Mozilla's SpiderMonkey JIT engine itself into WebAssembly and they've done some really nice work making that load as quickly as possible: https://bytecodealliance.org/articles/making-javascript-run-fast-on-webassembly https://bytecodealliance.org/articles/making-javascript-run-... The catch, of course, is that this still leaves a fair amount work which a JavaScript program has to deal with at runtime compared to a Rust program which the compiler can spend minutes optimizing long before deployment. This is a classic tradeoff for dynamic languages and a lot of people are satisfied with the approach of defaulting to faster developer turnaround and later converting hot spots to something like C or Rust, but I think it's definitely dodgy to use a single example of something you know to be this complex and present the results as generally representative of the entire platform. I have no knowledge of how Cloudflare ran their tests or reason to suspect malice but not disclosing test methodology and a “no benchmarks” license clause is going to make accusations about the results inevitable. Veterans of the benchmarketing wars have stories about comparisons where someone used a configuration option which happened to disable key performance optimizations for their competitors or used examples which favored their design decisions. Since nobody can read someone's mind to tell their real intent, the best way to avoid acrimony is to have full public disclosure of the benchmarks and their methodology — and to allow other people to conduct benchmarks so they can reproduce results or fill gaps. 1. https://github.com/bytecodealliance/wasmtime https://github.com/bytecodealliance/wasmtime

xwdv 5y ago

This vicious attack by Fastly on Cloudflare will not go unnoticed. IMO Fastly should prepare for the retaliation.

kailanb 5y ago
Vicious attack? Or defending your place in the market?
- floatingatoll 5y ago
  Based on their comment history, I interpreted this as unmarked-sarcasm slash pithy-witty driveby, that isn't meant to be taken seriously or contribute usefully to the discussion.
Twirrim 5y ago
Cloudflare started it by attacking Fastly in the first place.
- NicoJuicy 5y ago
  Benchmarking isn't attacking...

redm 5y ago

I think statistics like these tests are easy to sway in your favor, hence why Fastly is responding. Its not clear that Fastly didn't do the same thing, but a good ole' CDN rivalry does make for a good read. Someone get more mud to sling.

deft 5y ago

point #2 is silly. So basically your product is slower but its ok cuz its a beta only... Fine. But then cf did not mislead anyone. Your beta is slow.

invisible 5y ago
Not really. Cloudflare could have used their rust implementation for doing a fair comparison, instead they compared a beta product to a stable product. It's not like CF didn't have a choice to make a fair comparison.
- etchalon 5y ago
  Did CF pretend they were benchmarking something other than the product they said they were benchmarking?
  - invisible 5y ago
    > Today, we’re excited to report that Cloudflare Workers is 196% faster than Fastly’s Compute@Edge based on the time to first byte from the tests we ran on 50 nodes using Catchpoint’s data from across the world. Kind of? At minimum they made misleading representations. They used abbreviated explanations of how they're better and then expanded on what the test actually consisted of later on.
    - unityByFreedom 5y ago
      > At minimum they made misleading representations Fastly does not have a production-ready JavaScript product. From their blog post, > Their tests compare JavaScript running on Cloudflare Workers, a mature, generally available product, with JavaScript running on Compute@Edge. Although the Compute@Edge platform is now available for all in production, support for JavaScript on Compute@Edge is a beta product. We clearly identify in our documentation [2] that beta products are not ready for production use. A fairer test on this point would have compared Rust on Compute@Edge with JavaScript on Cloudflare Workers, which are at more comparable stages of the product lifecycle. Restricting any comparison isn't satisfactory since we care about both the supported languages and performance. Let there be writeups about both and allow users to make up our own minds. I think the Cloudflare article [2] could be amended to refer to the compared product as "Fastly's JavaScript on Compute@Edge" and all would be fine. Fastly will probably still say they advise against this comparison since it's "beta". Nonetheless, we should all feel free to compare publicly available products and write about them. [1] https://docs.fastly.com/products/fastly-product-lifecycle#:~:text=Fastly%20strongly%20advises,Terms%20of%20Service https://docs.fastly.com/products/fastly-product-lifecycle#:~.... [2] https://blog.cloudflare.com/network-performance-update-full-stack-week/ https://blog.cloudflare.com/network-performance-update-full-...

jjeaff 5y ago

As mentioned in the article, Cloudflare expressly prohibits benchmarking their services in the tos. I think it's rather disingenuous for Cloudflare to publish their own benchmarks calling out competitors when they won't allow anyone to run their own tests and comparisons.

charcircuit 5y ago
They don't prohibit it outright. They just require you to ask first. The article made no mention of if they asked or not.
- deadmutex 5y ago
  Why tho?
  - hysan 5y ago
    Probably for the same reason that I've seen processes put in place for performance testing a service (internally or on production) - because it puts a significant load on the service and automatic rate limiting, flagging, etc can be triggered.
  - dekhn 5y ago
    if you benchmark your competitor, your competitor should have the ability to provide tuning so their numbers look as good as possible, before you run to the press.
    - deadmutex 5y ago
      Unless your competitor was offering a complimentary service to do that for all customers (tuning) then perhaps it shouldn't be done. I believe the idea of benchmarking is to get a sense of how the service would perform for a customer. Consider the following scenario: You calling your ISP that you're going to do a benchmark. Then the ISP gives you the best service, at the expense of other customers bandwidth, for the duration of the test. Then after the test, the ISP removes the QoS modification, and reverts to the old behavior. That could end up with a biased, unrealistic result.

jbergstroem 5y ago

In the context of "should I use Cloudflare or Fastly as my edge", I lean Cloudflare. That said, I enjoyed listening to the GraphCDN crew choosing Fastly[1] – one crucial feather in the hat being cache invalidation[2] (miss ya, Phil Karlton) – and it sounds like a solid choice. [1]: https://www.youtube.com/watch?v=lpmpTJc_SP0 https://www.youtube.com/watch?v=lpmpTJc_SP0 [2]: https://graphcdn.io/docs/how-to/purge-the-cache https://graphcdn.io/docs/how-to/purge-the-cache

tylermenezes 5y ago
I've used both Cloudflare and Fastly for many years, and the cache invalidation at Fastly is a way bigger deal than people think. They're not competing for a 5% improvement in performance, they're competing with a totally different set of features which mean you can cache 80% more things, as long as you design for it. (Which I'm sure is why they don't aggressively compete on free plans -- if you try to just throw it in front like Cloudflare you won't see much of a difference from Cloudflare.) With some careful design to take advantage of it, I've gotten 5-10x faster page loads using Fastly. (The last time I pointed this out, people accused me of being a sockpuppet account. Feel free to Google me!)
- snowwrestler 5y ago
  Yes. Caching is a different animal when you can count on fast, reliable, targeted invalidation. You can cache basically everything with super long TTLs, and just sniper-shot individual cache objects as soon as the backend updates.
- deleted 5y ago
  [deleted]
- sadsoul21 5y ago
  I'm curious, what "totally different set of features" are you referring to?
  - tylermenezes 5y ago

breakingcups 5y ago

Cloudflare not allowing benchmarks in their TOS is very sketchy, that puts them in the same tier as Oracle. Cloudflare have pulled enough shady stuff now that they've fallen out of my favor. Their generous free product bought them a lot of community goodwill but their real face has been showing the past few years.

pm90 5y ago
I would give them a bit more wiggle room before giving up. We’re currently living under a monopoly of AWS (GCP is the most serious “competitor”, still far far behind in market share) and I would love to see more competition in this space. That said: this is definitely not the kind of behavior I expect from them and I’m hoping there’s a better reason than them not willing to submit their products to an impartial evaluation.
- varsketiz 5y ago
  You are forgetting Azure, which is much bigger than GCP.
  - pm90 5y ago
    I’m not forgetting Azure, it’s a deliberate omission because it is not a Tier 1 cloud product. It might be better than OCI or IBM Cloud but it simply does not have the maturity of either AWS or GCP.
    - robertlagrant 5y ago
      Interesting. As someone who's only really used Azure (other than a little GCP) - what's missing?
- latch 5y ago
  AWS is a monopoly if you're living in a HN bubble. The vast majority of "hosting" (internet or otherwise) doesn't use AWS. Large telco's (Telefonica, AT&T, China Unicom, ...) have numerous data centers. Conglomerates (Tata, ...) have numerous data centers. You have wholesalers (Equinix, Digital Realty, ...). You have dedicated providers (OVH, Hetzner, PhoenixNap, ...). VPS' (DO, Linode, Vultr,...) and shared hosting (GoDaddy, ...) Then you have a lot of private data centers, e.g. I worked at a major bank that had its own datacenters (multiple around the world). Admittedly, these are super small compared to an AWS DC, but there's literally thousands of these around the world (I know for banking and government, but I assume other industries do this too).

deleted 5y ago

[deleted]

merb 5y ago

btw. for what it's worth their javascript to wasm is opensource: - https://github.com/fastly/js-compute-runtime https://github.com/fastly/js-compute-runtime - https://github.com/tschneidereit/spidermonkey-wasi-embedding https://github.com/tschneidereit/spidermonkey-wasi-embedding and besides that it is slower than nodejs it is still plenty fast (no matter that it is not as fast as they want) btw. it's startup is faster than node. (maybe better pgo might help) its insane what they built, to do it and it will bring the whole wasm community forward if they succeed (and I hope they do)

NicoJuicy 5y ago
Gentle reminder: That's mostly because they acquired the team behind wasm. Which was already an opensource project at Mozilla. Post on HN: https://news.ycombinator.com/item?id=24897641 https://news.ycombinator.com/item?id=24897641
- tschneidereit 5y ago
  It's really the other way around: we joined Fastly because we knew it's a place where we could do this kind of work in the open. None of the code involved here existed a year ago, and none of it was somehow forced to be open source. (Also, the code in these two repositories is in many ways the most boring part of this solution. See this blog post for an excellent overview of the more interesting parts: https://bytecodealliance.org/articles/making-javascript-run-fast-on-webassembly https://bytecodealliance.org/articles/making-javascript-run-...)
  - NicoJuicy 5y ago
    The article I linked too ( and the hn comments) in my original post seemed that fastly acquired the wasm team. Sorry if I was mistaking.
    - tschneidereit 5y ago
      Oh, no worries at all! While there was no acquisition involved, a whole group of folks working on WebAssembly at Mozilla (myself included) moved to Fastly last Fall. What I tried to emphasize is that instead of the projects at hand here being open source because they somehow had to be, we all joined Fastly because it's a place where this kind of project can be made open source (and be created in the first place!) :)

kentonv 5y ago

OK... So, I'm the tech lead for Cloudflare Workers. In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don't know anything about either beyond what's publicly visible. But as long as we're already mud slinging, I'd like to take the opportunity to get a little something off my chest. Fastly has been trumpeting for years that Compute@Edge has 35 microsecond cold starts, or whatever, and repeatedly posting blog posts comparing that against 5 milliseconds for Workers, and implying that they are 150x faster. If you look at the details, it turns out that 35 microsecond time is actually how long they take to start a new request, given that the application is already loaded in memory. A hot start, not a cold start. Whereas Workers' 5ms includes time to load the application from disk (which is the biggest contributor to total time). Our hot start time is also a few microseconds, but that doesn't seem like an interesting number? We never called this out, it didn't seem worth arguing over. But excuse me if I'm not impressed by claims of false comparisons... On a serious note, I've been saying for decades that benchmarks are almost always meaningless, because different technologies will have different strengths and weaknesses, so you usually can't tell anything about how your use case will perform unless you actually test that use case. So, I would encourage everyone to run your own test and don't just go on other people's numbers. It's great that Fastly has opened up C@E for self-service testing so that people can actually try it out.

zimmerx 5y ago
I think this is all super fair commentary and very balanced. As a Compute@Edge user though how do they test CloudFlare when the ToS says otherwise? Test and break the ToS and hope for the best? It's a bit of a shame that it exists.
refulgentis 5y ago
I suggest deleting this. I'm a completely neutral third-party with little interest in the subject at hand, and their post doesn't come off like mud-slinging, at all. This then makes your response look accusatory and helps further a broader narrative post about Cloudflare's behavior here, which it seems your goal is to dispel, given your statement on whether engineering was involved.* A suggestion for something actionable that'll make things better in this difficult moment: email product counsel about starting the process to override your EULA and enable Compute@Edge to do benchmarking. It'll take forever and it's the only way to have an objective, evolving conversation about benchmarks, which your leads will (at least, should) want after this happened. * intentionally vague here, in order to give you space to delete and leave no record

deleted 5y ago

[deleted]

rileymat2 5y ago

> Cloudflare used a free Fastly trial account to conduct their tests. Free trial accounts are designed for limited use compared to paid accounts, and performance under load is not comparable between the two. https://www.fastly.com/pricing/ https://www.fastly.com/pricing/ does not make a distinction it says that "Any size company looking to give Fastly a try." They make distinctions on paying for more bandwidth, not that they are giving you an inferior product performance wise.

flumpcakes 5y ago
> They make distinctions on paying for more bandwidth, not that they are giving you an inferior product performance wise. I think this is a non-sequitur. Bandwidth is a fundamental property of the performance of a system, if you can "pay for more bandwidth" then you are literally getting an inferior product when you do not pay. Consider the implementation for "paying for more bandwidth". I assume that it is the same copper cabling hitting the same switches for fee paying and non-fee paying customers so the only way to "offer more bandwidth" is to implement some kind of quality-of-service metric on the network. This is explicitly giving the non-fee paying customer an inferior product.
- judge2020 5y ago
  > Test up to $50 of traffic for free. Pay as you go from there with bandwidth pricing. Looks like it's just post-use monitoring that'll cut off the account after it exceeds the trial limit or start billing per-GB; it doesn't say 'slower connection', so CF didn't realize that a trial account would affect their benchmarks regarding Compute@Edge
- rileymat2 5y ago
  People have stopped using "bandwidth" as we used to. It now means a total sum of the amount transferred in a lot of cases, not the "width" of the pipe. It is more evident here: https://www.fastly.com/pricing/#bandwidthpricing https://www.fastly.com/pricing/#bandwidthpricing (To be clear, I agree with your definition, but it is not used)

alophawen 5y ago

<--- CRIPPLE FIGHT

deleted 5y ago

[deleted]

atom_arranger 5y ago

As a consumer it’s great that both companies are benchmarking. Even if Cloudflare’s was flawed. Maybe a case of, the best way to get the right answer is to post the wrong answer. We’re at least moving towards objective comparisons, talking about numbers. Cloudflare prohibiting benchmarking is not good, obviously.

rkwasny 5y ago

Popcorn time ;) Let's wait for @jgrahamc to wake up Simple solution is for Cloudflare to change TOS to allow benchmarking and for 3rd party to publish a reproducible benchmark suite. However TBH benchmarking a distributed system like Cloudflare/Fastly is going to be REALLY HARD. There are 1000s of servers involved and reproducing the results might be impossible.

ksec 5y ago
>Popcorn time ;) Indeed. We already have comments claiming Fastly are dismissive and evasive while others claiming the fault of Cloudflare. The community is divided, which is quite usual on recent HN. Let there be another holy war of CDN between Cloudflare and Fastly.
- unityByFreedom 5y ago
  > The community is divided I don't see any division. It's clear to me who has experience to back up their words and who doesn't.
- saagarjha 5y ago
  Ok, but let’s ideally not have the comments from that war wash up here.
raggi 5y ago
It's a decent use case for planetlab.
- ethbr0 5y ago
  And reproducible benchmarks with cached, publicly accessible results. At some point, you either have to make free tiers worse, or deal with thousands of people using them to run the same benchmark over and over. Cloudflare loves alliances: start a distributed benchmarking alliance! :)
unityByFreedom 5y ago
> Simple solution is for Cloudflare to change TOS to allow benchmarking and for 3rd party to publish a reproducible benchmark suite. Why doesn't Fastly just gasp ask Cloudflare about performing a benchmark? If they say no, publish that. Fastly does not publish their pricing [1]. And, you need to contact a rep to sign up for commercial use [2], which implies everyone gets a different deal. It's absurd to say that contacting Cloudflare to do benchmarking means it's impossible for them to do a comparison, > [FASTLY BLOG] it's impossible for us to define our own terms for a comparison because Cloudflare’s terms of use prohibit benchmarking of their services >> [CLOUDFLARE TERMS] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent; That is not so different from Fastly's own sign-up process. [1] https://www.fastly.com/pricing/ https://www.fastly.com/pricing/ [2] https://news.ycombinator.com/item?id=29468455 https://news.ycombinator.com/item?id=29468455

technobabbler 5y ago

I was expecting a point-by-point comparison from Fastly. Instead, this comes off incredibly defensive and evasive, basically boiling down to two irrelevant points: * Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN) * Fastly performance throttles their free accounts (ok, but not Cloudflare's fault, and I definitely don't want to get into your pricing game if you're going to start offering different tiers of serverless workers... part of Cloudflare's allure is its simplicity in pricing). They do have two good points: * Benchmarks between CDNs should be done from the same client locations (fine, but why did you then use your own set) * Cloudflare should not prohibit benchmarking of their products (fair) But this whole post read like a feeble attempt at misdirection, and made me distrust Fastly as a company. The complaints about their methodology aren't solved by using your own, even more flawed, methodology that doesn't even use the same language. You could've just emailed them and asked to work together on an apple-to-apples test instead of creating an even more flawed benchmark.

tylerhou 5y ago
> * Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN) First, the JavaScript is compiled to hot bytecode (WASM-equivalent) by the VM, so the performance difference is smaller than you would expect than the normal performance difference between Rust and JavaScript. Second, when you’re measuring milliseconds for response latencies, the difference between JS taking 10us CPU and Rust-wasm taking 1us is negligible. And system overheads far outweigh the actual computation time.
- sroussey 5y ago
  > First, the JavaScript is compiled to hot bytecode (WASM-equivalent) by the VM, so the performance difference is smaller than you would expect than the normal performance difference between Rust and JavaScript. Can you explain this? JS intermediate is wildly different from WASM, not to mention WASM requires a max memory album and a lot of other things resulting from the different VMs
  - 5y ago

imilk 5y ago

This comes across as incredibly defensive in tone from Fastly. They could have just reported the results of their test. But calling their competitors outright liars reflects worse on them than anyone else.

bloodyplonker22 5y ago
They do have to punch up because they are the little guy in marketing, but they did it in a very sloppy manner with shaky points.
shah0048 5y ago
When you're sucker punched with misleading stats, ya kind have to hit back. And didn't sound defensive at all, btw.

thecompilr 5y ago

Awesome, now where do I sign for the free fastly account? Oh right, they only do enterprise, so who the hell cares? Cloudflare OTOH caters for the whole spectrum.

yeskia 5y ago

Only semi-related, but why does Cloudflare perform so poorly in Oceania? I've noticed less than stellar performance in Oceania with Cloudflare's CDN, so to see similar stats with workers makes it seem it's not a product-specific issue.

plasma 5y ago
I'm based Australia and have used Cloudflare for personal/business; if you aren't on the $200/month plan for your website then you don't get (consistent) access to the Australian-based servers for your visitors (they may be routed to eg Singapore instead during some times of the day) which actually hurts performance if your service is located in (say) AWS/Azure Australia region, at least for the CDN aspect, I assume the same for Workers too.
- kiwijamo 5y ago
  Same here in New Zealand too. Cloudflare peers only with certain ISPs. The ones they don't peer with sees some traffic (e.g. sites on free plans) routed to offshore Cloudflare POPs around the world instead of local Cloudflare POPs. I understand if the website is on a paid plan they will be more likely to route to a local POP (incurring more costs in doing so).
  - celsoazevedo 5y ago
    According to this 2016 post, the problem is Optus and Telstra, and the costs of peering with them (scroll down to "Oceania"): https://blog.cloudflare.com/bandwidth-costs-around-the-world/ https://blog.cloudflare.com/bandwidth-costs-around-the-world...
LordAtlas 5y ago
I'm in India and if you're on the free CF plan, you often seem to get served from the Singapore POP instead of the Indian ones. I've seen other people on various groups complain about this too.

foton1981 5y ago

Seriously, who cares about 5ms difference? Or even 25ms. I don't. And Asia is just used to longer latencies. Edge compute became a commodity even before it was born.

jlund-molfese 5y ago
I’d love to be able to (affordably) spin up a low-latency Windows desktop for a few hours at a time, that I could use for applications which don’t run well on my bottom-tier M1 Mac
cortesoft 5y ago
The whole point of edge workers is to be able to provide very low latency all around the world... once you have that, it opens up whole new classes of workloads. Yes, 25ms doesn't matter for most current use cases.... but that is partly because the use cases that required lower latency weren't possible before.

ddtaylor 5y ago

That is pretty scummy that Cloudflare won't allow benchmarks.

civilized 5y ago

It's been hilarious watching competing companies publish warring blog posts declaring each other full of shit. I mean, I knew, but thanks for airing each others' dirty underpants.

gary_0 5y ago
It reminds me of being on Slashdot 15 years ago. We've even got one of the engineers posting in the comment thread, starting their own little flamewar. The names of the characters have changed, but there's still a bit of the same old Internet here.
- unityByFreedom 5y ago
  The fact that a lead engineer and CEO took the time to respond here is clarity not a flamewar. edit maybe I'm wrong, https://news.ycombinator.com/item?id=29469174 https://news.ycombinator.com/item?id=29469174
  - gary_0 5y ago
    I did say little flamewar. :)
    - kentonv 5y ago
      I definitely started some sort of a flamewar but somehow not the one I was trying to start.
dmw_ng 5y ago
Most insightful comment here IMHO. Vaguely reminiscent of the Glasgow Ice Cream Wars, I'd rather not buy from anyone involved
- deleted 5y ago
  [deleted]
unityByFreedom 5y ago

boynamedsue 5y ago

Cloudflare seems to encourage benchmarks of Workers Sites: https://blog.cloudflare.com/workers-sites/ https://blog.cloudflare.com/workers-sites/

JediPig 5y ago

I am glad I said no to working for both. When I saw both are politically active and showed child like behavior during the interviews, I said no, and decided to dedicate some of my time to dog and cat rescue.

deleted 5y ago
[deleted]

soheil 5y ago

I'm regularly surprised why Cloudflare gets so much love specially here on HN. Cloudflare is slow, it man-in-the-middles websites, it shows a redirect page randomly for no apparent reason on websites that it powers, it does questionable things at the DNS level, it randomly shows captchas for no apparent reason, the list goes on. Almost every website nowadays uses this monstrosity. We're handing over the internet on a silver platter to them. I think there are better alternatives and I'm so glad at least Fastly is debunking their bs experiment.

celsoazevedo 5y ago
> it man-in-the-middles websites That's just how reverse proxies work. I'm not saying that this is good, but anyone offering the same service is doing the same thing.
sgtfrankieboy 5y ago
>no apparent reason A big reason is making sure the website behind it stays online. Having to deal with weekly/daily DDOS attacks yourself takes a lot of time.

speedgoose 5y ago

I also read that Fastly is slower at specific point of times during a week, slower for free test accounts, slower at executing javascript (hence the switch to rust), and slower in locations carefully selected by CloudFlare. I'm not saying that the benchmark from CloudFlare is fair, but it's a bit funny that their response contain a "let's switch to rust" or "let's benchmark next to our datacenters".

kamray23 5y ago
I mean they switched to Rust because a comparison of two fully developed systems is fairer than a production system and one in testing. Besides, I believe the blog post said that they selected nodes at actual random?

tomasreimers 5y ago

"A fairer test on this point would have compared Rust on Compute@Edge with JavaScript on Cloudflare Workers, which are at more comparable stages of the product lifecycle." Are there any widely agreed upon benchmarks on how emscriptened rust compares to untyped js? It feels like a large portion of the reported Delta boils down to that.

chinathrow 5y ago

This reads as someone who his losing lots of customers to CF and whines about it.

deleted 5y ago

[deleted]

qwertyuiop_ 5y ago

Another appendage measuring contest.

seando-vajina 5y ago

wgaf

nickdothutton 5y ago

The “DeWitt Clause” or sometimes also known as the “Oracle Clause” has been a fixture of ToS for years. The fact that it features in Cloudflare’s docs only serves to remind us that entrenched practices die hard. https://www.brentozar.com/archive/2018/05/the-dewitt-clause-why-you-rarely-see-database-benchmarks/ https://www.brentozar.com/archive/2018/05/the-dewitt-clause-...