4 ms·

> _We used a Wasm binary compiled from Rust rather than JavaScript. > We know support for JavaScript is important to many customers, but we're not yet satisf

by gopalv 5y ago

> _We used a Wasm binary compiled from Rust rather than JavaScript. > We know support for JavaScript is important to many customers, but we're not yet satisfied with the performance of Compute@Edge packages compiled from JavaScript. That's why it's in beta. When a product is ready for production, we remove the beta designation. I can't really square the idea that the 50-150ms time-delays in question comes down to the actual programming language, but it is absolutely believable that a longer test reduces the median latency rather than a high load test for a shorter duration. Having said that, I would notice anything over 150ms in my clicks, but wouldn't care whether something took under 100 or under 50 - except that the lower the latency the less the scale needed to serve the same number of active users (it becomes a question of cost rather than response time).

kailanb 5y ago
It definitely comes down to the language — JavaScript is an interpreted language, which requires an engine. Rust is statically compiled and runs by itself.
- azakai 5y ago
  In general you'd be right, but this is not a benchmark of code running in those languages. There is no loop, no heavy computation - it's just returning the headers. That should take less than 1ms in either language. Startup time might be a factor. But the measurements of the same language running on the same provider's infrastructure go from around 10 to 50ms (on one provider) and 10 to 100ms (on the other). Those huge differences don't look like CPU latencies (the machines aren't 5x or 10x faster in different locations) - they are likely network latencies. So programming language is probably a very minor factor here (at least in the non-beta services, which is what was measured here - maybe it was an issue in the older measurements).
  - acdha 5y ago
    I'd also expect unrelated load to be a factor, especially if there's a mechanism which priorities paid customer traffic over the free tier. Given the complexities of loading and running JavaScript, I'd be especially interested in what memory contention looks like — it seems like it could be as simple as Rust needing far less memory than loading all of SpiderMonkey and thus being less likely to hit both low-level CPU cache and high-level runtime cache evictions.
    - azakai 5y ago
      Ah, good point, memory consumption might indeed be a factor here. If that's the case I'd expect higher variance in the measurements perhaps - looking for that in the raw data could be interesting.
  - technobabbler 5y ago
    > In general you'd be right, but this is not a benchmark of code running in those languages. There is no loop, no heavy computation - it's just returning the headers. That should take less than 1ms in either language. They can't have it both ways... Fastly was the one that complained that their JS engine still wasn't up to par (being in beta), so clearly the language of choice (or interpreter/compiler/whatever) DOES matter. Either the language doesn't matter (in which case, replicate the JS test) or it does, in which case this isn't even a useful comparison.
    - azakai 5y ago
      Sorry, I should have been clearer. The new benchmark data does not look like it is influenced by the language in a noticeable way (since we see 5x or 10x differences in the mean even when using the same language, and which provider is faster or slower depends on which location you measure at). But it's very possible the old data may have been influenced by the language. That's not very interesting, though: as Fastly said, it's just beta performance. And anyhow this is a benchmark with almost no CPU work anyhow.
- vlovich123 5y ago
  On Cloudflare, JavaScript is a JITted language, not strictly interpreted (it uses V8). Code that's seeing any amount of traffic gets lowered into native code pretty darn quick & V8's profile-guided optimizer is pretty impressive at removing JS overheads. Rust is statically compiled but it runs under WASM. If I'm not mistaken, WASM is also typically interpreted & then JIT'ed to native code (at least it is on V8 IIRC). My guess would be that Fastly lowers it to native code AOT (or at least does a fair bit of optimization) but I'm not 100% sure. The question is whether Rust WASM will beat CF JS and that part is less clear because the workloads you'd choose for JS vs Rust are so different. Also, these platforms aren't static so it'll be interesting to watch the competition play out as we both push the boundaries of Edge compute. Disclaimer: I work at Cloudflare on R2. I think both teams do great technical work but I'm more excited about CF. I think our suite of Edge compute products built on top of Workers is stronger (cron triggers, unbound, durable objects & many other interesting things coming).
acdha 5y ago
> I can't really square the idea that the 50-150ms time-delays in question comes down to the actual programming language, but it is absolutely believable that a longer test reduces the median latency rather than a high load test for a shorter duration. It seems plausible to me: in Fastly's case, they're using WebAssembly via wasmtime[1] which does support AOT compilation but most JavaScript code is dynamic enough that they still need a runtime JIT engine. I believe the the current approach Fastly is using is to compile Mozilla's SpiderMonkey JIT engine itself into WebAssembly and they've done some really nice work making that load as quickly as possible: https://bytecodealliance.org/articles/making-javascript-run-fast-on-webassembly https://bytecodealliance.org/articles/making-javascript-run-... The catch, of course, is that this still leaves a fair amount work which a JavaScript program has to deal with at runtime compared to a Rust program which the compiler can spend minutes optimizing long before deployment. This is a classic tradeoff for dynamic languages and a lot of people are satisfied with the approach of defaulting to faster developer turnaround and later converting hot spots to something like C or Rust, but I think it's definitely dodgy to use a single example of something you know to be this complex and present the results as generally representative of the entire platform. I have no knowledge of how Cloudflare ran their tests or reason to suspect malice but not disclosing test methodology and a “no benchmarks” license clause is going to make accusations about the results inevitable. Veterans of the benchmarketing wars have stories about comparisons where someone used a configuration option which happened to disable key performance optimizations for their competitors or used examples which favored their design decisions. Since nobody can read someone's mind to tell their real intent, the best way to avoid acrimony is to have full public disclosure of the benchmarks and their methodology — and to allow other people to conduct benchmarks so they can reproduce results or fill gaps. 1. https://github.com/bytecodealliance/wasmtime https://github.com/bytecodealliance/wasmtime

by gopalv 5y ago

kailanb 5y ago

It definitely comes down to the language — JavaScript is an interpreted language, which requires an engine. Rust is statically compiled and runs by itself.

azakai 5y ago
In general you'd be right, but this is not a benchmark of code running in those languages. There is no loop, no heavy computation - it's just returning the headers. That should take less than 1ms in either language. Startup time might be a factor. But the measurements of the same language running on the same provider's infrastructure go from around 10 to 50ms (on one provider) and 10 to 100ms (on the other). Those huge differences don't look like CPU latencies (the machines aren't 5x or 10x faster in different locations) - they are likely network latencies. So programming language is probably a very minor factor here (at least in the non-beta services, which is what was measured here - maybe it was an issue in the older measurements).
- acdha 5y ago
  I'd also expect unrelated load to be a factor, especially if there's a mechanism which priorities paid customer traffic over the free tier. Given the complexities of loading and running JavaScript, I'd be especially interested in what memory contention looks like — it seems like it could be as simple as Rust needing far less memory than loading all of SpiderMonkey and thus being less likely to hit both low-level CPU cache and high-level runtime cache evictions.
  - azakai 5y ago
    Ah, good point, memory consumption might indeed be a factor here. If that's the case I'd expect higher variance in the measurements perhaps - looking for that in the raw data could be interesting.
- technobabbler 5y ago
  > In general you'd be right, but this is not a benchmark of code running in those languages. There is no loop, no heavy computation - it's just returning the headers. That should take less than 1ms in either language. They can't have it both ways... Fastly was the one that complained that their JS engine still wasn't up to par (being in beta), so clearly the language of choice (or interpreter/compiler/whatever) DOES matter. Either the language doesn't matter (in which case, replicate the JS test) or it does, in which case this isn't even a useful comparison.
  - azakai 5y ago
    Sorry, I should have been clearer. The new benchmark data does not look like it is influenced by the language in a noticeable way (since we see 5x or 10x differences in the mean even when using the same language, and which provider is faster or slower depends on which location you measure at). But it's very possible the old data may have been influenced by the language. That's not very interesting, though: as Fastly said, it's just beta performance. And anyhow this is a benchmark with almost no CPU work anyhow.
vlovich123 5y ago
On Cloudflare, JavaScript is a JITted language, not strictly interpreted (it uses V8). Code that's seeing any amount of traffic gets lowered into native code pretty darn quick & V8's profile-guided optimizer is pretty impressive at removing JS overheads. Rust is statically compiled but it runs under WASM. If I'm not mistaken, WASM is also typically interpreted & then JIT'ed to native code (at least it is on V8 IIRC). My guess would be that Fastly lowers it to native code AOT (or at least does a fair bit of optimization) but I'm not 100% sure. The question is whether Rust WASM will beat CF JS and that part is less clear because the workloads you'd choose for JS vs Rust are so different. Also, these platforms aren't static so it'll be interesting to watch the competition play out as we both push the boundaries of Edge compute. Disclaimer: I work at Cloudflare on R2. I think both teams do great technical work but I'm more excited about CF. I think our suite of Edge compute products built on top of Workers is stronger (cron triggers, unbound, durable objects & many other interesting things coming).

acdha 5y ago

> I can't really square the idea that the 50-150ms time-delays in question comes down to the actual programming language, but it is absolutely believable that a longer test reduces the median latency rather than a high load test for a shorter duration. It seems plausible to me: in Fastly's case, they're using WebAssembly via wasmtime[1] which does support AOT compilation but most JavaScript code is dynamic enough that they still need a runtime JIT engine. I believe the the current approach Fastly is using is to compile Mozilla's SpiderMonkey JIT engine itself into WebAssembly and they've done some really nice work making that load as quickly as possible: https://bytecodealliance.org/articles/making-javascript-run-fast-on-webassembly https://bytecodealliance.org/articles/making-javascript-run-... The catch, of course, is that this still leaves a fair amount work which a JavaScript program has to deal with at runtime compared to a Rust program which the compiler can spend minutes optimizing long before deployment. This is a classic tradeoff for dynamic languages and a lot of people are satisfied with the approach of defaulting to faster developer turnaround and later converting hot spots to something like C or Rust, but I think it's definitely dodgy to use a single example of something you know to be this complex and present the results as generally representative of the entire platform. I have no knowledge of how Cloudflare ran their tests or reason to suspect malice but not disclosing test methodology and a “no benchmarks” license clause is going to make accusations about the results inevitable. Veterans of the benchmarketing wars have stories about comparisons where someone used a configuration option which happened to disable key performance optimizations for their competitors or used examples which favored their design decisions. Since nobody can read someone's mind to tell their real intent, the best way to avoid acrimony is to have full public disclosure of the benchmarks and their methodology — and to allow other people to conduct benchmarks so they can reproduce results or fill gaps. 1. https://github.com/bytecodealliance/wasmtime https://github.com/bytecodealliance/wasmtime