3 ms·

I was expecting a point-by-point comparison from Fastly. Instead, this comes off incredibly defensive and evasive, basically boiling down to two irrelevant poin

I was expecting a point-by-point comparison from Fastly. Instead, this comes off incredibly defensive and evasive, basically boiling down to two irrelevant points: * Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN) * Fastly performance throttles their free accounts (ok, but not Cloudflare's fault, and I definitely don't want to get into your pricing game if you're going to start offering different tiers of serverless workers... part of Cloudflare's allure is its simplicity in pricing). They do have two good points: * Benchmarks between CDNs should be done from the same client locations (fine, but why did you then use your own set) * Cloudflare should not prohibit benchmarking of their products (fair) But this whole post read like a feeble attempt at misdirection, and made me distrust Fastly as a company. The complaints about their methodology aren't solved by using your own, even more flawed, methodology that doesn't even use the same language. You could've just emailed them and asked to work together on an apple-to-apples test instead of creating an even more flawed benchmark.

tylerhou 5y ago
> * Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN) First, the JavaScript is compiled to hot bytecode (WASM-equivalent) by the VM, so the performance difference is smaller than you would expect than the normal performance difference between Rust and JavaScript. Second, when you’re measuring milliseconds for response latencies, the difference between JS taking 10us CPU and Rust-wasm taking 1us is negligible. And system overheads far outweigh the actual computation time.
- sroussey 5y ago
  > First, the JavaScript is compiled to hot bytecode (WASM-equivalent) by the VM, so the performance difference is smaller than you would expect than the normal performance difference between Rust and JavaScript. Can you explain this? JS intermediate is wildly different from WASM, not to mention WASM requires a max memory album and a lot of other things resulting from the different VMs
  - tylerhou 5y ago
    Assuming that we're using V8 and the function is hot, both the JS code and the WASM will be compiled to optimized bytecode by V8's TurboFan optimizing compiler. The two big reasons why JS is inefficient vs Rust+WASM is because (1) everything is untyped -- without being able to statically resolve properties, the compiler is forced to resort to string lookups and (2) garbage collection. Compiling Rust to WASM will emit code that uses Rust's type system and lifetime analysis to create direct accesses and deallocate objects when necessary. For such a small program (write the current time to a socket), it's trivial for TurboFan to examine the types of the JS code at runtime [1]. Also, escape analysis should also be trivial [2]. So, at runtime, the optimizing compiler can (1) replace string lookups with very fast address dereferences, and (2) insert appropriate deallocations. My guess is that with these two optimizations the performance difference can be reduced significantly. (Of course there is still some overhead -- deoptimization checks, etc. -- but it's not nearly as big as between a 5 MB JS file vs the same amount of code in Rust.) [1] E.g. https://v8.dev/blog/fast-properties https://v8.dev/blog/fast-properties [2] https://www.jfokus.se/jfokus18/preso/Escape-Analysis-in-V8.pdf https://www.jfokus.se/jfokus18/preso/Escape-Analysis-in-V8.p...
    - saagarjha 5y ago
      > For such a small program (write the current time to a socket), it's trivial for TurboFan On the contrary, for short programs the JavaScript engine may not choose to do anything at all beyond run it through the baseline interpreter.
    - josephg 5y ago
      > For such a small program ... the performance difference can be reduced significantly. I think you're wrong. Small programs don't necessarily minimize the difference between languages. Small programs often exacerbate them. Often for weird, idiosyncratic reasons. Especially if you're measuring time-to-first-byte, and not letting the VM warm up. For example, in this case I wouldn't be surprised if most of the CPU time was going into node + v8's startup, rather than executing the user's javascript at all. Look at the plaintext techempower benchmarks[1]. These benchmarks test tiny amounts of code - the programs only have to respond with "hello world" HTTP responses. The best performers are hitting hardware limits. If your theory that small program = small relative cost of using javascript was true, nodejs's performance should be similar to rust / C++. It is not - node is only 13% of the performance of the top performers. Weirdly "justjs" (another v8 based runtime) is ~5x faster than nodejs on this test. The reason probably has nothing to do with javascript at all, and is because justjs has less overhead and talks to the OS in a more efficient way. (It probably uses io_ring.) But maybe this is evidence you're technically correct. The performance differences can be reduced. But we have every reason to assume nodejs will have way more overhead than rust at executing the same code. In all, I agree with other posters. A benchmark showing rust on fastly is faster than javascript on cloudflare tells us nothing about the underlying hosting providers. https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=plaintext https://www.techempower.com/benchmarks/#section=data-r20&hw=...
ksec 5y ago
>I was expecting a point-by-point comparison from Fastly. Instead, this comes off incredibly defensive and evasive, basically boiling down to two irrelevant points: Because it is not meant as a comparison but they are the one defending a benchmarks ran by their competitors?
makeitdouble 5y ago
> * Fastly performance throttles their free accounts (ok, but not Cloudflare's fault, and I definitely don't want to get into your pricing game if you're going to start offering different tiers of serverless workers... part of Cloudflare's allure is its simplicity in pricing). TBF, people who care about benchmarks usually expect high loads and value performance enough to put sufficient money on the table. I'd see the free tier people having different priorities. Sure, it's still valuable info to know what's the free account limits are, but it shouldn't be headline news (except if it's exceptionally fast...)
acdha 5y ago
> Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN) This was also addressed in the post: they don’t consider the JavaScript flavor production ready in part due to performance. Even if you’re an all JavaScript shop which will never consider something else, comparing a mature released service to one under development makes the comparison harder. It would have been interesting and more honest to compare other language runtimes to see whether there’s a trend for the entire platform or simply one language runtime.

tylerhou 5y ago

> * Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN) First, the JavaScript is compiled to hot bytecode (WASM-equivalent) by the VM, so the performance difference is smaller than you would expect than the normal performance difference between Rust and JavaScript. Second, when you’re measuring milliseconds for response latencies, the difference between JS taking 10us CPU and Rust-wasm taking 1us is negligible. And system overheads far outweigh the actual computation time.

sroussey 5y ago
> First, the JavaScript is compiled to hot bytecode (WASM-equivalent) by the VM, so the performance difference is smaller than you would expect than the normal performance difference between Rust and JavaScript. Can you explain this? JS intermediate is wildly different from WASM, not to mention WASM requires a max memory album and a lot of other things resulting from the different VMs
- tylerhou 5y ago
  Assuming that we're using V8 and the function is hot, both the JS code and the WASM will be compiled to optimized bytecode by V8's TurboFan optimizing compiler. The two big reasons why JS is inefficient vs Rust+WASM is because (1) everything is untyped -- without being able to statically resolve properties, the compiler is forced to resort to string lookups and (2) garbage collection. Compiling Rust to WASM will emit code that uses Rust's type system and lifetime analysis to create direct accesses and deallocate objects when necessary. For such a small program (write the current time to a socket), it's trivial for TurboFan to examine the types of the JS code at runtime [1]. Also, escape analysis should also be trivial [2]. So, at runtime, the optimizing compiler can (1) replace string lookups with very fast address dereferences, and (2) insert appropriate deallocations. My guess is that with these two optimizations the performance difference can be reduced significantly. (Of course there is still some overhead -- deoptimization checks, etc. -- but it's not nearly as big as between a 5 MB JS file vs the same amount of code in Rust.) [1] E.g. https://v8.dev/blog/fast-properties https://v8.dev/blog/fast-properties [2] https://www.jfokus.se/jfokus18/preso/Escape-Analysis-in-V8.pdf https://www.jfokus.se/jfokus18/preso/Escape-Analysis-in-V8.p...
  - saagarjha 5y ago
    > For such a small program (write the current time to a socket), it's trivial for TurboFan On the contrary, for short programs the JavaScript engine may not choose to do anything at all beyond run it through the baseline interpreter.
  - josephg 5y ago
    > For such a small program ... the performance difference can be reduced significantly. I think you're wrong. Small programs don't necessarily minimize the difference between languages. Small programs often exacerbate them. Often for weird, idiosyncratic reasons. Especially if you're measuring time-to-first-byte, and not letting the VM warm up. For example, in this case I wouldn't be surprised if most of the CPU time was going into node + v8's startup, rather than executing the user's javascript at all. Look at the plaintext techempower benchmarks[1]. These benchmarks test tiny amounts of code - the programs only have to respond with "hello world" HTTP responses. The best performers are hitting hardware limits. If your theory that small program = small relative cost of using javascript was true, nodejs's performance should be similar to rust / C++. It is not - node is only 13% of the performance of the top performers. Weirdly "justjs" (another v8 based runtime) is ~5x faster than nodejs on this test. The reason probably has nothing to do with javascript at all, and is because justjs has less overhead and talks to the OS in a more efficient way. (It probably uses io_ring.) But maybe this is evidence you're technically correct. The performance differences can be reduced. But we have every reason to assume nodejs will have way more overhead than rust at executing the same code. In all, I agree with other posters. A benchmark showing rust on fastly is faster than javascript on cloudflare tells us nothing about the underlying hosting providers. https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=plaintext https://www.techempower.com/benchmarks/#section=data-r20&hw=...

ksec 5y ago

>I was expecting a point-by-point comparison from Fastly. Instead, this comes off incredibly defensive and evasive, basically boiling down to two irrelevant points: Because it is not meant as a comparison but they are the one defending a benchmarks ran by their competitors?

makeitdouble 5y ago

> * Fastly performance throttles their free accounts (ok, but not Cloudflare's fault, and I definitely don't want to get into your pricing game if you're going to start offering different tiers of serverless workers... part of Cloudflare's allure is its simplicity in pricing). TBF, people who care about benchmarks usually expect high loads and value performance enough to put sufficient money on the table. I'd see the free tier people having different priorities. Sure, it's still valuable info to know what's the free account limits are, but it shouldn't be headline news (except if it's exceptionally fast...)

acdha 5y ago

> Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN) This was also addressed in the post: they don’t consider the JavaScript flavor production ready in part due to performance. Even if you’re an all JavaScript shop which will never consider something else, comparing a mature released service to one under development makes the comparison harder. It would have been interesting and more honest to compare other language runtimes to see whether there’s a trend for the entire platform or simply one language runtime.