6 ms·

Container Runtimes and WasmEdge benchmarking on Arm

82 points by 3Sophons 3y ago

oaiey 3y ago
For me this article is flawed. Hello Worlds application are not good metric foundation to make such a comparison. Also it is no surprising that a container cannot beat a sandboxed programming platform that wasm is. I do not know if there is some mode in Java that could be comparable.
- nunobrito 3y ago
  Yeah. Why bother with wasm when you can write the app from scratch in Java. Boot up time also tends to be irrelevant since the true slowdown comes from initializing other dependencies anyways. Wasm looks relevant for existing apps in C++ that you wish to run from a web browser.
- ridruejo 3y ago
  Meaningful benchmarks are hard. Having said that, in this particular context it is a good metric to measure the latency for the simplest possible function invocation as that is important for serverless at the edge (and why you put those workers/functions as close as possible to the clients in the first place)
jmillikin 3y ago
[flagged]
- 2fast4you 3y ago
  Actually there is support for using WASM as a container runtime. You could in theory have WASM containers and Linux containers next to each other on the same K8s cluster. Not sure that I’d recommend it in production just yet, but it’s certainly usable https://next.redhat.com/2023/02/01/running-webassembly-workloads-on-container-runtimes/ https://next.redhat.com/2023/02/01/running-webassembly-workl...
  - pjmlp 3y ago
    Yeah, re-inventing application servers, badly.
  - jmillikin 3y ago
    > Actually there is support for using WASM as a container runtime. Isn't that what I said? Some container runtimes (such as `runc`) execute a native binary. There are also container runtimes that execute a WASM+WASI binary. So trying to compare WASM with "containers" is like asking "should you deploy your code as x86 or as a container", which isn't a useful question.
    - foota 3y ago
      I think your comparison to CGI is unfair, since CGI in containers is quite different from "classic" CGI imo. Wikipedia describes CGI as "In computing, Common Gateway Interface (CGI) is an interface specification that enables web servers to execute an external program, typically to process user requests.[1] Such programs are often written in a scripting language and are commonly referred to as CGI scripts, but they may include compiled programs.[2] A typical use case occurs when a web user submits a web form on a web page that uses CGI. The form's data is sent to the web server within an HTTP request with a URL denoting a CGI script. The web server then launches the CGI script in a new computer process, passing the form data to it. The output of the CGI script, usually in the form of HTML, is returned by the script to the Web server, and the server relays it back to the browser as its response to the browser's request.[3]" Which bears little resemblance here imo.
      - jmillikin 3y ago
        Is it possible you don't have direct experience with CGI? The typical way to deploy CGI during the '90s was to have a compiled program in C/C++, which would be executed once per request. There were entire frameworks built around this "process per request" model, with various optimizations like pre-warming (start the process, connect to db, wait for request on stdin) and variant protocols (SCGI). Any time you have an execution model based on running a single address space per request, it's fundamentally CGI-ish. It doesn't have anything to do with the implementation language. The limitation of CGI is that state can't be persisted in-process between requests. --- Going back to the thread, there's multiple aspects when thinking about the structure of a hosted service: 1. Whether state is shared between requests (CGI vs long-lived process). 2. Whether the binary is native code (x86/ARM/) or bytecode (WASM/JVM/). 3. Whether the binary is standalone or comes with additional files. 4. If it needs additional files, whether it's distributed as a package (apt/rpm) or chroot (container image). Does breaking those different concepts down into a list help? You can see that trying to compare the performance of "WASM deployed as standalone binary with CGI execution model" and "native code deployed in container image" just isn't meaningful. The article is mixing up too many different ideas when trying to benchmark.
- pjmlp 3y ago
  Promote WASM no matter what, that is the point. And for CGI model, there is a reason why servlets, ISAPI handlers and Apache language modules came to be. Maybe compare those to start with, then sell how great WebAssembly is for server side. /s
- lmm 3y ago
  > WASM vs a container runtime isn't a meaningful comparison because they're at different layers of the stack. A container runtime runs a process tree in a chroot, and whether the process binaries are native code (x86, ARM) or bytecode (JVM, WASM, CLR) is an implementation detail of the runtime. It's a meaningful comparison because they often solve the same business problem; you need to run some code in an isolated/managed environment. Whether that's done through a process tree or a single process is an implementation detail for many purposes; not being able to run multiple processes might have implications for some use cases, but a lot of systems (e.g. docker/kubernetes, or especially anything serverless) are oriented around using a container for a single process.
  - jmillikin 3y ago
    A comparison between WASM and x86/ARM/etc native binaries is meaningful, because those are different formats for executable code. A comparison between WASM and "containers" is meaningless, because the binaries being executed by a container runtime could be written in WASM. The executable format and the packaging format are separate concerns. This might be easier to think about if you imagine the article was written as "native code vs containers" -- what conclusions (if any) could you draw from the benchmarks in an article with that premise?
    - hansvm 3y ago
      A comparison between WASM and "containers" is meaningful because in popular usage there's a cottage industry of WASM-based container solutions with the express goal (and co-evolved with the language to make this possible) of simplifying some of the thornier design decisions that make other containerization solutions easier to get started with (can just drop in any binary and dependencies) but also less performant for some set of tasks (have to set up network cgroups and other garbage). The phrase "WASM" isn't just being used to refer to the actual stack machine.
      - jmillikin 3y ago
        WASM is WebAssembly. Words have meanings. If someone wants to try to advertise some weird custom platform that uses WASM as an intermediate bytecode for its packaging format then that's fine, but they'll need to pick a different name.
- rektide 3y ago
  Things don't have to be the same to compare & contrast. There were dozens of articles a month comparing vm's Vs containers back in the day. I also think your fixed perspective that these are & always will be different levels of the stack is an inadequate fixedness, lacks the mental agility to comprehend many of the things wasm is growing into. For years already, standard libraries like wasi are attempting to become fairly complete platform libraries. Instances of wasm run in effectively process trees of chroots, so the simar parity is apparent. It's absolutely coom sense to be comparing startup times across the two. There have been endless efforts to try to make cold boot containers or container snapshots as fast as possible. And this again extends backwards in time to comparisons versus vm's as well, with efforts like the Xen Zerg demo trying to make vm's fast enough to boot a vm per request. https://news.ycombinator.com/item?id=5243360 https://news.ycombinator.com/item?id=5243360 Getting back to a process per request or process per session model is a holy grail, one that every virtualization/containerization technology has explored. It would greatly alter the application development model & potentially greatly enhance security, could help radically reshape resource management. This article is doing a fine job pointing to the future, to what we are unlocking. We absolutely should compare versus what the technology of today (containers) can do. The comparison is spot on.
- inkyoto 3y ago
  The author has carried out an assessment of how the use of the ARM assisted hardware virtualisation (normally used to isolate and run VM's but not containers) affects different metrics of running different runtimes in a container environment that is aware of and that can take advantage of the hardware virtualisation features: Kata Container is a secure container runtime that uses hardware virtualization technology as an additional layer of defense. This allows it to provide stronger workload isolation while still being lightweight and performing like regular containers. The author uses WASM runtimes as a conduit, and concludes the article with a compilation of results and the impact on each metric, namely: boot times, security, the ease of development in each setup, production readiness amongst others. The author does compare raw computational performance of each option, and it does not appear to be an objective. The article title could have been better worded, though.
- ridruejo 3y ago
  "Nobody is deploying large-scale services on top of what would be essentially the CGI model." Yes, they are. That's basically the Cloudflare workers model (which certainly qualifies as large-scale) and many other V8/Wasm based serverless solutions
  - jmillikin 3y ago
    The article is not describing the Cloudflare approach (v8 isolates) -- note the graph of startup times. Their "wasmedge" line takes 20 seconds to start 1500 instances, or about 13 milliseconds per instance. Cloudflare's blog post at https://blog.cloudflare.com/cloud-computing-without-containers/ https://blog.cloudflare.com/cloud-computing-without-containe... contains a graph in which requests are handled in 200 microseconds, orders of magnitude faster.
    - ridruejo 3y ago
      Same applies to Wasm. See Fastly equivalent of 35 microseconds https://www.fastly.com/edge-cloud-network/serverless#:~:text=With%20a%2035.4%20microsecond%20startup,fast%2C%20always%2Don%20computing https://www.fastly.com/edge-cloud-network/serverless#:~:text....
- jasode 3y ago
  >WASM vs a container runtime isn't a meaningful comparison because they're at different layers of the stack. It's not just comparing pure WASM (bytecode) but the WASM+WASI (runtime) which is what this article's WasmEdge (runtime) is about. It's a meaningful comparison when the original creator of Docker containers also compared them: >Solomon Hykes / @shykes@hachyderm.io @solomonstre >If WASM+WASI existed in 2008, we wouldn't have needed to created Docker. That's how important it is. Webassembly on the server is the future of computing. A standardized system interface was the missing link. Let's hope WASI is up to the task! -- from 2019-03-27t20:39:47 https://twitter.com/solomonstre/status/1111004913222324225 https://twitter.com/solomonstre/status/1111004913222324225
  - jmillikin 3y ago
    > It's not just comparing pure WASM (bytecode) but the WASM+WASI (runtime) WASI isn't a runtime, and it's not a packaging format. It's a standard ABI for a POSIX-like syscall interface, which is useful[0] but doesn't substitute for or relate to container runtimes. You still need a syscall shim to provide the WASI implementation, and you need a WASM VM to run it. > It's a meaningful comparison when the original creator of Docker > containers also compared them: I've seen that quote before and I think it's nonsense. Docker container images are architecture-specific, and it took years before their distribution protocol supported specifying the architecture. If the existence of an architecture-agnostic bytecode would have avoided the need for Docker, then the JVM would have filled that niche perfectly. And if you look at the Docker code, the vast majority is about configuring the runtime environment of the container (networking, bind-mounts, logging) and persisting that state across reboots. There is no syscall shimming like in gVisor or a WASI implementation, it's all about setting up a chroot and then calling `exec()`. If WASM had existed in 2008, maybe the Docker folks would have enabled it as an entry-point format, but you'd still need a way to ship all the other stuff that goes along with the main binary. It's baffling that people feel the need to fabricate ways for WASM/WASI to be important, when the projects themselves are already so interesting. Architecture-agnostic plugins! A clean slate for a capability-based syscall ABI! A genuine chance at running unmodified userland programs in memory-safe sandboxes! Get excited about that, not about someone poorly re-implementing AWS Lambda. [0] Maybe one day most small shell tools will be in WASM+WASI, and the typical Linux distribution will have relatively little native code installed by default.
- tecleandor 3y ago
  The article isn't about WASM vs a container runtime, it's about comparing two container runtimes. WasmEdge is a container runtime designed for running Webassembly If you check the whole article, you'll find the (simple and basic) benchmark comparing a rust "hello world" running with runc, and (apparently) the same code compiled to Wasm and running in WasmEdge.
politelemon 3y ago
> normal (debugging within the container is a problem) That's a medium difficulty then. Kata isn't a great development environment.
jensneuse 3y ago
Has anybody ever seen large scale applications in real life? Or is WASM more like a middleware layer target? I'm genuinely interested in this as I'm yet to see a company using WASM in production.
deleted 3y ago
[deleted]

82 points by 3Sophons 3y ago

For me this article is flawed. Hello Worlds application are not good metric foundation to make such a comparison. Also it is no surprising that a container cannot beat a sandboxed programming platform that wasm is. I do not know if there is some mode in Java that could be comparable.

nunobrito 3y ago
Yeah. Why bother with wasm when you can write the app from scratch in Java. Boot up time also tends to be irrelevant since the true slowdown comes from initializing other dependencies anyways. Wasm looks relevant for existing apps in C++ that you wish to run from a web browser.
ridruejo 3y ago
Meaningful benchmarks are hard. Having said that, in this particular context it is a good metric to measure the latency for the simplest possible function invocation as that is important for serverless at the edge (and why you put those workers/functions as close as possible to the clients in the first place)

jmillikin 3y ago

[flagged]

2fast4you 3y ago
Actually there is support for using WASM as a container runtime. You could in theory have WASM containers and Linux containers next to each other on the same K8s cluster. Not sure that I’d recommend it in production just yet, but it’s certainly usable https://next.redhat.com/2023/02/01/running-webassembly-workloads-on-container-runtimes/ https://next.redhat.com/2023/02/01/running-webassembly-workl...
- pjmlp 3y ago
  Yeah, re-inventing application servers, badly.
- jmillikin 3y ago
  > Actually there is support for using WASM as a container runtime. Isn't that what I said? Some container runtimes (such as `runc`) execute a native binary. There are also container runtimes that execute a WASM+WASI binary. So trying to compare WASM with "containers" is like asking "should you deploy your code as x86 or as a container", which isn't a useful question.
  - foota 3y ago
    I think your comparison to CGI is unfair, since CGI in containers is quite different from "classic" CGI imo. Wikipedia describes CGI as "In computing, Common Gateway Interface (CGI) is an interface specification that enables web servers to execute an external program, typically to process user requests.[1] Such programs are often written in a scripting language and are commonly referred to as CGI scripts, but they may include compiled programs.[2] A typical use case occurs when a web user submits a web form on a web page that uses CGI. The form's data is sent to the web server within an HTTP request with a URL denoting a CGI script. The web server then launches the CGI script in a new computer process, passing the form data to it. The output of the CGI script, usually in the form of HTML, is returned by the script to the Web server, and the server relays it back to the browser as its response to the browser's request.[3]" Which bears little resemblance here imo.
    - jmillikin 3y ago
      Is it possible you don't have direct experience with CGI? The typical way to deploy CGI during the '90s was to have a compiled program in C/C++, which would be executed once per request. There were entire frameworks built around this "process per request" model, with various optimizations like pre-warming (start the process, connect to db, wait for request on stdin) and variant protocols (SCGI). Any time you have an execution model based on running a single address space per request, it's fundamentally CGI-ish. It doesn't have anything to do with the implementation language. The limitation of CGI is that state can't be persisted in-process between requests. --- Going back to the thread, there's multiple aspects when thinking about the structure of a hosted service: 1. Whether state is shared between requests (CGI vs long-lived process). 2. Whether the binary is native code (x86/ARM/) or bytecode (WASM/JVM/). 3. Whether the binary is standalone or comes with additional files. 4. If it needs additional files, whether it's distributed as a package (apt/rpm) or chroot (container image). Does breaking those different concepts down into a list help? You can see that trying to compare the performance of "WASM deployed as standalone binary with CGI execution model" and "native code deployed in container image" just isn't meaningful. The article is mixing up too many different ideas when trying to benchmark.
pjmlp 3y ago
Promote WASM no matter what, that is the point. And for CGI model, there is a reason why servlets, ISAPI handlers and Apache language modules came to be. Maybe compare those to start with, then sell how great WebAssembly is for server side. /s
lmm 3y ago
> WASM vs a container runtime isn't a meaningful comparison because they're at different layers of the stack. A container runtime runs a process tree in a chroot, and whether the process binaries are native code (x86, ARM) or bytecode (JVM, WASM, CLR) is an implementation detail of the runtime. It's a meaningful comparison because they often solve the same business problem; you need to run some code in an isolated/managed environment. Whether that's done through a process tree or a single process is an implementation detail for many purposes; not being able to run multiple processes might have implications for some use cases, but a lot of systems (e.g. docker/kubernetes, or especially anything serverless) are oriented around using a container for a single process.
- jmillikin 3y ago
  A comparison between WASM and x86/ARM/etc native binaries is meaningful, because those are different formats for executable code. A comparison between WASM and "containers" is meaningless, because the binaries being executed by a container runtime could be written in WASM. The executable format and the packaging format are separate concerns. This might be easier to think about if you imagine the article was written as "native code vs containers" -- what conclusions (if any) could you draw from the benchmarks in an article with that premise?
  - hansvm 3y ago
    A comparison between WASM and "containers" is meaningful because in popular usage there's a cottage industry of WASM-based container solutions with the express goal (and co-evolved with the language to make this possible) of simplifying some of the thornier design decisions that make other containerization solutions easier to get started with (can just drop in any binary and dependencies) but also less performant for some set of tasks (have to set up network cgroups and other garbage). The phrase "WASM" isn't just being used to refer to the actual stack machine.
    - jmillikin 3y ago
      WASM is WebAssembly. Words have meanings. If someone wants to try to advertise some weird custom platform that uses WASM as an intermediate bytecode for its packaging format then that's fine, but they'll need to pick a different name.
rektide 3y ago
Things don't have to be the same to compare & contrast. There were dozens of articles a month comparing vm's Vs containers back in the day. I also think your fixed perspective that these are & always will be different levels of the stack is an inadequate fixedness, lacks the mental agility to comprehend many of the things wasm is growing into. For years already, standard libraries like wasi are attempting to become fairly complete platform libraries. Instances of wasm run in effectively process trees of chroots, so the simar parity is apparent. It's absolutely coom sense to be comparing startup times across the two. There have been endless efforts to try to make cold boot containers or container snapshots as fast as possible. And this again extends backwards in time to comparisons versus vm's as well, with efforts like the Xen Zerg demo trying to make vm's fast enough to boot a vm per request. https://news.ycombinator.com/item?id=5243360 https://news.ycombinator.com/item?id=5243360 Getting back to a process per request or process per session model is a holy grail, one that every virtualization/containerization technology has explored. It would greatly alter the application development model & potentially greatly enhance security, could help radically reshape resource management. This article is doing a fine job pointing to the future, to what we are unlocking. We absolutely should compare versus what the technology of today (containers) can do. The comparison is spot on.
inkyoto 3y ago
The author has carried out an assessment of how the use of the ARM assisted hardware virtualisation (normally used to isolate and run VM's but not containers) affects different metrics of running different runtimes in a container environment that is aware of and that can take advantage of the hardware virtualisation features: Kata Container is a secure container runtime that uses hardware virtualization technology as an additional layer of defense. This allows it to provide stronger workload isolation while still being lightweight and performing like regular containers. The author uses WASM runtimes as a conduit, and concludes the article with a compilation of results and the impact on each metric, namely: boot times, security, the ease of development in each setup, production readiness amongst others. The author does compare raw computational performance of each option, and it does not appear to be an objective. The article title could have been better worded, though.
ridruejo 3y ago
"Nobody is deploying large-scale services on top of what would be essentially the CGI model." Yes, they are. That's basically the Cloudflare workers model (which certainly qualifies as large-scale) and many other V8/Wasm based serverless solutions
- jmillikin 3y ago
  The article is not describing the Cloudflare approach (v8 isolates) -- note the graph of startup times. Their "wasmedge" line takes 20 seconds to start 1500 instances, or about 13 milliseconds per instance. Cloudflare's blog post at https://blog.cloudflare.com/cloud-computing-without-containers/ https://blog.cloudflare.com/cloud-computing-without-containe... contains a graph in which requests are handled in 200 microseconds, orders of magnitude faster.
  - ridruejo 3y ago
    Same applies to Wasm. See Fastly equivalent of 35 microseconds https://www.fastly.com/edge-cloud-network/serverless#:~:text=With%20a%2035.4%20microsecond%20startup,fast%2C%20always%2Don%20computing https://www.fastly.com/edge-cloud-network/serverless#:~:text....
jasode 3y ago
>WASM vs a container runtime isn't a meaningful comparison because they're at different layers of the stack. It's not just comparing pure WASM (bytecode) but the WASM+WASI (runtime) which is what this article's WasmEdge (runtime) is about. It's a meaningful comparison when the original creator of Docker containers also compared them: >Solomon Hykes / @shykes@hachyderm.io @solomonstre >If WASM+WASI existed in 2008, we wouldn't have needed to created Docker. That's how important it is. Webassembly on the server is the future of computing. A standardized system interface was the missing link. Let's hope WASI is up to the task! -- from 2019-03-27t20:39:47 https://twitter.com/solomonstre/status/1111004913222324225 https://twitter.com/solomonstre/status/1111004913222324225
- jmillikin 3y ago
  > It's not just comparing pure WASM (bytecode) but the WASM+WASI (runtime) WASI isn't a runtime, and it's not a packaging format. It's a standard ABI for a POSIX-like syscall interface, which is useful[0] but doesn't substitute for or relate to container runtimes. You still need a syscall shim to provide the WASI implementation, and you need a WASM VM to run it. > It's a meaningful comparison when the original creator of Docker > containers also compared them: I've seen that quote before and I think it's nonsense. Docker container images are architecture-specific, and it took years before their distribution protocol supported specifying the architecture. If the existence of an architecture-agnostic bytecode would have avoided the need for Docker, then the JVM would have filled that niche perfectly. And if you look at the Docker code, the vast majority is about configuring the runtime environment of the container (networking, bind-mounts, logging) and persisting that state across reboots. There is no syscall shimming like in gVisor or a WASI implementation, it's all about setting up a chroot and then calling `exec()`. If WASM had existed in 2008, maybe the Docker folks would have enabled it as an entry-point format, but you'd still need a way to ship all the other stuff that goes along with the main binary. It's baffling that people feel the need to fabricate ways for WASM/WASI to be important, when the projects themselves are already so interesting. Architecture-agnostic plugins! A clean slate for a capability-based syscall ABI! A genuine chance at running unmodified userland programs in memory-safe sandboxes! Get excited about that, not about someone poorly re-implementing AWS Lambda. [0] Maybe one day most small shell tools will be in WASM+WASI, and the typical Linux distribution will have relatively little native code installed by default.
tecleandor 3y ago
The article isn't about WASM vs a container runtime, it's about comparing two container runtimes. WasmEdge is a container runtime designed for running Webassembly If you check the whole article, you'll find the (simple and basic) benchmark comparing a rust "hello world" running with runc, and (apparently) the same code compiled to Wasm and running in WasmEdge.

politelemon 3y ago

> normal (debugging within the container is a problem) That's a medium difficulty then. Kata isn't a great development environment.

jensneuse 3y ago

Has anybody ever seen large scale applications in real life? Or is WASM more like a middleware layer target? I'm genuinely interested in this as I'm yet to see a company using WASM in production.

deleted 3y ago

[deleted]