23 ms·

Extism: Make all software programmable with WebAssembly

382 points by nilslice 4y ago

winnie_ua 4y ago
I'm thinking on this for second day, and this though can't leave my head. It sounds like sexism. And thous I think it's bad name. It continues ringing the bell for me.
tomrod 4y ago
I feel like I'm missing the bead. What does it mean to "make all software programmable" -- meaning make Wasm work with all languages? Is this really feasible? Maybe I'm simply getting old and don't understand recent jargon shifts.
- ctvo 4y ago
  The idea is to offer extensibility points in software by embedding their runtime into your current software. Say you have a database and instead of a DLS for querying, allow someone to upload a WebAssembly binary that has access to APIs you'd expose (`filter`, `map`, ...). I'm guessing they ease some of the complexity in moving data back and forth, their runtime is supported on N languages, and they use wasm's sandboxing properties. The problem, for me, is even though the runtime supports many languages, the languages the plugins can be written in is limited. Rust and Go support wasm well, but AssemblyScript* is not JavaScript / TypeScript. If the idea is to extend software and make it more accessible, they need better support for interpreted languages: JS, Python, Ruby, ... * - Closures are still not supported: https://github.com/AssemblyScript/assemblyscript/issues/798 https://github.com/AssemblyScript/assemblyscript/issues/798
  - meltedcapacitor 4y ago
    Yes! Shipping more copies of interpreter runtimes in trivial plugins is exactly what the world needs.
    - ctvo 4y ago
      Or solve it a different way. I don't think your users care. It doesn't change the fact that it's a limiting factor right now.
    - bhelx 4y ago
      There will be ways to handle this problem in the future with the Component Model spec. I've seen it working but it's not easy to do yet. As of today though, yes bringing your own runtime is costly.
- jamil7 4y ago
  No, it took me a bit to understand as well. It looks like a set of host SDKs and a sandboxed runtime (guessing this is why they're using wasm) you can drop into your own software and expose some hooks that would allow your users to write their own plugins for your app in a variety of languages.
- bhelx 4y ago
  Extism is a plug-in system. Plug-in systems make your software programmable by end users (such as your customers or your open source community). Currently plug-in systems are quite difficult to build and limited to certain host and guest languages. Extism makes this easier regardless of your language or domain knowledge.
  - robofanatic 4y ago
    > Plug-in systems make your software programmable by end users (such as your customers or your open source community) I am confused. Who is the "end user" or the "customer? It seems like "developer" who is writing software for a customer is your end user or may be not. Who will actually use this is not clear from the website. From couple of examples I think this comes close to something like a SQL CLR integration, where for example you can call a C# routine from SQL query. https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/introduction-to-sql-server-clr-integration https://learn.microsoft.com/en-us/dotnet/framework/data/adon...
    - bhelx 4y ago
      Your end user would be a user of your application, e.g. a customer. But it could also be a marketplace like situation where people are developing plugins for your customers. Examples of plugin systems are things like VS-Code Extensions, Figma plugins, firefox extensions, etc. Here is a good example of a wasm plugin system for an open source project: https://istio.io/latest/docs/reference/config/proxy_extensions/wasm-plugin/ https://istio.io/latest/docs/reference/config/proxy_extensio...
- pomatic 4y ago
  I think it would be better described as a 'universal framework for incorporating plugins into your application'. If I am right about these aspirations, they are quite grandiose. Shiver Shades of COM or corba reborn?
  - dboreham 4y ago
    If I understand it, this is more about the plumbing of calling the Rust wasm execution engine from various languages. And probably the secondary problem of how do you package wasm-hosted code such that it can interface with multiple calling languages. Rust can build a shared object with C linkage. All the languages have ways to call "native" code (code with C linkage). This is the sauce to connect those dots in a multi-language, batteries-included way. Hopefully I got that right. It's all in-process so not more like OLE than Corba ;)
    - nilslice 4y ago
      You got that right! :)
jonnycomputer 4y ago
"make all software programmable" doesn't mean anything to me, and I doubt it means much to a lot of others either. I'm just talking a basic marketing fail here. bhelx below made it make more sense that their whole intro page.
- nilslice 4y ago
  Yea, marketing is hard. Glad bhelx is here to help :)
- jonnycomputer 4y ago
  you mean like Emacs. (:
  - bhelx 4y ago
    If you release a competing plug-in system using emacs i'll definitely check it out :)
    - wtetzner 4y ago
      I think they meant that emacs is an example of programmable software.
- cormacrelf 4y ago
  Another marketing related comment: Extism is not a very nice name. It sounds like exorcism, which apart from being a rather unsympathetic concept, is more about ridding yourself of foreign stuff that’s controlling you rather than the other way round. If anyone has any suggestions, drop them here.
  - hestefisk 4y ago
    Yeah it’s a very strange and non catchy name. I guess naming things is hard… interesting project nonetheless.
- dqpb 4y ago
bhelx 4y ago
Hey folks, one of the authors here. Extism is a plug-in system. If you aren't familiar with this terminology, plug-in systems make your software programmable by end users such as your customers or your open source community (e.g. VS-Code Extensions). Currently plug-in systems are quite difficult to build and limited to certain host and guest languages. Extism makes this easier regardless of your language or domain knowledge.
- hestefisk 4y ago
  What would happen if you executed Extism on itself? Would the world implode? :)
  - nilslice 4y ago
    only one way to find out....
    - euroderf 4y ago
      Grey Goo: Wasm Edition
  - bhelx 4y ago
    I lay awake at night and think about how to do it. But maybe it's a goal best left unfulfilled :)
- ar_lan 4y ago
  Thank you - I think the intro page could help make this clearer. I’m not super familiar with the programmable nature of things beyond something like an Arduino, so I was very confused what “programmable software” meant in this context, and the landing page still isn’t super clear in that regard. Your comment helps - would make sense to recalibrate on the shared page.
sphinxster 4y ago
This page needs a working example use-case.
- nilslice 4y ago
  There are some examples, but the announcement is focused on the project itself - we will follow up over the next week+ with some end-to-end examples. Stay tuned!
- nilslice 4y ago
  We also have some example projects in the GitHub org, pointing out this one specifically, to create a UDF engine in sqlite3: https://github.com/extism/extism-sqlite3 https://github.com/extism/extism-sqlite3
davidkunz 4y ago
Is the interface defined through wit? https://github.com/bytecodealliance/wit-bindgen https://github.com/bytecodealliance/wit-bindgen
- bhelx 4y ago
  Currently no. It's up to you to define your own interface. We plan to work on WIT bindings and eventually push support though.
- nilslice 4y ago
  Currently it is not, but we are closely tracking the Component Model to see how we can best utilize it. There are a few places we know we can leverage it, namely in the HTTP interfaces and other imports we provide to plug-ins. But, we also want to contribute to the language support on the codegen side. In order to get the language support we have already, we couldn't invest the time up front to build all the codegen in wit that would have been necessary. Over time want to get there though!
  - davidkunz 4y ago
    Awesome, thanks for the answer!
    - nilslice 4y ago
      If you're interested in helping us get there, we'd love the help. Hop in the Discord and we can talk about what it would take.
nilslice 4y ago
Authors are here to answer questions! To add a little bit of detail: Extism is an open-source universal plug-in system with the goal of making all software programmable. Easily embed Extism into 13 popular programming languages (server or browser) and safely run high-performance WebAssembly plug-ins inside your code. What does it mean to "make software programmable"? Simply put, you can give end-users the ability to extend your software with their code. Plug-in systems are usually implemented in 3 ways: 1. execute a binary that is external to your process (like protoc) 2. re-compile a program with an implementation of some interface 3. dynamically link to native code / dlls All 3 have major trade-offs on the performance-to-security ratio. Extism provides a way to "have your cake and eat it too" in a sense, that it doesn't compromise security for performance. True, executing WebAssembly as we do here is not running at fully native speed, but it's darn close. And its sandboxed architecture offers the security you want if you're considering an alternative to dll/dlopen. We're only getting started here, and welcome feedback good and bad. Please join our Discord[0] if you want to chat, or open issues on GitHub[1]. [0]: https://discord.gg/cx3usBCWnc https://discord.gg/cx3usBCWnc [1]: https://github.com/extism/extism/issues https://github.com/extism/extism/issues
- RodgerTheGreat 4y ago
  There's at least one more approach to program extensibility that you should probably address: interpreted languages, especially Lua. Many games and utilities today use Lua or JS as an extension language. They have battle-tested runtimes, come with high-performance JITs, and provide the potential for rich interactive debugging and incremental development options that are not free with a WASM-based statically-compiled approach.
  - nilslice 4y ago
    Thank you - that's a great point, and totally slipped my mind when writing this.
  - bhelx 4y ago
    We think Lua is great and in many ways, Wasm has yet to catch up! I do think it's still early days for Wasm, but it will.
Octokiddie 4y ago
Interesting (I think), but the page lacks a problem statement that the project seeks to solve. As others have noted, "make all software programmable" seems a bit vague. And it's vague because there's no problem in there. When is software not programmable? I think the idea is to make application extensions easier to integrate (like photoshop plugins), but I'm not sure. Start with the problem statement, and everything else will fall into place. Leave out the problem statement, and leave your readers mystified.
- bhelx 4y ago
  Point taken. We've been focused on talking to people who have built, or tried to build, plug-in systems. But it's still a confusing concept to the general public without enumerating some examples.
intrepidhero 4y ago
How big is the WASM runtime? If I embed Extism in my application what is the impact on install size and memory usage?
- bhelx 4y ago
  for install size, it depends on your architecture, we use wasmtime under the hood so a lot of the size is related to that: https://wasmtime.dev/ https://wasmtime.dev/ You can also take a look at our binary releases to get an idea: https://github.com/extism/extism/releases/tag/v0.1.0 https://github.com/extism/extism/releases/tag/v0.1.0 In terms of memory usage, as the host you can constrain the memory that your plugin can use. There is a minimum of one Wasm page (64KB in this case).
mike_hearn 4y ago
Looks interesting. Based on prior experience, here are some concerns people will bring up with this approach: 1. Spectre. You may have to assume the plugin code can read anything in the address space including any secrets like passwords, keys, file contents etc. If the plugin can't communicate with the outside world this may not be a problem. 2. When you say WASM has a "sandboxing architecture", this is only partially true. It's quite easy to define a simple language that doesn't provide any useful IO APIs and then claim it's sandboxed - that's practically the default state of a new language that's being interpreted. The problems start when you begin offering actual features exposed to the sandboxed code. The app will have to offer APIs to the code being run by the WASM engine and those APIs can/will contain holes through which sandboxed code can escape. If you look at the history of sandboxing, most sandbox escapes were due to bugs in the higher privileged code exposed to sandboxed code so it could be useful, but you can't help devs with that. 3. WASM is mostly meant for low level languages (C, C++, Rust etc). Not many devs want to write plugins in such low level languages these days, they will often want to be using high level languages. Even game engines are like that: "plugin" code is often written in C#, Lua, Blueprint, etc. This is especially true because WASM doesn't try to solve the API typing/object interop problem (as far as I know?), which is why your example APIs are all C ABI style APIs - the world moved on from those a long time ago. You'll probably end up needing something like COM as otherwise the APIs the host app can expose will be so limited and require so much boilerplate that the plugin extension points will just be kind of trivial.
- fiedzia 4y ago
  >WASM doesn't try to solve the API typing/object interop problem There many initiative around wasm, one of them being WebAssembly interface types, aimed at that problem. This makes using wasm simple.
- bhelx 4y ago
  Lots of good points here. I'll try my best to address them. But you are right, There aren't really complete answers to each of these problems. Regarding #1, it's good to point out that spectre is still an ongoing problem. wasmtime, the runtime we use, has mitigations for spectre but things will likely come up. I think we need more time and tools to work on things like detecting attacks and mitigating future problems. Fortunately there are some big companies working on it. Our team cannot solve that problem. Regarding #2, sandboxing here refers to the memory access model and fault isolation. For each capability you give the plugin you may introduce risk. We don't claim to address this problem but I think education and mitigation are good goals for us. Regarding #3, I think there has been some good progress here. I have an experimental C# plugin PDK and a JS one based on quickjs. We also support Haskell. I think there will be improvement here. And regarding the ABI comment, there are going to be layers on top of this to make it more ergonomic.
jkcorrea 4y ago
Congrats on the launch! Just yesterday I was pondering if I could do something like this for a saas I’m working on: I’d like to offer users the ability to extend their dashboard in certain ways (a la WP/Shopify plugins), but those ways require extending the ui AND the server code. Extism seems like a great fit for this, I’ll definitely be checking it out :)
- nilslice 4y ago
  Thanks! I want to highlight that Extism supports both server-side AND browser environments, running the same plug-ins. So SaaS is a great example of a place where end-users may want to extend functionality that runs on the client, server, or both. Feel free to reach out if you have any questions.
- bhelx 4y ago
  Sounds very cool and like it could be a good fit for our browser based runtime! There are still a few limits, but you can currently run the same plugins in your backend and frontend.
nmstoker 4y ago
As others have said, the term "software" is confusing. Application or software product seem more apt. The other small point I'd raise: there's mention that due to the sandboxing of WASM it's "safe" to run untrusted code. Maybe I've misunderstood. To me that seems true in so far as the untrusted code cannot break out and can only do actions the hosting app decides to allow, but this isn't much consolation if the app allows actions that can be subverted. For instance, if the app allows control over saving a file, untrusted code only gets to handle that within the rails the app sets but that still means dodgy code could fill up a hard drive by saving over and over, or it could possibly save over important files.
- bhelx 4y ago
  > As others have said, the term "software" is confusing. Application or software product seem more apt. That's a great suggestion thanks! > To me that seems true in so far as the untrusted code cannot break out and can only do actions the hosting app decides to allow, but this isn't much consolation if the app allows actions that can be subverted. Yes i think safe is a relative term here. Any user input is "unsafe" to some degree. Your example is a good one! I think we wrote this in the context of other relative ways to do this but will be discussing better ways to talk about security and what Extism protects and what it doesn't.
hybid 4y ago
If I want to make all software programmable, I install Frida and easily hook into pretty much anything at the same privilege level. Still, misleading headline aside, this sounds pretty cool as a reasonably well sandboxed plug-in architecture for software that chooses to use it. Though if running in-process, it seems unlikely to be able to defend against hostile plugins that use e.g. speculative execution techniques to extract otherwise inaccessible data. But perhaps that's not the threat model here.
- warpech 4y ago
  For those who don't know what Frida is: https://frida.re/ https://frida.re/ I must say, this sentence from their docs describe it the best to me: Greasemonkey for native apps It has little to do with Extism, but still is interesting.
pavlov 4y ago
The manifest format could perhaps be less terse. It's really hard to guess what this actually means without reading the docs: "memory": { "max": 5 } Presumably the maximum amount of memory available to the plugin, but which unit? 5 megabytes? Turns out it's measured in pages of 64k each: "The units here are pages where a page is 64Kib." [1] So max==5 apparently means a hard upper limit of 320 kB. (For when you want to run a 1984 copy of WordPerfect as a plugin, of course.) Maybe a "max_MB" property might be more useful to plugin authors, and then just calculate the # of pages internally from that? [1] https://extism.org/docs/concepts/manifest#schema https://extism.org/docs/concepts/manifest#schema
- bhelx 4y ago
  We have an explanation of the manifest here: https://extism.org/docs/concepts/manifest https://extism.org/docs/concepts/manifest it's still a work in progress. the point about units is a good one. Might be a way to make that more clear.
- bhelx 4y ago
  > Maybe a "max_MB" property might be more useful to plugin authors, and then just calculate the # of pages internally from that? yeah, that's something to consider. You can only pass in parameters that are a multiple of the page size so that may not be an accurate representation. I do think we need to hint at the unit in the name though.
madmod 4y ago
I wonder how this compares to https://suborbital.dev/ https://suborbital.dev/
- callahad 4y ago
  [I work for Suborbital] We're both all in on extensible software powered by WebAssembly, but the scope and domains differ. Suborbital is more of a turnkey / batteries-included service that handles the full lifecycle of extending SaaS applications with user-defined plugins. So it includes a web-based editor widget, a module builder service, storage and compute, administrative dashboards, etc. Extism, from my lay understanding, is more tightly scoped toward being a great library for running WebAssembly modules, in the same way you might embed a Lua runtime into an application for user scripting. Another big difference is that you can use all of Extism right now. Suborbital's hosted platform is still in closed beta :) (Waitlist at https://form.typeform.com/to/DHxjRGKx https://form.typeform.com/to/DHxjRGKx)
  - bhelx 4y ago
    I haven't used suborbital yet but yeah, that sounds accurate. We're aimed at being a library and Extism itself is focused on building plug-in systems. I just signed up for the waitlist! Looking forward to learning more about it.
Jyaif 4y ago
Does it require JIT? In other words, does it work in iOS?
- bhelx 4y ago
  yes, Wasmtime does use a JIT. We haven't tried on mobile yet, but I think I have some paths to make it work. If this is something you need come join our discord and we can experiment on it!
  - Williams22 4y ago
    Yes i wanna Join
kumarski 4y ago
WebAssembly ecosystem booming it seems.
floober 4y ago
This looks fantastic! I'm working on a desktop app and I've been struggling with the extensibility story in the back of my mind for a couple of weeks now. Thanks for building this! edit: For what it's worth, the use case and value of something like this was immediately apparent to me.
- nilslice 4y ago
  Awesome! We'd love to see what you end up building, and if you use Extism please let us know what you find missing... > For what it's worth, the use case and value of something like this was immediately apparent to me. Thank you for saying so. Negative feedback doesn't bother me (I appreciate anyone taking time to speak up), but it's also usually the only kind you get on the Internet :) so we appreciate hearing that. Translating ideas into words is hard, especially in software.
beders 4y ago
I've built a plugin-system back in 2003 or so based on the JVM which can load/re-load bytecode relatively easily. (the concept became popular in the JVM world and lead to abominations like OSGi) The problems that arose quickly were: - Executing order of plugins attaching to the same extension point. - Third party library dependencies having different versions - Traceability and debugging - Thread-safety OTOH, it gave our custom dev team to develop integrations and extensions that could be deployed and maintained independently of the main product. So there's definitely a use-case in there, but it requires some coordination regardless.
- bhelx 4y ago
  Would love to hear more about your experience there. > (the concept became popular in the JVM world and lead to abominations like OSGi) I remember this, though it was a couple years before my time. Although some of the underlying technology has changed those problems you outlined have remained the same. These things are all relevant to our next steps. We've been thinking a lot about what developer experience problems need to be solved.
haberman 4y ago
I'm trying to figure out what Extism is actually providing here. The underlying runtime appears to be Wasmtime, an existing project that you can use separately. For the system API, Extism supports WASI (a POSIX-like API for WASM) which is already supported by Wasmtime. However it sounds like it also has a different, non-WASI API that provides similar functionality. I'm curious to hear a compare/contrast between these two system APIs, or what motivated inventing a new non-WASI system API. It does appear that WASI is in its early days; WASI was announced in 2019 (https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/ https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webas...) but it appears that all of the specific proposals are still early stage and far from being standardized (https://github.com/WebAssembly/WASI/blob/main/Proposals.md https://github.com/WebAssembly/WASI/blob/main/Proposals.md). For the API, Extism provides a C API/ABI that can be called through FFI, but AIUI Wasmtime already has such an API: https://docs.wasmtime.dev/c-api/ https://docs.wasmtime.dev/c-api/ Basically I'm trying to understand the conceptual difference between this project and Wasmtime.
- bhelx 4y ago
  It's a good question. It's hard to understand what all is involved in building something like that until you have to get into the details. You can think of Extism as a layer around wasmtime. There is an ABI of sorts that helps you communicate with the plugin: Send data to it, invoke functions, and get data out. With our system, you don't need to enable WASI to use a plugin. You can invoke functions directly. However, you can use WASI for other purposes. And we are working on a way to do POSIX-like stuff (e.g. programs with a main function that use stdin and stdout). Because we wrap wasmtime, we can also switch it out. That's how we are able to run the same plugins in the browser. There we use the browsers' implementation. I think one day we will replace some internal pieces of Extism with pieces of the Component Model spec. As of today it's not in a state where we can do all of this in this many languages. edit: rewording, grammar
  - syrusakbary 4y ago
    Hey! Super exciting to see your progress on Extism. I'm Syrus, from Wasmer. I did a quick review of your plugin architecture and it looks great (tons of languages supported, congrats!). I think Wasmer might be able able to help you so you have the same implementation of the runtime inside and outside of the browser (something similar as we did in wasmer-js [1], where we reuse same WASI implementation, but using the native Wasm engine from the browser). I'd love to see how we can integrate your plugins into Wasmer as well! [1] https://github.com/wasmerio/wasmer-js https://github.com/wasmerio/wasmer-js
fulafel 4y ago
How are you implementing JS platform support, is it a WebAssembly JS interpreter?
- nilslice 4y ago
  for JS on the Web, yes - we ported our runtime specific APIs to TypeScript and expose them to the WebAssembly runtime in the browser for JS in Node, we use wasmtime embedded within our runtime -- looking into what the tradeoffs are to re-implement the whole runtime in Node on its native WebAssembly runtime though. Feedback welcome & appreciated!
hestefisk 4y ago
Is it single threaded?
- bhelx 4y ago
  The plugins are, but not the host. You can create pools of plugin instances if you need concurrency. Threading inside a Wasm program is still a new concept but I think it will become more common.
satvikpendem 4y ago
Will WASM finally be the write once run everywhere system we've been promised for the past few decades? I can imagine every language compiling to the common format that is WASM, so you can mix and match libraries from other languages as you please, as well as chip manufacturers adding hardware accelerated WASM processing units if it gets popular enough. This really is the birth and death of Javascript.
- marcosdumay 4y ago
  No. WASM will have the same problem that every "run everywhere" system ever had: what makes software hard to port is not the ability to execute it, it's the environment differences. But it will be a powerful low trust portable VM, like none mainstream today. Odds are it kills Lua. (And yeah, it removes the Javascript monopoly on the browser. At this point, it can't be killed, but it will shrink.)
  - satvikpendem 4y ago
    Perhaps we can compile a Docker image into WASM too, we've seen Postgres recently. Or do you mean something else?
  - chubot 4y ago
    Lua is more like a C library and less like a VM. (Similarly, LLVM isn't a VM, which is a common misunderstanding, given the name) As far as I know, people don't ship code to remote Lua VMs and execute it. There is no compatibility guarantee across versions, and there are nontrivial compile-time options (like what the number type is). That doesn't mean Lua isn't useful; it's just not designed for the same things that WASM is.
    - marcosdumay 4y ago
      > As far as I know, people don't ship code to remote Lua VMs and execute it. So, you never used some software scripted with Lua? Or just never shared scripts? Lua does have other uses, but the vast majority of it is as a low trust VM.
NuSkooler 4y ago
Interesting. Just this week I've been building a little experiment in order to utilize WASM based plugins in a project I'm involved with. I'm using WasmEdge for the task and so far am having pretty good luck. Questions: 1. Can WASM plugins with Extism easily call back into host provided APIs? 2. Complex types are supported?
- nilslice 4y ago
  WasmEdge is really cool -- I'm super impressed with the work they are doing and hope to eventually provide an alternative runtime backend to Extism using WasmEdge (maybe they'd like to help!) To answer your questions: 1. Extism provides a specific set of host provided APIs, and currently we don't have support for user-supplied host functions beyond the ones we make available. This will change though, and we're looking for feedback. We held off because there wasn't a clear way to make it easy across ALL the languages we support. 2. Complex types are definitely supported, you need to encode the data into raw bytes and can use whichever format you want (only limited by which formats are available in host & plug-in languages..) Eventually we will move towards the Component Model which will make the complex types story much simpler.
  - NuSkooler 4y ago
    Thanks for the info! I'll be keeping an eye on this for sure. What I'm working on is a PoC and it's working well thus far. As Extism advances and supports some of these upcoming features, it may turn out to be "just right" for my needs.
meowface 4y ago
Three thoughts: * That's an absolutely viscerally horrifying logo * I absolutely love it and its design * I am confused about why it's used for a project like this and not some kind of internal NSA project, or maybe a nu-witch house band logo, or something
- nilslice 4y ago
  LOL Thank you on all 3 points!! I intended to invoke thoughts about how Extism enables one to "extend from within", thus the octopus crawling out from the skull. And also that the thing Extism should replace is rather cursed... loading DLLs / dlopen tons of plug-ins into your program to execute untrusted code is downright scary.
  - duxup 4y ago
    The logo is symbolic of the product or thing right (Extism), not other things? Not sure I would get the escaping from part by looking at a logo.
jayd16 4y ago
I'd love to see C# support for something like this as well. Would be nice if Unity games could start supporting wasm plugins. Not sure if something like that works for Ureal already. Another question... Is something like this possible on iOS or will these kinds of runtimes be banned by Apple?
- bhelx 4y ago
  C# support on which side? The host or the plugin side? I've got an experimental C# plugin almost working. I think I can do it. A Host SDK library would be pretty easy since it's just FFI to the runtime. > Would be nice if Unity games could start supporting wasm plugins That would be pretty neat! I go back and forth over whether this kind of thing will happen or not. Extism's goal is to be very generic (universal). A game engine may have some competing goals like performance or predictability. Still, I wouldn't be surprised if some alternative engines didn't do this in the next few years. > Another question... Is something like this possible on iOS or will these kinds of runtimes be banned by Apple? Not today, but I think I know how to get there. If you need this stuff for your project come join the discord and we can try to prioritize it!
- nilslice 4y ago
  It's already underway! https://github.com/extism/extism/pull/119 https://github.com/extism/extism/pull/119
zacharyvoase 4y ago
I'm genuinely curious: is the WebAssembly ecosystem just reinventing the JVM piecemeal? "Write once run anywhere"?
- bhelx 4y ago
  In many ways yes. We need to re-invent many things for this new world. But I'd encourage you to read a bit more about how Wasm is different. It's true it is a stack based virtual machine but that's about where the comparisons end.
- slaymaker1907 4y ago
  I think you're probably right, but I don't think many people would look at the JVM as a failure. Instead, I think WASM is trying to bring the good parts of the JVM such as being a simple compile target, small binaries, and a stable VM while fixing the bad parts like having a huge attack surface area and lacking certain features critical for performance (pointer chasing, GC, etc.).
  - zacharyvoase 4y ago
    Well as vendors add access to system APIs to WASM, that attack surface will indeed grow.
- moffkalast 4y ago
  No, it's just "let's take random languages and transpile them to Javascript but call it Webassembly and hope nobody notices". In that sense it's much worse.
rvcdbn 4y ago
Why not use protobuf for serializing data between plugin and host language?
- bhelx 4y ago
  You can bring your own serialization format. It's just bytes in and bytes out.
ynx 4y ago
Damn, this looks really cool. I've wanted to add wasm support to my project at work for some time now, but unfortunately with no good C# integrations (with Unity as the host), it just hasn't met the mark yet. Really hoping that can be addressed in the near future.
- bhelx 4y ago
  If you need C# support we can work on it! I've done some experimenting with it already. Please come join the discord or file an issue.
n4jm4 4y ago
Why not continue using long-standing C FFI? This is not so clear.
nilslice 4y ago
For those who want to go straight to the code: https://github.com/extism/extism https://github.com/extism/extism
ilyt 4y ago
Huh, just a week ago I thought it would be cool to add WASM plugins to my Go app, as a way to not be dependent on Go for plugins.
- nilslice 4y ago
  I hope Extism can help you make that happen! Here’s an example of a Go app that calls Extism plugins in case it helps: https://github.com/extism/extism-fsnotify https://github.com/extism/extism-fsnotify
Existenceblinks 4y ago
The pdk is missing Zig which is, for many developers, considerably the best for wasm target. It's as fast as Rust (from benchmark I saw, Rust is fastest), though Zig's binary size is as small as AssemblyScript's. Rust's binary size is considerably in a range of middle to large, mostly middle. It would be nice if there's an official wasm benchmark of source languages, showing speed and binary size comparison.
- bhelx 4y ago
  Yeah, that's a good idea. C is supported on the SDK and PDK side. We are wondering if Zig can utilize the c libraries for it.
  - Existenceblinks 4y ago
    I would say Zig is the nicest C friend! Zig can compile C code. Checkout: https://ziglang.org/learn/overview/#zig-is-also-a-c-compiler https://ziglang.org/learn/overview/#zig-is-also-a-c-compiler Also: https://ziglang.org/documentation/master/#toc-Freestanding https://ziglang.org/documentation/master/#toc-Freestanding
deleted 4y ago
[deleted]
cloudmike 4y ago
As others have mentioned, "making software programmable" isn't really landing for me. In the tradition of borrowing terms from the games industry (where we often see early popularization of good ideas), I wonder if "modding" could be a helpful addition somewhere in your marketing. This seems to have worked for the term "multiplayer", though "moddable" probably isn't as widely appreciated yet.
- Kinrany 4y ago
  Even in gaming modding is still somewhat niche.
computerdork 4y ago
If I understand it correctly, it is as overlap with the "sidecar" design-pattern, correct? (https://learn.microsoft.com/en-us/azure/architecture/patterns/sidecar https://learn.microsoft.com/en-us/azure/architecture/pattern...) If so, then super interesting product! Have actually had a somewhat similar idea myself for a product that allows you to create "subsystems" that are isolated from one another, allowing you great freedom in how the subsystems are deployed. They could be deployed all on the same machine, or, if greater scalability is needed, a subsystem (or plugin) could be moved to its own machine, or even scaled up to run multiple instances. Have only read through the simple code example on your website, but also, it would be great if there was a standard for calling and receiving requests through FFI's. Because this would give it the utmost flexibility on where the plugins can be deployed to. To me, this means both the actual mechanism for the call (REST over HTTP, graphQL...) and in the plugin and host themselves, with the syntax of the call. So for the syntax, it would also be great if this looked more like a regular method call instead of having to wrap parameters in an object: hostWrapper.somemethod(param1, param2) or hostWrapper.call("somemethod", param1, param2) (Although, not sure how difficult this is to do, given all the different method call syntaxes of the different languages). And, am glad that you and your company is really trying to make this happen, it's a great idea! Looking forward to giving it a try!
- nilslice 4y ago
  hey, thanks for checking out the project! it's not quite the sidecar pattern (assuming i'm interpreting the link you shared correctly) - since Extism is actually _embedded_ into your program's process, not isolated over a network boundary like a sidecar. the benefit of this is typically much faster execution speed, and thanks to WebAssembly's security model, it's safe to execute plug-in code - unlike a DLL or dlopen to a .dylib or .so file. in effect, it becomes an actual function call, not a RPC. please correct me if i've misinterpreted your post!
  - computerdork 4y ago
    Think you interpreted the Side Car pattern pretty well:) It definitely is typically done using HTTP (REST), and the benefit of this is it gives it great flexibility, allowing it to be deployed in many different ways. Like Extism, sidecars are typically developed in some type of container like Docker (I have a limited understanding of WASM but since it's also a container technology, think you could also use this instead). And since the calls into the container are done through a standard like HTTP REST (or GraphQL), it's no problem if you'd like to have the container deployed locally, or on the same machine as your host container, or even on it's own server if you need to run it as its own microservice. Although typically, sidecars are deployed locally and work almost exactly like Extism - simply as a plugin to the main application. And the performance of the local call is pretty good, because the call is usually made over a virtual network you'd setup in your docker configuration files. I came across sidecars for the first time a couple of years ago while doing a job at Atlassian. And, for me, creating an entire product like Extism to take plugin/sidecars to the next level seems like a great idea. Because sidecars are awesome and we need a way to make them easier and easier to develop! Hmm, was just thinking it over, wonder if it be useful if Extism did allow flexibility in deployment (meaning that the mechanism for calls wasn't just limited to using WebAssembly to do the call/security), but could switched to using HTTP REST or GraphQL by some simple changes to some config files? Because now the plugins can be deployed in many different ways, and even work in legacy systems (they may have their own security model. Seems like some type of REST security is pretty popular)... ... although, as mentioned, from what I saw, seemed like 95% of the time, sidecars are used locally as well, like Extism, so maybe just focusing on local calls is good enough?? ... Am also wondering if supporting both Docker and WASM might be useful/possible? And of course, I have very limited understanding of Extism, so of course, I may be waay off in what the future directions of your product, so please take everything I said with a grain of salt. But, Extism does seem like a great idea! Will give it a try.
est 4y ago
Is it possible to run wasm blobs directly on Linux?
deleted 4y ago
[deleted]
stevefan1999 4y ago
and...still no dotnet
- nilslice 4y ago
  it's underway! https://github.com/extism/extism/pull/119/files https://github.com/extism/extism/pull/119/files
garganzol 4y ago
Extism tries to cover two completely different things in one package: 1) runtime 2) component model. And this is a problem. Those things should be separated because the component model would be useful independently, without any ties to the specific runtime environment be it WebAssembly or something else. We already saw this playbook many times before. For instance, .NET. It has a handy set of system types that define the component model, but surprise, it can only be used in the realm of .NET which makes it unusable in, say, Go. Java - the same thing. C++ is the same. Everything more abstract than some primitive data types and plain functions just cannot cross the boundaries between the language ecosystems. If we had a standard component model then it would allow us to create a library in one language and then use it from many other languages without any manual work. But all we have now is the old plain C-like API as a common denominator between them. It's 1978 all over again for the last 40 years. But I still believe that we can do better than that. Instead of m * n complexity when every library is doomed to be manually ported to n languages, we may have just m + n complexity by leveraging the benefits of a standard component model. Just imagine the economic effects of such achievement. The main problem of an imaginary de-facto standard component model is the danger of being opinionated. We should solve that, otherwise all such efforts will be doomed to fail from the beginning. Data and functions are good starting points, and they are universal by the laws of math and algebra. That's why they are not opinionated. Consequently, they are used everywhere. We should build a similar formal apparatus for, say, objects to allow OOP interoperability between the languages once and for all.
- iecheruo 4y ago
  To be fair to .NET what you describe sounds an awful lot like COM/WinRT
Mikhail_K 4y ago
More like, "make all software exploitable"
winnie_ua 4y ago
To be honest, I don't see why it is advertised here at this stage. I appreciate effort, but project is too young and unable to do any serious plugin. I guess it is try to get new contributors on board? I see that project on github started few months ago. Isn't it too early to adwertise it to the world? But it already have fancy website with modern styles and so on. But it lacks good documentation. Article says that docs have "dozens of examples in different languages", but it just have dozen languages and helloworld in them. And basically examples are just simple function call from host to plugin. Operating on strings/basic types, and JSON. That's not what we want from plugins. Wee need some way to expose Host API some way, perhaps in OOP fashion. To let plugin interract with host app, perhaps even change UI. But not of examples show that, how to pass at least basic functions into plugin. As I understand it comes with it's own HTTP stack, presumably implemented in Rust. This can be used as starting point to make your own macro processor that would be able to expose API for plugins, and then dogfoof this codegen for your own HTTP lib. And passing JSON around is not thing that is needed. With same success IPC could be used, with relatively similar performance. We need proper marshaling. There much work to be done, sadly. That's big endeavor. Well, I hope this project would grow in something good, but it's in really early stage.

382 points by nilslice 4y ago

I'm thinking on this for second day, and this though can't leave my head. It sounds like sexism. And thous I think it's bad name. It continues ringing the bell for me.

tomrod 4y ago

I feel like I'm missing the bead. What does it mean to "make all software programmable" -- meaning make Wasm work with all languages? Is this really feasible? Maybe I'm simply getting old and don't understand recent jargon shifts.

ctvo 4y ago
The idea is to offer extensibility points in software by embedding their runtime into your current software. Say you have a database and instead of a DLS for querying, allow someone to upload a WebAssembly binary that has access to APIs you'd expose (`filter`, `map`, ...). I'm guessing they ease some of the complexity in moving data back and forth, their runtime is supported on N languages, and they use wasm's sandboxing properties. The problem, for me, is even though the runtime supports many languages, the languages the plugins can be written in is limited. Rust and Go support wasm well, but AssemblyScript* is not JavaScript / TypeScript. If the idea is to extend software and make it more accessible, they need better support for interpreted languages: JS, Python, Ruby, ... * - Closures are still not supported: https://github.com/AssemblyScript/assemblyscript/issues/798 https://github.com/AssemblyScript/assemblyscript/issues/798
- meltedcapacitor 4y ago
  Yes! Shipping more copies of interpreter runtimes in trivial plugins is exactly what the world needs.
  - ctvo 4y ago
    Or solve it a different way. I don't think your users care. It doesn't change the fact that it's a limiting factor right now.
  - bhelx 4y ago
    There will be ways to handle this problem in the future with the Component Model spec. I've seen it working but it's not easy to do yet. As of today though, yes bringing your own runtime is costly.
jamil7 4y ago
No, it took me a bit to understand as well. It looks like a set of host SDKs and a sandboxed runtime (guessing this is why they're using wasm) you can drop into your own software and expose some hooks that would allow your users to write their own plugins for your app in a variety of languages.
bhelx 4y ago
Extism is a plug-in system. Plug-in systems make your software programmable by end users (such as your customers or your open source community). Currently plug-in systems are quite difficult to build and limited to certain host and guest languages. Extism makes this easier regardless of your language or domain knowledge.
- robofanatic 4y ago
  > Plug-in systems make your software programmable by end users (such as your customers or your open source community) I am confused. Who is the "end user" or the "customer? It seems like "developer" who is writing software for a customer is your end user or may be not. Who will actually use this is not clear from the website. From couple of examples I think this comes close to something like a SQL CLR integration, where for example you can call a C# routine from SQL query. https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/introduction-to-sql-server-clr-integration https://learn.microsoft.com/en-us/dotnet/framework/data/adon...
  - bhelx 4y ago
    Your end user would be a user of your application, e.g. a customer. But it could also be a marketplace like situation where people are developing plugins for your customers. Examples of plugin systems are things like VS-Code Extensions, Figma plugins, firefox extensions, etc. Here is a good example of a wasm plugin system for an open source project: https://istio.io/latest/docs/reference/config/proxy_extensions/wasm-plugin/ https://istio.io/latest/docs/reference/config/proxy_extensio...
pomatic 4y ago
I think it would be better described as a 'universal framework for incorporating plugins into your application'. If I am right about these aspirations, they are quite grandiose. Shiver Shades of COM or corba reborn?
- dboreham 4y ago
  If I understand it, this is more about the plumbing of calling the Rust wasm execution engine from various languages. And probably the secondary problem of how do you package wasm-hosted code such that it can interface with multiple calling languages. Rust can build a shared object with C linkage. All the languages have ways to call "native" code (code with C linkage). This is the sauce to connect those dots in a multi-language, batteries-included way. Hopefully I got that right. It's all in-process so not more like OLE than Corba ;)
  - nilslice 4y ago
    You got that right! :)

jonnycomputer 4y ago

"make all software programmable" doesn't mean anything to me, and I doubt it means much to a lot of others either. I'm just talking a basic marketing fail here. bhelx below made it make more sense that their whole intro page.

nilslice 4y ago
Yea, marketing is hard. Glad bhelx is here to help :)
jonnycomputer 4y ago
you mean like Emacs. (:
- bhelx 4y ago
  If you release a competing plug-in system using emacs i'll definitely check it out :)
  - wtetzner 4y ago
    I think they meant that emacs is an example of programmable software.
cormacrelf 4y ago
Another marketing related comment: Extism is not a very nice name. It sounds like exorcism, which apart from being a rather unsympathetic concept, is more about ridding yourself of foreign stuff that’s controlling you rather than the other way round. If anyone has any suggestions, drop them here.
- hestefisk 4y ago
  Yeah it’s a very strange and non catchy name. I guess naming things is hard… interesting project nonetheless.
dqpb 4y ago

bhelx 4y ago

Hey folks, one of the authors here. Extism is a plug-in system. If you aren't familiar with this terminology, plug-in systems make your software programmable by end users such as your customers or your open source community (e.g. VS-Code Extensions). Currently plug-in systems are quite difficult to build and limited to certain host and guest languages. Extism makes this easier regardless of your language or domain knowledge.

hestefisk 4y ago
What would happen if you executed Extism on itself? Would the world implode? :)
- nilslice 4y ago
  only one way to find out....
  - euroderf 4y ago
    Grey Goo: Wasm Edition
- bhelx 4y ago
  I lay awake at night and think about how to do it. But maybe it's a goal best left unfulfilled :)
ar_lan 4y ago
Thank you - I think the intro page could help make this clearer. I’m not super familiar with the programmable nature of things beyond something like an Arduino, so I was very confused what “programmable software” meant in this context, and the landing page still isn’t super clear in that regard. Your comment helps - would make sense to recalibrate on the shared page.

sphinxster 4y ago

This page needs a working example use-case.

nilslice 4y ago
There are some examples, but the announcement is focused on the project itself - we will follow up over the next week+ with some end-to-end examples. Stay tuned!
nilslice 4y ago
We also have some example projects in the GitHub org, pointing out this one specifically, to create a UDF engine in sqlite3: https://github.com/extism/extism-sqlite3 https://github.com/extism/extism-sqlite3

davidkunz 4y ago

Is the interface defined through wit? https://github.com/bytecodealliance/wit-bindgen https://github.com/bytecodealliance/wit-bindgen

bhelx 4y ago
Currently no. It's up to you to define your own interface. We plan to work on WIT bindings and eventually push support though.
nilslice 4y ago
Currently it is not, but we are closely tracking the Component Model to see how we can best utilize it. There are a few places we know we can leverage it, namely in the HTTP interfaces and other imports we provide to plug-ins. But, we also want to contribute to the language support on the codegen side. In order to get the language support we have already, we couldn't invest the time up front to build all the codegen in wit that would have been necessary. Over time want to get there though!
- davidkunz 4y ago
  Awesome, thanks for the answer!
  - nilslice 4y ago
    If you're interested in helping us get there, we'd love the help. Hop in the Discord and we can talk about what it would take.

nilslice 4y ago

Authors are here to answer questions! To add a little bit of detail: Extism is an open-source universal plug-in system with the goal of making all software programmable. Easily embed Extism into 13 popular programming languages (server or browser) and safely run high-performance WebAssembly plug-ins inside your code. What does it mean to "make software programmable"? Simply put, you can give end-users the ability to extend your software with their code. Plug-in systems are usually implemented in 3 ways: 1. execute a binary that is external to your process (like protoc) 2. re-compile a program with an implementation of some interface 3. dynamically link to native code / dlls All 3 have major trade-offs on the performance-to-security ratio. Extism provides a way to "have your cake and eat it too" in a sense, that it doesn't compromise security for performance. True, executing WebAssembly as we do here is not running at fully native speed, but it's darn close. And its sandboxed architecture offers the security you want if you're considering an alternative to dll/dlopen. We're only getting started here, and welcome feedback good and bad. Please join our Discord[0] if you want to chat, or open issues on GitHub[1]. [0]: https://discord.gg/cx3usBCWnc https://discord.gg/cx3usBCWnc [1]: https://github.com/extism/extism/issues https://github.com/extism/extism/issues

RodgerTheGreat 4y ago
There's at least one more approach to program extensibility that you should probably address: interpreted languages, especially Lua. Many games and utilities today use Lua or JS as an extension language. They have battle-tested runtimes, come with high-performance JITs, and provide the potential for rich interactive debugging and incremental development options that are not free with a WASM-based statically-compiled approach.
- nilslice 4y ago
  Thank you - that's a great point, and totally slipped my mind when writing this.
- bhelx 4y ago
  We think Lua is great and in many ways, Wasm has yet to catch up! I do think it's still early days for Wasm, but it will.

Octokiddie 4y ago

Interesting (I think), but the page lacks a problem statement that the project seeks to solve. As others have noted, "make all software programmable" seems a bit vague. And it's vague because there's no problem in there. When is software not programmable? I think the idea is to make application extensions easier to integrate (like photoshop plugins), but I'm not sure. Start with the problem statement, and everything else will fall into place. Leave out the problem statement, and leave your readers mystified.

bhelx 4y ago
Point taken. We've been focused on talking to people who have built, or tried to build, plug-in systems. But it's still a confusing concept to the general public without enumerating some examples.

intrepidhero 4y ago

How big is the WASM runtime? If I embed Extism in my application what is the impact on install size and memory usage?

bhelx 4y ago
for install size, it depends on your architecture, we use wasmtime under the hood so a lot of the size is related to that: https://wasmtime.dev/ https://wasmtime.dev/ You can also take a look at our binary releases to get an idea: https://github.com/extism/extism/releases/tag/v0.1.0 https://github.com/extism/extism/releases/tag/v0.1.0 In terms of memory usage, as the host you can constrain the memory that your plugin can use. There is a minimum of one Wasm page (64KB in this case).

mike_hearn 4y ago

Looks interesting. Based on prior experience, here are some concerns people will bring up with this approach: 1. Spectre. You may have to assume the plugin code can read anything in the address space including any secrets like passwords, keys, file contents etc. If the plugin can't communicate with the outside world this may not be a problem. 2. When you say WASM has a "sandboxing architecture", this is only partially true. It's quite easy to define a simple language that doesn't provide any useful IO APIs and then claim it's sandboxed - that's practically the default state of a new language that's being interpreted. The problems start when you begin offering actual features exposed to the sandboxed code. The app will have to offer APIs to the code being run by the WASM engine and those APIs can/will contain holes through which sandboxed code can escape. If you look at the history of sandboxing, most sandbox escapes were due to bugs in the higher privileged code exposed to sandboxed code so it could be useful, but you can't help devs with that. 3. WASM is mostly meant for low level languages (C, C++, Rust etc). Not many devs want to write plugins in such low level languages these days, they will often want to be using high level languages. Even game engines are like that: "plugin" code is often written in C#, Lua, Blueprint, etc. This is especially true because WASM doesn't try to solve the API typing/object interop problem (as far as I know?), which is why your example APIs are all C ABI style APIs - the world moved on from those a long time ago. You'll probably end up needing something like COM as otherwise the APIs the host app can expose will be so limited and require so much boilerplate that the plugin extension points will just be kind of trivial.

fiedzia 4y ago
>WASM doesn't try to solve the API typing/object interop problem There many initiative around wasm, one of them being WebAssembly interface types, aimed at that problem. This makes using wasm simple.
bhelx 4y ago
Lots of good points here. I'll try my best to address them. But you are right, There aren't really complete answers to each of these problems. Regarding #1, it's good to point out that spectre is still an ongoing problem. wasmtime, the runtime we use, has mitigations for spectre but things will likely come up. I think we need more time and tools to work on things like detecting attacks and mitigating future problems. Fortunately there are some big companies working on it. Our team cannot solve that problem. Regarding #2, sandboxing here refers to the memory access model and fault isolation. For each capability you give the plugin you may introduce risk. We don't claim to address this problem but I think education and mitigation are good goals for us. Regarding #3, I think there has been some good progress here. I have an experimental C# plugin PDK and a JS one based on quickjs. We also support Haskell. I think there will be improvement here. And regarding the ABI comment, there are going to be layers on top of this to make it more ergonomic.

jkcorrea 4y ago

Congrats on the launch! Just yesterday I was pondering if I could do something like this for a saas I’m working on: I’d like to offer users the ability to extend their dashboard in certain ways (a la WP/Shopify plugins), but those ways require extending the ui AND the server code. Extism seems like a great fit for this, I’ll definitely be checking it out :)

nilslice 4y ago
Thanks! I want to highlight that Extism supports both server-side AND browser environments, running the same plug-ins. So SaaS is a great example of a place where end-users may want to extend functionality that runs on the client, server, or both. Feel free to reach out if you have any questions.
bhelx 4y ago
Sounds very cool and like it could be a good fit for our browser based runtime! There are still a few limits, but you can currently run the same plugins in your backend and frontend.

nmstoker 4y ago

As others have said, the term "software" is confusing. Application or software product seem more apt. The other small point I'd raise: there's mention that due to the sandboxing of WASM it's "safe" to run untrusted code. Maybe I've misunderstood. To me that seems true in so far as the untrusted code cannot break out and can only do actions the hosting app decides to allow, but this isn't much consolation if the app allows actions that can be subverted. For instance, if the app allows control over saving a file, untrusted code only gets to handle that within the rails the app sets but that still means dodgy code could fill up a hard drive by saving over and over, or it could possibly save over important files.

bhelx 4y ago
> As others have said, the term "software" is confusing. Application or software product seem more apt. That's a great suggestion thanks! > To me that seems true in so far as the untrusted code cannot break out and can only do actions the hosting app decides to allow, but this isn't much consolation if the app allows actions that can be subverted. Yes i think safe is a relative term here. Any user input is "unsafe" to some degree. Your example is a good one! I think we wrote this in the context of other relative ways to do this but will be discussing better ways to talk about security and what Extism protects and what it doesn't.

hybid 4y ago

If I want to make all software programmable, I install Frida and easily hook into pretty much anything at the same privilege level. Still, misleading headline aside, this sounds pretty cool as a reasonably well sandboxed plug-in architecture for software that chooses to use it. Though if running in-process, it seems unlikely to be able to defend against hostile plugins that use e.g. speculative execution techniques to extract otherwise inaccessible data. But perhaps that's not the threat model here.

warpech 4y ago
For those who don't know what Frida is: https://frida.re/ https://frida.re/ I must say, this sentence from their docs describe it the best to me: Greasemonkey for native apps It has little to do with Extism, but still is interesting.

pavlov 4y ago

The manifest format could perhaps be less terse. It's really hard to guess what this actually means without reading the docs: "memory": { "max": 5 } Presumably the maximum amount of memory available to the plugin, but which unit? 5 megabytes? Turns out it's measured in pages of 64k each: "The units here are pages where a page is 64Kib." [1] So max==5 apparently means a hard upper limit of 320 kB. (For when you want to run a 1984 copy of WordPerfect as a plugin, of course.) Maybe a "max_MB" property might be more useful to plugin authors, and then just calculate the # of pages internally from that? [1] https://extism.org/docs/concepts/manifest#schema https://extism.org/docs/concepts/manifest#schema

bhelx 4y ago
We have an explanation of the manifest here: https://extism.org/docs/concepts/manifest https://extism.org/docs/concepts/manifest it's still a work in progress. the point about units is a good one. Might be a way to make that more clear.
bhelx 4y ago
> Maybe a "max_MB" property might be more useful to plugin authors, and then just calculate the # of pages internally from that? yeah, that's something to consider. You can only pass in parameters that are a multiple of the page size so that may not be an accurate representation. I do think we need to hint at the unit in the name though.

madmod 4y ago

I wonder how this compares to https://suborbital.dev/ https://suborbital.dev/

callahad 4y ago
[I work for Suborbital] We're both all in on extensible software powered by WebAssembly, but the scope and domains differ. Suborbital is more of a turnkey / batteries-included service that handles the full lifecycle of extending SaaS applications with user-defined plugins. So it includes a web-based editor widget, a module builder service, storage and compute, administrative dashboards, etc. Extism, from my lay understanding, is more tightly scoped toward being a great library for running WebAssembly modules, in the same way you might embed a Lua runtime into an application for user scripting. Another big difference is that you can use all of Extism right now. Suborbital's hosted platform is still in closed beta :) (Waitlist at https://form.typeform.com/to/DHxjRGKx https://form.typeform.com/to/DHxjRGKx)
- bhelx 4y ago
  I haven't used suborbital yet but yeah, that sounds accurate. We're aimed at being a library and Extism itself is focused on building plug-in systems. I just signed up for the waitlist! Looking forward to learning more about it.

Jyaif 4y ago

Does it require JIT? In other words, does it work in iOS?

bhelx 4y ago
yes, Wasmtime does use a JIT. We haven't tried on mobile yet, but I think I have some paths to make it work. If this is something you need come join our discord and we can experiment on it!
- Williams22 4y ago
  Yes i wanna Join

kumarski 4y ago

WebAssembly ecosystem booming it seems.

floober 4y ago

This looks fantastic! I'm working on a desktop app and I've been struggling with the extensibility story in the back of my mind for a couple of weeks now. Thanks for building this! edit: For what it's worth, the use case and value of something like this was immediately apparent to me.

nilslice 4y ago
Awesome! We'd love to see what you end up building, and if you use Extism please let us know what you find missing... > For what it's worth, the use case and value of something like this was immediately apparent to me. Thank you for saying so. Negative feedback doesn't bother me (I appreciate anyone taking time to speak up), but it's also usually the only kind you get on the Internet :) so we appreciate hearing that. Translating ideas into words is hard, especially in software.

beders 4y ago

I've built a plugin-system back in 2003 or so based on the JVM which can load/re-load bytecode relatively easily. (the concept became popular in the JVM world and lead to abominations like OSGi) The problems that arose quickly were: - Executing order of plugins attaching to the same extension point. - Third party library dependencies having different versions - Traceability and debugging - Thread-safety OTOH, it gave our custom dev team to develop integrations and extensions that could be deployed and maintained independently of the main product. So there's definitely a use-case in there, but it requires some coordination regardless.

bhelx 4y ago
Would love to hear more about your experience there. > (the concept became popular in the JVM world and lead to abominations like OSGi) I remember this, though it was a couple years before my time. Although some of the underlying technology has changed those problems you outlined have remained the same. These things are all relevant to our next steps. We've been thinking a lot about what developer experience problems need to be solved.

haberman 4y ago

I'm trying to figure out what Extism is actually providing here. The underlying runtime appears to be Wasmtime, an existing project that you can use separately. For the system API, Extism supports WASI (a POSIX-like API for WASM) which is already supported by Wasmtime. However it sounds like it also has a different, non-WASI API that provides similar functionality. I'm curious to hear a compare/contrast between these two system APIs, or what motivated inventing a new non-WASI system API. It does appear that WASI is in its early days; WASI was announced in 2019 (https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/ https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webas...) but it appears that all of the specific proposals are still early stage and far from being standardized (https://github.com/WebAssembly/WASI/blob/main/Proposals.md https://github.com/WebAssembly/WASI/blob/main/Proposals.md). For the API, Extism provides a C API/ABI that can be called through FFI, but AIUI Wasmtime already has such an API: https://docs.wasmtime.dev/c-api/ https://docs.wasmtime.dev/c-api/ Basically I'm trying to understand the conceptual difference between this project and Wasmtime.

bhelx 4y ago
It's a good question. It's hard to understand what all is involved in building something like that until you have to get into the details. You can think of Extism as a layer around wasmtime. There is an ABI of sorts that helps you communicate with the plugin: Send data to it, invoke functions, and get data out. With our system, you don't need to enable WASI to use a plugin. You can invoke functions directly. However, you can use WASI for other purposes. And we are working on a way to do POSIX-like stuff (e.g. programs with a main function that use stdin and stdout). Because we wrap wasmtime, we can also switch it out. That's how we are able to run the same plugins in the browser. There we use the browsers' implementation. I think one day we will replace some internal pieces of Extism with pieces of the Component Model spec. As of today it's not in a state where we can do all of this in this many languages. edit: rewording, grammar
- syrusakbary 4y ago
  Hey! Super exciting to see your progress on Extism. I'm Syrus, from Wasmer. I did a quick review of your plugin architecture and it looks great (tons of languages supported, congrats!). I think Wasmer might be able able to help you so you have the same implementation of the runtime inside and outside of the browser (something similar as we did in wasmer-js [1], where we reuse same WASI implementation, but using the native Wasm engine from the browser). I'd love to see how we can integrate your plugins into Wasmer as well! [1] https://github.com/wasmerio/wasmer-js https://github.com/wasmerio/wasmer-js

fulafel 4y ago

How are you implementing JS platform support, is it a WebAssembly JS interpreter?

nilslice 4y ago
for JS on the Web, yes - we ported our runtime specific APIs to TypeScript and expose them to the WebAssembly runtime in the browser for JS in Node, we use wasmtime embedded within our runtime -- looking into what the tradeoffs are to re-implement the whole runtime in Node on its native WebAssembly runtime though. Feedback welcome & appreciated!

hestefisk 4y ago

Is it single threaded?

bhelx 4y ago
The plugins are, but not the host. You can create pools of plugin instances if you need concurrency. Threading inside a Wasm program is still a new concept but I think it will become more common.

satvikpendem 4y ago

Will WASM finally be the write once run everywhere system we've been promised for the past few decades? I can imagine every language compiling to the common format that is WASM, so you can mix and match libraries from other languages as you please, as well as chip manufacturers adding hardware accelerated WASM processing units if it gets popular enough. This really is the birth and death of Javascript.

marcosdumay 4y ago
No. WASM will have the same problem that every "run everywhere" system ever had: what makes software hard to port is not the ability to execute it, it's the environment differences. But it will be a powerful low trust portable VM, like none mainstream today. Odds are it kills Lua. (And yeah, it removes the Javascript monopoly on the browser. At this point, it can't be killed, but it will shrink.)
- satvikpendem 4y ago
  Perhaps we can compile a Docker image into WASM too, we've seen Postgres recently. Or do you mean something else?
- chubot 4y ago
  Lua is more like a C library and less like a VM. (Similarly, LLVM isn't a VM, which is a common misunderstanding, given the name) As far as I know, people don't ship code to remote Lua VMs and execute it. There is no compatibility guarantee across versions, and there are nontrivial compile-time options (like what the number type is). That doesn't mean Lua isn't useful; it's just not designed for the same things that WASM is.
  - marcosdumay 4y ago
    > As far as I know, people don't ship code to remote Lua VMs and execute it. So, you never used some software scripted with Lua? Or just never shared scripts? Lua does have other uses, but the vast majority of it is as a low trust VM.

NuSkooler 4y ago

Interesting. Just this week I've been building a little experiment in order to utilize WASM based plugins in a project I'm involved with. I'm using WasmEdge for the task and so far am having pretty good luck. Questions: 1. Can WASM plugins with Extism easily call back into host provided APIs? 2. Complex types are supported?

nilslice 4y ago
WasmEdge is really cool -- I'm super impressed with the work they are doing and hope to eventually provide an alternative runtime backend to Extism using WasmEdge (maybe they'd like to help!) To answer your questions: 1. Extism provides a specific set of host provided APIs, and currently we don't have support for user-supplied host functions beyond the ones we make available. This will change though, and we're looking for feedback. We held off because there wasn't a clear way to make it easy across ALL the languages we support. 2. Complex types are definitely supported, you need to encode the data into raw bytes and can use whichever format you want (only limited by which formats are available in host & plug-in languages..) Eventually we will move towards the Component Model which will make the complex types story much simpler.
- NuSkooler 4y ago
  Thanks for the info! I'll be keeping an eye on this for sure. What I'm working on is a PoC and it's working well thus far. As Extism advances and supports some of these upcoming features, it may turn out to be "just right" for my needs.

meowface 4y ago

Three thoughts: * That's an absolutely viscerally horrifying logo * I absolutely love it and its design * I am confused about why it's used for a project like this and not some kind of internal NSA project, or maybe a nu-witch house band logo, or something

nilslice 4y ago
LOL Thank you on all 3 points!! I intended to invoke thoughts about how Extism enables one to "extend from within", thus the octopus crawling out from the skull. And also that the thing Extism should replace is rather cursed... loading DLLs / dlopen tons of plug-ins into your program to execute untrusted code is downright scary.
- duxup 4y ago
  The logo is symbolic of the product or thing right (Extism), not other things? Not sure I would get the escaping from part by looking at a logo.

jayd16 4y ago

I'd love to see C# support for something like this as well. Would be nice if Unity games could start supporting wasm plugins. Not sure if something like that works for Ureal already. Another question... Is something like this possible on iOS or will these kinds of runtimes be banned by Apple?

bhelx 4y ago
C# support on which side? The host or the plugin side? I've got an experimental C# plugin almost working. I think I can do it. A Host SDK library would be pretty easy since it's just FFI to the runtime. > Would be nice if Unity games could start supporting wasm plugins That would be pretty neat! I go back and forth over whether this kind of thing will happen or not. Extism's goal is to be very generic (universal). A game engine may have some competing goals like performance or predictability. Still, I wouldn't be surprised if some alternative engines didn't do this in the next few years. > Another question... Is something like this possible on iOS or will these kinds of runtimes be banned by Apple? Not today, but I think I know how to get there. If you need this stuff for your project come join the discord and we can try to prioritize it!
nilslice 4y ago
It's already underway! https://github.com/extism/extism/pull/119 https://github.com/extism/extism/pull/119

zacharyvoase 4y ago

I'm genuinely curious: is the WebAssembly ecosystem just reinventing the JVM piecemeal? "Write once run anywhere"?

bhelx 4y ago
In many ways yes. We need to re-invent many things for this new world. But I'd encourage you to read a bit more about how Wasm is different. It's true it is a stack based virtual machine but that's about where the comparisons end.
slaymaker1907 4y ago
I think you're probably right, but I don't think many people would look at the JVM as a failure. Instead, I think WASM is trying to bring the good parts of the JVM such as being a simple compile target, small binaries, and a stable VM while fixing the bad parts like having a huge attack surface area and lacking certain features critical for performance (pointer chasing, GC, etc.).
- zacharyvoase 4y ago
  Well as vendors add access to system APIs to WASM, that attack surface will indeed grow.
moffkalast 4y ago
No, it's just "let's take random languages and transpile them to Javascript but call it Webassembly and hope nobody notices". In that sense it's much worse.

rvcdbn 4y ago

Why not use protobuf for serializing data between plugin and host language?

bhelx 4y ago
You can bring your own serialization format. It's just bytes in and bytes out.

ynx 4y ago

Damn, this looks really cool. I've wanted to add wasm support to my project at work for some time now, but unfortunately with no good C# integrations (with Unity as the host), it just hasn't met the mark yet. Really hoping that can be addressed in the near future.

bhelx 4y ago
If you need C# support we can work on it! I've done some experimenting with it already. Please come join the discord or file an issue.

n4jm4 4y ago

Why not continue using long-standing C FFI? This is not so clear.

nilslice 4y ago

For those who want to go straight to the code: https://github.com/extism/extism https://github.com/extism/extism

ilyt 4y ago

Huh, just a week ago I thought it would be cool to add WASM plugins to my Go app, as a way to not be dependent on Go for plugins.

nilslice 4y ago
I hope Extism can help you make that happen! Here’s an example of a Go app that calls Extism plugins in case it helps: https://github.com/extism/extism-fsnotify https://github.com/extism/extism-fsnotify

Existenceblinks 4y ago

The pdk is missing Zig which is, for many developers, considerably the best for wasm target. It's as fast as Rust (from benchmark I saw, Rust is fastest), though Zig's binary size is as small as AssemblyScript's. Rust's binary size is considerably in a range of middle to large, mostly middle. It would be nice if there's an official wasm benchmark of source languages, showing speed and binary size comparison.

bhelx 4y ago
Yeah, that's a good idea. C is supported on the SDK and PDK side. We are wondering if Zig can utilize the c libraries for it.
- Existenceblinks 4y ago
  I would say Zig is the nicest C friend! Zig can compile C code. Checkout: https://ziglang.org/learn/overview/#zig-is-also-a-c-compiler https://ziglang.org/learn/overview/#zig-is-also-a-c-compiler Also: https://ziglang.org/documentation/master/#toc-Freestanding https://ziglang.org/documentation/master/#toc-Freestanding

deleted 4y ago

[deleted]

cloudmike 4y ago

As others have mentioned, "making software programmable" isn't really landing for me. In the tradition of borrowing terms from the games industry (where we often see early popularization of good ideas), I wonder if "modding" could be a helpful addition somewhere in your marketing. This seems to have worked for the term "multiplayer", though "moddable" probably isn't as widely appreciated yet.

Kinrany 4y ago
Even in gaming modding is still somewhat niche.

computerdork 4y ago

If I understand it correctly, it is as overlap with the "sidecar" design-pattern, correct? (https://learn.microsoft.com/en-us/azure/architecture/patterns/sidecar https://learn.microsoft.com/en-us/azure/architecture/pattern...) If so, then super interesting product! Have actually had a somewhat similar idea myself for a product that allows you to create "subsystems" that are isolated from one another, allowing you great freedom in how the subsystems are deployed. They could be deployed all on the same machine, or, if greater scalability is needed, a subsystem (or plugin) could be moved to its own machine, or even scaled up to run multiple instances. Have only read through the simple code example on your website, but also, it would be great if there was a standard for calling and receiving requests through FFI's. Because this would give it the utmost flexibility on where the plugins can be deployed to. To me, this means both the actual mechanism for the call (REST over HTTP, graphQL...) and in the plugin and host themselves, with the syntax of the call. So for the syntax, it would also be great if this looked more like a regular method call instead of having to wrap parameters in an object: hostWrapper.somemethod(param1, param2) or hostWrapper.call("somemethod", param1, param2) (Although, not sure how difficult this is to do, given all the different method call syntaxes of the different languages). And, am glad that you and your company is really trying to make this happen, it's a great idea! Looking forward to giving it a try!

nilslice 4y ago
hey, thanks for checking out the project! it's not quite the sidecar pattern (assuming i'm interpreting the link you shared correctly) - since Extism is actually _embedded_ into your program's process, not isolated over a network boundary like a sidecar. the benefit of this is typically much faster execution speed, and thanks to WebAssembly's security model, it's safe to execute plug-in code - unlike a DLL or dlopen to a .dylib or .so file. in effect, it becomes an actual function call, not a RPC. please correct me if i've misinterpreted your post!
- computerdork 4y ago
  Think you interpreted the Side Car pattern pretty well:) It definitely is typically done using HTTP (REST), and the benefit of this is it gives it great flexibility, allowing it to be deployed in many different ways. Like Extism, sidecars are typically developed in some type of container like Docker (I have a limited understanding of WASM but since it's also a container technology, think you could also use this instead). And since the calls into the container are done through a standard like HTTP REST (or GraphQL), it's no problem if you'd like to have the container deployed locally, or on the same machine as your host container, or even on it's own server if you need to run it as its own microservice. Although typically, sidecars are deployed locally and work almost exactly like Extism - simply as a plugin to the main application. And the performance of the local call is pretty good, because the call is usually made over a virtual network you'd setup in your docker configuration files. I came across sidecars for the first time a couple of years ago while doing a job at Atlassian. And, for me, creating an entire product like Extism to take plugin/sidecars to the next level seems like a great idea. Because sidecars are awesome and we need a way to make them easier and easier to develop! Hmm, was just thinking it over, wonder if it be useful if Extism did allow flexibility in deployment (meaning that the mechanism for calls wasn't just limited to using WebAssembly to do the call/security), but could switched to using HTTP REST or GraphQL by some simple changes to some config files? Because now the plugins can be deployed in many different ways, and even work in legacy systems (they may have their own security model. Seems like some type of REST security is pretty popular)... ... although, as mentioned, from what I saw, seemed like 95% of the time, sidecars are used locally as well, like Extism, so maybe just focusing on local calls is good enough?? ... Am also wondering if supporting both Docker and WASM might be useful/possible? And of course, I have very limited understanding of Extism, so of course, I may be waay off in what the future directions of your product, so please take everything I said with a grain of salt. But, Extism does seem like a great idea! Will give it a try.

est 4y ago

Is it possible to run wasm blobs directly on Linux?

deleted 4y ago

[deleted]

stevefan1999 4y ago

and...still no dotnet

nilslice 4y ago
it's underway! https://github.com/extism/extism/pull/119/files https://github.com/extism/extism/pull/119/files

garganzol 4y ago

Extism tries to cover two completely different things in one package: 1) runtime 2) component model. And this is a problem. Those things should be separated because the component model would be useful independently, without any ties to the specific runtime environment be it WebAssembly or something else. We already saw this playbook many times before. For instance, .NET. It has a handy set of system types that define the component model, but surprise, it can only be used in the realm of .NET which makes it unusable in, say, Go. Java - the same thing. C++ is the same. Everything more abstract than some primitive data types and plain functions just cannot cross the boundaries between the language ecosystems. If we had a standard component model then it would allow us to create a library in one language and then use it from many other languages without any manual work. But all we have now is the old plain C-like API as a common denominator between them. It's 1978 all over again for the last 40 years. But I still believe that we can do better than that. Instead of m * n complexity when every library is doomed to be manually ported to n languages, we may have just m + n complexity by leveraging the benefits of a standard component model. Just imagine the economic effects of such achievement. The main problem of an imaginary de-facto standard component model is the danger of being opinionated. We should solve that, otherwise all such efforts will be doomed to fail from the beginning. Data and functions are good starting points, and they are universal by the laws of math and algebra. That's why they are not opinionated. Consequently, they are used everywhere. We should build a similar formal apparatus for, say, objects to allow OOP interoperability between the languages once and for all.

iecheruo 4y ago
To be fair to .NET what you describe sounds an awful lot like COM/WinRT

Mikhail_K 4y ago

More like, "make all software exploitable"

winnie_ua 4y ago

To be honest, I don't see why it is advertised here at this stage. I appreciate effort, but project is too young and unable to do any serious plugin. I guess it is try to get new contributors on board? I see that project on github started few months ago. Isn't it too early to adwertise it to the world? But it already have fancy website with modern styles and so on. But it lacks good documentation. Article says that docs have "dozens of examples in different languages", but it just have dozen languages and helloworld in them. And basically examples are just simple function call from host to plugin. Operating on strings/basic types, and JSON. That's not what we want from plugins. Wee need some way to expose Host API some way, perhaps in OOP fashion. To let plugin interract with host app, perhaps even change UI. But not of examples show that, how to pass at least basic functions into plugin. As I understand it comes with it's own HTTP stack, presumably implemented in Rust. This can be used as starting point to make your own macro processor that would be able to expose API for plugins, and then dogfoof this codegen for your own HTTP lib. And passing JSON around is not thing that is needed. With same success IPC could be used, with relatively similar performance. We need proper marshaling. There much work to be done, sadly. That's big endeavor. Well, I hope this project would grow in something good, but it's in really early stage.