5 ms·

You can't ship reasonable plugin for most of the popular content creation tools - be it audio, video effects, 3d modelling, etc. without support for dynamic/sha

by malkia 4y ago

You can't ship reasonable plugin for most of the popular content creation tools - be it audio, video effects, 3d modelling, etc. without support for dynamic/shared libs. Okay, maybe you can - at the risk of performance cost - e.g. your out-of process plugin that need to move data in/out with a cost, and then it over complicates - because the plugin process might've died - how do you restart it, how do you get it back to state, etc. etc. You just don't have to solve this with .dll or .so

greyw 4y ago
I'm not saying you need to never use dynamic libraries. Some tools (especially handling/supporting puligins) obviously should make use of dlopen and co. It's just doesn't make sense in my head why e.g. nano dynamically links against 10 libraries. Lots of pain for little gain. Latest dynamic linking bug I just encountered is that big picture mode of steam crashes somewhere in libstdc++ when said library is too new but works fine on older versions.
- deleted 4y ago
  [deleted]
pornel 4y ago
Apart from most perf critical cases, it's a good idea to run plugins in a separate process anyway. This way a bad plugin won't crash or corrupt your own process. You will be able to stop and unload it without risk of leaving locks locked or thread-locals leaking. And also there's WASM now which has even better sandboxing and OS/architecture independence.
- pjmlp 4y ago
  WASM is nothing new, just a recyling of old ideas with refueled marketing.
  - Ygg2 4y ago
    Doesn't it work on all platforms? In theory, at least?
    - pjmlp 4y ago
      Like plenty of other bytecode formats that predated it.
      - malkia 4y ago
        And good luck debugging / make it performant and still not clear how well the memory allocation is handled (in the dll-host situation, often the plugin may use the alloc/free functionality from the host, and give hints, in wasm case - this would be a bit of sandbox)
  - pornel 4y ago
    Something existing and succeeding is not automatically a claim that it's the only one possible thing, or even a novel idea, so dismissal "it's just recycling old ideas" is non-sequitur. I'm not even sure what you're trying to say with this comment. Is only 1958 LISP or maybe SmallTalk worthy and truly novel, and every other technology must come with a disclaimer that it's just a pale imitation and everything has been done before? Maybe there was some underappreciated pioneer PL/BYTECODE-1968, but in 2023 there's WASM, CIL, Java bytecode (with marketing as large as the dotcom bubble), maybe Lua or eBPF. For practical application it absolutely doesn't matter which one was first or the most humble. What matters is that WASM is common popular standard now. It has easy to embed runtimes and decent language support. The marketing around it is a huge plus too, because I can tell people "use WASM" and it'll be cool and they'll use it, instead of e.g. instinctively reacting "ewwww, Java".
- mike_hearn 4y ago
  Note that this is how things were done before dynamic linking was invented, and dynamic linking was seen as a big step forward. Processes+shared memory is a really painful and slow paradigm compared to just invoking function calls on the same stack and in the same process. "a bad plugin won't crash or corrupt your own process." It still can. The plugin just has to return bad data you're not prepared to handle, or crash in a way you didn't anticipate and code around. And note that with higher level languages the whole thing is moot. A plugin crashing just means it throws an exception which can be caught, reported and quite likely the host process will survive just fine even though it's all in-process and dynamically loaded.
Shorel 4y ago
You can, just change the strategy. Shared memory exists. That's how we read timing information from AMS, Project Cars 2, AMS2, rFactor2, etc.
- malkia 4y ago
  I would like to learn more! (gamedev here too) - Is there any GDC presentation about it?
  - Shorel 4y ago
    I use a Python script to read the shared memory and generate JSON with this information, then a web page using this JSON is overlaid using OBS, and this way we transmit races on YouTube. Here is how it looks like: https://youtu.be/ijSL8uN8cIw?t=5582 https://youtu.be/ijSL8uN8cIw?t=5582 If you want to read about a public project that uses this strategy, here is the repository link: https://gitlab.com/winzarten/SecondMonitor https://gitlab.com/winzarten/SecondMonitor I don't know C#, but it should work similarly. This also means you definitely don't need to use the same language to create and to consume the information.
justincormack 4y ago
Cross process shared memory is a thing on Unix so there is no need for the performance cost of copying across a pipe. Sure you have to manage if it dies, but you have to manage it crashing you in the dl case.
- malkia 4y ago
  But you don't have only Linux, you have (and probably first) OSX and/or Windows and then Linux, and you need an API that works well in that case. Being an in-process dll/so plugin, while fraught with perils gets you to avoid other issues (state, health, restart, identity, etc.) Also sometimes you don't have a choice, but have to make a dll, for example: https://github.com/steinbergmedia/vst3sdk https://github.com/steinbergmedia/vst3sdk or https://ae-plugins.docsforadobe.dev/ https://ae-plugins.docsforadobe.dev/ and many others. Sometimes it's the only viable choice. (I wish most have used grpc/flatbuffers/whatever to communicate, but then every RPC call have to be checked/retried/handled, and or shared memory well handled, with (?) locks, etc. - not a trivial thing for someone who is deeply specialized in making a very good effect/renderer/etc instead of dealing with this extra complexity on top).

by malkia 4y ago

greyw 4y ago

I'm not saying you need to never use dynamic libraries. Some tools (especially handling/supporting puligins) obviously should make use of dlopen and co. It's just doesn't make sense in my head why e.g. nano dynamically links against 10 libraries. Lots of pain for little gain. Latest dynamic linking bug I just encountered is that big picture mode of steam crashes somewhere in libstdc++ when said library is too new but works fine on older versions.

deleted 4y ago
[deleted]

pornel 4y ago

Apart from most perf critical cases, it's a good idea to run plugins in a separate process anyway. This way a bad plugin won't crash or corrupt your own process. You will be able to stop and unload it without risk of leaving locks locked or thread-locals leaking. And also there's WASM now which has even better sandboxing and OS/architecture independence.

pjmlp 4y ago
WASM is nothing new, just a recyling of old ideas with refueled marketing.
- Ygg2 4y ago
  Doesn't it work on all platforms? In theory, at least?
  - pjmlp 4y ago
    Like plenty of other bytecode formats that predated it.
    - malkia 4y ago
      And good luck debugging / make it performant and still not clear how well the memory allocation is handled (in the dll-host situation, often the plugin may use the alloc/free functionality from the host, and give hints, in wasm case - this would be a bit of sandbox)
- pornel 4y ago
  Something existing and succeeding is not automatically a claim that it's the only one possible thing, or even a novel idea, so dismissal "it's just recycling old ideas" is non-sequitur. I'm not even sure what you're trying to say with this comment. Is only 1958 LISP or maybe SmallTalk worthy and truly novel, and every other technology must come with a disclaimer that it's just a pale imitation and everything has been done before? Maybe there was some underappreciated pioneer PL/BYTECODE-1968, but in 2023 there's WASM, CIL, Java bytecode (with marketing as large as the dotcom bubble), maybe Lua or eBPF. For practical application it absolutely doesn't matter which one was first or the most humble. What matters is that WASM is common popular standard now. It has easy to embed runtimes and decent language support. The marketing around it is a huge plus too, because I can tell people "use WASM" and it'll be cool and they'll use it, instead of e.g. instinctively reacting "ewwww, Java".
mike_hearn 4y ago
Note that this is how things were done before dynamic linking was invented, and dynamic linking was seen as a big step forward. Processes+shared memory is a really painful and slow paradigm compared to just invoking function calls on the same stack and in the same process. "a bad plugin won't crash or corrupt your own process." It still can. The plugin just has to return bad data you're not prepared to handle, or crash in a way you didn't anticipate and code around. And note that with higher level languages the whole thing is moot. A plugin crashing just means it throws an exception which can be caught, reported and quite likely the host process will survive just fine even though it's all in-process and dynamically loaded.

Shorel 4y ago

You can, just change the strategy. Shared memory exists. That's how we read timing information from AMS, Project Cars 2, AMS2, rFactor2, etc.

malkia 4y ago
I would like to learn more! (gamedev here too) - Is there any GDC presentation about it?
- Shorel 4y ago
  I use a Python script to read the shared memory and generate JSON with this information, then a web page using this JSON is overlaid using OBS, and this way we transmit races on YouTube. Here is how it looks like: https://youtu.be/ijSL8uN8cIw?t=5582 https://youtu.be/ijSL8uN8cIw?t=5582 If you want to read about a public project that uses this strategy, here is the repository link: https://gitlab.com/winzarten/SecondMonitor https://gitlab.com/winzarten/SecondMonitor I don't know C#, but it should work similarly. This also means you definitely don't need to use the same language to create and to consume the information.

justincormack 4y ago

Cross process shared memory is a thing on Unix so there is no need for the performance cost of copying across a pipe. Sure you have to manage if it dies, but you have to manage it crashing you in the dl case.

malkia 4y ago
But you don't have only Linux, you have (and probably first) OSX and/or Windows and then Linux, and you need an API that works well in that case. Being an in-process dll/so plugin, while fraught with perils gets you to avoid other issues (state, health, restart, identity, etc.) Also sometimes you don't have a choice, but have to make a dll, for example: https://github.com/steinbergmedia/vst3sdk https://github.com/steinbergmedia/vst3sdk or https://ae-plugins.docsforadobe.dev/ https://ae-plugins.docsforadobe.dev/ and many others. Sometimes it's the only viable choice. (I wish most have used grpc/flatbuffers/whatever to communicate, but then every RPC call have to be checked/retried/handled, and or shared memory well handled, with (?) locks, etc. - not a trivial thing for someone who is deeply specialized in making a very good effect/renderer/etc instead of dealing with this extra complexity on top).