4 ms·

And don't forget, the plugin API has some super strong limitations. I wish that was fixed, otherwise platforms "similar to wordpress" can never become a thing o

And don't forget, the plugin API has some super strong limitations. I wish that was fixed, otherwise platforms "similar to wordpress" can never become a thing on Go

Xeoncross 4y ago
Plugins for a compiled binary are just plain hard. Changing the code works better with an interpreted language. Usually you see the software implement it's own interpreter or a scripting language on top to handle this.
- Fire-Dragon-DoL 4y ago
  I know, I was thinking C# does something like that though, so I was hoping it would be achievable. The annoying part about the scripting language is that if something is written in Go, you want developers to also work in Go, otherwise the people writing plugins for your software, knowing better the API and use-cases, have a disconnection with the language actually used for the software, losing potential opportunities for contributions.
  - int_19h 4y ago
    C# (usually) runs on top of a VM with a JIT, so the linking is effectively always done at runtime. It's also why it can do things like generic virtual methods - it can compile an instantiation and adjust the vtable at runtime.
- wahern 4y ago
  Plugins for ELF- and Mach-O-based systems aren't any different than regular shared libraries, modulo slightly different symbol precedence. AFAIU, while a quite different architecture, Windows PE linking treats them quite similarly, too. Perhaps you meant statically compiled binaries? Even then I'm not so sure about that. Right now I'm working on a project that statically compiles a plugin so its various libraries can't leak or be overridden by the loading application, yet which itself can load other plugins. OTOH, this is in C, C++, Objective-C, and Lua (basically C from the perspective of binary linking), all of which have mature linking semantics. (Well, at least this is true for C. Controlling symbol namespace pollution by C++ and Objective-C code requires more complexity than for C, but at least the necessary compiler and linker flags exist and are mature.) When you use languages or toolchains that don't invest in an ABI, or which make linking too automagic (with no or little recourse for exposing various linking features a platform may offer), then runtime linking is likely to become a major limitation in some solution areas. For example, there's probably no way to coax XCode to compile my plugins properly, without outsourcing some logic to additional scripts. Ultimately a Makefile is likely the best solution--and the one I chose, anticipating these headaches--as it keeps most of the build transparent. Fortunately, XCode seems to just be a wrapper around a bunch of command-line utilities, so you don't actually need XCode at all.
  - mattarm 4y ago
    Beyond linking it is also a problem with ABI compatibility of object layouts, etc. It is so much simpler with more dynamic languages.
    - int_19h 4y ago
      The C ABI on pretty much any platform defines object layout for structs, which is usually good enough.
      - yencabulator 4y ago
        The C ABI is nowhere near enough for passing arbitrary Go structs, all you need is a map or a sync.Mutex field and you need to exactly match plugin compilation to caller compilation. I wish https://pkg.go.dev/plugin https://pkg.go.dev/plugin was clearly documented as not being a viable 3rd party plugin solution.
        int_19h 4y ago
        It's not enough to pass arbitrary C++ classes, either. The point is that you can still design a fairly expressive API involving structured data within the constraints that the C ABI imposes on you - and that C ABI will guarantee stability. This is how plugins were usually handled in the C++ world, historically speaking. (There are also higher-level ABIs like COM, but they are usually reducible to the C ABI in practice - e.g. COM can be described entirely in terms of struct and function pointers.)
        yencabulator 4y ago
        You can write C plugins, or C-ABI plugins in any language that can do that, and call them from Go just fine. Nobody just considers that a very good way to write Go, or plugins for Go.
cube2222 4y ago
Actually, folks usually just use gRPC or Yaegi in Go. You can write an SDK for your software that abstracts all of that away for the plugin developer. See Terraform[0], Traefik[1], or OctoSQL[2]. Though I agree plugins would be welcome, especially for performance reasons, though also to be able to compile and load go code into a running go process (JIT-ish). [0]: https://github.com/hashicorp/terraform https://github.com/hashicorp/terraform [1]: https://github.com/traefik/traefik https://github.com/traefik/traefik [2]: https://github.com/cube2222/octosql https://github.com/cube2222/octosql Disclaimer: author of OctoSQL
- Fire-Dragon-DoL 4y ago
  Well aware of all that, but as you point out, real plugins would be better. And to be clear they exist, but you have to compile them with the same go version and if any dependency is shared, they need to be exactly the same version. There are also serious limitations in the sharable code
- zimbatm 4y ago
  I have had cases where Terraform would OOM because each Terraform provider instance maps to a fresh new Go process. Most of the time this approach is not an issue, and it works really well. But you need to be aware that it's best to keep the number of providers in the 0-20 range per root module.
morelisp 4y ago
If I was going to try to make something "similar to Wordpress" with Go I would replace plugins with optional packages triggered by build flags (or possibly a light codegen step), not an actual dynamic plugin system. The Wordpress model is solved with plugins in PHP (for reasons that were to some degree bad even at the time), but it's not actually "I want to inject some external code into this existing process environment", it's "I want a bespoke N of M features because some Ms are mutually exclusive or have overhead I can't afford". If you have a language that has a module ecosystem, compiles fast, and generates a single binary, take advantage of that; don't try to reimplement fragile dynamic linking.
- Fire-Dragon-DoL 4y ago
  WordPress is an example. For C# videogame bots, the software is delivered as a compiled executable file (no sources) and plugins are sold as DLLs that you can just copy-paste in a directory. Kerbal Space Program (videogame) is a big compiled executable and plugins are loaded as DLLs from a directory. I'm not trying to reimpostare anything, I just wish it was available.
- Fire-Dragon-DoL 4y ago
  I'd like to point out also that wordpress can install plugin directly from the UI, which is a lot more involved if it requires the whole software to be recompiled (totally different machine too)
  - morelisp 4y ago
    There’s no reason that UI can’t kick off a recompile on the backend, nor any reason it has to be a different machine.
    - Fire-Dragon-DoL 4y ago
      Recompiling the whole software is way more expensive than pasting a file on the filesystem, that was my point.
Philip-J-Fry 4y ago
I've written plugins in Go. You don't really need to use the plugin package, and you probably shouldn't anyway. One way I've done it in the past is via sub processes talking via stdin/stdout. Another way I've done it is just via a regular REST API. You'll never get enough performance for a lot of tasks if you need to pass a lot of data between applications. But depending on your needs it's more than serviceable. The benefit of this is that plugins don't need to be written in Go, as you have a simple interface. A good example of this is something like LSP. LSP plugins are abundant and each one is written in a different language. VSCode is Typescript, the Go language server is written in Go. Both can speak to eachother fast enough.
- Fire-Dragon-DoL 4y ago
  Unfortunately, the performance as you point out is not even close. Not only that, but the API is way more limited by the transport protocol, instead of being able to directly use language interfaces. If you write plugins for PHP, they are just as fast as the basic software. If you write plugins for C# (dlls), they are as fast as the language. Writing using grpc has substantial development overhead and performance overhead, so it makes Go a bad choice by default

And don't forget, the plugin API has some super strong limitations. I wish that was fixed, otherwise platforms "similar to wordpress" can never become a thing on Go

Xeoncross 4y ago

Plugins for a compiled binary are just plain hard. Changing the code works better with an interpreted language. Usually you see the software implement it's own interpreter or a scripting language on top to handle this.

Fire-Dragon-DoL 4y ago
I know, I was thinking C# does something like that though, so I was hoping it would be achievable. The annoying part about the scripting language is that if something is written in Go, you want developers to also work in Go, otherwise the people writing plugins for your software, knowing better the API and use-cases, have a disconnection with the language actually used for the software, losing potential opportunities for contributions.
- int_19h 4y ago
  C# (usually) runs on top of a VM with a JIT, so the linking is effectively always done at runtime. It's also why it can do things like generic virtual methods - it can compile an instantiation and adjust the vtable at runtime.
wahern 4y ago
Plugins for ELF- and Mach-O-based systems aren't any different than regular shared libraries, modulo slightly different symbol precedence. AFAIU, while a quite different architecture, Windows PE linking treats them quite similarly, too. Perhaps you meant statically compiled binaries? Even then I'm not so sure about that. Right now I'm working on a project that statically compiles a plugin so its various libraries can't leak or be overridden by the loading application, yet which itself can load other plugins. OTOH, this is in C, C++, Objective-C, and Lua (basically C from the perspective of binary linking), all of which have mature linking semantics. (Well, at least this is true for C. Controlling symbol namespace pollution by C++ and Objective-C code requires more complexity than for C, but at least the necessary compiler and linker flags exist and are mature.) When you use languages or toolchains that don't invest in an ABI, or which make linking too automagic (with no or little recourse for exposing various linking features a platform may offer), then runtime linking is likely to become a major limitation in some solution areas. For example, there's probably no way to coax XCode to compile my plugins properly, without outsourcing some logic to additional scripts. Ultimately a Makefile is likely the best solution--and the one I chose, anticipating these headaches--as it keeps most of the build transparent. Fortunately, XCode seems to just be a wrapper around a bunch of command-line utilities, so you don't actually need XCode at all.
- mattarm 4y ago
  Beyond linking it is also a problem with ABI compatibility of object layouts, etc. It is so much simpler with more dynamic languages.
  - int_19h 4y ago
    The C ABI on pretty much any platform defines object layout for structs, which is usually good enough.
    - yencabulator 4y ago
      The C ABI is nowhere near enough for passing arbitrary Go structs, all you need is a map or a sync.Mutex field and you need to exactly match plugin compilation to caller compilation. I wish https://pkg.go.dev/plugin https://pkg.go.dev/plugin was clearly documented as not being a viable 3rd party plugin solution.
      - int_19h 4y ago
        It's not enough to pass arbitrary C++ classes, either. The point is that you can still design a fairly expressive API involving structured data within the constraints that the C ABI imposes on you - and that C ABI will guarantee stability. This is how plugins were usually handled in the C++ world, historically speaking. (There are also higher-level ABIs like COM, but they are usually reducible to the C ABI in practice - e.g. COM can be described entirely in terms of struct and function pointers.)
        yencabulator 4y ago
        You can write C plugins, or C-ABI plugins in any language that can do that, and call them from Go just fine. Nobody just considers that a very good way to write Go, or plugins for Go.

cube2222 4y ago

Actually, folks usually just use gRPC or Yaegi in Go. You can write an SDK for your software that abstracts all of that away for the plugin developer. See Terraform[0], Traefik[1], or OctoSQL[2]. Though I agree plugins would be welcome, especially for performance reasons, though also to be able to compile and load go code into a running go process (JIT-ish). [0]: https://github.com/hashicorp/terraform https://github.com/hashicorp/terraform [1]: https://github.com/traefik/traefik https://github.com/traefik/traefik [2]: https://github.com/cube2222/octosql https://github.com/cube2222/octosql Disclaimer: author of OctoSQL

Fire-Dragon-DoL 4y ago
Well aware of all that, but as you point out, real plugins would be better. And to be clear they exist, but you have to compile them with the same go version and if any dependency is shared, they need to be exactly the same version. There are also serious limitations in the sharable code
zimbatm 4y ago
I have had cases where Terraform would OOM because each Terraform provider instance maps to a fresh new Go process. Most of the time this approach is not an issue, and it works really well. But you need to be aware that it's best to keep the number of providers in the 0-20 range per root module.

morelisp 4y ago

If I was going to try to make something "similar to Wordpress" with Go I would replace plugins with optional packages triggered by build flags (or possibly a light codegen step), not an actual dynamic plugin system. The Wordpress model is solved with plugins in PHP (for reasons that were to some degree bad even at the time), but it's not actually "I want to inject some external code into this existing process environment", it's "I want a bespoke N of M features because some Ms are mutually exclusive or have overhead I can't afford". If you have a language that has a module ecosystem, compiles fast, and generates a single binary, take advantage of that; don't try to reimplement fragile dynamic linking.

Fire-Dragon-DoL 4y ago
WordPress is an example. For C# videogame bots, the software is delivered as a compiled executable file (no sources) and plugins are sold as DLLs that you can just copy-paste in a directory. Kerbal Space Program (videogame) is a big compiled executable and plugins are loaded as DLLs from a directory. I'm not trying to reimpostare anything, I just wish it was available.
Fire-Dragon-DoL 4y ago
I'd like to point out also that wordpress can install plugin directly from the UI, which is a lot more involved if it requires the whole software to be recompiled (totally different machine too)
- morelisp 4y ago
  There’s no reason that UI can’t kick off a recompile on the backend, nor any reason it has to be a different machine.
  - Fire-Dragon-DoL 4y ago
    Recompiling the whole software is way more expensive than pasting a file on the filesystem, that was my point.

Philip-J-Fry 4y ago

I've written plugins in Go. You don't really need to use the plugin package, and you probably shouldn't anyway. One way I've done it in the past is via sub processes talking via stdin/stdout. Another way I've done it is just via a regular REST API. You'll never get enough performance for a lot of tasks if you need to pass a lot of data between applications. But depending on your needs it's more than serviceable. The benefit of this is that plugins don't need to be written in Go, as you have a simple interface. A good example of this is something like LSP. LSP plugins are abundant and each one is written in a different language. VSCode is Typescript, the Go language server is written in Go. Both can speak to eachother fast enough.

Fire-Dragon-DoL 4y ago
Unfortunately, the performance as you point out is not even close. Not only that, but the API is way more limited by the transport protocol, instead of being able to directly use language interfaces. If you write plugins for PHP, they are just as fast as the basic software. If you write plugins for C# (dlls), they are as fast as the language. Writing using grpc has substantial development overhead and performance overhead, so it makes Go a bad choice by default