6 ms·

Golang Package: plugin

107 points by pythonist 10y ago

orta 10y ago
This is awesome, a plugin structure like this is one of the fundamentals of macOS programming (they're called NSBundles in those cases.)
junke 10y ago
> A plugin is only initialized once, and cannot be closed. Why not?
- IshKebab 10y ago
  I guess it's because if you load a symbol there's no clean way to invalidate it, especially in a thread-safe way. Something like that anyway.
  - jerf 10y ago
    I think it's safe to say there is just plain no way to invalidate it. Even if it were hacked together, it would be unusably dangerous for even trivial cases. Module loading is something that can be added to a language later, but I suspect safe module unloading is something that has to go in from day one to ever work right. Even in languages that support it (Erlang) it's got a tricky list of caveats to keep track of that strike me as fairly essential to the problem, rather than accidental (in the Fred Brooks sense of the terms).
- pjmlp 10y ago
  Because unloading, while possible is a pain to get right. Hence why Java and .NET restrict it to Class Loaders/AppDomains.
  - Someone 10y ago
    For those who wonder why: some of the things you have to make sure of are if you want to implement this correctly: - no thread is running code in the plugin. - no thread has a return address on the stack somewhere in the plugin. - no variable stores a pointer pointing into the plugin (This includes objects whose destructor is implemented in the plugin) - no signal handler points into the plugin. - no kernel thread is reading the plug-ins memory. In addition there are race conditions to take care of (e.g. thread T unloads a plugin that thread U is loading) With a 64 bit address space, I guess you can get around most if not all problems by mapping the memory that was mapped to the plugin as "can't read, can't write, can't execute", so that you crash hard if somebody tries to access it, but if you do that, you can just as well keep the code around. Go is garbage-collected, so it likely already has a lot of the logic to do this, but even then, combining the functionality to get a robust implementation of plugin unloading is a lot of work.
gtrubetskoy 10y ago
Curious what it would take (if at all possible?) to load such a .so from C. Because that would open the door for writing Go libs for all your favorite interpreted languages, web servers, etc.
- fjl 10y ago
  You can already create shared libraries that expose a C API using -buildmode=shared.
- deleted 10y ago
  [deleted]
- dvirsky 10y ago
  That's already possible, I've written an API that encapsulates Go libraries as Redis modules.
  - Goopplesoft 10y ago
    Link?
    - dvirsky 10y ago
      It's a WIP and I haven't had the time to complete it, but here it is. You can load modules, handle redis commands and return responses, but you can't access redis itself yet. https://github.com/RedisLabs/rgm/blob/master/example/module.go https://github.com/RedisLabs/rgm/blob/master/example/module....
- robbles 10y ago
  here's a pretty cool example: https://blog.heroku.com/see_python_see_python_go_go_python_go https://blog.heroku.com/see_python_see_python_go_go_python_g...
speps 10y ago
I see an API fail coming. It seems plugin.Load takes the filename WITH extension where it shouldn't. Okay, right now it seems it's only supported on Linux but it ends up in Windows, you'd have to patch code to remove the .so. Mono handles it pretty well, the extension is optional.
- regecks 10y ago
  Not necessarily. https://tip.golang.org/src/plugin/plugin_dlopen.go https://tip.golang.org/src/plugin/plugin_dlopen.go " 5 // +build linux,cgo" The Windows version isn't there yet - it could encapsulate that logic.
- IshKebab 10y ago
  I don't think so. I think it is better that the file naming is handled in the application. It is less restrictive, e.g. you can use custom extensions this way. Consider another instance where file names are automatically mangled: GCC linking with libraries with the switch -lfoo. That has honestly caused me no end of problems. It would have been better if they required an explicit -llibfoo.so from the start. Go already has several easy methods for OS-specific code if you want.
  - dvirsky 10y ago
    Not sure it should be be part of this package, but it's trivial to write a function that exposes something like "SharedLibraryFile(libname string) string". Depending on the OS it just formats the default shared object name. For extra runtime speed you could build OS specific versions of that function using build flags, so for each target OS it has one implementation and you have no switch/case or anything in runtime.
- dsl 10y ago
  Assuming or magically guessing file extensions leads to all sorts of confusion attacks that could result in arbitrary code execution. I would go so far as to say an absolute path should be required.
SoapSeller 10y ago
Didn't one of the main selling points of Go is one big static executable with no external dependencies? What's next? Generics?
- dvirsky 10y ago
  Although this can be abused, the use case is not external dependencies IMO, but rather extending systems. Think of a web server etc. Each plugin is still a static binary that is self contained, but you can load capabilities into the server in runtime. This is not like dynamic linking hell you can get yourself into when writing C.
  - clappski 10y ago
    But isn't that exactly how dynamic linking works? By loading a DLL at runtime? The only issues I've ever had with dynamic linking are at _runtime_, where XXX.dll can't be found.
    - dvirsky 10y ago
      yes, my above comment wasn't phrased that well. but do distinguish between a plugin system where you load so's or DLLs to extend an app, and an application that has dynamic linking against them, where the linker handles this. So basically something like apache modules - either you can load the module or not - vs something like "cannot find symbol foo in foo.so.0.1."
  - wtbob 10y ago
    > Each plugin is still a static binary that is self contained, but you can load capabilities into the server in runtime. In my Go code, I've just used subprocesses for that. Stdin & stdout, plus a simple serialisation format, and I'm done. Not suitable for high performance stuff, of course, but good enough for my needs.
    - dvirsky 10y ago
      yeah, it could work for sure if latency is not an issue. It's also possible to do a go->C->go bridge to load Go plugins dynamically.
pjmlp 10y ago
Nice to see this coming, one step closer to have Oberon System 3 with Gadgets rewritten in Go. :)
guessmyname 10y ago
I agree with this comment from Reddit four days ago: > The inability to unload a plugin defeats the use I would have for this, unfortunately. A long running service with plugins may want to load new, updated versions of an existing plugin. Not being able to unload the old version, ends up creating stale references which will keep piling up for the lifetime of the process. As I understand it, this is essentially a memory leak. [...] > https://www.reddit.com/r/golang/comments/53adu8/a/d7rmmco https://www.reddit.com/r/golang/comments/53adu8/a/d7rmmco There is also this project by Hashicorp: https://github.com/hashicorp/go-plugin https://github.com/hashicorp/go-plugin
zupa-hu 10y ago
Amazing, so this is essentially a high performance alternative to Go scripting. One can already include the go build toolchain in a binary, right? Having JIT is only small steps away. Disagree with the problem of unloading plugins. From the sizes of Go binaries and memory sizes, one could easily load 10k+ plugins before the process needed a restart. Would be nice to have GC, but having plugins is such an increased benefit. To me this sounds like, Git is crap because you can't unload stored objects.

107 points by pythonist 10y ago

orta 10y ago

This is awesome, a plugin structure like this is one of the fundamentals of macOS programming (they're called NSBundles in those cases.)

junke 10y ago

> A plugin is only initialized once, and cannot be closed. Why not?

IshKebab 10y ago
I guess it's because if you load a symbol there's no clean way to invalidate it, especially in a thread-safe way. Something like that anyway.
- jerf 10y ago
  I think it's safe to say there is just plain no way to invalidate it. Even if it were hacked together, it would be unusably dangerous for even trivial cases. Module loading is something that can be added to a language later, but I suspect safe module unloading is something that has to go in from day one to ever work right. Even in languages that support it (Erlang) it's got a tricky list of caveats to keep track of that strike me as fairly essential to the problem, rather than accidental (in the Fred Brooks sense of the terms).
pjmlp 10y ago
Because unloading, while possible is a pain to get right. Hence why Java and .NET restrict it to Class Loaders/AppDomains.
- Someone 10y ago
  For those who wonder why: some of the things you have to make sure of are if you want to implement this correctly: - no thread is running code in the plugin. - no thread has a return address on the stack somewhere in the plugin. - no variable stores a pointer pointing into the plugin (This includes objects whose destructor is implemented in the plugin) - no signal handler points into the plugin. - no kernel thread is reading the plug-ins memory. In addition there are race conditions to take care of (e.g. thread T unloads a plugin that thread U is loading) With a 64 bit address space, I guess you can get around most if not all problems by mapping the memory that was mapped to the plugin as "can't read, can't write, can't execute", so that you crash hard if somebody tries to access it, but if you do that, you can just as well keep the code around. Go is garbage-collected, so it likely already has a lot of the logic to do this, but even then, combining the functionality to get a robust implementation of plugin unloading is a lot of work.

gtrubetskoy 10y ago

Curious what it would take (if at all possible?) to load such a .so from C. Because that would open the door for writing Go libs for all your favorite interpreted languages, web servers, etc.

fjl 10y ago
You can already create shared libraries that expose a C API using -buildmode=shared.
deleted 10y ago
[deleted]
dvirsky 10y ago
That's already possible, I've written an API that encapsulates Go libraries as Redis modules.
- Goopplesoft 10y ago
  Link?
  - dvirsky 10y ago
    It's a WIP and I haven't had the time to complete it, but here it is. You can load modules, handle redis commands and return responses, but you can't access redis itself yet. https://github.com/RedisLabs/rgm/blob/master/example/module.go https://github.com/RedisLabs/rgm/blob/master/example/module....
robbles 10y ago
here's a pretty cool example: https://blog.heroku.com/see_python_see_python_go_go_python_go https://blog.heroku.com/see_python_see_python_go_go_python_g...

speps 10y ago

I see an API fail coming. It seems plugin.Load takes the filename WITH extension where it shouldn't. Okay, right now it seems it's only supported on Linux but it ends up in Windows, you'd have to patch code to remove the .so. Mono handles it pretty well, the extension is optional.

regecks 10y ago
Not necessarily. https://tip.golang.org/src/plugin/plugin_dlopen.go https://tip.golang.org/src/plugin/plugin_dlopen.go " 5 // +build linux,cgo" The Windows version isn't there yet - it could encapsulate that logic.
IshKebab 10y ago
I don't think so. I think it is better that the file naming is handled in the application. It is less restrictive, e.g. you can use custom extensions this way. Consider another instance where file names are automatically mangled: GCC linking with libraries with the switch -lfoo. That has honestly caused me no end of problems. It would have been better if they required an explicit -llibfoo.so from the start. Go already has several easy methods for OS-specific code if you want.
- dvirsky 10y ago
  Not sure it should be be part of this package, but it's trivial to write a function that exposes something like "SharedLibraryFile(libname string) string". Depending on the OS it just formats the default shared object name. For extra runtime speed you could build OS specific versions of that function using build flags, so for each target OS it has one implementation and you have no switch/case or anything in runtime.
dsl 10y ago
Assuming or magically guessing file extensions leads to all sorts of confusion attacks that could result in arbitrary code execution. I would go so far as to say an absolute path should be required.

SoapSeller 10y ago

Didn't one of the main selling points of Go is one big static executable with no external dependencies? What's next? Generics?

dvirsky 10y ago
Although this can be abused, the use case is not external dependencies IMO, but rather extending systems. Think of a web server etc. Each plugin is still a static binary that is self contained, but you can load capabilities into the server in runtime. This is not like dynamic linking hell you can get yourself into when writing C.
- clappski 10y ago
  But isn't that exactly how dynamic linking works? By loading a DLL at runtime? The only issues I've ever had with dynamic linking are at _runtime_, where XXX.dll can't be found.
  - dvirsky 10y ago
    yes, my above comment wasn't phrased that well. but do distinguish between a plugin system where you load so's or DLLs to extend an app, and an application that has dynamic linking against them, where the linker handles this. So basically something like apache modules - either you can load the module or not - vs something like "cannot find symbol foo in foo.so.0.1."
- wtbob 10y ago
  > Each plugin is still a static binary that is self contained, but you can load capabilities into the server in runtime. In my Go code, I've just used subprocesses for that. Stdin & stdout, plus a simple serialisation format, and I'm done. Not suitable for high performance stuff, of course, but good enough for my needs.
  - dvirsky 10y ago
    yeah, it could work for sure if latency is not an issue. It's also possible to do a go->C->go bridge to load Go plugins dynamically.

pjmlp 10y ago

Nice to see this coming, one step closer to have Oberon System 3 with Gadgets rewritten in Go. :)

guessmyname 10y ago

I agree with this comment from Reddit four days ago: > The inability to unload a plugin defeats the use I would have for this, unfortunately. A long running service with plugins may want to load new, updated versions of an existing plugin. Not being able to unload the old version, ends up creating stale references which will keep piling up for the lifetime of the process. As I understand it, this is essentially a memory leak. [...] > https://www.reddit.com/r/golang/comments/53adu8/a/d7rmmco https://www.reddit.com/r/golang/comments/53adu8/a/d7rmmco There is also this project by Hashicorp: https://github.com/hashicorp/go-plugin https://github.com/hashicorp/go-plugin

zupa-hu 10y ago

Amazing, so this is essentially a high performance alternative to Go scripting. One can already include the go build toolchain in a binary, right? Having JIT is only small steps away. Disagree with the problem of unloading plugins. From the sizes of Go binaries and memory sizes, one could easily load 10k+ plugins before the process needed a restart. Would be nice to have GC, but having plugins is such an increased benefit. To me this sounds like, Git is crap because you can't unload stored objects.