6 ms·

Go Execution Modes

111 points by Rooki 12y ago

Rapzid 12y ago
I think the more relevant news in there for most is the support for dynamic/runtime loading. The title here makes it sound like it's just for dynamic linking but I read the article to article to be sure and was pleasantly surprised :)
- Rooki 12y ago
  Good point, I updated the title to include the actual title of the doc but left the rest to avoid confusion. On topic: I'm really excited to see this, even though it's made clear this is just a plan and not a promise it's still great news for things like Android development.
axaxs 12y ago
I would love to see the gc compiler support building dynamically linked executables against a libgo runtime Ala gccgo. Unless I'm misreading, this case doesn't seem covered. I love the standalone binary deployment ease as of now, but on a machine with tens of go applications, the binary size redundancy gets old. Or did I miss something?
- FooBarWidget 12y ago
  We're in an age where everybody is wasting gigabytes of memory through virtualization, and nobody cares. Would those few megabytes of memory saved by dynamically linking against libgo really be worth it? For past 5 years or so, it seems people value ease of deployment much more than efficiency.
  - davidw 12y ago
    The people who value their memory usage just aren't using that kind of system, so there's some selection bias there.
  - ColinCera 12y ago
    You have a point, but on mobile and embedded devices, nobody is "wasting gigabytes of memory through virtualization" and not caring.
    - laumars 12y ago
      It doesn't cost gigabytes of RAM, but Android applications do run on a Java virtual machine. But I'm being massively pedantic.
    - peterbotond 12y ago
      yes. +1
- mrweasel 12y ago
  How large are the binaries you deploy? Unless it's hundreds of MB per binary I does see the dynamically linked executables being worth the trouble. Of cause options are nice, I just worry that in the end I don't get to choose.
  - ColinCera 12y ago
    I think one of the motivations is that they want Go to be a language suitable for Android development and other use cases. In mobile and embedded devices, binary size still matters.
- ianlancetaylor 12y ago
  I believe that use case is covered via -buildmode=shared and -linkshared. Just run -buildmode=shared when building the standard library.
frik 12y ago
The doc doesn't mention a timeline or Go version when we may see the new planned features.
- enneff 12y ago
  That's unknown. Work will likely begin in December after Go 1.4 is released, so the earliest we'll see this work is in Go 1.5.
JuiceSSH 12y ago
Interesting that it will allow compilation of PIE executables. This is a mandatory requirement for any binaries run on newer versions of Android (L+).
- dmm 12y ago
  Is it for security reasons? I know that OpenBSD uses PIEs so that the runtime can randomized address space locations.
pauldix 12y ago
This would be awesome to have. +1! With InfluxDB we want to give users the ability to define custom functions and it looks like this could be used to let people do that in Go or any language really. Whereas right now we're limited to doing something like using LuaJIT.
personZ 12y ago
The dogma about preventing the sharing of pointers with C is a bit concerning. I've made use of that mechanism, to fantastic effect, albeit understanding the risks and consequences. The notion that it must be prevented because bad things can happen if you aren't careful isn't a starter, as there are essentially zero people doing that who've had problems. Instead a lot of people have enjoyed fantastic productivity because of the similarities of the implementations, and the synergy that allows. But the team wants move to a compacting GC. Fine, add pin/unpin idioms. This ground has been well covered by other languages. Don't completely destroy a very productive mechanism because in some oddball cases it might not work.
- pjmlp 12y ago
  I agree with the decision to forbid pointer sharing with C code. This is the type of issues that nicely lead to security bugs, in the hands of the typical average enterprise developer. Having a pin/unpin idiom might be good enough, though.
  - personZ 12y ago
    This is the type of issues that nicely lead to security bugs It's also the type of functionality that enables the efficient use of high performance C libraries and existing code (I use it to great effect with the Intel compiler for extremely high performance SIMD code). The "someone might not use it right, therefore it's anathema" notion is misguided, and is orthogonal with the philosophy of Go. Even C# and Java -- the pinnacle of "enterprise" languages -- understand and respect the notion of pointer sharing, and early on built in mechanisms to support it. Go is a league above, though, given that it follows C struct layout rules. But I see where the person proposing banning it got their notion from -- it's a classic hubris of "if it isn't important to me, it isn't important to anyone". A compacting GC has benefits, but the lazy notion of wholesale blocking broad and powerful uses because one doesn't personally benefit from them is how languages die.
    - pjmlp 12y ago
      Java and .NET offer GC free allocation for such purposes, via their interop APIs. This is also one area that is being improved in Java 9 (Arrays 2.0, Value Types and the new FFI). But as I mentioned, thus agreeing partially with you, pin/upin could be enough.
easytiger 12y ago
Is there a mirror? For those of us in the corporate wasteland this is not accesible
lobster_johnson 12y ago
> Building Go packages as a shared library Yes, please! We have a few potential Go projects which are blocked by the inability to separate Go code into plugins. I have toyed with the idea of forking child processes and implementing plugins via pipe I/O, but that's a very unfriendly API interface, and I don't feel very inspired. (Go isn't that great at POSIX stuff like forking, either.)
- _stephan 12y ago
  Why do the plugins have to be dynamically loaded? Would a source-based plugin/package system be an alternative?
  - lobster_johnson 12y ago
    Consider a data processing app. It calls little data-processing modules written in Go. You want to open-source it because it's general-purpose and reusable; but you don't want to open-source all your in-house, proprietary data-processing components, too; they're not of use to anyone but yourself, or they may be something you can't open-source for other reasons. That means the core project cannot list its plugins as dependencies. You will instead want to be able to load plugins from a configuration file. (The problem is actually equally awkward in languages that have package systems, such as Node and Ruby. In Ruby, for example, a program can only see gems declared in the Gemfile; same problem with npm-shrinkwrap.json.) I suppose you could set up a special build system where plugins are expected to be installed into a certain folder so that they're statically built in, with some kind of hook so that they're registered into the core. That's the Nginx approach. I'm not a big fan, especially since this method of building things need to be reinvented for each app that needs a plugin system. You could, of course, invert the relationship and make this system a library: To set it up with your plugins, you write a small program that runs the system's main entrypoint with a list of plugins. But that just makes it harder to configure, build, use and deploy.
bkeroack 12y ago
I realize it's just an early draft, but it's interesting (alarming?) that the proposed plugin API has no facilities for plugin discovery or separate namespaces. Nor any apparent ability for the plugin consumer to introspect into plugins. I suppose the namespace issue can be mitigated by a strict naming convention (something like "[application_name].[plugin_name]"), but the others are kind of a bummer. I don't have a whole lot of positive things to say about the Python setuptools API (perhaps better called "lack-of-API internal interface"), but at least it's possible to dynamically load python plugins from arbitrary locations in the filesystem and programmatically discover their contents.
- nteon 12y ago
  I'm not sure I see the benefit of plugin discovery or separate namespaces. Loading a plugin gives you a Plugin struct, which is a namespace to lookup symbols (functions + globals) from that plugin. Most plugins would I think by necessity be application specific, so I'm not sure how much gain there in in some sort of plugin discovery. But the process of loading a plugin would also allow the plugin to call into your code to register handlers & data structures in the main application through init() functions. From a security perspective, executing arbitrary code in your address space in order to discover the contents of a plugin sounds ill-advised. To me, it looks like they're building the low level building blocks that allow a number of different higher level APIs to be built, which is pretty neat.
- ianlancetaylor 12y ago
  The plugin API uses file names, so discovery would be at a higher level. Go packages impose their own namespace, so there shouldn't be any namespace issues. Introspection is useful, but would be a matter of examining the files using existing packages like debug/elf; it would not be part of the plugin API. Basically the plugin API is isomorphic to dlopen/dlclose. Everything else is built on top of that.
  - bkeroack 12y ago
    OK, if true that makes sense. I was interpreting the Name parameter to Open() as being something like a global plugin name rather than a filesystem location. That combined with the Plugin struct (which hopefully would have something like an iterable Map of exported symbols) resolves the issues I was concerned about. Nice.

111 points by Rooki 12y ago

I think the more relevant news in there for most is the support for dynamic/runtime loading. The title here makes it sound like it's just for dynamic linking but I read the article to article to be sure and was pleasantly surprised :)

Rooki 12y ago
Good point, I updated the title to include the actual title of the doc but left the rest to avoid confusion. On topic: I'm really excited to see this, even though it's made clear this is just a plan and not a promise it's still great news for things like Android development.

axaxs 12y ago

I would love to see the gc compiler support building dynamically linked executables against a libgo runtime Ala gccgo. Unless I'm misreading, this case doesn't seem covered. I love the standalone binary deployment ease as of now, but on a machine with tens of go applications, the binary size redundancy gets old. Or did I miss something?

FooBarWidget 12y ago
We're in an age where everybody is wasting gigabytes of memory through virtualization, and nobody cares. Would those few megabytes of memory saved by dynamically linking against libgo really be worth it? For past 5 years or so, it seems people value ease of deployment much more than efficiency.
- davidw 12y ago
  The people who value their memory usage just aren't using that kind of system, so there's some selection bias there.
- ColinCera 12y ago
  You have a point, but on mobile and embedded devices, nobody is "wasting gigabytes of memory through virtualization" and not caring.
  - laumars 12y ago
    It doesn't cost gigabytes of RAM, but Android applications do run on a Java virtual machine. But I'm being massively pedantic.
  - peterbotond 12y ago
    yes. +1
mrweasel 12y ago
How large are the binaries you deploy? Unless it's hundreds of MB per binary I does see the dynamically linked executables being worth the trouble. Of cause options are nice, I just worry that in the end I don't get to choose.
- ColinCera 12y ago
  I think one of the motivations is that they want Go to be a language suitable for Android development and other use cases. In mobile and embedded devices, binary size still matters.
ianlancetaylor 12y ago
I believe that use case is covered via -buildmode=shared and -linkshared. Just run -buildmode=shared when building the standard library.

frik 12y ago

The doc doesn't mention a timeline or Go version when we may see the new planned features.

enneff 12y ago
That's unknown. Work will likely begin in December after Go 1.4 is released, so the earliest we'll see this work is in Go 1.5.

JuiceSSH 12y ago

Interesting that it will allow compilation of PIE executables. This is a mandatory requirement for any binaries run on newer versions of Android (L+).

dmm 12y ago
Is it for security reasons? I know that OpenBSD uses PIEs so that the runtime can randomized address space locations.

pauldix 12y ago

This would be awesome to have. +1! With InfluxDB we want to give users the ability to define custom functions and it looks like this could be used to let people do that in Go or any language really. Whereas right now we're limited to doing something like using LuaJIT.

personZ 12y ago

The dogma about preventing the sharing of pointers with C is a bit concerning. I've made use of that mechanism, to fantastic effect, albeit understanding the risks and consequences. The notion that it must be prevented because bad things can happen if you aren't careful isn't a starter, as there are essentially zero people doing that who've had problems. Instead a lot of people have enjoyed fantastic productivity because of the similarities of the implementations, and the synergy that allows. But the team wants move to a compacting GC. Fine, add pin/unpin idioms. This ground has been well covered by other languages. Don't completely destroy a very productive mechanism because in some oddball cases it might not work.

pjmlp 12y ago
I agree with the decision to forbid pointer sharing with C code. This is the type of issues that nicely lead to security bugs, in the hands of the typical average enterprise developer. Having a pin/unpin idiom might be good enough, though.
- personZ 12y ago
  This is the type of issues that nicely lead to security bugs It's also the type of functionality that enables the efficient use of high performance C libraries and existing code (I use it to great effect with the Intel compiler for extremely high performance SIMD code). The "someone might not use it right, therefore it's anathema" notion is misguided, and is orthogonal with the philosophy of Go. Even C# and Java -- the pinnacle of "enterprise" languages -- understand and respect the notion of pointer sharing, and early on built in mechanisms to support it. Go is a league above, though, given that it follows C struct layout rules. But I see where the person proposing banning it got their notion from -- it's a classic hubris of "if it isn't important to me, it isn't important to anyone". A compacting GC has benefits, but the lazy notion of wholesale blocking broad and powerful uses because one doesn't personally benefit from them is how languages die.
  - pjmlp 12y ago
    Java and .NET offer GC free allocation for such purposes, via their interop APIs. This is also one area that is being improved in Java 9 (Arrays 2.0, Value Types and the new FFI). But as I mentioned, thus agreeing partially with you, pin/upin could be enough.

easytiger 12y ago

Is there a mirror? For those of us in the corporate wasteland this is not accesible

lobster_johnson 12y ago

> Building Go packages as a shared library Yes, please! We have a few potential Go projects which are blocked by the inability to separate Go code into plugins. I have toyed with the idea of forking child processes and implementing plugins via pipe I/O, but that's a very unfriendly API interface, and I don't feel very inspired. (Go isn't that great at POSIX stuff like forking, either.)

_stephan 12y ago
Why do the plugins have to be dynamically loaded? Would a source-based plugin/package system be an alternative?
- lobster_johnson 12y ago
  Consider a data processing app. It calls little data-processing modules written in Go. You want to open-source it because it's general-purpose and reusable; but you don't want to open-source all your in-house, proprietary data-processing components, too; they're not of use to anyone but yourself, or they may be something you can't open-source for other reasons. That means the core project cannot list its plugins as dependencies. You will instead want to be able to load plugins from a configuration file. (The problem is actually equally awkward in languages that have package systems, such as Node and Ruby. In Ruby, for example, a program can only see gems declared in the Gemfile; same problem with npm-shrinkwrap.json.) I suppose you could set up a special build system where plugins are expected to be installed into a certain folder so that they're statically built in, with some kind of hook so that they're registered into the core. That's the Nginx approach. I'm not a big fan, especially since this method of building things need to be reinvented for each app that needs a plugin system. You could, of course, invert the relationship and make this system a library: To set it up with your plugins, you write a small program that runs the system's main entrypoint with a list of plugins. But that just makes it harder to configure, build, use and deploy.

bkeroack 12y ago

I realize it's just an early draft, but it's interesting (alarming?) that the proposed plugin API has no facilities for plugin discovery or separate namespaces. Nor any apparent ability for the plugin consumer to introspect into plugins. I suppose the namespace issue can be mitigated by a strict naming convention (something like "[application_name].[plugin_name]"), but the others are kind of a bummer. I don't have a whole lot of positive things to say about the Python setuptools API (perhaps better called "lack-of-API internal interface"), but at least it's possible to dynamically load python plugins from arbitrary locations in the filesystem and programmatically discover their contents.

nteon 12y ago
I'm not sure I see the benefit of plugin discovery or separate namespaces. Loading a plugin gives you a Plugin struct, which is a namespace to lookup symbols (functions + globals) from that plugin. Most plugins would I think by necessity be application specific, so I'm not sure how much gain there in in some sort of plugin discovery. But the process of loading a plugin would also allow the plugin to call into your code to register handlers & data structures in the main application through init() functions. From a security perspective, executing arbitrary code in your address space in order to discover the contents of a plugin sounds ill-advised. To me, it looks like they're building the low level building blocks that allow a number of different higher level APIs to be built, which is pretty neat.
ianlancetaylor 12y ago
The plugin API uses file names, so discovery would be at a higher level. Go packages impose their own namespace, so there shouldn't be any namespace issues. Introspection is useful, but would be a matter of examining the files using existing packages like debug/elf; it would not be part of the plugin API. Basically the plugin API is isomorphic to dlopen/dlclose. Everything else is built on top of that.
- bkeroack 12y ago
  OK, if true that makes sense. I was interpreting the Name parameter to Open() as being something like a global plugin name rather than a filesystem location. That combined with the Plugin struct (which hopefully would have something like an iterable Map of exported symbols) resolves the issues I was concerned about. Nice.