6 ms·

The dogma about preventing the sharing of pointers with C is a bit concerning. I've made use of that mechanism, to fantastic effect, albeit understanding the ri

by personZ 12y ago

The dogma about preventing the sharing of pointers with C is a bit concerning. I've made use of that mechanism, to fantastic effect, albeit understanding the risks and consequences. The notion that it must be prevented because bad things can happen if you aren't careful isn't a starter, as there are essentially zero people doing that who've had problems. Instead a lot of people have enjoyed fantastic productivity because of the similarities of the implementations, and the synergy that allows. But the team wants move to a compacting GC. Fine, add pin/unpin idioms. This ground has been well covered by other languages. Don't completely destroy a very productive mechanism because in some oddball cases it might not work.

pjmlp 12y ago
I agree with the decision to forbid pointer sharing with C code. This is the type of issues that nicely lead to security bugs, in the hands of the typical average enterprise developer. Having a pin/unpin idiom might be good enough, though.
- personZ 12y ago
  This is the type of issues that nicely lead to security bugs It's also the type of functionality that enables the efficient use of high performance C libraries and existing code (I use it to great effect with the Intel compiler for extremely high performance SIMD code). The "someone might not use it right, therefore it's anathema" notion is misguided, and is orthogonal with the philosophy of Go. Even C# and Java -- the pinnacle of "enterprise" languages -- understand and respect the notion of pointer sharing, and early on built in mechanisms to support it. Go is a league above, though, given that it follows C struct layout rules. But I see where the person proposing banning it got their notion from -- it's a classic hubris of "if it isn't important to me, it isn't important to anyone". A compacting GC has benefits, but the lazy notion of wholesale blocking broad and powerful uses because one doesn't personally benefit from them is how languages die.
  - pjmlp 12y ago
    Java and .NET offer GC free allocation for such purposes, via their interop APIs. This is also one area that is being improved in Java 9 (Arrays 2.0, Value Types and the new FFI). But as I mentioned, thus agreeing partially with you, pin/upin could be enough.
    - voidlogic 12y ago
      Although I would perfer pin/unpin, You could easily implmement GC free allocation in Go and people do. Its not hard to write a library for explicit memory management: 1. mmap some memory 2. Write or port malloc and free like functions This implementation could probably even use sync.Pool to good effect.
ominous_prime 12y ago
The current state of the CGO interface is that you can't safely pass Go pointers to C code. That's all this document is repeating. You may have used this successfully in the past, I know I have, and it works with the GC toolchain, but it is still technically incorrect to do so. I think there's another design doc being worked on that will document the situation, and possibly have proposals to make sharing memory between Go and C more convenient. There's are many projects that make heavy use of this, where copying is slow or infeasible, and malloc is inconvenient.
- personZ 12y ago
  The current state of the CGO interface is that you can't safely pass Go pointers to C code. That's all this document is repeating. Where is this stated? Where is it said that it's technically incorrect? Just to be clear, I'm not passing it and then running a thread in C that just runs with the pointer to an object that no longer exists in Go. Instead I'm calling a C function synchronously -- there is no possible way the element is going to be collected during that run. It could be compacted (though Go up until now has explicitly had a non-compacting GC, so not a current risk), of course, which is what brought up this issue, but that's the reason such interop languages have pinning (flagging that something shouldn't be moved). It wasn't by chance -- it was by design, and it was purposefully understanding the lifetime of objects.
  - ominous_prime 12y ago
    You're right, I can't find where it's explicitly forbidden in the docs, but it also isn't stated that you can pass pointers to C code either. There has been more recent discussion here (https://code.google.com/p/go/issues/detail?id=8310 https://code.google.com/p/go/issues/detail?id=8310) and on the mailing lists too: > +rsc > Passing a Go pointer like that to C is problematic because eventually > we will want the garbage collector to be able to move things and later on with a response from Dmitry: >> In my opinion, the current behavior should be covered by the >> compatibility guarantee. In Go 1, passing pointers to Go-allocated >> memory into C was not a problem. > > This was never fully working. Only some individual cases were working. Even though your C thread is being run synchronously, the GC is running concurrently in another thread (concurrent GC was introduced in go1.3), and has the ability to alter the memory that was passed to your C function.
crawshaw 12y ago
You can do your own pinning, which is what I do in gobind[1]. The basic idea is to keep a map in Go of the pointers: var ptrs = make(map[uintptr]unsafe.Pointer) When passing a pointer from Go to C: var p unsafe.Pointer = ... id := uintptr(p) ptrs[id] = p C.Fn(id) When C returns the pointer to Go to be used, run it through the ptrs map again //export GoFn func GoFn(id uintptr) { p := ptrs[id] // ... use p } Don't forget a cleanup function where you delete(ptrs, id). If you want to pass the same pointer multiple times to C and keep it comparable, you'll need a second map. All of this is requires being very careful, but the hard part is the notion of holding references to memory outside the realm of the GC. I don't believe a runtime-assisted pinning API can do any better than what you can do yourself with a map. [1] https://godoc.org/code.google.com/p/go.mobile/cmd/gobind https://godoc.org/code.google.com/p/go.mobile/cmd/gobind
- ominous_prime 12y ago
  I don't think this would really handle the pinning aspect if a compacting GC is implemented, would it? It does keep the pointer alive, but it technically doesn't do anything to prevent the GC from changing the pointer.
  - crawshaw 12y ago
    The uintptr version of the pointer is not moved by the GC, so it acts as a stable identifier.
- pjmlp 12y ago
  This does not work if the GC updates the pointer concurrently.
  - crawshaw 12y ago
    The GC will not update a pointer once it is converted to the type uintptr. It becomes invisible to the precise collector, so the uintptr acts as an id value.

by personZ 12y ago

pjmlp 12y ago

I agree with the decision to forbid pointer sharing with C code. This is the type of issues that nicely lead to security bugs, in the hands of the typical average enterprise developer. Having a pin/unpin idiom might be good enough, though.

personZ 12y ago
This is the type of issues that nicely lead to security bugs It's also the type of functionality that enables the efficient use of high performance C libraries and existing code (I use it to great effect with the Intel compiler for extremely high performance SIMD code). The "someone might not use it right, therefore it's anathema" notion is misguided, and is orthogonal with the philosophy of Go. Even C# and Java -- the pinnacle of "enterprise" languages -- understand and respect the notion of pointer sharing, and early on built in mechanisms to support it. Go is a league above, though, given that it follows C struct layout rules. But I see where the person proposing banning it got their notion from -- it's a classic hubris of "if it isn't important to me, it isn't important to anyone". A compacting GC has benefits, but the lazy notion of wholesale blocking broad and powerful uses because one doesn't personally benefit from them is how languages die.
- pjmlp 12y ago
  Java and .NET offer GC free allocation for such purposes, via their interop APIs. This is also one area that is being improved in Java 9 (Arrays 2.0, Value Types and the new FFI). But as I mentioned, thus agreeing partially with you, pin/upin could be enough.
  - voidlogic 12y ago
    Although I would perfer pin/unpin, You could easily implmement GC free allocation in Go and people do. Its not hard to write a library for explicit memory management: 1. mmap some memory 2. Write or port malloc and free like functions This implementation could probably even use sync.Pool to good effect.

ominous_prime 12y ago

The current state of the CGO interface is that you can't safely pass Go pointers to C code. That's all this document is repeating. You may have used this successfully in the past, I know I have, and it works with the GC toolchain, but it is still technically incorrect to do so. I think there's another design doc being worked on that will document the situation, and possibly have proposals to make sharing memory between Go and C more convenient. There's are many projects that make heavy use of this, where copying is slow or infeasible, and malloc is inconvenient.

personZ 12y ago
The current state of the CGO interface is that you can't safely pass Go pointers to C code. That's all this document is repeating. Where is this stated? Where is it said that it's technically incorrect? Just to be clear, I'm not passing it and then running a thread in C that just runs with the pointer to an object that no longer exists in Go. Instead I'm calling a C function synchronously -- there is no possible way the element is going to be collected during that run. It could be compacted (though Go up until now has explicitly had a non-compacting GC, so not a current risk), of course, which is what brought up this issue, but that's the reason such interop languages have pinning (flagging that something shouldn't be moved). It wasn't by chance -- it was by design, and it was purposefully understanding the lifetime of objects.
- ominous_prime 12y ago
  You're right, I can't find where it's explicitly forbidden in the docs, but it also isn't stated that you can pass pointers to C code either. There has been more recent discussion here (https://code.google.com/p/go/issues/detail?id=8310 https://code.google.com/p/go/issues/detail?id=8310) and on the mailing lists too: > +rsc > Passing a Go pointer like that to C is problematic because eventually > we will want the garbage collector to be able to move things and later on with a response from Dmitry: >> In my opinion, the current behavior should be covered by the >> compatibility guarantee. In Go 1, passing pointers to Go-allocated >> memory into C was not a problem. > > This was never fully working. Only some individual cases were working. Even though your C thread is being run synchronously, the GC is running concurrently in another thread (concurrent GC was introduced in go1.3), and has the ability to alter the memory that was passed to your C function.

crawshaw 12y ago

You can do your own pinning, which is what I do in gobind[1]. The basic idea is to keep a map in Go of the pointers: var ptrs = make(map[uintptr]unsafe.Pointer) When passing a pointer from Go to C: var p unsafe.Pointer = ... id := uintptr(p) ptrs[id] = p C.Fn(id) When C returns the pointer to Go to be used, run it through the ptrs map again //export GoFn func GoFn(id uintptr) { p := ptrs[id] // ... use p } Don't forget a cleanup function where you delete(ptrs, id). If you want to pass the same pointer multiple times to C and keep it comparable, you'll need a second map. All of this is requires being very careful, but the hard part is the notion of holding references to memory outside the realm of the GC. I don't believe a runtime-assisted pinning API can do any better than what you can do yourself with a map. [1] https://godoc.org/code.google.com/p/go.mobile/cmd/gobind https://godoc.org/code.google.com/p/go.mobile/cmd/gobind

ominous_prime 12y ago
I don't think this would really handle the pinning aspect if a compacting GC is implemented, would it? It does keep the pointer alive, but it technically doesn't do anything to prevent the GC from changing the pointer.
- crawshaw 12y ago
  The uintptr version of the pointer is not moved by the GC, so it acts as a stable identifier.
pjmlp 12y ago
This does not work if the GC updates the pointer concurrently.
- crawshaw 12y ago
  The GC will not update a pointer once it is converted to the type uintptr. It becomes invisible to the precise collector, so the uintptr acts as an id value.