4 ms·

Any slice implementation that references to the original data has to be some kind of "fat pointer". Is there any other way that can be done? String.substring i

Any slice implementation that references to the original data has to be some kind of "fat pointer". Is there any other way that can be done? String.substring in Java does exactly that. I'm sure numpy arrays in Python are implemented like that, perhaps slightly more fatter.

chewxy 7y ago
Not numpy, but Gorgonia tensors are just pointers. The reason: it's cheaper to pass around one word than the absolute minimum "fatness" to pass around a multidimensional array. The absolute minimum meta info required for a multidim array is shape, stride. Numpy implements them as a [HUGENUMBER] array of integers for both. Gorgonia implements them as slices. Both of these incur a lot of additional movement of memory. More here: https://youtu.be/fd4EPh2tYrk?t=654 https://youtu.be/fd4EPh2tYrk?t=654
chrisseaton 7y ago
> String.substring in Java does exactly that. It used to - it doesn’t these days, where it now copies.
masklinn 7y ago
> Any slice implementation that references to the original data has to be some kind of "fat pointer". Is there any other way that can be done? Yes, add one more level of indirection. For instance traditional "oo" languages don't usually use fat pointers for dynamic dispatch, you have a single pointer to an instance which holds a pointer to its vtable. In Rust however, the "object pointer" is a fat pointer of (vtable, instance). Go's interface pointers are the same (which combined with nils leads to the dreaded "typed nil" issue).
- weberc2 7y ago
  I don’t understand people’s beef with “typed nil”. If an interface is just a fat pointer and the data is itself a pointer, then it stands to reason that the interface can be not nil while the data can be nil. If you understand the concept of pointers (or nillable pointers, anyway), surely this should not be surprising to anyone? It would have the same states as a asterisk-asterisk-int (HN formatting prohibits asterisk literals), would it not?
  - jerf 7y ago
    I think it's a combination of several things: First, you've got a good chunk of Go programmers who haven't dug down into the details of the internal representation of interfaces, because for the most part, they don't need to, and they either don't generally dig in to that level, or just haven't gotten around to it yet. (There's nothing wrong with this and this isn't a criticism. It's just that not everybody is an expert in everything they do or use.) Secondly, I think that for a lot of people, when they have an interface that is itself nil, they mentally tag it with the label "nil", and when they have an interface that contains a typed value that is nil, they mentally tag this with "nil", and from the there the confusion is obvious. Mix in some concepts from other languages like C where nil instances are always invalid, so that you don't mentally have a model for a "legitimate instance of a type whose pointer is nil" [1], and it just gets worse. Third, I think it's an action-at-a-distance problem. I'd say the point at which you have, say, an "io.Reader" value, and you put an invalid nil struct pointer in there that doesn't work, the problem is there. Values that are invalid in that way should never be created at all, so "interfaceVal == nil" should be all the check that is necessary. So you have a problem where the code that is blowing up trying to use the "invalid interface" is actually caused by an arbitrary-distant bit of code that created the value that shouldn't have been created in the first place, and that pattern generally causes problems with proper attribution of "fault" in code; the temptation to blame the place that crashed is very strong, and, I mean, that's generally a perfectly sensible default presumption so it's not like that's a crazy idea or anything. [1]: For those who don't know, in Go, you can have a legitimate value of a pointer which is nil, and implements methods, because the "nil" still has a type the compiler/runtime tracks. I have a memory pool-type thing, for instance (different use case than sync.Pool, and predates it) which if it has an instance does its pooling thing, but if it is nil, falls back to just using make and letting the GC pick up the pieces.
    - inlined 7y ago
      This. Nil is the one case where you can seriously screw your self up in Go because it’s an untyped literal that gets typed on assignment. Interfaces are fat pointers and a “true” nil pointer is nil interface, nil implementation. If your function is going to return a known subtype and you store that nil in a temporary you now return a known interface and nil implementation. Then your == nil check on the interface pointer will fail. I royally screwed up a deploy once because my functions were returning nil error implementations that were != nil and triggered failure paths.
xxs 7y ago
>String.substring in Java does exactly that. Not since 2007 (java 7). The extra ints (8bytes) used for marking the beginning/end of the string and the fact the subscring leaked the ref. to the original string caused extra memory footprint. So now it's just a copy unless the substring is exactly the same as the original. Previously (around 2004) I used to call new String(string.substring) in specific cases to prevent leaks. >Is there any other way that can be done? <-- yes copies.

chewxy 7y ago

Not numpy, but Gorgonia tensors are just pointers. The reason: it's cheaper to pass around one word than the absolute minimum "fatness" to pass around a multidimensional array. The absolute minimum meta info required for a multidim array is shape, stride. Numpy implements them as a [HUGENUMBER] array of integers for both. Gorgonia implements them as slices. Both of these incur a lot of additional movement of memory. More here: https://youtu.be/fd4EPh2tYrk?t=654 https://youtu.be/fd4EPh2tYrk?t=654

chrisseaton 7y ago

> String.substring in Java does exactly that. It used to - it doesn’t these days, where it now copies.

masklinn 7y ago

> Any slice implementation that references to the original data has to be some kind of "fat pointer". Is there any other way that can be done? Yes, add one more level of indirection. For instance traditional "oo" languages don't usually use fat pointers for dynamic dispatch, you have a single pointer to an instance which holds a pointer to its vtable. In Rust however, the "object pointer" is a fat pointer of (vtable, instance). Go's interface pointers are the same (which combined with nils leads to the dreaded "typed nil" issue).

weberc2 7y ago
I don’t understand people’s beef with “typed nil”. If an interface is just a fat pointer and the data is itself a pointer, then it stands to reason that the interface can be not nil while the data can be nil. If you understand the concept of pointers (or nillable pointers, anyway), surely this should not be surprising to anyone? It would have the same states as a asterisk-asterisk-int (HN formatting prohibits asterisk literals), would it not?
- jerf 7y ago
  I think it's a combination of several things: First, you've got a good chunk of Go programmers who haven't dug down into the details of the internal representation of interfaces, because for the most part, they don't need to, and they either don't generally dig in to that level, or just haven't gotten around to it yet. (There's nothing wrong with this and this isn't a criticism. It's just that not everybody is an expert in everything they do or use.) Secondly, I think that for a lot of people, when they have an interface that is itself nil, they mentally tag it with the label "nil", and when they have an interface that contains a typed value that is nil, they mentally tag this with "nil", and from the there the confusion is obvious. Mix in some concepts from other languages like C where nil instances are always invalid, so that you don't mentally have a model for a "legitimate instance of a type whose pointer is nil" [1], and it just gets worse. Third, I think it's an action-at-a-distance problem. I'd say the point at which you have, say, an "io.Reader" value, and you put an invalid nil struct pointer in there that doesn't work, the problem is there. Values that are invalid in that way should never be created at all, so "interfaceVal == nil" should be all the check that is necessary. So you have a problem where the code that is blowing up trying to use the "invalid interface" is actually caused by an arbitrary-distant bit of code that created the value that shouldn't have been created in the first place, and that pattern generally causes problems with proper attribution of "fault" in code; the temptation to blame the place that crashed is very strong, and, I mean, that's generally a perfectly sensible default presumption so it's not like that's a crazy idea or anything. [1]: For those who don't know, in Go, you can have a legitimate value of a pointer which is nil, and implements methods, because the "nil" still has a type the compiler/runtime tracks. I have a memory pool-type thing, for instance (different use case than sync.Pool, and predates it) which if it has an instance does its pooling thing, but if it is nil, falls back to just using make and letting the GC pick up the pieces.
  - inlined 7y ago
    This. Nil is the one case where you can seriously screw your self up in Go because it’s an untyped literal that gets typed on assignment. Interfaces are fat pointers and a “true” nil pointer is nil interface, nil implementation. If your function is going to return a known subtype and you store that nil in a temporary you now return a known interface and nil implementation. Then your == nil check on the interface pointer will fail. I royally screwed up a deploy once because my functions were returning nil error implementations that were != nil and triggered failure paths.

xxs 7y ago

>String.substring in Java does exactly that. Not since 2007 (java 7). The extra ints (8bytes) used for marking the beginning/end of the string and the fact the subscring leaked the ref. to the original string caused extra memory footprint. So now it's just a copy unless the substring is exactly the same as the original. Previously (around 2004) I used to call new String(string.substring) in specific cases to prevent leaks. >Is there any other way that can be done? <-- yes copies.