16 ms·
a[low:high:max] in Golang – A Rare Slice Trick
- gus_massa 4y agoThis paragraph is scary. It looks like a good idea for the Golang version of the Underhanded C Contest. > This trick is useful for returning a slice from an immutable array; if you accidentally append to the supposedly immutable slice, a copy is forced and no data is overwritten because there is no more capacity left.
- gleenn 4y agoI don't know much Go but what does it mean to "accidentally" append to an immutable array? Are there just no particular guards for arrays and they are saying something which should be treated as immutable?
- UncleEntity 4y agoI know even less go but it sounds like it prevents accidentally writing past the end of the array by making a copy and appending to that.
- tedunangst 4y agoThere are no immutable arrays. If you have a slice of capacity ten with four elements, you can write a fifth element to it. If you reduce the capacity to four, then a new slice must be allocated to store five elements. The backing store for a slice is just a pointer. The rule is you don't give people pointers that you don't want them to write to.
- andrewstuart2 4y agoThat's just not true. There are immutable arrays in length. [4]int cannot be appended to. The backing of all slices are array types and array doubling is used for appends that go beyond the capacity. https://go.dev/ref/spec#Array_types https://go.dev/ref/spec#Array_types
- tedunangst 4y agoYeah, that was a mistake.
- yencabulator 4y agoNo array can change its length, the length is part of the type. There really are no "immutable array"s.
- Groxx 4y agoAppend mutates or copies-and-appends based on the capacity of the slice it's given. And since slicing an array or a slice gives you just a view, not a copy, it can cause "spooky action at a distance" and mutate things you didn't expect, especially if it was a slice with extra capacity of data used somewhere else. Which is what this article is describing. Plus the knowledge of the change in length (to see the appended data) is only visible with the returned slice (a new view with the larger length), but the underlying data is changed either way. It's not often a source of errors, but when it is it can be extremely hard to diagnose.
- DougBTX 4y agoSo in other words, appending to a slice may overwrite data in the middle of a different slice?
- Groxx 4y agoYep. Depending on how the slice was constructed, not how it is used. Slices (largely) behave like this in most languages, Go's contribution is mostly that slices and append are ubiquitous, so pretty much every Go coder is exposed to it. Prior to generics, literally any alternative was so much more work they essentially haven't been used. That may change in the future now that we do have (very simplistic) generics, but only time will tell.
- iainmerrick 4y agoSlices (largely) behave like this in most languages Is that really the case? I can’t think of many languages that let you construct a view into the middle of array, pass that view as an argument to a function call, and allow that function to add new values into your array via the view. In Python or JS, for example, the “slice” would just be a brand new array, right?
- Groxx 4y agoIf the slice syntax creates a copy, I would argue that it's simply syntactic sugar for array copying, not actually producing a slice (i.e. there is no slice type). But yes, `somefunc(ary[1:])` in Python produces a copy, not a reference to the underlying value. You could build a more "true" slice class, but the builtin stuff doesn't do that. JavaScript is similar. Java however has `List<>.sublist` which largely behaves like Go: https://docs.oracle.com/javase/8/docs/api/java/util/List.html#subList-int-int- https://docs.oracle.com/javase/8/docs/api/java/util/List.htm... . Sometimes these kinds of things are also referred to as "views". Go calls them slices though, and this is a Go article. edit: C# apparently has "spans": https://learn.microsoft.com/en-us/archive/msdn-magazine/2018/january/csharp-all-about-span-exploring-a-new-net-mainstay https://learn.microsoft.com/en-us/archive/msdn-magazine/2018...
- TwentyPosts 4y agoGo has no way to ensure "const correctness". If you pass around a slice (which you'll be doing a lot, since it's essentially Go's equivalent of a vector), everyone can modify the slice however they please. It's just a fat pointer. So yes, something which should be treated as immutable. The way Go works here easily leads to bugs, especially when concurrency is involved and slices get captured or used by goroutines. If you send a slice to a function, they essentially receive a pointer pointing at the same block of memory as in your calling function. They can modify it. If they append to the slice, however, and surpass the capacity of the slice they received, then their slice gets reallocated and doesn't point at the same block of memory anymore. In other words, if you receive a slice, append to it, and then change the first value of the slice, then this might modify a slice (or array) somewhere completely different, depending on whether your append surpasses the allocated capacity of the slice or not. Welcome to Golang.
- Groxx 4y agoBut also strings are actually immutable sometimes (at the very least when hardcoded), and they can be converted to a `[]byte` slice that looks mutable, but panics when modified. AFAIK no other slice-like data in Go behaves like this. Fun!
- wyager 4y agoThis is a pretty good vignette of how Go is designed. "Users don't need this feature (immutability, polymorphism, etc.), let's not include it. Ah crap, turns out we actually need it for a core language feature. Is there a lesson we can take away here? Nah, just make an ad-hoc implementation of the functionality in this one place."
- Groxx 4y ago"X for me but not for thee" is very much the vibe Go gives me, yeah. I mean, I have completely replaced my adhoc Python use with it and I'm much happier. But it's terrifying to use to build large systems.
- chrsig 4y agoHere's a succinct demonstration: https://go.dev/play/p/Li6_Rpe2R5L https://go.dev/play/p/Li6_Rpe2R5L Basically: a slice is a triple of {ptr to allocation, number of elements used, size of allocation} the slices will share the same underlying allocation. By specifying the third parameter in the slice function, you set the capacity equal to the number of elements in your slice. This forces the next append to reallocate and copy the contents into a new region. Without bounding the capacity when creating a new slice, the append operation could possibly continue using the same allocation, shared with the original slice. It could possibly resize and copy as well. It depends on how full the slice is, and is an implementation detail subject to change between versions.
- ErikCorry 4y agoHere another illustration of how slices sometimes alias each other, sometimes not. https://mobile.twitter.com/erikcorry/status/1561635841495236610 https://mobile.twitter.com/erikcorry/status/1561635841495236...
- secondraft 4y agoand yet another one (thanks to Erik): https://github.com/tlk/go-wat/blob/41cc21294a953dc7ad07a9b40a2486acf816b8e1/main.go#L27 https://github.com/tlk/go-wat/blob/41cc21294a953dc7ad07a9b40...
- Izkata 4y agoSounds like they're just badly describing copy-on-write. The original slice uses no extra memory because as long as you treat it as immutable it won't actually make a copy. As soon as you try to modify it, then the copy is made and the extra memory used.
- deleted 4y ago[deleted]
- mook 4y agoYes-ish; because go has no immutable slices, it's just copy-on-realloc. Which is if course obvious if realize that's what it's actually doing…
- masklinn 4y ago> Sounds like they're just badly describing copy-on-write. It’s not quite copy-on-write, and what it’s really describing is a workaround / safeguard. The problem is that by default Go slices will have as large a capacity as they can based on the parent, this is a problem if you return a slice to a still-in-use slice or array, and the caller decides to use that slice as a vector (either because the contract is not well documented or because they fucked up): appending to the “borrow” will happily go and stomp over the backing array, which may be holding in-use data of an other slice or the original array. By forcing the slice to have no extra capacity, if a caller tries to append it’ll force a “fork” by realloc-ing and avoid the issue.
- jjice 4y agoTangential, but if OP is the owner of the site, could you talk a little bit about your book writing process? - How you write - How you render the PDF - How you develop your plans That kind of thing. I find it super interesting to self-publish software books and I've been slowly writing one for about a year now. Really curious about this stuff in general and it looks like you've got a solid process down.
- iKlsR 4y agoFrom the author of Crafting Interpreters and Game Programming Patterns, some interesting stuff here about how he went about his two books which are excellent quality http://journal.stuffwithstuff.com/category/book/ http://journal.stuffwithstuff.com/category/book/
- deleted 4y ago[deleted]
- nutate 4y agoMy fave golang slice trick is the len of an empty slice is 0, but the slice itself is == to nil, but the len of nil won't compile. Can't understand that one. https://go.dev/play/p/MslCkBphl7q?v=gotip https://go.dev/play/p/MslCkBphl7q?v=gotip
- morelisp 4y agoBecause the bare value `nil` has no type. Typing it, e.g. `len([]int(nil))` works fine.
- nutate 4y agoThis is the most satisfying answer, but it doesn't make the implications less complex.
- morelisp 4y agoWhat implications did you have in mind? In languages with null and type inference, `var x = null` is probably not going to infer the type you want. In languages with function overloading (which is essentially the case for Go `len`), `f(null)` is going to be a compile-time error if multiple overloads are potentially null.
- nutate 4y agothe ability to cast nil as a zero length array means there's no difference between a function that returns a zero length array and a function that returns nil (assuming some casting process takes place). It could be a subtle and annoying bug to track down the difference.
- morelisp 4y agonil isn't cast to an empty slice (Go doesn't have casts, except maybe the new pointer-to-array syntax if you want to count that), nil is the default value of a slice, and that value is also empty. Other empty slices may be non-nil, because they may have capacity, or have been sliced out of another buffer, etc. Of the various legitimate issues around nil (box vs. unboxed, nil receivers, nilability of all pointers), this is the most not-actually-ever-an-issue.
- sacnoradhq 4y ago0. Only 2 of 3 values are necessary. 1. Can 1 value be unspecified? 2. Can 2 values be unspecified? 3. Can 3 values be unspecified? 4. Are conflicting values interpreted as intersection of ranges rather than union? Note 1-3. With 2 values, I believe x[:] is how to lift a sized array into a more generic slice type.
- masklinn 4y agoYour questions are really unclear. In the 2-parameters form `a[low:high]`, both values can be left out, defaulting to respectively 0 and len(a). In the 3-parameter form, only the leading value can be left out, defaulting to 0. I've no idea what (4) is asking about.
- klodolph 4y agoRegarding conflicting ranges— In a[i:j:k], 0 <= i, i <= j, j <= k, and k <= cap(a). If not, then the operation will panic.
- deleted 4y ago[deleted]
- tommodev 4y agoQuestion: what is the reason for the silent copy when append exceeds the original slice cap? It's a footgun avoided by reading the spec and (maybe) remembering it in practice, but it feels like it would be safer to throw a comp error and force the user to deal with it when a user is trying to exceed the cap of the underlying array? Alternative is defensively using len() and cap() for slice ops in which case error-ing out feels more ergonomic.
- sa46 4y agoErroring on appending to a slice would require checking every call to append for an error. I'd find it more surprising for append to error on resize since append implies a growable array.
- shakow 4y agoBecause you would not have any growable vector/list structure otherwise. The real problem is that Go merrily lets you have copy-and-append operation on a slice (good), subviews of a slice so that you can share subsets of the data without copying it (good), at the same time (very bad: any operation on either will lead to confusion). In most languages, subslicing gives you something of another type that can't be modified (or at least not accidentally). But in Go, if I call a function `do_smth_with_slice([]byte xs)`, there is no way for me to know whether this function expects `xs` to be mutable or not. I'm sure that e.g. a C++ function taking an std::view can do some forbidden magic to still modify the underlying data, but at least the original intent is made clear by the argument type.
- masklinn 4y ago> Question: what is the reason for the silent copy when append exceeds the original slice cap? Because Go slices play double duty as vectors. And that is the usual behaviour of a vector. And the issue is the opposite situation, when appending does not exceed the original slice cap. The entire point of the slice trick is to force a resize (and thus a copy) on append. > it feels like it would be safer to throw a comp error and force the user to deal with it when a user is trying to exceed the cap of the underlying array? It would be safer to have not confused slices and vectors, but half-adding that confusion sounds even worse, your suggestion would only keep the worst parts, and would require hand-rolling the rest every time.
- operator-name 4y agoSo this is the same as a[low:min(high, max)] ?
- guipsp 4y agoNo.
- yarg 4y agoThis to me is a glaring case of a poorly named variable. a[low:high:capacity] would be much easier to understand at first sight.
- masklinn 4y agoAs high is not len, max is not capacity, low gets subtracted from both.
- H4ZB7 4y agowhy do people write articles about go features? when PHP was in its prime, almost nobody wrote blogs explaining how they found some philosophy in PHP. there's a reason for that. when you see someone open their article explaining a language feature by talking of the implementation details or specific use cases, that's a language smell (of course all industrial PLs stink). ironically go is the only post 80s language that uses "memory safe" as a marketing point (even though they all are), yet go has the most memory unsafety of post 80s industrial languages. you can parse something and pass on a slice somewhere. if you mistakenly slice that slice with a bigger size - this incorrect size being the programmer bounds check error - you restore some of the original array that was supposed to be cut off and teh next operation working on that slice will thus modify or leak data: package main import "fmt" func main(){ a := [3]int{1,2,3} b := a[0:2] fmt.Println(b[1]) c := b[0:3] fmt.Println(c[2]) } $ go run a.go 2 3 the other example of memory unsafety in go being that modifying slices between threads can lead to actual memory corruption, not just simulated memory corruption as above the point here is that this footgun doesnt even have a real point outside of some insane performance argument. nobody would ever design something like this without massive cognitive dissonance (aside from industrial PLs, which just copy and modify the previous industrial PL, C in this case). all go's primitives are rigged like this with unintuitive behaviors. its amazing how much such a simple language with small scope can get wrong. and i expect nothing less from people who go around saying "zeroeth". DAY OF THE BOBCAT SOON
- preseinger 4y agocalling articles about language features and/or their implementations a "smell" is some pretty insane stuff the slice behavior you demonstrate there is well-defined by the language spec, it's totally memory safe, it doesn't demonstrate memory corruption or anything like that go is probably the most successful new language since java, if you don't like it that's fine, but it's nonsensical to call its design decisions "wrong"
- iainmerrick 4y agoI’m pretty sure the most successful language since Java is JavaScript. Of course it has even more bad design decisions than Go, but people don’t leap out to defend them quite so much.
- deleted 4y ago[deleted]